CNBH Acoustic Scale Wiki - Recent changes [en]

Auditory-Based Processing of Communication Sounds/Compressive Auditory Filtering

Rdp1 — Wed, 08 Sep 2010 17:25:51 GMT

Results

←Older revision		Revision as of 17:25, 8 September 2010
(4 intermediate revisions not shown.)
Line 10:		Line 10:
	'Zooming in' to look at another level of detail, in Chapter 3 the process of strobed temporal integration was investigated and placed on firmer theoretical ground. In doing this, it was hypothesised that the auditory images generated using strobed temporal integration should be more robust to noise than features generated from more simple, purely spectral models. This hypothesis was investigated in Chapter 4 by adapting the auditory features developed in Chapter 2.		'Zooming in' to look at another level of detail, in Chapter 3 the process of strobed temporal integration was investigated and placed on firmer theoretical ground. In doing this, it was hypothesised that the auditory images generated using strobed temporal integration should be more robust to noise than features generated from more simple, purely spectral models. This hypothesis was investigated in Chapter 4 by adapting the auditory features developed in Chapter 2.

-	Having modelled an observed large-scale behaviour of the auditory system, and then developed a particular model of the post-cochlear neural processing, in this chapter I '~~zoom~~ in' again to look in a finer ~~level of~~ detail at the behaviour of the cochlea, and in particular its response at very short timescales. The cochlea is perhaps one of the more well-understood components of the auditory system, and there is a wealth of data on the spectral shape of the human auditory filter, and the fine-timing properties of the mammalian cochlea.	+	Having modelled and observed the large-scale behaviour of the auditory system, and then developed a particular model of the post-cochlear neural processing, this chapter 'zooms in' again to look in finer detail at the behaviour of the cochlea, and in particular its response at very short timescales. The cochlea is perhaps one of the more well-understood components of the auditory system, and there is a wealth of data on the spectral shape of the human auditory filter, and the fine-timing properties of the mammalian cochlea.

-	The dynamic range of audio signals is ~~many~~ orders of magnitude larger than the dynamic range available to encode those signals in the auditory nerve. This means that the auditory system has to perform some sort of compression on the incoming signal in order to represent it effectively with a neural code. It is perhaps surprising to find that the auditory system ~~has~~ this compression within the auditory filter itself~~, using~~ mechanical feedback from the outer hair cells (OHCs) to dynamically modify the motion of the basilar membrane, and so the signal encoded by the inner hair cells. One important advantage of this approach is that it makes it possible to perform the dynamic range compression with an extremely fast time-constant. The auditory filter is able to compress the peaks of the waveform within a single cycle, and leave the zero-crossings unchanged.	+	The dynamic range of audio signals is orders of magnitude larger than the dynamic range available to encode those signals in the auditory nerve. This means that the auditory system has to perform some sort of compression on the incoming signal in order to represent it effectively with a neural code. It is perhaps surprising to find that the auditory system performs this compression within the auditory filter itself; it uses mechanical feedback from the outer hair cells (OHCs) to dynamically modify the motion of the basilar membrane, and so the signal encoded by the inner hair cells. One important advantage of this approach is that it makes it possible to perform the dynamic range compression with an extremely fast time-constant. The auditory filter is able to compress the peaks of the waveform within a single cycle, and leave the zero-crossings effectively unchanged.

	Models of auditory filtering attempt to describe mathematically the processing performed by the cochlea on an incoming sound, which ultimately leads to a neural response. They are a mathematical abstraction of the response of the complex physiological systems in the cochlea to stimulation by an incoming pressure wave. There exist a number of excellent descriptions of various parts of the history of these models, for example <citet r="lyon:1996"/> and <citet r="patterson:2003"/>. The introduction to this chapter briefly covers the major points of the various models, and introduces a set of increasingly more complex criteria that an auditory model must fulfil in order to accurately model the human auditory system.		Models of auditory filtering attempt to describe mathematically the processing performed by the cochlea on an incoming sound, which ultimately leads to a neural response. They are a mathematical abstraction of the response of the complex physiological systems in the cochlea to stimulation by an incoming pressure wave. There exist a number of excellent descriptions of various parts of the history of these models, for example <citet r="lyon:1996"/> and <citet r="patterson:2003"/>. The introduction to this chapter briefly covers the major points of the various models, and introduces a set of increasingly more complex criteria that an auditory model must fulfil in order to accurately model the human auditory system.
Line 188:		Line 188:

	===== Comparison to human pitch perception =====		===== Comparison to human pitch perception =====
-	~~In order to bear any relevance to human pitch perception, it is also necessary that this~~ normalised pitch strength ~~measure does a reasonable job of modelling human perception~~. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise <citep r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. Subjects compared IRN with different numbers of iterations to a tonal stimulus (256-iteration IRN) masked with noise. Subjects were asked to select the stimulus with the stronger pitch strength as the SNR of the noise-masked tonal stimulus was changed. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results from the perceptual ~~experiments~~ are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure ~~and~~ plotted in <figureRef name='irn_ps_model_results'/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]] [[Image:Pzfc ps 800Hz plot prop.eps\|thumb\|400px\|right\|<figure name='irn_ps_model_results'/> --- Predicted pitch strengths from the normalised pitch strength measure used in the experiments in this section. The results follow the same pattern as those reported in <citet r='patterson:1996a'/>, but the measure is slightly noisier. ]]. The ~~pitch strength~~ predictions made by the normalized pitch strength measure have the same form as the perceptual results reported in <citet r='patterson:1996a'/>. The curves predicted by this model are a little noisier than those from the model of <citet r='patterson:1996a'/>, but the model has the advantage of being normalised ~~so that~~ it ~~is easy~~ to compare ~~the~~ results across ~~different filterbank, which may have differing~~ output levels.	+	The normalised pitch-strength measured for IRN stimuli was compared with the pitch strength measured for IRN in perceptual experiments. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise <citep r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. Subjects compared IRN with different numbers of iterations to a tonal stimulus (256-iteration IRN) masked with noise. Subjects were asked to select the stimulus with the stronger pitch strength as the SNR of the noise-masked tonal stimulus was changed. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results from the perceptual experiment are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure are plotted in <figureRef name='irn_ps_model_results'/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]] [[Image:Pzfc ps 800Hz plot prop.eps\|thumb\|400px\|right\|<figure name='irn_ps_model_results'/> --- Predicted pitch strengths from the normalised pitch strength measure used in the experiments in this section. The results follow the same pattern as those reported in <citet r='patterson:1996a'/>, but the measure is slightly noisier. ]] The predictions made by the normalized pitch strength measure have the same form as the perceptual results reported in <citet r='patterson:1996a'/>. The curves predicted by this model are a little noisier than those from the model of <citet r='patterson:1996a'/>, but the model has the advantage of being normalised making it easier to compare results across filterbanks when their output levels differ.

	====Experiments====		====Experiments====
Line 219:		Line 219:
	\|}		\|}

-	~~This suggests~~ that the processing performed by the dcGC is fundamentally different to that performed by the PZFC for these stimuli. Since IRN is inherently noisy, I now move to using an alternative stimulus: a harmonic complex. This stimulus is used to test the effect of modifying various parameters related to the automatic gain control in the PZFC.	+	The results suggest that the processing performed by the dcGC is fundamentally different to that performed by the PZFC for these stimuli. Since IRN is inherently noisy, I now move to using an alternative stimulus: a harmonic complex. This stimulus is used to test the effect of modifying various parameters related to the automatic gain control in the PZFC.

	===== Harmonic complexes =====		===== Harmonic complexes =====

Auditory-Based Processing of Communication Sounds/Scale-shift Invariant Auditory Features

Tcw24 — Tue, 07 Sep 2010 07:08:50 GMT

Results:

←Older revision		Revision as of 07:08, 7 September 2010
(One intermediate revision not shown.)
Line 100:		Line 100:
	Performance of the HMM-based recogniser on the two sets of features was markedly different, leading to overall differences in recognition rate which were in general much larger than those due to the changes in the HMM parameters. Optimum performance, of 72.3% syllable accuracy on the MFCC features, was achieved with an HMM with 2 emitting states and with 4 components in the output distribution. This performance was achieved after 15 iterations of the training algorithm.		Performance of the HMM-based recogniser on the two sets of features was markedly different, leading to overall differences in recognition rate which were in general much larger than those due to the changes in the HMM parameters. Optimum performance, of 72.3% syllable accuracy on the MFCC features, was achieved with an HMM with 2 emitting states and with 4 components in the output distribution. This performance was achieved after 15 iterations of the training algorithm.

-	For the AIM features, the optimum recognition performance was 93.2% across all syllables using an HMM with 2 emitting states, 6 output distribution components and after 10 training iterations. However, <figureRef name='mfcc_vs_nap_performance'/> shows the overall syllable recognition rate for the MFCCs and AIM features respectively as a function of the various HMM parameters. For the MFCCs, the performance for all HMM parameters is between 64% and 72.3%. For the same set of parameters with the AIM features, performance is between 78% and 93.2%. Performance is consistently better for the AIM features across all tested HMM configurations. (In addition to the HMM configurations shown in this graph, odd-numbered values of the HMM parameters were also tried but, for simplicity of representation, they are not plotted here. These results cluster in the same way.) [[Image:Nap vs mfcc performance.pdf\|thumb\|300px\|right\|<figure name='mfcc_vs_nap_performance'/> --- Performance of HMMs with varying parameters, trained using AIM features (black) and MFCCs (red). The vertical axis shows overall recognition performance. The horizontal axis shows the number of HMM training iterations. Solid lines denote HMMs with 2 emitting states, dashed lines 4 emitting states and dotted lines 8 emitting states. Stars show performance with 2 Gaussian components in the output distribution, circles are for ~~four~~ components and plus symbols show performance with 6 components).]]	+	For the AIM features, the optimum recognition performance was 93.2% across all syllables using an HMM with 2 emitting states, 6 output distribution components and after 10 training iterations. However, <figureRef name='mfcc_vs_nap_performance'/> shows the overall syllable recognition rate for the MFCCs and AIM features respectively as a function of the various HMM parameters. For the MFCCs, the performance for all HMM parameters is between 64% and 72.3%. For the same set of parameters with the AIM features, performance is between 78% and 93.2%. Performance is consistently better for the AIM features across all tested HMM configurations. (In addition to the HMM configurations shown in this graph, odd-numbered values of the HMM parameters were also tried but, for simplicity of representation, they are not plotted here. These results cluster in the same way.) [[Image:Nap vs mfcc performance.pdf\|thumb\|300px\|right\|<figure name='mfcc_vs_nap_performance'/> --- Performance of HMMs with varying parameters, trained using AIM features (black) and MFCCs (red). The vertical axis shows overall recognition performance. The horizontal axis shows the number of HMM training iterations. Solid lines denote HMMs with 2 emitting states, dashed lines 4 emitting states and dotted lines 8 emitting states. Stars show performance with 2 Gaussian components in the output distribution, circles are for 4 components and plus symbols show performance with 6 components).]]

	For the AIM features, 92.8% accuracy was achieved with the 2 emitting state, 4 component HMM model after 15 training iterations. Since this performance is almost at the ceiling for the AIM features, and the same HMM parameters lead to optimal performance for the MFCC features, all further comparisons are made using these parameters. This allows for a direct comparison of the features using an otherwise identical recognition system.		For the AIM features, 92.8% accuracy was achieved with the 2 emitting state, 4 component HMM model after 15 training iterations. Since this performance is almost at the ceiling for the AIM features, and the same HMM parameters lead to optimal performance for the MFCC features, all further comparisons are made using these parameters. This allows for a direct comparison of the features using an otherwise identical recognition system.
Line 110:		Line 110:

	unlike the MFCC recogniser, performance along most of the spokes is near ceiling after optimisation. The worst performance, for the speaker with the longest VTL, was 66.5%, which compares with 3.8% in the MFCC case. There is a drop in performance at the extremes of VTL, although the drop is small in comparison to that seen in the MFCC case.		unlike the MFCC recogniser, performance along most of the spokes is near ceiling after optimisation. The worst performance, for the speaker with the longest VTL, was 66.5%, which compares with 3.8% in the MFCC case. There is a drop in performance at the extremes of VTL, although the drop is small in comparison to that seen in the MFCC case.
-
-
-

	==== Comparison with standard VTLN ====		==== Comparison with standard VTLN ====
Line 154:		Line 151:

	==== Conclusions ====		==== Conclusions ====
-	~~The scale~~-shift invariant Gaussian features derived from AIM spectral profiles proved ~~extremely~~ effective for a small-scale size-invariant syllable recognition task. This result demonstrates that size-invariant features are useful in allowing recognition systems to generalise over a range of speakers. The ~~fact that the AIM~~ features consistently outperform ~~the~~ MFCC features on the ~~syllable~~ recognition ~~task shows~~ that the ~~exact~~ parameters of the HMM ~~have~~ only a ~~secondary~~ effect to the feature ~~change~~.	+	Scale-shift invariant Gaussian features derived from AIM spectral profiles have proved effective for a small-scale size-invariant syllable recognition task. This result demonstrates that size-invariant features are useful in allowing recognition systems to generalise over a range of speakers. The scale-shift invariant features derived from AIM consistently outperform standard MFCC features. This ability to outperform the MFCCs appears to be an intrinsic property of the features themselves, rather than an effect of the recognition system being used, evidenced by the observation that changing the parameters of the HMM recognition system has only a minor effect compared with that of changing the feature itself.
		+
		+	However, when optimal vocal tract length normalisation is performed on the MFCCs, their performance is extremely good - becoming better than the performance with the scale shift invariant features. Such performance could be achieved in practice if the recognition system was able to correctly identify the correct scaling factor for every speaker. With current speech recognition systems, performing vocal tract length normalisation involves pre-calculating features for a range of scalings of the input sound <citep r="welling:2002"/> which adds complexity to the recognition system. As they stand, the scale-shit invariant AIM features are approximately 30 times slower to compute than MFCC features with vocal tract length normalisation. It is not clear how many discrete vocal tract warping values would have to be used to generate the full range of inputs for a vocal tract length normalising speech recognition system, but it is likely to be of the same order of magnitude, so for short segments of speech, where it is not possible to TODO (put this back-of-the-envelope calculation on firmer ground, and put it in an earlier section).
		+
		+
		+

-	It is interesting to note that the performance of the HMM-based recogniser with the MFCCs is considerably improved when the system is trained on a wide range of speakers - suggesting that the utility of VTL-invariant features may be limited to the case where there is only data from a single training speaker.

-	When the frequency axis is warped optimally, it is possible to achieve performance which exceeds that of the AIM features when using the MFCC features. However, such performance could only be achieved in practice if the recognition system was able to correctly identify the correct scaling factor for every speaker. With current systems this would involve pre-calculating features for a range of scalings of the input sound <citep r="welling:2002"/>. The

	There is still some sensitivity to change in VTL in the scale-shift invariant feature vectors. Since it affects only the extreme VTL conditions, it seems likely that it is due to edge effects at the Gaussian fitting stage. That is, when a formant occurs near the edge of the spectrum, the tail of the Gaussian used to fit the formant prevents it from shifting sufficiently to centre the Gaussian on the formant. If this proves to be the reason, it suggests that performance is not limited by the underlying auditory representation but rather by a limitation in the feature extraction process.		There is still some sensitivity to change in VTL in the scale-shift invariant feature vectors. Since it affects only the extreme VTL conditions, it seems likely that it is due to edge effects at the Gaussian fitting stage. That is, when a formant occurs near the edge of the spectrum, the tail of the Gaussian used to fit the formant prevents it from shifting sufficiently to centre the Gaussian on the formant. If this proves to be the reason, it suggests that performance is not limited by the underlying auditory representation but rather by a limitation in the feature extraction process.
		+
		+	It is interesting to note that the performance of the HMM-based recogniser with the MFCCs is considerably improved when the system is trained on a wide range of speakers - suggesting that the utility of VTL-invariant features may be limited to the case where there is only data from a single training speaker.

	Based on this small-scale set of recognition experiments, there are many possibilities for future work. With the AIM-C software that I developed and the processing pipeline that Jess Monaghan and I worked on, it is now possible to easily create features from large databases of sounds. For these recognition experiments, the Amazon EC2 cloud computing service was used to generate the features quickly, as it is simple to parallelise the task over many compute cores.		Based on this small-scale set of recognition experiments, there are many possibilities for future work. With the AIM-C software that I developed and the processing pipeline that Jess Monaghan and I worked on, it is now possible to easily create features from large databases of sounds. For these recognition experiments, the Amazon EC2 cloud computing service was used to generate the features quickly, as it is simple to parallelise the task over many compute cores.

-	The simplest addition to the system would be to test the Gaussian features with other filterbanks, including compressive filters. I have carried out some preliminary studies in this area, and with some modification to the Gaussian fitting procedure, similar features can be calculated using the dcGC and PZFC filterbanks. Beyond the simple database of scaled syllables, there is potential to extend the use of AIM Gaussian features to more general speech recognition tasks, such as the TIMIT database.

-	The features used in the study described above are computed simply from the output of the NAP stage of AIM. In the following chapter, the process of strobed temporal integration is investigated, and In Chapter 4, the feature generation system is extended to create features from ~~the~~ spectral ~~profile~~ of the AIM stabilised auditory image~~. By using the SAI, the features generated are found to be more robust to interfering noise~~.	+	The features used in the study described above are computed simply from the output of the NAP stage of AIM. In the following chapter, the process of strobed temporal integration is investigated, and In Chapter 4, the feature generation system is extended to create features from various spectral profiles of the AIM stabilised auditory image.

	=== Notes ===		=== Notes ===

Auditory-Based Processing of Communication Sounds/Strobes and Stabilised Auditory Images

Tcw24 — Tue, 07 Sep 2010 06:42:36 GMT

←Older revision		Revision as of 06:42, 7 September 2010
Line 5:		Line 5:

	{{Underconstruction}}</noinclude>		{{Underconstruction}}</noinclude>
		+
		+	The last chapter introduced a feature representation based on the smoothed output from a simulation of the cochlea and a simple hair-cell model. The signal was temporally averaged over a short window by means of a low-pass filter. Strobed temporal integration, leading to a stabilised auditory image (SAI), is an alternative, more complex, system for processing the signal leaving the cochlea. A set of strobe points are identified in each channel of the filterbank output, and these strobe points act as triggers for a temporal integration process in which shifted copies of the signal are overlaid on one another.
		+
		+	Since strobe points tend to occur at or near the pulses in a pulse-resonance sound, representing a signal containing a pulse-resonance sound as an SAI will tend to accentuate the periodic pulse-resonance signal relative to any background noise. Noise-robustness is an extremely useful property in any machine hearing system, and it would be beneficial if it were possible to make use of some of the inherent noise-robustness in the SAI. In order to do this, it is first necessary to understand the optimise the process by which strobe points are identified in a signal.
		+
		+	In this chapter, various existing systems for performing strobed temporal integration are assessed and compared, and a new model is proposed based on a simple criterion for optimal strobe generation. The goal is to create a good stabilised auditory image which can be used as the basis for a noise-robust machine hearing system.

	=== Strobe-finding in AIM ===		=== Strobe-finding in AIM ===
Line 295:		Line 301:
	\|-		\|-
	\| \|\| 0.94 \|\| 0.90 \|\| 1.01 \|\| 0.91		\| \|\| 0.94 \|\| 0.90 \|\| 1.01 \|\| 0.91
		+	\|-
	! ''lyon''		! ''lyon''
	\| 0.0% \|\| 0.0% \|\| 0.1% \|\| 0.1%		\| 0.0% \|\| 0.0% \|\| 0.1% \|\| 0.1%
	\|-		\|-
	\| \|\| 4.23 \|\| 4.23 \|\| 4.23 \|\| 4.23		\| \|\| 4.23 \|\| 4.23 \|\| 4.23 \|\| 4.23
		+	\|-
	\|}		\|}

Line 342:		Line 350:
	\|-		\|-
	\| \|\| 0.94 \|\| 0.92 \|\| 1.01 \|\| 0.91		\| \|\| 0.94 \|\| 0.92 \|\| 1.01 \|\| 0.91
		+	\|-
	! ''lyon''		! ''lyon''
	\| 5.5% \|\| 5.1% \|\| 5.4% \|\| 5.5%		\| 5.5% \|\| 5.1% \|\| 5.4% \|\| 5.5%
	\|-		\|-
	\| \|\| 3.99 \|\| 3.99 \|\| 3.99 \|\| 3.99		\| \|\| 3.99 \|\| 3.99 \|\| 3.99 \|\| 3.99
		+	\|-
	\|}		\|}

Line 389:		Line 399:
	\|-		\|-
	\| \|\| 1.24 \|\| 0.91 \|\| 1.10 \|\| 0.94		\| \|\| 1.24 \|\| 0.91 \|\| 1.10 \|\| 0.94
		+	\|-
	! ''lyon''		! ''lyon''
	\| 0.0% \|\| 0.0% \|\| 0.1% \|\| 0.1%		\| 0.0% \|\| 0.0% \|\| 0.1% \|\| 0.1%
	\|-		\|-
	\| \|\| 4.23 \|\| 4.23 \|\| 4.23 \|\|4.23		\| \|\| 4.23 \|\| 4.23 \|\| 4.23 \|\|4.23
		+	\|-
	\|}		\|}

Line 460:		Line 472:
	An alternative strobe detection system was also introduced that compares the NAP to the impulse response of the filterbank at every time step. While this system is extremely effective at 'back-projecting' to find the original strobe time accurately, it is not at all computationally efficient and so cannot be used in a complete machine hearing system. Further work would also be required in tuning the parameters of this system in order to make it work correctly with the PZFC filterbank.		An alternative strobe detection system was also introduced that compares the NAP to the impulse response of the filterbank at every time step. While this system is extremely effective at 'back-projecting' to find the original strobe time accurately, it is not at all computationally efficient and so cannot be used in a complete machine hearing system. Further work would also be required in tuning the parameters of this system in order to make it work correctly with the PZFC filterbank.

-	~~Having investigated the properties of strobed temporal integration in the auditory image model, in~~ the next chapter, the ~~original syllable recognition system described~~ in Chapter 2 ~~is extended to make use of features~~ computed from the stabilized auditory image~~. The hypothesis is that~~, ~~due~~ to the strobed temporal integration process, these features will be more noise-robust than those generated from the NAP alone.	+	In the next chapter, the features developed in Chapter 2 are computed from the stabilized auditory image, rather than from a simple filterbank output. Due to the strobed temporal integration process, we hypothesise these features will be more noise-robust than those generated from the NAP alone. The syllable recognition system developed in chapter 2 provides an excellent test-bed for the alternative features.

	<noinclude><bibliography t="== Bibliography =="/></noinclude>		<noinclude><bibliography t="== Bibliography =="/></noinclude>

Auditory-Based Processing of Communication Sounds/Compressive Auditory Filtering

Tcw24 — Tue, 07 Sep 2010 06:12:00 GMT

Comparison to human pitch perception:

Auditory-Based Processing of Communication Sounds/Scale-shift Invariant Auditory Features

Tcw24 — Tue, 07 Sep 2010 02:42:16 GMT

Spectral resolution:

←Older revision		Revision as of 02:42, 7 September 2010
(One intermediate revision not shown.)
Line 100:		Line 100:
	Performance of the HMM-based recogniser on the two sets of features was markedly different, leading to overall differences in recognition rate which were in general much larger than those due to the changes in the HMM parameters. Optimum performance, of 72.3% syllable accuracy on the MFCC features, was achieved with an HMM with 2 emitting states and with 4 components in the output distribution. This performance was achieved after 15 iterations of the training algorithm.		Performance of the HMM-based recogniser on the two sets of features was markedly different, leading to overall differences in recognition rate which were in general much larger than those due to the changes in the HMM parameters. Optimum performance, of 72.3% syllable accuracy on the MFCC features, was achieved with an HMM with 2 emitting states and with 4 components in the output distribution. This performance was achieved after 15 iterations of the training algorithm.

-	For the AIM features, the optimum recognition performance was 93.2% across all syllables using an HMM with 2 emitting states, 6 output distribution components and after 10 training iterations. However, <figureRef name='~~mfcc_performance~~'/> ~~and <figureRef name='aimfeat_performance'/> show~~ the overall syllable recognition rate for the MFCCs and AIM features respectively as a function of the various HMM parameters. For the MFCCs, the performance for all HMM parameters ~~(apart from the 1-emitting-state HMM with a single Gaussian in the output distribution) clusters~~ between 64% and 72.3%. For the same set of parameters with the AIM features, performance ~~clusters~~ between 78% and 93.2% ~~accuracy~~. Performance is consistently better for the AIM features, across all tested HMM configurations.	+	For the AIM features, the optimum recognition performance was 93.2% across all syllables using an HMM with 2 emitting states, 6 output distribution components and after 10 training iterations. However, <figureRef name='mfcc_vs_nap_performance'/> shows the overall syllable recognition rate for the MFCCs and AIM features respectively as a function of the various HMM parameters. For the MFCCs, the performance for all HMM parameters is between 64% and 72.3%. For the same set of parameters with the AIM features, performance is between 78% and 93.2%. Performance is consistently better for the AIM features across all tested HMM configurations. (In addition to the HMM configurations shown in this graph, odd-numbered values of the HMM parameters were also tried but, for simplicity of representation, they are not plotted here. These results cluster in the same way.) [[Image:Nap vs mfcc performance.pdf\|thumb\|300px\|right\|<figure name='mfcc_vs_nap_performance'/> --- Performance of HMMs with varying parameters, trained using AIM features (black) and MFCCs (red). The vertical axis shows overall recognition performance. The horizontal axis shows the number of HMM training iterations. Solid lines denote HMMs with 2 emitting states, dashed lines 4 emitting states and dotted lines 8 emitting states. Stars show performance with 2 Gaussian components in the output distribution, circles are for four components and plus symbols show performance with 6 components).]]
-		+
-	For the AIM features, 92.8% accuracy was achieved with the 2-emitting-state, 4-component HMM model after 15 training iterations. Since this performance is almost at the ceiling for the AIM features, and the same ~~model provides~~ optimal performance for the MFCC features, all further comparisons are made using these ~~HMM~~ parameters. This allows for a ~~more~~ direct comparison of the features.	+	For the AIM features, 92.8% accuracy was achieved with the 2 emitting state, 4 component HMM model after 15 training iterations. Since this performance is almost at the ceiling for the AIM features, and the same HMM parameters lead to optimal performance for the MFCC features, all further comparisons are made using these parameters. This allows for a direct comparison of the features using an otherwise identical recognition system.

	[[Image:MFCC Results 2.png\|thumb\|300px\|right\|<figure name='mfcc_results_2'/> --- Recognition results for the MFCC features for a 2-emitting-sate HMM with 4 output distribution components after 15 training iterations.]]		[[Image:MFCC Results 2.png\|thumb\|300px\|right\|<figure name='mfcc_results_2'/> --- Recognition results for the MFCC features for a 2-emitting-sate HMM with 4 output distribution components after 15 training iterations.]]
	With these parameters, recognition performance with the MFCC feature vectors was (TODO:check numbers here) ??% correct on the training voices, as shown in <figureRef name='mfcc_results_2'/>. Performance held up well along the spokes where VTL does not vary much from that of the reference speaker. This subset of the results illustrates the standard finding that MFCCs are robust to changes in GPR, primarily because the process of extracting MFCCs eliminates most of the GPR information from the features. As VTL varies further from the training values, performance degrades rapidly, particularly on the spokes with large VTL change, where recognition falls close to a minimum of ??% for the extreme VTL values. This provides a practical demonstration of the known lack of robustness to changes in VTL associated with the lack of scale-shift covariance in MFCCs.		With these parameters, recognition performance with the MFCC feature vectors was (TODO:check numbers here) ??% correct on the training voices, as shown in <figureRef name='mfcc_results_2'/>. Performance held up well along the spokes where VTL does not vary much from that of the reference speaker. This subset of the results illustrates the standard finding that MFCCs are robust to changes in GPR, primarily because the process of extracting MFCCs eliminates most of the GPR information from the features. As VTL varies further from the training values, performance degrades rapidly, particularly on the spokes with large VTL change, where recognition falls close to a minimum of ??% for the extreme VTL values. This provides a practical demonstration of the known lack of robustness to changes in VTL associated with the lack of scale-shift covariance in MFCCs.

-	Recognition performance with the auditory feature vectors was ??% ~~correct~~ on the training voices. As with the MFCCs, performance remained high along the spokes associated with major changes in GPR. However, performance along all of the spokes also remains near ceiling, and only drops off significantly at the extremes of VTL. Performance at the longest VTL was ??% and performance at the shortest was ??%.	+	Recognition performance with the auditory feature vectors was 100% on the training voices. As with the MFCCs, performance remained high along the spokes associated with major changes in GPR. However, performance along all of the spokes also remains near ceiling, and only drops off significantly at the extremes of VTL. Performance at the longest VTL was 68.1% and performance at the shortest was 73.0%.
-		+
-		+
-	~~===== Spectral resolution =====~~	+

-	~~TODO (A further experiment was carried out using MFCCs produced from a 200 channel mel filterbank to check that~~ the performance of the ~~recogniser was not being limited by a lack of spectral resolution~~. The performance ~~using these MFCCs was 67.7%~~ for the ~~initial topology~~ with ~~three emitting states and 73~~.3% ~~using~~ the ~~best topology from~~ the ~~previous experiments~~, ~~indicating~~ that ~~26-channel resolution was not a serious limitation~~.)	+	unlike the MFCC recogniser, performance along most of the spokes is near ceiling after optimisation. The worst performance, for the speaker with the longest VTL, was 66.5%, which compares with 3.8% in the MFCC case. There is a drop in performance at the extremes of VTL, although the drop is small in comparison to that seen in the MFCC case.

-	~~===== Scale-shift invariant features =====~~
-	As with the MFCC features, a `baseline' recogniser was first used. This was based on an HMM with a topology with three emitting states and a single Gaussian output distribution for each state. Recognition performance using this HMM topology was 84.6% over all speakers. Once again, the HMM topology was varied to find the best performance. This was found when using an HMM with two emitting states and a mixture of four Gaussians in the output distribution. After optimisation of the topology and nine iterations of the training algorithm, performance rose to 90.7%. This level of performance is well above the 73.5% achieved after similar optimisation with the MFCC feature vectors.

-	~~[[Image:AIM-Features Results.png\|thumb\|300px\|right\|<figure name='gmm_results'/> --- Recognition results for the AIM GMM features using a gammatone filterbank.]]~~

-	Performance obtained using this topology for the individual speakers across the GPR-VTL plane, is shown in <figureRef name='gmm_results'/>. As with the MFCC recogniser, performance is best when only GPR varies. However, unlike the MFCC recogniser, performance along most of the spokes is near ceiling after optimisation. The worst performance, for the speaker with the longest VTL, was 66.5%, which compares with 3.8% in the MFCC case. There is a drop in performance at the extremes of VTL, although the drop is small in comparison to that seen in the MFCC case.

	==== Comparison with standard VTLN ====		==== Comparison with standard VTLN ====

Image:Nap vs mfcc performance.pdf

Tcw24 — Tue, 07 Sep 2010 01:29:08 GMT

uploaded "[[Image:Nap vs mfcc performance.pdf]]" Performance of HMMs with varying parameters, trained using AIM features (black) and MFCCs (red). The vertical axis shows overall recognition performance. The horizontal axis shows the number of HMM training iterations. Solid lines denote HMMs with 2 emitt

New page

Auditory-Based Processing of Communication Sounds/Compressive Auditory Filtering

Tcw24 — Tue, 07 Sep 2010 00:35:09 GMT

Comparison to human pitch perception:

Show changes

Image:Pat Yost Human Results.png

Tcw24 — Sat, 04 Sep 2010 23:56:57 GMT

uploaded "[[Image:Pat Yost Human Results.png]]" Psychometric functions for the perception of the pitch strength of a tonal stimulus in noise compared with IRN with various parameters. Figure 1 from <citet r='patterson:1996a'/>.

New page

Image:Pat Yost Predictions.png

Tcw24 — Sat, 04 Sep 2010 23:55:49 GMT

←Older revision		Revision as of 23:55, 4 September 2010
Line 1:		Line 1:
-	~~Psychometric~~ functions for pitch strength of IRN. Figure 1 from Patterson, Handel, Yost and Datta, 1996.	+	Predicted psychometric functions for pitch strength of an IRN stimulus compared with an tonal stimulus masked with noise. Figure 5 from Patterson, Handel, Yost and Datta, 1996.

Auditory-Based Processing of Communication Sounds/Compressive Auditory Filtering

Tcw24 — Sat, 04 Sep 2010 23:43:14 GMT

Comparison to human pitch perception:

←Older revision		Revision as of 23:43, 4 September 2010
Line 176:		Line 176:

	===== Comparison to human pitch perception =====		===== Comparison to human pitch perception =====
-	In order to bear any relevance to human pitch perception, it is also necessary that this normalised pitch strength measure does a good job of modelling human perception. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise. TODO: add more description of the experiments here see <citet r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure and plotted in <figureRef name=' '/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]]	+	In order to bear any relevance to human pitch perception, it is also necessary that this normalised pitch strength measure does a good job of modelling human perception. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise. TODO: add more description of the experiments here see <citet r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure and plotted in <figureRef name='irn_ps_model_results'/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]] [[Image:Pzfc ps 800Hz plot prop.eps\|thumb\|400px\|right\|<figure name='irn_ps_model_results'/> --- Predicted pitch strengths from the normalised pitch strength measure used in the experiments in this section. The results follow the same pattern as those reported in <citet r='patterson:1996a'/>, but the measure is slightly noisier. ]]. The pitch strength predictions made by the normalized pitch strength measure have the same form as the perceptual results reported in <citet r='patterson:1996a'/. The curves predicted by this model are a little noisier than those from the model of <citet r='patterson:1996a'/>, but the model has the advantage of being normalised so that it is easy to compare the results across different filterbank, which may have differing output levels.

	====Experiments====		====Experiments====

←Older revision		Revision as of 06:12, 7 September 2010
(One intermediate revision not shown.)
Line 10:		Line 10:
	'Zooming in' to look at another level of detail, in Chapter 3 the process of strobed temporal integration was investigated and placed on firmer theoretical ground. In doing this, it was hypothesised that the auditory images generated using strobed temporal integration should be more robust to noise than features generated from more simple, purely spectral models. This hypothesis was investigated in Chapter 4 by adapting the auditory features developed in Chapter 2.		'Zooming in' to look at another level of detail, in Chapter 3 the process of strobed temporal integration was investigated and placed on firmer theoretical ground. In doing this, it was hypothesised that the auditory images generated using strobed temporal integration should be more robust to noise than features generated from more simple, purely spectral models. This hypothesis was investigated in Chapter 4 by adapting the auditory features developed in Chapter 2.

-	Having modelled an observed large-scale behaviour of the auditory system, and then developed a particular model of the post-cochlear neural processing, in this chapter I 'zoom in' again to look in a finer level of detail at the behaviour of the cochlea, and in particular its response at very short timescales. The cochlea is perhaps one of the more well-understood components of the auditory system, and there is a wealth of data on the spectral shape of the human auditory filter, and the fine-timing properties of the mammalian cochlea. ~~If we are to accurately model the human auditory system, it is clear that~~	+	Having modelled an observed large-scale behaviour of the auditory system, and then developed a particular model of the post-cochlear neural processing, in this chapter I 'zoom in' again to look in a finer level of detail at the behaviour of the cochlea, and in particular its response at very short timescales. The cochlea is perhaps one of the more well-understood components of the auditory system, and there is a wealth of data on the spectral shape of the human auditory filter, and the fine-timing properties of the mammalian cochlea.

-	The dynamic range of audio signals is many orders of magnitude larger than the dynamic range available to encode those signals in the auditory nerve ~~(TODO: good reference for this - Moore's book, perhaps)~~. This means that the auditory system has to perform some sort of compression on the incoming signal in order to represent it effectively with a neural code. It is perhaps surprising to find that the auditory system has this compression within the auditory filter itself, using mechanical feedback from the outer hair cells (OHCs) to dynamically modify the motion of the basilar membrane, and so the signal encoded by the inner hair cells ~~(TODO: citation)~~. One important advantage of this approach is that it makes it possible to perform the dynamic range compression with an extremely fast time-constant. The auditory filter is able to compress the peaks of the waveform within a single cycle, and leave the zero-crossings unchanged ~~(TODO: citation)~~.	+	The dynamic range of audio signals is many orders of magnitude larger than the dynamic range available to encode those signals in the auditory nerve. This means that the auditory system has to perform some sort of compression on the incoming signal in order to represent it effectively with a neural code. It is perhaps surprising to find that the auditory system has this compression within the auditory filter itself, using mechanical feedback from the outer hair cells (OHCs) to dynamically modify the motion of the basilar membrane, and so the signal encoded by the inner hair cells. One important advantage of this approach is that it makes it possible to perform the dynamic range compression with an extremely fast time-constant. The auditory filter is able to compress the peaks of the waveform within a single cycle, and leave the zero-crossings unchanged.

	Models of auditory filtering attempt to describe mathematically the processing performed by the cochlea on an incoming sound, which ultimately leads to a neural response. They are a mathematical abstraction of the response of the complex physiological systems in the cochlea to stimulation by an incoming pressure wave. There exist a number of excellent descriptions of various parts of the history of these models, for example <citet r="lyon:1996"/> and <citet r="patterson:2003"/>. The introduction to this chapter briefly covers the major points of the various models, and introduces a set of increasingly more complex criteria that an auditory model must fulfil in order to accurately model the human auditory system.		Models of auditory filtering attempt to describe mathematically the processing performed by the cochlea on an incoming sound, which ultimately leads to a neural response. They are a mathematical abstraction of the response of the complex physiological systems in the cochlea to stimulation by an incoming pressure wave. There exist a number of excellent descriptions of various parts of the history of these models, for example <citet r="lyon:1996"/> and <citet r="patterson:2003"/>. The introduction to this chapter briefly covers the major points of the various models, and introduces a set of increasingly more complex criteria that an auditory model must fulfil in order to accurately model the human auditory system.
Line 18:		Line 18:
	An important feature of the more recent models of the auditory filter is their ability to deal with dynamic compression performed by the cochlea. In this chapter, two recent models of the auditory filter that perform dynamic compression are discussed and analysed. The models are the dynamic, compressive gammachirp (dcGC) <citepL r="irino:2006,irino:2007"/> and the pole-zero filter cascade (PZFC) <citep r="lyon:2010"/>. The two filter models are compared in their response to a number of test stimuli to assess the response of the dynamic, time-varying compression that they both implement.		An important feature of the more recent models of the auditory filter is their ability to deal with dynamic compression performed by the cochlea. In this chapter, two recent models of the auditory filter that perform dynamic compression are discussed and analysed. The models are the dynamic, compressive gammachirp (dcGC) <citepL r="irino:2006,irino:2007"/> and the pole-zero filter cascade (PZFC) <citep r="lyon:2010"/>. The two filter models are compared in their response to a number of test stimuli to assess the response of the dynamic, time-varying compression that they both implement.

-	The studies presented in this chapter provide some evidence that dynamic, within-cycle, compression is a feature of auditory processing which is important for correctly modelling human perception of certain stimuli. The stimuli used in this chapter are iterated rippled noise (IRN) and high-pass filtered harmonic complexes in which the fundamental and lower harmonics of the stimulus are not present. ~~Both of these~~ stimuli	+	The studies presented in this chapter provide some evidence that dynamic, within-cycle, compression is a feature of auditory processing which is important for correctly modelling human perception of certain stimuli. The stimuli used in this chapter are iterated rippled noise (IRN) and high-pass filtered harmonic complexes in which the fundamental and lower harmonics of the stimulus are not present. These stimuli illustrate well the ability of the auditory system to process temporal regularity in a signal despite the lack of a strong fundamental harmonic, and thus provide a good test for temporal models of audition.

-	~~The study in this~~ chapter does not directly address the problem of whether compressive filtering is a crucial element for a good machine-hearing system, but rather looks at the application of compressive filtering to a particular class of stimuli. ~~The detailed nature~~ of ~~the~~	+	This chapter does not directly address the problem of whether compressive filtering is a crucial element for a good machine-hearing system, but rather looks at the application of compressive filtering to a particular class of stimuli, the correct representation of which is required in a system which accurately models human auditory perception. This is an incremental step towards understanding exactly which aspects of human auditory processing are necessary for effective machine hearing.

	=== A short history of models of auditory filtering ===		=== A short history of models of auditory filtering ===
Line 188:		Line 188:

	===== Comparison to human pitch perception =====		===== Comparison to human pitch perception =====
-	In order to bear any relevance to human pitch perception, it is also necessary that this normalised pitch strength measure does a reasonable job of modelling human perception. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise~~. Subjects compared TODO~~ <~~citet~~ r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results from the perceptual experiments are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure and plotted in <figureRef name='irn_ps_model_results'/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]] [[Image:Pzfc ps 800Hz plot prop.eps\|thumb\|400px\|right\|<figure name='irn_ps_model_results'/> --- Predicted pitch strengths from the normalised pitch strength measure used in the experiments in this section. The results follow the same pattern as those reported in <citet r='patterson:1996a'/>, but the measure is slightly noisier. ]]. The pitch strength predictions made by the normalized pitch strength measure have the same form as the perceptual results reported in <citet r='patterson:1996a'/. The curves predicted by this model are a little noisier than those from the model of <citet r='patterson:1996a'/>, but the model has the advantage of being normalised so that it is easy to compare the results across different filterbank, which may have differing output levels.	+	In order to bear any relevance to human pitch perception, it is also necessary that this normalised pitch strength measure does a reasonable job of modelling human perception. <citet r='Patterson:1996a'/> performed a series of experiments on the human perception of IRN by comparing the pitch strength of IRN stimuli and tonal stimuli masked with noise <citep r='Yost:1996, Yost:1998, Patterson:2000, Handel:2000'/>. Subjects compared IRN with different numbers of iterations to a tonal stimulus (256-iteration IRN) masked with noise. Subjects were asked to select the stimulus with the stronger pitch strength as the SNR of the noise-masked tonal stimulus was changed. The pitch strength measure described above was used to model the data of <citet r='Patterson:1996a'/>. The original results from the perceptual experiments are plotted in <figureRef name='pat_yost_predictions '/>, and the results using the current measure and plotted in <figureRef name='irn_ps_model_results'/>. [[Image:Pat Yost Predictions.png\|thumb\|400px\|right\|<figure name='pat_yost_predictions'/> --- Pitch strength predictions for the perception of IRN in noise, from <citet r='patterson:1996a'/>.]] [[Image:Pzfc ps 800Hz plot prop.eps\|thumb\|400px\|right\|<figure name='irn_ps_model_results'/> --- Predicted pitch strengths from the normalised pitch strength measure used in the experiments in this section. The results follow the same pattern as those reported in <citet r='patterson:1996a'/>, but the measure is slightly noisier. ]]. The pitch strength predictions made by the normalized pitch strength measure have the same form as the perceptual results reported in <citet r='patterson:1996a'/>. The curves predicted by this model are a little noisier than those from the model of <citet r='patterson:1996a'/>, but the model has the advantage of being normalised so that it is easy to compare the results across different filterbank, which may have differing output levels.

	====Experiments====		====Experiments====
Line 316:		Line 316:
	One interesting approach to this problem would be to modify the AGC of the PZFC so that its architecture was more like that of the dcGC. The AGC activity could be shifted so that the state of an AGC stage which takes input from one frequency is used to affect the PZFC filter stage at a lower frequency. This would align the PZFC AGC architecture more closely with models of compression in the cochlea.		One interesting approach to this problem would be to modify the AGC of the PZFC so that its architecture was more like that of the dcGC. The AGC activity could be shifted so that the state of an AGC stage which takes input from one frequency is used to affect the PZFC filter stage at a lower frequency. This would align the PZFC AGC architecture more closely with models of compression in the cochlea.

-	The experiments presented in this chapter do not, on their own, justify the use of a compressive filterbank in a machine hearing system. However, they provide ~~some~~ insight into the ~~behaviour~~ of compressive filterbanks in	+	The experiments presented in this chapter do not, on their own, justify the use of a compressive filterbank in a machine hearing system. However, they provide an insight into the utility of compressive filterbanks in the processing of stimuli in which the temporal fine structure is important.

-	A potential continuation of this work would be to use the compressive filterbanks described above in combination with the features generated from the Gaussian mixture model used in the earlier chapters of this thesis. Preliminary experiments to this end, which directly swapped the gammatone filterbank for the PZFC filterbank, led to recognition results which were significantly worse than the results gained with the gammatone filterbank~~. (TODO: find the ball-park percentages for this and put them in here)~~. However the exact parameters of the Gaussian fitting procedure were tuned to the output of a simple gammatone filterbank and these parameters were not modified in the initial experiments. The tuning of these parameters for use with the PZFC and dcGC filterbanks, ideally using a complete search of the parameter space for the Gaussian fitting system, is another potential direction for future work.	+	A potential continuation of this work would be to use the compressive filterbanks described above in combination with the features generated from the Gaussian mixture model used in the earlier chapters of this thesis. Preliminary experiments to this end, which directly swapped the gammatone filterbank for the PZFC filterbank, led to recognition results which were significantly worse than the results gained with the gammatone filterbank. However the exact parameters of the Gaussian fitting procedure were tuned to the output of a simple gammatone filterbank and these parameters were not modified in the initial experiments. The tuning of these parameters for use with the PZFC and dcGC filterbanks, ideally using a complete search of the parameter space for the Gaussian fitting system, is another potential direction for future work.

	While the modification of the PZFC AGC circuits, and the evaluation of compressive filterbanks in a speech recognition task are both fruitful potential avenues of further study, at this stage, the opportunity presented itself to work with auditory features at a much larger scale. The research team of Dick Lyon at Google had been investigating the use of MFCC features in a large-scale sound effects recognition task, making use of the PAMIR machine learning system which is optimized for use on large datasets. The team was working to extend the model to work with features generated from a version of the auditory image; I joined the team for an internship, working with them on the evaluation of the auditory features within the sound effects ranking task.		While the modification of the PZFC AGC circuits, and the evaluation of compressive filterbanks in a speech recognition task are both fruitful potential avenues of further study, at this stage, the opportunity presented itself to work with auditory features at a much larger scale. The research team of Dick Lyon at Google had been investigating the use of MFCC features in a large-scale sound effects recognition task, making use of the PAMIR machine learning system which is optimized for use on large datasets. The team was working to extend the model to work with features generated from a version of the auditory image; I joined the team for an internship, working with them on the evaluation of the auditory features within the sound effects ranking task.