1.1. Exemplars store fine phonetic details

Phonologists have long attempted to describe and model how words are represented in the mental lexicon. One of the most enduring proposals is that words are stored as abstract, formal representations, composed of strings of contrastive segments, a principle borrowed from generative phonology (Chomsky & Halle, 1968) and applied to models of speech perception involving normalization (Gerstman, 1968; Tranmüller, 1981, etc.). Contemporaneous to these developments, phoneticians were becoming aware of the vast acoustic variability of the speech signal. Liberman et al. (1967) demonstrated the variability of particular phonemes produced by speakers due to coarticulatory effects, while Stevens (1972) and Klatt (1979) noted the variability due to idiosyncrasies of a speaker’s voice, gender, and vocal tract size (even among speakers from the same speech communities). This acoustic variability proved problematic without an additional system by which phonological representations of words could be extracted from the messy acoustic signal. The solution was in the normalization of the speech signal: Coarticulatory effects were undone, and the idiosyncratic features of talkers’ voices were stripped away by the listener, allowing the activation of an invariant phonological form.

In the following decades, exemplar models challenged the concept of abstract phonological forms as the mental representations of words. The core principle of exemplar-based models is that percepts—in the case of speech perception, words—are stored in memory as collections of individual labeled instances, rather than as a single abstraction. Goldinger (1996) provided evidence that exemplars were used in speech perception, finding that subjects were more accurate in identifying whether a word had been repeated or not if the repetitions were produced in the same voice. This suggests not only that voice information was stored in memory, but that this information was an integrated part of the percept along with the phonological form of the word itself. If only an abstract representation had been activated, with voice information stripped away, no voice effect should have been found.

These findings led to a line of research in phonetic accommodation which extended and further corroborated the predictions of the exemplar model. The theory of phonetic accommodation asserts that an individual’s pronunciation will drift towards that exhibited by his or her interlocutors after being exposed to it. Phonetic accommodation is predicted by exemplar theory because if prior stored instances of words are what comprise an individual’s mental representation of a word, a new instance will then (slightly) shift the mean pronunciation of future utterances of that word. Goldinger (1998) observed just such an effect in an AXB task, such that speakers’ repetitions of words (B) heard after the stimulus (X) sounded more like the stimulus than the original pronunciation (A). Furthermore, Goldinger found other effects that suggest the structure of exemplar clouds. Immediate shadowing of the stimuli produced a stronger imitative effect than delayed shadowing, suggesting new exemplars fade from memory with time. Secondly, more repetitions of words resulted in stronger accommodation, suggesting a greater number of new exemplars have a greater ability to shift the previous production averages. Lastly, low frequency words also displayed greater accommodation, suggesting a smaller pre-existing cloud of exemplars would be more prone to change than those containing more prior exemplars. While Goldinger’s (1998) accommodation study judged similarity to the model based on the qualitative impressions of a panel of judges, later studies were able to quantify phonetic drift due to accommodation, measuring small changes in particular phonetic features such as VOT (Shockley, Sabadini, & Fowler, 2004; Nielsen, 2011) and vowel quality (Tilsen, 2009).

While many studies have established strong evidence for the existence of word exemplars and their relevance in speech perception, it is less clear precisely what information is stored in one. Generally speaking, a standard exemplar model assumes that exemplars are fairly veridical representations, being stored “as they occur, without any abstraction at all” (Johnson, 2007, p. 27). However, the abstraction that Johnson mentions refers to the transformation of newly perceived percepts into prototypical forms, as needed for speech recognition, a separate phenomenon from exemplars themselves undergoing some sort of perceptual transformation or abstraction while stored in memory as part of clouds of instances of particular words. Nevertheless, studies by Johnson (1997a) and (1997b) propose exemplars with purely veridical information, though with varying degrees. Johnson (1997a) considers reduced representations of vowel exemplars composed only of formant values. However, Johnson (1997b) argues for the storage of ‘auditory spectra’ containing all auditory information that the ear gleans from the acoustic signal. He states that this more detailed representation “is realistic because it is based on psychoacoustic data, and it also avoids making assumptions about which of the many potential acoustic features should be measured and kept in an exemplar of heard speech” (2007, p. 34), though he concedes that more data-driven evidence could lend support for a more compact representation.

Others propose models which simultaneously contain veridical exemplars and abstract phonological representations. For example, Pierrehumbert (2002, 2016) argues in favor of a hybrid model containing both phonological encoding and exemplar representations. This helps to account both for generalizations such as Neogrammarian type sound changes which affect all instances of a particular phoneme, as well as word-specific phonetics that might be influenced by word frequency, a problem better handled by exemplar representations. McLennan and Luce (2005) and Luce and McLennan (2005) find that both abstract representations and exemplars may co-exist, being activated at different stages in speech perception. While such hybrid models include both phonological and exemplar levels of representation, it is unclear if exemplars in these models would have purely veridical information or if they may include some degree of abstraction as well.

Goldinger (1996) does not directly define the contents of exemplars, although he describes the perceptual process of encoding voice information in episodes as being automatic, which could suggest there is no specificity in what details are stored. Nevertheless, he also notes that episodic encoding of phonetic details may occur “only to the extent that they matter in original processing,” and that these memory traces will usually “emphasize elements of meaning, not perception” (p. 1180). Goldinger (2007) further considers the question of what exemplars encode, asserting that any ‘raw data’ will necessarily undergo some abstract transformation, such that “each stored ‘exemplar’ is actually a product of perceptual input combined with prior knowledge, the precise balance likely affected by many factors” (p. 50). Hawkins (2003, 2010), proposes that exemplars with fine phonetic detail are first stored in memory, but are processed for signal-to-structure mapping, only to the extent needed to achieve an understanding of the linguistic meaning. This can be achieved a number of ways depending on the context, whereas processing of the individual words, phonemes, or subphonemic details may not be necessary depending on the context. In some cases, top-down processing will be faster in identifying words and phonemes as opposed to actual processing of the speech signal, in which case the veridical phonetic details may not have been used to understand the linguistic meaning. This suggests a possible bias in the storage and maintenance of different phonetic information present in the speech signal which might in some way be modulated by contextual information.

Some previous work has similarly indicated that the availability of top-down information could be a factor in the storage and maintenance of phonetic detail. Nye and Fowler (2003) is a rare case of an accommodation study which used sentence rather than word stimuli, and as such, provided insight into how top-down information influences phonetic accommodation. In their experiment, subjects were presented with nonce word stimuli that varied in how closely they approximated English, particularly with regard to phonotactic similarity—e.g., low orders of approximation grossly violated English phonotactic structure, such as [ə tb oɪmɛkʌnd prən vʊʃəl], whereas higher order stimuli closely resembled English, such as [hiz ə pɪnto ænd hi fotəgræs wʌndɚfəlli]. The results showed that subjects more closely imitated the lower orders of approximation, which suggested that more information about the higher order stimuli may have undergone some degree of abstraction due to high level linguistic knowledge of English, allowing for less detailed processing of higher order stimuli.

Manker (2019) directly assessed the effect of contextual knowledge on the storage of phonetic details in exemplar memory. In a discrimination task, subjects heard sentences with one of the words repeated, and were asked to determine if the word sounded exactly the same or somewhat different when repeated. The results showed that subjects displayed better discrimination when the words had been heard in an unpredictable context, e.g., “Joe turned and saw the cabins,” as opposed to a predictable context, e.g., “Pioneers built log cabins.” In a second accommodation experiment with the same stimuli, it was revealed that subjects also displayed better VOT and pitch contour accommodation for words that had been heard in unpredictable context, corroborating the results of the discrimination task. Ultimately, this suggested that more detailed, veridical exemplars were stored or maintained in memory for unpredictable words, because more reliance on processing of fine phonetic detail was needed in the first place in order to identify these words, whereas the recognition of predictable words could be achieved via top-down processing using contextual information.

Thus, these studies showed that semantic contextual information is one possible bias affecting what details might be stored in exemplar memory. Other linguistic information may also make certain details of the speech signal prone to being ignored or abstracted. The current study considers the effect of coarticulation in a similar way: If a particular coarticulatory effect is expected, does it need to be processed, stored, and maintained in exemplar memory when such detail could easily be abstracted or reconstructed based on higher level linguistic knowledge?

To answer this question, I conducted a discrimination task that compared listeners’ abilities to store and maintain the acoustic details of expected coarticulation compared to unexpected acoustic detail. This should test whether coarticulatory predictability modulates the details that are stored in exemplar memory, by either stripping them away or abstracting them. I specifically examined the coarticulatory phenomenon of f0 raising versus lowering following voiceless and voiced consonants respectively, a phenomenon which can lead to tonogenesis following neutralization of consonant voicing (Hombert, Ohala, & Ewan, 1979). I hypothesized that subjects would show better discrimination of stimuli that differ in acoustic detail that would not arise from coarticulation, which would suggest more veridical details of the auditory signal are stored in exemplar memory for unpredictable speech.

3.1. D’ statistical analysis

The sensitivity index, d’, was calculated to assess the statistic significance of the data. This statistical measure is useful for determining a subject’s ability to perceive similarity or difference while taking into account the possibility of response biases—for example, subjects who primarily think everything sounds the ‘same.’ Thus, this statistic takes into account not only a subject’s ‘hits’—correctly identifying when the word repetition is different—but also ‘false alarms,’ where the subject believes the stimuli sounded different when they were in fact the same. Thus, a subject with a 100% hit rate but also a 100% false alarm rate would have a very low d’ score, whereas a 100% hit rate and a 0% false alarm rate would result in a high score, indicating a high level of sensitivity in detecting the acoustic differences in the stimuli (MacMillan & Creelman, 2005). Two d’ values were calculated for each subject, one quantifying sensitivity towards differences in the voiced stimuli and the other for the voiceless stimuli, using the same-different d’ equation (rather than ‘yes-no’) via the sensR package in R. Thus, we can compare over all subjects to see if their d’ scores are significantly higher for the voiced tokens, as is hypothesized.

Additionally, since d’ is less accurate for very high hit and false alarm rates approaching 100% or 0% due to resulting in infinite z-scores (Stanislaw & Todorov, 1999), I followed a similar method to the log-linear approach detailed in Hautus (1995). In order to avoid 0% and 100% rates, a value of 1 was added to each of the false alarm, correct rejection, miss, and hit totals for each subject. Thus, a perfect hit rate of 24/24 and 0/24 misses would be corrected to 25/26 and 1/26 respectively.

D’ was calculated for all 100 subjects in both the voiced and voiceless stimuli conditions. The mean d’ for the voiced stimuli over all subjects was 1.59, whereas the mean for the voiceless stimuli was only 0.96. Subjects also showed a bias towards responding ‘same,’ rather than ‘different,’ reflected in a criteria location value, c, of 0.5938, the positive value here indicating a higher miss rate than false alarm rate. In order to assess statistical significance, a paired, two-tailed t-test was conducted comparing the individual voiced versus voiceless stimuli scores for each of the 100 subjects. The results show subjects’ d’ for the voiced stimuli was significantly higher than the d’ for voiceless stimuli (p < 0.0001, t = 6.37, df = 100), corroborating the assessment that subjects more readily could perceive the raised pitch contour on voiced rather than voiceless stimuli. The scores for all subjects are shown below in Figure 6, with d’ values for voiceless stimuli along the y-axis and values for voiced stimuli along the x-axis. Each dot represents one of the 100 participants, and the diagonal line represents an equal d’ score for both the voiced and voiceless stimuli—thus those falling above the line demonstrated better discrimination of the voiceless tokens, while those falling below the line demonstrated better discrimination of the voiced tokens. The greater number of dots below the line reflects the bias towards better discrimination of the voiced tokens among individual subjects.

Figure 6

d’ scores for voiced and voiceless stimuli over all subjects. Each dot represents a single subject.

3.2. Principal Components Analysis

A post-hoc Principal Components Analysis (PCA) was also conducted in order to determine if subjects used different listening and/or response strategies. The PCA was run using the prcomp function in R. The model included twelve variables, which were the total number of correct responses (out of eight) for each of the six phoneme stimuli (/p t k b d g/) repeated either the same or different (6 phonemes × 2 conditions). The results revealed just two principle components that accounted for more than 5% of the data.

As shown in the biplot in Figure 7, PC1 explains 57.1% of the variation in the data, whereas PC2 accounts for 18.4%. The red arrows show the contribution of each variable to the two PCs. For example, higher accuracy of the ‘different’ stimuli led to a higher PC1 score (thus the leftward points of the ‘same’ variables and the rightward points of the ‘different’ variables). Thus, we can conclude that PC1 is the component indicating either a ‘same’ or ‘different’ response bias. For example, subject 6 responded ‘different’ to all 96 stimuli, whereas subject 10 responded ‘same’ to all but one of the 96 stimuli.

Figure 7

Principal Components Analysis of subjects.

Higher accuracy of all stimuli contributed to a higher PC2 score, suggesting that PC2 correlates with how accurately the subjects responded overall. For example, subject 8, with the highest PC2 score, correctly discriminated 89 of the 96 stimuli. The overall triangular shape of the subjects’ spread is indicative of the fact that the ones who did not have a strong ‘same’ or ‘different’ response bias usually did better overall. Further inspecting PC2, we see that the three voiced ‘different’ variables contributed more to a higher PC2 since subjects were more successful in discriminating those stimuli (thus the higher upward tilt of those arrows).

4.1. Consequences for exemplar theory

The main findings from this study suggest that either some acoustic detail is filtered from exemplar memory or that certain details fade more rapidly from those exemplars. This departs from some other exemplar models in suggesting some degree of abstraction occurs in the process of storing exemplars. For example, Johnson (1997b) suggests that the entire auditory spectrum is stored in memory. The current findings suggest that, contrary to this, that not all information is stored in its raw, veridical form. However, Johnson’s assertion cannot be ruled out entirely. It could be the case that the raw auditory spectrum is briefly stored in a sort of working memory before being committed to exemplar storage. All that can be deduced for sure is that different features of the auditory signal, particularly those which are more predictable in some way, fade from exemplar memory more quickly. Johnson’s (1997a) suggestion of reduced exemplars that store only formant information also requires some amendment. It is unclear that there is any automatic means of compressing the auditory data for storage in memory; rather, the listener’s attention guides what is stored, and details that are less predictable in a particular context are more likely to be committed to longer term storage of the veridical details. Generally speaking, these findings align well with hybrid exemplar models (e.g., Pierrehumbert, 2002), which suggest listeners store both exemplars and abstract representations in memory. However, it remains unclear whether both types of exemplars exist, or if instead exemplars are typically a patchwork quilt of veridical and abstracted information. While further study is needed to understand the relationship between veridical and abstracted detail in memory, it seems probable that details of individual exemplars will continue to fade from memory and become abstracted over time. At some point, the memory trace may primarily contain phonemic information, or further abstract to represent mere words or ideas. However, this does not preclude the possibility that individual exemplar clouds—containing thousands of traces of varying degrees of abstraction—may be linked to some sort of fully abstract representation of the word.

The current findings also fit well with Goldinger’s (2007) statement that “each stored ‘exemplar’ is actually a product of perceptual input combined with prior knowledge” (p. 50), as well as Hawkins’ (2003, 2010) observations that the speech signal is processed only to the extent needed for extracting linguistic meaning. This also concurs with Goldinger & Azuma’s (2003) application of Adaptive Resonance Theory (ART), which suggests that there is no fixed unit of speech perception—speakers adaptively process speech units in whatever way achieves the quickest comprehension of the linguistic meaning of the speech signal. We could further apply the current findings to this model to suggest that the details processed in speech perception are what end up being encoded in the memory traces themselves. Ultimately, acoustic information that is predictable based on coarticulation or context (such as found in Manker, 2019) may not survive in the exemplar.

It should also be noted that the current findings do not suggest that any automatic process of abstraction must act in transforming the speech signal before word recognition, as is typical in models of speech perception including normalization (Gerstman, 1968; Tranmüller, 1981, etc.). Rather, those details that are particularly predictable and/or redundant in speech perception may be most likely to fade from exemplar memory. In fact, some subphonemic information is often facilitative in speech recognition (Johnson, 1997a), and in such cases, I would expect these details would more likely survive in exemplar memory. Further study will examine this question more closely.

While the results strongly demonstrate an effect of perceptual salience of coarticulatory details in a given phonetic context, an alternative analysis could challenge whether the observed perceptual bias is relevant to exemplar storage at all. For example, perhaps listeners at some point in the experiment became aware that ‘different’ always meant a raised pitch contour, at which point they began to ignore the initial utterance and only focus on the repetition. In this case, the difference in perceptual salience was all that motivated the observed bias, with no bias in what was originally stored or maintained in exemplar memory. An experimental design including initial utterances with raised pitch which is then repeated would be able to rule out or confirm this possible explanation. However, I believe this alternative account is unlikely, primarily due to the inclusion of the filler ‘word-identification’ stimuli. In these cases, no word was repeated at all, so it encouraged listeners to pay close attention to the initial utterance of the word and not only its (possible) repetition. Additionally, the dual tasks would likely distract from subjects’ attempts to determine any patterns in the repetitions. In any case, further research can explore and disambiguate this competing interpretation.

4.2. Relation of findings to compensation for coarticulation

Compensation for coarticulation is a phenomenon whereby listeners perceptually ‘undo’ coarticulatory effects of neighboring sounds in order to determine the intended underlying segments. This was famously observed in Mann and Repp (1980), in which subjects were more likely to perceive an acoustically ambiguous fricative as the sound [s] rather than [ʃ] following [u], arguably due to the coarticulatory effect caused by the rounded vowel [u], which tends to cause the neighboring sounds to lower in frequency. The mechanics of this process suggest something similar to the effect found in the current paper—that listeners perceptually remove an initial pitch raise following voiceless sounds as an expected effect of coarticulation, thus stripping away this detail in exemplar memory. However, Holt, Lotto, and Kluender (2001) find sensitivity to the coarticulatory relationship between F0 and voicing in Japanese quail, suggesting awareness of this relationship is not rooted in human speech perception. As a result, it is not clear if the phenomenon observed in the present paper is the result of compensation for coarticulation or a distinct phenomenon rooted in more general acoustic predictability, though the perceptual consequences may be quite similar. Future research and analysis are needed to investigate the relationship of predictability-modulated acoustic awareness and compensation for coarticulation.

4.3. Consequences for sound change

Ohala’s (1981, 1983) account of perceptual correction, and other studies of compensation for coarticulation (Yu, 2010; Yu, Abrego-Collier, & Sonderegger, 2013), have considered the role of perceptual biases in sound change. For example, Ohala (1981) claims that hypocorrection is one source of sound change—when speakers notice certain coarticulatory details but do not attribute them to their phonological environment. Yu (2010) found that female subjects with low Autism Quotient scores demonstrated less compensation for coarticulation, being more likely to notice certain articulatory effects (e.g., hearing [ʃ] before [u] instead of [s] and not attributing the lowered fricative frequencies to coarticulation). The results of the current study could be applied to either of these accounts, following the proposal that such ‘misperception’ results from encoding predictable coarticulatory effects in exemplar memory.

The current results make some additional predictions, however. New variation that is phonologically predictable, such as coarticulation, even when exaggerated a bit beyond what listeners are used to hearing, is more likely to evade notice and fail to be retained in memory, at least shortly after perceiving these details. However, new variation that is phonologically unconditioned would be more likely to be stored and maintained in exemplar memory. If, following the phonetic accommodation paradigm, we assume that new exemplar traces inform future productions, then we might expect conditioned versus unconditioned sound changes to spread differently within a language. For example, a conditioned change like the nasalization of vowels before nasal consonants (followed by their eventual loss) should more likely evade the notice of listeners since it is a predictable result of coarticulation, though a change of this sort is under the constant articulatory pressure that causes coarticulation in the first place. On the other hand, an unconditioned change, like a chain shift causing /p t k/ to become /p^h t^h k^h/ in all phonological environments should be more likely to be encoded into listeners’ exemplar memories since the change is not phonologically predictable. However, whereas we might predict its spread from speaker to speaker may happen more rapidly, it is not clear how strong the motivation is to begin in the first place, since unconditioned changes may be influenced more by phonological considerations (e.g., pressures within the sound system) rather than articulatory pressure.

Additionally, it is not clear how the effect observed in this study would be maintained over longer periods of time, necessary for eventually permanent changes in a language. For example, predictable detail, while stored in memory less faithfully and possibly abstracted in some way, may actually be retained for longer, whereas the more veridical memories of unpredictable memory may fade more quickly. In any case, the results may suggest some differences in the way that conditioned and unconditioned sound changes spread, though much more work is needed to understand the relevance, if any, of predictability-based perceptual biases in exemplar storage on sound change.

4.4. Future research

Several important questions remain in order to understand the nature and contents of exemplars. First of all, it is necessary to continue to survey how different acoustic cues are stored in exemplar memory, and the various perceptual biases that facilitate or impede the storage of various auditory information. One question of particular interest will be whether or not certain coarticulatory cues are in fact stored in memory. While I have suggested that predictable coarticulatory cues may be ignored and perceptually filtered, there is a lot of research showing that coarticulation can be used to facilitate speech recognition (Johnson, 1997a; Beddor, Krakow, & Lindemann, 2001). Beddor et al., for example, state that “listeners use coarticulatory variation as information about sounds that are further up or down the speech stream” (p. 56). If coarticulation aids in speech recognition in this way, following my previous proposal that details that are used in speech recognition will more likely survive in exemplar memory, this should result in better storage of these details. However, it is unclear how the events of auditory storage and maintenance unfold. For example, perhaps once phoneme or word recognition occurs, such coarticulatory details are rapidly lost from memory or abstracted. Alternatively, there could be some difference in anticipatory versus confirmatory coarticulation. In the present study, the f0 modulation occurred after the voicing and VOT cues had provided ample evidence as to the initial sounds in the target words (e.g., ‘bath,’ ‘path,’ etc.). The f0 cue only served as an expected confirmation. On the other hand, nasality on a vowel that cues an upcoming nasal consonant occurs at a point in time when the listener does not know the upcoming sound (such as is shown for English by Lahiri & Marslen-Wilson, 1991). Thus, anticipatory coarticulation of this sort might be more faithfully stored and maintained in memory.

Further research should also consider additional aspects of predictability and expectation. For example, predictability of a person’s voice may lead to lower awareness of certain acoustic information—more acoustic information might be stored when listening to different speakers produce stimuli, similar to the stronger imitative effect found in Goldinger (1998) when subjects heard multiple model speakers. In addition to the contextual predictability effect found in Manker (2019), which was based on word priming, we could consider whether general situational context also results in lower attention and less faithful storage of the auditory signal. For example, if objects are visually presented before they are referred to, will listeners store less auditory detail of these words? Finally, we might consider how the predictability of semantic context interacts with other forms of predictability, such as phonologically predictable coarticulation.

Lastly, further work in the phonetic accommodation paradigm will complement the findings of the current study. For example, as in Manker (2019), should we expect to find greater accommodation of predictable coarticulation compared to unpredictable phonetic variation? Zellou, Scarborough, and Nielsen (2016), for example, did in fact find imitation of ‘hyper-nasalized’ vowel coarticulation, such that speakers did in fact increase their own vowel nasality after hearing greater vowel nasality produced. From this study alone, it is not clear whether such imitation of coarticulation is weaker in magnitude than imitation of unconditioned acoustic variation. Secondly, this is again a case of an anticipatory coarticulation effect, so would the effect be weaker for perseverative coarticulation? Further research will help to interpret the growing body of literature in phonetic accommodation as a whole and its relevance in the storage and maintenance of detail in exemplar memory.