1.1. Multiple cues: Individual variability and sound change

Individual variability in speech perception has been well documented (see Yu & Zellou, 2019 for a review). There is also evidence that individuals may vary in their use of secondary cues in perception (see Schertz & Claire, 2019 for a review of individual variability in cue weights). Some individuals use a secondary cue more than others for F0 in English stop voicing (Kapnoula, Winn, Kong, Edwards, & McMurray, 2017; Kong & Edwards, 2016; Shultz, Francis, & Llanos, 2012) and vowel duration for English tense/lax vowels (Kim & Clayards, 2019). A common method to quantify individual cue weights is to use regression coefficients fit to each individual’s responses (e.g., Shultz et al., 2012), or to use by-individual deviations from the population coefficient for a single regression model fit to all data (‘random slopes,’ e.g., Clayards, 2018; see also Schertz & Claire, 2019 for discussion of different methods). Using these methods, researchers have tried to determine if individuals differ from each other systematically by looking at whether individuals’ cue weights are correlated across dimensions. In other words, they asked whether those with larger than average primary cue coefficients have smaller than average or larger than average secondary cue coefficients.

Results have been mixed, with some studies finding a weak or non-significant relationship between cues in perceiving a contrast (Shultz et al., 2012; Clayards, 2018 for F0 and VOT for English stop voicing) or positive correlations (Clayards, 2018; Kim & Clayards, 2019 for vowel formants and duration for English tense-lax vowels). For the Korean three-way laryngeal contrast, one study found positive, negative, or no correlation between cues depending on the contrast pair (Kong & Lee, 2018). Clayards (2018) examined coefficients for individuals across different contrasts in English (e.g., tense/lax vowels and word-final fricative voicing) and found that positive correlations were the most common for both primary and secondary cues (e.g., individuals with larger coefficients for the secondary cue in tense/lax vowels also had larger coefficients for the secondary cue in final fricatives). This suggests that differences between individuals may be systematic and not tied to particular contrasts or dimensions (cf. Hazan & Rosen, 1991). Some researchers have argued that more use of a secondary cue is associated with more gradient sensitivity to primary cues, using a visual analog scaling task rather than a categorical decision task (Kapnoula et al., 2017; Kong & Edwards, 2016). Thus, the positive correlations in Clayards (2018) and the relationship between gradient sensitivity and cue use (Kapnoula et al., 2017; Kong & Edwards, 2016) both point to some listeners’ speech perception being more closely tied to the acoustics of the stimulus than others. However, as noted above, not all studies have found this relationship.

Individual variability in secondary cue use may also play an important role in sound change. Some sound changes involve a non-primary cue taking over the role of a primary cue as occurs in ‘tonogenesis’ (Kingston, 2011), for example, the case of some younger speakers of Afrikaans shifting from VOT to f0 to signal a voicing contrast (Coetzee, Beddor, Shedden, Styler, & Wissing, 2018). In contrast, in the case of Shanghai Wu, a non-primary cue (i.e., breathiness) to the register contrast is losing importance (Gao, 2016). We thus may expect to observe increased individual variability in the use of non-primary cues in a variety undergoing sound change (though see Coetzee et al., 2018 perception data). Conversely, since Shanghai Wu is undergoing a loss of a non-primary cue, rather than an increase in non-primary cue importance, we may see more individual variability in a contrast in a variety that is not undergoing the loss of a non-primary cue (Jiashan Wu), and is therefore more dependent on multiple dimensions (e.g., Mayo & Turk, 2005 find larger differences between individuals, in this case adults and children, on contrasts with a larger role for the non-primary cue).

1.2. The perception of voice quality cues

Voice quality, one of the cues of interest in this study, plays different roles in speech perception in different languages. Furthermore, whether voice quality is used to signal a contrast and whether listeners are perceptually sensitive to voice quality are only partially related. This section summarizes the perception of voice quality in three types of languages.

First, in some tone languages, voice quality can be either a phonemic dimension that is independent of pitch, or it can be the main cue to a contrast that also has pitch differences. For example, in Yi, tones with the same pitch can be associated with different phonation types, and listeners rely on phonation cues to distinguish the tones (Kuang, 2011). Jalapa Mazatec (Garellek & Keating, 2011) provides a similar case, where phonation contrasts are independent of pitch contrast, with each tone (low, mid, high) associating with different phonations (laryngealized, modal, breathy). In addition, phonation cues also appear to be perceptually important when different voice qualities are associated with two tones of similar pitch (although not identical). For example, White Hmong has two tones with similar pitch (21 versus 22 in Chao numbers) that also contrast in modal versus breathy voice. Here listeners were found to attend to voice quality information but ignore changes in pitch height or contour (Garellek, Keating, Esposito, & Kreiman, 2013). Similarly, in Sgaw Karen, pitch information is limited and instead, voice quality is crucial for listeners to distinguish the tonal contrast (Brunelle & Finkeldey, 2011). Thus, voice quality can be an independent contrast, or it can be the primary cue in cases where the contrast also includes some difference in pitch.

Second, some languages use phonation as a redundant or non-primary cue associated with certain tones. For example, in Mandarin Chinese, creaky voice facilitates the perception of tone 3, a dipping tone (Kuang, 2013). Similarly, creaky voice in Cantonese tone 4 syllables increases tonal identification accuracy (Yu & Lam, 2011). In Black Miao, tones contrasting in both phonation and pitch are better distinguished than tones contrasting only in pitch (Kuang, 2013). In Northern Vietnamese, creakiness turns out to be as important as pitch in tonal categorization for one pair of tones (Brunelle, 2009).

Third, some languages like English do not use phonation to mark contrasts. Nonetheless, English listeners’ perception of talker pitch is influenced by spectral slope such that sounds with tenser/flatter spectral slope are heard as having a higher pitch for both synthetic and resynthesized speech (Kuang & Liberman, 2015; Kuang, Guo, & Liberman, 2016). This may allow listeners to normalize for pitch range (Honorof & Whalen, 2005) by making use of changes in voice quality that occur in the higher parts of speakers’ pitch range (Hollien, 1974). However, listeners were not sensitive to the steepness of the spectral slope (as long as it was not entirely flat), i.e., the degree of breathiness/tenseness.

To summarize, voice quality plays a variety of roles cross-linguistically, and is used by listeners for different purposes. The next section gives a brief introduction to the target language of this study, Wu Chinese, in which voice quality facilitates tonal register contrasts.

1.3. Wu Chinese

Wu Chinese is spoken in Shanghai, Zhejiang province and southern Jiangsu province of China. The two target dialects, Shanghai and Jiashan, are both sub-dialects of Wu. According to Yip (1980, 2002), a register feature [+/– Upper] divides the tonal space according to pitch: [+Upper] indicates a higher pitch range (corresponding to the historical yin tones) and [–Upper] indicates a lower pitch range (corresponding to the historical yang tones) creating a tonal register contrast. Historically, the register was also related to initial consonant voicing, that is, voiceless consonants only occurred in upper registers (yin tones) and voiced consonants only occurred in lower registers (yang tones). While the restriction on the distribution of consonant voicing and register still exists in non-initial position, the voicing contrast has been lost in initial position (e.g., Chen & Gussenhoven, 2015).

The tone inventory of Shanghai and Jiashan dialect are represented using the register features in Table 1 together with Chao numbers (Chao, 1930) indicating the pitch contour (numbers 1–5 stand for lower–higher pitch). Tone notations of Shanghai Wu are from Xu and Tang (1988); as there is no previous study on Jiashan Wu, the transcriptions are based on the first author’s experience as a native speaker and Yu (1988) on Jiaxing Wu, a highly similar dialect spoken nearby.

There are four types of tones: falling, level, rising, and checked. Checked tones are relatively short in vowel duration and end in a glottal stop. Both dialects have fewer than eight tones due to historical mergers. Shanghai dialect merged the level and the rising tones in the upper register, while falling, level, and rising tones were merged into one rising tone in the lower register (Qian, 1992), leaving a five-tone system. In Jiashan, only the lower-register level and rising tones were merged, resulting in a rising tone (Yu, 1988) and a seven-tone system.

Note that the tone notations in Table 1 do not accurately match the F0 trajectories, but are rather an abstract representation. Various other notations are proposed by different researchers (as discussed in Chen & Gussenhoven, 2015 for Shanghai Wu), as the pronunciations are varied across individuals and generations. However, no one notation can accurately reflect the exact F0 trajectories given the large variation. Therefore, similar to differences between formant values and distinct IPA symbols, the tones represented in Chao numbers are phonemic and do not necessarily correspond to the actual phonetic realization. For example, related to the current study, the Jiashan falling tone is realized with a steeper contour in the upper register than the lower register while the Shanghai rising tone is realized with a steeper contour in the lower register than the upper register, although the tone notations do not reflect such differences. For the checked tone, while it is always produced with shorter vowel duration, speakers vary in how audible the glottal stop is.

While pitch range signals the register contrast, it is not the only cue. Others involve voice quality, pitch contour, and duration. Crucially, the upper tonal register is produced with modal voice whereas the lower tonal register is produced with breathy voice (Cao & Maddieson, 1992; Chen, 2010; Gao, Hallé, Honda, Maeda, & Toda, 2011; Zhang & Yan, 2015; Jiang & Kuang, 2016). In addition, in many Wu dialects, the steepness of the contour in contour tones differs in the two registers² (e.g., Chen & Gussenhoven, 2015).

While both Shanghai and Jiashan dialects share the same characteristics of register contrast, the Shanghai dialect is argued to be going through a loss of breathiness in the lower register, at least in production (Gao, 2016; Gao et al., 2011). Based on their acoustic and electroglottographic data, Gao and colleagues found that younger speakers used less breathy voice in the lower register compared to older speakers, possibly due to contact with Mandarin Chinese which does not employ breathy voice in tonal contrasts. However, Shanghai Wu listeners do seem to use voice quality in perception despite the change in production. An early study by Ren (1992) found that breathy voice was a perceptual cue for the lower register. A later perception study by Gao, Hallé, and Draxler (2019) found that voice quality did influence perception for both natural (produced by a trained phonetician) and synthesized stimuli. While Zhang and Yan (2015) found that Shanghai listeners heavily relied on F0 information for the register distinction, they also used breathiness as one of the non-primary cues. It is not known how much Shanghai listeners use breathiness relative to other Wu listeners, for whom this cue is not being lost. Direct comparison between listener groups is required to determine this.

To summarize, Shanghai Wu and Jiashan Wu both incorporate a breathy-modal distinction into the tonal register contrast as a non-primary cue. The two dialects have similar phonological systems, although Jiashan has a slightly richer tone inventory and stronger voice quality distinction. Moreover, as Shanghai is in the process of losing breathiness, its listeners may also exhibit some characteristics of listeners of non-phonatorily contrastive languages like English, that is, relying less on breathiness when compared with Jiashan listeners.

1.4. Current study

This paper examines the perceptual difference between dialects and across individuals on the tonal register contrast in Chinese Wu dialects, by manipulating three cues: pitch height (F0 onset), voice quality, and contour (pitch slope). We compare the falling tone pair in Jiashan Wu and the rising tone pair in Shanghai Wu (Experiment 1) and the checked tone pair in both languages (Experiment 2). Because the checked pair is the same in both dialects we can compare both groups of listeners on the same stimuli in Experiment 2. Unlike previous work on perception of voice quality, we explore breathiness as a perceptual cue in fine detail by creating a continuum from natural endpoints that changes in all aspects of the acoustic space for breathiness. Previous studies either only have two levels (breathy versus modal, e.g., Gao et al., 2019; Zhang & Yan, 2015), or manually modified only certain parameters (e.g., Garellek et al., 2013, who modulated H1–H2, H2–H4, H4–2 kHz, and 2 kHz–5 kHz). We also investigate how different dialectal experience caused by sound change is manifested in the perception of multiple cues at both the group level and the individual level by comparing Shanghai and Jiashan Wu listeners on stimuli from both dialects. This paper addresses the three questions raised earlier.

First: Averaging over listeners within each dialect, how important is a non-primary cue (i.e., voice quality) in a multidimensional contrast and does it vary by tone contrast, or is it consistent for different tone pairs? The current study manipulates the three cues independently and includes a five-step breathy-modal continuum. This extends previous studies on cue weighting in Chinese Wu dialects by allowing different combinations of ambiguous cues. We predict that voice quality will significantly influence listeners’ perception, at least for Jiashan listeners, but we expect it to be less important than pitch height. The relative importance of voice quality and pitch contour may change depending on the tone pair.

Second: As sound change is taking place in Shanghai, does the decreasing use of the voice quality in production result in different cue weighting than for traditional Wu (i.e., Jiashan) listeners? This study investigates how a dialectal background difference in degree of a non-primary cue production (i.e., voice quality) affects listeners’ cue weights. To understand whether listeners from the two dialect groups show different perceptual strategies, and whether the potential difference is caused by the acoustic cues in the stimuli or differences in the listeners, this study examines listeners’ cue weightings when they are exposed only to their native dialect (Experiment 1) and when they listen to both dialects (checked tones in Experiment 2). Because the Shanghai dialect is thought to be less breathy, we expect voice quality to have only a small effect on listeners’ responses when they listen to the Shanghai stimuli. If both sets of listeners respond to voice quality in the same way, we expect voice quality to have a bigger effect when they listen to Jiashan stimuli, which have a larger voice quality difference. However, if Shanghai listeners are not as sensitive to voice quality as Jiashan listeners we expect their responses to be less affected by voice quality than Jiashan listeners when listening to Jiashan stimuli.

Relevant to the comparison in Experiment 2 is the question of to what extent listeners have experience with the other dialect. It should be noted that both dialects are mutually intelligible, and there are TV shows broadcast in both dialects. Since Shanghai is a large city it may be the case that Jiashan speakers have more opportunities to be exposed to Shanghai Wu than vice-versa. However, it is also the case that Shanghai listeners have opportunities to be exposed to dialects with larger breathy-modal contrasts, although not necessarily Jiashan Wu. Within the city of Shanghai, there are other dialects of Shanghai Wu that differ from the more common ‘downtown Shanghai Wu’ used in this study in that they are more typical Wu dialects with larger breathy-modal contrast. Moreover, older Shanghai speakers produce breathier lower register words than younger speakers (Gao, 2016) and residents of Shanghai could also be exposed to speakers of other, mutually-intelligible Wu dialects with a greater breathy-modal contrast. Given these factors it is difficult to say whether one dialect group is more familiar with the other dialect or not.

Third: Do individuals show structured differences in cue ordering or cue magnitude, or are differences between individuals random variation? Does the structure of individual variability differ by dialect? To address these issues, this study examines individual variability in relative cue weighting. Previous work on individual variability has examined only two cues to a contrast. By examining three cues we are able to better observe how cue weights relate to each other. Group-level results are sometimes insufficient to understand how a multidimensional contrast is perceived, as the results are averaged over all participants, giving only one pattern of cue weighting. How individual cue weights diverge from the average pattern is important for fully understanding the phonological contrast of the target language as well as the perceptual system of individuals. Individual variability in cue weights may be completely random, or structured (correlated). Individual differences could be small, with individuals only differing in the magnitude of the cues, while sharing the same ordering of cue weights, or larger, with individuals differing in the ordering of cue importance. Because so little is known about individual variability in cue use in perception, especially with more than two cues, it is difficult to make a priori predictions. To examine individual differences, we use the correlation matrix of the random effects of a mixed-effects model fit to the data. A methodological contribution of this paper is showing how this component of mixed-effects models, which is typically ignored in phonetic studies, can be used to understand the structure of individual differences.

3.1. Experiment 1: Group results

Experiment 1 examines listeners’ perception of tonal register contrast (upper versus lower) of their native dialect when exposed to stimuli with different degrees of breathiness and pitch and with two contours. In order to investigate the relative importance of the three cues, two mixed-effects logistic model were fit for the two dialect groups respectively to model participants’ response as a function of the three variables.

The data were fitted with a Bayesian mixed-effects logistic regression (MELR) model in R using version 1.0-4 of the blme package (Chung, Rabe-Hesketh, Dorie, Gelman, & Liu, 2013) using the default settings for priors. One advantage of a Bayesian MELR over the more widely used non-Bayesian MELR (e.g., using lme4 in R) is the ability to fit a maximal random-effect structure (all possible slopes and correlation terms; Barr, Levy, Scheepers, & Tily, 2013) without convergence issues which frequently arise for non-Bayesian methods (including when applied to our data), usually related to ‘impossible’ 1/–1 correlation values (Nicenboim & Vasishth, 2016; see Vasishth et al., 2018 for a tutorial). The dependent variable was the register response (upper = 1, lower = 0). All predictors were centered and standardized by subtracting the mean and dividing by two standard deviations. This makes the coefficients comparable (e.g., in Table 3, the coefficient estimates of pitch are the highest among the three cues, indicating that it is the primary cue) and minimizes collinearity between main effects and interactions (Gelman & Hill, 2006). More importantly, centering means the interpretation of a certain main effect is averaged over other variables this specific variable interacts with. The fixed-effect predictors were voice quality step breathiness (coding for five steps are scaled in range [–0.7, 0.7]), pitch height step pitch (coding for five steps were scaled in range [–0.7, 0.7]) and pitch contour step contour (two levels: upper = 0.5 lower = –0.5). Two-way and three-way interactions were also included because changes in one cue may have an effect on participants’ perception of the other cues. The random effects included by-participant random intercepts and random slopes for breathiness, pitch, and contour and their interactions, to account for individual differences in cue weights, as well as all possible correlation terms (the maximal structure), to measure potential correlations in cue usage among individuals.⁵

3.1.1. Cue weights in Shanghai Wu

Figure 3 shows the mean responses from the Shanghai participants. Table 3 shows the model results for fixed-effects terms. All three cues had a significant effect, and only the pitch-breathiness term was significant among all interactions. Note that since all variables are centered, the main-effect coefficient estimates indicate how much the log-odds ratio of responding with the upper register changes as the variable shifts by one unit, averaging over other factors (e.g., the β coefficient of contour means the effect of contour at the average pitch and breathiness values).

Figure 2

Contour continua of Shanghai (left) and Jiashan (right) used in Experiment 2.

Figure 3

Percentage of upper register response from Shanghai participants. The x axis is the breathiness continuum (1 = breathy, 5 = modal). The y axis is the pitch continuum (1 = low pitch, 5 = high pitch).

According to the coefficient estimates, pitch was the most important cue (β = 6.67, SE = 0.36, p < 0.001), much stronger than contour (β = 1.26, SE = 0.18, p < 0.001), while breathiness was the least important (β = 0.55, SE = 0.11, p < 0.001).⁶ Moreover, the two-way interaction between breathiness and pitch (β = 0.74, SE = 0.32, p = 0.024) shows that the higher the pitch, the more important voice quality is. There was a trend for a three-way interaction between the cues (β = 1.43, SE = 0.72, p = 0.052), indicating that how much the effect of breathiness is affected by pitch is itself modulated by the specific contour. Together these interactions suggest that the higher pitch onsets and the upper register contour were slightly more ambiguous than the lower pitch onsets and lower register contour.

Figure 3 shows the corresponding heat map of the participants’ categorization. The left panel represents the results for the lower register tone contour and the right panel represents the results for the upper register contour. The lighter color indicates higher percentage of upper register responses. In terms of cue weighting, compatible with the model predictions, there was indeed a large effect of pitch, as indicated by the change of colors along the y axis in both panels. Effects of breathiness and contour are much weaker, and are most visible in regions with ambiguous values of pitch (step 3). The effect of breathiness can be seen in a color change along the x-axis and the effect of contour can be seen in lighter colors in the right panel, consistent with model predictions.

3.1.2. Cue weights in Jiashan Wu

The Jiashan participants, however, show a different pattern. As displayed in Table 4, the Jiashan model found that contour had the biggest effect (β = 3.08, SE = 0.46, p < 0.001) followed by pitch (β = 1.63, SE = 0.30, p < 0.001) and breathiness (β = 0.86, SE = 0.16, p < 0.001).⁷

Table 4

Summary of the fixed effects for the model of Jiashan participant data from Experiment 1.

	Estimate	SE	z-value	Pr(>|z|)
Intercept	0.73	0.11	6.51	<0.001
breath	0.86	0.16	5.14	<0.001
pitch	1.63	0.30	5.45	<0.001
contour	3.08	0.46	6.62	<0.001
breath × pitch	–0.69	0.20	–3.37	<0.001
breath × contour	–0.52	0.23	–2.22	0.02
pitch × contour	–0.74	0.24	–3.06	0.002
breath × pitch × contour	–0.30	0.39	–0.76	0.44

Heat maps of the empirical data also reflect this pattern (Figure 4). It is obvious that contour has a large effect on the perception of tonal register for Jiashan listeners. When listening to the upper register contour (right panel), listeners consistently hear it as an upper register word. The heat map also shows that pitch and breathiness affect categorization, more so for the lower register contour, as captured by the significant interaction terms between breathiness and contour (β = –0.35, SE = 0.18, p = 0.04) and between pitch and contour (β = –0.62, SE = 0.19, p < 0.001), which predict lower breathiness and pitch effects for the upper register contour.

Figure 4

Percentage of upper register response from Jiashan participants (axes as in Figure 3).

The significant breathiness by pitch interaction (β = –0.69, SE = 0.2, p < 0.001) suggests that the effect of breathiness is smaller at higher pitch than at lower pitch. The right panel reflects this prediction in that there is a larger change in color at step 1 of the pitch continuum (the bottom row) than at step 5. As with the Shanghai responses, these interactions likely reflect the fact that the continua were not symmetric in how ambiguous the steps were. In particular, the lower register contour seems to have been more ambiguous than the upper register contour.

3.2. Experiment 2: Group results

This section extends the previous experiment by directly comparing the two dialect groups. In this experiment, listeners listen to the same stimuli (i.e., checked tone) produced by talkers from both dialects. Note that checked tones are marked with much shorter syllable duration and overall flatter contour (as shown in Figure 2 and Table 2), which may reduce the use of the contour cue. Checked tones are used for Experiment 2, however, because they are the only pair of tones sharing the same phonological representation between the two dialects (see Table 1). When exposed to stimuli differing in the degree of phonation contrast, we expect listeners who are sensitive to breathiness to show smaller breathiness cue weights for Shanghai stimuli with a weak breathy-modal distinction, than for Jiashan stimuli which differ more in breathiness.

Experiment 2 also clarifies one remaining problem in Experiment 1. Specifically, some participants relied almost solely on contour (see Section 3.3 on individual differences below). One possibility is that the effects of pitch and breathiness were masked by a dominant contour cue and weren’t observed because of the lack of an intermediate/ambiguous contour value. By breaking down the contour cue into a five-step continuum (while holding the other two cues at mid-point), Experiment 2 better examines how these listeners use all three cues.

The data were again fitted with a Bayesian mixed-effects logistic regression model in R using the blme package (Chung et al., 2013). As with Experiment 1, the dependent variable was the register response (upper = 1, lower = 0). Independent variables were breathiness, pitch, and contour (continuum step as a numerical variable from 1 to 5) and their interactions with talker dialect and participant dialect (for both talker and participant dialect, Jiashan was coded as 0 and Shanghai was coded as 1) as the effect of the cues may vary by talker and participant dialect. Random effects include by-participant random intercepts, and random slopes for the talker dialect and the three cues. Correlations among all random slopes were included. All predictors were centered and standardized by subtracting the mean and dividing by two standard deviations, as in Model 1 (making Jiashan = –0.5 and Shanghai = 0.5 for the talker and participant dialect variables).⁸ Table 5 shows the fixed effects of the model.

The results show that all the main effects are significant. This means that all three cues, although redundant, are important in perceiving the tonal register contrast for this tone pair. Secondly, responses depended on listener dialect and talker dialect. Specifically, listeners tended to perceive more lower register words when they listened to a Shanghai talker (indicated by the negative talkerDialect coefficient) probably because the talker had an overall lower pitch range. Furthermore, Shanghai listeners tended to hear more upper register words (indicated by the positive listenerDialeect coefficient). However, of main interest were the interactions between the cues and the talker and listener variables.

We found that talker dialect influenced use of contour (β = 0.63, SE = 0.18, p < 0.001) and pitch (β = 0.86, SE = 0.22, p < 0.001), indicating that listeners were more influenced by both contour and pitch when listening to the Shanghai talker. Listener dialect interacted with breathiness (β = –0.43, SE = 0.16, p = 0.008) and pitch (β = –0.87, SE = 0.38, p = 0.023), which reveals that Shanghai listeners were less influenced by breathiness and pitch. Moreover, there was a significant three-way interaction between talker dialect, listener dialect, and breathiness (β = 0.97, SE = 0.31 p = 0.002) indicating that listeners were more influenced by breathiness when listening to their own dialect than when listening to the non-native dialect, at least for Shanghai listeners (see Figure 5).

Figure 5

Cue weights for Jiashan (JS) and Shanghai (SH) talker by listener with error bars showing 95% speaker variability intervals, capturing individual variation in cue weights. Experiment 2.

Given that the model contains many two-way and three-way interactions, and the cue weighting of specific listener-talker combinations require taking these interactions into consideration, it is useful to calculate the coefficient estimates for usage of each cue as a function of talker and listener dialects. Table 6 below (also visualized in Figure 5) shows the coefficient estimates (with standard errors) of the three cues for different talker-listener combinations, calculated using the emmeans package (Lenth, Love, & Hervé, 2017) in R.

Pitch is the primary cue for all talker/listener dialect pairs, being much more important than the secondary cues (breath, contour). This suggests that when listening to checked tones, both Shanghai and Jiashan listeners mainly use pitch to distinguish the two registers. This is in contrast to Experiment 1 in which only Shanghai listeners used pitch as the primary cue. Secondly, when listening to the Jiashan talker, the Jiashan listeners had on average a positive cue weight for breathiness while the Shanghai listeners had on average a coefficient close to zero. Thus even when listening to the same stimuli, Jiashan listeners’ responses are more affected by breathiness. Finally, when listening to the Shanghai talker, Jiashan listeners had a larger coefficient for contour (their secondary cue for that contrast) than the Shanghai listeners. Thus, regardless of the stimuli, Jiashan listeners seem to have larger secondary cue coefficients. In fact, comparing within stimuli and dimensions (e.g., Jiashan talker pitch) in almost every case the Jiashan listeners have larger coefficients than the Shanghai listeners for the same stimuli and dimension.

Furthermore, Jiashan listeners appear to rely more on breathiness than Shanghai listeners as indicated by the change of cue weighting for different talker dialects. When exposed to a larger breathy-modal contrast (Jiashan Wu talker), Jiashan listeners indeed had a larger coefficient for breathiness than they did when listening to Shanghai Wu. Shanghai listeners on the other hand, had a smaller coefficient for breathiness when listening to Jiashan Wu talker, the opposite tendency to what one would expect given the greater role of breathiness in Jiashan Wu in production. Thus, Shanghai listeners did not use breathiness when they were actually exposed to a larger breathy-modal contrast, while they managed to better perceive such a contrast in their own dialect with a smaller difference.

To summarize, Jiashan and Shanghai listeners share a similar order of cue preference for both talkers, with pitch always being the primary cue. However, Jiashan listeners’ coefficients more closely relate to the amount of breathy-modal contrast in the stimuli than Shanghai listeners. Thirdly, both listeners show greater use of voice quality cue when listening to their own dialect.

3.3. Individual variability

3.3.1. Experiment 1

Group-level results found in Experiment 1 indicate that while all three cues significantly affect both groups of listeners’ tonal register categorization, Shanghai listeners’ perception was more restricted to a single cue (pitch) while Jiashan listeners on average had a more balanced cue weighting. What is not clear from the group results is whether these average patterns were an equally good description of individuals in the two groups. In the Introduction, we hypothesized that the ongoing loss of breathiness in Shanghai Wu might lead to inter-listener variability. On the other hand, having a contrast that depends on more cues might lead to greater listener variability in the Jiashan listeners. In order to test the two hypostheses, we compare variability in individual cue weights between the two listener groups. In addition, we wanted to test if there are patterns across cues in how individuals use them. In other words, does having a large primary cue weight predict having a large secondary cue weight, or does it predict having a small secondary cue weight? Examining the structure of the variability between cues allows us to test these competing hypotheses.

The random effects fit in the models in the previous section, summarized in Tables 7 and 8, allow us to examine individual variability in both the ways detailed above. Specifically, random effects are the part of the statistical model that captures the variability within groups (in this case, the variability of cue weights among different participants). We can examine the standard deviation in estimates for each cue (SD column), which capture the degree of variability in cue weights across participants, as well as the correlations between cues (across participants).

Table 7

Summary of random effects and correlations of Shanghai participants, Experiment 1. Correlations between interaction terms are not shown.

Name	Variance	SD	Correlation
Intercept	0.91	0.95
Breathiness	0.17	0.41	0.50
Pitch	3.91	1.97	0.61	0.83
Contour	0.86	0.93	0.54	0.72	0.63
Breath × Pitch	0.61	0.78
Breath × Contour	0.21	0.45
Pitch × Contour	1.89	1.37
Breath × Pitch × Contour	6.42	2.53

Table 8

Summary of random effects and correlations of Jiashan participants, Experiment 1. Correlations between interaction terms are not shown.

Name	Variance	SD	Correlation
Intercept	0.34	0.59
Breathiness	0.68	0.82	0.25
Pitch	2.71	1.64	0.29	0.93
Contour	6.86	2.61	–0.23	–0.12	–0.15
Breath × Pitch	0.46	0.67
Breath × Contour	0.87	0.93
Pitch × Contour	0.89	0.94
Breath × Pitch × Contour	1.28	1.13

Figure 6 summarizes the variability across individuals in the weight of each of the three cues, for each listener group. For each cue, the fixed-effect estimate from the model plus or minus twice the random effect standard deviation gives the range of cue values predicted for 95% of speakers, which we call the ‘speaker variability interval.’ As seen before for the group-level pattern, the importance of cues within dialect varies, and furthermore, Jiashan listeners consistently show larger variability than Shanghai listeners for the primary (pitch for Shanghai, contour for Jiashan), the second important (contour for Shanghai, pitch for Jiashan) and the least important cue (both are breathiness), with variance scaling with the mean.

Figure 6

Group-level cue weights for Shanghai (left) and Jiashan (right) listeners with error bars showing 95% speaker variability intervals, capturing individual variation in cue weights. Experiment 1.

For Shanghai listeners, pitch is always the most important cue for all participants, and higher pitch always leads to more upper register percepts. The findings are also reflected in the individual cue weighting plots (Figure A3 in the Appendix) where pitch always has the highest weighting while the reliance on contour and breathiness varies by individual. For Jiashan listeners, plots of individual cue weights (Figures A3 in the Appendix) further support the larger individual variability for these listeners. Unlike Shanghai listeners, Jiashan listeners show different preferences on the primary cue as well as the use of other cues. Moreover, plots of individual responses also display larger variation for Jiashan listeners (Figures A1 and A2 in the Appendix). It is worth noting that while there seem to be large differences in cue weighting, all participants (except for one participant, 62107, who displayed the opposite categorization) were successful in marking the contrast between the two natural tokens. This suggests that different individuals may have different perceptual strategies, an idea we explore further by inspecting the correlation terms.

Tables 7 and 8 indicate that the models fit some large positive correlations between the cues across individuals. In order to evaluate the significance of the correlations, we extracted the coefficient estimates from the main effects (no interactions) of the fixed effects and the random effects to produce the cue weights from each individual predicted by the model. This yields four estimates per individual, three slopes for three cues (breathiness, pitch, contour) and one intercept. We used non-parametric correlation tests using the Spearman method to test all correlations, since non-parametric models are robust to outliers. The results yielded significant correlations among all three cues for the Shanghai listeners (breathiness and pitch: r = 0.87, p < 0.001; breathiness and contour: r = 0.87, p < 0.001; pitch and contour: r = 0.67, p < 0.001) and only between pitch and breathiness for Jiashan listeners (r = 0.94, p < 0.001). The relationship between the three cues and the intercepts for each language group is also visually presented in Figure 7. Figure 7 is the result of a Principal Component Analysis (PCA) on the weights calculated for each language separately. PCA is a technique to reduce the dimensionality of the data by transforming possibly correlated variables into a smaller number of uncorrelated variables called ‘Principal Components’ (Jolliffe, 2002). The first principal component accounts for as much of the variability in the data as possible, and each subsequent component accounts for as much of the remaining variability as possible. Figure 7 shows that most of the variability in weights for the three dimensions and the intercept can be captured in two dimensions, i.e., the first two principle components (93.68% for Shanghai and 97.12% for Jiashan) and that the two language groups differ in how closely contour patterns with the other two cues and the magnitude of variation in the intercept.

Figure 7

Principal component analysis (PCA) for Shanghai listeners (left) and Jiashan listeners (right) cue weights in Experiment 1. Each dot represents the cue weights of one individual projected onto the first two principal components. The length of the arrows reflects the amount of variation of the term, and the angle between two arrows reflects how much the two terms are correlated (more acute means higher positive correlation; more obtuse means higher negative correlation; the closer to right angle, the more the terms are independent).

The results show that despite the fact that pitch and breathiness go together (are positively correlated across listeners) for both groups, there are different patterns for the two groups of listeners. Jiashan listeners have a competing primary cue (e.g., participant 62310 has pitch as primary cue and participant 73002 in the same figure has contour; see Figure A2 and A4 in the Appendix), where they may prefer contour or pitch, while for Shanghai listeners, both secondary cues are correlated, and are further correlated with the use of the primary cue (which is always pitch).

In summary, for Jiashan listeners, while contour is the primary cue in aggregate results, this pattern is not representative of all individual listeners. Whether or not a listener relies on contour is independent of their use of pitch and breathiness, while the importance of the other two cues is always positively correlated. For Shanghai listeners, on the other hand, all three cues are correlated, with pitch and breathiness having a stronger correlation.

3.3.2. Experiment 2

Similar to Experiment 1, this section explores individual variability and the pattern of cue usage using standard deviations reported in the random effects of the mixed-effects models and per-individual coefficient estimates predicted from the main effects (no interactions) of the fixed effects and the random effects. For the purpose of examining the two groups of participants separately, two new models were fit, one for Shanghai participants and one for Jiashan participants, each using the corresponding subset of the data.⁹ Tables 9 and 10 show the summary of random effects for Shanghai and Jiashan participants respectively.

Table 9

Summary of random effects and correlations of Shanghai participants, Experiment 2. Correlations between interaction terms are not shown.

Name	Variance	SD	Correlation
intercept	0.87	0.93
Breath	0.13	0.37	0.57
Contour	0.12	0.35	0.63	0.46
Pitch	2.13	1.46	0.31	0.48	0.23
talkerDialect	5.45	2.33	0.13	–0.18	0.49	–0.55
Breath × talkerDialect	0.35	0.59
Contour × talkerDialect	0.88	0.94
Pitch × talkerDialect	1.34	1.15

Table 10

Summary of random effects and correlations of Jiashan participants, Experiment 2. Correlations between interaction terms are not shown.

Name	Variance	SD	Correlation
intercept	0.62	0.79
Breath	0.31	0.56	–0.71
Contour	0.16	0.40	0.52	–0.02
Pitch	2.19	1.48	0.13	0.40	0.59
talkerDialect	4.45	2.11	–0.12	–0.06	–0.25	–0.61
Breath × talkerDialect	0.98	0.99
Contour × talkerDialect	0.99	0.99
Pitch × talkerDialect	1.56	1.25

As shown by the standard deviations reported in the two tables, the magnitude of individual variability is similar between the two listener groups. Note also that there is large variation in the effect of talker dialect: Listeners differ more in the effect of listening to different talkers than in the size of any cue.

In terms of correlations among cues, the positive correlation between breathiness and pitch is consistently found in both dialect groups and both experiments (Shanghai: r = 0.61, p < 0.001; Jiashan: r = 0.4, p = 0.01; correlation and significance are from the same Spearman test as Experiment 1), suggesting that listeners’ reliance on breathiness is proportional to their reliance on pitch. However, there are some differences between the two experiments. While Shanghai listeners showed a positive correlation between all three cues in Experiment 1, they only show significant correlations between contour and breathiness (r = 0.58, p < 0.001) and between pitch and breathiness in Experiment 2. There is also a trend that all correlations are smaller relative to Experiment 1, with the smallest correlation (i.e., contour and pitch) in Experiment 1 being not significant in Experiment 2. Jiashan listeners, on the other hand, display a positive correlation between pitch and contour (r = 0.69, p < 0.001), which is not found in Experiment 1. Visualization of the Experiment 2 individual variation using PCA, shown in Figure 8, further confirms the results of correlations between individuals’ cue weights. The small degree of variability in contour weight shown in Figure 8 probably arises because the checked tones in Jiashan Wu have even shorter duration than those in Shanghai Wu, which makes contour less salient.

Figure 8

Principle component analysis (PCA) for Shanghai listeners (left) and Jiashan listeners (right), Experiment 2.

In summary, pitch and breathiness are consistently correlated in a positive manner for both groups (although less strongly for Experiment 2). Different from Experiment 1, contour is not correlated with Shanghai listeners’ primary cue (i.e., pitch), but instead with the other secondary cue, breathiness. Jiashan listeners, however, display positive correlations between the primary cue and the other two secondary cues.

4.1. Jiashan listeners rely on multiple dimensions

The two experiments show that breathiness, while being a secondary cue to the multidimensional tonal register contrast, is nevertheless used in both Shanghai Wu and Jiashan Wu listeners. The difference between the two dialects is that Jiashan listeners’ perception is overall more multidimensional with relatively more similar cue weights, while Shanghai listeners’ perception is mostly dominated by pitch.

Jiashan listeners as a group have a multidimensional percept of register for the contrasts we studied. This is observed in their relatively large cue weights for the non-primary cues across both experiments, especially when listening to their own dialect. The smaller difference between the cue weights suggests that secondary cues play a bigger role in their perception than for Shanghai listeners.

Jiashan listeners also seem to be more context-dependent in how different cues are used, in this case depending on the particular tone. When comparing the cue weights for falling tones (Experiment 1) and checked tones (Experiment 2), Jiashan listeners use contour as the primary cue for falling tones, but contour is the least important cue for checked tones (likely because of the short syllable duration); meanwhile, the importance of pitch and breathiness is greater in checked tones relative to falling tones. Jiashan listeners also show context dependence in their use of secondary cues when listening to stimuli with different degrees of breathy-modal contrast (Jiashan versus Shanghai talker). Specifically, Jiashan listeners rely more on breathiness when they listened to Jiashan Wu talker—with a larger phonation contrast—than when they listened to Shanghai Wu talker—with a less distinctive contrast in this dimension. It should be noted that these two sets of stimuli vary in more than just the voice quality dimension, as they are different talkers. Nonetheless, what is relevant is that the two listener groups treat the stimuli differently. Moreover, such variation of cue weights under different contexts and speakers/dialects further shows that the secondary cues are indeed learnt from language-specific experience. Shanghai listeners, on the other hand, do not show such variation according to the degree of breathiness in the signal. In fact, their perception strategy appears to be more similar to listeners of languages with no phonation contrasts, a point which we turn to in the next section.

4.2. Shanghai listeners’ perception is dominated by pitch height

Shanghai listeners’ perception of the tonal register contrast differs from Jiashan listeners in that their perception is dominated by pitch, consistent with previous observations (e.g., Zhang & Yan, 2015). In other words, while Shanghai listeners rely on all three cues including breathiness, they exhibit an invariance in their cue weightings for all three sets of stimuli (i.e., Shanghai contour tones, Shanghai checked tones, and Jiashan checked tones): Pitch is always the primary cue and breathiness and contour are always the secondary cues. Furthermore, the secondary cues have quite small cue weights across experiments, especially when listening to the other dialect.

In terms of perception of breathiness, Shanghai listeners may be more like listeners of a language where voice quality is not a cue to lexical contrasts (the third type mentioned in Section 1.2). They seem to rely on voice quality only for the purpose of facilitating pitch location (see Kuang & Liberman, 2015; Kuang et al., 2016 for the influence of voice quality on pitch perception), while Jiashan Wu listeners behave like listeners of a typical language with a phonation contrast. Since there is limited use of breathiness in Shanghai Wu production, it is perhaps not surprising that Shanghai listeners are not very influenced by the breathiness dimension for the Shanghai talker; those stimuli likely do not have very large differences in breathiness, much like stimuli used in past studies (Zhang & Yan, 2015). What is perhaps surprising is that they also are not very influenced by the breathiness dimension for the Jiashan talker in Experiment 2. This provides evidence that these listeners may simply be less sensitive to breathiness, probably using it to facilitate pitch location, indirectly contributing to the tonal register contrast.

Another important piece of evidence supporting the different roles that voice quality plays in Shanghai and Jiashan Wu is the different influence of pitch on breathiness (the breath × pitch interaction terms in Tables 3 and 4) in the dialects. At lower pitch, breathiness has a smaller cue weight for Shanghai listeners (compared to higher pitch), but a higher weight for Jiashan listeners. This difference in direction may reflect how voice quality is treated differently in the two dialects: Breathiness, which is always associated with lower pitch, contributes less to the register categorization in Shanghai, as low pitch is already indicative for lower register; on the other hand, high pitch is likely to be perceived lower when produced with breathy voice. In this way breathiness affects the perception of tonal register. The opposite pattern in Jiashan listeners, however, reveals that listeners are more likely to perceive lower register when lower pitch is accompanied by breathiness, suggesting that breathiness is not merely facilitative for pitch location, but directly influences register categorization. Nevertheless, we also acknowledge an alternative explanation, which may require further study: The Jiashan talker has an overall higher pitch range, so breathiness may be more important to identify the lower register, as pitch itself may be ambiguous, while breathiness is less important when the pitch is already high enough to identify a higher register. It is also possible that both these factors contribute to the different pitch-breathiness relationship observed for the two groups of listeners.

We found that the Shanghai listeners were not very influenced by the contour dimension either. However, unlike breathiness which in Experiment 2 was enhanced for the Jiashan talker relative to the Shanghai talker, Experiment 2 did not provide very compelling contour cues in either set of stimuli, limiting our ability to assess use of contour by Shanghai listeners. It is also possible that there is an intrinsic difference between rising tones (heard by the Jiashan listeners of Experiment 1) and falling tones (heard by the Shanghai listeners of Experiment 1), as studies have found that Chinese listeners have smaller Just Noticeable Differences for falling tones than rising tones in terms of both pitch height and pitch contour (Jongman, Qin, Zhang, & Sereno, 2016). Comparison of different tone pairs would be necessary to understand the contribution of contour to the perception of the register contrast in Shanghai Wu.

The above evidence indicates that Shanghai Wu listeners in general rely less on breathiness than Jiashan Wu listeners (and possibly contour as well), regardless of the acoustic signals they heard. Thus, their register contrast seems to be primarily based on pitch height. The decreased cue weight of the voice quality contrast in perception reflects the on-going loss of breathiness in Shanghai speakers’ production of the tonal register contrast (Gao, 2016). One possibility is that the difference between the dialects is due to Shanghai listeners’ relatively greater contact with Mandarin which does not employ a voice quality contrast (Gao, 2016). Another possibility is that Jiashan listeners’ use of more cues in a context-dependent way compared to Shanghai listeners may be because Jiashan Wu has a relatively more complex tone inventory than Shanghai Wu, which may require multiple acoustic dimensions to distinguish between them.

4.3. Individual variability and processing of acoustic cues

We have two major findings on variability between individuals: The variability is highly structured, and there is more variability when the contrast is more multi-dimensional. First, we generally found positive correlations between coefficients of the different cues, and in many cases, these correlations were quite high (more discussion below). Second, in three of our four contrasts (Shanghai rising tones in Experiment 1 and checked tones for all listeners in Experiment 2), the coefficient of the primary cue was much larger than the other two cues in the group data. For these contrasts, the individual data (Figures A3, A5, and A6 in the Appendix) showed that almost all listeners used the same primary cue (with only a few exceptions). In the fourth contrast (Jiashan falling tones in Experiment 1), the group data found that the coefficients for the three cues were more similar, and in the individual data, listeners differed in which cue they used as primary. Inspection of the data in Figure A4 in the Appendix shows that 20/33 participants had contour as the primary cue while 11/33 had pitch as the primary cue (the last participant had an ambiguous pattern). This may be due to the falling tone contrast having a more multidimensional nature, with each of the cues providing some, and likely redundant, information. Thus, listeners may come to different solutions to the problem.

We now turn to correlations between individuals’ cue coefficients. As has been assumed in previous studies, we think it is correct to assume that group coefficient estimates indicate the degree to which the group uses particular acoustic dimensions. However, we argue that an individual’s coefficient estimates are influenced by two factors: the relative importance of a cue with respect to other cues (i.e., the relative magnitude of cues) and the individual’s ability to categorize speech stimuli consistently (i.e., overall magnitude, averaged across cues) (see also Kapnoula et al., 2017 for similar arguments). We argue that our analysis shows both of these components and that they have not been considered together before in analyses that report only group-level regression coefficients.

First, when we consider the individual-level coefficients, for both groups of listeners, whenever there is a correlation between two cues, it is always positive. We argue that positive correlations between individuals’ coefficients is the default when perception is influenced by more than one cue. This is consistent with previous findings for English showing correlations between regression coefficients for pairs of cues (Clayards, 2018; Hazan & Rosen, 1991; Kim & Clayards, 2019). Furthermore, both Hazan and Rosen (1991) and Clayards (2018) found correlations between the primary cue weights across different contrasts. Together these studies support the idea that some listeners are better able to extract and attend to acoustic phonetic details in a categorization task, regardless of the acoustic dimension. This leads to a more consistent stimulus-response relationship (i.e., sharper categorization functions) and thus larger correlation coefficients.

There are, however, several cases of no significant correlation between cues in the two experiments. Some of these are due to relatively small coefficients of the cues involved, which leads to reduced variability and thus weak correlations (e.g., correlations in Experiment 2 involving contour where contour played only a small role). The more interesting exception is the case of Jiashan listeners’ use of pitch and contour in Experiment 1. As discussed above, in this experiment, both cues were important to the contrast and there was very large individual variability in individuals’ cue weights. We think the reason for this apparent contradiction comes from the fact that listeners differed in which cue was primary, pitch or contour. Note that previous studies only examine two cues, in which case individuals rarely differ in cue ordering, but in absolute magnitude. Our study, however, focuses on three cues, which provides a venue to examine listeners’ cue weights when both cue ordering and weight magnitude are different across individuals. For the Jiashan listeners in Experiment 1, by definition, those who had contour as a primary cue had smaller cue weights for pitch than contour while those who had pitch as a primary cue had smaller cue coefficients for contour than pitch. Thus, there must be some negative relationship between the cue coefficients for these two cues. We think that this negative relationship in the relative cue coefficients may have been offset by an overall positive relationship between cue coefficients due to individual differences in response consistency (coefficient magnitude) that we observed in all of the other conditions.

In summary, we find that listeners vary in how consistently they respond to cues. Specifically, when listeners agree in cue ordering but only differ in weight magnitude, there is a clear positive correlation; when listeners differ in both cue ordering and weight magnitude, a positive correlation may be masked by their different choice of the primary cue, in which case the correlation pattern may not surface.

4.4. Limitations

One may worry whether or not the talkers chosen in the study are representative of the dialect, in other words, whether listeners’ responses are specific to the talker, or generalizable for the dialect. We acknowledge that any choice of talker will introduce talker-specific acoustic characteristics that may not be representative for the dialect, and it is hard to verify which specific acoustic dimensions are idiosyncratic. The ideal case would be to use a number of talkers that could jointly be ‘representative’ of the dialect, although this would be impractical in a single study. In order to eliminate the possibility of talker effects, a follow-up study is needed to explicitly test how listeners react to different talkers. However, even though a single talker cannot stand for the dialect, the acoustics indeed reflect the difference we expect from the two dialects: The Jiashan stimuli have a larger breathy-modal contrast and the Shanghai stimuli have a smaller contrast.