1. Introduction

Researchers in sociophonetics and variationist sociolinguistics have increasingly turned to computational methods to automate time-consuming research tasks such as data extraction (e.g., Fromont & Hay, 2012), phonetic alignment (e.g., McAuliffe, Socolof, Mihuc, Wagner, & Sonderegger, 2017; Rosenfelder, Fruehwald, Evanini, & Yuan, 2011), transcription (e.g., Reddy & Stanford, 2015), and measurement of vowels (e.g., Labov, Rosenfelder, & Freuhwald, 2013), consonants (e.g., Schuppler, Ernestus, Scharenborg, & Boves, 2011; Schuppler, van Dommelen, Koreman, & Ernestus, 2012; Sonderegger & Keshet, 2012), and suprasegmentals (e.g., Rosenberg, 2017). In the cases of forced phonetic alignment and automated transcription (both of which use machine learning methods), the technique rests on the assumption that there is some learnable, predictable pattern in the input that can be used to predict new cases; if an acoustic pattern surfaces in test data (data that the model was not exposed to in training) that the model has reliably seen associated with a certain label in training data, then a well-performing model will assign the same label to this pattern in the test data.

We propose that the same logic can apply to another common, time-consuming step in the process of conducting sociophonetic research: coding, or assigning categories (usually allophones), to variable data based on acoustic features. Complicating this process are two common properties of sociophonetic variables: acoustic complexity and gradient variability. When listeners (whether trained or lay) hear tokens of a sociophonetic variable, they do not only hear one or two salient acoustic measures, but rather a constellation of acoustic cues. Moreover, these cues’ distributions seldom divide neatly into bins corresponding to individual allophonic variants, but are rather characterized by continuous, gradient variation; this fact belies the long-standing categorical treatment of variables like /r/, a treatment that hides from researchers’ view the potential meaningfulness of tokens that exist in the gray area between cardinal variants. We thus argue that, just as human coders contend with acoustic complexity and gradient variability in making coding judgments—not always with success, as evidenced by low inter-rater reliability for variables like /r/ (see Section 3.1)—so too must an automated method for sociophonetic coding contend with acoustic complexity and gradient variability if its predictions are to have any validity.

Like the established methods of forced phonetic alignment and automated transcription, we apply a machine learning method to find learnable, predictable patterns in acoustically complex input data, in order to predict new cases—here, codes for sociophonetic variables—in test data. While several machine learning methods are likely appropriate for this application, we use the method of random forest classification (Breiman, 2001), which does not suffer from overfitting when predictors are collinear (Matsuki, Kuperman, & Van Dyke, 2016; Strobl & Zeileis, 2008); in this respect, random forests are unlike methods like generalized linear modeling (e.g., linear regression, logistic regression), for which collinearity in predictors can cause poor prediction of unseen data and misleading estimates of variable importance (Dormann et al., 2013). This property of random forests is especially important for acoustically complex variables, for which it is unclear which acoustic cues most meaningfully underlie variation; although it is not an invitation to indiscriminately include every possible acoustic parameter, this property of random forests nevertheless allows us to test large numbers of acoustic measures.

We propose that such an automated method for sociophonetic coding should differ in an important respect from the other computational methods mentioned above. In essence, those methods seek to replicate the labor of a competent human (aligning phonetic boundaries, assigning words to acoustic signals, etc.) with greater efficiency. While an automated method for coding should be able to do the same (e.g., assigning Present or Absent codes to tokens of English non-prevocalic /r/ with greater efficiency than a human could), to do so would be to ignore gradient variability. The acoustic correlates of many purportedly ‘categorical’ sociophonetic variables exist on a gradient, with acoustic correlates often exhibiting unimodal or overlapping distributions rather than sorting neatly into separate distributions by class (e.g., Love & Walker, 2013; see also Purse, 2019 for an articulatory case); this suggests that some variables themselves may also exist on a gradient, varying between tokens exemplary of one class to those exemplary of neither class to those exemplary of another class (Hashimoto, 2019; Hay & Clendon, 2012; Hay & Maclagan, 2010, 2012; Love & Walker, 2013). In essence, an automated method for sociophonetic coding should capture this gradient variability by learning from the acoustic complexity of training data.

Our research questions are:

1. Can we use machine learning methods to effectively code sociophonetic variables (in particular, New Zealand English non-prevocalic /r/ and word-medial intervocalic /t/)?

2. If machine learning methods can effectively code these sociophonetic variables, how much data is necessary to make it work?

3. How well do classifiers’ gradient predictions predict trained listeners’ coding judgments?

In the following sections, we present an automated method for sociophonetic coding that successfully meets the desiderata outlined above. We first provide a background on random forests and classification (Sections 2 and 3). Section 4 is devoted to describing the classifiers, including the results (Section 4.3) and simulations to explore the relationship between the size of the training data and classifier performance (Section 4.5). In Section 5, we describe a listening experiment that assessed the accuracy of the /r/ classifier’s predictions and the degree to which the /r/ classifier’s gradient predictions matched trained listeners’ coding judgments. We conclude with a general discussion in Section 6. Readers can find additional information in the online supplementary materials, which detail the hyperparameter settings used in the classifiers and give additional information about simulations described in Sections 4.3.2 and 4.5. Readers can also find a step-by-step guide to training random-forest classifiers (including all the data for both variables in this paper, and all the code needed to recreate the /r/ classifier) at https://nzilbb.github.io/How-to-Train-Your-Classifier/How_to_Train_Your_Classifier.html.

2. Random forests and classification

Random forests are an extension of the method of classification and regression trees, which recursively partition data into successively smaller subsets at each tree node by finding the independent variable that minimizes variation in the subbranches under the node (Breiman, 2001). While individual trees are useful for describing variation in the contexts of classification (categorical dependent variable) or regression (continuous dependent variable), they suffer from overfitting to training data and so perform poorly at predicting unseen data. Random forests are an ensemble method that grow a multitude of systematically ‘weaker’ trees—which are only permitted to select from among random subsets of independent variables at each split—that together form a consensus on the classes of previously unseen observations and which predictors are most important (Tagliamonte & Baayen, 2012). This method reduces variance in estimates without the overfitting that individual stronger trees suffer from (Breiman, 2001). Importantly, unlike modeling techniques like generalized linear modeling, random forests do not suffer from overfitting when predictors are collinear (Dormann et al., 2013; Matsuki et al., 2016; Strobl & Zeileis, 2008); as a result, collinearity does not hinder random forests’ ability to predict unseen data and to determine the relative importance of independent variables.¹ This property is particularly important for the present study, as the acoustic complexity of /r/ and /t/ led us to include many acoustic measures, including those that are naturally correlated with one another (e.g., formant values at multiple timepoints), in our random forests; indeed, we demonstrate in Section 4.3.2 that our data is characterized by a considerable degree of collinearity. We should note here that other classification methods have this property, and we do not claim that random forests are the only approach to automatically coding sociophonetic variation; as the goal of the present research is to present one method for automated sociophonetic coding and not to compare methods, we welcome future research that looks more closely into the benefits and drawbacks of different machine-learning tools for this application.

In this paper, we use random forests for three specific functions: classification (predicting the response values of observations in a test set based on the patterning of predictors and responses in a training set), feature selection (determining which predictors are most influential in classification), and gradience (mapping categorical classes to continuous predictions). (Again, these functions are not exclusive to random forests.) The classification function has been used in fields like photogrammetry (Rodriguez-Galiano, Ghimire, Rogan, Chica-Olmo, & Rigol-Sanchez, 2012) and sleep medicine (Fraiwan, Lweesy, Khasawneh, Wenz, & Dickhaus, 2012) but not, to our knowledge, in phonetics or sociolinguistics. In a related approach, German, Carlson, and Pierrehumbert (2013) used optimal discriminant analysis to calculate speaker-specific F3 thresholds for distinguishing retroflex versus flapped realizations of English /r/.

The feature selection function of random forests has been used in several studies of cue weighting (often with acoustic cues). For example, Brown, Winter, Idemaru, and Grawunder (2014) explored which of nine phonetic cues were most predictive of Korean honorific judgments for both Korean and English listeners; Al-Tamimi (2017) contrasted 13 acoustic correlates of pharyngealization in Jordanian and Moroccan Arabic; and Baumann and Winter (2018) explored 17 prosodic and non-prosodic cues affecting perceptions of prominence. In a slightly different function, Tagliamonte and Baayen (2012) used random forests to compare the importance of linguistic and extralinguistic factors on was/were variation in York English, finding that individual variation outranked all other predictors in importance.

Finally, in the gradience function of machine-learning models like random forests, a model trained on categorical data generates continuous, probabilistic predictions, which can potentially be used to predict listener responses on new data. This function is related to our earlier observation that the acoustic correlates of many purportedly ‘categorical’ sociophonetic variables exist on a gradient (see Section 1); if the continuous nature of these correlates indeed accrues to the gradience of the variables themselves—which is by no means a given—then these models can predict this gradience. For example, van Alphen and Smits (2004) use individual regression trees to predict the likelihood that listeners hear tokens of Dutch labial and alveolar plosives as voiced (although they do not apply these probabilistic predictions to new tokens).

Within sociolinguistics, we are aware of only one study that attempts to use the gradience function of machine-learning models; McLarty, Jones, and Hall (2019) train a support vector machine (SVM) classifier of African American Language non-prevocalic /r/s. Using 36 mel-frequency cepstral coefficients as acoustic measures, the authors obtained binary classifications and continuous measures of how far tokens were from the decision boundary for non-prevocalic /r/s (defined as the vowel+/r/ sequence, as in the present research). However, their classifier was trained on oral vowels (representing the Absent class) and onset /r/s (representing the Present class). We note that this approach rests on the premise that the acoustic continuum of variation in non-prevocalic /r/s is identical to the continuum of variation between onset /r/s and vowels; there are several reasons to doubt this premise. First, even a fully constricted non-prevocalic /r/ will take up less of the time-course of its token than an onset /r/; this dissimilarity between onset and Present coda /r/ is problematic given the coarse temporal resolution—three time points—of the acoustic measures fed into this classifier. Second, there is no oral vowel analogue for the Absent versions of contexts like SQUARE; it is not clear how well a classifier whose training set of Absent tokens does not include [ɛə] could classify such tokens as Absent. We are skeptical that a classifier whose training and test sets differed in such a fundamental way could succeed, regardless of the specific machine-learning implementation used, the size of the training set, or the number of acoustic measures. Because the authors neither report cross-validation results on non-prevocalic /r/ data with known codes nor compare the classifier’s predictions against human coders, it is not possible to validate the predictions of the classifier that they report.

3. Background: /r/ and /t/

Our classifier efforts focused on two sociophonetic variables: non-prevocalic /r/ and medial /t/. Both of these variables are prime candidates for this method: Variants of these variables are sociolinguistically meaningful in New Zealand English (NZE) (and numerous Englishes worldwide), making the automation of coding advantageous for research; and they are acoustically complex, meaning for neither variable is there a single acoustic measure that can stand in as a proxy for its variability. In this section, we discuss each variable’s acoustic features and sociolinguistic context in NZE, as well as our corpora.

4. Building classifiers

We trained random forest classifiers in an attempt to automate coding for our two sociophonetic variables: non-prevocalic /r/ and medial /t/. Using training sets of several thousand hand-coded tokens (/r/: 4,689; /t/: 4,218), we trained classifiers on large numbers of acoustic measures (180 measures of /r/, 113 measures of /t/). Our question is whether this method is viable for automatically coding these variables. In this section, we first outline the data used for this automatic classification, before outlining the acoustic measures that were extracted from each token and the performance metric that we used to tune the classifiers.

4.3. Results

In this section, we discuss the final classifiers’ performance, variable importance, and auto-coding.

4.3.1. Classifier performance

Table 1 displays performance measures for the classifiers. The overall accuracy for the /r/ classifier, 84.5%, compares favorably with human coders’ inter-rater reliability for coding rhoticity, as noted in Section 3.3. It is also worth noting that the class accuracy for Present lagged Absent, as the classifier correctly coded 62.2% of actual Present tokens as Present, compared to correctly coding 93.1% of actual Absent tokens as Absent. This result is not surprising given that Absent tokens outweighed Present by roughly 2.6 to 1, as the classifier had fewer Present tokens in its training set.

Table 1

Classifier results. Baseline performance metrics reflecting a naive classifier (one always choosing the majority class) are given in parentheses; the .5000 baseline for AUC assumes performance equal to a coin flip.

Performance measure	/r/ classifier	Six-class /t/ classifier	Two-class /t/ classifier
Overall accuracy × AUC	.7423 (.3608)	.7107 (.1962)	.8918 (.2527)
Overall accuracy	.8447 (.7217)	.7847 (.3924)	.9182 (.5055)
AUC	.8786 (.5000)	.9056 (.5000)	.9711 (.5000)
Class accuracies	Absent 93.1%; Present 62.2%	[ɾ] 91.9%; [s] 92.7%; [d] 20.1%; [t] 37.3%; [ʔ] 47.5%; [∅] 0.0%	Voiced 92.2%; Voiceless 91.5%

The six-class medial /t/ classifier achieved high class accuracies for the majority classes [ɾ s] but dismal accuracies for the minority classes [d t ʔ ∅]; in fact, none of actual [∅] tokens were correctly coded as [∅]. This classifier’s confusion matrix (Table 2) shows that the minority classes were consistently miscoded as the majority classes (e.g., most [d] were miscoded as [ɾ], most [t] as [s]), indicating that the sizes of these classes contributed to the classifier’s performance in identifying them. Clearly, this six-class classifier would be inadequate for deploying across a corpus, especially if it is important to capture rare but sociolinguistically salient variants. For example, [t] is a conservative variant in NZE, so it is useful to differentiate [t] from other variants to measure the time-course of sociolinguistic change in a corpus, but less than 40% of hand-coded [t]s were successfully detected by the classifier.¹⁰ The poor performance of this six-class classifier motivated the attempt to treat medial /t/ as a two-class variable (following Clark, 2018; Hay & Foulkes, 2016).

Table 2

Confusion matrix for six-class medial /t/ classifier (percentages of overall data). Actual classes in columns, predicted classes in rows.

	ɾ	s	d	t	ʔ	∅
ɾ	36.2	2.1	6.4	0.3	1.0	1.4
s	1.9	36.2	0.4	3.7	0.2	0.2
d	0.7	0.2	1.9	0.5	0.3	0.1
t	0.2	0.5	0.5	2.7	0.2	0.0
ʔ	0.4	0.0	0.3	0.0	1.5	0.0
∅	0.0	0.0	0.0	0.0	0.0	0.0

By contrast, the two-class /t/ classifier performed very well, with nearly identical class accuracies for Voiced versus Voiceless. Again, to the extent that classifier performance on a given class is related to the size of that class, this result is not surprising, as the Voiced and Voiceless classes were nearly equal in size. However, this increased performance comes at the cost of failing to capture rare but sociolinguistically salient variants like [t], as with the six-class /t/ classifier.

4.3.2. Acoustic measures

Figure 1 displays the 20 most important acoustic measures for the /r/ and two-class /t/ classifiers, in order of decreasing importance, as measured by the Gini index (see the online classifier-training tutorial for more details).¹¹ For /r/, nearly all of these measures involved the difference between (raw) F3 and F1 (11 measures) or F3 and F2 (6 measures). Figure 2 demonstrates that F3–F1 was most important just after the midpoint, whereas F3–F2 was most important toward the end of the token. Also represented in the most important measures for /r/ were token duration, intensities at F3 minimum and F3 maximum, and speaker-normalized F3 minimum.

Figure 1

Top 20 most important acoustic measures (Gini index) for the non-prevocalic /r/ and two-class medial /t/ classifiers.

Figure 2

Variable importance for selected groups of time-varying acoustic measures for the non-prevocalic /r/ and two-class medial /t/ classifiers.

For the two-class /t/ classifier, the most important acoustic measures likewise cluster toward the end of the token, including spectral moments (center of gravity: 8 measures; kurtosis: 4) and voicing (7); speaker-normalized token duration is also among these top 20 measures. These measures indicate that for this variable, the phonological categorization of these classes as ‘voiced’ and ‘voiceless’ (Clark, 2018; Hay & Foulkes, 2016) is only partially rooted in phonetic voicing (periodicity); instead, Voiced and Voiceless variants are distinguished by spectral characteristics that characterize fricatives.

As mentioned in Section 2, random forests do not suffer from overfitting when predictors are collinear (Matsuki et al., 2016; Strobl & Zeileis, 2008). As a check on collinearity, we calculated pairwise correlations for the acoustic measures in Figure 1. Out of the 190 pairwise correlations per variable, 64 correlations for /r/ and 49 for /t/ exceeded the 0.7 rule-of-thumb threshold for ‘severe’ collinearity (Dormann et al., 2013). A technique like generalized linear modeling that is sensitive to collinearity would clearly not be appropriate for this data.

Moreover, there is some evidence that this technique produces reliable measures of variable importance. In simulations with subsets of the /r/ and two-class /t/ data, we found high correlations between the variable importance scores (rs > .7728), indicating stability between estimates of variable importance (even with much smaller datasets). (Details of these simulations can be found in the online supplementary materials.¹²) This result validates our use of random forests for sociophonetic applications of feature selection, such as determining which acoustic features are most influential in classifying variants of sociophonetic variables like /r/ and /t/.

4.3.3. Auto-coding data

After training the classifiers, we used them to predict binary classes and classifier probabilities for 27,516 /r/ tokens and 4,888 /t/ tokens that had not been previously hand-coded. In terms of binary predictions, the predicted distribution of /t/ in the auto-coded test set (Voiced 52.6%; Voiceless 47.4%) closely matched the observed distribution in the /t/ training set (Voiced 50.5%; Voiceless 49.5%), whereas the predicted distribution of /r/ in the auto-coded test set (Absent 79.5%; Present 20.5%) was somewhat more skewed toward Absent than the observed distribution in the training set (Absent 72.2%; Present 27.8%). Figure 3 displays the distributions of classifier probabilities underlying these predictions. For both variables, the distributions are similar for training and test sets, with a lower (more Absent-like) mode in the /r/ training set than the /r/ test set that may indicate greater differentiation between Absent and Present classes in the training set than the test set. However, there is a noticeable difference between the unimodal distributions for /r/ and the sharply bimodal distributions for /t/.

Figure 3

Distribution of classifier probabilities in training set and test set, for /r/ and two-class /t/ classifiers (greater classifier probability = more Present and more Voiced, respectively). Dotted vertical lines denote binary classification cutoffs for each classifier.

We present these comparisons between the classifier probability distributions of training sets and test sets with a caveat: Classifier probabilities are generated via slightly different processes for training versus test sets. For example, the /r/ classifier used repeated k-fold cross validation, meaning the test set is exposed to the entire classifier whereas the training set is exposed to just a subset. As a result, while there is no reason not to combine hand-coded and auto-coded binary codes, we caution against commingling training set and test set classifier probabilities in the same data.

4.5. How much data is necessary?

We conducted an analysis of sample sizes to explore how the size and consistency of the training set affects the performance of the classifier. Practically speaking, if a researcher wants to use a random forest classifier on sociophonetic variation in their own data, they will need to know how much data to hand-code in order to train a classifier that will yield reliable predictions on non-hand-coded data. We carried out this analysis via simulations in which we re-ran the final /r/ and binary /t/ classifiers at sample sizes between 100 and 1,200 (with 100 random subsamples per sample size). Details of the implementation of these simulations can be found in the online supplementary materials. While it is impossible to come up with a ‘magic number’ minimum sample size that will work for all variables and any research question, we present these results as a general guide to future users of the classifier method.

As Figure 4 indicates, some classifiers trained on as few as 100 tokens matched or even eclipsed the performance of the classifiers trained on the full data sets of over 4,200 tokens. As sample size increases, not only does performance improve on average, performance becomes more reliable. Improvement is more pronounced for the /t/ classifiers than the /r/ classifiers. However, for both /r/ and /t/ the average performance of classifiers run on 1,200 tokens sits well below the performance of the classifiers run on over 4,200 tokens, suggesting that more data is a sure way to improve classifier performance (albeit with diminishing returns).

Figure 4

Overall accuracy × AUC for /r/ and /t/ simulations, 100 classifiers at each sample size. Dotted line marks performance of best classifier for each variable.

Additional simulations of /r/ controlling for whether the Present and Absent classes are internally homogeneous or heterogeneous furthermore suggest that the homogeneity of the training set is associated with classifier success (plots in the online supplementary materials). When the minority Present class is homogeneous (i.e., characterized by less acoustic variability), greater classifier performance and greater class accuracies can be achieved with smaller samples. This finding suggests that the classifier has an easier time discriminating between classes when there is less variation in one class.

5. Experimental assessment of classifier predictions

In order to assess the accuracy of the /r/ classifier’s predictions, and also the degree to which continuous variability in classifier probability might be sociolinguistically meaningful, we conducted a small-scale perception task. We chose to investigate the /r/ classifier and not the /t/ classifier because of the apparent gradience of /r/ and categoricity of /t/; moreover, the scarcity of /t/ tokens with intermediate classifier probabilities would make stimulus selection difficult.

Stimuli were selected from among classifier-coded tokens to represent a range of classifier probabilities and to control for additional independent factors that affect /r/ in this community. Preliminary mixed-effects modeling of production in Southland /r/ (including both hand-coded and auto-coded tokens) uncovered several significant linguistic and social factors; this analysis indicated that experimental stimuli needed to be controlled for gender, stress, morphological status, preceding vowel, and following segment. Rather than generate a complex model with interactions of all possible control factors, we chose to set each factor constant. We thus restricted our stimuli to tokens uttered by men in stressed syllables in content words, with preceding NURSE and a following sonorant. To control for any effects of metrical structure, we selected only monosyllables. Out of 27,516 classifier-coded tokens, this gave us a stimulus population of 330. Sixty tokens were then selected to span the range of classifier probabilities, with 31 tokens coded Absent and 29 coded Present by the classifier. Stimuli were created by extracting the relevant word from the audio file, resampling the word to 22,050 Hz, and scaling the word’s intensity to 70 Pa.

Eleven phonetically trained listeners were asked to judge each stimulus as Present or Absent and to rate their confidence on a scale from 1 to 5. Listeners heard each stimulus word twice, with a 750 ms buffer between repetitions. Listeners performed the task with headphones on individual computers. All listeners were proficient English users who lived in a predominantly non-rhotic country at the time of the experiment. Five listeners self-reported speaking English with rhotic accents, six with non-rhotic accents; seven listeners self-reported as native speakers, four as non-native speakers.

5.2. Results

5.2.1. Judgments

The baseline model of judgments revealed a significant positive effect of classifier probability on trained listeners’ judgments (β = 3.64, z = 5.64, p < .0001), indicating that stimuli with greater classifier probabilities were more likely to be judged Present; this relationship is evident in Figure 5. Following segment was not significant (ps > .56). This baseline model proved to be the best model of human judgments. The model with a restricted cubic spline for classifier probability failed to significantly improve the model fit (χ²[1] = 0.10, p = .76). Models that added listener rhoticity (χ²[2] = 1.45, p = .48) or listener native speakerhood (χ²[2] = 2.80, p = .25) also failed to significantly improve fit. While these nonsignificant results nominally fail to indicate either nonlinearities in the relationship between classifier probability and human judgments, or an effect of these particular dimensions of listener experience on judgments, they must be interpreted with caution; the relatively small sample size and small number of listeners means that the models have low power, especially when testing for effects of properties of speakers (Brysbaert & Stevens, 2018; Judd, Westfall, & Kenny, 2017; Kirby & Sonderegger, 2018; Westfall, Kenny, & Judd, 2014).¹⁴

Figure 5

Classifier probability versus trained listeners’ coding judgments for each stimulus (dots); fitted effects line and 95% confidence-interval bands calculated by R effects package (Fox & Weisberg, 2019) from best model of judgments.

5.2.2. Confidence

The model of confidence ratings revealed no significant main effect of classifier probability on confidence ratings, with the intercept term corresponding to the Absent judgment (β = –0.16, t[33.43] = –0.44, p = .66); however, there was a significant interaction effect of classifier probability with the Present judgment (β = 1.78, t[462.58] = 4.09, p < .0001). In other words, listeners were more confident in judging stimuli with greater classifier probabilities as Present, but there is no evidence that classifier probability had an effect on listeners’ confidence in Absent judgments. The model additionally revealed a significant main effect of Present judgments (β = –0.88, t[476.83] = –3.26, p < .005), indicating that listeners were less confident in judging tokens with low classifier probabilities Present than Absent. These relationships are evident in Figure 6.

Figure 6

Classifier probability versus mean confidence rating for stimuli judged Absent versus Present (darker dots received more judgments); fitted effects lines and 95% confidence-interval bands calculated by R effects package (Fox & Weisberg, 2019) from best model of confidence.

As with judgment, the baseline models of confidence revealed no significant effect of following segment on confidence (ps > .24). The baseline models were also the best models of confidence, as adding a restricted cubic spline for classifier probability, listener rhoticity, and listener native speakerhood failed to significantly improve the baseline confidence model (ps > .69). As with the judgment models, these results nominally fail to indicate nonlinearities in the relationship between classifier probability and confidence, or that these particular dimensions of listener experience affected confidence ratings—but we again urge caution in not over-interpreting these null results given the likelihood of low power in this model (Brysbaert & Stevens, 2018; Judd et al., 2017; Kirby & Sonderegger, 2018; Westfall et al., 2014).¹⁵

5.2.3. Classifier probability versus individual measures

As mentioned in Section 3.1, sociophonetic modeling of /r/ as a continuous phenomenon (i.e., beyond the Present/Absent binary) typically treats F3 minimum as an index of rhoticity (Hay & Maclagan, 2012; Love & Walker, 2013). It is an empirical question, then, whether F3 minimum significantly predicts human judgments in the same way that classifier probability does—and if so, which measure predicts human judgments better. To test this question, we ran six additional models of judgments: three models with an individual acoustic measure (centered and scaled) instead of classifier probability, and three with both classifier probability and an individual measure. Two measures were F3 minimum: the raw F3 minimum, and the speaker-normalized F3 minimum that was entered into the classifier. Since the latter was ranked only 14th out of 180 variables in importance by the classifier, we also assessed the top-ranked variable in importance (see Section 4.3.2): the difference between F3 and F1 at 55% of the token’s duration. To parallel the models reported in Section 5.2.1, these models included random slopes of the acoustic measure by listener that were uncorrelated with the listener random intercept, even when the correlated slopes did not result in a singular fit; in no case did removing this correlation significantly hurt the model fit.

In all of these models, the individual measure significantly or near-significantly affected trained listeners’ coding judgments in the expected (negative) direction (Table 3); for example, tokens with greater raw F3 minimum were significantly less likely to be judged Present. However, likelihood ratio tests revealed that adding classifier probability to these models resulted in significantly better fits; for example, the model with raw F3 minimum plus classifier probability was significantly better than the model with just raw F3 minimum (χ²[2] = 9.27, p < .01).¹⁶ In other words, classifier probability captured something about human judgments above and beyond what these individual measures can capture.

Table 3

Significance of main predictors (centered and scaled) in individual-measure judgment models. “Log-lik 1/2” refers to log-likelihoods of models without/with classifier probability, respectively.

Main predictor	Estimate	Log-lik 1	Log-lik 2	Likelihood ratio test
Raw F3 minimum	β = –0.80, z = –4.39, p < .0001	–386.87	–382.24	χ2(2) = 9.27, p < .01
Normalized F3 minimum	β = –0.67, z = –3.73, p < .001	–390.68	–382.52	χ2(2) = 16.33, p < .001
F3–F1 at 55% duration	β = –0.45, z = –2.68, p < .01	–395.35	–383.04	χ2(2) = 24.62, p < .0001

We should note that the stimulus sample was chosen to systematically represent a range of classifier probabilities, while we did not specifically sample for F3 minimum and other individual measures. To control for this, we resampled the stimuli such that the shape of the classifier probability distribution more closely matched the shape of the raw F3 minimum distribution, and we repeated the above comparison with models of this resampled data set. The models with classifier probability and individual measures were significantly better than the models with just individual measures for normalized F3 minimum (χ²[2] = 9.22, p < .01) and F3–F1 at 55% duration (χ²[2] = 24.16, p < .0001), but not for raw F3 minimum (χ²[2] = 4.79, p = .09).

6. General discussion and conclusion

In this paper, we have discussed the method of random forest classification as a means to automatically code sociophonetic variables, which is a welcome alternative to the tedious, time-consuming process of hand-coding. We have validated this method by showing that binary classifiers of two sociophonetic variables of English (non-prevocalic /r/ and word-medial intervocalic /t/) perform well in cross-validation and that the /r/ classifier’s gradient predictions reflect variability in trained listeners’ coding judgments. This method also sheds light on the low-level acoustic properties that best characterize variation in /r/ and /t/ in New Zealand English; contrary to the common practice of describing gradient variation in /r/ via F3 minimum, our /r/ classifier indicates that several measures involving differences between F3 and F1 or F2 at various timepoints are more informative in classification than F3 minimum. Over and above the importance of any individual acoustic measures, we find that classifier probability, as a composite of numerous acoustic measures, more accurately modeled trained listeners’ coding judgments of /r/.

While it remains to be seen to what extent this finding of meaningful gradience in /r/ extends to other variables, this method raises the possibility of transforming a binary treatment of sociolinguistic variation to a continuous index of variability. This continuous treatment more accurately reflects the fact that individual tokens lie on an acoustic continuum that is not adequately captured by merely sorting tokens into Present versus Absent bins. As discussed in Section 3.1, there is growing evidence for treating /r/ as meaningfully gradient, focusing specifically on F3 minimum (Hashimoto, 2019; Hay & Clendon, 2012; Hay & Maclagan, 2010, 2012; Love & Walker, 2013). Moreover, as an editor points out, these findings questioning the assumption of categoricity in /r/ dovetail with recent evidence questioning the categoricity of English coronal stop deletion based on both acoustic (Temple, 2014) and articulatory (Purse, 2019) approaches. In light of this external evidence that gradience in English /r/ (and possibly other variables) is phonetically and sociolinguistically meaningful, the random forest classifier method represents a potential means to operationalize ‘categorical’ variation as gradient (at least where rhoticity is concerned) beyond individual acoustic correlates.

We further argue that treating ‘categorical’ sociophonetic variables as gradient surmounts another hurdle typically faced by hand-coding these variables as ‘categorical’: poor inter-rater reliability (see Section 3.3). Given a set of tokens, some listeners will assign larger or smaller numbers of tokens to one category or the other, in part due to asymmetries of expectation and experience among different listeners (Hay et al., 2018). In other words, while some tokens of a variable may clearly, unambiguously belong to a single class, for a considerable number of tokens there is no unambiguous ground truth. This is additional evidence in favor of reconceptualizing ‘categorical’ variation in probabilistic, continuous terms—and the listening experiment suggests that our classifier model successfully facilitates this reconceptualization. Out of 60 tokens, only one was categorically perceived as Absent by 11 listeners, and only one was categorically perceived as Present; the classifier was accurate in assigning these tokens very low and very high probabilities, respectively. Crucially, the tokens that listeners perceived in the gray area were assigned middling probabilities by the classifier. In a typical sociophonetic coding situation, tokens with such low levels of inter-coder agreement would be seen as problematic, requiring extensive intervention to help coders arrive at an agreed-upon ‘correct’ class; a continuous treatment, by contrast, recognizes the potential sociolinguistic meaningfulness of these gray area tokens. Lending weight to this interpretation is Hall-Lew and Fix’s (2012) finding that the /l/ tokens that experienced the greatest disagreement among coders were those that were, on average, judged to be intermediate between vocalized and consonantal realizations. Moreover, it is worth remembering that the hand-coded classes that were used to train the classifier were generated by a single human rater (albeit a well-trained rater), but the probabilities generated by the model trained on this single rater’s coding judgments nevertheless accurately predicted group patterns among a larger set of trained coders. These results suggest, as an empirical question, that this gradient metric may be a more reliable measure of rhoticity than any individual rater’s binary ratings alone.

A gradient operationalization is not necessarily appropriate for all variables, however, as indicated by the differences in classifier probabilities for auto-coded /r/ versus /t/ tokens. While the distribution of classifier probabilities for auto-coded /r/ tokens was unimodal, with most tokens in the gray area between Absent and Present, auto-coded /t/ tokens’ classifier probabilities were bimodally distributed, with most tokens clustered toward being highly Voiceless-like or highly Voiced-like. As a result, the probabilities returned by the /t/ classifier are less likely to meaningfully represent gradient variation between two endpoints than the /r/ classifier probabilities. In other words, while the /t/ classifier is well suited to generating binary predictions, the /r/ classifier’s predictive capabilities are better suited to generating classifier probabilities. However, this does not eliminate the possibility that the /t/ classifier may be capturing interesting gradience within the two /t/ categories; some Voiced tokens, for example, are associated with a higher probability of voicing than others. It remains for further work to determine whether this variation is perceptually or linguistically meaningful.

Although we present what we find to be compelling proof-of-concept evidence for the use of random forest classifiers for automated coding of sociophonetic variation, we emphasize that readers should exercise caution when deploying this method on their own datasets.¹⁸ A classifier’s predictions are only as good as the coder(s) who create the training data, meaning that (as with traditional hand-coding) there is always an element of subjectivity involved. Even with unimpeachable training data, readers’ mileage may vary with respect to classifier performance, partially as a function of the different properties of different variables. Indeed, while binary classifiers of /r/ and /t/ performed well, the /t/ classifier with six classes (some quite rare in the data) failed to perform up to acceptable levels, suggesting room for improvement in accounting for minority classes. Random forests’ ability to handle collinear data, while useful for variables for which it is unclear which acoustic cues most meaningfully underlie variation, is also not an invitation to indiscriminately include every possible acoustic parameter in the model. In short, random forest classifiers are no ‘magic wand.’ As a result, we urge readers to conduct checks on the outputs of their classifiers (perhaps conducting their own listening experiments) before using this method to auto-code new tokens, assess variable importance, and/or generate gradient predictions.

Indeed, our intent with this paper is certainly not to have the last word on this method. While promising, our finding that the classifier method works for two variables (both of which have large training sets) in one variety is not an indication that this method is suitable for use in all conditions. Although a comparison of methods is outside the scope of this paper, machine-learning implementations of classifiers other than random forests may prove to be more appropriate for this application; indeed, as an editor suggests, a bottom-up clustering method may be more appropriate than a classification approach. It is further possible that, as anonymous reviewers suggest, classifiers trained on a smaller number of composite measures common in signal processing (e.g., discrete cosine transforms and mel-frequency cepstral coefficients) may outperform models trained on large numbers of ‘traditional’ acoustic-phonetic measures. Finally, while we intentionally avoided introducing social or linguistic factors above the level of phonetics directly into our classifiers to prevent over-learning of top-down extra-phonetic features (see Section 4.2.1), it remains to be seen if extra-phonetic features can find their way indirectly into the model regardless (i.e., via the acoustic measures we did introduce into the model). As these are all rich areas for future study, our online classifier-training tutorial (https://nzilbb.github.io/How-to-Train-Your-Classifier/How_to_Train_Your_Classifier.html) contains the /r/ and /t/ data, and all the code needed to recreate the /r/ classifier reported in this paper, to allow researchers to explore and expand on this method on their own.

In conclusion, the method of random forest classification provides us with the ability to automatically code vast quantities of sociolinguistic variables with accuracy comparable to that of a trained linguist, opening research questions that would otherwise be time-prohibitive to pursue. To the extent that variables are meaningfully gradient (which remains an open question), this method also provides a means to capture that gradience more comprehensively than is possible in any individual acoustic measure. We argue that this method represents a potentially powerful new method in the sociolinguistic toolkit.

Competing Interests

The authors have no competing interests to declare.