1.1 Background

Early studies on gesture-prosody synchronisation were interested in the synchronisation of intervals in these modalities such as strokes (meaningful gesture parts with maximum dynamic effort) and (stressed) syllables (Creider, 1986; McClave, 1991; McNeill, 1992; Tuite, 1993, amongst others). McNeill’s (1992) phonological synchrony rule summarises the general claim of these studies: “…the stroke of the gesture precedes or ends at, but does not follow, the phonological peak syllable of speech” (p. 26). In other words, prominences in gesture and prosody were claimed to be synchronised. Later, re-analysing McNeill’s data, Nobe (1996) refined this rule into a stroke-acoustic peak synchrony rule, which stated that strokes co-occur with (or precede) F0 and intensity peaks. Following on from this refinement, many later studies have focused on the synchronisation of atomic landmarks (i.e., smallest possible anchors) rather than intervals, testing whether prominent points in time in gesture and prosody are synchronised (see Türk, 2020, for an overview).

In these studies, the prominent point in time in gesture has been commonly identified as the apex, which is dynamically the most prominent instant corresponding to the target of gesture (e.g., endpoint as in Figure 1), and/or abrupt directional changes in the gesture stroke (Loehr, 2004; Shattuck-Hufnagel, Yasinnik, Veilleux, & Renwick, 2007) (see Section 2.4.1).

Figure 1

Two frames showing a deictic gesture apex.

Defining prosodic prominence and selecting measurable cues, on the other hand, have been fundamental problems for studies on apex synchronisation, and in many cases, the cues used were not informed by phonological theories of prominence. Experimental studies have often assumed phonetic definitions of what prominence is. They characterised it with single acoustic parameters measured over stressed (prominent) syllables such as articulatory vocalic targets (Roustan & Dohen, 2010), jaw openings (Rochet-Capellan, Laboissière, Galván, & Schwartz, 2008), or vowel onsets or midpoints (Leonard & Cummins, 2011; Rusiewicz, 2010; Rusiewicz, Shaiman, Iverson, & Szuminsky, 2013) without providing much insight into why those parameters were chosen. This view is in conflict with prosodic research, which has established a complex relationship of acoustic cues to prominence where multiple cues are shown to contribute to the perception of prominence (Breen, Fedorenko, Wagner, & Gibson, 2010; Cole, 2015; Arnhold & Kyröläinen, 2017; Baumann & Winter, 2018; Kügler & Calhoun, 2020). The cues to prominence depend on multiple factors such as the language in question and the position in prosodic structure. Accordingly, if particular points in time need to be measured for apex synchronisation tests, this measure should not ignore prosodic structure and how different acoustic cues are used in the language in question.

In addition to using different units of measure, previous studies have also adopted different methodologies to define and test synchronisation. In studies concerned with the synchronisation of intervals, the temporal overlaps of intervals were interpreted as synchronisation, regardless of the durational difference between the intervals and the duration of the overlap (e.g., syllables and strokes as in Jannedy & Mendoza-Denton, 2005; Shattuck-Hufnagel & Ren, 2018). Studies on the synchronisation of precise time points tended to use actual time distance measurements (Loehr, 2004; Rochet-Capellan et al., 2008; Roustan & Dohen, 2010; Rusiewicz, 2010; Leonard & Cummins, 2011; Rusiewicz et al., 2013). In some of these, synchronisation was determined as a relative temporal proximity in binary conditions where production of speech and gesture was heavily controlled (e.g., Rochet-Capellan et al., 2008; Rusiewicz, 2010). Namely, the time distances between apexes and phonetic/prosodic anchors were measured over words with varying stress positions (e.g., stress on the first vs. the second syllable). If these distances were significantly less in one condition and more in the other, then this reduction was considered as synchronisation. In other studies, the closest pre-selected phonetic/prosodic anchor to an apex was considered to be in synchrony with it (e.g., Roustan & Dohen, 2010). Such an approach, in general, is indicative of synchronisation but less absolute as it disregards the actual time distance between anchors (e.g., two units may be closest but in fact seconds away from each other). Loehr’s (2004) approach counters this by considering two closest events (i.e., apexes and pitch accents) synchronised only if they occur within a pre-defined duration (i.e., 275 ms). This approach seems to be more plausible in that it provides a standard timeframe to base synchronisation interpretations on. It is also tolerant of timing differences potentially caused by other gestural and prosodic factors, which in general has not been explored in depth so far.

In fact, emerging evidence on synchronisation suggests that prosodic phrasing might be one such prosodic factor affecting gesture synchronisation. In English, gesture phrases (i.e., single meaningful unit of bodily action, Kendon, 2004) were found to be synchronised with single intermediate phrases, meaning that gestures are sensitive to prosodic boundaries (Loehr, 2004). Krivokapić, Tiede, and Tyrone (2017) showed that manual gestures lengthen under prominence and at prosodic boundaries, which potentially implies an effect on synchronisation with co-occurring prosodic units. Esteve-Gibert and Prieto (2013) and Esteve-Gibert, Borràs-Comes, Asor, Swerts, and Prieto (2017) showed that prosodic structure shapes the patterns of synchronisation as the positions of apexes and pitch accents were found to change in tandem, depending on the prosodic phrasing. Rohrer, Prieto, and Delais-Roussarie (2019) showed that in French, apexes were synchronised with pitch accents and that this synchronisation was more with (accentual) phrase-final pitch accents than phrase-initial ones, although both can be used to mark prominence. Interestingly, they also noted that synchronisations with apexes were at much lower rates compared to the rates reported in studies in English (Jannedy & Mendoza-Denton, 2005; Shattuck-Hufnagel & Ren, 2018). Moreover, in cases of asynchronisation, apexes coincided with phrase-initial positions that contain no pitch accentuation. Their interpretation of this pattern was that apexes might be synchronised with phrase onsets marking a prosodic domain independently from the encoding of prominence. The prospect of apexes synchronising with boundaries is interesting and requires further exploration as many previous studies tested apex synchronisation with the prior assumption that apexes are only synchronised with prosodic prominence, as mentioned above. Following on from this, it might also be that in addition to phrasing, prosodic hierarchy and the relative positioning of phrases with regard to sentence prominence (e.g., nuclear vs. pre-nuclear, see Section 1.2) may also affect synchronisation patterns of apexes, which has not been investigated systematically so far.

Similarly, there might be gestural factors that can affect apex synchronisation. For instance, the semantic functions of gesture (i.e., iconics, metaphorics, deictics, and beats, see Section 2.4.1) have been mostly overlooked in many previous studies on apex synchronisation. They either did not distinguish between any gesture types in their analysis (e.g., Loehr, 2004; Jannedy & Mendoza-Denton, 2005), or only investigated apex synchronisation for one or two gesture types (Rochet-Capellan et al., 2008; Roustan & Dohen, 2010; Rusiewicz, 2010; Leonard & Cummins, 2011; Rusiewicz et al., 2013; Esteve-Gibert & Prieto, 2013; Esteve-Gibert et al., 2017). Most of these studies have analysed the synchronisation of either deictics (i.e., pointing gestures) or beats (i.e., simple flicks of hand) separately with the exception of Roustan and Dohen (2010). They compared apex synchronisation patterns of beats and deictics and found different synchronisation anchors for these where deictic apexes were synchronised with articulatory vocalic targets and beat apexes with prosodic peaks. This implies that gesture type can impact apex synchronisation. However, to our knowledge, there are not any other studies that focus on comparing apex synchronisation across different gesture types, especially for representational gestures (i.e., iconics and metaphorics – gestures that represent concrete or abstract concepts in speech). Overall, all of these highlight that gesture and prosody may be connected in more ways than previously assumed. These also imply that in terms of synchronisation, gesture and prosody must be treated as complex systems where the continuous stream of gestural and prosodic events and their organisations can influence each other.

Despite their methodological differences, however, most of the previous studies have reported a prominence-based synchronisation between gesture and prosody. That is, in these studies, dynamically prominent apexes were claimed to be synchronised with certain events that were considered to bear prominence (though see Rusiewicz, 2010; Rusiewicz et al., 2013). This indicates a consistent relationship between categories in phonology and gesture. Given this synchronisation, multimodal cues are likely to be part of the implementation of phonological structure at different levels of planning (Türk, 2020). Therefore, they could play a role in how intonational categories are produced and represented by speakers, and identified and perceived by listeners (e.g., Krahmer & Swerts, 2007 for evidence of the effects of visual beat gestures on the perception of prosodic prominence). Furthermore, this strong relationship can also aid us as analysts. Taking advantage of tight synchronisation patterns, multimodal cues can be used to assist phonological analyses in the identification of phonological categories. In particular, if apexes are systematically synchronised with only prominence lending events, then this can be used for the identification of such phonological events in cases where there may be disagreement in literature.

The present study builds on these interpretations and predictions. Its primary goal is to show there are consistent multimodal cues to intonational phonological categories in Turkish. For this purpose, the present study interrogates the claim that atomic prominent events in gesture (i.e., apexes) and in prosody are synchronised. This investigation diverges from others in multiple ways: (1) How it selects prosodic anchors for synchronisation, (2) by accounting for the effect of gestural and prosodic context on synchronisation, and (3) how it determines synchronisation. Further, apex synchronisation has only been tested for a limited number of languages with similar intonational structures, excluding Turkish. Therefore, investigating synchronisation in Turkish enables cross-linguistic comparison with previous studies. Turkish also provides an interesting challenge to the prominence-based synchronisation claim as not every prosodic phrase is claimed to contain a prominent event (Section 1.2.1), making apex synchronisation in these cases unclear. Finally, the present study also demonstrates how any consistent multimodal cues to intonational categories can be exploited to inform phonological analyses (Section 1.2.2). In the next section, all of these points are detailed.

1.2 Present study

The present study aims to test whether gesture and prosody are synchronised at their prominent units, as claimed previously, or whether there are other anchors in prosody that gesture can be synchronised with in natural speech data. Therefore, the way in which anchors in gesture and prosody are defined is crucial for establishing meaningful synchronisation relationships. The present study diverges from previous studies in how it selects these anchors. It does not presume that there can be a synchronisation relationship only between prominent units, so it does not test synchronisation just between two pre-selected anchors. Instead, it adopts the apex as the prominent gestural anchor (Section 2.4.1) and aims to find the best synchronisation anchor amongst a set of prosodic events that have prosodic functions within Turkish prosody. The advantage of this approach is that any prosodic event, not just prominent ones, can be synchronised with apexes, which enables the discovery of other potentially crucial synchronisation patterns.

Research has shown that multiple acoustic cues contribute to the perception of prominence. It has also been shown that these cues cannot be expected to be the same for every syllable, regardless of (1) position in prosodic structure, and (2) how those acoustic cues are used in that language (Breen et al., 2010; Arnhold & Kyröläinen, 2017; Baumann & Winter, 2018; Kügler & Calhoun, 2020). Therefore, the prosodic anchors that are tested for synchronisation should be sensitive to these factors (Section 1.1). The present study uses F0 minima and maxima (i.e., F0 turning points or tonal events) as prosodic anchors. These have been shown to be sensitive to both of these factors, and to be associated with syllables even though they are not necessarily aligned within them (Prieto, Van Santen, & Hirschberg, 1995; Arvaniti, Ladd, & Mennen, 1998; Xu, 1998; Arvaniti, Ladd, & Mennen, 2000; Atterer & Ladd, 2004; Prieto & Torreira, 2007; Ladd, 2008).

Tonal events have different functions within prosodic structure. Within the Autosegmental-Metrical (AM) framework of intonational phonology, intonation contours are construed as a sequence of high (H) and low (L) tonal targets (i.e., tonal events). These tonal events either mark prominence (i.e., pitch accents) or mark the boundaries of prosodic phrases (i.e., phrase accents or boundary tones). Within the present study, the analysis of synchronisation considered any tonal event, regardless of function, as a potential anchor for an apex. This meant that any consistent observation of synchronisation would be the result of a genuine systematic behaviour. The inventory and functions of tonal events are language specific. The next section introduces the intonational structure of Turkish.

1.2.1 Turkish phonology

To the authors’ knowledge, there are no current complete intonational descriptions for Turkish within the AM framework but only partial descriptions (Özge & Bozsahin, 2010; Kamali, 2011; Ipek & Jun, 2013; Güneş, 2013, 2015). These earlier studies employed read-out tasks of pre-set sentences in their analyses, and they differ clearly from each other in their representation of Turkish tonal inventory. In the present study, it was not possible to follow any of these descriptions entirely because the study uses spontaneous natural speech data. There were many cases where the existing descriptions fell short or starkly conflicted with our naturalistic data. Therefore, the lack of consensus on the descriptions of Turkish phonology and uncontrolled variability introduced by our data led us to develop a new annotation scheme. The scheme takes earlier studies in consideration but diverges from them in the form of simplifications and generalisations that explained our data best (these will be touched upon in Section 2.4.2). In particular, where there were disagreements between previous descriptions, and/or our data did not fit any of the previous schemes, we adopted a broad phonetic approach to be reasonably transparent to the phonetic cues at the surface level (see Hualde & Prieto, 2016 and references therein). This approach can be seen as a form of transcription “…that includes a certain amount of redundant, phonologically non-contrastive detail that is nevertheless a systematic aspect of the language” (Hualde & Prieto, 2016, p. 3). It aims to be surface-transparent, and is precisely useful in cases like Turkish where the correct phonological analysis is not yet known or is disputed. We follow this approach also in order to account for non-predictable phenomena introduced by the nature of our data. Our annotation scheme can be considered as a low-level prosodic transcription that does not aim to describe the full range of phonological units and contrasts, but to capture prosodic representations that are transparent at the surface/phonetic level (see Kügler et al., 2015 for a scheme for German with a similar motivation). We do not claim to be exhaustive in our description nor is it the only analysis possible. Our focus was on representing the common intonational phenomena found in our data even if they did not fit any of the existing descriptions. In general, however, the scheme is more similar to Ipek and Jun’s (2013) descriptions than others. In the rest of this section, Turkish intonation is introduced briefly (see further Section 2.4.2) in order to demonstrate the potential tonal event candidates for apex synchronisation and what consistent synchronisation patterns with these candidates imply for Turkish intonation (Section 1.2.2).

In Turkish, there are three levels of prosodic phrasing, each usually associated with a corresponding level of morphosyntactic structure: (1) Prosodic word (PW) (≈ morphological word), (2) intermediate phrase (ip) (≈ syntactic phrase), and (3) intonational phrase (IP) (≈ syntactic clause). Each of these is marked by specific tonal events. The IP is the largest phrase in the hierarchy, and it is marked by either a low or high boundary tone at the right edge (L% or H%). Similar to the IP, the ip is marked at the right edge with a phrase accent (L- or H-). The PW, on the other hand, is marked with a L tone at its left edge. The only other tonal event within the PW is the pitch accent (H*, !H*, or L*). Pitch accents are associated with the prominent syllables of PWs. A basic schematic for the prosodic structure of Turkish and an example pitch track are shown in Figure 2 (Section 2.4.2 for details).

Figure 2

A schematic for the prosodic structure of Turkish shown on the utterance “There are kitchen cupboards above the fridge” and its pitch track.

Figure 2 shows that in Turkish, there can be multiple prominence-lending and boundary marking tonal events occurring near each other (PW initial L tones, pitch and phrase accents, and boundary tones). Moreover, the realisations of these tonal events are sensitive to the organisation of prosodic structure. This is observed in the differences in tonal marking due to the position of ips relative to the ip that contains sentence level prosodic prominence, the nuclear ip (nip). For instance, pre-nuclear ips (prips) are usually marked with a H- phrase accent, whereas nuclear and post-nuclear ips (ptips) are usually marked with a L-, and the pitch range of pitch accents in nips is substantially compressed (!H*) (see Figure 2b).

The present study tests whether apexes are synchronised with any of these events with different functions and positions in the prosodic organisation. By adopting tonal events as anchors for apexes, we are able to test whether apexes are indeed synchronised with prominent pitch accents when other types of tonal events (e.g., boundary-marking events) are included in the analysis. We also test whether apex synchronisation shows any difference depending on the organisation of prosodic phrases, which has not been reflected in earlier studies on gesture-prosody synchronisation. Turkish presents an interesting prosodic context to test synchronisation as there can be multiple tonal events occurring close to each other within a short duration in PWs. This crowding is potentially a curious challenge for synchronisation as it generates noise for the mechanism that manages synchronisation, especially in naturalistic data such as is used in the present study. However, consistent synchronisation with one of these tonal events over others in such conditions would indicate a systematic selection of particular tonal events for apex synchronisation, which resists contextual effects.

Such systematic selection of anchors for synchronisation is inherently a cueing behaviour. For instance, if apexes are synchronised with pitch accents, as earlier studies would predict, then the locations of pitch accents can be predicted from the visual signal alone provided that there is a gesture accompanying speech. In other words, the systematic behaviour of apexes can be used in the identification of tonal events they are associated with, assisting phonological analyses. It is possible to show how apex synchronisation can shed light on intonational categories in a case study of a disputed feature related to accenting in Turkish.

1.2.2 A case study on accentlessness in Turkish

As noted in Section 1.1, earlier studies on apex synchronisation claimed a synchronisation of prominences in gesture and prosody. This is interrogated in Turkish in the present study. Turkish presents an interesting ground for this interrogation since its prosodic structure poses a challenge for this claim — there can be prosodic phrases without a pitch accent in Turkish. In such cases, there would be no tonal events that encode prominence, and therefore, there are no targets for apexes to be synchronised with. For instance, post-nuclear ips in Turkish do not contain pitch accents (see İşsever, 2003; Özge & Bozsahin, 2010; Section 2.4.2; and also Kabak & Vogel, 2001 for other prosodic processes that can lead to deaccentuation). Another example of phrases with no prominence pertains to lexical stress in Turkish. Two types of lexical stress are prescribed in Turkish: (1) Regular stress where the stress is on the PW-final syllable, and (2) irregular stress where the stress is on a non-final syllable (Sezer, 1981; Kabak & Vogel, 2001). In some studies, a pitch accent is associated with both types of lexical stress (Ipek & Jun, 2013). However, in others, words with regular stress are claimed to be accentless (Levi, 2005; Kamali, 2011; Güneş, 2013). In this case study, we concentrate on this claim and show how apex synchronisation can be employed to inform us about the presence or absence of accents in pre-nuclear ips in Turkish.

Pre-nuclear ips (prips) typically end in a pitch rise in Turkish (Figure 2b). In cases of regular stress, this rise is hard to dissociate from the hypothesised word-final pitch accent, which is also associated with a pitch rise. In other words, word-final pitch accents and phrase-final phrase accents coincide in words with regular stress, and therefore, it is unclear what the status is of the rise in pitch (i.e., whether the H tone is a part of the pitch accent or the phrase accent). There are two views regarding this issue. Ipek and Jun (2013) suggests that the H tone functions both as a part of the pitch accent and as part of the phrase accent. On the other hand, Kamali (2011) claims that the H tone is a property of the ip only, which in turn proposes that words with regular stress are accentless in Turkish (see Section 2.4.2 for details).

We claim that apex synchronisation can be used to shed light on this disagreement. In these prips, the synchronisation options (if at all) are limited to the PW-initial L and the H tone (see Figure 3 for an illustration). The prominence-based synchronisation claim, which the present study interrogates, predicts that apexes would be synchronised with prominent pitch accents since the L tones are only demarcative and do not lend prominence (see Section 2.4.2). Assuming that this claim is true for Turkish as well (which is tested first in this study), there are three possibilities:

1. If apexes are synchronised with the H tone (see Figure 3a), this implies that the H tone may actually be a part of the pitch accent as well as the phrase accent, supporting Ipek and Jun’s (2013) double function claim. The presence of a prominent pitch accent at the location of the rise attracts apexes.

2. If apexes are synchronised with PW-initial L tones instead (see Figure 3b), this synchronisation pattern would also imply that synchronisation is not managed by prominence only. In the absence of pitch accents associated with PWs, apexes may be synchronised with PW-initial L tones which are also associated with PWs. This would suggest that apex synchronisation is sensitive to the prosodic hierarchy because synchronisation occurs only between the members of the PW. Namely, apexes do not synchronise with tonal events that are markers of higher level constituents such as phrase accents and boundary tones.

3. If apexes do not show any discernible synchronisation pattern (either the L tones or the H tone), this implies that there is no prominent pitch accent with which the apex can be synchronised. Therefore, synchronisation cannot be reliably predicted. This would support that the H tone is part of the phrase accent only, and not prominence-lending (Kamali, 2011; Güneş, 2013, 2015). This would also suggest no effect of prosodic hierarchy, as apex synchronisation floats between boundary events marking constituents at different levels.

Figure 3

Two hypothetical gesture apex synchronisation cases with L or H on the word üzerinde ‘above’ in Figure 2b.

As can be inferred from these, consistent synchronisation with either of these tonal events (or lack thereof) can be used to identify whether words with regular stress are accentless or not in Turkish. Moreover, (a) synchronisation has implications for which levels in their organisation gesture and prosody are coupled with while also exploring apex synchronisation behaviour in cases where there is no prescribed anchor.

To summarise, the present study tests whether apexes are synchronised with pitch accents or with other tonal events in Turkish. The analysis examines synchronisation in different prosodic (phrase types such as pre-nuclear and nuclear) and gestural contexts (gesture types such as deictic and iconic) accounting for their potential effects on apex-tonal event timings (see Section 1.1). Following, systematic coordination of apexes with these events is used to determine what their role is as multimodal cues in the representation and identification of intonational categories in Turkish. Based on its findings, the study then moves on to demonstrate how these multimodal cues can be utilised in phonological analyses focusing on the disagreement about accentless words in Turkish.

1.3 What constitutes synchronisation?

Section 1.2 stated the aims of the present study and clarified the anchors in gesture and prosody for synchronisation. In this section, we introduce what constitutes synchronisation within this study while making comparisons with approaches adopted in earlier studies.

First, unlike many earlier studies (Section 1.1), the present study does not define a single pre-specified target (e.g., a target accented/edge syllable) with which apexes are assumed to be synchronised. Instead, apexes are free to be synchronised with any event in the prosodic signal, enabling us to capture the range of synchronisation behaviours in spontaneous data. Second, the study is interested in the synchronisation of points in time, not intervals (e.g., syllables and strokes). The analyses of synchronisation through overlaps of intervals with potentially incomparable durations can lead to less precise conceptualisation of synchronisation. For instance, the stroke of an iconic gesture can easily reach one second in duration, and comparing the overlap of any one syllable with such a stroke is not likely to return a meaningful result in general. This is further highlighted if any amount of overlap between intervals is considered as synchronisation. For example, with such an approach, only 1ms of overlap between a syllable and a stroke would count as synchronisation, although most of the syllable would fall outside of the stroke (the stroke would align with many other syllables, too). In contrast, an analysis of synchronisation of gestural points in time (i.e., apexes) with points in time that are associable with (but do not fully equate to) intonational categories (e.g., tonal events) provides a more refined look into gesture-prosody synchronisation at the smallest unit level, as intended in the present study.

Apexes and tonal events only need to consistently co-occur near each other as synchronisation is generally defined as the systematic co-occurrence of units in time. This, on the other hand, necessitates a measurable definition of “near” — how near these units should be in order to be considered synchronised. There is no set number in the literature that shows how near apexes and tonal events should be in order to be considered synchronised. However, there have been other studies that approached synchronisation in a similar manner (Section 1.1). For instance, Loehr (2004) considered two anchors synchronised only if they occurred within a pre-defined duration, 275ms, which was the average duration between gestural and prosodic markings in his data.

The present study adopts the same approach to synchronisation but uses the average syllable duration, 160ms, as its synchronisation criterion. This means that if the time distance between an apex and the nearest tonal event to it is less than the average syllable duration then they are considered synchronised. This duration was selected because it is phonologically meaningful: The syllable is the smallest phonological unit that can carry prominence, and prominence is the main concept that gesture synchronisation has been built on so far (Section 1.1). Therefore, it is plausible to look for temporal relationships within this duration. The use of such a duration is also plausible from the perspective of studies on the perception of synchronisation. For instance, Leonard and Cummins (2011) reported that apexes that occur up to 200ms before target syllables can still be perceived as prosodically aligned, which supports using an even shorter duration in the present study. To reiterate, 160ms is a duration that only describes what “near” is in the present study and not a claim of synchronisation within any one syllable. We argue that apexes and tonal events need not occur within the same syllable to indicate a cueing behaviour. Otherwise, an analysis can unnecessarily predict asynchronisation when the actual time distance between two anchors is as short as 10ms if there happens to be a syllable boundary between them. Moreover, tonal events themselves do not need to align within the syllables to which they are related (Prieto et al., 1995; Arvaniti et al., 2000). Therefore, points in time in another modality should also not be strictly expected to align within syllables in order to signal (temporal) relationships, especially in spontaneous speech.

Having a fixed duration as a synchronisation criterion enables statistical analyses (Section 2.5) where this duration sets a tolerance zone only large enough to predict meaningful timing relationships, ignoring spurious effects that might cause slight asynchronies. In addition, this makes it possible to determine the effect of gestural and prosodic contexts on synchronisation. As implied in Sections 1.1 and 1.2, we hypothesise that tone types (e.g., L, H*), prosodic phrase types (e.g., pre-nuclear, nuclear), and gesture types (e.g., iconic, deictic) may affect the time distances between apexes and tonal events. Further, the present study also considered information structure as a speech component that might have an effect on apex-tonal event synchronisation due to the strong link between information structure and prosody (see Kügler & Calhoun, 2020 for an overview). Accordingly, the study also tested whether apex synchronisation is sensitive to information structural categories such as topic, focus, contrast, and givenness (see Götze et al., 2007 for definitions). However, no such effects were observed; therefore, this will not be reported further in this paper.

3.1 Which tones do apexes tend to be paired with?

A pairing consists of an apex and the tonal event that is nearest to it. In this section, we report whether apexes were paired with certain types of tones more than others, indicating a pattern. In Section 2.4.2, tonal events marking prominence or boundaries in Turkish were described (e.g., H-, H*). To give a general look into the data and the pairing behaviour, we first group them under four main categories (one for prominence and three for prosodic phrases): Pitch accents, phrase accents, boundary tones, and prosodic word-initial low tones (L hereon). These are referred to as tone types. Grouped by tone type, Table 2 shows the total number of tones annotated for each type (Annotated N), what number/percentage of these annotated tones were paired with apexes (Paired N/%), and what percentage they constituted out of 820 pairing instances (Frequency %).

In the data, 22% of tones were paired with apexes within intonational phrases that were gestured; 36% of pitch accents were paired with apexes, and these pairings constituted nearly half (46%) of all pairing instances. Most apexes were paired with pitch accents, closely followed by Ls; 21% of Ls were paired, which constituted 34% of all pairing instances. Based on these numbers, it is possible to argue that apexes tended to be paired with pitch accents as well as with Ls. Therefore, the claim that apexes are synchronised only with pitch accents (as the only prominence markers) was not fully supported here. However, in line with our arguments in Section 1.1, it is possible that the pairing patterns (hence synchronisation) is affected by prosodic and gestural contexts — there might be different modes of pairings that alternate depending on these contexts within which tones and apexes existed.

Our next analysis looked for potential modes of pairing and revealed a bimodal distribution where apexes were paired with pitch accents when a pitch accent was available in the PWs. However, when there was no pitch accent available, Ls were greatly favoured for pairings (not every phrase that a gesture may accompany has a pitch accent, see Section 1.2.1). In fact, 80% of the pairing instances with Ls in the data occurred when there was no pitch accent in the PWs. To demonstrate, Table 3 breaks down the distribution in Table 2 by whether or not there was a pitch accent in the PW that the paired tone was in. The bimodal distribution sensitive to the presence of a prominence marker is evident in the table. This change in the pairing patterns establishes an effect of prosodic context on apex timing behaviour.

Apex pairings with Ls marking PW onsets imply that apexes are sensitive to prosodic phrasing. Note that both pitch accents and Ls are associated with the PW (Section 2.4.2). The pairing preference shifts from one to the other, depending on the accentlessness of the PW — apexes were not paired with phrase accents and boundary tones, which are associated with phrases at higher levels in the prosodic hierarchy. This suggests that apex synchronisation is also sensitive to prosodic hierarchy.

Our analysis also tested whether the pairing patterns change depending on: (1) Subcategories of four tonal events (e.g., L- vs. H- and ^H- for phrase accents, or !H* vs. H* for pitch accents including final/non-final distinction); (2) Available tones in the prosodic phrases (e.g., L H* H- vs. L H- vs. L); (3) Intermediate phrase type that the paired tone is in (e.g., pre-nuclear ip vs. nuclear vs. post-nuclear); (4) Gesture types (e.g., iconic vs. deictic). However, we observed no sensitivity to any of these factors.

A reviewer noted that the association of 21% of apexes with phrase accents and boundary tones seems surprisingly high (Table 2). In fact, some earlier studies reported ∼10-20% of alignment with unpredicted anchors (e.g., 17% in Loehr, 2004) and some did not report what happens in cases of non-alignment with the pre-selected anchor (e.g., ∼17% non-alignment with stressed syllables in Shattuck-Hufnagel & Ren, 2018). We argue that the number of pairings with these was too low to constitute an overall pattern and that this much variation is to be expected in studies using natural data.

To summarise, there is a bimodal pairing pattern that alternates between pitch accents and Ls depending only on the accenting of phrases. This finding has implications for accented/accentless pre-nuclear intermediate phrases in Turkish (Sections 1.2.2 and 2.4.2). This will be addressed in Section 3.3 together with the findings of the synchronisation analysis in the next section, where we report whether the paired apexes and tonal events achieved synchronisation given the synchronisation criterion adopted in the present study.

3.2 Are apexes synchronised with their nearest tone?

The findings in Section 3.1 have shown pitch accents and Ls as the preferred targets of apex pairings, and phrase/boundary tones as dispreferred targets. As explained in Section 2.5, pairings indicate a proximity and semantic based relation of an apex with a tonal event. Synchronisation, on the other hand, deals with the actual measurements of time distances between paired units. In Section 2.5, the present study introduced a synchronisation criterion in order to define what it considers “synchronised”. Namely, if the time distance between the paired units is less than the average syllable duration (160ms), then the pairing achieves synchronisation; if not, the members of the pairings are not synchronised. In this section, the present study first gives an overview of measured time distances between paired apexes and tonal events. Then, it tests whether these distances were affected by prosodic and gestural factors (Section 2.5), and finally tests whether the pairings achieved synchronisation given the effects of these factors.

3.2.1 Overview of time distances between pairs

The pairing patterns presented in Section 3.1 indicated preferences depending on the tone type. It may also be the case that the time distance between the paired units may be greater or smaller depending on the tone type. In order to get a general view, Figure 10 shows normalised (scaled) distributions of time distances between the paired units for each tone type. Phrase accent and boundary tone pairings were grouped together because annotations of boundary tones coincide with phrase accents (Section 2.4.2), and there were not enough observations for these tones to enable meaningful comparisons separately.

Figure 10

Time-normalized histograms of the time distances of the nearest tone from an apex.

In the figures, the mean distance for pitch accents and Ls were very close to zero. The distribution for phrase/boundary tones showed a higher mean and standard deviation, exhibiting a lead of tones over apexes about a syllable duration (i.e., tones occur before their paired apexes). For each type condition, most of the observations were between ±200ms range. The distribution for pitch accents in Figure 10a showed the most compact peak followed by Ls in Figure 10b, which had a slight spread into the positive direction on the x-axis. The distribution for phrase/boundary tones had smaller peaks further away from the main peak, reaching time distances as far as 800ms. Following on this, it was checked whether these small peaks and spreads were indicators of bimodality using Hartigan’s dip test (Hartigan, Hartigan, et al., 1985). However, no significant evidence of bimodality was found (a: D = 0.018, p = 0.481; b: D = 0.015, p = 0.935; c: D = 0.016, p = 0.992).

Overall, these findings agree with the pairing preferences in that the time distances between apexes and the preferred tonal targets (pitch accents and Ls) were closer to zero than the dispreferred targets (phrase accent and boundary tones). Therefore, the type of tones involved in the pairings seems to be a factor that affects the time distances between paired units. In Section 1.1, we also predicted that there might be gestural, prosodic, and information structural contexts that can affect these measured time distances. The next section reports our findings of this analysis.

3.2.4 Effect of tone type

In Figure 11, the equivalence test output is plotted for the significant effect of tone type (see Table 5) where the dashed lines indicate the equivalence bounds at ±160ms. Note that the results of this simple effect on the time distances present a general view of synchronisation for apexes and tones, showing the overall (i.e., standard) synchronisation pattern of apexes and tonal events. This is because the estimates plotted here were averaged over the levels of gesture type and ip type effects (Sections 3.2.5, and 3.2.6) since the tone type was the common variable in all significant terms.

Figure 11

The estimated means of tone type term with the CIs at 95%. The dashed lines are the upper (160ms) and lower (-160ms) equivalence bounds.

Table 5

The equivalence test results for apex-tone pairings for each tone type.

	t	df	p
Pitch.acc	6.226	373	<0.001*
L	4.109	216	<0.001*
Phrasal	–2.001	126	0.976

As can be seen in the figure, the estimates for pitch accents and Ls were one to two frames away from zero (–9ms and –34ms). However, the estimate for phrasal tones was at –229ms, falling beyond the lower bound. Consequently, the equivalence test results showed that the time distances of pairings with pitch accents and Ls were statistically equivalent to zero, and therefore synchronisation was achieved. However, the same was not true for phrasal tones. These tones tended to start more than an average syllable duration earlier than apexes. Therefore, no equivalence was observed: Apexes were not synchronised with these tones when they were paired with them.

3.2.5 Effect of gesture type

Figure 12 plots the equivalence test output for the significant interaction of tone type and gesture type. The estimates of phrasal tones in iconics (N = 225) and deictics (N = 280) showed an average lead for tones of about –252ms in iconics and –226ms in deictics not fitting between the equivalence bounds. For pitch accent and L pairings, the tone lead was a lot less with –11ms/–43ms in iconics and –9ms/35ms in deictics. From the equivalence test, it can be concluded that for iconic and deictic gesture apexes, the time distances of pairings with pitch accents and Ls were statistically equivalent to zero (see Table 6), meaning that apexes were synchronised with these tonal events. However, for phrasal tones, the synchronisation was not achieved.

Figure 12

The estimated means of the Tone type: Gesture type interaction with confidence interval at 95%. The dashed lines are the upper (160ms) and lower (–160ms) equivalence bounds.

Table 6

The equivalence test results for apex-tone pairings in metaphorics and beats.

	t	df	p
(a) Iconics
Pitch.acc	5.139	115	<0.001*
L	3.665	74	<0.001*
Phrasal	–2.167	33	0.982
(b) Deictics
Pitch.acc	6.226	159	<0.001*
L	4.109	66	<0.001*
Phrasal	–2.001	52	0.975
(c) Metaphorics
Pitch.acc	2.164	31	0.018*
L	3.175	24	<0.001*
Phrasal	1.537	24	0.067
(d) Beats
Pitch.acc	3.395	65	<0.001*
L	2.937	49	<0.01*
Phrasal	2.103	15	<0.05*

For metaphoric (N = 82) and beat apexes (N = 132), all estimates were within the equivalence bounds ranging between –89ms and 58ms, and the equivalence test were significant for all conditions except for the pairings of the apexes of metaphoric gestures with phrasal tones (see Table 6). These findings meant that beat apexes were synchronised with their nearest tone regardless of type. For metaphorics, the pairings with pitch accents and Ls satisfied the synchronisation criterion, but phrasal tones did not because the lower bound was crossed for these (see Table 12). However, despite this, the present study interprets metaphoric apex pairings with phrasal tones as a successful synchronisation because the bound was crossed by only 23ms, and most of the confidence interval fell within the bounds. We deem a deviation as small as almost one video-frame length (i.e., 17ms) as acceptable in this case. Therefore, it was concluded that the apex-tone pairings for metaphoric and beat gestures achieved synchronisation in the present study.

Taken together, the equivalence test results revealed two patterns of synchronisation with tone type depending on gesture type. The synchronisation in iconics and deictics mirrored the pairing preference — pitch accents and Ls were the preferred targets of apexes for pairing, and these pairings achieved synchronisation. The dispreferred targets (i.e., phrasal tones) were not synchronised with apexes when they were paired. On the other hand, this situation was not the same for the apexes of metaphorics and beats. The pairing preference was not mirrored in synchronisation since it was achieved with all tonal events.

In line with Section 3.2.4, the present study interprets the synchronisation pattern in iconics and deictics as the general apex synchronisation behaviour (i.e., in line with overall findings in Section 3.1), and the pattern in metaphorics and beats as a different pattern that results from the rhythmic behaviour of apexes observed in metaphorics and beats in the data.

Beats occurring consecutively in clusters, as in Figure 13 (see Loehr, 2004 and references therein), impose rhythmic apex productions. Similarly, metaphorics occurred mostly when participants wanted to express meta-narrative content (McNeill, 1992), such as uncertainty or repetition, which were expressed with jerky gestures, causing multiple consecutive rhythmic apexes. The production of apexes in such rhythmic sequences can force pairings with the dispreferred tones since apexes have to occur at a location that is imposed by the rhythm, and the nearest tone to that location will form a pairing with that apex regardless of preference. These series of apex productions follow their own rhythm, but usually at least one of the apexes in the series (often the very first one) tended to show synchronisation with a pitch accent (see McClave, 1994 for similar observations). In terms of synchronisation, the time distances between apexes and tonal events are likely to be shorter in these rhythmic productions because Turkish can offer several tonal events over short durations as potential targets. Consecutive apex productions at fixed distances are able to find a tone occurring nearby in every case, ensuring synchronisation.

Figure 13

An ELAN screenshot showing that rhythmic pattern of apexes in consecutive beat gestures.

Overall, the metaphoric/beat apex synchronisations differed from the general synchronisation pattern observed for iconic and deictic apexes as a result of these common rhythmic formations of apexes over short periods of time. For metaphorics, we do not claim that the pattern we observed is representative of the gesture type as a whole. It is only the case that in our data, they behaved more like beats and displayed rhythmic behaviour within their formation (a single apex vs. repeating apexes in series). This implies that form-related properties of gestures can impact synchronisation as well. We leave such an analysis for a future study.

3.2.6 Effect of ip type

The final significant effect was the tone type and ip type interaction. Figure 14 plots the equivalence test output for this effect. For nuclear ips (nips, N = 344), the estimates of pitch accents and Ls were only about one frame duration away from zero (20ms, –18ms respectively). The phrasal tone estimate was slightly further away from zero with –76ms. Accordingly, the equivalence test confirmed that all observed time distances under this condition were statistically equal to zero (see Table 7). For pre-nuclear ips (prips, N = 375), the estimated means of pitch accents and Ls were again closer to zero – they were only one to two frames away from it (–9ms and –34ms respectively). However, the phrasal tone estimate was much further away with –229ms, and fell outside of the lower bound. Consequently, the equivalence test was not significant for phrasal tones. However, the results were significant for pitch accents and Ls, achieving synchronisation.

Figure 14

The estimated means of the Tone type: ip type interaction with CIs at 95%. The dashed lines are the upper (160ms) and lower (-160ms) equivalence bounds.

Table 7

The equivalence test results for apex-tone pairings in pre-nuclear and nuclear ips.

	t	df	p
(a) Pre-nuclear
Pitch.acc	6.226	151	<0.001*
L	4.109	141	<0.001*
Phrasal	–2.001	80	0.976
(b) Nuclear
Pitch.acc	6.490	221	<0.001*
L	3.811	74	<0.001*
Phrasal	2.214	46	<0.05

The expected pattern was observed for prips where the synchronisation results reflected the preference indicated in the pairing pattern: Pitch accents and Ls were synchronised with the apexes they were paired with. The less common pairings with phrasal tones exhibited a lead for tones with a distance that was more than the average syllable duration, and therefore these failed to synchronise. Unlike in prips, the apex-tone synchronisation was successful for all pairings in nips. One possible explanation might be that the nuclear prominence had an effect on the distance between the members of the pairings. Regardless of the tone type, apexes were more tightly coupled with tones if they were in the nuclear area. This effect was not clearly observed for pitch accents and Ls as these were already tightly synchronised with apexes. However, the phrasal tone estimate moved almost an average syllable duration (153ms) closer to zero in nips compared to prips, achieving synchronisation. The effect of nuclearity presented here can be seen as evidence that the apex-tone synchronisation is sensitive to phrasal prominence as well.

3.3 The case of accenting in pre-nuclear ips

In Section 1.2.2, we introduced a disagreement in earlier studies on Turkish phonology about whether accentlessness is a feature of PWs with word-final stress. In the case of prips which are marked with a high tone at the right edge, some argued that this final rise is not a pitch accent and only marks the end of the prip; whereas others claimed that this rise has a double function marking both a pitch accent and the end of the prip. We argued that the pairing patterns and synchronisation of apexes can bring some insight into this discussion. In what follows, we concentrate on accented and accentless prips in our data, examining in more detail the pairing patterns in these phrases and the hypothesis they support. We will also see that the findings from this analysis are consistent with those shown in the synchronization analysis in Section 3.2.

Regarding the apex pairings in prips, the present study has hypothesised three scenarios which could inform us about the accenting of these ips. One hypothesis was that if the final rise has a double function then the apexes should be attracted to this rise, pairing with the phrase accent and the pitch accent claimed to be there (Figure 15a). This would mean that all prips contain a pitch accent. The second hypothesis was that there is no double function of the final rise, and therefore these phrases can lack prominence, in which case the apexes should be attracted to Ls (Figure 15b) as the only other tonal event associated at the level of the PW. This would also mean that apex synchronisation is sensitive to prosodic phrasing and hierarchy. The third hypothesis was that apexes will not show any clear synchronisation when the phrases were without a prominence marker — prominence is the only attractor for apexes. In what follows, we concentrate on accented and accentless prips in our data, examining pairing and synchronisation patterns in these phrases respectively, and then discuss which hypothesis they support.

Figure 15

Two hypothetical gesture apex synchronisation cases (modified from Figure 3).

There were 405 prips containing a pitch accent in the data (out of 648), and 69% (N = 278) of these contained tonal event paired with apexes. Table 8a shows the pairing pattern of apexes in these prips (boundary tones were again grouped together with phrase accents). As seen in the table, the pairing preference was consistent: Apexes tended to be paired with pitch accents when the phrases were deemed to contain a pitch accent in our annotation which was independent from the annotation of apexes (Section 2.4).

Here, we also checked whether the pairing pattern in accented prips was affected by the location of the pitch accent (final versus non-final syllable, see Section 1.2.2). The position of pitch accents within PWs could potentially have an effect on the distributions because in the non-final condition, pitch accents would be further away in time from the phrase accents at the edges of prips compared to the prips with word-final accents (compare the distance of the apex to the final rise in Figures 16a and 16b). This increased distance in time amplifies the preferentiality of tones for pairing. That is, if apexes are paired with pitch accents even when they were further separated from the ip-final phrase accents, this would support that they were not attracted to the ip-final rises per se but to the pitch accents.

Figure 16

The apexes pairing with the tones in accented prips.

Table 8b breaks down Table 8a by whether the pitch accents in prips were word-final or non-final. The table shows the pairing instances where there was only one of each tone type available for pairing in the phrase (i.e., one L, one pitch accent, one phrase accent) to account for a potential effect of other available tonal events in the phrases. No major effect of pitch accent location on the pairing preference could be observed in the table — the preference was pitch accents in both conditions. As shown in the pairing examples in Figures 16a and 16b, when pitch accents moved away from the phrase final rises (non-final condition), the apex locations tended to move away from the prip final rises along with pitch accents. This finding reinforces the claim that pitch accents and apexes are tightly coupled in that apex coordination is responsive to the shifts of accent locations within words. Similarly, the findings also show that apexes are not necessarily attracted to acoustic peaks themselves but to the prominence they cue (Section 1.1). Phrase accents and boundary tones in prips (and also generally) often presented higher pitch values than pitch accents (e.g., ^H- and L-H%, see Section 2.4.2), yet apexes were not paired with these.

In prips without a pitch accent, the phrase final rises still persist in the absence of pitch accents, marking the phrase boundary. Therefore, apexes occurring during these prips could be paired with either Ls or H- phrase accents (see Figure 17). There were 244 such accentless prips in our data (out of 648), and only 39% (N = 96) of these contained tones that were paired with apexes. This revealed the first difference between accented and accentless prips, which was that accentless prips were gestured less frequently than the accented ones — phrases bearing prosodic prominence attracted gestures (also observed across ip types, i.e., prips/nips vs. ptips, see Section 3.2.6).

Figure 17

An apex pairing within an accentless prip.

As shown in Table 9, the dispreference for phrase-final rises was apparent in accentless prips. Apexes tended to be paired with Ls rather than with pitch rises (H-X%) in these prips, in line with the pairing preference reported in Section 3.1 (see Figure 17 for an example). In these cases, the apex locations consistently shifted away from the phrase-final rises towards Ls. This is an example of a systematic behaviour of apexes, rejecting the third hypothesis which predicted that apexes would not show any consistent synchronisation when there is no marker of prominence (Section 1.2.2). Moreover, apexes did not stay anchored at the pitch rises. This indicates that there were no pitch accents at the location of the phrase final rises, which would result in more pairings with H-X%. Instead, the pairing preference shifted to Ls, as has been shown to happen in the absence of prominence. Therefore, these findings suggest that these rises did not double function as part of a pitch accent in line with our annotation, rejecting the first hypothesis.

Overall, the findings of the present study support the second hypothesis (Figure 15b). Apexes were chiefly paired with pitch accents when prips contained one. However, if prips were marked as accentless in our independent annotation of intonation (Section 2.4.2), then apexes were largely paired with L, indicating a distinct pattern in these cases. This hypothesis is further supported in the analysis of synchronisation (Figure 14 in Section 3.2.6). There, it was shown that synchronisation was achieved when apexes were paired with both pitch accents and Ls in prips. Since in the majority of cases where apexes were paired with Ls in prips, the prip was unaccented, we can infer that apexes were both paired and synchronized with Ls in unaccented prips. On the other hand, even when apexes were paired with (i.e., nearest to) phrase accents and boundary tones, they were not synchronised with them.

4.1 Summary

The present study investigated the synchronisation between gesture and prosody focusing on apexes and tonal events. There were two steps in the analysis. In the first step, the study analysed whether there were patterns in the pairings of apexes with tonal events. The second step consisted of the statistical testing of synchronisation.

The pairing analysis in Section 3.1 showed that apexes tended to be nearest to prominent pitch accents rather than other possible tonal events. If pitch accents were not available in the prosodic phrase, then apexes were paired with Ls, which are the only other tonal event at the level of the PW. Pairings with phrase accents and boundary tones which are associated with prosodic phrases above the level of the PW were avoided. This pattern was found to be consistent across different contexts. Based on these observations, pitch accents and Ls were identified as the preferred anchors of apexes for pairing, and phrase accents and boundary tones as dispreferred anchors. In the second step of the analysis (Section 3.2), it was shown that synchronisation patterns were in line with these pairing patterns. That is, if an apex was paired with a preferred tonal event, then these were synchronised given the synchronisation criterion (Section 1.3). In other words, apexes tended to occur within an average syllable duration of pitch accents and Ls. In cases where apexes were nearest to dispreferred anchors, apex-tone synchronisation was not achieved, meaning that apexes tended to occur farther than an average syllable duration away from these tone types.

Including the possible effects of gestural and prosodic context in the analysis revealed exceptions to the general synchronisation pattern. Rhythmic apical productions in metaphorics and beats imposed synchronisation of apexes with tonal events regardless of type. Moreover, in nuclear ips, apex-tonal event pairings tended to be synchronised regardless of type. The general pairing pattern was still observed here: A great majority of apexes were paired/synchronised with pitch accents. Yet, under the effect of maximum prosodic prominence, apexes and tones were more tightly coupled in time.

4.2 Implications

Studies on gesture-prosody synchronisation have generally concentrated on prominence-based synchronisation (Section 1.1). These studies often isolated stressed syllables in the continuous event-rich prosodic signal and only checked whether apexes are synchronised with measures taken from these syllables when isolated from their prosodic context (cf. Loehr, 2004). Consequently, other possible synchronisations with anchors, such as the tonal events that mark boundaries, have been ignored. The results of the present study were in line with the prominence-based synchronisation claims for the most part (i.e., consistent apex synchronisation with pitch accent peaks).

Yet, further analyses revealed another synchronisation pattern where apexes were synchronised with boundary marking events, indicating the lack of pitch accents. Current claims about synchronisation cannot account for the pairings with Ls because they are not prominent events nor acoustic peaks. To the authors’ knowledge, the possibility that apexes may also be synchronised with boundary-marking tonal events has only been mentioned recently for French by Rohrer et al. (2019) (see Section 1.1). The findings of the present study are the first to offer evidence that apexes can also be synchronised with boundary-marking tonal events. Interestingly, PWs in Turkish and accentual phrases in French show some intonational similarity. They both have default final stress and are marked with L tones at their left edge (Delais-Roussarie et al., 2015). These L tones do not encode prominence and create a contrast with phrase-final pitch rises (Ipek & Jun, 2013; Delais-Roussarie et al., 2015). Our findings were in line with Rohrer et al’s (2019) prediction in that through synchronisation, apexes marked a prosodic domain independently from the encoding of prominence.

Our findings were also important as they have shown for the first time that synchronisation at the micro level was managed by the prosodic hierarchy in addition to prominence. Pitch accents and Ls are both associated with PWs in Turkish. When the highest priority target for synchronisation (i.e., a pitch accent) was not present, apexes stayed anchored at the PW level by synchronising with Ls. In fact, apex synchronisation displayed sensitivity to the prosodic hierarchy by not synchronising with the markers of other prosodic constituents (i.e., phrase accents and boundary tones) that are higher in the prosodic hierarchy (see Figure 18). This result is interpreted to mean that both prosodic prominence and hierarchy are active agents that play a role in the anchoring of gesture to speech. Altogether, these findings hint at a typology of synchronisation that adapts to the prosodic structure of languages. More research focusing on prosodically different languages is required to uncover different synchronisation patterns.

Figure 18

A schematic that shows the domain of apex synchronisation as the prosodic word.

The fact that apex synchronisation was not arbitrary and clearly informed both by prosodic prominence and phrasing shows how multimodal cues are intertwined with phonological structure. The present study has shown that in terms of production, phonological categories impact gesture production since the timing of gesture components relies on feedback from phonological encoding processes. Moreover, in terms of implementation, interlocutors methodically implement multimodal cues along with phonological ones to code communicative intent for other interlocutors to decode. It follows that multimodal cues can influence the perception of intonational categories by listeners, when they are available (Vaissière, 2008; Cruz, Swerts, & Frota, 2017; Kelly, Bailey, & Hirata, 2017). The systematic and tight synchronisation with prosodic structure shown here suggests these cues could be an even more effective cue to prosodic events than has previously been explored, and play a role in the identification of both prominence-marking and boundary-marking prosodic events by listeners. The present study has shown evidence for this at the smallest unit level but there are also a few studies that reported temporal relationships between gestural phrases and prosodic phrases as well as information structural categories such as focus and topic (see Türk, 2020 and references therein). These findings support the general argument being made in the present study: Multimodal cues play a role in the implementation, production, and perception of intonational categories at multiple levels of the phonological structural hierarchy.

The present study has argued that such consistent behaviour of multimodal cues can be used to assist phonological analyses in identification of categories. In particular, it has shown an example case where looking at apex synchronisation can shed light on the accenting of words in Turkish focusing on pre-nuclear ips. The study made use of the bimodal apex synchronisation pattern to check whether words in pre-nuclear ips bear prominence. It was found that these phrases can be accented or accentless based on a systematic shift in the synchronisation behaviour of apexes. Note that the present study did not aim to explain why some words had pitch accents and some did not. Accentless realisations of these words can be a result of a variety of factors, which is out of the scope of this study to analyse. More research is required on the meaning and function (in terms of information structure) of these different realisations. The present study only showed that apexes were not synchronised with the pitch accent argued by some to be a constituent part – together with the phrase accent – of the tonal events at the end of the ip. Rather, apex synchronisation behaved as if there was no pitch accent in these words, offering multimodal evidence in support of the claim that words with final stress may not bear a pitch accent.

Our findings highlight the importance of a linguistically informed selection of anchors for synchronisation tests. Using measurements such as jaw displacement, vowel onsets, and acoustic peaks may provide methodological convenience; however, the understanding that gestural and prosodic units exist as members of complex systems where events, constituents, and their positioning influence each other can add valuable insight into our understanding of the relationship between multimodal cues and phonological categories, and more generally how speech and gesture are temporally coordinated. Furthermore, since the prosodic structures of languages exhibit different features cross-linguistically, more studies in different languages are needed to understand these in full. Investigations in this manner may shed light on the complex relation between acoustic and visual information, and how this relation is used by speakers and listeners as well as by us analysts.