Okay, hands up those of you who’ve heard of the McGurk Effect… Okay, then, how about those who watch faces when people are speaking -actually see them speak? And, does it make a difference? Some things are just so obvious they no longer register until they’re pointed out. Some things are multimodal; speech is multimodal and uses information not only from sound, but also vision to enable effective communication.
The McGurk Effect -described in a 1976 paper entitled Hearing lips and seeing voices, by McGurk and MacDonald- points out that visual input really does influence the perception of speech sounds. A subsequent study I happened upon in a 2021 edition of the Proceedings of the Royal Society B -this one by Bosker and Peeters- added yet another visible modality to the list: movement of the hands –the so-called ‘beat gestures’. https://royalsocietypublishing.org/doi/10.1098/rspb.2020.2419
‘Beat gestures—spontaneously produced biphasic movements of the hand—are among the most frequently encountered co-speech gestures in human communication. They are closely temporally aligned to the prosodic characteristics of the speech signal, typically occurring on lexically stressed syllables… we demonstrate that beat gestures influence the explicit and implicit perception of lexical stress (e.g. distinguishing OBject from obJECT), and in turn can influence what vowels listeners hear. Thus, we provide converging evidence for a manual McGurk effect: relatively simple and widely occurring hand movements influence which speech sounds we hear.’ Or, for those of you (like me) who had difficulty wading through the chaff, let me reiterate the last few words: ‘hand movements influence which speech sounds we hear…’
In this time of wrath and tears, I have begun to wonder just how much Covid restrictions have hindered our interpersonal communications. It seems trivial to point to the lack of visual cues that might otherwise have added to effective dialogue, even with so many of us resorting to video conversations. And yet, on Skype, or Zoom, we seldom see more than a face -a talking head. And yes, in case this seems familiar to those of you who’ve blundered into my work in the past, I admit I have thought about communicative gestures before: https://musingsonretirementblog.com/2020/11/22/a-wink-and-a-nod/amp/ but I’m not sure I realized at the time just how much speech required them beyond their obvious roles as addendal colour and emphasis. But it does!
For example, in the classic example McGurk et al reported, ‘participants were made to repeatedly listen to the sound /ba/. At the same time, they watched the face of a speaker in a video whose mouth movements corresponded to the sound /ga/. When asked what they heard, a majority of participants reported actually perceiving the sound /da/. This… indicates that during language comprehension our brains take both verbal (here: auditory) and non-verbal (here: visual) aspects of communication into account, providing the addressee with a best guess of what exact message a speaker intends to convey. We thus listen not only with our ears, but also with our eyes.’
Of course, we’ve long intuited that speakers ‘make use of different channels (mouth, eyes, hands, torso) to get their messages across, while addressees have to quickly and efficiently integrate or bind the different ephemeral signals they concomitantly perceive. This process is skilfully guided by top-down predictions that exploit a lifetime of communicative experience, acquired world knowledge and personal common ground built up with the speaker.’
Anyway, buoyed by my new-found scientific knowledge, I wondered whether or not I could perform a similar, but covert, validation of how extra communication is added with my hands on a Skype visit with a friend. Pandemic restrictions and the in-person mask mandates, were merely the excuses I exploited.
The problem, as it turned out, was not so much a validation as the distraction of constantly having to maneuver my hands into a position normally occupied by my ears for the camera. My addressee -I use the term for added scientific weight to my observations- thought I was making a face at her, and thus failed to identify the gravity of my purpose.
So, in a much needed attempt at experimental redesign, the next thing I tried was to sit way back from the screen so the computer camera could encompass more of my body. Unfortunately, that required more appropriate clothing than I had hitherto been used to -a properly-buttoned shirt, for example, and a decent pair of pants other than pyjama bottoms that would not detract from the gravity of my manual appurtenancing.
Her major objection to this maneuver, though, was to complain that she couldn’t see my eyes well enough to judge whether my cleverly-designed confusing hand signals were meant to convey something meaningful to her, or were largely idiosyncratic.
It’s difficult to follow through with an experiment on communication when the recipient of my expertise chooses to laugh, and mock the sincerity of my intentions. I decided that the best course of action, under the circumstances, was to resort to a scientific no-no with her: instead of hiding that I was trying to disguise that I was assessing her interpretation of what I was saying with non-conforming hand gestures, I decided to go anti-inductive and plunge right into deductive reasoning -start with a broad generalization and then look for specific instances to prove it… Or is it the other way around? Anyway, she’d already blind-sided me with the unfair vocabulary-laden ‘eccentricity’ without my ever seeing her hands, so I figured it was fair.
I could tell she was getting annoyed, though, because every so often she’d roll her eyes, or pretend-slap her cheek with her hand. The first time she did it, I just thought that she was showing me her growing understanding of the utility of manual signals in effective communication. Still, her eyes didn’t begin to narrow until she saw me pretend-biting my index finger -I usually do that when I am trying to make an important decision.
In retrospect, however, I wonder if she took that as permission to respond in kind. I’m not sure that fits in with the scientific paradigm I had intended, but a true scientist does not ignore something just because it fails to illustrate his original hypothesis. In fact, there is a growing movement in reputable journals to accept for publication the reporting of negative findings. It no doubt indicates the fact that not all experiments can be expected to corroborate their author’s initial expectations. The public likes honesty; it is also a needed gesture on the part of both the scientists and the journals which hope to publish their findings for their otherwise sceptical subscribers.
But in this case, the addressee responded with her middle finger, accompanied by an abrupt hissing outflow of a proto-word from her mouth. I suspect it may have been an admission that my point about the value of the hand in speech recognition had been made, and really, any subsequently recognizable words on her part would have been unnecessary.
At any rate, her screen suddenly went blank, so I have decided to temporarily delay any further work on my project… And besides, she won’t answer my calls anymore…
Musings on Retirement 