auditory perception and sound as event:

15/10/2012 by theearoftheduck Leave a comment

– THEORISING SOUND IMAGERY IN PSYCHOLOGY

Michael A. Forrester. Department of Psychology, Keynes College, Canterbury

Abstract: The study of sound in psychology has been dominated by the auditory perception view of psycho-acoustics. This paper considers the nature of the relationship between sound as event and associated processes of imagery, imagination and memory. Through a consideration of sound(s) as ecological event(s), the role of sound in film and radio, and our earliest experiences of sound as language, the discussion centres on whether psychology can contribute to our understanding of sound imagery. Concluding comments touch on the observation that when hearing a sound, our imagination often plays an important part in recognising what it might be.

* Sections of this paper are to appear in a forthcoming book ‘Psychology of the Image’ published by Routledge.

AUDITORY PERCEPTION AND SOUND AS EVENT: THEORISING SOUND IMAGERY IN PSYCHOLOGY

Intro-

Within psychology the study of sound falls under the umbrella term ‘auditory perception’ where the research focus is centred upon the presumed relationships between the psychophysics of sound and associated cognitive processes of recognition and interpretation. While the benefits of such an approach can be identified in certain specific applied areas, such as in neuropsychology, it can be argued that there remains something of a theoretical vacuum in our understanding of the relationship between hearing sound and the images or imagery that is conjured up by our experience. This paper asks whether psychology can develop a theoretical outlook which moves beyond the ‘stimulus driven’ orientation of the traditional approach, an orientation which helps highlight the role of imagery in our everyday perception of sound(s) as event.

The emphasis on the visual in Western culture makes it difficult for those not visually impaired, to recognise that the world of sound is an event-world while the world of sight is an object world (Ong, 1971). Reflecting on the relationship between sound and imagery provokes the observation that ours is a visually dominant representational culture. There is no reason to believe, however, that sound perception is any less complicated than visual perception, where the relationship between perception and discursive representations of perceptual experience remain philosophically problematic (Sharrock and Coulter, 1998). Although we understand scientific descriptions of auditory perception, phenomenally we don’t ‘hear’ acoustic signals or sound waves, we hear events: the sounds of people and things moving, changing, beginning and ending, forever interdependent with the dynamics of the present moment. We ‘hear’ the sound of silence.

From an evolutionary perspective sound has at least two distinct qualitative dimensions, one nurturing, supportive and indicative of comfort, care and safety the other dissonant, disruptive and likely to provoke anxiety. Nurturing sounds might include blood flow (from our time in the womb), rhythm, intonational prominence and all those many sounds associated with the presence of others involved in our care. The preference new-born infants display for their own mother’s voices has been well documented (DeCasper and Fifer, 1980). Parents in many cultures spontaneously produce ‘baby-talk’ when soothing infants, a form of speech characterised by rhythmic intonational patterns, short sentences, often spoken softly (Snow and Ferguson, 1977). In adult life the beneficial effect of meditative or calming mood music is promoted as an aid to reducing stress, and sufferers of insomnia know the value of listening to music or a late-night radio discussion show in order to lull themselves to sleep. The inherent rhythm to the sound of speech can have a comforting or soothing effect on us when we’re anxious (although not all the time, e.g., Baker, et al, 1993).

In contrast, it makes evolutionary sense that we are be highly sensitive to those sounds that might indicate the presence of potential predators, not dissimilar to our keen visual sensitivity to the detection of movement in peripheral vision. Some sounds appear to be intrinsically appealing and pleasurable, otherwise discomforting and annoying. We are very easily disturbed by loud and disruptive noises. In particular, sounds in our environment which presuppose danger in some way, e.g., screeching car-tyres from behind as we walk on the highway, are exceptionally attention grabbing, and for good reason. In what sense however, do we ‘imagine’ the cause of the sound or the sound-event? When woken in the night by a scratching noise we might quickly decide that we are listening to the sound of a mouse or rat under the floorboards or behind the wall. But consider, it is on hearing the noise that we then imagine that the sound is the kind of noise a rodent might make when scraping or scratching around for food. Our knowledge of such sounds has come from the cultural repertoire of all those available imaginable sounds, i.e., we don’t in reality have to have seen a rat or mouse making such a sound, a great deal of our knowledge comes from the available cultural discourses about sounds and their causes. Again, in the same way that visual perception of an event is interdependently linked with labels, names, discourses about that event, so it is for sound. We might even say that there is no such thing as silence, except an imaginary silence – a pure, abstract absence of sound, arguably we cannot jump out of our discursive representational knowledge of sound into a ‘soundless’ void.

Here, I want to begin by comparing the traditional approach to sound (auditory) perception within psychology with more recent attempts inspired by Gibson’s (1979) realist metaphor, and which focus on sound as event. After some discussion on the differences between these approaches, I then consider the relationship between sound, affect and our earliest experiences, followed by a look at specific contexts where sound effects are deliberately manipulated in service of the imagination, e.g., film and radio. Reflecting on our response to sound in such contexts provokes a brief look at the role of affect and sounds that evoke particular meaning or significance for us. By way of conclusion, towards the end of the paper a number of comments are made regarding the cultural basis of auditory perception, i.e., sound as ‘meaning and event’ within a particular social-discursive context.

SOUND AS PSYCHOPHYSICAL OBJECT

Psychology studies the nature of sound as the psychophysics of wave form analysis. The essential focus is on the nature of the computation said to take place as a result of sound waves creating vibrations in our eardrums. In line with other areas of sensory perception, the more dominant theories of auditory perception focus on how the cognitive system constructs appropriate auditory representations, that is, given the potentially confusing, degraded or redundant information made available to the ears. In light of the observation that sound waves from any source will reach each ear at a different time, the question of how sound is located is normally framed within a ‘deprivation’ model. The established practice of viewing auditory perception in terms of sound waves underlies the rather curious image we have where humans can only ‘hear’ sounds within a certain frequency range, and dogs, bats, porpoises and other mammals able to hear much higher frequencies. As sound wave frequency increases, pitch increases, providing the template for Western musical scales, and interestingly one of the earliest theories of pitch perception (pitch is described as the prime quality of sound measured on a scale of high to low), proposed that the ear contained a structure formed like a stringed instrument:

Different parts of this structure are tuned to different frequencies, so that when a frequency is presented to the ear, the corresponding part of the structure vibrates-just as when a tuning fork is struck near a piano, the piano string that is tuned to the frequency of the fork will begin to vibrate. This idea proved to be essentially correct; the structure turned out to be the basilar membrane, which unlike a set of strings, is continuous (Atkinson, et al 1990:143).

Even such a cursory examination of the images, metaphors and ideas informing current theory in auditory perception reminds us that the scientific study of sound is linked in a very particular way with what is said to constitute, subjectively, our perception of sound events in the first place. Consider for example, what must influence the calibration of any instrument for measuring the intensity of sounds in decibels (table1).

Table 1. Decibel ratings and common sounds

Decibel ratings and common sounds

Decibel

Level

Example

Lowest sound audible to human ear

Quiet library, soft whisper

Quiet office, living room, bedroom away from traffic

Light traffic at a distance, refrigerator, gentle breeze

Air conditioner at 20 feet, conversation, sewing machine

Busy traffic, office tabulator, noisy restaurant

Subway, heavy city traffic, alarm clock at 2 feet, factory noise

100

Truck traffic, noisy home appliances, shop tools, lawnmower

110

Chain saw, boiler shop, pneumatic drill

120

Rock concert in front of speakers, sandblasting, thunderclap

140

Gunshot blast, jet plane

180

Rocket launching pad

(adapted from Atkinson, et al, 1990)

The index on the left hand side of the table is qualitatively linked to sound experiences but clearly do not map onto the scale in some sort of additive fashion, i.e., the difference between a soft whisper and a refrigerator is hardly perceived as the same sound scale difference between heavy city traffic and a chain saw. It should not escape our attention that one of the most often employed dependent measures within the psychophysics of sound is the ‘JND’ (a just noticeable difference), an amusing example of the transformation of a social-cultural practice into an objective measure (Krueger, 1992; Drake and Botte, 1993). People in psychophysics experiments are invited to indicate when a difference between two sounds is ‘just noticeable’, thereby providing the basis for the measuring instrument.

Essentially auditory perception research is dominated by sound as ‘abstracted information’ in the information processing sense of cognition. To hear is to perceive (albeit in a sub-conscious way) sound wave frequencies, pitch intensity and so on. The measurement of relevant parameters concentrate on the sensory qualities of sound (loudness, tone, intensity, timing and pitch), where the ‘brain-mind’ transforms the neural excitations, caused by sound wave pressure into sound perception in an as yet unknown way. A reading of such work leads quickly to the conclusion that the psycho-acoustics of sound will have little bearing on developing a theoretical understanding of the relationship between sound perception and images, leaving aside one or two studies in marketing psychology which look at this relationship in advertising (Halpern, 1988; Miller and Marks, 1992).

We only have to think of our everyday experience of sound to see why there are major difficulties in developing a psychological theory of sound imagery. Consider how we might explain our experience of sound and associated imagery processes when we are listening to music through headphones, particularly headphones where there is no experience of pressure on our ears. Although we know the source of the music is external to our bodies, our phenomenal experience is of music playing in our heads, sounds and images intermeshed with thoughts, reflections and associated responses to the music. What is inside and what is outside becomes unclear, an observation which should remind us that to listen is not the same thing as to hear in a passive sense. We can then ask, how are we to conceive of sound as event?

PERCEIVING SOUNDS AS EVENTS

In the comparatively recent past psychologists have taken note that the relationship between the ‘outside’ and ‘inside’ nature of sound perception (imagery) is not simple. Gaver (1993a), for example, criticises the overemphasis on the computational approach in auditory perception. He points out, that we hear events in the world rather than hear sounds, noting that it remains unknown how sounds close to the ear can indicate remote physical events, e.g. hearing the sound of distant traffic through a closed window. In developing his proposals Gaver (1993a) contrasts everyday listening with musical listening, as in table 2.

Table 2. Everyday and musical listening

Musical experience

Everyday listening

Musical listening

(perception of sound producing events)

(experience of sounds themselves)

* hear a sound in terms of its sources

* hear a sound in terms of it’s sensory qualities

* rarely addressed in psychology

* traditional approach in psycho-acoustics

* reports research which focuses on people’s perception of ‘everyday’ sounds

* sound do not convey enough information to specify their sources: must be supplemented by memory, unconscious processes and problems solving

The traditional approach in psycho-acoustics, as noted earlier, has been to study sound with regard to measurable sensory qualities, waveform; pitch and so on, and no surprise to note that the primary focus within the psychology of music has been musical perception, tonality, pitch and so on (Boltz, 1998; Krumhsanl, 1991). Again, and in line with the constructivist approach of visual perception (Hochberg, 1978), the sound information in perceiving music is said to be impoverished in some way, i.e., not adequate enough to specify the source of the sound. Top-down cognitive processes supplement the raw primary data activating the senses.

In contrast, our experience of listening to sounds in everyday life is in terms of the sources that produce them, the whirring of a fridge; rustling of leaves on the road and so on. To hear a sound is often to hear the cause of the sound. The wind is a good example particularly as it is the effect of ‘the wind’ on objects which constitutes the sound of the wind as an event or events, e.g., compare the sound of leaves blowing with the intermittent but continual sound of a door swinging and banging – both caused by the wind.

Gaver (1993a) calls for an approach in auditory perception which focuses on the consistent structure of the world that allows sounds to relate reliably to their sources. Two distinct questions are raised. What do we hear, and how do we hear sound(s)? In answer to what, he develops a framework for describing ecologically relevant perceptual entities in the dimensions and features of events that we actually hear (this could be viewed as a content question). In answer to how do we hear, Gaver (1993b) developed an ecological acoustics, one that describes the acoustic properties of sounds conveying information about the events we hear (a perceptual structure question). His proposal relies on an ‘analysis by synthesis’ method which attempts to formalise the relationship of dimensions of physical objects to how they are perceived as event-sounds. The theoretical procedure involves an iterative process of analysing the sound of an object, e.g., a hammer hitting metal, and then synthesising a duplicate on the basis of the sound patterns (spectograms composed of frequencies, amplitude and time dimensions). Subsequently the analysis data can be systematically reduced, and the resultant synthesised sound compared to the original. If there are no perceptible differences between the original and the synthesised versions, then the discarded data are considered irrelevant for perception. By proceeding in this way, an understanding of the aspects of a given sound crucial for perception of that sound can be obtained (Gaver, 1993b, p. 289).

This attempt at formalising an ecological approach to auditory perception resonates with similar work within geography. Rodaway (1994) in particular has incorporated Gibson’s approach in his analysis of what he calls auditory geographies. He asks us first to consider that within North American/European culture, we focus on the visual to the extent that even our labelling of sound events emphasises the nature of the object making the sound, rather than the phenomena. For example, as a description we are more likely to say, ‘the bell sounds’ rather than ‘the sound bells’ – in the former the focus is on the bell, in the latter on the sound itself. In addition, he notes that with sight or vision there is always a phenomenal sense of being at the edge of our visual field, and thus viewing our perceptual landscape in front and to the sides of the direction we’re facing. Not so with sound, where we are always at the centre of the perceptual experience, i.e., we are as sensitive, if not more so, to sounds behind us as well as in front. While vision is an object world, sound is an event world. Auditory experience is always of a flow of sound, constant at times, rising or falling in intensity, noticeable when absent or excessive, but never truly ‘silent’. The relationship between sound and attention becomes important whenever we use the terms ‘background’ or ‘foreground’ noise.

Noting the extension of the visual metaphor in the image of the landscape, Schafer (1977) provides a framework for understanding sound as event, with his notion of the soundscape. Soundscapes ‘surround and unfold in complex symphonies or cacophonies of sound’ (p.86) and his framework emphasises the fact that we inhabit the centre of our soundscape. In one sense we might say that we feel more detached from the visual world. Sound experience is always a sensuous experience at one level, an interdependent time/space geography of constant and continuous dynamic events. Even in circumstances where you might imagine the experience of complete silence, a moment’s reflection highlights the nature of such a fantasy. Consider for example that if you were sitting in a sensory deprivation chamber, at the very least you would nevertheless hear the sound of your own blood flowing through your veins and the beating of your heart. Auditory experience is a special sensory key to interiority and as noted earlier, when listening to sound through earphones one quickly realise the borders between the ‘external’ and the ‘internal’ are as much determined by language and discourse as they are by phenomenal experience.

In thinking through the complex nature of auditory perception, Rodaway (1994) provides a helpful classification of phenomenal experience in his outline matrix describing an auditory geography (figure 1).

Figure 1. Dimensions of an auditory geography

sensation

To hear

To be

heard

perception

presence

To listen

To sound,

voice

meaning

Classifying auditory experience in this way may help clarify the position or rather the conceptual relationship between sound and imagery. On the one hand there is a dimension spanning perception as raw experience with presence and embodiment. At the same time a second dimension distinguishes between simple auditory sensation with sound event as meaning. Rodaway (1994) points out that hearing (to hear) may be described as the basic passive sensation, the condition which make it possible to ask ‘what’s that sound?’ Listening (to listen) however implies active attentiveness to auditory information and the very act of listening draws attention to our desire to establish or mark out meaning.

Correspondingly however and with respect to the second dimension of presence, we are organisms which acquire a knowledge of auditory information about the environment that surrounds us. People, animals and things all emit sounds even when they are unwelcome (e.g., when your stomach begins to rumble during an important interview for a job). We also, through sound, project ourselves into the auditory soundscapes of those around us – we have a voice capable of denoting our presence in the world and specific ‘meanings’ interdependent with the projection of ‘our sounds’ denoting our presence. It becomes easier to understand why we feel more detached from the visual than the auditory world. Our experience of sound is often emotional and Rodaway (1994) suggests that while sight paints a picture of the world, sound(s) (along with touch, taste and smell) are life itself. We might ask, where does our sensitivity to sound begin and what form does it take?

IN THE BEGINNING WAS THE ‘SOUND’

While we can applaud the attempts by ecological psychology to draw attention to sound event perception, it remains unclear why we do not have conceptually rich theories for understanding the relations between sound and imagery. This is rather surprising and may in part be due to what Beloff (1994) has called psychology’s overemphasis on written language (i.e., text). Language is first and foremost sound, or at least the business of making meaning through the use of sound. Consider for a moment though, how the study of language acquisition is conceived in psychology and linguistics. The central constructs within child language research focus on the ‘lexicon’, syntactic structure, lexical processing and so on, all terms predicated on a view of language as formal (often written) object. In other words, what underlies the very definitions which inform the study of language acquisition are notions of structure and language as ‘text/object’, an approach which derives from formal linguistics, despite the fact that Saussure’s (1974) structuralism grew out of his analysis of the segmentation of sound in the vocal tract. In contrast to the formalist emphasis of psycholinguistics, from the child’s viewpoint what she hears are sounds (not words), and one of her first tasks is probably to learn what sounds to pay attention to (mother’s voice) and what sounds to ignore (background noise of traffic outside the window).

Alongside touch, sound perception in wrapped up with emotional response and affect. Dore (1983) argues that the earliest attempts at overcoming separation and anxiety are centred on the infants sound making attempts. And as the child grows her main task is to learn under what conditions making this (and not that) sound, will lead to those around her responding to her ‘soundings’ as intentional communicative acts. Some work within developmental psychology has mapped out the circumstance where the interactive structuring of the infants early sounds (by parents) leads on to their making accountable (word) sounds (e.g., Golinkoff, 1883).

It would be a mistake, of course, to think that parent and child are focused solely upon intentional communication above everything else. Our emotional development is deeply embedded within ‘sound-sensitive’ contexts. The child’s whole sensual environment is effused with sound. Folklorists and psychoanalysts have studied the nature of early lullabies sung by mothers noting the unique (usually but not always), positive intonational contours of nursery rhymes (Sircar, 1997; Bowey, 1990). Baby infants spend considerable time playing with the sounds the make, displaying amusement and interest when others copy their noises (Romeflanders and Cronk, 1995). Repetition, sound play and noise experimentation all appear to be important aspects of the child’s pre-linguistic development, despite the absence of any direct correlation between early sound use and later language development (Kuhl and Meltzoff, 1996; Ingram, 1985). We continue to find it difficult to remember that children learn language as accountable sound performance, and only later learning that these noises are described as words, sentences and all other such constructs which derive from the invention of writing.

The significance of the first sounds for early affective development has been stressed by developmental psychologists and psychoanalysts. Sabbadini (1998) for example, argues that the earliest exposure to sound patterning has profound effects on later psychopathology. He notes:

Let us come back to the nursery. Silent feeding sessions with a mother never addressing sounds or words to her baby may be distressing for the child and could amount to perceptual and emotional deprivation, possibly leading to future problems in the development of linguistic and musical skills, or to overt psychopathology. (Sabbadini 1998:5).

While recognising it is unlikely that any direct evidence linking sound deprivation with later psychopathology would ever be forthcoming (for methodological if not any other reason), we need to keep in mind that the infant’s primary sensation environment is tactile and auditory before it is visual. We feel and ‘sound’ our way into the world before we perceive that world visually.

FACILITATING THE IMAGINATION IN SOUND MEDIA

Moving on from reflecting on our primary sensory experience, if we aim to examine contexts where the relationship between sound and imagery is paramount then we only have to turn to two influential communication media, i.e., film and radio. One might expect that a consideration of the sound techniques and practices deliberately employed within film should highlight certain aspects of the assumed relationships between visual and auditory perception. And where one strips away dependence on the visual within a communicative context (radio production) the relationship between ‘image production’ or the imaginary should come to the foreground. We can begin with sound and film, confining our discussion to the relationship between music and film. Dialogue as ‘sound’ in film serves as the primary vehicle for narrative construction and may be secondary to music when it comes to techniques for facilitating and manipulating the audience’s imagination.

Sound and film

A consideration of sound imagery would certainly be incomplete without discussing the interdependence of imagery domains in film (sound and visual). Film directors and producers work hard at representing (re-presenting) our acoustic world in film. To paraphrase Balazs (1985), the acoustic landscape in which we live is often revealed through the sound film. He comments that all that has speech beyond human speech, all that speaks to us with the vast conversational powers of life and incessantly influences and directs our thoughts and emotions, is made available in the medium of film. From the muttering of the sea to the din of a great city, from the roar of machinery to the gentle patter of autumn rain on a windowpane, all are recognisable in an instant given the richness of our imagery associations and rememberings. The sound film can encapsulate the significant sounds of life, in a way that seems much more direct than say, the lyrical poem.

We can also note that the sounds heard in a film do not need to be explained. Balazs (1985) makes the point that the timbre of a sound changes in accordance with the gesture of the visible source of the sound observed in the film, in a way akin to the way the shade and value of a colour changes according to what other colours border it in a painting. In a sense the acoustic and optical images are effused together in a single ‘sound-vision’ picture. And the intimacy of sound can be manipulated quite deliberately in the sound film,

We are all familiar with the ‘acoustic’ close-up where we are made to perceive sounds which we would never normally hear (e.g. in building up the atmosphere in a horror movie). And of course subtle associations and interrelations of thoughts and emotions can be conveyed by means of very low, soft sound effects. Such emotional or intellectual linkages often play a decisive dramaturgical part. They can be anything – in the ticking of a clock in an empty room, a slow drip from a burst pipe, or the moaning of a little child in it’s sleep: we possess such an array of deep associations of the most common everyday sounds. We have learned a language of sound imagery, where we simply seem to know ‘immediately’ what any given sound ‘means’ (Balazs 1985:121).

Such observations should remind us that there is a considerable difference between our visual and acoustic education. One reason for this is we often see without hearing (e.g., through a window) but we very rarely hear the sounds of nature and of life without seeing something, if not with the eyes then with the imagination.

There are a number of more formal approaches to the study of music in film (Neumeyer, 1997; Steinberg, 1997; Libscomb, 1997). Kendall and Lipscomb (1997) for example, propose that any given musical and visual relationship can be represented in a two-dimensional space representing the degree of abstraction of the musical and the visual dimension (see figure 2.). Motion pictures as audio-visual composites can be placed within such a grid, where,

one extreme of the musical dimension might be exemplified by the use of “La Marsailles” and “Deutschland über Alles” in the bar scene from “Casablanca”. Each melody is associated with the country which it represents (HIGHLY referential). The other end of this dimension might be represented by musical sound that has no conventional association, yet is perceived as organized simply because of the relationship of one tone to the others or because of the superimposition of musical sound on a sequence of visual images (Lipscomb 1997:2).

Lipscomb and Kendall (1994) raise two considerations concerning the combination of musical sound and visual images, attempting to take into consideration what they call the referential and syntactical aspects of the motion picture experience. In figure 2, they place Norma McLaren’s ‘Synchrony’ in a position of high visual and musical abstraction. The film is a piece of experimental animation where abstract shapes appear on the screen at the same time as identical images pass over the (idealised) photoelectric cell of the ‘sound track’ portion of the film celluloid, ‘the resultant tones are not intended to have any conventional association for the viewer’ (Lipscomb 1997:2). In contrast, they situate the final scenes from Zwick and Horner’s ‘Glory’ (1989) towards an extreme ‘concrete’ visual and musical position on the grid. In these scenes a troop of soldiers prepare to march off to war, and the themes and motifs associated with the various characters throughout the film are heard on the soundtrack, ‘the scene is extremely concrete (i.e., a low degree of abstraction) in its representation of human drama while the music is referential both stylistially and thematically’ (p. 3).

Figure 2. Composition dimensions for film music

Lipscomb and Kendall’s (1994) main point is that an effective film score, in its interactive association with the visual element, need not attract the audience’s attention to the music itself. Lipscomb (1989) goes on to argue that most successful film composers have made a fine art of manipulating audience perception and emphasising important events in the dramatic action without causing a conscious attentional shift. It remains uncertain however whether formalisms of this kind will provide a sufficiently rich theoretical framework for understanding the relationship between music and film. Although a two dimensional representation of this kind can help place or position a film within a particular genre (concrete/abstract), as a theory we are left in the dark when it comes to explaining why we might ‘see’ an image on hearing a piece of music (or vice versa). Notwithstanding developments in audience reception theory a convincing account of this phenomenon has not yet emerged.

Radio

Unlike film where sound helps to widen representational space by extending the ‘film-frame’ on screen, radio is what Beck (1998) calls the blind medium – confined to sound alone. There is no way in which the visual scene, the ‘positioned perspective’ of the viewer can be enhanced through the simultaneous employment of sound and image. Radio plays in particular, stand on their own and have to create the unique impressions associated with engaging the listener in the ‘act of sound imagining’. A number of writers in radio drama and criticism have drawn attention to the poetics of radio drama and provide interpretative tools for a systematic analysis of production systems in radio drama (Rodger, 1982; Drakakis; 1981; Beck, 1997, Chion, 1994). One idea within this work, the construction of a ‘point of listening’ helps articulate certain kinds of metaphors or ideas we hold regarding the nature of ‘external’ and ‘internal’ auditory experiences. Ferrington (1993) for example interprets the effect of listening to a radio play, claiming,

An effectively designed audio work may facilitate a listener’s integration of life-based experiences into a ‘movie’ created within the ‘theatre of the mind’. Each individual becomes his or her own movie director with no two people having the same imaginary experience (Ferrington 1993:45).

Building on the work of Chion (1994), Beck (1998) summarises three important elements involved in the production of a point-of-listening: the sound frame, acousmatic sound and the listening zone. Asking how is it that radio manages to create the equivalent of ‘on-screen vs. off-screen’, Beck notes that every sound event in radio drama requires an acoustic, a context and a verbal description to fill out the situation for a listener. Producing sound events that are ‘in-frame’ (rather than out-frame) involves balancing together the sounds with explanations for their occurrence within a ‘sound perspective’ based on what we would normally expect with our everyday acoustic environment, i.e., sounds close by us. In contrast ‘out-frame’ sounds are produced as if external to the immediate present (for the participants in the drama), e.g., the sound of a dog barking in the distance, seagulls and so on.

Commenting on the second category of sound, Chion (1994) calls this ‘acousmatic sound’: a sound you hear without seeing its cause, such as railway station and other loudspeaker announcements, a bomb exploding at a distance, the voice down the telephone line, the noise in the attic, someone coming up the stairs, the knock at the door, and what is overheard in the next room. Beck (1998) points out that in radio drama acousmatic sound is categorically different from film,

because we hear every sound event in radio drama and connect it with its cause, within an overall sound space. Here, the acousmatic inhabits an area beyond the main frame of the sound picture, and it is ‘out’ and ‘unseen’, but of course is still bound by the sound space’s ‘outer frame’. I use the terms ‘main frame’ of the sound picture and ‘outer frame’ of the overall sound space (Beck 1998:23).

Notice the distinction made here between ‘sound picture’ and ‘sound space’, the first more object/image like, the second event-determined, and needless to say, both in relation to an interpreting listener. Underpinning the idea of a ‘point of listening’ rests the question, from where does the listener listen? There are at least two aspects of this positioning as Chion (1994) describes them – one a spatial sense, in the space represented in the soundtrack, the second a subjective sense involving the positioning of the listener with the relevant character at a given moment in the narrative, i.e., which character is hearing what I hear? Beck (1998) points out that the listener is always positioned at the centre of the sound which determines the mixing and balancing of sound technically. The task for the director is to transform potentially confusable hearing into active radio ‘listening’, which involves of course retention, interpretation and an immediate reliance on short-term memory.

The listener however occupies a corresponding ‘listening zone’, which Gray (1990) claims is the imagination, the creativity of the mind, or to paraphrase Beck (1998), this is the second play in the audience’s head where the listener is the ‘final actor’ and ‘director’. Here at least is one context where the relationship between sound and imagery is articulated with deliberation and aesthetic consideration. Commenting on the historical and social practices which surround the activity of radio listening, Beck (1998) comments that radio listening zones are personal and not socially ritualised in the same way that cinema or the theatre is,

Listening to radio plays is rarely now a collective activity. Each listening zone has its own disattention factors, extraneous noise and accompanying activities, and in spite of these, radio listening is accessible and pleasurable. Does being a solitary listener incline one to strong or weak identification? Surely the environment influences our reception. The fact is that many Radio 4 afternoon plays have domestic plots, what in America are called ‘hearth’ plays. These cross over to the domestic zones of the listeners and suggest at least a relationship with the – usually – strongly sympathetic protagonists (Beck 1998:5).

The second ‘subjective’ aspect of the listening zone is reflected in the technical notes of radio drama scripts (indicated as ‘thoughts’). A subjective point-of-listening is produced where the listener is (as if) given an internal dialogue of what a character might be thinking. Again, Beck (1998) notes that technically the actor-speaker for ‘thoughts’ is always close to the microphone rather in the normal position (about arms length or further). In such a close position the voice sounds very different, more detailed, intimate and ‘as if’ we are voyeuristically inhabiting the speaker’s mind. This production convention has been termed ‘interiorization’, what Beck calls radio’s ‘fourth dimension’ through which a particular complicity with the listener can be established, ‘a process as familiar as our own inner ruminations’ (p.6).

To summarise, in both film and radio drama we can identify particular aspects of the relationship between sound and imagery (or the imagination). In fact, presupposed in the design and use of the technical procedures for ‘sound imaging’ in these domains, exists an idealised conception of the spectator/listener’s imagination processes. Radio in particular makes manifest important distinctions of auditory perception (internal ‘thoughts’, sound distancing, point of listening), ideas which inform our understanding of sound imagery.

AFFECT, IMAGERY AND THE SIGNIFICANCE OF SOUND MEMORY

We hear sounds as events but at the same time many of the sounds we hear conjure up particular memories, associations and images of significant moments in the past. There is a particular kind of relationship between certain sounds we hear and their significance in our lives. Developing insights into the social significance of sounds outlined by Ferrington (1995), Thorn (1998) points to the ‘marking’ significance that some sounds can have in our daily life. The associations of sounds made by people accomplish what he calls the ‘marking off’ of social boundaries in everyday life, constituting a largely unremarked part of our soundscape, or to give some examples of such ‘sound marks’ provided by one of his respondents,

…”dad’s razor being scraped night and morning, giving me a sense of security”,

…”the sound of the key in the front-door meaning dad was home and everything was all right”,

and

…”the sound of family moving about the house after I’d gone to bed giving me a sense of security and belonging” (Ferrington 1995:23)

Such references to notions of security are good examples of the deeply emotional and personal associations that people give in response to questions about the significance of sound. As a further investigation of such associations, in a recent pilot study of the relationship between sounds and affect (Forrester, 1998), I asked students attending a University class in England what kind of emotional responses they have to particularly significant sound marks. Their responses included,

Example 1: ” The sound of a recorder – when I hear it now I get images of myself and old school friends practising in our music room – I feel the whole presence of the school and that era at that time. I can actually feel what it was like to be 8 years old again”.

Example 2: ” The sound of a lawnmower in the summer somewhere in the distance. Brings back memories of the summers playing in the garden and staying out as long as possible at our cottage.”

Example 3: “Piano music – my sister and I trying to get our practice done so that we could go out and play.”

Example 4: “Cross-channel car ferry horn. This meant great excitement at going on holiday, the start of an adventure, or feelings of safety and comfort on the way home after the adventure”.

Example 5: “Grandfather’s clock making a noise every hour. Whenever I heard it I was filled with joy that I was there. Usually happened in the morning”

Example 6: “Parent’s arguing – emotional worry and stress. School bell – pleased happy time to go.

Example 7: “Noises of children playing at playtime in school – associate this with fun…..Conflicting music and speech from the radio. My three older brothers all liked different music, my mother listened to radio 4 and my father watched television. A cacophonous sound, indicative of my family being all together”

Example 8: “A train whistle – steam train noises were very exciting. Reminds me of playing very freely in our ‘garden’ and was closely involved in our fantasy games, e.g. train could be bringing ‘Cowboys'”.

We can see that in these examples, simply remembering significant sounds serves to reproduce particular images and associations for people. And the images themselves are often effused with affect (examples 4 and 5), point to important aspects of past relationships (examples 6 and 7), and appear to have the potential to make people ‘feel again’ sensations from the distant past (examples 1 and 8). We perceive sound as event and through sound we can relive earlier associations and feelings, good, bad and indifferent. Sounds can act as signifiers for conscious and unconscious meanings.

COGNITION, SOUND AND CULTURE

Beyond recognising that there seems to be something of a theoretical vacuum in the study of sound imagery in psychology, critically reflecting on the nature of sound imagery may have implications for our ideas about cognition more generally. Rodaway (1994) points out that the Inuit do not conceptualise space and time separately but instead perceive any situation as a dynamic whole. Commenting on an early study by Carpenter (1973) into the navigational abilities of the Inuit, he notes that people are often astounded by the Inuit ability to follow a trail across an apparently featureless tundra waste, suggesting that the idea of a ‘map’ was for them quite alien since their world was much clearer in auditory than in visual form,

This was an auditory world of events, processes and actions, not the visual world of places, patterns and objects. The wind was more important than the vista, offering environmental information from its noise, force and direction, and from its olfactory content as well. The long periods of darkness in the tundra winter and the snow and ice expanses where sky and land and sea merge make visual sensitivity less useful, especially when the individual is hidden well into his or her parka to keep out of the cold and biting wind (Rodaway 1994:110).

Arguably, for psychology knowing is predicated on the visual, reflected in the metaphors we using when discussing whether we comprehend or understand each other (e.g., I see what you mean; can’t you understand my perspective; she seems to have second sight; and so on). We can speculate as to the kind of knowing derived from auditory perception. Gell (1995) argues that the Umeda of Papua New Guinea possess a quite different model of the relationship between perception, language and cognition (compared to Western presuppositions and assumptions), noting his own ‘methodological deafness’ which caused him to fail to appreciate,

the auditory domain, including natural sounds, language and song, as cultural systems in their own right, and not just adjuncts to culture at large, but as foundations, thematic at every level of cultural experience (Gell, 1995:233).

We can also recognise that what we mean by silence is culturally embedded given Imada’s (1994) observation that for the Japenese, silence can have a very particular suggestive force. In a brief introduction to Japanese ideas about sound, Imada (1994) points to the suggestive force of the absence of discernible sound, through two illustrations. The first, is of people gathering to listen to the sound of the bloom of a lotus flower at a pond in Tokyo. The blooming actually occurs at a pitch below the level of human hearing, but people ‘wanted to listen to that phantom sound. The experience was a kind of communal auditory hallucination’ (Imada 1994:5). We are reminded of John Cage’s musical experiment with his 4` 33″ piece of silent music. Cook (1990) cites the occasion when the pianist created havoc when he ‘played’ his silent piece to considerable effect. As an event, the pianist approaches the piano, opens the instrument as if to play, then sits in complete silence for 4 minutes and 33 seconds precisely. The effect as a ‘performance’ is to draw the audience’s attention to the accompanying sounds being made by them as they await the playing. To paraphrase Cook, the piece creates a musical event from whatever is heard, and does this by creating in the listener an openness to the qualities of sounds heard for their own sake, an awareness normally absent in our everyday surroundings. Anything can be music if the listener chooses to hear it in a particular way, and the opposite can also be true, nothing can be music unless it is heard as such.

Secondly, in describing a sound installation in a garden, Imada (1994) draws attention to the delay between water being introduced to the ‘suikinkutsu’ and its effect being heard. He says that not only did people ‘appreciate the sound of the suikinkutsu itself, but also the time spent creating the sound.’ the delay ‘had the effect of directing [their] listening to other environmental sounds in the garden’ (Imada, 1994, p. 7). We are reminded that sound is not only an event, it is always an event understood within specific cultural discourses which define the meaning of an particular sound as ‘the sound’ for x, y, or z. We have yet to formulate a socio-psychophysics of sound perception.

CONCLUDING COMMENTS

Attempting to think of a sound as a sign may be possible only in rather specific social-cultural contexts, for example, the sound of a hooter blowing denoting the end of the working day. If anything, it seems that only when sounds are inscribed as ‘text’ (phonetically, musical scores, sound-wave patterns or whatever) do we find it easy to formalise the language of sound as a semiotic enterprise. Representing in text the speech event as communicative meaning makes semiotic analysis realisable, and explains in part why linguistics and psycholinguistics is dominated by formalist accounts of language. But focusing on language in this ‘object’ like way may loose sight of the fact that speech is sound first and ‘text’ second. It is rather difficult to formulate an understanding of sound as imagery where we only have rather impoverished theories of auditory perception as our conceptual starting point. The work of Rodaway (1994) and others reminds us that there is a conceptual gap between the ‘sensation of sound’ as perceptual experience and the recognition of sound as ‘event and meaning’. Within this gap we have inserted discourses of sound, and thus a ‘sound as sign’ can only be understood with respect to the cultural discourses within which the sound is embedded. And the answer we give ourselves to the question ‘what’s that sound?’ will often rest upon our imagination, in the sense that in hearing a sound as belonging to a recognisable category will engender imagining the cause of whatever produces that kind of sound. We cannot yet talk of a theory of sound imagery but we can at least begin to articulate conceptions of sound which move away from psychology’s overemphasis on auditory perception.

Bibliography:

Atkinson, R., Atkinson, R. C., Smith, E. E., Bem, D. J., & Hilgard, E. R. (1990). Introduction to psychology (10th Edition). San Diego, Calif., London: Harcourt Brace Jovanovich.

Baker, C. F., Garvin, B. J., Kennedy, C. W., & Polivka, B. J. (1993). The effect of environmental sound and communication on ccu patients heart-rate and blood-pressure. Research In Nursing & Health, 16(6), 415-421.

Balazs, B. (1985). Theory of the film: sound. In E. Weis & J. Belton (Eds.), Film sound: Theory and practice . New York: Columbia University Press.

Beck, A. (1997). Radio acting. London: A.& C. Black.

Beck, A. (1998). Point-of-listening in radio plays. Sound Journal, 1(http://speke.ukc.ac.uk/sais/sound-journal/beck981.html), 15-25.

Beloff, H. (1984). The social psychology of the personal photograph. Bulletin of the British Psychological Society, 37(February), 47.

Boltz, M. G. (1998). The processing of temporal and non-temporal information in the remembering of event durations and musical structure. Journal of Experimental Psychology: Human Perception and Performance, 24, 1087-1104.

Bowey, J. A. (1990). On rhyme, language, and childrens reading. Applied Psycholinguistics, 11(4), 439-448.

Chion, M. (1994). Audio-Vision: Sound on Screen: Columbia University Press.

De Casper, A. J., & Fifer, W. (1980). Of human bonding: Newborns prefer their mothers’ voices. Science, 208, 1174-1176.

Drakakis, J. (Ed.). (1981). British Radio Drama. Cambridge: Cambridge University Press.

Drake, C., & Botte, M. C. (1993). Tempo sensitivity in auditory sequences: Evidence for a multiple-look model. Perception and Psychophysics, 54, 277-286.

Ferrington, G. (1993). ‘Audio design: Creating multi-sensory images for the mind’, electronic publication ‘http://interact.uoregon.edu/MediaLit/WFAEResearch/sndesign’ in the World Forum for Acoustic Ecology (WFAE).

Ferrington, G. (1995). Soudscape experiences. : ‘Soundscape Experiences Questionnaire’ Internet: WFAE Discussion Group. http://interact.uoregon.edu/MediaLit/WFAEHomePage.

Gaver, W. W. (1993a). How do we hear in the world: Explorations in ecological acoustics. Ecological Psychology, 5(285-313).

Gaver, W. W. (1993b). What in the world do we hear: An ecological approach to auditory event perception. Ecological Psychology, 5, 1-29.

Gibson, J. J. (1979). The Ecological Approach to Visual Perception: MIT Press.

Golinkoff, R. M. (Ed.). (1983). The transition from pre-linguistic to linguistic communication. Hillsdale, N.J.: Lawrence Earlbaum Associates.

Halpern, A. (1988). Mental scanning in auditory imagery for songs. Journal of Experimental Psychology: Learning, Memory and Cognition, 14, 434-443.

Hochberg, J. E. (1978). Perception. (Second ed.). New York: Prentice Hall.

Ingram, D. (1985). On childrens homonyms. Journal Of Child Language, 12(3), 671-680.

Kendall, R., & Lipscomb, S. D. (1997). Music and Film. .

Krueger, L. E. (1992). The word-superiority effect and phonological recoding. Memory and Cognition, 20, 685-694.

Krumhansl, C. L. (1991). Tonal structures in perception and memory. Annual Review of Psychology, 42, 277-303.

Kuhl, P. K., & Meltzoff, A. N. (1996). Infant vocalizations in response to speech: Vocal imitation and developmental change. Journal of the Acoustical Society of America, 100(4 Pt1), 2425-2438.

Lipscomb, S. D. (1989). Film Music Paradigm. Los Angelas: UCLA.

Lipscomb, S. D. (1997). How music can fit film, Internet discussion group: Analysis of film music and sound : izzy2of@mv.oac.ucla.edu.

Lipscomb, S. D., & Kendall, R. A. (1994). Perceptual judgement of the relationship between musical and visual components in film. Psychomusicology, 13, 60-98.

Miller, D. W., & Marks, L. J. (1992). Mental imagery and sound effects in radio commercials. Journal of Advertising, 21, 83-93.

Neumeyer, D. (1997). Source music, background music, fantasy and reality in early sound film. College Music Symposium, 37, 13-20.

Ong, W. J. (1971). World as view and world as event. In P. Shepard & D. McKinley (Eds.), Environ/mental: Essays on the Planet as Home. (pp. 61-79). New York: Houghton.

Rodaway, P. (1994). Sensuous geography: body sense and place. London: Routledge.

Rodger, I. (1982). Radio Drama. London: Macmillan.

Sabbadini, A. (1998). On Sounds, Children, Identity and a ‘Quite Unmusical’ Man. Sound Journal, 1(http://speke.ukc.ac.uk/sais/sound-journal/articleindex.html), 1-10.

Saussure, F. (1974). A course in general linguistics. London: Fontans.

Schafer, R. M. (1977). The tuning of the world. New York: Alfred A. Knopf.

Sharrock, W. C., J. (1998). On what we can see. Theory & Psychology, 8(2), 147-164.

Sircar, S. (1997). An annotated ‘Chhara-Punthi’ + India, folklore, oral tradition, music – Nursery rhymes from Bengal. Asian Folklore Studies, 56(1), 79-108.

Snow, C., & Ferguson, M. (Eds.). (1977). Talking to children: From input to acquisition. Cambridge: Cambridge University Press.

Steinberg, M. P. (1997). Music, musicology and film. Musical Quarterly, 81(2), 170-172.

Thorn, R. (1998). Hearing is Believing – the Evidence. Sound Journal, 1(http://speke.ukc.ac.uk/sais/sound-journal/thorn981.html), 35-45.

Zwick, E., & Horner, J. (1989). Glory: Tristar Pictures. USA.

AUDITORY PERCEPTION AND SOUND AS EVENT: THEORISING SOUND IMAGERY IN PSYCHOLOGY

Abstract:

The study of sound in psychology has been dominated by the auditory perception view of psycho-acoustics. This paper considers the nature of the relationship between sound as event and associated processes of imagery, imagination and memory. Through a consideration of sound(s) as ecological event(s), the role of sound in film and radio, and our earliest experiences of sound as language, the discussion centres on whether psychology can contribute to our understanding of sound imagery. Concluding comments touch on the observation that when hearing a sound, our imagination often plays an important part in recognising what it might be.

Michael A. Forrester

Department of Psychology

Keynes College

Canterbury

CT2 7NP

* Sections of this paper are to appear in a forthcoming book ‘Psychology of the Image’ published by Routledge.

AUDITORY PERCEPTION AND SOUND AS EVENT: THEORISING SOUND IMAGERY IN PSYCHOLOGY

INTRODUCTION