Sound—Image—Text—Motion. An Overview of Audiovisual Explorations.

In this brief paper, I intend to map the concept of complementary relationships of sound, image, text and motion as the languages of communication, in the context of design education. What has brought me to this discourse — and to Motyf conference — is a need to recognize the “scholarship of teaching” (Ernest Boyer¹) as a design research area, associated with developing new pedagogies and curricula in response to the shifting paradigms of design-based professions. A unique concept of Motyf (Motion typography festival) provides a platform of such discourse in the area of kinetic typography and motion graphics. However, the need for “scholarship of teaching” pertains to all disciplines of design and beyond.

At Motyf 2013, in the paper entitled “Type on Wheels”², the authors postulated that motion (and motion design) is a language of communication. The paper proposed that the language of motion, as integral to design, should be taught at all levels of design curriculum, rather than teaching kinetic typography alone, as an “end product.” This current paper, originally submitted to Motyf 2014, (the 2014 conference’s subtitle was “Type in Music”), expands the scope of investigating language of motion into broader contexts of “audiovisual explorations.” The word “overview” in the title, suggests a wide angle lens of observation, a sort of “google earth” view on the subject, beyond the topic of typography in motion.

Audio-visuality

Stimuli in the domain of sight and sound make up the core of human experience. Consequently, vision and hearing are the most involved modalities in multimedia communication. John Cage wrote about vision and hearing as the “public senses.”³ “We have eyes as well as ears, and it is our business while we are alive to use them.” (John Cage⁴) Indeed, relations of vision and hearing, both as a sensory experience as well as a linguistic expression, evoke multiple questions within multiple theories and histories of communication and media.

We can trace in the history of the last two centuries, a noticeable effort to develop tools and technologies of capturing, reproducing and circulating content in the domain of image, sound and motion. The invention of “technical image” (as Vilém Flusser⁵ describes it), begins a sequence of rapid changes in the spectrum of communication tools — via invention of photography, cinematography, phonographic technology, sound film, video, digital media, etc. — bringing the forms of expression and communication closer and closer to its natural environment of audio-visuality. “What phonographs and cinematographs, whose names not coincidentally derive from writing, were able to store was time: time as a mixture of audio frequencies in the acoustic realm and as the movement of single-image sequences in the optical.” (Friedrich Kittler⁶) Ultimately time, and therefore motion as an integrating element of sequentiality, contributes an essential structure to audiovisual narrative, with its beginning, middle and end.

Time and Motion

Motion is integral to design. The notion of time, intertwined with motion, is considered the organizing principle to which all other design elements must relate. Audiovisual experience is about the process of forming rather than a static form itself. The meaning of motion and concepts of sequentiality already explored within multiple disciplines of art and science have become part of the core vocabulary of communication design.

As the mind perceives visual, sonic, and kinetic information over a period of time, it continuously organizes discrete units or messages into a narrative, however abstract that “cinematic” story might be. Indeed, in its more than a century-long history, the language of cinema has evolved into a complex, universally understood system of communication, capable of translating a multi-sensory human experience into a kinetic sequence of audiovisual events, where motion serves to integrate all other channels of communication.

The language of motion involves issues of “what” is moving and “how” that something is moving. The how question refers to the kinetic form and its grammar, defined by space and time dimensions. Kinetic “behavior” itself, contributes an additional layer of meaning to the objects that already convey messages expressed in their own native language of pictures, or words and numbers, as in case of kinetic typography (the core focus of Motyf conferences). Indeed, kinetic typography as a result of the evolution of verbal communication — from spoken language (sound) to text (image), to typography (technical text) and to typography in motion (audiovisual experience) — can be itself a case study of complex relationships of phonemes and graphemes (phonetic and visual elements) in linguistic communication.

Audiovisual explorations seem to have a new life in the the 21st century, due to proliferation of authoring tools and technologies of audiovisual content, for both personal and professional use, as well as rapid development of accessible platforms for publishing and distribution of dynamic media.

In the following paragraphs, I intend to present a brief overview of a limited number of publications, as well as a very limited selection of historic audiovisual work, related to the topic of audio-visuality. This is a personal and certainly idiosyncratic list of materials, which I use in my studio and liberal arts courses. I consider this listing of resources, a project in progress aiming at assembling a new infrastructure for dynamic media design, research and pedagogy.

Selected Literature on the Subject

The 2009 book titled Audio.Visual — On Visual Music and Related Media,⁷ explores a relatively new genre of VJ-ing, that is mixing sound with moving image — live, and often through improvisations — in public audiovisual spectacles, concerts etc. The term “visual music” in the title, has become lately a generic term to describe not only the genre of VJ-ing but the entire audiovisual spectrum.

This 2013 monograph: Brian Eno Visual Music,⁸ uses the same term again. The book gives us an insight, as well as evidence, of an inseparable nature of moving image and sound in the work of the artist. “The Aesthetics of Time” is the term the author uses while discussing Eno’s work.

The 2005 book Visual Music. Synaesthesia in Art and Music Since 1900,⁹ was the original trigger of the popularity of the term. It is a monumental catalog of the 2005 exhibition presented by Hirshhorn Museum and Sculpture Garden, Smithsonian Institution in Washington DC, and the Museum of Contemporary Art in Los Angeles.

The exhibition and the book propose a revision, or perhaps an update, in how we should reference the history of art in the context of dynamic media pedagogy. The curators traced an idea of correspondence between non-representational aspects in fine art and music, instead focusing on individual artists or art movements. To describe visual music the curators use the term “synesthesia.” Synesthesia is a neurological phenomenon in which stimulation of one sensory input triggers automatically, and involuntarily, the sensory experience in another domain — very often associating sound and color (known as chromesthesia). The term synesthesia is used here as a metaphor related to all multi-sensory experiences.

Another interesting exhibition took place in Lentos Kunstmuseum, in Linz, Austria. The 2009 publication titled See This Sound. Promises in Sound and Vision,¹⁰ was the original catalog of the show. The following years, two additional volumes were published as well, sponsored by Ludvig Boltzmann Institute (Media.Art.Research). The first (2010) titled Audiovisuology: Compendium,¹¹ was described in its subtitle as “an interdisciplinary survey of audiovisual culture.” The second (2011) titled Audiovisuology 2: Essays,¹² was described as “histories and theories of audiovisual media and art.”

These three books together constitute a comprehensive educational resource, which combines research and expertise from multiple relevant fields of art and design history, theory and philosophy of media, musicology, cinema and theatre studies, pop culture, and many others. Such blend of disciplines represents a new cluster of content, as well as research methodologies, of how we should approach dynamic media discourse in education.

But of course, there are some classics on the subject, such as the 1980 book Digital Harmony: On the Complementarity of Music and Visual Art,¹³ by John Whitney. The book explores a single hypothesis, that through code and mathematical expressions, the harmony and complementarity of sound and visual domain reached its new representation. This is one of the earliest works on the subject of digital visualization of music, and it refers, of course, to the cinematic work of the author (which I will examine in more detail later). In the introduction, John Whitney refers to the conventions of harmony founded upon laws of Pythagoras, and the brief history of redefinitions of the concept of harmony through the work of artists and musicians including Richard Wagner, dadaists, or Igor Stravinsky who is known to remark “harmony is dead.”¹⁴

Another classic is the 1967 book entitled Verbi-Voco-Visual Explorations¹⁵ by Marshal McLuhan. This text refers to various expressions of language through various technologies of communication. It points, as majority of the work of McLuhan, at the multi-sensory nature of human experience of language and communication.

Selected Audiovisual Work

This very selective list of samples used in my classroom, I must begin with a classic Diagonal Symphony, film by Viking Eggeling, Germany 1921–1924. This experimental abstract film gained him a reputation of a pioneer and one of the most original artist of the time-base medium. Born in Sweden in 1880, he worked in Paris, Zurich and Berlin.¹⁶ In Diagonal Symphony Eggeling used a simple and rather limited visual vocabulary of abstract forms to “orchestrate” (his term) their rhythmic and precise visual permutations on screen. The film delivers a dynamic sequence of contrasting opposites constantly changing their visual appearance and relationships over time. The moving lines and shapes convey an abstract narrative similar to a musical experience. I consider this work an audiovisual experiment, although the intention of Eggeling was to project this film as silent. Eggeling must have believed that there is a synesthetic ability in all of us.

Soon after the premiere of Diagonal Symphony Eggeling died prematurely in 1925. In the book entitled “Painting, Photography and Film“ published in Munich the same year (originally in German, published as the Bauhaus Book, Volume 8, 1925), Laszlo Moholy-Nagy, himself a very influential artist and educator and proponent of photography and film as “new media” of his time, described Eggeling’s experiments as “a new instrument which […] produced […] the articulation of space in motion.”¹⁷ According to Moholy-Nagy “Eggelling […] further developed the importance of the time problem, which revolutionized the whole existing aesthetic” (translation by Janet Seligman, MIT Press, 1969).¹⁸

Born in Germany in 1900, Oskar Fischinger¹⁹ elevated the abstract film and audiovisual experience to a new and unprecedented level. In each of the Film Studies (1927–1932) Fischinger explored different forms of visual and kinetic vocabulary, precisely synchronized with a particular musical movement (and for that reason, the Film Studies can be seen as a precursor to music videos of the 1980s). These films were shown in theaters in Europe, the United States and Japan to a high acclaim of critics and audiences.

In the 1931 black and white film Film Study No. 8 (he added color to his vocabulary in later films), Oskar Fischinger (abstract painter himself) experiments on screen with a sequence of very complex kinetic gestures to complement and interpret a syncopated rhythm of Paul Dukas’ “The Sorcerer’s Apprentice” (the same music was used later in Fantasia sequence, on which Fischinger worked while on contract with Walt Disney). Through balancing tension and release, the film presents a narrative of sight and sound unified by the language of motion. In that audiovisual experience the domains of sound and vision seem inseparable. Later, in the Art in Cinema catalog (San Francisco Museum of Art, 1947) Fischinger described his understanding of the concept of visual music:

“Under the guidance of music […] there came the speedy discovery of new laws—the application of acoustical laws to optical expression was possible. As in the dance, new motions and rhythms sprang out of the music—and the rhythms became more and more important. I named these absolute films Studies; and I numbered them Study No. 1, Study No. 2, and so forth. These early black and white studies drew enthusiastic response at the time from the most famous art critics of England and Europe. Then came the color film. Of course, the temptation was great to work in color, and I made thereafter a number of absolute color films. But I soon found out that the simplicity of my own black and white films could never be surpassed.”²⁰

Obviously, there is a difference between Eggeling and Fischinger in how audio and visual components are mapped to each other. The rigor and precision of Fishinger’s film-scores²¹ was unprecedented as well. The film scores, or maps, synchronize the elements of visual form with the elements of music to a millisecond. Those film scores are reminiscent of contemporary musical scores. Indeed, John Cage credited Oskar Fischinger as a very influential in his own work, especially in relation to the exhibition and a seminal book on visual music notations.²²

The work titled Ornament Sound — was Oskar Fischinger’s radical experiment of creating sound generated from visual source. He called these modular, drawn manually geometric strips “ornaments.”²³ The ornaments were transferred photographically onto optical track of the film and played as sound through film projector. That was a reversal of the process — the idea which was later continued by Norman MacLaren, a Canadian animator and filmmaker (and many others).

The work of Mary Ellen Bute (1906–1983) deserves to be better known.²⁴ She made her first black and white experimental films in the 1930s and from early 1940s she continued working in color. She was head of Expanding Cinema, the company she organized in the mid-1930s. Almost all of her abstract films were set to specific musical movements of various composers and in those cases music was the defining, structural component of the film sequence. In a sense she continued a long modernist tradition of exploring synesthetic relationship of sound and image. However, her interests seem influenced not only by modern art but by technologists, engineers and scientist who informed and inspired her work.

At some point, she collaborated with Leon Theremin, the inventor of the first electronic musical instrument. Eventually, Mary Ellen Bute explored an experimental oscilloscope technology (with help from Bell Telephone Laboratories) to control source of light as a drawing instrument while visualizing music.

“The figures and form on the oscilloscope can be made to move on the horizontal and vertical planes, toward or away from the spectator; their shapes can be varied as much and as often as one pleases; the tempo of their movement can be changed at will (the physics of these tempos is a study in itself); luminescence and shadow can be deployed; and the illusion of three dimensional space can be aroused. And all this can be synchronized to music.”²⁵

Especially relevant to the topic of kinetic typography is Mary Ellen Bute’s film “Tarantella” (1940),²⁶ the early color film to the music of Edwin Gershefsky with an unconventional treatment of movie titles. It is worth to note that although “Tarantella” as an independent production and had only a limited distribution the experimental and innovative movie titles design by Mary Ellen Bute preceded the work of Saul Bass.

John Whitney (1917–1995),²⁷ was an influential artist, engineer, and explorer of advanced technology aiming to synthesize the domains of vision and sound. He successfully integrated musical composition with time-base medium of computer-generated imagery. He coined the term “motion graphics” (and founded Motion Graphics Incorporated in 1960) and by many he is considered the “father of computer graphics.”

John Whitney’s first abstract films were made in collaboration with his brother James Whitney (1921–1981) between 1939 and 1944. The Whitney brothers built a unique set of specifically designed optical and acoustic devises — the optical printer and the pendulum sound recorder — to meet the artistic goal of their experiments. The result of their collaborative work, Five Abstract Film Exercises, was recognized internationally and won first prize (Grand Prix) at experimental film festival in Belgium in 1949.

In the 1950s John Whitney began producing commercial 16mm films. During that time he collaborated with Saul Bass on the animated sequence for Alfred Hichcock’s Vertigo. The rotating and expanding mathematical figure superimposed on Kim Novak’s pupil seems a signature form of John Whitney. In 1957 he worked with Charles Eames as a film consultant to create a seven-screen presentation for the Buckminster Fuller dome at the exhibition in Moscow — the large scale, multiple-screen projection.

However, it would be the late 1960s and the 1970’s that defined John Whitney’s oeuvre—the synthesis of sounds and visuals in the domain of computation. First, using the analog computing machine and the process of coloring his films in post production with the optical printer, he eventually turned to digital computing.

From an amazing body of John Whitney work, two are considered “the seminal computer films” — the original audiovisual explorations in the digital domain. The film Permutations (5-minute, color, 16mm),²⁸ premiered in 1968, and the film Arabesque (7-minute, color, 16mm)²⁹ premiered in 1975, which John Whitney created with help of the National Endowment for the Arts grant and IBM sponsorship. Both films synthesize a flow of exotic music with the perfectly harmonized, dynamic patterns of colored abstract forms, and in both John Whitney managed to balance aesthetics with the scientific and engineering aspects of digital production. These films define the computer-based dynamic media as a legitimate form of art and design.

Coda

Eventually, in digital communication, information will blur the differences among individual channels. Sound and image, text and voice will reside side by side within consumer interface fulfilling digital paradigm that “any medium can be translated into any other.” (Friedrich Kittler³⁰) Indeed, the interface — an audiovisual experience — can be considered a tool for narration and narrative itself. Or, to paraphrase Marshall McLuhan, ”the interface is the message” (Aaron Koblin³¹).

Described above and listed below is a partial list of my teaching resources — a modest evidence of never ending project of designing and redesigning of “what” and “how” and “why” we teach what we teach, including the topic of audiovisual communication. The truly challenging work in front of us design educators, is education itself. The curriculum of design, as well the process of designing the curriculum. Moreover, the pedagogy of design, as well as the process of designing new pedagogies. And there is another challenge: To share our thoughts with everybody else. Please find my contribution below (with full awareness of how incomplete it is).

 

References

1. Ernest L. Boyer. Scholarship Reconsidered: Priorities of the Professoriate, (New York: The Carnegie Foundation for the Advancement of Teaching, 1990), 16.
2. Jan Kubasiewicz, Brian Lucid, “Type on Wheels: Two Voices on Teaching the Language of Motion,” in: Motyf 2013, (Warsaw, 2014), 149.
3. William Fetterman, John Cage’s Theatre Pieces, (New York, London: Routledge, 2010), 21.
4. John Cage, “Experimental Music,” in: Silence, (Wesleyan University Press, 1973), 12.
5. Vilém Flusser, Toward a Philosophy of Photography, (London: Reaktion Books, 2000), 14.
6. Friedrich A. Kittler, Gramophone, Film, Typewriter, (Stanford, CA: Stanford University Press, 1999), 3.
7. Cornelia Lund, Holger Lund, Ed., Visual—On Visual Music and Related Media, (Stuttgart: Arnoldsche Art Publishers, 2009).
8. Christopher Scoates, Brian Eno Visual Music, (San Francisco: Chronicle Books, 2013).
9. Kerry Brougher, Olivia Mattis, Jeremy Strick, Ari Wiseman, Judith Zilczer,Visual Music: Synaesthesia in Art and Music Since 1900, (New York: Thames and Hudson, 2005).
10. Cosima Rainer, Stella Rollig, Dieter Daniels, Manuela Ammer, Ed., See This Sound: Promises in Sound and Vision, (Cologne: Walther König, 2009).
11. Dieter Daniels, Sandra Nauman, Ed., Audiovisuology: Compendium, (Cologne: Walther König, 2010).
12. Dieter Daniels, Sandra Nauman, Ed., Audiovisuology 2: Essays, (Cologne: Walther König, 2011).
13. John Whitney, Digital Harmony: On the Complementarity of Music and Visual Art, (Peterborough, NH: Byte Books / A McGraw-Hill publication, 1980).
14. Ibid, 5.
15. Marshal McLuhan, Verbi-Voco-Visual Explorations, (New York: Something Else Press, 1967).
16. Louise O’Konor, Viking Eggeling 1880–1925: Artist and Film-maker, Life and Work, (Stockholm, Almvqvist & Wiksell, 1971).
17. Laszlo Moholy-Nagy, Painting, Photography and Film, (Cambridge, MA: MIT Press, 1969), 21.
18. Ibid, 21.
19. William Moritz, Optical Poetry: The Life and Work of Oskar Fischinger, (Bloomington, IN: Indiana University Press, 2004).
20. Oskar Fischinger, “My Statements Are in My Work,” in: Art in Cinema catalog (San Francisco Museum of Art, 1947), 38.
21. Cindy Keefer, Jaap Guldemond, Ed., Oskar Fischinger 1900–1067: Experiments in Cinematic Abstraction, (Amsterdam: EYE Filmmuseum, 2012), p.148.
22. John Cage, Notations, (New York: Something Else Press, 1969).
23. Cindy Keefer, Jaap Guldemond, Ed., Oskar Fischinger 1900–1067: Experiments in Cinematic Abstraction, (Amsterdam: EYE Filmmuseum, 2012), p.98.
24. Sandra Neuman, “Seeing Sound: The Short Films of Mary Ellen Bute,” in: Cornelia Lund, Holger Lund, Ed., Visual—On Visual Music and Related Media, (Stuttgart: Arnoldsche Art Publishers, 2009).
25. Mary Ellen Bute, “Abstronics” in: Film in Review, June-July 1954
26. Mary Ellen Bute, Dir. Tarantella, (1940), in: Cornelia Lund, Holger Lund, Ed., Visual—On Visual Music and Related Media, DVD, (Stuttgart: Arnoldsche Art Publishers, 2009).
27. “The John Whitney Biographical Website,” https://www.siggraph.org/artdesign/profile/whitney/whitney.html (January, 2015).
28. John Whitney, Dir., The John Whitney Collection, DVD, (Santa Monica, CA: Pyramid Media, 2009).
29. Ibid.
30. Friedrich A. Kittler, Gramophone, Film, Typewriter, (Stanford, CA: Stanford University Press, 1999), 3.
31. Aaron Koblin, “TED Lecture: Artfully Visualizing Our Humanity,” 2011, http://www.ted.com/talks/lang/en/aaron_koblin.html

 

Download PDF (11MB) of the essay excerpted from the book Motyf 2014, published by University of Applied Sciences, Mainz, Germany and Polish-Japanese Academy of Information Technologies, Warsaw, Poland.