There is some debate whether
cases of sensory substitution (1) are the results of imaginativeness,
psychological effects or neurological (mis-)wiring. For our purposes,
this distinction need not be relevant. We have evidence such substitutions
do happen and that trumps any theory answering why. Whether or
not Wagner, Klee or Kandinsky actually had synesthesia (2), there
is a rich history of people equating one type of sensory stimuli
for another. Most often we hear about organs, such as Scriabin's,
that emitted colored light shows with each note. There has been
a lot of experimentation over the years in search of correspondences
between absolute frequencies of light, and absolute frequencies
in sound, as well as in the cyclical orbit of planets and chakras
(3). We can chalk it up to over-eager spiritualism, but who can
really argue if in the end this proves inspirational to the composers.
It does seem peculiar that relative
frequencies (for example musical intervals) are rather rare. Joseph
Albers famous color studies (4) touch on relativistic issues,
but the book is certainly not about music. (Or does that matter?)
"A ‘transformation' of 4 reds to 4 blues of a slightly
lower key seems comparable to a transformation of a tetrachord
in music from one instrument to another. Therefore we are also
concerned with ‘intervals.'" (5) Though not altogether
literally true or even a very well articulated point, the analogy
is a telling one. Simply it reveals that there is a metaphorical
correlation we tend to make in language (for instance, extension
of the metaphor we use to say "a brighter hue" can readily
be extended to describe a tone). Extension of metaphors that act
as building blocks for comprehension is hardly a radical notion
(6). If we understand how a piece of string works, we can use
this concept as a metaphor when imagining a number line. In a
similar way, both traditional musical and/or visual art students
will recognize the idea of "intervals".
As for music, the relativistic case
is far more formalized thanks to Western music theory. Within
a key, obviously the root note has a distinct sound from the minor
second above it. But C in the key of G is furthermore perceived
differently than the same C in the key of F#. Even accounting
for perfect pitch or the ability to identify the name of the note,
the context is the over all music. Naming the notes doesn't make
them sound any more like music than noise. To detect musical-ness
requires recognition of a context. This doesn't only apply to
the pitches but also rhythms. "In addition, some experimenters
have noted in passing that changing the metrical context of a
melody by presenting it with a different accompaniment, can make
it sound totally different." (7). Interestingly that context
is simply a cluster of more notes, symbiotically dependant on
the context, just as individual cars both part of and a delineate
traffic. It is a theme that recurs in studies of networking.
Investigations on the perception
of color are also enlightening. For instance, though languages
vary in the amount of vocabulary devoted to naming colors and
the number of them named even in cultures that only have words
for "black" and "white", there is a remarkable
agreement as to what the archetypal red is and the bluest blue.
There is also a peculiar agreement as to what the basic colors
are. The rainbow could be segmented an infinite number of ways,
but the divisions tend toward the same number and uneven spacing
on the spectrum.
predecessors in this work, Brent Berlin and Paul Lay,
had previously discovered a universality to the order
in which primitive cultures come to assign color names,
with very primitive cultures only having words for black
and white. More advanced cultures sequentially add red,
then yellow and green,(in either order), then blue, then
brown, and finally pink, orange and grey (these last four
in any order).
… There is, it appears, a natural way to
divide the spectrum into color categories, a division
that appears to have more to do with the color itself
and our human mechanism for experiencing it (our three
types of cones and how they are wired to our neurons)
than it has to do with any particular culture or language.
Be reassured: members of the Navajo Nation experience
blue and green just like everyone else who isn't color
… These biological
facts of our cognitive life are completely consistent
with Wittgenstein's insight that our language categories
have such prototypical ‘centers' (a ‘good
red', a ‘good chair') but fuzzy boundaries (reddish-orange,
a piano stool)." (8)
One aim of psychology and Cognitive
Science is to discover precisely why we so often do understand
such a correlation. There clearly is no physical reason that we
are so influenced by this adopted linguistic metaphor. We conceive
of pitches as having locations on a line, though neither sound
nor light behave linearly at all. These lines are merely conventions
of convenience, but ultimately imaginary abstractions.
Note that many cultures recognize
and extract scales of seven notes. They may choose 5 (often in
Africa) or even less. When cultures employ quarter notes, pitches
between the notes of our twelve note chromatic scale (as in Middle
Eastern music), these pitches are not treated like the others
but lead the listener to more "stable" notes (9). Hence,
they are not precisely what we would call a chromatic scale (all
the notes). The vast majority of music throughout the world then
employs a subset of 5-7 of usually 12. The construction of musical
theories across the globe, whether explicitly formalized or unconsciously,
like color perception, tend to delineate the perceptible spectrum
using a very similar strategy, though the physical events that
trigger these stimuli differ enormously.
Aesthetic and subjective issues
aside, it is hard to top the impressive results obtained by Paul
Bach-y-Rita and Carter Collins in the ‘70s. Theirs was a
very different approach. Rather than look for a correlation that
may or may not exist, they created conditions so that brains,
with their impulsive organization of incoming information, would
be forced to create/customize functions (and physical brain modules)
in order to use the new information. In both cases, they substituted
sight for touch using the brain's somatosensory map (basically
the sense of touch). There is a map of our bodies in our brains
deformed as larger neural territory that is necessary to correspond
to more sensitive body parts. The "somatosensory homonculous"
has enormous lips and very small ear lobes. Their idea allows
for individuals to "remap" touch stimuli to visial stimuli
by means of neuroplasticity (10), and absolutely blind people
as well as blind folded subjects could identify objects and play
"rock, paper, scissors.
Bach-y-Rita created a contraption
that "displays" a 7 x 7 pixel image in a configuration
of tiny jolts from a grid of electrodes placed on the tongue (11).
Though Bach-y-Rita claimed that the low resolution was insignificant
(12), even a cursory brush with computer graphics will show this
is hardly the case (13). Nonetheless, his boast that "I can
connect anything to anything." (14) is impressively justified.
Carter Collins faced the opposite
obstacle. Though his invention was also an impressive one by any
standard (15), it was less limited by a 100 x 100 pixel resolution.
A grid of approximately 1/2 square inch covered the back. This
is about as densely as the nerves on the back can locate though
(16). There seems to be no ideal location on the body where the
nerves are distributed densely enough over a wide enough area.
Though the back offered a convenient spread of skin, especially
given the size, this proves unwieldy. Subjects had to remain seated
and stationary to accommodate the larger cameras and stand back
then. The hands and lips offer far more information, but then
we are back to a physical limitation of the electronics. These
2 legendary researchers have since worked together, though Bach-y-Rita's
work has been far better publicized.
It should be noted though that there
really is nothing like "visual stimuli" once inside
the brain. The brain doesn't actually ever deal in images. It
sends information of all kinds to the prefrontal cortex (among
other places, but this is generally what where the action happens
that we are talking about). The prefrontal cortex is a black box
that does it's magic (we don't know much about how, but don't
need to at this point) and assembles an updated mental model of
the world from the stream of incoming bits of information (17).
We tend to picture this world, and thus imagine that what we see,
is a world "out there" that is intrinsically visible.
The visable-ness is merely a byproduct of our brains, not something
we can verify in the a priori universe. The same is true for sound
and the other senses. Thus these experiments are able to exploit
this scheme, all they need to do is fool the brain into thinking
the incoming stimulus is something to picture in the mental model.
The drive to limit color palettes
and musical scales in similar ways is not likely to be coincidental.
Just as phone numbers cannot be any number of digits or we tend
to name our children a number of syllables relatively normal for
our culture, there is a concept called chunking (18). Humans tend
to remember around 7 things. This number can be pushed or pulled
in a variety of ways, the most important being culture.
While there is absolutely nothing
similar about color and sound in the real physical world (even
the concept of frequencies is a bit imprecise, albeit useful in
conversing), the peculiar limitations in the way human brains
process these has features which we can use. There is even debate
if anything like qualia (the feeling that red-ness or
the smell of burnt toast evokes in the mind) actually occurs,
and is not simply a theoretical concept (despite Wittgenstein's
insightful theorizing about this). However it may not be intuitively
obvious, we can easily brush these notions aside. We can successfully
draw parallels between the ways the mind receives a chosen sensory
stimulus and another. Not the way they occur in the physical world
nor the way the brain processes them, but the way they are subjectively
No computer can predict what you
will feel from looking at a painting by Monet as opposed to a
painting by Lucian Freud obviously. Nor can it analyze a Beach
Boys tune and a piece by Bartok and come up with rules to generate
create a happier mood. But a computer can compare things like
warmth, brightness, shapes/textures (the number of hard edges
and softer edges). These things are likely to contribute to the
mood, though we don't need to know what that specific mood is.
Likewise, we can guess, employing the concepts of sensory substitution
outlined above, that hues are comparable to scale degrees,
how distant or usual a color is when compared to the general color
scheme correlates to how far from the root of a scale along that
scale is a given note. The result is neither a better or more
emotive image or sound but one that is likely to trigger emotions
in the same ways. The emotions need not be the same, but the audience
is then able to compare the way one stimulus feels with another.
(1) Bavelier, D. & Neville, H.: Cross-Modal Plasticity: Where
and How. Department of Brain and Cognitive Sciences, University
of Rochester (2002)
(2) Cytowic, R.: The Man Who Tasted Shapes. MIT Press, Cambridge
(3) Sacks, O: Musicophelia: Tales of Music and the Brain. p 171n,
Alfred A Kniopf, New York (2007)
Godwin, J: Music, Mysticism and Magic A Sourcebook. pp 216—229,
Penguin Group, New York (1986)
(4) Albers, J.: The Interaction of Color. Yale University Press,
New Haven (1963)
(5) ibid, p. 80
(6) Lakoff, G & Nunez, R: Where Mathematics Comes from. ch.
2, Basic Books, New York (2000)
(7) Temperley, D.: The Cognition of Basic Musical Structures.
MIT Press, Cambridge (2004) refereeing to referring to Pavel &
Essens, p. 432. (1985))
(8) Hundert, E. Lessons from an Optical Illusion. pp. 169-170.
Harvard University Press, Cambridge (1995)
(9) May, E. (Ed.): Music of Many Cultures. Chapters 14, 15. University
of California Press, Berkeley (1980)
Levitin, D.: This Is your Brain on Music. p. 61. Penguin Books,
New York (2006)
(10) Doidge, N.: The Brain that Changes Itself. chapter 1. Viking,
New York (2007)
(11) Bach-y-Rita, P.: Brain Mechanisms in Sensory Substitution.
Academic Press, New York (1972)
Bach-y-Rita, P. & Kaczmarek, K. & Tyler, M. & Garcia-Lara,
J.: Form perception with a 49-point electrotactile stimulus array
on the tongue: A technical note. Journal of Rehabilitation Research.
(35) pp. 427-430. (1998)
(12) #10, p. 12
(13) Myler, H. & Weeks, A.: The Pocket Handbook of Image Processing
Algorithms in C. Prentice Hall, Upper Saddle River (1993)
Gonzales, R. & Wintz, P.: Graphics Programming. Addison Wesley,
(14) #10, p. 15
(15) Ornstein, R.: Psychology. pp. 255-256. Harcourt Brace Javonovich
Publishers, USA (1985)
(16) Stafford, T. & Webb, M.: Mind Hacks: Tips and Tools for
Using your Brain. pp. 27-31. O'Reilly, Sebastopol (2005)
(17) Solso, R.: The Psychology of Art and the Evolution of the
Human Brain. pp. 254—259. MIT Press, Cambridge, MA (2003)
(18) Calvin, W.: How Brains Think. pp. 92—93. Basic Books,
New York, NY (1999)