New Tools to Help Patients Reclaim Damaged Senses via Sensory Substitution

November 23, 2004
By SANDRA BLAKESLEE





Cheryl Schiltz vividly recalls the morning she became a
wobbler. Seven years ago, recovering from an infection
after surgery with the aid of a common antibiotic, she
climbed out of bed feeling pretty good.

"Then I literally fell to the floor," she said recently.
"The whole world started wobbling. When I turned my head,
the room tilted. My vision blurred. Even the air felt
heavy."

The antibiotic, Ms. Schiltz learned, had damaged her
vestibular system, the part of the brain that provides
visual and gravitational stability. She was forced to quit
her job and stay home, clinging to the walls to keep from
toppling over.

But three years ago, Ms. Schiltz volunteered for an
experimental treatment - a fat strip of tape, placed on her
tongue, with an array of 144 microelectrodes about the size
of a postage stamp. The strip was wired to a kind of
carpenter's level, which was mounted on a hard hat that she
placed on her head. The level determined her spatial
coordinates and sent the information as tiny pulses to her
tongue.

The apparatus, called a BrainPort, worked beautifully. By
"buzzing" her tongue once a day for 20 minutes, keeping the
pulses centered, she regained normal vestibular function
and was able to balance.

Ms. Schiltz and other patients like her are the
beneficiaries of an astonishing new technology that allows
one set of sensory information to substitute for another in
the brain.

Using novel electronic aids, vision can be represented on
the skin, tongue or through the ears. If the sense of touch
is gone from one part of the body, it can be routed to an
area where touch sensations are intact. Pilots confused by
foggy conditions, in which the horizon disappears, can
right their aircraft by monitoring sensations on the tongue
or trunk. Surgeons can feel on their tongues the tip of a
probe inside a patient's body, enabling precise movements.

Sensory substitution is not new. Touch substitutes for
vision when people read Braille. By tapping a cane, a blind
person perceives a step, a curb or a puddle of water but is
not aware of any sensation in the hand; feeling is
experienced at the tip of the cane.

But the technology for swapping sensory information is
largely the effort of Dr. Paul Bach-y-Rita, a
neuroscientist in the University of Wisconsin Medical
School's orthopedics and rehabilitation department. More
than 30 years ago, Dr. Bach-y-Rita developed the first
sensory substitution device, routing visual images, via a
head-mounted camera, to electrodes taped to the skin on
people's backs. The subjects, he found, could "see" large
objects and flickering candles with their backs. The
tongue, sensitive and easy to reach, turned out to be an
even better place to deliver substitute senses, Dr.
Bach-y-Rita said.

Until recently sensory substitution was confined to the
laboratory. But electronic miniaturization and more
powerful computer algorithms are making the technology less
cumbersome. Next month, the first fully portable device
will be tested in Dr. Bach-y-Rita's lab.

The BrainPort is nearing commercialization. Two years ago,
the University of Wisconsin patented the concept and
exclusively licensed it to Wicab Inc., a company formed by
Dr. Bach-y-Rita to develop and market BrainPort devices.
Robert Beckman, the company president, said units should be
available a year from now.

Meanwhile, a handful of clinicians around the world who are
using the BrainPort on an experimental basis are effusive
about its promise.

"I have never seen any other device do what this one does,"
said Dr. F. Owen Black, an expert on vestibular disorders
at the Legacy Clinical Research and Technology Center in
Portland, Ore. "Our patients are begging us to continue
using the device."

Dr. Maurice Ptito, a neuroscientist at University of
Montreal School of Optometry, is conducting brain imaging
experiments to explore how BrainPort works.

Dr. Eliana Sampaio, a neuroscientist at the National
Conservatory of Arts and Métiers in Paris, is using the
BrainPort to study brain plasticity. Sensory substitution
is based on the idea that all sensory information entering
the brain consists of patterns carried by nerve fibers.

In vision, images of the world pass through the retina and
are converted into impulses that travel up the optic nerve
into the brain. In hearing, sounds pass through the ear and
are converted into patterns carried by the auditory nerve
into the brain. In touch, nerve endings on skin translate
touch sensations into patterns carried into the brain.

These patterns travel to special sensory regions where they
are interpreted, with the help of memory, into seeing,
hearing and touch. Patterns are also seamlessly combined so
that one can see, hear and feel things simultaneously.

"We see with the brain, not with the eyes," Dr. Bach-y-Rita
said. "You can lose your retina but you do not lose the
ability to see as long as your brain is intact."

Most important, the brain does not seem to care if patterns
come from the eye, ear or skin. Given the proper context,
it will interpret and understand them. "For me, it happened
automatically, within a few minutes," said Erik
Weihenmayer, who has been blind since he was 13.

Mr. Weihenmayer, a 35-year-old adventurer who climbed to
the summit of Mount Everest two years ago, recently tried
another version of the BrainPort, a hard hat carrying a
small video camera. Visual information from the camera was
translated into pulses that reached his tongue.

He found doorways, caught balls rolling toward him and with
his small daughter played a game of rock, paper and
scissors for the first time in more than 20 years. Mr.
Weihenmayer said that, with practice, the substituted sense
gets better, "as if the brain were rewiring itself."

Ms. Schiltz, too, whose vestibular system was damaged by
gentamicin, an inexpensive generic antibiotic used for
Gram-negative infections, said that the first few times she
used the BrainPort she felt tiny impulses on her tongue but
still could not maintain her balance. But one day, after a
full 20-minute session with the BrainPort, Ms. Schiltz
opened her eyes and felt that something was different. She
tilted her head back. The room did not move. "I went
running out the door," she recalled. "I danced in the
parking lot. I was completely normal. For a whole hour."
Then, she said, the problem returned.

She tried more sessions. Soon her balance was restored for
three hours, then half a day. Now working with the
BrainPort team at the University of Wisconsin, Ms. Schiltz
wears the tongue unit each morning. Her balance problems
are gone as long as she keeps to the regimen.

How the device produces a lasting effect is being
investigated. The vestibular system instructs the brain
about changes in head movement with respect to the pull of
gravity. Dr. Bach-y-Rita speculated that in some patients,
a tiny amount of vestibular tissue might survive and be
reactivated by the BrainPort.

Dr. Black said he had seen the same residual effect in his
own pilot study. "It decays in hours to days," he said,
"but is very encouraging."

Blind people who have used the device do not report lasting
effects. But they are amazed by what they can see. Mr.
Weihenmayer said the device at first felt like candy pop
rocks on his tongue. But that sensation quickly gave way to
perceptions of size, movement and recognition.

Mr. Weihenmayer said that on several occasions he was able
to find his wife, who was standing still in an outdoor
park, but he admitted that he also once confused her with a
tree. Another time, he walked down a sidewalk and almost
went off a bridge.

Nevertheless, he is enthusiastic about the future of the
device. Mr. Weihenmayer likes to paraglide, and he sees the
BrainPort as a way to deliver sonar information to his
tongue about how far he is from the ground.

Dr. Ptito is scanning the brains of congenitally blind
people who, wearing the BrainPort, have learned to make out
the shapes, learned from Braille, of capital letters like
T, B or E. The first few times they wore the device, he
said, their visual areas remained dark and inactive - not
surprising since they had been blind since birth. But after
training, he said, their visual areas lighted up when they
used the tongue device. The study has been accepted for
publication in the journal Brain.

Dr. Ptito says he would like to see if he could teach his
subjects how to read drifting letters like those in
advertising displays. Not seeing motion is a big problem
for the blind, he said.

In another approach, Dr. Peter Meijer, a Dutch scientist
working independently, has developed a system for blind
people to see with their ears. A small device converts
signals from a video camera into sound patterns delivered
by stereo headset to the ears. Changes in frequency connote
up or down. Changes in pixel brightness are sensed as
louder or softer sounds.

Dr. Yuri Danilov, a neuroscientist and engineer who works
with Dr. Bach-y-Rita, said the research team had thought of
dozens of applications for the BrainPort, which he called a
"USB port to the brain."

In one experiment, a leprosy patient who had lost the
ability to experience touch with his fingers was outfitted
with a glove containing contact sensors. These were coupled
to skin on his forehead. Soon he experienced the data
coming from the glove on his forehead, as if the feelings
originated in his fingertips. He said he cried when he
could touch and feel his wife's face.

The federal government has also shown interest in sensory
substitution technology. The Navy is exploring the use of a
tongue device to help divers find their way in dark waters
at night, said Dr. Anil Raj, director of the Institute for
Human and Machine Cognition at the University of West
Florida in Pensacola.

The sensors detect water surges, informing Navy Seals if
they are following the correct course. The Army is thinking
about sending infrared signals from night goggles directly
to the tongue, Dr. Raj said.

In another application, student pilots have been fitted
with body sensors attached to aircraft instruments. When
the airplane starts to pitch or change altitude, they can
feel the movements on their chests.

Sensory substitution technology may eventually help
millions of people overcome their sensory disabilities. But
the devices may also have more frivolous uses: in video
games, for example.

Dr. Raj said the tongue unit had already been tried out in
a game that involved shooting villains. "In two minutes you
stop feeling the buzz on your tongue and get a visual
representation of the bad guy," he said. "You feel like you
have X-ray vision. Unfortunately it makes the game boring."