A

Seminarreporton

“BRAINPORT VISION DEVICE”

Submitted to
R aj a st ha n T e ch ni cal U ni v er si ty , K ot a

In Partial Fulfillment BachelorofterequiremofTechnologyntfortheaward of the degree of

In
“Electronics & Communication Engineering”

2010-2011

Submitted by:- Supervised by:-

Koushal Singh Kiroula “Mr. Mansingh Negi”

(Dept. of Electronics & Communication Engg.)

ModiInstituteofTechnology
N a ya ga o n, R aw at b h at a Ro a d , Ko ta

1
 Chapter 1

INTRODUCTION

The BrainPort vision device provides information to blind individuals via a neuro-
stimulating array placed on the tongue. This unique device provides immediate benefits to
its blind users in areas of safety, mobility, and recreation and opens a new world of sensory
experience and exploration.

The BrainPort vision device operates by acquiring an image stream from a camera, similar
to a camcorder. Like a camcorder, the moving images are sent to a display, which, in this
case, is the electrode array on the tongue. The image stream is displayed on the tongue by
converting light information to electrical stimulation, which feels like microscopic bubbles
to the user. With minimal training, users learn to interpret the images on their tongue as
information about the scene in front of them.

The BrainPort vision device includes an imaging system capable of working both indoors
and outdoors, with a field of view spanning 3-75 degrees (magnified versus wide angle
views). The tongue array contains 400 electrodes and is connected to the controller via a
flexible cable. The control system is approximately the size of a PDA and runs for about 3
hours per charge, with swappable batteries.

An artist’s concept of the tongue array (or Intra-Oral Device, IOD) and camera mounting is
shown below alongside the device. The IOD is attached to a flexible boom by a thin wire.
The camera unit is mounted on a pair of eyeglasses frames. The user controls and the power
supply are connected to a belt-worn, pager-style, controller.

A blind woman sits in a chair holding a video camera focused on a scientist sitting in front of
her. She has a device in her mouth, touching her tongue, and there are wires running from that
device to the video camera. The woman has been blind since birth and doesn't really know what
a rubber ball looks like, but the scientist is holding one and when he suddenly rolls it in her
direction, she puts out a hand to stop it. The blind woman saw the ball through her tongue. Well,
not exactly through her tongue, but the device in her mouth sent visual

2
input through her tongue in much the same way that seeing individuals receive visual input
through the eyes. In both cases, the initial sensory input mechanism -- the tongue or the eyes -
- sends the visual data to the brain, where that data is processed and interpreted to form
images. Braille is a typical example of sensory substitution -- in this case, you're using one
sense, touch, to take in information normally intended for another sense, vision. Electro
tactile stimulation is a higher-tech method of receiving somewhat similar (although more
surprising) results, and it's based on the idea that the brain can interpret sensory information
even if it's not provided via the natural channel.

3
2.1 Statistics on the Blind:

 37 million: People in the world are blind India (9 million), Africa (7 million)
and China (6 million)
 Every 5 seconds: One person in our world goes blind
 75 million: People will be blind by
2020 (if trends continue)

2.2 Cybernetics:

 Cybernetics is about having a goal and taking action to achieve that goal.
 "Cybernetics" comes from a Greek word meaning "the art of steering“.
 Ironically but logically, AI and cybernetics have each gone in and out in the search
for machine intelligence
 So “I Can Read” can be termed as a “Cybernetics System For Disabled (blind)”

4
 Chapter 2

What is BrainPort Vision Device?

 "BrainPort device" is a technology developed in US, which is making the world

visible to the ones who lose their sight due to some accidental incidents.
 A Neuro scientist has developed the BrainPort Vision Device that allows the blinds to
“see” using their tongue.
 Craig Lundberg, 24, is the first British soldier to test the BrainPort system, which is
billed as the next best thing to sight
 The technology has made the dark-dependent world come alive and independent to
Craig Lundberg who completely lost his sight after a grenade attack in Iraq, as he is
now able to sense the visuals with his tongue. The soldier admits that his world has
been transformed because of the technology.
 The device which sends visual input through tongue in much the same way that
seeing individuals receive visual input through the eyes is called the “BrainPort
Vision Device”.
 BrainPort could provide vision-impaired people with limited forms of sight.
 Technically, this device is underlying a principle called “electro tactile stimulation
for sensory substitution”.
 To produce tactile vision, BrainPort uses a camera to capture visual data.

5
 Chapter 3

Modes of stimulation and its various forms

From the very beginning of the electro tactile stimulation this journey has travelled a lot
and the various forms may be described as follows:

 TVSS-Tactile Vision Substitution Systems.

 Vibrotactile (FINGERTIPS).
 Electro Tactile Stimulation for Tongue

3.1 Tactile vision substitution system

The TVSS may be characterized as a humanistic intelligence system. It represent a symbiosis

between instrumentation-for example, an artificial sensor array (TV camera)-computational
equipment, and the human user. Consistent with the terminology of this issue, this is made
possible by "instrumental sensory plasticity," the capacity of the brain to reorganize when there
is: (a) functional demand,(b) the sensor technology to fill that demand, and (c) the training and
psychosocial factors that support the functional demand. To constitute such systems then, it is
only necessary to present environmental information from an artificial sensor in a form of
energy that can be mediated by the receptors at the human-machine interface, and for the brain,
through a motor system (e.g., a head-mounted camera under the motor control of the neck
muscles), to determine the origin of the information.

This can now be extended into other domains with modern technology and the availability
of artificial sensory receptors, such as:

1. A miniature TV camera for blind persons,

2. A MEMS technology accelerometer for providing substitute vestibular information for
persons with bilateral vestibular loss,
3. Touch and shear-force sensors to provide information for spinal cord injured persons,
4. Instrumented condom for replacing lost sex sensation, or
5. A sensate robotic hand (Bach-y-Rita, 1999).

6
In first sensory substitution project, they developed tactile vision substitution Systems
(TVSS) to deliver visual information to the brain via arrays of stimulators in contact with
the skin of one of several parts of the body (abdomen, back thigh). Optical images picked
up by a TV camera were transuded into a form of energy (vibratory or direct electrical
stimulation) that could be mediated by the skin receptors. In these sensory substitute
systems, the visual information reaches the perceptual levels for analysis and interpretation
via somatosensory pathways and structures.

3.2 Vibrotactile

After sufficient training with the TVSS, our subjects reported experiencing the image in
space, instead of on the skin. They learn to make perceptual judgments using visual means
of analysis, such as perspective, parallax, looming and zooming, and depth judgments
(Bach-y-Rita, Collins, Saunders, White, & Scadden, 1969; cf., Bach-y-Rita, 1972, 1989,
1995, 1996, 1999; Bach-y-Rita, Kaczmarek, & Meier, 1998; Bach-y-Rita, Kaczmarek,
Tyler, & Garcia-Lara, 1998; Bach-y-Rita, Webster, Tompkins, & Crabb, 1987; Kaczmarek
& Bach-y-Rita, 1995; White, Saunders, Scadden, Bach-y-Rita, & Collins, 1970). Although
the TVSS systems have only had between 100 and 1032 point arrays, the low resolution has
been sufficient to perform complex perception and "eye"-hand coordination tasks. These
have included facial recognition, accurate judgment of speed and direction of a rolling ball
with over 95% accuracy in batting a ball as it rolls over a table edge, and complex
inspection-assembly tasks. The latter were performed on an electronics company assembly
line with a 100 point vibrotactile array clipped to the work-bench against which the blind
worker pressed the skin of his abdomen, and through which information from a TV camera
(Substituting for the ocular piece of a dissection microscope) was delivered to the human-
machine interface (Bach-y-Rita, 1995, pp. 187-193).

7
Child reproducing perceived image of a teachers hand as displayed on a modified Optacon.
The tactile image is picked up with one finger statically placed on the 6 × 24 vibrotactile
array. LED monitor in foreground is visual representation of active pattern on the tactile
display, which is obtained by the child's head-mounted camera.

8
3.3 Electrotactile Stimulation for Tongue

In the TVSS studies cited above, the stimulus arrays presented only black-white
information, without gray scale. However, the tongue electrotactile system does present
gray-scaled pattern information, and multimodal and multidimensional stimulation is
possible. Simultaneously, we have also modelled the electrotactile stimulation parameter
space to determine how we might elicit tactile "colors."

Aiello (1998a, 1998b) has identified six stimulus parameters: the current level, the pulse width,
the interval between pulses, the number of pulses in a burst, the burst interval, and the frame
rate. All six parameters in the waveforms can, in principle, be varied independently within
certain ranges, and may elicit potentially distinct responses. For example, in a study of electrical
stimulation of the skin of the abdomen, Aiello (1998a) suggested that the best way to encode
intensity information independent of other percept qualities with a multidimensional stimulus
waveform was through modulation of the energy delivered by the stimulus. In that case, the
energy was varied in such a way that the displacement in the parameter space, corresponding to
a given transition between energy levels, was minimal (gradient mode of stimulation). Although
the gradient mode of stimulation requires a real-time fulfilment of mathematical constraints
among all the parameters, its implementation could be included within a microelectronic
package for signal treatment. The tongue interface overcomes many of these. The tongue is very
sensitive and highly mobile. Since it is in the protected environment of the mouth, the sensory
receptors are close to the surface. The presence of an electrolytic solution, saliva, assures good
electrical contact. The results obtained with a small electrotactile array developed for a study of
form perception with a finger tip demonstrated that perception with electrical stimulation of the
tongue is somewhat better than with finger-tip electrotactile stimulation, and the tongue requires
only about 3% of the voltage (5-15 V), and much less current (0.4-2.0 mA), than the finger-tip.
The electronic system has been described elsewhere (Bach-y-Rita, Kaczmarek, Tyler, et al.,
1998).

9
Electrotactile stimuli are delivered to the dorsum of the tongue via flexible electrode arrays
(Figure 3) placed in the mouth, with connection to the stimulator apparatus (TDU) via a flat
cable passing out of the mouth. The tongue electrode array and cable are made of a thin
(100 μm) strip of polyester material (Mylar®) onto which a rectangular matrix of gold-
plated copper circular electrodes has been deposited by a photolithographic process similar
to that used to make printed circuit boards. The electrotactile stimulus consists of 40-μs
pulses delivered sequentially to each of the active electrodes in the pattern. Bursts of three
pulses each are delivered at a rate of 50 Hz with a 200 Hz pulse rate within a burst.

This structure was shown previously to yield strong, comfortable electrotactile precepts.
Positive pulses are used because they yield lower thresholds and a superior stimulus quality
on the fingertips.

Close-up of 144-point (12 x 12) "virtual ground" electrotactile tongue display

10
Action potentials (AP's) thus recorded had amplitudes from 0.1 to 1.0 mV and a 5 : 1 signal-to-
noise ratio (SNR).A circular electrode surrounding the recording site served as the ground
reference. Following pre amplification and band pass filtering (200-10 000 Hz), a differential
amplitude detector identified AP's, producing an output pulse whenever the recorded signal
entered a predefined amplitude-time window. In the first experiment, electrotactile entrainment
currents (iEN) were determined by adjusting the stimulation current from near zero to the
minimal value resulting in one AP for each stimulation pulse. These currents exceeded the
absolute thresholds (the currents causing occasional AP's) by approximately5%.

The entrainment current was determined twice for positive- and negative-polarity
stimulation pulses of ten different widths: 20, 30, 40, 50, 70, 100, 150, 200, 300, and 500
_s, delivered at a rate of 10 pulses/s. The width sequence was reversed during the second
run on each of the three fibers.

Relative timing between simultaneous mechanical and electrotactile stimulation. The

top trace represents the sinusoidal, 30-Hz, 50-100-_m (0-P) mechanical displacement.

11
 Chapter 4

Electrotactile Stimulation for Visual Substitution

When a human looks at an object, the optical image entering the eyes does not go beyond
the retina. Instead, it would turn into spatio-temporal nerve patterns of impulse along the
optic nerve fibers. By analysing the impulse patterns, the brain recreates the images.
Indeed, the channels such as eyes, ears and skin those carry sensory information to the brain
are set up in a similar manner to perform similar activities. To substitute one sensory input
channel for another, the big challenge to the scientists is how to correctly encode the nerve
signals for the sensory event and send them to the brain through the alternate channel as the
brain appears to the flexible when it comes to interpreting sensory input.

The concepts at work behind electro tactile stimulation for sensory substitution are complex.
The idea is to communicate non-tactile via electrical stimulation of the sense of touch.

In practice, this typically means that "an array of electrodes receiving input from a non-
tactile information source (a camera, for instance) applies small, controlled, painless
currents (some subjects report it feeling something like soda bubbles) to the skin at precise
locations according to an encoded pattern."

For a blind person, it means the encoding of the electrical Pattern essentially attempts to mimic
the input that would normally be received by the non-functioning sense – vision. So patterns of
light picked up by a camera to form an image are replacing the perception of the eyes and
converted into electrical pulses that represent those patterns of light. In other words, when the
encoded pulses are applied to the skin, the skin is actually receiving image data which would be
then sent to the brain in the forms of impulse. Under normal circumstances, the parietal lobe in
the brain receives touch information, while the occipital lobe receives vision information. When
the nerve fibers forward the image-encoded touch signals to the

12
parietal lobe, "the electric field thus generated in subcutaneous tissue directly excites the
afferent nerve fibres responsible for touch sensations".

Within the system, arrays of electrodes can be used to communicate non-touch information
through pathways to the brain normally used for the touch related impulses. The
breakthrough of the BrainPort technology is to use the tongue as the substitute sensory
channel.

13
 Chapter 5

THE STRUCTURE OF THE BRAINPORT DEVICE

The figure below shows the structure of the BrainPort Vision Device.

The optical information that would normally hit the retina is picked up by the digital
camera in digital form. It uses radio signals to send the ones and zeros to the CPU for
encoding. Each set of pixels in the camera's light sensor corresponds to an electrode in the
array. After that, the CPU runs a program that turns the camera's electrical information into
a spatially encoded signal. "The encoded signal represents differences in pixel data as
differences in pulse characteristics such as frequency, amplitude and duration.

14
Multidimensional image information takes the form of variances in pulse current or voltage,
pulse duration, intervals between pulses and the number of pulses in a burst, among other
parameters." Then, the electrode array (shown in Fig) receives the resulting signal via the
stimulation circuitry and applies it to the tongue. At last, the brain interprets and uses the
information coming from the tongue as if it were coming from the eyes.

HOW DOES THE BRAINPORT VISION DEVICE WORK?

The BrainPort vision system consists of a postage-stamp-size electrode array for the top
surface of the tongue (the tongue array), a base unit, a digital video camera, and a hand-held
controller for zoom and contrast inversion. Visual information is collected from the user-
adjustable head-mounted camera (FOV range 3–90 degrees) and sent to the BrainPort base
unit. The base unit translates the visual information into an stimulation pattern that is
displayed on the tongue. The tactile image is created by presenting white pixels from the
camera as strong stimulation, black pixels as no stimulation, and gray levels as medium
levels of stimulation, with the ability to invert contrast when appropriate. Users often report
the sensation as pictures that are painted on the tongue with Champagne bubbles.

15
16
With the current system (arrays containing 100 to 600+ electrodes), study participants have
been able to recognize high-contrast objects, their location, movement, and some aspects of
perspective and depth. Trained blind participants use information from the tongue display to
augment understanding of the environment. Our ongoing research with the BrainPort vision
device demonstrates the great potential of tactile vision augmentation and we believe that
these findings warrant further exploration. As a result, we are currently working on
improvements to the tongue display hardware, software, and usability, and on overall
device miniaturization.

WHY TONGUE IS USED IN BRAINPORT TECHNOLOGY

Compare to all other skin areas, the tongue skin area is the most sensitive one because there
are more nerve fibres and they are much closer to the surface. Moreover, there is no stratum
conium (an outer layer of dead skin cells) on the tongue which act as an insulator. With
these characteristics, it requires less voltage to stimulate nerve fibres in the tongue (5 to 15
volts) compared to areas like the fingertips or abdomen (40 to 500 volts). Also, since the
tongue is surrounded by saliva which contains electrolytes, it would help to maintain the
current flow between the electrode and skin tissues. Last but not least, the area of the
cerebral cortex that interprets touch data from the tongue is larger than the areas serving
other body parts. Therefore, the tongue is the best choice for conveying tactile-based data to
the brain until this moment

17
 Chapter 6

ADVANTAGES:

Blind users can use the Brain Port vision device independently - at home, at work, and in
public spaces indoors and out as a tool for improved safety, mobility and object recognition.
Secondary benefits include applying the technology toward specific hobbies and
recreational situations. These benefits enable greater independence at home, school and in
business, greatly improving quality of life.

SAFETY

 Navigating difficult environments, such as parking lots, traffic circles, complex

intersections
 Recognizing quiet moving objects like hybrid cars or bicycle

MOBILITY

 Finding doorways, hall intersections, lobby or restaurant in an office or hotel.

 Finding continuous sidewalks, sidewalk intersections and curbs.

18
 Chapter 7

OBJECT RECREATION

 Locating people
 Locating known objects such as shoes, cane, coffee mug, keys

APPLICATIONS

 Just a few of the current or foreseeable applications include providing elements of

sight for the visually impaired in the medical field.

 The BrainPort electrodes would receive input from a sonar device to provide not
only directional cues but also a visual sense of obstacles and terrain.

 BrainPort may also provide expanded information for military pilots, such as a pulse
on the tongue to indicate approaching aircraft or to indicate that they must take
immediate action.

 BrainPort applications include robotic surgery. The surgeon could wear electro tactile
gloves to receive tactile input from robotic probes inside someone's chest cavity.

19
20
 Chapter 8

CURRENT AND POTENTIAL APPLICATIONS

 While the full spectrum of BrainPort Vision Technology applications has yet to
realized, the device has the potential to lessen an array of sensory limitations and to
alleviate the symptoms of a variety of disorders.
 Just a few of the current or foreseeable applications include providing elements of
sight for the visually impaired in the medical field.
 Beyond medical applications, scientists have been exploring potential military uses
with a grant from the Defence Advanced Research Projects Agency (DARPA). They
are looking into underwater applications that could provide the Navy Seals with
navigation information and orientation signals in dark, murky water.
 The BrainPort electrodes would receive input from a sonar device to provide not
only directional cues but also a visual sense of obstacles and terrain.
 BrainPort may also provide expanded information for military pilots, such as a pulse
on the tongue to indicate approaching aircraft or to indicate that they must take
immediate action.
 Other potential BrainPort applications include robotic surgery. The surgeon could
wear electro tactile gloves to receive tactile input from robotic probes inside
someone's chest cavity. In this way, the surgeon could feel what he's doing as he
controls the robotic equipment.
 Race car drivers might use a version of BrainPort to train brains for faster reaction
times, and gamers might use electro tactile feedback gloves or their controllers to
feel what they're doing in a video game.

21
 Chapter 9

CONCLUSION

Even though this is a field of scientific study that has been around for nearly a century, it
has just picked up in this decade due to the miniaturization of electronics and dramatic
improvement of the computer's speed. Already more streamlined than any previous setup
using electro tactile stimulation for sensory substitution, BrainPort envisions itself even
smaller and less obtrusive in the future. In the case of the BrainPort Vision Device the
electronics might be completely embedded in a pair of glasses along with a tiny camera and
radio transmitter, and the mouthpiece would house a radio receiver to receive encoded
signals from the glasses. It's not exactly a system on a chip, but gives it 20 years -- we
might be seeing a camera with the size of a grain of rice embedded in people's foreheads.

22
REFERENCES

 Bach-y-Rita, Paul et al. "Form perception with a 49-point electro tactile stimulus
array on the tongue: A technical note." Journal of Rehabilitation Research and
Development.
 http://howstuffworks.com
 http://india.cchem.berkeley.edu/~reimer/courses/present/samples/sample2.pdf
 http://wistechnology.com/printarticle.php?id=2319
 http://www.itechnews.net/2009/08/16/brainport-vision-device-lets-blinds-see-
with-tougues/
 http://www.gizmowatch.com/entry/brainport-vision-device-for-the-
visually-impaired-to-feel-images/
 http://accessibletravel.suite101.com/article.cfm/brainport_vision_device
 http://www.scientificamerican.com/article.cfm?id=device-lets-blind-see-with-tongues
 http://topnews.net.nz/content/22766-brainport-device-transforms-world-blinds
 http://www.telegraph.co.uk/health/healthnews/7350218/British-soldier-blinded-
in-Iraq-trials-new-technology-to-see-using-his-tongue.html

23