Biological Model The basis for the vWISP neural network device

Peter AJ van der Made, Chief Scientist.

Introduction.
The human brain is amazingly complex and wonderfully designed. We can recognize people, places and circumstances in a fraction of a second, make decisions based on reasoning and act on those decisions. A computer is a calculating machine and cannot do these things. After more than 150 years of research, there is no shortage of documentation. If anything, too many papers deal with anything from the complete brain to minute details on the internal mechanisms of a synapse. Some papers are contradictory. The human brain is the most complex creation known in the universe. The central nervous system exists out of an estimated 100 billion (1011) specialized cells called neurons interconnected through up to 10,000 synapses each, and with 1 trillion (1012) glial support cells. The glial cells (Greek for glue) support the neurons with nutrition, remove dead neurons, form myelin that insulates axons from the surrounding tissue and destroy pathogens. There are an estimated 100 trillion (1014) synapses in the brain. Synapses are more than just connections; they hold the key to learning and knowledge. This paper describes the biological basis for NeoCortex digital devices. The design objective of the NeoCortex devices was to develop a new processing architecture that simulates the functions of a biological neural cell to a high degree of accuracy, and to form an array of such devices to simulate the function of the brain. . Purpose-built hardware performs at many times the speed of a computer executing a mathematical model of the brain in software. This makes realistic cortical processing feasible for small to medium sized computer systems, much in the same way that the introduction of signal processors in the 1980's has allowed small computers to process complex audio and video streams in real time.

Brain Physiology.
The brain consists of neurons organized in a hierarchy. The hierarchy consists of neurons organized in columns and groups. A full description of the mamalian brain will fill a large encyclopedia. Thousands of papers, books and articles have been written on the subject, varying in detail from the minute migration of charged particles in synaptic vesicles to global descriptions of the function of complete modules or the entire brain. For the purpose of building a digital model of a neuron we don't really care how ions migrate from one part of a synapse to another, or the biology of a neural cell. We are more concerned with the processing in input signals in a synapse and how this results in an output signal.

When we look at an image of the human brain it looks like an enormous greyish-pink wallnut. The wrinkly-looking area on the outside is the Cortex (bark), and is also referred to as the Neocortex. The cortex contains high-level modules for processing visual and audio information, Brocas area for the processing of complex grammar, the temporal lobe (awareness), Wernickes area for language comprehension and the sensory associative cortex. The cortex consists of 6 subdivided layers of neurons and is 1-4 mm thick. All the modular areas are in roughly the same place between individuals. Cradled within the Cortex lies the Limbic system and the brain stem (Pons), which connects to the spinal cord. The limbic system performs much of the pre-processing of sensory input, as well as post-processing and transmission of cortex output to the spinal cord and then to muscles and glands. Behind the Cortex, just above the neck, lies the Cerebellum (little brain). The Cerebellum primary role is to coordinate movement but is also involved in many learned automatic actions. Often, when we are cruising on automatic it is our cerrebellum that is in control. This allows us to perform one task such as driving a car, while thinking about other things. The Cerebellum has massive connections to other parts of the brain. The limbic system contains the Corpus Callosum, the Amygdala, the Thalamus, and the Hippocampus. The Corpus Callosum connects the two brain halves and is larger in females. The Hippocampus is involved in the formation of new memories. The Thalamus is a sorting center for the senses. Peope with damage to the Thalamus may see odours and smell colours. People with damage to the hippocampus remember all they have learned in the past, but can not form any new memories. The amygdala is the emotional centre. Measuring about the size of a 30 by 30 cm handkerchief if it were flattened, the cortex is folded to save space and therefore has a distict wrinked appearance. The six layers of the cortex appear to have a distict connection pattern that is the same for all areas. These six major layers are: 1) 2) 3) 4) Molecular layer: horizontal Thalamus connections, axons and dendrites External granular layer: Pyramidal Interneurons & Feedback External pyramidal layer: vertical connections & neurons (3A + 3B) Internal granular layer: horizontal connections & stellar neurons (4A,4B,4C) 5) Internal pyramidal layer: Pyramidal Interneurons and timing (5A + 5B) 6) Fusiform layer: Column connections

The layers are made visible by different staining techniques as shown here. Not all parts of the brain have 6 layers; the Hippicampus for instance has 5 layers. The layers have no clear boundaries. During early infancy another layer of neurons exists in the white matter. That layer disappears in early childhood. Sensory input appears to be connected to the neurons in the dense middle layers, which are separated into seven sub-layers:

Layers in the Primary visual cortex (V1). Layer 4C connects to the LGN (Lateral Geniculate Nucleus). The LGN is a sorting centre for the eye. Simple cells respond to static images while complex cells respond to moving images (time shifted)

Neurons.
Neurons are electrically and chemically excitable cells with the function to receive and to process information. Each is a single, specialized nerve cell that has many properties. The intensity of input pulses, the interval of those pulses, the properties of the soma, the delays in the dendrites and the axon (output) are all relevant factors in the information processing of the brain. A number of different types of neurons exist which are classified by their shape or their location in the central nervous system, such as Bipolar, Basket, Betz, Medium Spiny, Purkinje, Pyramidal and Renshaw neurons. Though each type is

specialized in some way, the basic structure is the same and each of these neurons has a soma, a nucleus, one or more dendrites covered in synapses and a axon. Synapses may also be attached to the soma and even the axon. The cell body of a neuron is called the soma. It contains the nucleus, mitochondria and other structures necessary for the cell to survive. Extending from the soma are the dendrites and the axon. If the neuron is a sensory neuron, then the dendrites are specialized for receiving sensory information (light, sound). If the neuron is a motor neuron or an inter-neuron, then the dendrites are specialized for receiving information from other neurons through synapses and the axon connects to muscle tissue. Communication takes place through synapses, which generally connect to the dendrites, but also to the soma and the axon. The information passed through the synapses can be either electrical or chemical, depending on the type of the synapse. Most synapses are electro-chemical, whereby chemical neurotransmitters are pushed into the synaptic cleft when a current excites the junction.. The axon is responsible for sending signals, although some people believe that signals can also travel in the opposite direction. In some neurons the axon is covered by a myelin sheath, a layer of fatty substance that serves as an electrical insulation. Axons can be long, extending all the way down the spinal cord. Nerves are large bundles of axons, connecting for instance our little toe to our spinal cord. This means that these cells are more than a meter long. There are up to 10,000 synapses per neuron. Synapses can be Excitatory, Inhibiting or Modulatory. Excitatory neurons cause an excitatory action in connected cells. Inhibitory neurons inhibit the connected neurons. Inhibitory signals override excitory signals. Modulatory neurons modify the behaviour of groups of neurons by secreting the neurotransmitters Dopamine, Acetylcholine and Serotonin. Such chemically transmitted signals need to be terminated in one of two ways; the chemical neurotransmitter is either taken back up into the terminal button or the neurotransmitter is broken down by a chemical that exists for that purpose. For example, acetylcholine is rapidly broken down by acetylcholinesterase. Of course, the acetylcholine first stimulates the receptor sites. During the re-uptake period the neuron is unresponsive to input signals.

An image of a complete neuron (source Wikipedia.com) A function of the neuron nucleus is the generation of a precision-timed pulse train, whereby the period and duration is a function of the integrated temporal input pattern and the delay of the soma and axon. A delay value t is stored in the synapses and determines the period and intensity of the pulse train on which that synapse triggers the neuron. The delay in the axon determines the period of the output pulse train. A slow variable determines the period of oscillation. A better temporal-spatial match generally results in a longer period of oscillation. .

When a single neuron is isolated in vitro and depolarized by a current injection (Trigger) it will generate a rapid sequence of pulses, a so-called

spike train. This is a typical response from the nucleus in-vitro when stimulated by an electrical current. The output waveform is in this instance not related to an input pattern. The output waveform is typical for the neuron and the input pattern and is instrumental in selecting other neurons relative to the selected group (memory or action)

Waveforms in vitro: 1=presynaptic ll = postsynaptic. Observe how the post-synaptic voltage sharply increases and then slowly decreases. The rate of decrease is variable. Multiple pulses increase the post sysnaptic voltage incrementally.

In vivo, the pulse train shape and modulation depends directly on the input pattern, or its coincidence to a learned temporal pattern, whereby the number of pulses and interval are indicative of the match. The input is not a static weighted value, as it is in most current artificial (Hopfield-style) neural networks. The input exists out of a temporal sequence, whereby the period, duration and intesity of input pulses are all relevant information carriers. The neuron can be triggered by a single synapse that is pulsed intensely, or any number of synapses that are receiving less intense pulses. The neuron is most likely to be triggered by a fast sequence of pulses, as the synapse is discharged in the interval. The rate of depolorasation is variable, as is the delay in the axon. Inhibiting synapses take preference over excitatory synapses that are active at the same time. The output is a sequence of pulses, which vary in duration and period.

A simple cell in the primary visual cortex responds most intensely when the image matches the stored temporal pattern, for instance a horizontal line segment as in (A) above. The cell also responds if the vertical line segment has a slight offset (B), but does not respond at all to diagonal lines or when the line is outside of the sensor area (C and D). Complex cells respond when the stimulus is within the sensor area as shown below in (A) and (B). Complex cells are also movement sensitive, in that they respond when, for instance in (C), the line moves from right to left, but not when it moves from left to right (D). Other cells respond to (D) instead of (C).

1.

The Synapse.

Synapses can be electrical or chemical. All signalling in the cortex takes place through synapses. An exception to this rule are nerve cells that detect light (Visual sensory cells), smells and tastes (olfactory sense), tactile (touch and pain) and muscular feedback. There are approximately 1014 synapses in the human brain. Synapses are dynamic, in that their parameters are not constant. Learning takes place constantly as synapses are modified and new pathways are established or strenghtened. A single neuron may have as many as 10,000 synapses. Signalling across chemical synapses is slow, somewhere around 2 mS. Electrical synapses are ten times faster at 0.2 mS. Electrical synapses are

found only where neurons are in close proximilty and are part of the same data structure. Chemical synapses are the predominant communication method used in the brain.

Electrical synapse. The gap is small enough for ions to pass from one cell to the other. (Source: Wikipedia)

In the electrical synapse the gap is much smaller, about 3.5 nanometer vs. the 20 to 40 nanometer in the chemical synapse. The pore of the gap function is large enough for ions and small molecules to pass from one cell to the next.

A chemical synapse converts an electrical current into a chemical signal

In the chemical synapse the synaptic cleft is 20 to 40 nM wide. When an electric charge arrives, calcium ions trigger a chemical cascade causing synaptic vesticles to release their content into the synaptic cleft. Receptors on the opposite side of the synaptic cleft respond by opening ion channels in the post-synaptic cell membrane. The resulting change in voltage is called the post-synaptic potential. Whether a synapse is excitatory or inhibitory depends

on the type of ion channel that conducts the post-synaptic current, the type of receptors and the neurotransmitter present in the synapse. Following fusion of the synaptic vesicles and release of transmitter molecules into the synaptic cleft, the neurotransmitter is rapidly cleared from the space for recycling by specialized membrane proteins in the pre-synaptic or postsynaptic membrane. Re-uptake prevents blocking or desensization of the post-synaptic receptors and ensures that succeeding action potentials cause the same response. The necessity of re-uptake and the phenomenon of desensitization in ion channels means that the strength of a synapse may in effect diminish as a train of action potentials arrive in rapid succession--which gives rise to the so-called frequency dependence of synapses. The nervous system exploits this property for computational purposes, and can tune its synapses through chemical means. The size, number and replenishment rate of vesicles also are subject to change, and are elements of the computational and memory model, known as synaptic plasticity. Each neuron forms synaptic connections to many others. When action potentials fire simultaneously in several neurons that weakly connect to a single cell, they may initiate an impulse in that cell even though the synapses are weak. This process is known as summation. The timing of such signals is important, since we are not talking about static signals, as in ANNs. The different delays in the synapses will cause two or more signals to be summed that do not neccesary coincide in time which may result in a trigger event. More details follow in the signalling section.

Electrical Principle of Operation diagram.

Circuit description. Boxes A, B, and C represent three synapses, and box D forms the neuron Soma. Capacitor (Cin) across each input is charged by an excitory signal and discharged by an inhibitory signal, through resistor Rin_EX and Rin_INH respectively. Capacitor (Cin) is also discharged through resistor (Rdis). The values of these resistors and capacitors set up the characteristics of each synapse. The values are not fixed but are constantly updated. The resistors and capacitors in this representation stand for the number of vesticles and available Na+ or Ca+ ions. The summed output of all synapses is input to a gated oscillator. When the summed output of the synapses is sufficient to

trigger the oscillator a set of pulses is output. The duty cycle and frequency of this oscillator are set through (Rgain) and the values of Rcfreq, while capacitors represent the delays in the dendrite and axon. The OpAmp has a variable hysterisys. In the digital representation the charging and discharging of capacitors is simulated using up/down counters. Signalling Synapses are sensitive to specific pulse sequences, with specific timing and interval. A pulse with a 5mS period may not trigger a specific synapse, nor will a pulse with a 15 mS period. The following diagramme is borrowed from the Berger-Liaw dynamic synapse description:

If waveform (a) were applied to the input of the principle of operation circuit described above, a charge would be generated across the capacitor Cin. When the pulses are sufficiently close together this charge would accumulate until a

trigger level is reached in (e). However, this circuit does not limit the upper frequencies which will cause Ca+/ Na+ deficiency in real synapses and thereby limit the response of the synapse. The Berger-Liaw dynamic synapse was implemented using Digital Signal Processors and does not allow for selforganisation nor large-scale integration. In their research they obtained significant results in speech recognition in very noisy environments using just three DSP-based neurons, which communicate through dynamic synaptic junctions. The vWISP Temporal Digital artificial Neuron. The vWISP digital artificial neuron (DAN) is the result of seven years of research and experimentation. The device is a unified building block for the vWISP neural processor, in which a matrix of artificial neurons is used and controlled by a small microcontroller running software comprising a modified Genetic Algorithm. This microcontroller could be replaced with fuzzy logic that performs the same function. At this stage a microcontroller was preferred to allow for easy modification of the algorithm during the research period. The vWISP neuron is implemented in logic gates. This was one of the design criteria to allow large scale integration. Its function is close to the function of a biological neuron, except that the voltage levels are at binary levels while biological neurons operate between -70 to +40 mV. Input signals will need to be conditioned to be within the correct logic levels. The vWISP DAN exists out of two distinctive parts: a large number of identical synapse circuits and a neuron body (soma). These are connected through a summing mechanism analogues to a biological neuron dentrite. In the matrix, six layers of neurons form a column. A column is triggered by sensory inputs, or by other columns or both. Neurons are also organised in groups, but groups have weaker connections than columns. Large groups are organised in modules. Each column has 6 outputs and 6 inputs which are also monitored by the microcontroller. The microcontroller collects historial trigger information from the synapse and executes a genetic algorithm to eliminate synapses that are never or seldom triggered. Eliminated synapses are kept in a synapse pool, and recycled. DAN: The vWISP artificial Synapse circuit The synapse circuit block diagram is shown on the next page. Circuit description: A sequence of pulses from an external source or a previous neuron is input to the 'LOAD' input of a binary down counter. This counter also receives a clock

input. The clock input is nominal larger than 2n the load pulse frequency, whereby n is the width of the counter in number of bits. On each cycle of the clock the down-counter decrements the load value. The down counter simulates discharging of the Post Synaptic Potential in the biological synapse. An adder circuit adds the current output of the down-counter to the current value of the Na+ depletion counter on each occurance of a LOAD signal. The Na+ depletion counter is reset by the same signal, but one clock period later. The summed output of the adder circuit is thus loaded into the down counter on the edge of an input pulse and subsequently decremented on each clock cycle. Thus the circuit forms a leaky integrator, integrating the input pulse train (a..b..c..d..e) to a constantly declining set of values. The sawtooth shape shown (a..b..c..d..e) above the down counter in the following block diagram is not a waveform, but represents the decrementing output value of the down-counter expressed as a number of bits, depending on the width of the counter and the adder circuit. The timing is the same as the input pulse train. The depletion counter counts up to the value stored in the Na+ register. The depletion counter is cleared 1 cycle after each input LOAD pulse. The output of the depletion counter is maximum at Na+/clk cycles, and remains there until the next input pulse. At this time the value is added to the down counter value and stored in the down counter and the depletion counter is reset. The output of the down counter is also input to a second parallel adder, where it is summed with the output of other synapses within the same neuron circuit. An output trigger event can thus result from a single synapse or from multiple synapses. The Na+ register sets the ideal pulse or burst frequency on which the synapse triggers. In the circuit simulation outputs in figures 6a to 6d the output burst is a function of Na+ availability in the synapses, and the neuron soma Pulse Count register. Soma Circuit: The soma receives a common clock signal. The soma Pulse Count register effectively divides the clock signal by a stored value, and this becomes the soma interval frequency. The soma outputs the interval frequency as sharp, one cycle pulses when a trigger event occures, for the duration of the trigger event, thus simulating a relaxation oscillator.

IMAGES:

Synapse Waveforms The vWISP DAN performs like a biological synapse and soma. In the figure above the charging of the Na+ reservoir can be observed (Red), and the Post

Synaptic Potential (Magenta). The Na+ reservoir value is reset on each input pulse and may never reach the maximum count stored in the Na+ register. The (magenta) trace is the PSP, decreasing between input pulses. The yellow diamond indicates the start of a trigger event which is input to the neuron soma. Also, PSP values are output to an adder circuit, hence many synapse outputs can be added and the sum may trigger the soma at an earlier time. The output of the soma is not shown in this timing chart. These are the actual waveforms observed in the artificial synapse.

Dynamic synapse block diagram.

In a typical artificial neuron a large number of these synapse circuits are used, one for each input. The same synaptic circuit is used for both excitory and inhibitory inputs. The input pulse train waveform defines the timing of (a..b..c..d..e) further in the circuit. Please note that all signals are binary, and that the sawtooth in these diagrammes represent the combined value of a number of binary digits. The width of the counter circuits is not relevant to the opration method of the circuit. The depletion counter is cleared by every input pulse one cycle after the value has been added to the value in the 'downcounter'. The 'leaky integrator' effect is simulated by linear decrementing the down-counter.

Figure 1b shows the same synapse circuit, but a higher frequency input signal is applied. The result is that the trigger output does not change and no trigger event is generated. The output of the 'Down-counter' never reaches the trigger level because the value in the depletion counter is smaller than the decline in the 'leaky integrator'. The resulting ripple value may still contribute to a trigger event when all values are added together

The synapse has also no trigger output when the pulse frequency is lower than the parameter set up in the registers. The integrated value output value of the down-counter declines before the trigger value is reached. The Na+ value reaches the maximum value and levels out, but adding this value does not put the counter output above the trigger level. However, the value may contribute to a trigger event when output of all other synapses is summed. SOMA The soma simulates a relaxation oscillator that generates a sequence of precision timed pulses that are one period wide. The pulse-count register contains the 'charge time' of the relaxation oscillator. When the counter reaches 0 a single pulse is output and the process repeats.

Soma Circuit. The soma circuit generates a sequence of pulses. The frequency depends on the value in the Pulse Count register. If the pulse count register is 10 then 10 is loaded into the down counter, and the output is high for one clock cycle every 10 clock cycles.

Many synapses can be connected to a single soma. The output value of each synapse is added to all others using parallel adders. The number of bits in the counters and adders represent the synapse resolution: 6 bits means that there are 26 or 64 steps in the cycle. In most cases 8 or more bits would be desirable. 6 Somas are connected in a column. Each column feeds back to its input synapses. Columns are organised in groups. At least 1000 columns need to be integrated in a FPGA. The registers are connected to a small CPU core which executes a Hebbian modified genetic algorithm from internal storage. The connection pattern is generated by the CPU and optimized with a genetic algorithm. In the figure below the actual output of the soma, in combination with the input to a single synapse. The synapse was manually set up to respond strongly to the input pulse, and with a short output pulse interval (burst).

In biological systems up to 10,000 synapses are connected to each neural soma. In the vWISP processor there are many synapses conncted to many somas connected in a matrix. The organisation of the matrix is not determined and is not a constant. In the first prototype an organiation of six somas per column and 16 synapses per soma was used. At each level the output was fed back to one of the inputs. The next prototype is planned for end 2007 and will contain at least 5000 soma circuits and 80,000 synapses on a single chip, with fast serial I/O to enable larger matrices existing out of multiple chips.

Another possible avenue that has presented itself is the automatic replication of vWISP DAN synapse and soma circuit in virgin FPGA devices, This would enable the neural circuit to 'grow' new circuitry as it learns new tasks and when it is required. At present, small prototypes have exhibited behaviours found in biological brains, such as spontaneous organization. The expectation is that larger matrices will display more complex neural patterns. Applications for this technology include audio and video pattern recognition (voice and face recognition), security, control applications and robotics.