The Extensive Connectivity Neural Network Architecture

Marius Myburg www.ComplexComputation.com MariusMyburg@gmail.com

Abstract
Artificial neural networks have been shown to be effective at various computational tasks, such as classification, feature extraction, and recognition. In this paper I present a new neural network architecture/design (which I call extensive connectivity or EXCON for short) that deviates from conventional neural networks in various ways. I discuss the motivation behind EXCON and then I continue to show that EXCON is superior to most conventional neural networks in some important ways.

1. Introduction
In some problem spaces, such as the problem of everyday living and everything that goes hand-in-hand with that, such as locomotion and navigation, object recognition, memory and even consciousness, biological nervous systems are the most impressive computational devices known. Not only do they somehow perform these amazing feats, but they also adapt what we would call learning in animals. So it is no surprise that the construction of computationally useful artificial neural networks has become a science in itself. Conventional artificial neural networks can be used to find solutions to many different types of problem, the most common probably being classification of input into one or more output categories. And importantly, artificial neural networks can also adapt; ANNs can be taught. This is because ANNs are crude abstractions of biological neural networks and some of the adaptability of biological neural networks has been captured in artificial neural networks. But artificial neural networks are only very crude abstractions of biological neural networks, not only in size, but importantly also in dynamics of the underlying elements. In fact, most conventional neural networks abstract a neuron into a simple transfer-function element, while in biological systems neurons are much more dynamic systems. EXCON, the neural network design outlined here, builds upon conventional neural network structures in several important ways, that gives an EXCON neural network access to computational variables and opportunities that is not available to conventional neural networks.

2. Conventional vs. EXCON Nodes

In spite of their many successes and uses, artificial neural networks are sometimes regarded as failing. This is because they are, understandably, compared to biological neural networks, and the fact that they fall short is undeniable. According to Wikipedia: A.K. Dewdney, a former Scientific American columnist, wrote in 1997: Although neural nets do solve a few toy problems, their powers of computation are so limited that I am surprised anyone takes them seriously as a general problem-solving tool [1]. Most conventional neural networks are abstracted away, that is simplified, from their biological counterparts to a high enough degree that they lose many important capabilities. In this section I will briefly discuss the design of the typical conventional neural network and specifically, the nodes and adaptation process, and then I will go on to compare this to EXCON, and to show the benefits of an EXCON implementation.

2.1. Conventional Neural Networks

In most implementations, the conventional node (abstraction of a neuron) can be depicted as follows:

Figure 1. The perceptron (conventional node). The conventional node is a structure that has multiple (n) inputs, and a single output. To produce the output, the node simply sums its inputs and then passes the result through a so-called transfer function, which clips certain values (and outputs 0), and accepts other values (and outputs 1). Surprisingly, thats it; most conventional neural networks consists of large numbers of these elements, connected with links with weights, which is simply a mechanism to multiply the output by some value (usually between 0 and 1 inclusive) and pass the resulting value to an input of another node. So a typical neural network would be depicted something like this:

Figure 2. A simple network. Training a conventional neural network consists of presenting an input together with a corresponding correct output, and adjusting the weights of the network so that the input will produce the desired output. In other words, there is a data presentation cycle followed by a separate training cycle. Conventional neural networks implements various learning algorithms; these will not be discussed as conventional neural network learning algorithms are not the topic of this paper; for a review of the most common learning algorithms see [2].

2.2. Extensive Connectivity Neural Networks

Like conventional neural networks, EXCON networks consist of the two basic structures; node and link. However, the EXCON node functions very differently from the conventional node. The EXCON link performs the same function as the conventional link, except that it is somewhat simpler; it does not incorporate a multiplication factor. This hints at another network-level difference between EXCON and conventional networks; learning is not about weight adaptation, but about pathway formation more on this later.

First off, an EXCON node takes n binary inputs, as opposed to the weight-dependent analog inputs of conventional neural networks. It is envisioned that the EXCON model can be extended to make use of weights and thus analog inputs, but this is not the topic of this paper. Another difference on the level of the node, but with implications for the functioning of the whole network, is the fact that an EXCON node makes use of events. An EXCON network is not dependent on a separate cycle to compute the on-off states of each of its nodes. Instead, as soon as a node is activated, it triggers an event in all of its first-level downstream connected nodes. This is the OnParentSignallingStateChanged event, and it accomplishes just what the name suggests it informs the node that a parent nodes signaling state has changed. The node, in turn, now re-evaluates its own signaling state to make sure that it correctly reflects the effect of its inputs, which might have changed since one of those parentss signaling states have changed. Such an event-based network functions differently as a whole than a conventional, cycle-based network. Every signaling event causes a re-evaluation of child nodes, instead of the network being static and idle until the user or the software enters a signaling state reevaluation cycle for all nodes.

3. Extensive Connectivity in More Detail

Now that we have reviewed the basic structure and dynamics of conventional neural networks as well as some of the key differences between conventional networks and EXCON, it is time to delve a little deeper into EXCON. Nodes in EXCON are mathematically simpler than conventional nodes. Whereas a conventional node does a summation of its inputs which then gets passed through a transfer function, an EXCON nodes only job is to ask two simple questions: am I inhibited? If not, am I excited? Differently stated, a node is off when inhibited, and on when not inhibited and excited. As a single network, conventional neural networks are more about the weights, and thus the links, than about the nodes. The links are modified to cause certain nodes to be either signaling or not. In EXCON, the links play an equally important part, but in a different sense. In EXCON, everything is about pathways, and learning is all about pathway formation. A good example is to compare a conventional logical XOR network with an EXCON XOR network and note the differences. A conventional neural network can implement XOR using 6 nodes, 2 for input, one for output, and three hidden. Additionally, 7 links are required. By setting the weights properly, it is easy to construct a network that will give the XOR output:

Figure 3. XOR in a Conventional Neural Network An EXCON network can implement XOR using 5 nodes, 2 for input, one for output, and two hidden. 6 links are required.

It can be seen that, by making use of a node with threshold 2 in the hidden layer, the conventional neural network implements an AND and this AND inhibits the output. In contrast, the EXCON network implements pathway inactivation if an input node is on, then the pathway that consists of the other input node, its hidden node, to the output node, gets disabled. The input nodes own pathway is not disabled, unless the OTHER input node is on. In that case, both pathways are disabled, and the output node is not activated at all. It can be shown that all logical operations (AND, OR, XOR, and NOT) can be implemented in EXCON. Arguably, the EXCON implementation of XOR is simpler (there are less elements), more efficient (since it is eventbased and nodes evaluate their own activation state automatically), and more elegant.

In an EXCON network things look somewhat different. A reliance on weights and thresholds to implement XOR are not made, since EXCON does not make use of weights and thresholds; instead, the connectivity between nodes plays a vital role. As will be discussed shortly, EXCON implements different effective pathways which get opened given different XOR inputs (pathway inputs are referred to as contexts in EXCON). Figure 4 below depicts a XOR network, but this time implemented using the EXCON architecture:

Figure 4. XOR in an EXCON Neural Network

Lets have a look at some of the basic principles that underlies EXCON networks.

3.1. Extensive Connectivity Principles

The concept of the signaling pathway forms the cornerstone of EXCON. EXCON networks consist of many effective pathways living side-by-side in a single network.

Figure 5. The unit pathway. The unit pathway is the simplest and least flexible pathway possible; it consists simply of two nodes connected by a single link in one direction (in this case, A to B). Even though this is the unit pathway, because of its limitations

there is another pathway, the functional unit pathway, that extends the unit pathway somewhat to accomplish the functional unit in an EXCON network.

Figure 6. The Functional Unit Pathway The FUP consists of three nodes, A, B and C, A connected to B, and B connected to C. This is the functional unit pathway because it is the simplest pathway that can be used to construct multiple functional pathways that exist side-by-side. The B node which sits between A and C can be inhibited by a fourth node, blocking the A-C pathway. The reader is referred back to the XOR implementation; note that XOR actually consists of two FUPs sharing the C node.

Figure 7. One half of the XOR network, depicting the simple construction of the A-B-C FUP with the inhibitory link from D. As mentioned earlier, weights play no part in EXCON, and inhibition takes precedence over excitation, so node B in figure 7 will be inhibited when node D is active, even when node A is active concurrently. So the inhibitory DB link closes the contra lateral A-B-C FUP. The result: Exclusive OR (in die diagram, one half of it).

3.2. Sequence, Time and Oscillation

The concept of the Functional Unit Pathway is exciting; imagining a network with hundreds or thousands of interconnected FUPs, it is easy to see that in the 3 dimensional mesh there exists thousands upon thousands of FUPs (remembering that ANY three nodes connected to eachother forms a FUP), there are hundreds of thousands (even millions) of possible pathways that connect any two points. To demonstrate:

Figure 8. 5 Pathways connecting A to D.

Figure 8 shows that in an exceedingly small EXCON network, there are 5 pathways from A to D. If you add just one more node, in a fully connected network the number of pathways increase to 9.

Figure 9. 4 more pathways for 1 more node. Since EXCON makes use of pathways, as you add small numbers of nodes and links you add many alternate pathways between any two nodes, and this means your EXCON network becomes more powerful. But there is more, and in the domain of sequence, time and oscillation, EXCON becomes even more exciting. I mentioned earlier that EXCON nodes implement an event, OnParentSignallingStateChanged. This event-based model enables the network to make use of sequence, which is now an implicit variable in the network; one event triggers another, which may trigger another, which may trigger another, etc in sequence. This is actually only half the story. In biological systems, not only are sequence important, but actual delays are important also. Because of physiological factors, it takes time for a neuron to conduct an action potential; or for a synapse to release its neurotransmitter. EXCON implements another event which is called right before the parents signaling state changes, OnParentSignallingStateChanging. A node has a property, SignallingStateChangeDelay, that determines the delay that will occur between StateChanging and StateChanged. This simulates the duration that elapses before the signaling state change of the node in question actually happens. That is sequence and time; there is another dynamic that an EXCON network can implement that is important and that is oscillation. A single node can not oscillate, but it is possible to construct an oscillator with two nodes. However I will demonstrate a simple three-node oscillator in this text in order to make explaining a little simpler.

An oscillator is a node that fires multiple signals, each signal being separated from the preceding signal by a certain amount of time, which determines the rate of oscillation. In biological systems oscillators are important for rhythmic behaviour such as controlling heartbeat, breathing, and even locomotion. An oscillator is referred to as a Central Pattern Generator or CPG for short.

Figure 10. A simple EXCON CPG Figure 10 illustrates a simple three-node EXCON CPG. Node A, which is always on, excited node B. Node B in turn excites node C, and node C inhibits node B. As soon as node B is inhibited, node C is not excited any more, so the inhibition from node B ceases. This pattern repeats, resulting in node B (and C) oscillating, the rate (and other aspects) of oscillation being determined by the signaling state change delay of nodes B and C.

4. Conclusions
In this paper I presented a very brief introduction to some of the features of the Extensive Connectivity neural network architecture. There are more to Extensive Connectivity that I have not dealt with here, but this paper should make the reader somewhat familiar with what this neural network architecture is all about. Things to be done in future papers include: Demonstrating implementations of all of the logical gates in EXCON. Speaking in more detail about what context means and Demonstrate conventional networks implemented in EXCON and Demonstrate the unique power of an EXCON network in which optimal use is made of context and dynamic FUP switching and Node-local, synapse-induced learning.

5. References
1 2
http://en.wikipedia.org/wiki/Neural_network Neural Networks: A Systematic Introduction, http://page.mi.fu-berlin.de/rojas/neural/index.html

Please feel welcome to send comments to the author at email address MariusMyburg@gmail.com.