Digital Hardware Design

Report for Reading assignment and final project

Srikrishna Acharya B

Acharya 10/24/2012

Introduction: In order to solve any problem we need to follow a systematic procedure, but this is possible if we know the procedure and hinders we would face in that process. What if the procedure is not known and the problems we face are abstract. This is what in general we face in real times and the answer for this problem is Neural Networks which resembles the human neuron. The network is composed of a large number of highly interconnected processing elements (neurons) working in parallel to solve a specific problem. Neural networks learn by example. They cannot be programmed to perform a specific task. The examples must be selected carefully otherwise useful time is wasted or even worse the network might be functioning incorrectly [1]. Implementation of Neural networks can be done using analog or digital systems. The major advantages of digital implementation are higher accuracy, better repeatability, lower noise sensitivity, better testability, higher flexibility, and compatibility with other types of preprocessors. The digital NN hardware implementations are further classified as (i) (ii) (iii) FPGA-based implementations DSP-based implementations ASIC-based implementations.

The most important aspects of NN is Parallelism, Modularity and Dynamic adaption, Now out of these three implementations DSP based implementation is sequential and hence does not preserve the parallel architecture of the neurons in a layer. ASIC implementations do not offer re-configurability by the user. FPGA is a suitable hardware for neural network implementation as it preserves the parallel architecture of the neurons in a layer and offers flexibility in reconfiguration[2]. Field programmable gate arrays are high-density digital integrated circuits that can be configured by the user; they combine the flexibility of gate arrays with desktop programmability. Their architecture consists mainly of: Configurable Logic Blocks (CLB's) where Boolean functions can be realized, Input/output Blocks (IOB's) serve as input/output ports, and programmable interconnection between the CLB's and IOB's[3]. Architectures of NNs Before going into implementation aspects of NN in FPGA lets see different architectures of NN. They way you interconnect neurons plays an extremely important role in implementation aspects. In fact, researchers try to find new architectures that can be used for many purposes.

Basically NN architecture is determined from the perspective of the training i.e. learning to be more precise.
i) ii)

Supervised learning Unsupervised learning

Supervised learning we train the NN with proper inputs and expected outputs and change the weights of the interconnections accordingly. In Supervised learning basic architecture is the dictomizer. It consists of a single neuron, and can classify between two different sets.

The improved version of the dictomizer is Multilayer feedforward networks. Here, several neurons are arranged in layers. The output of one layer is the input for the next one.

Another popular architecture in same taxonomy is Hopfield networks. This network is an example of a single layer-feedback network. That is, the output of a neuron is recycled as the input for other neurons.

All images source: [7]

The other class of NN architecture is Unsupervised learning, where the NN trains itself from the situation it faces and forms a map by itself. The network then adapts accordingly. In this network, no expected output is required. The network learns on itself [7]. Based on the hardware requirement and the implementation constraints we generally employ feed forward multilayered perceptron in FPGA. FPGA realization of ANNs with a large number of neurons is still a challenging task because ANN algorithms are multiplication-rich and it is relatively expensive to implement [2]. Implementation of NN in FPGA The artificial neural network implemented is a three layer perceptron. Basically a perceptron is simple unit of the dictomizer.

Image source: [2]

Where X=PiWi +B and the output y is the activation function(f(x)) of the weighted sum of the inputs. This activation function can be Linear, Log-sigmoid and Tan-sigmoid excitation functions. The network classifies selected input patterns. It consists of three layers: an input layer with 5 nodes, a hidden layer with 4 nodes and an output layer with 2 nodes. The input to the network is a continuous valued vector x1,x2,x3,x4,x5 . The output of the network is the class of the current input. Output of node yi is calculated by y=f(x) forms a complex interconnect network.

The implementation of the complex network is illustrated by the following block diagram.

Image source: [3] Architecture of Neural network.

These nodes are fully interconnected to each other between adjacent layers. We have trained the network off-line by simulating the network on a PC and obtain the final values of weights at the end of training session. The input layer does no processing but simply buffers the data. The nodes in the other layers form a weighted sum of their inputs. The 8-bit input to the hidden and output nodes y, is multiplied by 8-bit signed weights Wij to form a 16-bit signed product. The products (five in the hidden and four in the output layer) and a 16-bit signed bias value are accumulated into a 20-bit sum. We have scaled down the 20-bit sum to a 10-bit value. We have selected the 10-bit scaling after running a simulated software network on a PC using the same input data of the hardware network. The 10 bit are the minimum number of bits that can be retained without deteriorating the accuracy of the sum. This 10-bit scaled sum serves as the address of a 1 K x 8 EPROM where a sigmoid activation function f is realized as a lookup table[3]. Overall for one node we perform a multiplication and fast addition with accumulator. For that we need one 2s complement multiplier, carry lookhead adder and accumulator. And an 1K 8 EPROM. In the following block diagram implementation of a single hidden layer is illustrated.

Image source:[3]

The detailed implementation differs from application to application for example in [2] NN is used to implement Space Vector Modulation for a Voltage source invertor. In that application it requires single input node and 3 output nodes, so they have employed 1 6 6 6 3 structure. Similarly in [6] the application is image processing and it requires 3 input nodes and one output node, so used 3 3 1 structure. It all depends on the application and the level of computations we need in the process. In [5] the entire NN is implemented in a DSP and a PID controller, control unit and PWM generator is implemented in FPGA. The basic understanding of a NN and its implementation is similar but based on the application and the requirements the number of layers, nodes of in each layer and the activation function are chosen accordingly.

References: [1] http://www.doc.ic.ac.uk/~nd/surprise_96/journal/vol4/cs11/report.html [2] Neural Network Implementation Using FPGA:Issues and Application - A. Muthuramalingam, S. Himavathi, E. Srinivasan [3] Hardware Implementation of an Artificial Neural Network Using Field Programmable Gate Arrays (FPGA's) - Naleih M. Botros and M. Abdul-Arir [4] A NEURAL NETWORK FPGA IMPLEMENTATION -S. CoriC*, I. LatinoviC**, A. PavasoviC** [5] Hardware Implementation of a Real Time Neural Network Controller with a DSP and an FPGA - Sung Su Kim* and Sed Jug** [6] FPGA Implementation of a Neural Network for a Real-Time Hand Tracking System - Marco Krips, Thomas Lammert, and Anton Kummert [7] http://www.aishack.in/2009/12/the-different-network-architectures/