Agenda

Hopfield Neural Network Gaussian Machine Quadratic Assignment Problems How to Solve Problem Computation Results Conclusion

Artificial Neural Network (ANN)

An mathematic model Inspired by the biological nervous systems Acquires knowledge through learning

ANNs knowledge is stored within inter-neuron connection strengths (Synaptic weights)

Hopfield Neural Network

Use continuous activation function Fully connected recurrent network Notion of energy

Each state has an energy Computes recursively until a stable state reached Converges to stable states

Hopfield Network Model:

N du ( t ) Dynamic i = -ui (t ) + Ti j xi (t ) + I i dt Equation: j =1

Output:

ui (t ) 1 xi (t ) = f (ui (t )) = 1 + tanh 2 a 0

Hopfield Neural Network

Activation Function
ui ( t ) 1 xi (t ) = f (ui (t )) = 1 + tanh 2 a 0

xi =f(ui)
1

a0 parameter determines behavior of the gain function Higher ~ gentle Lower ~ steep

ui

Hopfield Network: Energy Function

Quantification of current energy of the network Energy surface determines stable states Stable states are local minima Each update converges to stable state Symmetric connections Ti j = T ji , Tii = 0
N 1 N N E ( x) = - Ti j xi x j - I i xi 2 i =1 j =1 function: i =1

Energy

Lyapunov Condition:

N dE dxi = - 0 dt i =1 dt

Gaussian Machine: An Improvement of Hopfield NN

Gaussian Machine Objective: Allow the system to escape from local minima

Added Gaussian noises that its power vary in time Vary the activation function gain in time
Dynamic Equation:
N dui (t ) = -ui (t ) + Ti , j x j (t ) + I i + i (t ) dt j =1

Output:

1 xi (t ) = f (ui (t )) = 1 + tanh 2

( )

Gaussian Machine: An Improvement of Hopfield NN

Gaussian Machine

Added Gaussian noises that its power vary in time

Dynamic Equation:
N dui (t ) = -ui (t ) + Ti , j x j (t ) + I i + i (t ) dt j =1

Temperature

1.5

hi (t ) = N (0, s i (t ))

0.5

hi (t )

-0.5

-1

-1.5

0.5

1.5

2.5

Gaussian Machine: An Improvement of Hopfield NN

Gaussian Machine

Vary the activation function gain in time

Output: 1 xi (t ) = f (ui (t )) = 1 + tanh 2 ( )

xi =f(ui)
1

ui

Quadratic Assignment Problems-QAP

Quadratic Assignment Problems-QAP

Assign N Facilities to N locations

Minimum sum of product of flow between facilities and distance between locations

Cost = f AB d12 + f BC d 23 + f AC d13

N! Possible Solutions

Computationally hard problem, grows exponentially

N = 12, 479001600 solutions N = 20, 2432902008176640000 solutions

How to find a solution from this ocean ?

Quadratic Assignment Problems-QAP

Problem Representation:

D = [d kl ] ~ the distance between location k and l

and d kk = 0 , d kl = dlk F = [ fij ] ~ the traffic flow between facility i and j and fii = 0 , fij = f ji
1 if i is assigned to k xik = 0 otherwise

A solution: Permutation Matrix (NxN Matrix)

P = [ xik ]

Quadratic Assignment Problems-QAP

Objective: Assign N Facilities to N locations Minimize the total cost of assignment
1 N N N N min C = fij d kl xik x jl x 2 i =1 j =1 k =1 l =1

Quadratic function QAP

The constraints
N

x
i =1 N

ik

= 1 for i = 1,..., N

: Only one location k is assigned in each facility i

x
k =1

ik

= 1 for k = 1,..., N : Only one facility i is assigned in each location k

: Output level boundary

xik [0,1] "i, k

QAP Example

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Representation of QAP

Permutation matrix represents an assignment

Rows ~ Facilities Columns ~ Locations

P = [ xik ] =

1 if i is assigned to k xik = 0 otherwise

Cost = f DA d12 + f BC d 23 + fCB d34 + f BE d 45 + f ED d51

Representation of QAP

Use a neuron to represent each entry of the matrix

xik If the entry is 1, neuron is on ( 1) xik If the entry is 0, neuron is off ( 0)

N-facilities problem represented using N2 neurons

Hopfield Neural Network for QAP

Arrange the neuron in a matrix form

Neurons addressed with double indices

N N duik (t ) = -uik (t ) + Tik , jl x jl (t ) + I ik Equation: dt j =1 l =1

Dynamic

Output:

uik (t ) 1 xik (t ) = f (uik (t )) = 1 + tanh 2 a 0

N N 1 N N N N E ( x) = - Tik , jl xik x jl - I ik xik 2 i =1 j =1 k =1 l =1 i =1 k =1

Energy function:

Gaussian Machine for QAP

Same notation as Hopfield network

Dynamic Equation: Output:
N N duik (t ) = -uik (t ) + Tik , jl x jl (t ) + I ik + dt j =1 l =1

ik

(t )

uik (t ) 1 xik (t ) = f (uik (t )) = 1 + tanh 2 ( )

N N 1 N N N N E ( x) = - Tik , jl xik x jl - I ik xik 2 i =1 j =1 k =1 l =1 i =1 k =1

Energy function:

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Energy Function for QAP

Its minima must correspond to the valid solutions Shorter paths and flow must have lower energy So, break it down into

E ( x) = Epenalty + Ecost

Energy Function for QAP

Constraint Satisfaction:
Epenalty A A C N N = xik - 1 + xik - 1 + xik (1 - xik ) 2 i =1 k =1 2 k =1 i =1 2 k =1 i =1
Only one 1 in each row Only one 1 in each column Output level close to 1
N N 2 N N 2

Cost:
Ecost B N N = fij d kl xik x jl 2 i , j =1 k ,l =1

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Mapping QAP onto Neural Network

N 1 N N E ( x) = - Tik , jl xik x jl - I ik xik 2 i , j =1 k ,l =1 i , k =1

Quadratic term

Linear term

Quadratic terms for T values Linear terms for I values

Mapping QAP onto Neural Network

QAP Energy Function

A N N A N N C N N E ( x) = xik - 1 + xik - 1 + xik (1 - xik ) 2 i =1 k =1 2 k =1 i =1 2 k =1 i =1 B N N + fij d kl xik x jl 2 i , j =1 k ,l =1
2 2

Network Energy Function

N 1 N N E ( x) = - Tik , jl xik x jl - I ik xik 2 i , j =1 k ,l =1 i , k =1

Mapping QAP onto Neural Network

QAP Energy Function

A N A N C N B N N E ( x) = xik x jl + xik x jl xik x jl + fij d kl xik x jl 2 i , k ,l =1 2 k ,i , j =1 2 i , j ,k ,l =1 2 i , j =1 k ,l =1 C - A xik - A xik - xik 2 k =1 i =1
N N

Network Energy Function

N 1 N N E ( x) = - Tik , jl xik x jl - I ik xik 2 i , j =1 k ,l =1 i , k =1

Mapping QAP onto Neural Network

Linear term

Network Energy Function

E ( x) = T x x I x

Mapping QAP onto Neural Network

Linear term

Network Energy Function

E ( x) = T x x I x

Derived T and I matrices

Tik , jl = - Ad i , j - Ad k ,l + Cd i , jd k ,l - Bfij d kl I ik = 2 A + C 2
Kronecker Delta

1 , i = j d ik = 0 , i j

Neural Network as QAP solver

Find a representation for the problem

Define a problem energy function

Derive T and I matrixes from the energy function

Construct the network using T and I matrixes

Agenda

Hopfield Neural Network Gaussian Machine Quadratic Assignment Problems How to Solve Problem Computation Results Conclusion

Computation Results

Conclusion