Академический Документы
Профессиональный Документы
Культура Документы
Summer 2019
Combinatorial optimization
1 Heuristic searches
3 Tabu algorithms
4 Up next
Outline
1 Heuristic searches
3 Tabu algorithms
4 Up next
More generally
A trade-off
Choice of neighborhood
2
This defines a so-called k-neighborhood, and any move is called a k-change.
Statistics and Econometrics (CAU Kiel) Summer 2019 10 / 41
Heuristic searches
The moves
You may come up with many ways of updating the current candidate; you
typically want an upward move:
Random starts
Worth emphasizing: The random starts strategy can be combined with any
optimization method!
400
yaxislabel
350
300
1 31 16 46 61
xaxislabel
Results of random-starts local search, 1-change steepest ascent, 15
iterations max per start. (AIC ranges between −300 and −420.)
Statistics and Econometrics (CAU Kiel) Summer 2019 13 / 41
Heuristic searches
Method 1 2 3 6 7 8 9 10 12 13 14 15 16 18 19 20 21 22 24 25 26 AIC
LS (2,4) ••• • • • • • • • • • −416.95
LS (1) •• • • • • • • • • −414.60
LS (3) •• • • • • • • • • • −414.16
LS (5) • •• • • • • • • • −413.52
S-Plus • •• • • • • • • • • • −416.94
Efroymson • •• • • • • • • −400.16
Random starts local search [LS] model selection results using 1-change
steepest ascent, as well as two standard procedures (greedy stepwise
ascent). (Bullets indicate inclusion of predictor j.)
We may want to
break out of a bad neighbourhood, or
move towards other (more promising) neighbourhoods.
Of course, while ensuring that you don’t slide back immediately...
Outline
1 Heuristic searches
3 Tabu algorithms
4 Up next
Annealing
Wikipedia says:
Annealing, in metallurgy and materials science, is a heat
treatment that alters the physical and sometimes chemical
properties of a material [...].
In annealing, atoms migrate in the crystal lattice [grid] and the
number of dislocations decreases, leading to a change in ductility
and hardness. As the material cools it recrystallizes.
An analogy
Simulated annealing
The algorithm
1 Pick (randomly) a candidate solution θ̃ from the current
neighborhood of θ (t) , according to a proposal distribution g (t) .
(Often the discrete uniform over N (θ (t) ).)
2 If proposal leads to an improvement, update θ (t+1) = θ̃
3 If proposal does not lead to an improvement,
randomly choose whether to update using θ̃ or not;
the current temperature controls the probability of updating!
So let θ (t+1) = θ̃ with probability equal to
( !)
f (θ (t) ) − f (θ̃)
min 1, exp .
τj
Some details
The neighbourhoods:
should be small and easily computed.
any two candidates must be connectable via a sequence of
neighborhoods.
−300
yaxislabel 1.0
otherlabel
−350
0.5
−400
0.0
0
1000 2000
xaxislabel
Discrete uniform proposal, 1-neighbourhoods, 15-stage cooling schedule
with stage lengths of 60, 120, and 220, α(τj−1 ) = 0.9τj−1 , τ0 is 1 (bottom
solid) or 10 (top solid). (Dashed: baseline temperature at each step.)
Statistics and Econometrics (CAU Kiel) Summer 2019 23 / 41
Simulated annealing and genetic algorithms
Genetic algorithms
In this example,
1 The genetic representation of θ is simply inclusion/exclusion of the
respective regressor (obvious codes 0/1).
2 The AIC may play the role of the fitness function.
3 The initial population consists of several candidate models, say picked
at random.
4 To renew the population, we look at the best performing models from
the current populations, and change them suitably to yield a new
generation (“breeding”).
5 We stop if no improvement has been observed for the last say 15
generations or a total number of generations has been reached (etc.).
400
300
yaxislabel
200
100
0 50 100
xaxislabel
Population size P = 20, 100 generations breeding with fitness-rank based
selection probability, simple crossover, and 1% mutation.
Statistics and Econometrics (CAU Kiel) Summer 2019 28 / 41
Tabu algorithms
Outline
1 Heuristic searches
3 Tabu algorithms
4 Up next
So, after a downhill step, let’s (temporarily) forbid (or make tabu/taboo)
moves that have the potential of taking us back. We need to
characterize moves to this end, and
keep an eye on the recent history (say H (t) ) of moves.
Move attributes
The tabu list is a list of attributes, not moves, so a single tabu attribute
may forbid entire classes of moves.
Examples of attributes
Generic Attribute Model Selection Example
(2-change neighbourhood)
(t)
A change in the value of θi . The at- A1 : Whether the ith predictor is added
tribute may be the value from which (or deleted) from the model.
the move began, or the value to which
it arrived.
(t) (t)
A swap in the values of θi and θj A2 : Whether the absent variable is ex-
when
(t)
θi 6=
(t)
θj . changed for the included variable.
History matters:
Current neighbourhoods may depend on search history; denote this
N (θ (t) , H (t) ).
The choice of θ (t+1) in N (θ (t) , H (t) ) may depend on the (recent)
search history too.
This effectively leads to using an augmented target function, fH (t) .
Workflow
√ √
Typical tabu validity in practice: 7 ≤ τ ≤ 20, or .5 p ≤ τ ≤ 2 p. If it’s
large enough it will prevent cycles.
Refinements: Aspiration
Most common:
permit a tabu move ...
if it provides a higher value of the objective function than has been
found in any iteration so far.
Aspiration by influence:
A very recent high increase in target may imply that algorithm parses
a new (promising) region in Ω
So cancel the tabu of not reversing the last downward move: further
exploration of new region is meaningful.
Refinements: Diversification
Refinements: Intensification
Construct fH (t) to extend search near good solutions with high residency.
Reward retention of attributes that correlate with high-quality
high-freqency solutions;
penalize removal of such attributes.
To sum up
Identify a list of problem-specific attributes, initialize tabu list. Then
1 Construct fH (t) (θ) based on f (θ), and perhaps
diversification/intensification penalties or incentives.
2 Identify neighbors of θ (t) , namely the members of N (θ (t) ).
3 Rank the neighbors in decreasing order of improvement, as evaluated
by fH (t) .
4 Select the highest ranking neighbor.
5 Is this neighbor currently on the tabu list? If not, go to step 8.
6 Does this neighbor pass an aspiration criterion? If so, go to step 8.
7 If all neighbors of θ (t) have been considered and none have been
adopted as θ (t+1) , then stop. Otherwise, select the next best neighbor
and go to step 5.
8 Adopt this solution as θ (t+1) and update the tabu list.
9 Has a stopping criterion been met? If so, stop. Otherwise, increment t
and go to step 1.
Statistics and Econometrics (CAU Kiel) Summer 2019 38 / 41
Tabu algorithms
400
yaxislabel
350
300
0 20 40 60
xaxislabel
Outline
1 Heuristic searches
3 Tabu algorithms
4 Up next
Coming up