Nature-Inspired Optimization Algorithms

2020

Chapter 1: Introduction to Algorithms

Abstract

Many problems in science and engineering can be formulated as optimization problems, subject to complex, nonlinear constraints. The solutions of optimization problems often require sophisticated optimization techniques. Traditional algorithms may struggle to deal with such highly nonlinear problems. Nature-inspired algorithms can be good alternatives, and they are flexible and efficient for solving problems in optimization, data mining and machine learning. This chapter introduces the fundamentals of algorithms, classification of optimization problems and algorithms as well as a brief history of metaheuristics.

Keywords

Algorithm; bat algorithm; cuckoo search; differential evolution; firefly algorithm; flower pollination algorithm; genetic algorithm; gradient-based method; nature-inspired computation; optimization; optimality; particle swarm optimization; swarm intelligence

Optimization is paramount in many applications such as engineering, business activities, and industrial designs. Obviously, the aims of optimization can be anything—to minimize the energy consumption, costs, and environmental impact, to maximize the profit, output, performance, sustainability, and efficiency. It is no exaggeration to say that optimization is everywhere, from engineering design to business planning and from Internet routing to holiday planning. Because resources, time, and money are always limited in real-world applications, we have to find solutions to optimally use these valuable resources under various constraints, ideally in a sustainable way. Mathematical optimization or programming is the study of such planning and design problems using mathematical tools. Since most real-world applications are often highly nonlinear, they require sophisticated optimization tools to tackle. Nowadays, computer simulations become an indispensable tool for solving such optimization problems with various efficient search algorithms.

Behind any computer simulation and computational methods, there are always some algorithms at work. The basic components and the ways of they interact determine how an algorithm works and the efficiency and performance of the algorithm.

This chapter introduces algorithms and analyze the essence of the algorithms. Then we discuss the general formulation of an optimization problem and describe the modern approaches in terms of swarm intelligence and bio-inspired computation. A brief history of nature-inspired algorithms is reviewed.

1.1 What Is an Algorithm?

In essence, an algorithm is a step-by-step procedure of providing calculations or instructions. Many algorithms are iterative. The actual steps and procedures depend on the algorithm used and the context of interest. However, in this book, we mainly concern ourselves with the algorithms for optimization, and thus we place more emphasis on iterative procedure for constructing algorithms.

or x, can be written as

(1.1)

. Here, t is the iteration counter or index, also called pseudo-time or generation counter.

in the following form:

(1.2)

, we have

(1.3)

(1.4)

(1.5)

, which shows that this iteration method is very efficient.

due to the fact that

(1.6)

.

. This method is similar to the well-known bisection method.

It is worth pointing out that the final result, though converged beautifully here, may depend on the starting (initial) guess. This is a very common feature and disadvantage of deterministic procedures or algorithms. We will come back to this point many times in different contexts in this book.

was converted to Eq. (1.1)? Why not write the iterative formula as simply the following:

(1.7)

, we have

(1.8)

and k. This clearly demonstrates that the way to design a good iterative formula is very important.

From a mathematical point of view, an algorithm A at iteration or time t. That is,

(1.9)

where A . In fact, A can be a set of mathematical equations in general. In some literature, especially those in numerical analysis, n does not mean x . Such notations will become useful and no confusion will occur when used appropriately. We use such notations when appropriate in this book.

1.2 Newton's Method

Newton's method . It was formulated by Newton in 1669, and later Raphson applied this idea to polynomials in 1690. This method is also referred to as the Newton-Raphson method.

,

(1.10)

which leads to

(1.11)

or

(1.12)

. Thus we have the standard Newton iterative formula

(1.13)

and continues until a certain criterion is met.

can be used as the starting point. But if the initial value is too far away from the true zero, the iteration process may fail. So it is a good idea to limit the number of iterations.

Newton's method is very efficient and is thus widely used. For nonlinear equations, there are often multiple roots, and the choice of initial guess may affect the root into which the iterative procedure could converge. For some initial guess, the iteration simply does not work. This is better demonstrated by an example.

We know that the following nonlinear equation

(1.14)

. Let us now try to solve it using Newton's method. First, we rewrite it as

(1.15)

, and

(1.16)

(1.17)

(1.18)

is very close to the true solution e. The convergence is very slow.

and the iterative formula

(1.19)

we get

, though the expression may become singular if we continue the iterations.

since it leads to a negative value in the second step and thus the formula breaks down afterwards. This highlights the importance of choosing the right initial starting point.

in a d-dimensional space. That is,

(1.20)

is a vector of d variables and the superscript T means the transpose to convert a row vector into a column vector. This current notation makes it easier to extend from univariate functions to multivariate functions since the form is identical and the only difference is to convert a scalar x into a vector x (in bold font now). It is worth pointing out that in some textbooks x can be interpreted as a vector form, too. However, to avoid any possible confusion, we will use x in bold font as our vector notation.

. In general, this Newton-Raphson method has a quadratic convergence rate. Sometimes the true convergence rate may not be as quick as it should be, it may have nonquadratic convergence property (Yang and He, 2019).

A way to improve the convergence in this case is to modify the above formula slightly by introducing a parameter p so that

(1.21)

, we can take p as

(1.22)

The above iterative equation can be written as

(1.23)

It is worth pointing out that the optimal convergence of Newton-Raphson's method leads to an optimal parameter setting pto be optimized.

1.3 Formulation of Optimization Problems

Let us first discuss the general formulations of optimization problems, then we can look at different ways of classifying various optimization problems.

1.3.1 Optimization Formulation

Mathematically speaking, it is possible to write most optimization problems in the generic form

(1.24)

subject to

(1.25)

(1.26)

are functions of the design vector

(1.27)

of the D-dimensional vector x are called design or decision variables, and they can be real continuous, discrete or the mixed of these two.

are called the objective functions or simply cost functions, there is only a single objective. The space spanned by the decision variables is called the design space or search space , whereas the space formed by the objective function values is called the solution space or response spaceare called constraints. It is worth pointing out that we can also write the inequalities in the other way, ≥0, and we can also formulate the objectives as a maximization problem.

In a rare but extreme case where there is no objective at all, there are only constraints. Such a problem is called a feasibility problem because any feasible solution is an optimal solution.

. Multi-objective optimization is also referred to as multicriteria or even multi-attribute optimization in the literature. In real-world problems, most optimization tasks are multi-objective. Though the algorithms we will discuss in this book are equally applicable to multi-objective optimization with some modifications, we mainly place the emphasis on single-objective optimization problems.

, then it is called an unconstrained optimization , it is called an equality-constrained becomes an inequality-constrained problem.

. However, equality constraints have special properties and require special care. One drawback is that the volume of satisfying an equality is essentially zero in the search space, thus it is very difficult to get sampling points that satisfy the equality exactly. Some tolerance or allowance is used in practice.

are nonlinear, we have to deal with a nonlinear optimization problem.

1.3.2 Classification of Optimization Problems

The classification of optimization problems can also be done by looking at the modality of the objective landscape, which can be divided into multimodal problems and unimodal problems (including convex optimization). In addition, classification can also be about the determinacy of the problem. If there is not any randomness in the formulation, the problem is called deterministic and all these previous problems are essentially deterministic. However, if there is any uncertainty in the variables or function forms, then such optimization involves probability distributions and expectation. In this case, they are often called stochastic optimization or robust optimization. Classification of optimization problems and the terminologies used in the optimization literature can be diverse and potentially confusing. We summarize most of these terms in Fig. 1.1.

Figure 1.1 Classification of optimization problems.

Whether an optimization problem is considered easy or hard, it can depend on many factors and the actual mathematical formulation or problem functions. In fact, three factors that make a problem more challenging are: nonlinearity of the objective function, the high dimensionality of the problem, and the complex shape of the search domain.

•  Nonlinearity and multimodality. The high nonlinearity of a function to be optimized usually means multimodality with multiple local modes with multiple (even infinite) optima. In most cases, algorithms to solve such problems are more likely get trapped in local modes.

•  High dimensionality. High dimensionality is associated with the so-called curse of dimensionality where the distance becomes large, whereas the volume of any algorithm that can actually search becomes insignificant compared to the vast feasible space.

•  Constraints. Multiple constraints can make the feasible region complicated with irregular geometry or shapes. In some cases, feasible regions can be split into multiple disconnected regions with isolated islands, making it harder for algorithms to search all the feasible regions (thus potentially miss the true optimality).

Other factors such as the evaluation time of an objective are also important. In many applications, such as protein folding, bioinformatics, aerospace engineering, and deep machine learning, the evaluation of a single objective can take a long time (from a few hours to days or even weeks), therefore the computational costs can be very high. Thus, the choice of efficient algorithms becomes paramount.

1.3.3 Classification of Optimization Algorithms

Algorithms for solving optimization problems are typically iterative, and thus multiple evaluations of objectives are needed, usually hundreds or thousands (or even millions) of evaluations. Different algorithms may have different efficiency and different requirements. For example, Newton's method, which is gradient-based, is very efficient for solving problems with smooth objective functions, but it can get stuck in local modes if the objective is highly multimodal. If the objective is not smooth or has a kink, then the Nelder-Mead simplex method can be used because it is a gradient-free method and can work well even for problems with discontinuities, but it can become slow and get stuck in a local mode.

Algorithms for solving nonlinear optimization are diverse, including the trust-region method, interior-point method, and others, but they are mostly local search methods. In a special case when the objective becomes convex, they can be very efficient. Quadratic programming (QP) and sequential quadratic programming (SQP) use such convexity properties to their advantages.

The types of algorithms that can solve various optimization problems are summarized in Fig. 1.2, where we also include some machine learning algorithms, such as the artificial neural network (ANN), regression, and support vector machine (SVM).

Figure 1.2 Classification of optimization algorithms.

1.4 Optimization Algorithms

There are many different types of optimization algorithms. We now introduce some of these algorithms in this section.

1.4.1 Gradient-Based Algorithms

Newton's method introduced earlier is for single-variable functions. Now let us extend it to multivariate functions.

:

is the solution of the following linear equation:

(1.28)

This leads to

(1.29)

is the Hessian matrix, which is defined as

(1.30)

This matrix is symmetric due to the fact that

(1.31)

iteration can be written as

(1.32)

is quadratic, then the solution can be found exactly in a single step. However, this method is usually not efficient for nonquadratic functions.

and we have the modified Newton's method

(1.33)

, and we have the quasi-Newton method

(1.34)

which is essentially the steepest descent method.

, we have

(1.35)

is the increment vector. Since we are trying to find a better approximation to the objective function, it requires that the second term on the right hand is negative. So,

(1.36)

of two vectors u and v is the largest negative when they are parallel but in opposite directions, so as to make their dot product negative. Therefore, we