The LQR algorithm is a classic algorithm in optimal control, and LQR is a control algorithm with many applications, so here I will introduce the LQR control algorithm starting from optimal control.

Note that the basis of the LQR control algorithm is that you must have a foundation in modern control theory and need to know the state space.

1 Example Analysis of Optimal Control Problem

Consider a train W with a mass of m, moving along a horizontal track. Ignore air resistance and the friction of the ground on the train. Consider the train as a point mass moving along a straight line. x(t) represents the position of the train at time t, and u(t) is the external control force applied to the train. Assume that the initial position and speed of the train are x(0)=x0, x' (0)=0, respectively. It is required to select a suitable external control function (t) so that the train can reach and stop at the origin of the coordinate system in the shortest time, that is, the speed is zero when it reaches the origin of the coordinate system.

According to Newton's second law, the equation of motion of the train is:

Initial conditions

Terminal conditions

Due to technical reasons, the external thrust cannot be as large as desired; it is bounded in quantity, i.e.

Where M is a positive constant

The problem is to find a control function u(t) that satisfies equation (1.1.4), transfers W from the initial state (x0,0)' to the final state (0,0)', and minimizes the performance index J(u). Any control function that can meet the above requirements is called optimal control. The rapid lifting and lowering of elevators, the rapid control of rolling mills, and the rapid vibration elimination of mechanical vibrations can all be explained by the above problem.

2 Mathematical description of optimal control

From the above four examples of optimal control problems, it can be seen that the problem to be solved by optimal control theory is to select an admissible control law based on the dynamic characteristics of the controlled object (system mathematical model), so that the controlled object can run according to the predetermined requirements (from the initial state to the terminal state) and make a given performance index reach the optimal value. Therefore, the mathematical description of the optimal control problem should include the mathematical model of the controlled object, the boundary conditions of the system (initial state and final state), the performance index to measure the effect of the "control action" and the admissible control.

2.1 Mathematical Model

The mathematical model of the controlled object, that is, the differential equation of the dynamic system, reflects the physical or chemical laws that the dynamic system should follow during the motion process. Its motion laws can be expressed by the state equation.

Let x=(x1,…,xn)^T ∈ R^n represent the state variable of the control system, and u=(u1,…,um)^T ∈ R^m represent the control variable of the control system. Then the state equation of the control system can usually be described by a set of first-order differential equations:

Equation (1.2.1) summarizes the situations of equation (1.1.1), equation (1.1.6) and equation (1.1.12).

When f does not explicitly contain t, equation (1.2.1) is called a steady-state system (or a time-invariant system). When f is a linear relationship with respect to x and u, equation (1.2.1) is called a linear system. In this case, the equation can be written as

Among them, A(t) is an n-order square matrix, and B(t) is an n-row and m-column matrix. When A and B are independent of time t, equation (1.2.2) is called a linear steady-state system or a linear autonomous system.

In some practical problems, the state variables and control variables of the system are discrete with respect to time. Such a control system is called a discrete control system. Let x(k)=[x1(k),…,xn(k)]^T ∈ R^n represent the state variables of the control system, and u(k)=[u1(k),…,um(k)]^T ∈ R^m represent the control variables of the control system. Then the state equation of the discrete control system can be described by the difference equation as

Equation (1.2.3) generalizes the situation of equation (1.1.15).

2.2 Boundary conditions

The initial and final states of a dynamic system are the boundary conditions of the state equation. The motion of a dynamic system is ultimately a transition from one state to another in the state space, and its motion changes over time corresponding to a trajectory in the state space. The initial state of the trajectory can be recorded as x(t0), where n is the initial time; the terminal state of the trajectory can be recorded as x(s), where tf is the time to reach the final state.

In the optimal control problem, the initial state at t = t0 is usually known, that is, x(t0) = x, while the time t and state x(tn) to reach the terminal vary from problem to problem. As for the terminal time t, it can have two cases: one is fixed, such as = 80 years in Example 1.3 and = 3 years in Example 1.4; the other is variable or free, such as Examples 1.1 and 1.2. As for the terminal state x(tn), the situation is much more complicated and can be summarized into the following three cases:

** (1) Terminal status fixed**

The terminal state is fixed when the terminal state x(tno) corresponds to a fixed point in the state space, that is, x(trsr) is known, such as x(tf)=0 in Examples 1.1 and 1.2, and x(tf)=0.5 in Example 13.

** (2) Terminal state is constrained**

The terminal state is constrained when the terminal state x(tf) is subject to certain conditions, such as the following equation that represents the constraints that x(tf) must satisfy.

** (3) Terminal state freedom**

The terminal state is free when the terminal state x(sn) is no longer a point but a moving point.

For the above situations, they can all be summarized by a target set S. If the terminal state is fixed, then the target set S has only one element; if the terminal state is constrained by certain conditions, then the target set S is a surface in the state space; if the terminal state is free, that is, not constrained by any conditions, then the target set S extends to the entire state space.

2.3 Performance Indicators

The transition from the initial state to the terminal state in the state space can be achieved through different control actions. How to measure the quality of the system under control requires a standard to quantify it. We call this evaluation scale or standard a performance indicator.

It is worth emphasizing that: First, we cannot prescribe a unified format of performance indicators for various optimal control problems. In fact, there is no such thing as an optimal control that covers all aspects. Second, the content and form of the performance indicators depend on the main contradiction to be solved by the optimal control problem. Third, even for the same problem, its performance indicators may vary depending on the designer's focus. For example, some designers focus on shortening time, some designers focus on saving fuel, and some designers take into account both shortening time and saving fuel. Therefore, in order to make the performance indicators just right, theoretical knowledge is indispensable, but the accumulation of experience and skills is particularly important. Performance indicators are generally represented by J, and are given different names in many technical materials, such as performance functional, value function, objective function, benefit function, etc.

The mathematical expression of performance indicators mainly has the following three forms:

** (1) Terminal value performance index, also known as Mayer performance index**

In Example 1.1, in the problem of rapid train arrival, J(u)=tf-t0, which is a terminal value performance indicator.

** (2) Integral performance index, also known as Lagrange performance index**

Example 1.3 In the optimal management problem of a fund,

Integral performance index

** (3) Composite performance index, also called Bolza performance index**

The composite performance index is actually a combination of the final value performance index and the integral performance index. From formula (1.2.7), it can be seen that this index has requirements for the state quantity x(t), control quantity u(t) and terminal state x(tf) of the control process.

2.4 Admissibility Control

For a practical control problem, the control variable u(t) is usually a physical quantity. According to the range of the control variable, control problems can be divided into two categories: one is the control with limited range of control variable, such as the rudder angle of the ship steering and the current of the motor; the other is the control with unlimited or practically unlimited range of control variable, such as the thrust control angle of the missile, which can be changed by +/-360° and is not restricted.

For each control problem, a value of the control action u(t) that satisfies the condition corresponds to a point in the m-dimensional space R^m. All values ​​of the control action u(t) that satisfy the condition constitute a set in the m-dimensional space, denoted as, which is called the admissible control set. All controls that belong to the admissible control set Ω are admissible controls. In the two types of control mentioned above, the former belongs to closed set control, and the latter belongs to open set control. We will see later that these two types of control problems have essential differences in their treatment methods. Optimal control must be admissible control, that is,

2.5 General formulation of optimal control

Assume that the state equation of the known system is

The initial conditions and terminal states satisfy

The control function is

Here, function f is a continuous function of x(t), u(t) and t, and is continuously differentiable with respect to x(t) and t. If there exists a piecewise continuous control function u(t) in the interval [t0, tf] that can transfer the system state x(t) from the initial state x0 to the final state xf∈S, and make the performance index