Take off in 18 seconds!

Over the past decade, quadcopter drones have evolved dramatically. Today, they are widely used in all areas of life, from government inspections and accident mapping to filming, express delivery, and search and rescue.

At least from a commercial or research perspective, you can buy a good drone for just a few thousand dollars, and all you need to do is teach it to fly. But if you want a quadcopter to achieve stable takeoff, you usually need to spend a lot of time on parameter debugging, program development, and flight tests.

Recently, researchers from New York University and TII (Agile Robotics and Perception Laboratory and Technology Innovation Institute) provided new ideas for simplifying the flight process of drones, and successfully sent robots into the sky through rapid simulation on consumer-grade laptops.

By applying deep reinforcement learning and simulation technology, the system they developed can teach a quadcopter drone that has never flown how to take off and hover stably in just 18 seconds. Not only is the takeoff time short, but the hardware requirements for the entire process are also very low: a MacBook Pro can do it.

Deep reinforcement learning, “learning without a teacher”

If you want a computer to independently control complex quadcopter flight, the key is to implement algorithms for autonomous learning and decision-making.

What is reinforcement learning? Simply put, it is a training method that interacts with the environment. The algorithm tries different actions and adjusts its strategy based on the rewards or penalties brought by the actions to maximize the accumulated rewards during the training process.

Researchers do not need to define a complex control model in advance, or manually specify all the characteristics and parameters required for flight. The algorithm will autonomously explore the optimal strategy, allowing the quadrotor to learn without a teacher.

Simulation environment, safe and efficient

The most direct way to teach a quadcopter to fly is to test it in real life and adjust the algorithm based on the flight results. However, the risks and costs of doing so are extremely high. First, it requires a lot of money, and second, each test flight and crash is a loss of equipment.

The solution proposed by the New York University research team is to build a digital simulation platform that is extremely realistic with the real environment.

In this simulated world, the quadcopter can take off, hover, and land continuously, and the drone can obtain status information and make decisions just like in reality. In the event of an accident, you only need to reset the system without worrying about equipment damage.

Such an efficient and secure training environment provides great convenience for algorithm iteration. The team also took advantage of the hardware acceleration provided by Apple M series processors to further shorten the training time and greatly improve the simulation efficiency.

Course-driven, effectively avoiding overfitting

In addition to deep reinforcement learning and simulation technology, the "curriculum-driven" strategy designed in this study also played an important role in rapid training.

The so-called curriculum-driven approach is to dynamically adjust the reward mechanism according to different stages of the training process. At the beginning, more behaviors are rewarded to encourage the algorithm to explore sufficiently. In the later stage, only the optimization goals are rewarded, and the penalties are gradually increased to emphasize robustness and reliability.

The benefit of this approach is that it can effectively avoid overfitting to a specific environment. Some systems perform well in complex simulations, but fail in real scenarios because they rely too much on certain simulation parameters. Dynamically adjusting the reward target can encourage the algorithm to focus on the truly important performance indicators.

And within 18 seconds, not only does the take-off take off, in fact the quadcopter can already maintain a stable hover, avoid nudges and fly along a specific trajectory.

18 seconds, just the beginning

Although the number of 18 seconds is amazing, you may wonder: Is training completed in just 18 seconds really enough to cope with various unknown situations in complex environments?

Indeed, there are still some limitations. For example, the existing training is still too idealistic and cannot cover all situations, such as the response ability under different load conditions and the impact of different ambient light on vision. The training samples still need to be further expanded.

But looking to the future, 18 seconds is just the beginning. Perhaps the next 18 seconds will bring more unpredictable surprises.