Designing a Robotic Device for Studying Flying Insects Using LabVIEW

Schematic diagram of information flow in the fly control robot experiment

Using NI's LabVIEW software and CompactRIO hardware, we built a fast, modular, and easy-to-use biomimetic robotics platform that involves various industrial protocols and real-time closed-loop stimulus signal generation.

"With the help of CompactRIO controllers and LabVIEW, we studied how flying insects achieve excellent flight control."

The maneuverability of flies, which can chase at high speeds and land precisely on the edge of a plate, is of great interest. Flies can be used as model systems to study neural information processing, aerodynamics, and genetics, and they can use their biosensors, controllers, and actuators quickly and precisely. Such capabilities are of great interest but difficult to study. Measurement and stimulation devices must have high bandwidth, low latency, and flexible interfaces. Ease of use and modularity are also key to interdisciplinary and collaborative research.

We used CompactRIO controllers and LabVIEW graphical system design software to study how flying insects achieve their remarkable flight control. We used digital I/O modules to interface with an LED-based visual stimulus field, which has precise temporal and spatial resolution, allowing us to effectively stimulate the fly's visual system. Recording the insect's response requires a fast and flexible acquisition system. LabVIEW provides the speed and modularity required to record these signals and generate stimulation signals as real-time feedback. This allows us to use the fly as a living sensor and embed it into a scientific system.

We developed an experiment. In the experiment, we tethered a fruit fly and used the fly's movements to control the e-puck robot. The e-puck is a small mobile robot that was developed as a university research project and is designed to navigate an environment full of obstacles. Feedback from cameras and proximity sensors attached to the robot is used to determine the visual stimuli presented to the fly and flight parameters such as wingbeat frequency and amplitude to control the robot's movements (Figure 1). The transfer function between the fly and the robot is varied to enable a range of experimental modes.

High-speed movies of flies: accelerated LED visual fields

The visual stimulus field consists of eight green LED panels connected to a custom controller via I2C protocol. In previous designs, all flies were controlled by a single bus. To achieve higher frame rates and adjust the visual stimulus based on the fly's feedback, we had to use multiple parallel buses. Ultimately, we chose to replace the original controller with the NI cRIO-9014 real-time controller and the all-in-one NI cRIO-9104 reconfigurable embedded chassis.

Fly Control Robot: From Flies to Robots

In the experimental setup (Figure 2), a fruit fly is tethered to the center of a circular array of LED panels. Although the insect cannot move, it can still flap its wings and fly in the same manner as in free flight. A digital wingbeat analyzer obtains the current frequency, amplitude, position mean, and phase of the fly's wingbeats. These behavioral state vectors are transmitted via user datagram protocol (UDP) packets to a host computer running LabVIEW. Custom transfer functions can be applied on the host computer to calculate the updated wheel speed of the e-puck robot. These values ​​are then sent to the robot via Bluetooth.

From robot to fly

When we use insect behavior to manipulate the robot, feedback from the robotic device modifies the visual display facing the insect. Feedback is provided by three linear cameras and eight proximity sensors mounted on the top of the robot. The cameras acquire at 10Hz, with 102 pixels per frame. The proximity sensors output calibrated data at 20Hz. The host receives these signals via Bluetooth and applies a second custom transfer function to generate the next frame of the image displayed on the LED visual field. The

host application sends the new image pattern to the real-time controller via Ethernet. This image pattern is then divided into 8×8 pixel blocks, each corresponding to an LED panel, and converted into I2C commands. For maximum throughput, this data is passed to the FPGA (field programmable gate array) via a FIFO (first-in-first-out) queue of DMA (direct memory access). Interrupt vectors ensure synchronization between the real-time controller command generation and the FPGA underlying hardware communication. The FPGA backplane then controls 12 buses using the I2C protocol, each bus controlling five panels. Thus, the environment seen by the robot determines the visual stimulus for the fly, and the fly's response to the visual stimulus changes the robot's path.

The frame rate of the visual stimulus is between 30Hz and 400Hz, depending on the depth of the pattern and whether it is vertically symmetrical. The cumulative delay in the control loop is less than 50 milliseconds and this is mainly caused by the transmission of sensor information from the robot to the host via Bluetooth.

Efficient design: flexible interface and modular structure

With LabVIEW and CompactRIO, we can connect to a range of research tools through a variety of protocols. The great flexibility and many example programs provided by the NI and LabVIEW online user community make applications designed based on LabVIEW an effective alternative to custom controllers in experimental biology.

We designed a friendly GUI (graphical user interface) that provides the experimenter with the necessary control means and information, thereby simplifying the complexity of running code on multiple hardware platforms (Figure 3). This feature is very effective in some interdisciplinary applications, which can enhance the close cooperation between biologists, mathematicians, physicists, and engineers. In addition, the modularity and portability of LabVIEW code also make it possible to share and reuse between laboratories. For example, in a customized version of this solution, the operation mode can be pre-generated and saved in a USB flash drive, then downloaded to the RAM of the real-time controller and then transferred to the LED panel to obtain a higher refresh rate.

A hybrid adaptive controller

Because some of the fly's neural circuits are highly plastic, it can be considered an adaptive controller. Using the new biomimetic robotic platform, we were able to evaluate the controller's performance under various external transfer functions that mimic almost all of the fly's natural flight environments, such as determining the up and down movement of a grid in the visual field based on the position of the obstacle closest to the robot. Surprisingly, the transfer function that is closest to intuition does not necessarily give the best results.

LabVIEW and CompactRIO provide an ideal solution for constructing this control loop that includes live insects and allows us to perform a variety of experiments. CompactRIO is responsible for acquiring and generating a variety of signals that apply different industrial standards and expands custom research tools. In addition, because our applications on the computer, real-time controller, and FPGA are implemented in the same programming environment and development language, it greatly saves our learning time and improves efficiency. In addition, a large number of accessories and external interfaces also provide great potential for future expansion and adaptability.