Cognex Invents Automatic "Hand-Eye Calibration" for Robotic Arm Vision Systems

A robotic arm is a complex system with high precision, multiple inputs and outputs, high nonlinearity, and strong coupling. Due to its unique operational flexibility, it has been widely used in industrial assembly, safety and explosion protection, and other fields.

At the same time, the robot is also a complex system, with uncertainties such as parameter perturbation, external interference and unmodeled dynamics. Therefore, the modeling model of the robot also has uncertainties. For different tasks, it is necessary to plan the motion trajectory of the robot joint space, so as to cascade to form the terminal posture. This terminal posture often requires the support of hardware sensors and related technologies such as machine vision.

The vision system can be used to inspect the planar objects being operated in the imaging scene. One of the important tasks that needs to be completed with the help of machine vision is hand-eye calibration. This is because when the robot arm performs related tasks, it is often necessary to keep the object to be operated within the visual range of the operating table. However, the robot arm also needs to operate the object to be operated at the same time, which often inevitably causes the position of the object to change, which in turn affects the needs of the operation.

At the same time, due to the cantilever effect in the robot arm, the "hand-eye calibration" task has become more difficult. To this end, Cognex applied for an invention patent on February 3, 2020, entitled "Automatic hand-eye calibration system and method for robot motion vision system" (application number: 202010079136.1), and the applicant is Cognex Corporation.

This patent invents a hand-eye calibration system that can be used for robotic arms. Based on the currently available information, let’s take a look at this invention.

As shown in the figure above, it is a schematic diagram of the robot manipulator assembly of the vision system camera installed on the end actuator invented in this patent. The figure shows the overall process of the multi-axis robot arm performing work on the fixed base 111. Its main task is to align the optical axis and field of view of the camera assembly 120 to the standard plate 130. At the same time, the actuator 112 at the end also includes manipulators such as clamps and suction cups, which can move the parts being aligned.

The advantage of placing the end manipulator and the visual camera in the same position is that the vision system camera assembly can move with the manipulator, and when the actuator moves around the overall workspace of the robotic arm, the camera's alignment function can always keep the manipulator aligned with the part that needs to be aligned, which is the "hand-eye calibration" mentioned in the patent.

It can be noticed that the bottom of the robotic arm is also connected to the computer used by the user. When the visual system uses hand-eye calibration, the user can see the motion information captured by the visual system in real time on the computer. The actual operating space of the robotic arm and the motion picture seen by the user on the computer are actually two different interfaces. Therefore, in order to maintain the consistency of the two interfaces, the coordinate systems of the two screens need to be synchronized, as shown in the figure below.

The hand-eye calibration method invented in this patent can automatically determine the motion parameters of the robot arm and calculate the spatial position to keep the characteristic pattern of the calibration object within the camera field of view, thereby compensating for the cantilever effect. What is the cantilever effect? ​​When the cantilever effect exists, the movement of the manipulator sometimes positions part of the calibration object outside the field of view of the visual system, which will cause a visual blind spot when the user handles the task.

It can also be seen from the figure that the end of the robotic arm and the visual system processor have their own coordinate system management solutions to truly display the information captured by the camera to the user.

Finally, let's look at the flowchart of the process of performing hand-eye calibration in the presence of cantilever effect. As shown in the figure above, the system first uses the pre-calibration process to move the robot so that the calibration pattern is in the center area of ​​the camera's field of view. The system then searches for spatial points for three-point calibration. This step will establish a small predetermined step size based on the size of the entire field of view and the spacing of features within the field of view.

Thereafter, process 500 further determines the availability of the spatial point in the calibration system, and then automatically adjusts the step size based on the feedback from the vision system until the target object is guaranteed to be in the center of the field of view.

The above is the solution for automatic hand-eye calibration in the robot motion vision system invented by Cognex. This solution can be used for hand-eye calibration of the vision system performed together with robot operation. At the same time, it solves the cantilever problem of the robot arm very well, so that when the user uses the robot arm to perform specific operations, the operation object can always be kept in the user's field of view to complete the task efficiently.