Application and development of foreign CCD detection technology in industry

0 Introduction

A charge-coupled device (CCD) is a miniature image sensor that uses electric charge as a signal carrier. It has the functions of photoelectric conversion and signal charge storage, transfer and readout. Its output signal is usually a video signal that meets television standards and can be stored in an appropriate medium or input into a computer to facilitate image storage, enhancement, recognition and other processing [1].

Since the birth of CCD in Bell Labs in 1970, CCD technology has developed rapidly with the development of semiconductor microelectronics technology. The pixel integration, resolution, geometric accuracy and sensitivity of CCD sensors have been greatly improved, and the operating frequency range has been significantly increased. High-speed imaging can be achieved to meet the needs of shooting high-speed moving objects [2]. It has the advantages of wide spectral response, large dynamic range, high sensitivity and geometric accuracy, low noise, small size, light weight, low voltage, low power consumption, strong impact resistance, vibration resistance, anti-electromagnetic interference resistance, durability, long life, small image distortion, no residual image, long-term operation in harsh environments, easy digital processing and connection with computers, etc. It has been widely used in image acquisition, non-contact measurement and real-time monitoring, and has become one of the most active and fruitful research fields in modern optoelectronics and testing technology [1,3].

1 Detection principle of CCD sensor

CCD is an integrated photoelectric conversion device composed of photosensitive units, input structures and output structures. Its outstanding feature is that it uses charge as a signal carrier. Its basic working principle can be found in literature [4,5]. When incident light hits the CCD photosensitive unit, a photocharge Q will be generated in the photosensitive unit. Q is proportional to the photon flow rate Δn0, the illumination time TC, and the photosensitive unit area A, that is:

Q = ηqΔn0ATc (1)

where η is the quantum efficiency of the material; q is the amount of electron charge. The photoelectric conversion characteristics of the CCD image sensor are shown in Figure 1, where the horizontal axis is the illumination, lx.s; the vertical axis is the output voltage, and V0 satisfies the following in the non-saturated region:

f(s) = d1sτ + d2 (2)

where f(s) is the output signal voltage (V); s is the exposure (lx.s); d1 is the slope of the straight line segment (V/lx.s), indicating the light response of the CCD; τ is the photoelectric conversion coefficient, τ≈1; d2 is the output voltage of the CCD when there is no light, called the dark output voltage. The exposure SE corresponding to the inflection point G of the characteristic curve is called the saturated exposure, and the corresponding output voltage VSAT is called the saturated output voltage. When the exposure is higher than SE, the CCD output signal no longer increases. It can be seen that the photoelectric conversion characteristics of the CCD image sensor in the non-saturation area are close to linear. Therefore, the working state of the CCD should be controlled in the non-saturation area.

2 Application of CCD

CCD detection technology, as a non-contact detection technology that can effectively realize dynamic tracking, is widely used in the fields of size, displacement, surface shape detection and temperature detection.

2.1 Dimension measurement

The dimension measurement device composed of CCD sensor, optical imaging system, data acquisition and processing system has the characteristics of high measurement accuracy, fast speed, convenient and flexible application, which is incomparable to the existing mechanical, optical and electromagnetic measuring instruments. In dimension measurement, a suitable lighting system is usually used to make the object to be measured imaged on the CCD target surface through the objective lens. By properly processing the signal output by the CCD, the geometric information of the measured object is extracted, and the measured size can be calculated by combining the transformation characteristics of the optical system [2].

2.1.1 Accurate measurement of part size

In 1997, JBLiao [6] et al. applied the CCD camera system to the three-dimensional coordinate measuring machine (CMM) to realize the automatic measurement of three-dimensional coordinates. They installed a CCD array at a fixed position at a 45° angle to the three axes of the CMM, and connected the computer vision system to the original control system of the CMM to control the movement of the probe and the workpiece, thereby detecting the three-dimensional position of the probe and the workpiece. This method does not require changes to the original CMM system, and only requires the CCD vision system to be connected to the original measuring machine. Since only one CCD array is used in the measurement system, the measurement system structure is simplified, the system cost is reduced, the error caused by manual operation is reduced, the measurement efficiency is improved, and the edge detection error caused by light diffraction when using CCD alone for measurement can be avoided. It can be used for accurate measurement of the three-dimensional size of the workpiece. However, this method requires certain prior knowledge of the working environment and the shape of the workpiece, which greatly limits its application scope. To this end, VH Chan, C. Bradley et al. [7] proposed an automatic measurement method using a composite sensor. This method installs a black and white CCD and a coordinate probe at the end of the Z-axis working arm of the CMM. Before detection, the CCD first images the workpiece from the front, back, left, right and top, and determines the surface position and area of ​​the workpiece through a neural network-based stereo pairing algorithm, thereby determining the detection path of the probe. This method has a high degree of intelligence and can efficiently measure the three-dimensional dimensions of workpieces with complex shapes, and can construct a CAD model of the workpiece based on the measured data. However, the calculation is complex and requires a computer with fast computing speed and large memory capacity. In addition, the accuracy of the algorithm's stereo matching needs to be improved.

Although the above measurement system has been significantly improved by the introduction of CCD technology, it is still a contact measurement and cannot accurately measure certain elastic and soft workpieces. Recently, PFLuo et al. [8] replaced the CMM probe with a CCD camera and combined it with laser ranging technology to achieve non-contact and accurate measurement of one-dimensional dimensions. This method uses sub-pixel precision detection technology and uses a laser rangefinder for distance correction, which effectively improves the detection accuracy. Its accurate measurement range is 1 to 300 mm, but this method can only measure one-dimensional dimensions. PFLuo et al. believe that the system can achieve accurate measurement of two-dimensional dimensions after improvement, and the data fluctuation caused by vibration caused by the sliding of the worktable can also be effectively reduced, but no successful examples have been seen.

2.1.2 Measurement of micro-sizes

In order to detect whether the pin heights of BGA (ball grid array) chips are coplanar, RVSI, an American company, developed a single-point offline detection device based on laser triangulation [1]. This device can only measure one pin at a time, has a slow measurement speed, and cannot achieve online measurement. In 1999, Kim, Pyunghyun [9] and others proposed a new stereo measurement method. This method uses a laser line source to illuminate the chip pins. The illuminated pin images are collected by two sets of CCD camera systems at a certain angle to each other and then input into a computer for stereo matching. The pin height and longitudinal spacing are calculated using the perspective transformation model and coordinate transformation relationship. The chip under test is then driven by a stepper motor to move in one direction, thereby achieving three-dimensional size measurement. A capacitive micrometer is introduced to monitor the position change of the workbench in real time, and dynamic error compensation is performed, effectively reducing the error caused by vibration. In 2001, CJ Tay, X. He [10] and others used image recognition and digital correlation technologies to simplify the calculation process, making it possible to calculate the height of hundreds of pins in just a few seconds, thereby effectively improving the practicality of the detection system. Recently, CJ Tay [11] and others proposed a method for simply measuring the height of BGA pins based on optical shadows, based on the inherent relationship between the image and shadow of an object illuminated by oblique light. This method uses laser to illuminate the pins of the chip under test obliquely to produce pin shadows. The pins and their shadows are collected by a CCD camera with a telephoto microscope and input into a computer. The computer software calculates the pin height based on the relationship between the shadow and the image. The author proposed two concise calculation methods that can avoid edge detection errors caused by light diffraction. The calculation is simple and fast, but requires a high-precision mechanical positioning device, and can only detect a few pins at a time. In addition, the chip flatness and detection environment are very high, and further improvements are needed before it can be put into practical use.

In recent years, the technology of combining CCD technology with moiré fringe, digital holography, electronic speckle interferometry and other technologies to accurately measure micro-sizes is becoming a research and development direction with great potential [12]. [page]

2.2 Deformation measurement

Although the use of linear array CCD to measure material deformation has the advantages of non-contact, no wear, high precision, no introduction of additional errors, and the ability to measure the entire process of material stretching, especially measuring the stress-strain curve of the material before and after fracture, and obtaining various limit characteristic parameters of the material, it can only measure the uniform deformation of the material in the axial direction when it is stretched. For this reason, Scheday, Miehe and Chevalier et al. [13] conducted research on measuring material deformation using area array CCD. On this basis, Stefan Hartmann et al. [14] used area array CCD to study the deformation of rubber materials during stretching and compression. That is, several white dots are equidistantly marked in the axial direction of the cylindrical black test sample, and the corresponding images are captured by CCD and sent to the computer for processing. The axial deformation of the sample when subjected to force is calculated by detecting the distance between the white dot marks, and the contour size of the axisymmetric cylindrical sample is obtained by the contour detection algorithm. After data correction, the deformation of the sample in the radial direction can be calculated. This method can obtain the deformation in two directions at the same time and measure the non-uniform deformation of the material when it is compressed. S. Claudinon, P. Lamesle et al. [15] used a similar method to study the deformation of quenched steel samples during air cooling, solved the problem of dimensional measurement of high-temperature samples, and could continuously measure the deformation at different temperatures, but at low temperatures, measurement errors are prone to occur. J.-M. Siguier et al. [16] used two CCD cameras to capture the surface image of the object being tested to study the properties of the surface material of a large scientific balloon, and obtained the three-dimensional deformation of the sample by stereo correlation method. However, this measurement method is technically complex, and the data obtained in the normal direction perpendicular to the material surface is relatively small.

2.3 Mechanical wear measurement

Although the above methods can measure the size or deformation of various workpieces, they are subject to many limitations when measuring certain special workpieces. For example, when detecting the wear of tools on high-speed cutting machines, the tools need to be removed for measurement. To this end, some researchers have devoted themselves to the study of using machine vision to detect the degree of tool wear. In 2000, T. Pfeifer and L. Wiegers [17] compared various measurement methods and pointed out that the detection system based on machine vision has the most advantages and potential. They also built a non-contact detection system consisting of a CCD camera, lighting equipment, and a fixture. The system images the side of the tool edge at an appropriate position, inputs the collected tool image signal into the computer, calculates the tool wear profile, and uses this to determine the tool wear level and the tool replacement time. However, the image processing process of this system is complex, the application range is narrow, and the detection accuracy and efficiency need to be improved. In 2002, Jeon-Ha Kim et al. [18] conducted experimental analysis on the error factors one by one on this basis, determined the optimal light intensity, angle, shooting angle, etc., and inserted the light source into the periphery of the lens through the optical fiber to reduce the error caused by the shadow, so that the free rotation angle of the fixture is increased, the size of the imaging device is reduced, and the scope of use of the system is improved. At the same time, by using the lateral dimension difference of the tool before and after wear to calculate the wear degree, the image processing process is greatly simplified. Experimental measurements of four different tools show that the measurement signal-to-noise ratio of the system can reach 46 dB, the measurement accuracy and speed are significantly improved, and real-time online measurement can be achieved, but it is not suitable for measuring tools with too complex geometric shapes.

2.4 Three-dimensional surface measurement

Since the CCD sensor can simultaneously obtain the brightness and phase information of the measured surface, the combination of CCD and computer image processing technology with the traditional three-dimensional surface non-contact optical measurement method can measure the deformation, vibration and shape of the object in real time. In the early 1990s, Yamaguchi et al. [19] used linear array CCD to measure the Parson's ratio of different materials in speckle interferometry, but the linear array CCD can only record one-dimensional orthogonal correlation information. With the improvement of CCD technology, area array CCD has been widely used in three-dimensional surface measurement [19]. In 1996, B. Skarman et al. [20] proposed the phase change digital holographic measurement method. Since then, F. Chesn [21], C. Quan [22], PS Huang [23], G. Pedrini et al. [24] have applied CCD technology in relevant measurement methods, and new methods for obtaining phase images from CCD images [24, 26, 27] have also emerged. When phase change technology is used in the fringe pattern projection method, only the surface contour of static objects can be detected, which is not suitable for real-time detection of vibration and changing surface shapes. For this reason, CJ Tay et al. [28] established a system for three-dimensional detection of low-frequency vibrating object surfaces. The system consists of an oscillation generation system, a liquid crystal display fringe emitter, a special telecentric lens, a high-speed CCD, an image acquisition card and a computer. The telecentric lens used in the system can keep the magnification constant, so that the measurement result is independent of the distance between the measured object and the CCD, thereby reducing the measurement error caused by the change in depth of field when the object vibrates during measurement. At the same time, the phase scanning method is used to calculate the phase of the fringe pattern point by point, which can obtain the vibration frequency and amplitude of the measured object in real time and reconstruct the surface profile of the object in real time. Its measurement accuracy can reach 1/500 of the amplitude value. However, the system can only measure the positive fringe plane, and requires high-quality sinusoidal emission fringes and CCD image acquisition frequency greater than the vibration frequency of the measured object. Subsequently, they applied a similar method in the shadow moiré fringe interferometry to measure the three-dimensional surface of the vibrating object and achieved good results [29]. Although this method is simpler and more practical than the digital holographic method [30] and has relatively low requirements for the measurement environment, its measurement range is limited by the CCD acquisition speed and is not practical for measuring the surface of objects with high-speed vibration and irregular deformation.

2.5 High-temperature measurement

The wavelength and intensity of the radiation light of an object have a specific relationship with the temperature of the object. Therefore, CCD, as a photoelectric conversion device, can be used for temperature measurement. In 1993, Tenchov et al. [31] used CCD to indirectly measure the surface temperature of the solution; in 1995, KYHsu and LDChen [32] used an enhanced CCD that can measure infrared bands to measure the combustion flame temperature of liquid metal, but the measurement error reached 400-200K, which lacked practicality. Since then, using infrared CCD to measure temperature fields has become the mainstream of CCD temperature measurement research. In 2001, Takeshi Azami et al. [33] used the brightness fluctuation information of CCD to study the heat flow condition on the surface of molten silicon bridge and obtained good results. In 2002, D. Manca et al. [34] proposed a practical method for measuring and controlling the flame temperature field of the combustion chamber using infrared CCD. In 2003, G. Sutter et al. [35] used an enhanced CCD to measure monochromatic light of a certain wavelength emitted from the surface of an object similar to a black body, thereby obtaining the radiation temperature of the object. The difference between the measurement result and the actual temperature of the object was almost negligible, and it was used to measure the tool temperature field of a right-angle high-speed cutting machine. However, the author did not specify the image processing and temperature calculation methods, nor did he conduct error analysis. The experimental error reached 16 °C. When this method measures temperatures in different ranges, it is necessary to find different optimal wavelengths and use a filter with a very narrow band to obtain a single wavelength light radiation signal. In 1996, B. Skarman et al. [36, 37] proposed using CCD to shoot a hologram of a fluid and reconstruct the three-dimensional temperature field of the fluid through image processing technology. Since the CCD acquisition speed, image processing speed and storage speed were relatively low at that time, and the laser interference quality was not high, this method lacked practicality. By 1998, this method entered the practical stage and could measure the three-dimensional temperature of stable transparent liquids and obtain data such as flow rate and fluid density. In 2002, CHhmann et al. [38] used the characteristics of high-resolution temperature sensing liquid crystal color changing with temperature to sense the temperature of the measured area, and then used a color CCD to capture the color image of the liquid crystal surface to indirectly measure the temperature of the meniscus when the liquid evaporates. This method can achieve temperature measurement of a small area, but it requires precise calibration. Other scholars have proposed using CCD in conjunction with a laser-sensing phosphor to measure temperature [39]. In fact, from the spectral response characteristics and photoelectric conversion characteristics of CCD, it can be seen that the brightness and chromaticity information in the surface image of the measured object can be obtained using the RGB output value, and the surface temperature field of the object can be calculated based on the colorimetric temperature measurement principle. Although someone has proposed a three-dimensional temperature field construction algorithm based on a CCD temperature measurement system [40], instruments that directly use color CCD to measure temperature are still in the experimental development stage. Nevertheless, since CCD technology can measure the temperature of moving objects, provide a two-dimensional or three-dimensional temperature field, and achieve non-contact high-temperature measurement, CCD temperature measurement technology has great development potential and application prospects.

3 Conclusion

In summary, CCD application technology has become a comprehensive technology integrating optics, electronics, precision machinery and computer technology, and is widely used in the field of modern optical and optoelectronic testing technology. In fact, CCD can be applied almost anywhere where film and photoelectric detection technology can be used. With the development of semiconductor materials and technology, especially the continuous progress of ultra-large-scale integrated circuit technology, the performance of CCD image sensors is also rapidly improving. Combining CCD technology, computer image processing technology and traditional measurement methods can obtain more information about the object being measured, realize fast and accurate non-contact measurement, and significantly improve the level of measurement technology and intelligence. Therefore, CCD technology will surely be more and more widely used in industrial measurement and control, machine vision, multimedia technology, virtual reality technology and many other fields with its outstanding advantages.

References
[1] Wang Qingyou. CCD application technology [M]. Tianjin: Tianjin University Press, 2000.
[2] Wang Yueke, Yang Huayong. Current status and application prospects of CCD image sensing technology [J]. Optical Instruments, 1996, 18(5): 32-36.
［3］The past, present and future of scientific CCD［J］. Laser and Optoelectronics Progress, 1995, (10): 8-10．
［4］Yan Lei, Zhang Boxu, Chang Bingguo. CCD image sensor and its digital camera technology［J］. Information Recording Materials, 2002, 3(1): 45-49．
［5］Central Optical Digital Imaging Company CCD & CMOS Imaging and Machine Vision Product Manual［M］.