A new method for high-precision real-time stitching of large-format scanners based on multi-line CCDS cameras

0 Introduction

The development of high-precision large-format scanners based on multi-line array CCDS cameras has always been a key area of ​​concern for domestic and foreign scholars and the industry. However, it involves five key technologies: precise optical path design, complex mechanical platform, complete hardware system, friendly human-machine interface, and intelligent algorithm implementation, among which high-precision real-time stitching technology is the key difficulty. So far, all key technologies of high-precision large-format scanners based on multi-line array CCDS cameras are basically monopolized by foreign multinational companies and large research institutions, and domestic ones rely entirely on imports.

High-precision real-time stitching technology is the key difficulty in large-format scanners with multi-line array CCDS cameras. Due to the limited field of view of line array CCDS cameras, in the context of large-format scanner applications that require a large field of view and high scanning accuracy, a single three-line array CCDS camera obviously cannot meet the requirements, so multiple three-line array cameras need to be stitched together to achieve this.

The most important indicator of stitching is to ensure that image information is not lost or misaligned. The ideal situation is that all three-line array CCDS cameras remain on the same horizontal plane and the pixels of the images captured by two adjacent cameras are continuous.

In order to meet the application requirements of splicing multi-line array CCDS large-format scanners, optical splicing or optical splicing plus software post-processing is usually used¨01. Optical splicing connects multiple line array CCDS cameras in the first place through a beam splitter prism, that is, the last effective pixel of the previous CCDS camera is connected to the first effective pixel of the next CCDS camera. The accuracy requirement is within 1 pixel, and all pixels of multiple CCDS cameras must be on the same horizontal plane. This requires very high mechanical processing accuracy and the corresponding cost is also very high. The splicing accuracy decreases with the increase of usage time. Optical splicing plus software post-processing overlaps the effective pixels of two adjacent CCDS cameras to ensure that the image information is not lost, and then saves the collected image information in the computer memory. The image data collected by each camera is moved up and down and left and right for splicing through the software algorithm. This splicing method reduces the accuracy requirements of the machine, but has high requirements for computer configuration. The splicing time is long and takes 2 s to 3 s, which has a certain impact on the real-time performance of large-format scanners¨2|.

To solve this problem, the author deeply studied the internal structure of FPGA/SDRAM/CCDS, and through many experiments, proposed a new method for high-precision real-time stitching of large-format scanners based on multi-line CCDS cameras - hardware stitching method.

1 Basic theory

1.1 Splicing Principle

In order to ensure that data is not lost at the end where adjacent cameras are connected, the effective pixels between the two cameras must be partially overlapped, as shown in the splicing principle diagram in Figure 1.

Camera 1 overlaps with camera 2 by W1 pixels, and camera 2 overlaps with camera 3 by W2 pixels. The address of SDRAM is divided into three address blocks, the initial address corresponding to camera 1 is ADDRESS-1, the initial address corresponding to camera 2 is ADDRESS-2, and the initial address corresponding to camera 3 is ADDRESS_3. The three-line CCDS camera scans line by line, and stores each line of data collected by the three cameras in three address blocks opened by SDRAM. Because the address of SDRAM corresponds to the data one by one, knowing the first address of each line of data, by setting the offset, the data of the three cameras can be spliced ​​in real time by continuously reading the data from the first address bit of each line of data.

1.2 Actual situations and solutions during splicing

In the actual process, due to the accuracy of mechanical platform processing, the three cameras may not be on the same horizontal plane, and the overlapping pixels of two adjacent cameras may not be the same, which causes misalignment in conventional data transmission. According to experiments and analysis, there are 6 possible situations in the stitching process, as shown in Figures 2(a), 2(b), 2(c), 2(d), 2(e), and 2(f).

Figure 2(a) shows the image valid data misalignment when the horizontal position of camera 1 is greater than that of camera 2 and the horizontal position of camera 2 is greater than that of camera 3. Figure 2(b) shows the image valid data misalignment when the horizontal position of camera 1 is less than that of camera 2 and the horizontal position of camera 2 is less than that of camera 3. Figure 2(c) shows the image valid data misalignment when the horizontal position of camera 2 is greater than that of camera 1 and the horizontal position of camera 1 is greater than that of camera 3.

Figure 2(d) shows the misalignment of valid image data when the horizontal position of camera 2 is greater than that of camera 3, and the horizontal position of camera 3 is greater than that of camera 1. Figure 2(e) shows the misalignment of valid image data when the horizontal position of camera 3 is greater than that of camera 1, and the horizontal position of camera 1 is greater than that of camera 2. Figure 2(f) shows the misalignment of valid image data when the horizontal position of camera 1 is greater than that of camera 3, and the horizontal position of camera 3 is greater than that of camera 2.

Therefore, it is necessary to first determine the position of the three cameras of the scanner. The number of pixels in each row of the camera is L=5340. The host computer sets the values ​​of H1, H2, W1, and w2 according to the specific situation. The maximum number of rows of data that need to be cached is calculated based on the preset values ​​of H1, H2, W1, and W2. The read-out first and last addresses of each row of cached data of camera 2 and the read-out first and last addresses of each row of cached data of camera 3 are calculated. Finally, the cached data of each camera is read and written sequentially and transmitted to the host computer for real-time display.

The initial addresses of the write buffers in the six cases shown in Figure 2 are all consistent, that is, the initial address of the write buffer for camera 1 is ADDRESS-WRITEBUFFER.1=ADDRESS-1, the initial address of the write buffer for camera 2 is ADDRESS-WRITEBUFFER_2=ADDRESS_2, and the initial address of the write buffer for camera 2 is ADDRESS-WRITEBUFFER-3=ADDRESS_3.

For the situation in Figure 2(a), it is necessary to cache BUFFER-LINE=H1+H2+1 line data. After camera 2 continuously writes H1+1 line data, it is necessary to initialize the write address. After camera 3 continuously writes H1+H2+1 line data, it is necessary to initialize the write address. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1+(H2-H1)·L+H2-H1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS-2+H2·L+H2, camera 3 writes the initial address ADDRESS-WRITE_3=ADDRESS_3. Camera 1 reads the initial address ADDRESS-READ-1=ADDRESS-1+(H1+H2)·L+H1+H2, camera 2 reads the initial address ADDRESS-READ_2=ADDRESS_2+H2·L+W1+H2, camera 3 reads the initial address ADDRESS-READ_3=ADDRESS._3+W2.

For the situation in Figure 2(b), it is necessary to cache BUFFER-LINE=HI+H2+1 line data. After camera 1 continuously writes H1+H2+1 line data, it is necessary to initialize the write address. After camera 2 continuously writes H2+1 line data, it is necessary to initialize the write address. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS_2+H1·L+H1, camera 3 writes the initial address ADDRESS-WRITE-3=ADDRESS_3+(H1+H2)·L+H1+H2. Camera 1 reads the initial address ADDRESS-READ-1=ADDRESS-1, camera 2 reads the initial address ADDRESS-READ-2=ADDRESS-2+H1·L+W1+H1, camera 3 reads the initial address ADDRESS-READ-3=ADDRESS 3+(H1+H2)·L+H1+H2+W2.

For the situation in Figure 2(c), BUFFER-LINE=H2+1 line data needs to be cached. After camera 1 continuously writes H1+1 line data, the write address needs to be initialized. After camera 3 continuously writes H2+1 line data, the write address needs to be initialized. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1+(H2-H1)·L+H2-H1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS-2+H2·L+H2, camera 3 writes the initial address ADDRESS-WRITE_3=ADDRESS-3. Camera 1 reads the initial address ADDRESS_READ-1=ADDRESSs-1+(H2-H1)·L+H2-H1, camera 2 reads the initial address ADDRESS-READ-2=ADDRESS-2+I-12·L+w1+H2, camera 3 reads the initial address ADDRESS_READ_3=ADDRESS_3+W2.

For the situation in Figure 2(d), BUFFER-LINE=H1+1 lines of data need to be cached. After camera 1 continuously writes H1+1 lines of data, the write address needs to be initialized. After camera 3 continuously writes H2+1 lines of data, the write address needs to be initialized. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS-2+H1·L+W1+HI, camera 3 writes the initial address ADDRESS-WRITE-3=ADDRESS._3+(H1-H2)·L+HI-H2. Camera 1 reads the initial address ADDRESS-READ-1=ADDRESS-1, camera 2 reads the initial address ADDRESS-READ-2=ADDRESS-2+H1·L+W1+H1, camera 3 reads the initial address ADDRESS-READ-3=ADDRESS_3+(H1-H2)·L+H1-H2+W2.

For the situation in Figure 2(e), it is necessary to cache BUFFER-LINE=H2+1 line data. After camera 1 continuously writes H2-HI+1 line data, it is necessary to initialize the write address. After camera 2 continuously writes H2+1 line data, it is necessary to initialize the write address. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1+H1·L+H1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS-2, camera 3 writes the initial address ADDRESS-WRITE-3=ADDRESS-3+H2·L+H2. Camera 1 reads the initial address ADDRESS-READ-1=ADDRESS-1+HI·L+H1, camera 2 reads the initial address ADDRESS-READ-2=ADDRESS_2+W1, camera 3 reads the initial address ADDRESS-READ-3=ADDRESS_3+H2·L+H2.

For the situation in Figure 2(f), BUFFER-LINE=H1+1 line data needs to be cached. After camera 2 continuously writes H1+1 line data, the write address needs to be initialized. After camera 3 continuously writes H1-H2+1 line data, the write address needs to be initialized. Camera 1 writes the initial address ADDRESS-WRITE-1=ADDRESS-1+H1·L+H1, camera 2 writes the initial address ADDRESS-WRITE-2=ADDRESS-2, camera 3 writes the initial address ADDRESS_WRITE_3=ADDRESS_3+H2·L+H2. Camera 1 reads the initial address ADDRESS-READ-1=ADDRESS-1+H1·L+H1, camera 2 reads the initial address ADDRESS-READ_2=ADDRESS_2+W1, camera 3 reads the initial address ADDRESS-READ-3=ADDRESS-3+H2·L+H2+W2.

2 Experimental results and discussion

2.1 Method Execution Process

The process flow of the new method for realizing high-precision real-time stitching of large-format scanners with multi-line array CCDS cameras is shown in Figure 3. According to the characteristics of the main components, the FPGA and SDRAM need to be initialized after power-on to make them in normal working state. The preset values ​​of H1, H2, W1, and W2 are obtained from the host computer to determine the situation of the camera position. When the highest bit of H1 is defined as 1, the horizontal position of camera 1 is higher than the horizontal position of camera 2, and when the highest bit of H1 is 0, the horizontal position of camera 1 is lower than the horizontal position of camera 2. When the highest bit of I-12 is defined as 1, the horizontal position of camera 2 is higher than the horizontal position of camera 3, and when the highest bit of H2 is 0, the horizontal position of camera 2 is lower than the horizontal position of camera 3. The 6 situations shown in Figure 2 are divided into 6 states corresponding to the preset values ​​of H1 and H2. When the highest bit of H1 is 1 and the highest bit of H2 is 1, it corresponds to Figure 2(a); when the highest bit of H1 is 0 and the highest bit of H2 is 0, it corresponds to Figure 2(b); when the highest bit of HI is 0, the highest bit of I-12 is 1, and H1 is less than I-12, it corresponds to Figure 2(c); when the highest bit of H1 is 0, the highest bit of H2 is 1, and H1 is greater than or equal to H2, it corresponds to Figure 2(d); when the highest bit of H1 is 1, the highest bit of H2 is 0, and H1 is less than H2, it corresponds to Figure 2(e); when the highest bit of H1 is 1, the highest bit of H2 is 0, and H1 is greater than or equal to H2, it corresponds to Figure 2(f). After obtaining the preset value, data cache is performed, image data is read out line by line according to the camera number, and it is determined whether the read address needs to be initialized. Image data is written line by line according to the camera number, and it is determined whether the write address needs to be initialized. Finally, it is determined whether the end signal sent by the host computer is received.