Start learning machine vision with Feiteng Pi (4): Motion detection

Xinchacha has opened the Feitengpai communication community. This article is an excerpt of community articles. Everyone is welcome to communicate in the community.

In the previous article, the basic algorithm for separating objects from static images - the binarization method - was introduced. This article will introduce the separation of moving objects from dynamic videos . Separating moving targets or changing areas from background images is also called motion detection. Motion detection is often used in video surveillance, anomaly detection, tracking photography and other fields. In addition to playing an important role in the industrial field, motion detection is also the basis for applications in many fields , such as drone flight and three-dimensional reconstruction. At present, the main methods of motion detection include: background subtraction, frame difference and optical flow method.

Background subtraction, frame difference

Background subtraction and frame difference are classic methods for motion detection and are currently the mainstream methods ¹ . As the names suggest, background subtraction and frame difference mainly use the difference results obtained by picture subtraction to detect targets. The difference is that background subtraction subtracts the current frame in the video from the determined or real-time updated background reference model. According to Threshold to find different areas, for example, the area that differs from the background image by more than a certain threshold is regarded as the motion area, and the part smaller than the threshold is regarded as the background area. Frame difference uses two consecutive frames, three frames, or multiple frames of images to perform difference operations and obtains motion areas from the difference results based on thresholds. Background subtraction and frame difference algorithms are simple and are mainly used in
scenes where the camera is fixed. The detection speed is fast, but it is susceptible to interference from changes in external light.

Optical flow algorithm

Optical flow algorithm (Optical flow, OF) refers to the movement of target pixels in the image caused by the movement of objects in the image or the movement of the camera in two consecutive frames of images. Different from background subtraction and frame difference, the core of the optical flow algorithm is to find the correspondence between the previous frame and the current frame by calculating the changes in the time domain of pixels in the video and the correlation between adjacent frames. Method ² to calculate the motion information of objects between adjacent frames . The optical flow method works based on two basic assumptions: 1) The pixel intensity of the scene is basically unchanged between adjacent frames 2) Adjacent pixels have similar motion.

Use P(x,y) to represent the image pixel value, P(x,y,t) to represent the image pixel value of a certain frame at time t. After \text{d}t time, the pixel has moved (dx in the next frame ,dy) , since these pixels are the same and the intensity is basically unchanged, P(x,y,t) can be expressed as:

P(x, y, t) = P(x+dx, y+dy, t+dt)

Use Taylor expansion:

P(x+dx, y+dy, t+dt) = P(x, y, t) + \frac{\partial P}{\partial x} \Delta x + \frac{\partial P}{\partial y}\Delta y + \frac{\partial P}{\partial t} \Delta t + O

Due to the OF assumption: the pixel intensity of the scene is basically unchanged between adjacent frames

\frac{\partial P}{\partial x} \Delta x + \frac{\partial P}{\partial y}\Delta y + \frac{\partial P}{\partial t} \Delta t = 0

Divide the left and right sides by \Delta t at the same time to get

\frac{\partial P}{\partial x} \frac{\Delta x}{\Delta t} + \frac{\partial P}{\partial y} \frac{\Delta y}{\Delta t} + \frac{\partial P}{\partial t} = 0

The above formula is called the optical flow equation, \frac{\partial P}{\partial x} and \frac{\partial P}{\partial y} are the gradients of the image along x, y , \frac{\partial P} {\partial t} is the gradient along time. The optical flow method mainly calculates \frac{\Delta x}{\Delta t} , \frac{\Delta y}{\Delta t} to obtain the velocity estimate of each pixel to detect moving objects. Currently, there are many solutions to the optical flow equation. This article will mainly introduce a classic optical flow solution - the Lucas-Kanade algorithm ³ . Compared with the two basic assumptions of the optical flow method, the LK algorithm adds a basic assumption of spatial assumptions: adjacent points on a scene are also adjacent points when projected onto the image, and the speed of adjacent points is consistent. opencv provides functions to implement the LK algorithm and calculate optical flow. calcOpticalFlowPyrLK

The function prototype is as follows:

void cv::calcOpticalFlowPyrLK	(	InputArray 	prevImg,
InputArray 	nextImg,
InputArray 	prevPts,
InputOutputArray 	nextPts,
OutputArray 	status,
OutputArray 	err,
Size 	winSize = Size(21, 21),
int 	maxLevel = 3,
TermCriteria 	criteria = TermCriteria(TermCriteria::COUNT+TermCriteria::EPS, 30, 0.01),
int 	flags = 0,
double 	minEigThreshold = 1e-4 
)

code time

When using the function, you need to first call the function to find key points, and then use to calculate the key point optical flow. At the same time, this article still uses the computational efficiency used in the previous article . calcOpticalFlowPyrLK GoodFeaturesToTrack calcOpticalFlowPyrLK Timer

#include <iostream>
#include "timer.hpp"
#include <opencv2/video/tracking.hpp>
#include <opencv2/video.hpp>
#include <opencv2/imgproc.hpp>
#include <opencv2/highgui.hpp>
#include <opencv2/imgcodecs.hpp>
#include <opencv2/opencv.hpp>

#include "timer.hpp"

using namespace cv;
using namespace std;

int main(const int argc, const char *argv[])
{
    const string file_name = std::string(argv[1]);

    VideoCapture capture(file_name);
    if (!capture.isOpened())
    {
        // error in opening the video input
        cerr << "Unable to open file!" << endl;
        cerr << file_name << endl;
        return 0;
    }
    // Create some random colors
    vector<Scalar> colors;
    RNG rng;
    for (int i = 0; i < 100; i++)
    {
        int r = rng.uniform(0, 256);
        int g = rng.uniform(0, 256);
        int b = rng.uniform(0, 256);
        colors.push_back(Scalar(r, g, b));
    }
    Mat old_frame, old_gray;
    vector<Point2f> p0, p1;
    // Take first frame and find corners in it
    capture >> old_frame;
    cvtColor(old_frame, old_gray, COLOR_BGR2GRAY);
    goodFeaturesToTrack(old_gray, p0, 100, 0.3, 7, Mat(), 7, false, 0.04);
    // Create a mask image for drawing purposes
    Mat mask = Mat::zeros(old_frame.size(), old_frame.type());
    while (true)
    {
        Mat frame, frame_gray;
        capture >> frame;
        if (frame.empty())
            break;
        cvtColor(frame, frame_gray, COLOR_BGR2GRAY);
        // calculate optical flow
        vector<uchar> status;
        vector<float> err;
        TermCriteria criteria =
            TermCriteria((TermCriteria::COUNT) + (TermCriteria::EPS), 10, 0.03);
        solution::TimerTracker tracker("LK");
        calcOpticalFlowPyrLK(old_gray, frame_gray, p0, p1, status, err,
                             Size(15, 15), 2, criteria);
        tracker("optical flow");
        vector<Point2f> good_new;
        for (uint i = 0; i < p0.size(); i++)
        {
            // Select good points
            if (status[i] == 1)
            {
                good_new.push_back(p1[i]);
                // draw the tracks
                line(mask, p1[i], p0[i], colors[i], 2);
                circle(frame, p1[i], 5, colors[i], -1);
            }
        }
        Mat img;
        add(frame, mask, img);
        imshow("Frame", img);
        int keyboard = waitKey(30);
        if (keyboard == 'q' || keyboard == 27){
            cv::imwrite("of.png", frame);
            cv::imwrite("of_line.png", img);
            break;
        }
        // Now update the previous frame and previous points
        old_gray = frame_gray.clone();
        p0 = good_new;
    }
    return 0;

    return 0;
}

Compile and run

$ cmake -Bbuild -S . -DCMAKE_BUILD_TYPE=Release
$ cmake --build build

user@phytiumpi:~/Documents/cv/track$ ./build/track_cross carseq.avi
LK, tracked
LK: optical flow elapsed: 5.328 ms.
LK, tracked
LK: optical flow elapsed: 2.135 ms.
LK, tracked
LK: optical flow elapsed: 2.663 ms.
LK, tracked
LK: optical flow elapsed: 1.501 ms.
LK, tracked
LK: optical flow elapsed: 2.044 ms.
LK, tracked
LK: optical flow elapsed: 1.684 ms.

Tracking effect

Optical flow effect

Thesis (In-depth understanding)

1. Zhang Jing, Cai Bogen, Wu Jianping. Research on motion detection technology [J]. Journal of Beijing Jiaotong University, 2003, 27(003):80-83.DOI:10.3969/j.issn.1673-0291.2003.03.019. ↩

2. Zhang Wenxiang, Lu Weijiang. Object moving speed detection technology based on optical flow[J]. Mechanical Engineer, 2012(8):3.DOI:10.3969/j.issn.1002-2333.2012.08.039. ↩

3. Lucas BD, Kanade T. An iterative image registration technique with an application to stereo vision[J]. 1981. ↩

   
   

4. END

   
   
5. Remember to share, like and watch!
6. 
   
7. About CLP Port
   

8. 
   

9. 
   

10. China Electronics Port (stock code: 001287) is the industry's leading comprehensive service platform for electronic component application innovation and modern supply chain. Relying on more than 30 years of industrial upstream and downstream resource accumulation, technology precipitation, and application innovation, it has developed into a comprehensive service platform covering electronic components. A comprehensive service provider of distribution, design chain services, supply chain coordination and industrial data services.
    

11. 
    

12. China Electronics Port adheres to the business philosophy of "serving customers and sharing with partners". While fulfilling its social responsibilities, it strives to build a component supply chain ecosystem to help the development of China's electronic information industry.
    

13. 
    

14. 
    

15. 
    
16. Click below to follow the CLP Port official account
17. Get more industry information
18. 
    

20.