Epoll usage example: TCP server handles multiple client requests

DDZZ669

Epoll usage example: TCP server handles multiple client requests [Copy link]

The previous article: Is it possible to perform socket communication without IP and port number? introduced socket communication in Unix domain and tested TCP and UDP transmission methods through examples. This article, based on the previous example, will learn the multiplexing function of epoll and realize the reception of data from multiple clients by adding epoll listening function to the server.

epoll stands for eventpoll, which is an implementation of IO multiplexing in the Linux kernel. epoll is an upgraded version of select and poll. Compared with these two predecessors, epoll has improved the working method and made it more efficient. This article will not introduce the internal implementation principle of epoll, but first introduce how to use epoll to implement multiplexing.

1 epoll Introduction

1.1 epoll creation

int epoll_create(int size); //Number of listeners

1.2 epoll event settings

int epoll_ctl(int epfd, int op, int fd, struct epoll_event *event)

The first parameter epfd is the return value of epoll_create(), and the second parameter op represents the action, which is represented by three macros:

• EPOLL_CTL_ADD: register new fd to epfd;

• EPOLL_CTL_MOD: Modify the monitoring event of the registered fd;

• EPOLL_CTL_DEL: delete an fd from epfd;

The third parameter is the fd to be monitored, and the fourth parameter tells the kernel what to monitor. The struct epoll_event structure is as follows:

struct epoll_event { 
  __uint32_t events; /* Epoll events */
  epoll_data_t data; /* User data variable */
};

Events can be a collection of the following macros:

• EPOLLIN: Indicates that the corresponding file descriptor can be read (including the normal closure of the peer SOCKET);

• EPOLLOUT: indicates that the corresponding file descriptor can be written;

• EPOLLPRI: Indicates that the corresponding file descriptor has urgent data to read (this should indicate that out-of-band data is arriving);

• EPOLLERR: Indicates that an error occurred in the corresponding file descriptor;

• EPOLLHUP: Indicates that the corresponding file descriptor is hung up;

• EPOLLET: Set EPOLL to edge triggered mode, which is relative to level triggered mode.

• EPOLLONESHOT: Only listen to the event once. After listening to the event, if you need to continue listening to this socket, you need to add this socket to the EPOLL queue again.

1.3 epoll monitoring

int epoll_wait(int epfd, struct epoll_event * events, int maxevents, int timeout)

Waiting for events to be generated is similar to the select() call. The events parameter is used to get the set of events from the kernel, maxevents tells the kernel how big the events are, and the value of maxevents cannot be greater than the size when creating epoll_create(), and the timeout parameter is the timeout (milliseconds, 0 will return immediately, -1 will be uncertain, and some say it is permanently blocked). The function returns the number of events that need to be processed, and if it returns 0, it means the timeout has occurred.

2 Programming Example Test

This test is modified based on the Unix domain socket communication code in the previous article, and only TCP socket communication is used for testing.

In the test code of the previous article, after the server receives a connection from a client, it only serves the client and does not receive processing logic for other clients. What we want to achieve in this article is a server that can receive data from multiple clients.

Before programming, let's take a look at the program structure to be implemented. The yellow part is the part that needs to be added to the previous example.

You only need to modify the server program and add the epoll listening function. The client program does not need to be modified.

2.1 Add epoll monitoring function to the socket server

The TCP server code is modified as follows. The main modification is after listen, creating an epoll, and then adding the server's socketfd to epoll for listening:

• When a new client requests a connection, the server's socketfd will receive an event, and then epoll will receive the EPOLLIN event of the server's socketfd. At this time, the server can accept the client's request and add the created client fd to epoll for monitoring.

• When the client is successfully connected and monitored by epoll, the client sends a message again, and epoll will receive the EPOLLIN event corresponding to the client fd, which allows the server to read the client's message.

#define LISTEN_MAX 5
#define EPOLL_FDSIZE LISTEN_MAX
#define EPOLL_EVENTS 20
#define CLIENT_NUM 3

void EpollAddEvent(int epollfd, int fd, int event)
{
  PRINT("epollfd:%d add fd:%d(event:%d)\n", epollfd, fd, event);
  struct epoll_event ev;
  ev.events = event;
  ev.data.fd = fd;
  epoll_ctl(epollfd, EPOLL_CTL_ADD, fd, &ev);
}

void TcpServerThread()
{
    //------------socket
    int sockfd = socket(AF_UNIX, SOCK_STREAM, 0);
    if (sockfd < 0)
    {
        PRINT("create socket fail\n");
        return;
    }
    PRINT("create socketfd:%d\n", sockfd);

    struct sockaddr_un addr;
    memset (&addr, 0, sizeof(addr));
    addr.sun_family = AF_UNIX;
    strcpy(addr.sun_path, UNIX_TCP_SOCKET_ADDR);

    //------------bind
    if (bind(sockfd, (struct sockaddr *)&addr, sizeof(addr)))
    {
        PRINT("bind fail\n");
        return;
    }
    PRINT("bind ok\n");

    //------------listen
    if (listen(sockfd, LISTEN_MAX))
    {
        PRINT("listen fail\n");
        return;
    }
    PRINT("listen ok\n");

    //------------epoll---------------
    int epollfd = epoll_create(EPOLL_FDSIZE);
    if (epollfd < 0)
    {
        PRINT("epoll create fail\n");
        return;
    }
    PRINT("epoll create fd:%d\n", epollfd);

    EpollAddEvent(epollfd, sockfd, EPOLLIN);

    struct epoll_event events[EPOLL_EVENTS];
    while(1)
    {
        PRINT("epoll wait...\n");
        int num = epoll_wait(epollfd, events, EPOLL_EVENTS, -1);
        PRINT("epoll wait done, num:%d\n", num);
        for (int i = 0;i < num;i++)
        {
            int fd = events.data.fd;
            if (EPOLLIN == events.events)
            {
                //Accept the client's connection request
                if (fd == sockfd)
                {
                    //------------accept
                    int clientfd = accept(sockfd, NULL, NULL);
                    if (clientfd == -1)
                    {
                        PRINT("accpet error\n");
                    }
                    else
                    {
                        PRINT("======> accept new clientfd:%d\n", clientfd);
                        
                        EpollAddEvent(epollfd, clientfd, EPOLLIN);
                    }
                }
                //Read the data sent by the client
                else
                {
                    char buf[BUF_SIZE] = {0};
                    //------------recv
                    size_t size = recv(fd, buf, BUF_SIZE, 0);
                    //size = read(clientfd, buf, BUF_SIZE);
                    if (size > 0)
                    {
                        PRINT("recv from clientfd:%d, msg:%s\n", fd, buf);
                    }
                }
            }
        }
    }

    PRINT("end\n");
}

2.2 Start multiple clients for testing

Modify the main program, create multiple client threads, generate multiple clients, and connect to the same server to test the function of epoll to monitor multiple events.

int main()
{
    unlink(UNIX_TCP_SOCKET_ADDR);

    //Create a server
    thread thServer(TcpServerThread);

    //Create multiple clients
    thread thClinet[CLIENT_NUM];
    for (int i=0; i<CLIENT_NUM; i++)
    {
        thClinet = thread(TcpClientThread);
        sleep(1);
    }

    while(1)
    {
        sleep(5);
    }
}

In this example, CLIENT_NUM is 3, and three clients are used to test the epoll function.

2.3 Test Results

Compile and run on Ubuntu, and the program prints as follows:

[TcpServerThread] create socketfd:3
[TcpServerThread] bind ok
[TcpClientThread] create socketfd:4
[TcpServerThread] listen ok
[TcpServerThread] epoll create fd:5
[EpollAddEvent] epollfd:5 add fd:3(event:1)
[TcpServerThread] epoll wait...
[TcpClientThread] create socketfd:6
[TcpClientThread] connect ok
[TcpServerThread] epoll wait done, num:1
[TcpServerThread] =====> accept new clientfd:7
[EpollAddEvent] epollfd:5 add fd:7(event:1)
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:1
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)1
[TcpServerThread] epoll wait...
[TcpClientThread] create socketfd:8
[TcpClientThread] connect ok
[TcpServerThread] epoll wait done, num:2
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)2
[TcpServerThread] =====> accept new clientfd:9
[EpollAddEvent] epollfd:5 add fd:9(event:1)
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:1
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)3
[TcpServerThread] epoll wait...
[TcpClientThread] connect ok
[TcpServerThread] epoll wait done, num:3
[TcpServerThread] =====> accept new clientfd:10
[EpollAddEvent] epollfd:5 add fd:10(event:1)
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)5
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)6
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:1
[TcpServerThread] recv from clientfd:10, msg:helloTCP(fd:8)4
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:3
[TcpServerThread] recv from clientfd:10, msg:helloTCP(fd:8)7
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)8
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)9
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:1
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)10
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:2
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)12
[TcpServerThread] recv from clientfd:10, msg:helloTCP(fd:8)11
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:3
[TcpServerThread] recv from clientfd:10, msg:helloTCP(fd:8)14
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)13
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)15
[TcpServerThread] epoll wait...
[TcpServerThread] epoll wait done, num:3
[TcpServerThread] recv from clientfd:10, msg:helloTCP(fd:8)16
[TcpServerThread] recv from clientfd:7, msg:helloTCP(fd:4)17
[TcpServerThread] recv from clientfd:9, msg:helloTCP(fd:6)18
[TcpServerThread] epoll wait...

It is easier to understand the program running process by marking the results:

It can be seen that the server accepts the connection requests from three clients in turn, and then can receive data sent by the three clients.

3 Conclusion

This article introduces the use of epoll function in Linux software development. By adding epoll function to TCP server, one server can handle multiple clients.

freebsder

Thanks for sharing, very detailed!

lugl4313820

I have read your work carefully, it is really easy to understand, thank you for sharing.