Preface

Some time ago, we discussed in the group how to upload a certain file to the server at a scheduled time. I remember doing a project before: : Implementing a module that can remotely download and upgrade image packages to achieve version upgrade functions. At that time, a super powerful tool cURL was used. On a whim, I decided to write a special article for friends in need. 为高通的iot模组编写FOTA功能

The article is mainly divided into two parts:

1. First, introduce the syntax of cURL;

2. Implement a simple program to upload log files at a scheduled time.

concept

cURL is a commonly used command line tool used to request web servers. Its name means client URL tool.

The original author of cURL is Daniel Stenberg (currently the core developer of cURL), who is also a senior member of the IETF HTTPbis working group. Daniel created the curl project in 1998. He wrote the initial version of curl and created the libcurl library. So far, more than half of the 24,000 commits included in the code repository were submitted by Daniel himself, and he is still the core developer of the project. Daniel said that he has regarded curl as his own child.

As a powerful tool, it supports the following protocols: (DICT, FILE, FTP, FTPS, GOPHER, HTTP, HTTPS, IMAP, IMAPS, LDAP, LDAPS, POP3, POP3S, RTMP, RTSP, SCP, SFTP, SMTP, SMTPS, TELNET and TFTP), as well as POST, cookies, authentication, downloading part of a file from a specified offset, etc. It has features such as user agent string, speed limit, file size, progress bar, cookie support, user authentication, and breakpoint resume.

1. Command installation

sudo apt-get install curl

2. cURL command syntax:

curl [options] [URL...]

3. URL Format

On the WWW, every information resource has a unified and unique address on the Internet. This address is called URL (Uniform Resource Locator). It is the uniform resource locator of the WWW, which refers to the network address. The format definition of URL can be found in RFC 1808.

地址：http://www.w3.org/Addressing/rfc1808.txt
《Relative Uniform Resource Locators 》 

A URL consists of three parts: the resource type, the host domain name where the resource is stored, and the resource file name. It can also be considered to consist of four parts: The general grammatical format of a URL is: 协议、主机、端口、路径

protocol :// hostname[:port] / path / [;parameters][?query]#fragment

(Items in square brackets [] are optional).

protocol

Specifies the transport protocol to use. The following table lists the valid scheme names for the protocol attribute. The most commonly used is the HTTP protocol, which is also the most widely used protocol in the WWW.

• The file resource is a file on the local computer. The format is file:///. Note that there should be three slash marks at the end.
• ftp Access resources via FTP. Format FTP://
• gopher Access the resource via the Gopher protocol.
• http Access the resource via HTTP. The format is HTTP://
• https Access this resource via secure HTTPS. Format: https://
• The mailto resource is an email address, accessed via SMTP. The format is mailto:
• MMS plays this resource by supporting the MMS (streaming media) protocol. (Representative software: Windows Media Player) Format MMS://
• ed2k Access this resource through P2P software that supports the ed2k (dedicated download link) protocol. (Representative software: eMule) Format ed2k://
• Flashget Access this resource through P2P software that supports Flashget: (dedicated download link) protocol. (Representative software: Express) Format Flashget://
• thunder Access this resource through P2P software that supports the thunder (dedicated download link) protocol. (Representative software: Thunder) Format thunder://
• news Access this resource via NNTP.

hostname

It refers to the Domain Name System (DNS) host name or IP address of the server where the resource is stored. Sometimes, the host name can also include the username and password required to connect to the server (format: username:password@hostname).

port

Integer, optional. If omitted, the default port of the scheme is used. Various transport protocols have default port numbers, such as the default port of http is 80. If omitted when input, the default port number is used. Sometimes for security or other reasons, the port can be redefined on the server, that is, a non-standard port number is used. In this case, the port number cannot be omitted in the URL.

path

A string separated by zero or more "/" symbols, generally used to represent a directory or file address on the host.

parameters

This is optional and is used to specify special parameters.

query

Optional. Used to pass parameters to dynamic web pages (such as web pages created using CGI, ISAPI, PHP/JSP/ASP/ASP.NET, etc.). There can be multiple parameters, separated by the "&" symbol, and the name and value of each parameter are separated by the "=" symbol.

fragment

A string used to specify a fragment in a network resource. For example, if there are multiple definitions of terms on a web page, you can use fragment to directly locate a specific definition of a term.

4. Detailed explanation of curl command parameters:

Since Linux curl is very powerful, it has many command parameters. The following table only lists some commonly used parameters. For more parameters, please run the "man curl" command to view them.

In the following options, (H) means only for HTTP/HTTPS, (F) means only for FTP

5. Linux cURL command exit code:

The following are the error codes of the Linux curl command and their corresponding error messages. When the command execution fails, you can use the error code to view the cause of the error, which is convenient for development and debugging.

6. Usage demonstration:

To save space, the execution results of some operations are no longer attached.

1. View the webpage source code

Simply add the URL after the curl command to see the source code of the web page. Let's take the URL www.sina.com as an example (we chose this URL mainly because its web page code is shorter):

root@ubuntu:/home/peng# curl www.sohu.com
<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

The execution result shows 307 Temporary Redirect that the URL needs to be redirected.

If you want to save this webpage, you can use -o the parameter, which is equivalent to using the wget command.

curl -o [文件名] www.sohu.com

2. Automatic jump

Some URLs are automatically redirected. Use -L the parameter and curl will redirect to the new URL.

curl -L www.sohu.com

Type the above command and the result will automatically jump to www.sohu.com.cn.

3. Display header information

-i The parameter can display the header information of the http response, together with the web page code.

root@ubuntu:/home/peng/driver/test#  curl -i www.sohu.com
HTTP/1.1 307 Temporary Redirect
Content-Type: text/html
Content-Length: 180
Connection: keep-alive
Server: nginx
Date: Tue, 25 Aug 2020 10:10:54 GMT
Location: https://www.sohu.com/
FSS-Cache: from 9790436.18244590.10468709
FSS-Proxy: Powered by 2384755.3433341.3062915

<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

-I The parameter only displays the header information of the http response.

4. Display communication process

-v The parameter can display the entire process of an http communication, including port connection and http request header information.

root@ubuntu:/home/peng/driver/test#  curl -v www.sohu.com
* About to connect() to www.sohu.com port 80 (#0)
*   Trying 240e:83:201:3700::5... connected
> GET / HTTP/1.1
> User-Agent: curl/7.22.0 (i686-pc-linux-gnu) libcurl/7.22.0 OpenSSL/1.0.1 zlib/1.2.3.4 libidn/1.23 librtmp/2.3
> Host: www.sohu.com
> Accept: */*
> 
< HTTP/1.1 307 Temporary Redirect
< Content-Type: text/html
< Content-Length: 180
< Connection: keep-alive
< Server: nginx
< Date: Tue, 25 Aug 2020 10:11:49 GMT
< Location: https://www.sohu.com/
< FSS-Cache: from 9855973.18375663.10534247
< FSS-Proxy: Powered by 2450292.3564414.3128453
< 
<html>
<head><title>307 Temporary Redirect</title></head>
<body bgcolor="white">
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>
* Connection #0 to host www.sohu.com left intact
* Closing connection #0

If you feel that the above information is not enough, the following command can view a more detailed communication process.

curl --trace output.txt www.sohu.com

or

curl --trace-ascii output.txt www.sohu.com

After running, please open the output.txt file to view it.

5. Send form information

There are two methods for sending form information: GET and POST. The GET method is relatively simple, just append the data to the URL.

curl example.com/form.cgi?data=xxx

The POST method must separate the data and the URL, so curl needs to use the --data parameter.

curl -X POST --data "data=xxx" example.com/form.cgi

If your data is not form-encoded, you can also let curl encode it for you with the --data-urlencode .

curl -X POST--data-urlencode "date=April 1" example.com/form.cgi

6. HTTP Verbs

The default HTTP verb for curl is GET, and -X other verbs can be supported using the parameter.

curl -X POST www.example.com
curl -X DELETE www.example.com

7. File upload

Assume the file upload form is as follows:

<form method="POST" enctype='multipart/form-data' action="upload.cgi">
 <input type=file name=upload>
 <input type=submit name=press value="OK">
</form>

You can upload files using curl like this:

curl --form upload=@localfilename --form press=OK [URL]

8. Referer field

Sometimes you need to provide a referer field in the http request header to indicate where you are redirected from.

curl --referer http://www.example.com http://www.example.com

9. User Agent Field

This field is used to indicate the client's device information. The server sometimes returns web pages in different formats for different devices based on this field, such as mobile and desktop versions.

The User Agent of iPhone 4 is

Mozilla/5.0 (iPhone; U; CPU iPhone OS 4_0 like Mac OS X; en-us) AppleWebKit/532.9 (KHTML, like Gecko) Version/4.0.5 Mobile/8A293 Safari/6531.22.7

curl can simulate this:

curl --user-agent "[User Agent]" [URL]

10. Cookies

Using --cookie the parameter, you can have curl send cookies.

curl --cookie "name=xxx" www.example.com

As for the specific cookie value, it can Set-Cookie be obtained from the field of the http response header information.

-c cookie-file You can save the cookie returned by the server to a file, and -b cookie-file use this file as cookie information for subsequent requests.

curl -c cookies http://example.com
curl -b cookies http://example.com

11. Add header information

Sometimes you need to add a header to the http request. --header Parameters can do this.

$ curl --header "Content-Type:application/json" http://example.com

12. Certification

Use curl option -u to complete HTTP or FTP authentication. You can specify a password or not and enter the password in subsequent operations:

curl -u user:pwd http://man.linuxde.net
curl -u user http://man.linuxde.net

13. FTP

1) List the directories on the ftp server

curl ftp://www.xxx.com/ --user name:passwd
curl ftp://www.xxx.com/ –u name:passwd #简洁写法
curl ftp://name:passwd@www.xxx.com #简洁写法2

For example: Set up an FTP server on the IP address 192.168.43.117, and set the username and password to . user 123456

Now we want to display all file information in the root directory on the server. The command is as follows:

curl -u user:123456 ftp://192.168.43.117

The execution results are as follows: Concise writing:

curl ftp://user:123456@192.168.43.117

The execution results are as follows:

2) Only list the directory without displaying the progress bar

curl ftp://www.xxx.com –u name:passwd -s

3) Download a file:

The command is as follows:

curl ftp://www.xxx.com/size.zip –u name:passwd -o size.zip

The example is as follows: Download the file test.c from the root directory of the server and save it locally. The local file name is also test.c. [Note] If there is no -o option, the program will direct the data stream to stdout, that is, directly display the file content on the terminal.

curl ftp://user:123456@192.168.43.117/test.c -o test.c

The execution results are as follows: Simple mode

curl -u user:123456 ftp://192.168.43.117/list.h -o list.h

The execution results are as follows:

4) Upload a file:

curl –u name:passwd -T size.mp3 ftp://www.xxx.com/mp3/

Here are some examples:

curl -u user:123456 ftp://192.168.43.117/ -T list.h

You can see that the file was not uploaded successfully and the error code returned is 25. Please refer to Chapter 5

25 FTP couldn't STOR file. The server denied the STOR operation, used for FTP uploading.

It can be seen that this is because the server does not grant storage permissions, so just set the server's write permissions.

5) Delete files from the server (use curl to pass the DELE command of the FTP protocol):

curl –u name:passwd ftp://www.xxx.com/ -X 'DELE mp3/size.mp3'

6) In addition, curl does not support recursive downloading, but you can use arrays to download files. For example, if we want to download files named 1-10.gif consecutively: **

curl –u name:passwd ftp://www.xxx.com/img/[1-10].gif –O #O字母大写

7) To download multiple files continuously:

curl –u name:passwd ftp://www.xxx.com/img/[one,two,three].jpg –O #O字母大写

6. Implement scheduled upload of log files

Function

1. When the program is running, the last modification time of the current log file should be recorded;
2. Check every 10 seconds whether the log file has been modified. If it has not been modified, sleep for 10 seconds.
3. If the log file is modified, the current log file is copied to a backup file, and the backup file name is added with the current time.
4. Send the backup file to the ftp server via curl;
5. Delete the backup file and repeat step 2.

The program flow chart is as follows:

Function introduction

init()

First, record the current log file time and save it last_mtime in a global variable.

check_file_change()

Read the last modification time of the file and last_mtime compare it with . If they are the same, return 0, otherwise return 1.

file_name_add_time()

Copy the current log file to a backup file, and add the current time to the backup file name.

stat()

Get the attribute information of the corresponding file and store it in the struct stat structure variable.

system()

Execute the command corresponding to the string in the parameter.

The code is as follows:

/*   Copyright (C)   公众号: yikoulinux  */
#include <sys/stat.h>
#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <time.h>
 
typedef struct stat ST;
unsigned long last_mtime;

/*用户名密码暂时写死，实际应该保存在配置文件*/
char name[32]="user";
char pass[32] ="123456";
char ip[32]     ="192.168.43.117";
char filename[32]="t.log";
char dstfile[256]  ={0};

int init(void)
{
 //准备结构体
 ST status;
 
 //调用stat函数
 int res = stat(filename,&status);
 if(-1 == res)
 {
  perror("error:open file fail\n");
  return 0;
 }
 last_mtime = status.st_mtime;
 printf("init time:%s \n",ctime(&last_mtime));
 return 1;
}
 
int  check_file_change(void)
{
 //准备结构体
 ST status;
 
 //调用stat函数
 int res = stat(filename,&status);
 if(-1 == res)
 {
  perror("error:open file fail\n");
  return 0;
 }
// printf("old:%s new:%s",ctime(&last_mtime),ctime(&status.st_mtime));
 if(last_mtime == status.st_mtime)
 {
  printf("file not change\n");
  return 0;
 }else{
  printf("file updated\n"); 
  last_mtime = status.st_mtime;
  return 1;
 }

}
void file_name_add_time(void)
{
 ST status;
 time_t t;  
 struct tm *tblock; 
 char cmd[1024]={0};
  
 t = time(NULL);
 tblock = localtime(&t);
 
 sprintf(dstfile,"t-%d-%d-%d-%d-%d-%d.log",
  tblock->tm_year+1900,
  tblock->tm_mon,
  tblock->tm_mday,
  tblock->tm_hour,
  tblock->tm_min,
  tblock->tm_sec);
 sprintf(cmd,"cp %s %s",filename,dstfile);
// printf("cdm=%s\n",cmd);
 system(cmd);
}
int main(void)
{

 char cmd[1024]={0};

 init();
 while(1)
 { 
  if(check_file_change() == 1)
  {
   file_name_add_time();
   sprintf(cmd,"curl -u %s:%s ftp://%s/ -T %s",name,pass,ip,dstfile);
 //  printf("cdm=%s\n",cmd);
   system(cmd);
   unlink(dstfile);
  }
  sleep(10); 
 }
}

Run screenshot:

Step 1: Because the log file has not been modified, the program will not upload.

Step 2: Manually input the string yikoulinux into the log file t.log. Step 3: Because the file has changed, it prints "file updated", and you can also see the log information of the file uploaded by curl. The following is the root directory of the FTP server, and you can see the uploaded log file: t-2020-7-26-1-19-45.log .

【Replenish】

1. Configuration information is hard-coded in the code and should usually be read from the configuration file. For the convenience of readers, this code does not add this function.
2. FTP server setup is not described in this article. There are many related documents. You can search for them yourself. I use File Zilla.
3. Usually, such a program that needs to run for a long time needs to be set as a daemon process. This article does not add the corresponding function, and readers can search for it on their own. If there is a strong demand, a separate article can be opened for detailed introduction.