How to check which process is occupying and writing data in a file in Linux?

Latest update time：2024-07-11

Reads：


Author : jiaxin_12
Link: https://www.cnblogs.com/YangJiaXin/p/10531197.html

Table of contents

background
step
Get the process ID of the file being written

The file is used by that process, and the data written can be found by lsof, but the reality is that lsof can't find it T_T

background

CentOS 7 monitors and warns that disk usage is 99% for a period of time. Since monitoring is summary information and there is no snapshot information monitoring (which can find the I/O and CPU consumption of a process), it is necessary to periodically execute statistical commands on the server to obtain snapshot information.
You need to use iostat -dx -k to view avgqu-sz, await, svctm, and %util;
sar -u to view %iowait and %user;
pidstat -d to view the snapshot information of process I/O read and write

step

Generate statistics file

cat>/tmp/at_task.sh<<EOF
pidstat -d 2 >/tmp/pidstat_\`date +%F_%T\`.log 2>& 1 &
sar -u 2  >/tmp/sar_\`date +%F_%T\`.log 2>& 1 &
while [ 1 ];do echo -n \`date +%T\` >>/tmp/iostat_\`date +%F\` 2>& 1  && iostat -dx -k 1 1 >>/tmp/iostat_\`date +%F\` 2>& 1; sleep 2; done &
EOF

The reason for using iostat in a while loop is to output date +%T the time. Otherwise, it is useless if there is only data but no time information.

Use the at command to execute at a fixed time

at 15:14 today -f /tmp/at_task.sh

An error occurred

Can't open /var/run/atd.pid to signal atd. No atd running?

Restart the atd service

service atd restart

Restart the AT scheduled task

at 15:14 today -f /tmp/at_task.sh
job 2 at Wed Mar 13 15:14:00 2019

Get the following snapshot information
iostat

15:13:35Linux 3.10.0-862.14.4.el7.x86_64 (ip-xxxxx)     03/13/2019      _x86_64_        (4 CPU)

Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await r_await w_await  svctm  %util
vda               0.12     0.07   17.31   19.41   580.79    90.52    36.57     0.09    2.39    4.42    0.57   0.72   2.63
scd0              0.00     0.00    0.00    0.00     0.00     0.00     6.00     0.00    0.28    0.28    0.00   0.25   0.00

sar

03:14:00 PM     CPU     %user     %nice   %system   %iowait    %steal     %idle
03:14:02 PM     all      0.25      0.00      0.38      0.00      0.00     99.37
03:14:04 PM     all      1.25      0.13      0.63      0.00      0.00     97.99
03:14:06 PM     all      0.25      0.13      0.50      0.00      0.00     99.12
03:14:08 PM     all      0.50      0.00      0.50      0.63      0.00     98.37

pidstat

03:14:00 PM   UID       PID   kB_rd/s   kB_wr/s kB_ccwr/s  Command
03:14:02 PM  5700      9089      0.00      6.00      0.00  uxxx
03:14:02 PM  5700      9140      0.00      6.00      0.00  uxxx
03:14:02 PM  5700      9292      0.00     10.00      0.00  uxxx
03:14:02 PM     0     18084      0.00      2.00      0.00  bash

Kill the command to collect information

ps -ef | egrep 'iostat|sar|pidstat|while' | grep -v grep | awk '{print $2}' | xargs -l kill

However, the ps -ef | egrep command does not obtain the pid of the while loop. If the while loop is not killed, data will continue to be written to /tmp/iostat_2019-03-13-_-

No process that opens the file is located by lsof

lsof /tmp/iostat_2019-03-13 
[root@ip-10-186-60-117 ~]#
[root@ip-10-186-60-117 ~]#

Through lsof, you can locate the process that opens mysql-error.log

lsof /opt/mysql/data/5690/mysql-error.log 
COMMAND   PID                USER   FD   TYPE DEVICE SIZE/OFF     NODE NAME
mysqld  12858 actiontech-universe    1w   REG  253,1     6345 20083533 /opt/mysql/data/5690/mysql-error.log
mysqld  12858 actiontech-universe    2w   REG  253,1     6345 20083533 /opt/mysql/data/5690/mysql-error.log

It can be seen that a process has only one inode holding a file, so lsof can be used to view which processes are using the file.

Get the process ID of the file being written

Install sysemtap

yum -y install systemtap

SystemTap is a tool for monitoring and tracking the Linux kernel.
Use the inodewatch.stp tool in systemtap to find the process number of the file being written.

Get the file's inode

stat -c '%i' /tmp/iostat_2019-03-13 
4210339

Get the major and minor of the device where the file is located

ls -al /dev/vda1
brw-rw---- 1 root disk 253, 1 Jan 30 13:57 /dev/vda1

Get the pid of the file being written

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1  4210339

Checking "/lib/modules/3.10.0-862.14.4.el7.x86_64/build/.config" failed with error: No such file or directory
Incorrect version or missing kernel-devel package, use: yum install kernel-devel-3.10.0-862.14.4.el7.x86_64

Download the corresponding kernel-devel package from the kernel-devel rpm build for: Scientific Linux 7 website according to the system kernel version.

wget ftp://ftp.pbone.net/mirror/ftp.scientificlinux.org/linux/scientific/7.2/x86_64/updates/security/kernel-devel-3.10.0-862.14.4.el7.x86_64.rpm

rpm -ivh kernel-devel-3.10.0-862.14.4.el7.x86_64.rpm

Execute stap again

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339
......
Missing separate debuginfos, use: debuginfo-install kernel-3.10.0-862.14.4.el7.x86_64
Pass 2: analysis failed. [man error::pass2]
Number of similar error messages suppressed: 2.

Install debuginfo kernel

debuginfo-install kernel-3.10.0-862.14.4.el7.x86_64
  Verifying  : kernel-debuginfo-common-x86_64-3.10.0-862.14.4.el7.x86_64                        1/3 
  Verifying  : yum-plugin-auto-update-debug-info-1.1.31-50.el7.noarch                           2/3 
  Verifying  : kernel-debuginfo-3.10.0-862.14.4.el7.x86_64                                      3/3 

Installed:
  kernel-debuginfo.x86_64 0:3.10.0-862.14.4.el7
  yum-plugin-auto-update-debug-info.noarch 0:1.1.31-50.el7

Dependency Installed:
  kernel-debuginfo-common-x86_64.x86_64 0:3.10.0-862.14.4.el7

Execute stap again

stap /usr/share/systemtap/examples/io/inodewatch.stp 253 1  4210339
ERROR: module version mismatch (#1 SMP Tue Sep 25 14:32:52 CDT 2018 vs #1 SMP Wed Sep 26 15:12:11 UTC 2018), release 3.10.0-862.14.4.el7.x86_64
WARNING: /usr/bin/staprun exited with status: 1

添加 -v查看详细报错
stap -v  /usr/share/systemtap/examples/io/inodewatch.stp 253 1  4210339
Pass 1: parsed user script and 471 library scripts using 240276virt/41896res/3368shr/38600data kb, in 300usr/20sys/320real ms.
Pass 2: analyzed script: 2 probes, 12 functions, 8 embeds, 0 globals using 399436virt/196284res/4744shr/197760data kb, in 1540usr/560sys/2106real ms.
Pass 3: using cached /root/.systemtap/cache/f5/stap_f5c0cd780e8a2cac973c9e3ee69fba0c_7030.c
Pass 4: using cached /root/.systemtap/cache/f5/stap_f5c0cd780e8a2cac973c9e3ee69fba0c_7030.ko
Pass 5: starting run.
ERROR: module version mismatch (#1 SMP Tue Sep 25 14:32:52 CDT 2018 vs #1 SMP Wed Sep 26 15:12:11 UTC 2018), release 3.10.0-862.14.4.el7.x86_64
WARNING: /usr/bin/staprun exited with status: 1
Pass 5: run completed in 0usr/10sys/38real ms.
Pass 5: run failed.  [man error::pass5]

Revise

vim /usr/src/kernels/3.10.0-862.14.4.el7.x86_64/include/generated/compile.h

#define UTS_VERSION "#1 SMP Tue Sep 25 14:32:52 CDT 2018"
改为
#define UTS_VERSION "#1 SMP Wed Sep 26 15:12:11 UTC 2018"

rm -rf  /root/.systemtap/cache/f5/stap_f5c0cd780e8a2cac973c9e3ee69fba0c_7030*

Execute again

step /usr/share/systemtap/examples/io/inodewatch.stp 253 1 4210339

iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4671) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4677) vfs_write 0xfd00001/4210339
iostat(4683) vfs_write 0xfd00001/4210339
............

It can be seen that the process ID of the process that writes the /tmp/iostat_ file has been obtained date +%F , but the process ID is always printed out because the background process iostat -dx -m is in a while loop, and it will execute iostat every 2s after sleep to generate a new pid. So how can I stop iostat -dx -m from writing the /tmp/iostat_ file? In addition to restarting Dafa$_$
date +%F

rm -rf also cannot terminate the background while iostat process writing files. After deleting the files, the while loop will generate new files.

rm -rf  /tmp/iostat_2019-03-1*

stat /tmp/iostat_2019-03-1*
  File: ‘/tmp/iostat_2019-03-13’
  Size: 146700    	Blocks: 512        IO Block: 4096   regular file
Device: fd01h/64769d	Inode: 4210339     Links: 1
Access: (0644/-rw-r--r--)  Uid: (    0/    root)   Gid: (    0/    root)
Access: 2019-03-14 16:07:26.211888899 +0800
Modify: 2019-03-14 16:18:17.854019793 +0800
Change: 2019-03-14 16:18:17.854019793 +0800

The right approach

cat>/tmp/iostat.sh<<EOF
while [ 1 ];do echo -n \`date +%T\` >>/tmp/iostat_\`date +%F\` 2>& 1  && iostat -dx -m 1 1 >>/tmp/iostat_\`date +%F\` 2>& 1; sleep 2; done &
EOF

at  now + 1 minute  today
bash /tmp/iostat.sh

#这样就能方便的获取到进程号pid了
ps -ef | grep iostat
root      8593     1  0 16:16 pts/2    00:00:00 bash /tmp/iostat.sh

autumn The recruitment has already begun. If you are not well prepared, autumn It is difficult to find a good job.

Here is a big employment gift package for everyone. You can prepare for the spring recruitment and find a good job!

Latest articles about

■Master Docker image skills: create efficient and lightweight containers

■Improving website response speed and reliability: Nginx load balancing best practices

■The Linux version of WeChat has been officially announced - developed based on Qt, starts in seconds, and provides a smooth experience, beating QQ next door?

■HTTP will add a new method type!

■Break the bottleneck! Practical methods to comprehensively improve MySQL performance

■These are some ridiculously difficult programming languages. If you learn them, I lose.

■Explain in detail why the process needs to sleep?

■Arrange files neatly! A very detailed tutorial on how to sort file contents using the Sort command in Linux

■The customer's computer broke down, so he took out the hard drive and connected it to another computer. He never expected that he did not have access permissions to access the folder. What should he do?

■Comprehensively master the Top command: common techniques and methods for system performance monitoring

How to check which process is occupying and writing data in a file in Linux?

Author : jiaxin_12Link: https://www.cnblogs.com/YangJiaXin/p/10531197.html

Latest articles about

Author : jiaxin_12
Link: https://www.cnblogs.com/YangJiaXin/p/10531197.html