Linux 系统负载查询及分析

  • A+
所属分类:Linux
高性能企业级服务器首台5折

本文先对云服务器 ECS Linux 的总体负载情况的查询和分析进行简要说明。然后分别从 CPU、IO 和网络等多个维度分别进行负载的分析说明。

Linux 系统总体负载的查询及分析

云服务器 ECS Linux 系统如果总体负载过高,可能会引发死机或卡顿等异常。可以参阅如下步骤进行整体排查:

  1. 检查服务器进程与服务否占用了过多内存,或者内存没有正常释放,导致出现内存溢出,系统宕机。
  2. 检查 /var/spool/cron 等系统配置中是否有 cron(计划任务、自动任务)在对应时间段内执行。
  3. 检查 Web 服务器的参数是否超过了服务器的性能。比如最大连接数过高等。
  4. 检查进程数是否非常高,导致服务瘫痪,机器假死。
  5. 查看系统日志中是否有异常记录。
  6. 检查磁盘是否有坏块。
  7. 内核消耗过大,查看是否有瞬间资源占用过大的进程或服务。
  8. 查看是否有异常进程,是否存在被攻击或入侵症状。

使用 sar 查看 Linux 系统各项资源使用情况

sar 是System Activity Reporter(系统活动情况报告)的缩写。sar 工具对系统状态进行取样后,=通过计算数据和比例来表达系统当前的运行状态。其特点是可以连续对系统取样,获得大量的取样数据。其取样数据和分析结果可以存入文件,所需的负载很小。

sar 是 Linux 系统中较为全面的性能分析工具,可以从多个方面对系统的活动进行监控和报告,包括文件读写情况、系统调用使用情况、串口、CPU 效率、内存使用情况、进程活动及 IPC 有关的活动情况等等。

  • 安装
    使用 yum install sysstat 进行安装,然后使用 /etc/init.d/sysstat start 启动服务。
  • 查看 CPU 负载
    用法:

    1. 1
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">u </span><span class="lit">1</span> <span class="lit">5</span>

    示例输出:

    1. 1
      2
      3
      4
      5
      6
      7
      8
      9
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">u </span><span class="lit">1</span> <span class="lit">5</span>
      <span class="typ">Linux</span> <span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.9</span><span class="pun">.</span><span class="lit">3.el7.x86</span><em><span class="lit">64</span> <span class="pun">(</span><span class="pln">iZ23pddtofdZ</span><span class="pun">)</span>     <span class="lit">07</span><span class="pun">/</span><span class="lit">04</span><span class="pun">/</span><span class="lit">2016</span><span class="pln">     _x86_64</span></em>    <span class="pun">(</span><span class="lit">1</span><span class="pln"> CPU</span><span class="pun">)</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">35</span><span class="pln"> AM     CPU     </span><span class="pun">%</span><span class="pln">user     </span><span class="pun">%</span><span class="pln">nice   </span><span class="pun">%</span><span class="pln">system   </span><span class="pun">%</span><span class="pln">iowait    </span><span class="pun">%</span><span class="pln">steal     </span><span class="pun">%</span><span class="pln">idle</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">36</span><span class="pln"> AM     all     </span><span class="lit">14.14</span>      <span class="lit">0.00</span>      <span class="lit">1.01</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>     <span class="lit">84.85</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">37</span><span class="pln"> AM     all     </span><span class="lit">14.14</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">1.01</span>      <span class="lit">0.00</span>     <span class="lit">84.85</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">38</span><span class="pln"> AM     all      </span><span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">1.01</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>     <span class="lit">98.99</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">39</span><span class="pln"> AM     all      </span><span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>    <span class="lit">100.00</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">16</span><span class="pun">:</span><span class="lit">40</span><span class="pln"> AM     all      </span><span class="lit">1.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>      <span class="lit">0.00</span>     <span class="lit">99.00</span>
      <span class="typ">Average</span><span class="pun">:</span><span class="pln">        all      </span><span class="lit">5.86</span>      <span class="lit">0.00</span>      <span class="lit">0.40</span>      <span class="lit">0.20</span>      <span class="lit">0.00</span>     <span class="lit">93.54</span>

    回显说明:

    • %user:用户模式下消耗的 CPU 时间的比例。
    • %nice:通过 nice 改变了进程调度优先级的进程,在用户模式下消耗的 CPU 时间的比例。
    • %system:系统模式下消耗的 CPU 时间的比例。
    • %iowait:CPU 等待磁盘 I/O 导致空闲状态消耗的时间比例。
    • %steal:利用 Xen 等操作系统虚拟化技术,等待其它虚拟 CPU 计算占用的时间比例。
    • %idle:CPU 空闲时间比例。

 

  • 查看平均负载
    用法:

    1. 1
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">q </span><span class="lit">1</span> <span class="lit">60</span>

    示例输出:

    1. 1
      2
      3
      4
      5
      6
      7
      8
      9
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">q </span><span class="lit">1</span> <span class="lit">6Linux</span> <span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.9</span><span class="pun">.</span><span class="lit">3.el7.x86</span><em><span class="lit">64</span> <span class="pun">(</span><span class="pln">iZ23pddtofdZ</span><span class="pun">)</span>     <span class="lit">07</span><span class="pun">/</span><span class="lit">04</span><span class="pun">/</span><span class="lit">2016</span><span class="pln">     _x86_64</span></em>    <span class="pun">(</span><span class="lit">1</span><span class="pln"> CPU</span><span class="pun">)</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">13</span><span class="pln"> AM   runq</span><span class="pun">-</span><span class="pln">sz  plist</span><span class="pun">-</span><span class="pln">sz   ldavg</span><span class="pun">-</span><span class="lit">1</span><span class="pln">   ldavg</span><span class="pun">-</span><span class="lit">5</span><span class="pln">  ldavg</span><span class="pun">-</span><span class="lit">15</span><span class="pln">   blocked</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">14</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">15</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">16</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">17</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">18</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">23</span><span class="pun">:</span><span class="lit">19</span><span class="pln"> AM         </span><span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>
      <span class="typ">Average</span><span class="pun">:</span>            <span class="lit">0</span>       <span class="lit">142</span>      <span class="lit">0.00</span>      <span class="lit">0.01</span>      <span class="lit">0.05</span>         <span class="lit">0</span>

    回显说明:
    指定 -q 后,可以查看运行队列中的进程数、系统上的进程大小、平均负载等信息。与其它命令相比,能查看各项指标随时间变化的情况。

    • runq-sz:运行队列的长度(等待运行的进程数)。
    • plist-sz:进程列表中进程(processes)和线程(threads)的数量。
    • ldavg-1:最后 1 分钟的系统平均负载。
    • ldavg-5:过去 5 分钟的系统平均负载。
    • ldavg-15:过去 15 分钟的系统平均负载。

 

  • 查看内存负载
    用法:

    1. 1
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">r </span><span class="lit">1</span> <span class="lit">3</span>

    示例输出:

    1. 1
      2
      3
      4
      5
      6
      7
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">r </span><span class="lit">1</span> <span class="lit">3</span>
      <span class="typ">Linux</span> <span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.9</span><span class="pun">.</span><span class="lit">3.el7.x86</span><em><span class="lit">64</span> <span class="pun">(</span><span class="pln">iZ23pddtofdZ</span><span class="pun">)</span>     <span class="lit">07</span><span class="pun">/</span><span class="lit">04</span><span class="pun">/</span><span class="lit">2016</span><span class="pln">     _x86_64</span></em>    <span class="pun">(</span><span class="lit">1</span><span class="pln"> CPU</span><span class="pun">)</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">27</span><span class="pun">:</span><span class="lit">34</span><span class="pln"> AM kbmemfree kbmemused  </span><span class="pun">%</span><span class="pln">memused kbbuffers  kbcached  kbcommit   </span><span class="pun">%</span><span class="pln">commit  kbactive   kbinact   kbdirty</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">27</span><span class="pun">:</span><span class="lit">35</span><span class="pln"> AM    </span><span class="lit">275992</span>    <span class="lit">740664</span>     <span class="lit">72.85</span>    <span class="lit">181552</span>    <span class="lit">315340</span>    <span class="lit">362052</span>     <span class="lit">35.61</span>    <span class="lit">471216</span>    <span class="lit">115828</span>        <span class="lit">60</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">27</span><span class="pun">:</span><span class="lit">36</span><span class="pln"> AM    </span><span class="lit">276024</span>    <span class="lit">740632</span>     <span class="lit">72.85</span>    <span class="lit">181552</span>    <span class="lit">315340</span>    <span class="lit">362052</span>     <span class="lit">35.61</span>    <span class="lit">471220</span>    <span class="lit">115828</span>        <span class="lit">64</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">27</span><span class="pun">:</span><span class="lit">37</span><span class="pln"> AM    </span><span class="lit">276024</span>    <span class="lit">740632</span>     <span class="lit">72.85</span>    <span class="lit">181552</span>    <span class="lit">315340</span>    <span class="lit">362052</span>     <span class="lit">35.61</span>    <span class="lit">471220</span>    <span class="lit">115828</span>        <span class="lit">64</span>
      <span class="typ">Average</span><span class="pun">:</span>       <span class="lit">276013</span>    <span class="lit">740643</span>     <span class="lit">72.85</span>    <span class="lit">181552</span>    <span class="lit">315340</span>    <span class="lit">362052</span>     <span class="lit">35.61</span>    <span class="lit">471219</span>    <span class="lit">115828</span>        <span class="lit">63</span>

    回显说明:

    • kbmemfree:该值和 free 命令中的 free 值基本一致,它不包括 buffer 和 cache 空间
    • kbmemused:该值和 free 命令中的 used 值基本一致,它包括 buffer 和 cache 空间。
    • %memused:物理内存使用率,该值是 kbmemused 和内存总量(不包括 swap)的百分比。
    • kbbuffers 和 kbcached:这两个值和 free 命令中的 buffer 和 cache 一致。
    • kbcommit:保证当前系统所需要的内存,即为了确保不溢出而需要的内存(RAM+swap)。
    • %commit:该值是 kbcommit 与内存总量(包括swap)的百分比。

 

  • 查看页面交换发生状况
    用法:

    1. 1
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">W </span><span class="lit">1</span> <span class="lit">3</span>

    示例输出:

    1. 1
      2
      3
      4
      5
      6
      7
      <span class="pln">sar </span><span class="pun">-</span><span class="pln">W </span><span class="lit">1</span> <span class="lit">3</span>
      <span class="typ">Linux</span> <span class="lit">3.10</span><span class="pun">.</span><span class="lit">0</span><span class="pun">-</span><span class="lit">123.9</span><span class="pun">.</span><span class="lit">3.el7.x86</span><em><span class="lit">64</span> <span class="pun">(</span><span class="pln">iZ23pddtofdZ</span><span class="pun">)</span>     <span class="lit">07</span><span class="pun">/</span><span class="lit">04</span><span class="pun">/</span><span class="lit">2016</span><span class="pln">     _x86_64</span></em>    <span class="pun">(</span><span class="lit">1</span><span class="pln"> CPU</span><span class="pun">)</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">28</span><span class="pun">:</span><span class="lit">59</span><span class="pln"> AM  pswpin</span><span class="pun">/</span><span class="pln">s pswpout</span><span class="pun">/</span><span class="pln">s</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">29</span><span class="pun">:</span><span class="lit">00</span><span class="pln"> AM      </span><span class="lit">0.00</span>      <span class="lit">0.00</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">29</span><span class="pun">:</span><span class="lit">01</span><span class="pln"> AM      </span><span class="lit">0.00</span>      <span class="lit">0.00</span>
      <span class="lit">10</span><span class="pun">:</span><span class="lit">29</span><span class="pun">:</span><span class="lit">02</span><span class="pln"> AM      </span><span class="lit">0.00</span>      <span class="lit">0.00</span>
      <span class="typ">Average</span><span class="pun">:</span>         <span class="lit">0.00</span>      <span class="lit">0.00</span>

    回显说明:

    • pswpin/s:每秒系统换入的交换页面(swap page)数量。
    • pswpout/s:每秒系统换出的交换页面(swap page)数量。

 

  • 其它常见 sar 参数说明(区分大小写)-A 汇总所有的报告
    -a 报告文件读写使用情况
    -B 报告附加的缓存的使用情况
    -b 报告缓存的使用情况
    -c 报告系统调用的使用情况
    -d 报告磁盘的使用情况
    -g 报告串口的使用情况
    -h 报告关于buffer使用的统计数据
    -m 报告IPC消息队列和信号量的使用情况
    -n 报告命名cache的使用情况
    -p 报告调页活动的使用情况
    -q 报告运行队列和交换队列的平均长度
    -R 报告进程的活动情况
    -r 报告没有使用的内存页面和硬盘块
    -u 报告CPU的利用率
    -v 报告进程、i节点、文件和锁表状态
    -w 报告系统交换活动状况
    -y 报告TTY设备活动状况

 

使用 htop 查看系统负载情况

htop 是 Linux 系统中的一个互动进程查看器,可以让用户进行交互式操作,可横向或纵向滚动浏览进程列表,支持鼠标操作。用户可以在安装 htop 来监控服务器的负载。

Linux系统默认不带有 htop 工具,需要手动安装通过 yum install htop 等方法安装后才能使用。安装方法本文不再详述。

安装成功后,可以在命令行输入 htop 来启动 htop 监控工具。htop 启动后的界面如下图所示:

blob.png

返回结果左侧显示 CPU、内存、交换区 swap 的使用情况,右侧显示任务、负载、开机时间,下面的主体部分就是进程实时状况,底端是 F1-F10 功能键。

相关快捷功能键的说明如下:

功能键 对应功能 说明
F1 Invoke htop Help 查看htop帮助说明
F2 Htop Setup Menu htop 配置菜单
F3 Search for a Process 搜索进程
F4 Incremental process filtering 进程过滤器
F5 Tree View 显示树形结构
F6 Sort by a column 选择排序方式
F7 Nice - (change priority) 可减少nice值,用于提高对应进程的优先级
F8 Nice + (change priority) 可增加nice值,用于降低对应进程的优先级
F9 Kill a Process 对进程传递信号
F10 Quit htop 结束htop

在 htop 界面,用户可以通过鼠标点击相关进程、列、功能键,也可以通过上下键或 PgUP、PgDn  键选定想要的进程,左右键或 Home、 End 键移动字段,常用的快捷键如下:

  • Space    标记/取消标记一个或多个进程。
  • s    选择某一进程,按 s 后,用 strace 追踪进程的系统调用。
  • l    显示进程打开的文件。如果安装了 lsof,按此键可以显示进程所打开的文件。
  •    按 Memory 使用排序。
  • P    按 CPU 使用排序。
  •    按 Time+ 使用排序。
  • F    跟踪进程: 如果排序顺序引起选定的进程在列表上随意移动,让选定条跟随该进程。这对监视一个进程非常有用。通过这种方式,用户可以让特定进程在屏幕上一直可见。使用方向键会停止该功能。
  • K    显示/隐藏内核线程。
  •    显示/隐藏用户线程。
  • Ctrl-L    刷新。

鼠标点击 Help 或者按 F1 ,可以显示自带帮助:

blob.png

鼠标点击 Setup 或者按下 F2 可以进入 htop 配置页面。例如最后一项的设定是调整 Columns(数据列)的显示,用于自定义 htop 进程列表中可以看到哪些字段的数据及信息。

blob.png

鼠标点击 Search 或者按下 F3  或者输入 “/“, 可以通过输入进程名进行搜索,例如搜索 ssh 进程

blob.png

输入 “t” 或按下 F5,显示树形结构,与 pstree 显示效果类似,可以看到所有程序树状执行的结构。

blob.png

按下 F6 可以选择依照哪一列来排序,最常用的排序内容就是 CPU 和 Memory。

blob.png

 

CPU 负载的查询分析与常见案例

CPU 负载的查询分析

使用 vmstat 查看系统纬度的 CPU 负载

可以通过 vmstat 从系统维度查看 CPU 资源的使用情况。

用法说明:

  1. 1
    2
    3
    4
    5
    6
    7
    8
    <span class="pun">格式:</span><span class="pln">vmstat </span><span class="pun">-</span><span class="pln">n </span><span class="lit">1</span><span class="com"><span class="hljs-comment"># -n 1 表示结果一秒刷新一次。示例输出:$ vmstat -n 1 </span></span>
    <span class="pln"><span class="hljs-comment">procs </span></span><span class="pun"><span class="hljs-comment">—————-</span></span><span class="pln"><span class="hljs-comment">memory</span></span><span class="pun"><span class="hljs-comment">—————</span></span> <span class="pun"><span class="hljs-comment">—-</span></span><span class="pln"><span class="hljs-comment">swap</span></span><span class="pun"><span class="hljs-comment">—</span></span> <span class="pun"><span class="hljs-comment">——-</span></span><span class="pln"><span class="hljs-comment">io</span></span><span class="pun"><span class="hljs-comment">——</span></span> <span class="pun"><span class="hljs-comment">-</span></span><span class="pln"><span class="hljs-comment">system</span></span><span class="pun"><span class="hljs-comment">—</span></span> <span class="pun"><span class="hljs-comment">———</span></span><span class="pln"><span class="hljs-comment">cpu</span></span><span class="pun"><span class="hljs-comment">——-</span></span>
    <span class="pln"><span class="hljs-comment"> r  b   swpd   free   buff  cache   si   so    bi    bo   </span></span><span class="kwd"><span class="hljs-comment">in</span></span><span class="pln"><span class="hljs-comment">   cs us sy id wa st</span></span>
    <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>      <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">70352</span></span> <span class="lit"><span class="hljs-comment">169448</span></span> <span class="lit"><span class="hljs-comment">448452</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">4</span></span>   <span class="lit"><span class="hljs-comment">10</span></span>   <span class="lit"><span class="hljs-comment">11</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span> <span class="lit"><span class="hljs-comment">99</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>      <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">70376</span></span> <span class="lit"><span class="hljs-comment">169448</span></span> <span class="lit"><span class="hljs-comment">448484</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">175</span></span>  <span class="lit"><span class="hljs-comment">406</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span> <span class="lit"><span class="hljs-comment">100</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>      <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">70376</span></span> <span class="lit"><span class="hljs-comment">169448</span></span> <span class="lit"><span class="hljs-comment">448484</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">173</span></span>  <span class="lit"><span class="hljs-comment">414</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">1</span></span> <span class="lit"><span class="hljs-comment">99</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>      <span class="lit"><span class="hljs-comment">0</span></span>  <span class="lit"><span class="hljs-comment">70376</span></span> <span class="lit"><span class="hljs-comment">169448</span></span> <span class="lit"><span class="hljs-comment">448484</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>    <span class="lit"><span class="hljs-comment">0</span></span>     <span class="lit"><span class="hljs-comment">0</span></span>   <span class="lit"><span class="hljs-comment">128</span></span>  <span class="lit"><span class="hljs-comment">212</span></span>  <span class="lit"><span class="hljs-comment">429</span></span>  <span class="lit"><span class="hljs-comment">3</span></span>  <span class="lit"><span class="hljs-comment">0</span></span> <span class="lit"><span class="hljs-comment">96</span></span>  <span class="lit"><span class="hljs-comment">1</span></span>  <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="pun"><span class="hljs-comment">^</span></span><span class="pln"><span class="hljs-comment">C</span></span>

回显说明:

返回结果中的主要数据列说明:

  • r: 表示系统中 CPU 等待处理的线程。由于 CPU 每次只能处理一个线程,所以,该数值越大,通常表示系统运行越慢。
  • us:用户模式消耗的 CPU 时间百分比。该值较高时,说明用户进程消耗的 CPU 时间比较多,比如,如果该值长期超过 50%,则需要对程序算法或代码等进行优化。
  • sy:内核模式消耗的 CPU 时间百分比。
  • wa:IO 等待消耗的 CPU 时间百分比。该值较高时,说明 IO 等待比较严重,这可能磁盘大量作随机访问造成的,也可能是磁盘性能出现了瓶颈。
  • id:处于空闲状态的 CPU 时间百分比。如果该值持续为 0,同时 sy 是 us 的两倍,则通常说明系统则面临着 CPU 资源的短缺。

使用 top 查看进程纬度的 CPU 负载

可以通过 top 从进程纬度来查看其 CPU、内存等资源的使用情况。

用法说明:

  1. 1
    2
    3
    4
    5
    6
    7
    8
    9
    <span class="pun">格式:</span><span class="pln">top</span><span class="pun">示例输出:</span>
    <span class="pln">top </span><span class="pun">-</span> <span class="lit">17</span><span class="pun">:</span><span class="lit">27</span><span class="pun">:</span><span class="lit">13</span><span class="pln"> up </span><span class="lit">27</span><span class="pln"> days</span><span class="pun">,</span>  <span class="lit">3</span><span class="pun">:</span><span class="lit">13</span><span class="pun">,</span>  <span class="lit">1</span><span class="pln"> user</span><span class="pun">,</span><span class="pln">  load average</span><span class="pun">:</span> <span class="lit">0.02</span><span class="pun">,</span> <span class="lit">0.03</span><span class="pun">,</span> <span class="lit">0.05</span>
    <span class="typ">Tasks</span><span class="pun">:</span>  <span class="lit">94</span><span class="pln"> total</span><span class="pun">,</span>   <span class="lit">1</span><span class="pln"> running</span><span class="pun">,</span>  <span class="lit">93</span><span class="pln"> sleeping</span><span class="pun">,</span>   <span class="lit">0</span><span class="pln"> stopped</span><span class="pun">,</span>   <span class="lit">0</span><span class="pln"> zombie</span>
    <span class="pun">%</span><span class="typ">Cpu</span><span class="pun">(</span><span class="pln">s</span><span class="pun">):</span>  <span class="lit">0.3</span><span class="pln"> us</span><span class="pun">,</span>  <span class="lit">0.1</span><span class="pln"> sy</span><span class="pun">,</span>  <span class="lit">0.0</span><span class="pln"> ni</span><span class="pun">,</span> <span class="lit">99.5</span><span class="pln"> id</span><span class="pun">,</span>  <span class="lit">0.0</span><span class="pln"> wa</span><span class="pun">,</span>  <span class="lit">0.0</span><span class="pln"> hi</span><span class="pun">,</span>  <span class="lit">0.0</span><span class="pln"> si</span><span class="pun">,</span>  <span class="lit">0.1</span><span class="pln"> st</span>
    <span class="typ">KiB</span> <span class="typ">Mem</span><span class="pun">:</span>   <span class="lit">1016656</span><span class="pln"> total</span><span class="pun">,</span>   <span class="lit">946628</span><span class="pln"> used</span><span class="pun">,</span>    <span class="lit">70028</span><span class="pln"> free</span><span class="pun">,</span>   <span class="lit">169536</span><span class="pln"> buffers</span>
    <span class="typ">KiB</span> <span class="typ">Swap</span><span class="pun">:</span>        <span class="lit">0</span><span class="pln"> total</span><span class="pun">,</span>        <span class="lit">0</span><span class="pln"> used</span><span class="pun">,</span>        <span class="lit">0</span><span class="pln"> free</span><span class="pun">.</span>   <span class="lit">448644</span><span class="pln"> cached </span><span class="typ">Mem</span>
    <span class="pln">  PID USER      PR  NI    VIRT    RES    SHR S </span><span class="pun">%</span><span class="pln">CPU </span><span class="pun">%</span><span class="pln">MEM     TIME</span><span class="pun">+</span><span class="pln"> COMMAND  </span>
    <span class="lit">1</span><span class="pln"> root      </span><span class="lit">20</span>   <span class="lit">0</span>   <span class="lit">41412</span>   <span class="lit">3824</span>   <span class="lit">2308</span><span class="pln"> S  </span><span class="lit">0.0</span>  <span class="lit">0.4</span>   <span class="lit">0</span><span class="pun">:</span><span class="lit">19.01</span><span class="pln"> systemd  </span>
    <span class="lit">2</span><span class="pln"> root      </span><span class="lit">20</span>   <span class="lit">0</span>       <span class="lit">0</span>      <span class="lit">0</span>      <span class="lit">0</span><span class="pln"> S  </span><span class="lit">0.0</span>  <span class="lit">0.0</span>   <span class="lit">0</span><span class="pun">:</span><span class="lit">00.04</span><span class="pln"> kthreadd </span>

回显说明:

默认界面上第三行会显示当前 CPU 资源的总体使用情况,下方会显示各个进程的资源占用情况。

可以直接在界面输入大小字母 P,来使监控结果按 CPU 使用率倒序排列,进而定位系统中占用 CPU 较高的进程。最后,根据系统日志和程序自身相关日志,对相应进程做进一步排查分析,以判断其占用过高 CPU 的原因。

CPU 负载异常的常见案例

使用 top 直接终止 CPU 消耗较大的进程

如前面所述,可以通过 top 命令查看系统的负载问题,并定位耗用较多 CPU 资源的进程。

可以直接在 top 运行界面快速终止相应的异常进程。说明如下:

  1. 想要终止某个进程,只需按下小写的 k 键。
  2. 输入想要终止的进程 PID (top 输出结果的第一列)。比如,如下图所示,假如想要终止 PID 为 23 的进程,输入 23 后按回车。
  3. 如下图所示,操作成功后,界面会出现类似 “Send pid 23 signal [15/sigterm]” 的提示信息让用户进行确认。按回车确认即可。

CPU 使用率较低但负载较高

  • 问题描述:
    Linux 系统没有业务程序运行,通过 top 观察,类似如下图所示,CPU 很空闲,但是 load average 却非常高:
  •  处理办法
    load average 是对 CPU 负载的评估,其值越高,说明其任务队列越长,处于等待执行的任务越多。
    出现此种情况时,可能是由于僵死进程导致的。可以通过指令 ps -axjf  查看是否存在 D 状态进程。
    D 状态是指不可中断的睡眠状态。该状态的进程无法被 kill,也无法自行退出。只能通过恢复其依赖的资源或者重启系统来解决。

 

kswapd0 进程占用 CPU 较高

操作系统都用分页机制来管理物理内存,操作系统将磁盘的一部分划出来作为虚拟内存,由于内存的速度要比磁盘快得多,所以操作系统要按照某种换页机制将不需要的页面换到磁盘中,将需要的页面调到内存中,由于内存持续不足,这个换页动作持续进行,kswapd0是虚拟内存管理中负责换页的,当服务器内存不足的时候kswapd0会执行换页操作,这个换页操作是十分消耗主机CPU资源的。如果通过top发现该进程持续处于非睡眠状态,且运行时间较长,可以初步判定系统在持续的进行换页操作,可以将问题转向内存不足的原因来排查。

  • 问题描述:
    kswapd0 进程占用了系统大量 CPU 资源。
  • 处理办法:
    Linux 系统通过分页机制管理内存的同时,将磁盘的一部分划出来作为虚拟内存。而 kswapd0 是 Linux 系统虚拟内存管理中负责换页的进程。当系统内存不足时,kswapd0 会频繁的进行换页操作。而由于换页操作非常消耗 CPU 资源,所以会导致该进程持续占用较高 CPU 资源。
    如果通过 top 等监控发现 kswapd0 进程持续处于非睡眠状态,且运行时间较长并持续占用较高 CPU 资源,则通常是由于系统在持续的进行换页操作所致。则可以通过 free 、ps 等指令进一步查询系统及系统内进程的内存占用情况,做进一步排查分析。

IO 负载的查询与分析与操作案例

IO 负载的查询与分析

使用 iostat 从系统纬度查看磁盘 IO 负载

可以通过 iostat 从系统维度查看 IO 负载情况。

iostat 并非常见 Linux 发行版本自带工具,其包含在 sysstat 软件包中,需要先通过 yum 或 apt-get 等方式进行安装后才能使用。 具体安装方法本文不再详述。

用法说明:

  1. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    <span class="pun">示例用法:</span>
    <span class="pln">iostat </span><span class="pun">-</span><span class="pln">d </span><span class="pun">-</span><span class="pln">k </span><span class="lit">2</span>
    <span class="com"><span class="hljs-comment"># -d 表示,显示设备(磁盘)使用状态。</span></span>
    <span class="com"><span class="hljs-comment"># -k 表示让某些使用 block 为单位的列强制使用 kB 为单位。</span></span>
    <span class="com"><span class="hljs-comment"># 2表示,数据显示每隔2秒刷新一次。</span></span>
    <span class="pun"><span class="hljs-comment">示例输出:</span></span>
    <span class="typ"><span class="hljs-comment">Linux</span></span> <span class="lit"><span class="hljs-comment">3.10</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">0</span></span><span class="pun"><span class="hljs-comment">-</span></span><span class="lit"><span class="hljs-comment">123.9</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">3.el7.x86</span></span><em><span class="lit"><span class="hljs-comment">64</span></span> <span class="pun"><span class="hljs-comment">(</span></span><span class="pln"><span class="hljs-comment">centos</span></span><span class="pun"><span class="hljs-comment">)</span></span>     <span class="lit"><span class="hljs-comment">06</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">27</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">2016</span></span><span class="pln"><span class="hljs-comment">     _x86_64</span></span></em>    <span class="pun"><span class="hljs-comment">(</span></span><span class="lit"><span class="hljs-comment">1</span></span><span class="pln"><span class="hljs-comment"> CPU</span></span><span class="pun"><span class="hljs-comment">)</span></span>
    <span class="typ"><span class="hljs-comment">Device</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">            tps    kB_read</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    kB_wrtn</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    kB_read    kB_wrtn</span></span>
    <span class="pln"><span class="hljs-comment">xvda              </span></span><span class="lit"><span class="hljs-comment">0.58</span></span>         <span class="lit"><span class="hljs-comment">0.12</span></span>         <span class="lit"><span class="hljs-comment">3.75</span></span>     <span class="lit"><span class="hljs-comment">278001</span></span>    <span class="lit"><span class="hljs-comment">8820028</span></span>
    <span class="pln"><span class="hljs-comment">xvdb              </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>        <span class="lit"><span class="hljs-comment">740</span></span>          <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="pln"><span class="hljs-comment">xvdc              </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>       <span class="lit"><span class="hljs-comment">1388</span></span>          <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="pln"><span class="hljs-comment">xvde              </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>       <span class="lit"><span class="hljs-comment">2035</span></span>          <span class="lit"><span class="hljs-comment">0</span></span>
    <span class="pln"><span class="hljs-comment">xvdf              </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>         <span class="lit"><span class="hljs-comment">0.00</span></span>        <span class="lit"><span class="hljs-comment">740</span></span>          <span class="lit"><span class="hljs-comment">0</span></span>

更多参数及返回结果说明,可以参阅 iostat 的 man 帮助。

使用 iotop 从进程纬度查看磁盘 IO 负载

也可以通过 iotop 从进程维度对系统内进程的 IO 使用情况进行排序。

iotop 也并非常见 Linux 发行版本自带工具,也需要先通过 yum 或 apt-get 等方式进行安装后才能使用。 具体安装方法本文不再详述。

用法说明:

直接使用指令 iotop 运行后,可以看到类似以下界面:

回显说明:

  • 默认情况下按照 IO 使用量倒序排序,可以用左右箭头操作排序的字段。
  • 按 r 切换排序方式。
  • 按 o 只显示有磁盘 IO 活动的进程。
  • 更多参数及返回结果说明,可以参阅 iotop 的 man 帮助。

IO 负载相关操作案例

kjournald 进程占用 IO 资源高问题

  • 问题描述:
    使用 iotop 排查分析,发现 kjournald 进程占用了大量 IO 资源。
  • 处理办法:
    kjournald 进程是 ext3 文件系统进行 IO 数据操作的内核进程,?它会在向磁盘内写入和读取数据时占用 CPU 和 内存资源。
    该问题通常是由于 ext3 文件系统循环的写数据,导致 Journal size 不断增大,进而占满导致。
    可以通过 dumpe2fs 指令查看相应分区的 Journal size 配置情况,然后尝试调大该值,看问题是否缓解。查询方法如下所示:

    1. 1
      2
      3
      4
      5
      6
      7
      8
      9
      <span class="pln">$ dumpe2fs </span><span class="pun">/</span><span class="pln">dev</span><span class="pun">/</span><span class="pln">xvda1 </span><span class="pun">|</span><span class="pln"> grep </span><span class="typ">Journal</span>
      <span class="pln">dumpe2fs </span><span class="lit">1.42</span><span class="pun">.</span><span class="lit">9</span> <span class="pun">(</span><span class="lit">28</span><span class="pun">-</span><span class="typ">Dec</span><span class="pun">-</span><span class="lit">2013</span><span class="pun">)</span>
      <span class="typ">Journal</span><span class="pln"> inode</span><span class="pun">:</span>            <span class="lit">8</span>
      <span class="typ">Journal</span><span class="pln"> backup</span><span class="pun">:</span><span class="pln">           inode blocks</span>
      <span class="typ">Journal</span><span class="pln"> features</span><span class="pun">:</span><span class="pln">         journal_incompat_revoke</span>
      <span class="typ">Journal</span><span class="pln"> size</span><span class="pun">:</span>             <span class="lit">128M</span>
      <span class="typ">Journal</span><span class="pln"> length</span><span class="pun">:</span>           <span class="lit">32768</span>
      <span class="typ">Journal</span><span class="pln"> sequence</span><span class="pun">:</span>         <span class="lit">0x00010ffb</span>
      <span class="typ">Journal</span><span class="pln"> start</span><span class="pun">:</span>            <span class="lit">10953</span>

     

通过 4K 对齐提高 IO 性能

可以通过如下脚本对磁盘进行格式化并自动配置 4K 对齐。

注意:运行此脚本会自动格式化所有数据盘磁盘。如果并非新购磁盘,请在操作前,务必确认已经完成相关磁盘上的数据备份。可以通过快照进行磁盘的备份,操作方法可以参阅创建快照

  1. 下载脚本 auto_fdisk.zip
  2. 解压 auto_fdisk.zip 后,将相应脚本上传到目标服务器。
  3. 通过 chmod +x 等指令,为脚本添加执行权限。
  4. 以root身份,通过如下方式为脚本添加执行权限,然后运行即可:
    1. 1
      2
      <span class="pln">chmod </span><span class="pun">+</span><span class="pln">x </span><span class="pun">./</span><span class="pln">auto_fdisk</span><span class="pun">.</span><span class="pln">sh</span>
      <span class="pun">./</span><span class="pln">auto_fdisk</span><span class="pun">.</span><span class="pln">sh</span>

网络负载的查询及分析

sar 的使用

sar 可以从网络接口层面来分析数据包的收发情况、错误信息等。

使用 sar 来监控网络流量的常用命令为:

  1. 1
    2
    <span class="pln">sar </span><span class="pun">-</span><span class="pln">n DEV </span><span class="pun">[</span><span class="pln">interval</span><span class="pun">]</span> <span class="pun">[</span><span class="pln">count</span><span class="pun">]</span>
    <span class="pun">参数</span><span class="pln"> interval </span><span class="pun">是统计间隔,</span><span class="pln">count </span><span class="pun">是统计次数。</span>

示例:

使用以下命令,可以使用 sar 每两秒统计一次网络接口的活动状况,连续报告 3 次:

  1. 1
    2
    3
    4
    5
    6
    7
    8
    9
    10
    11
    12
    13
    14
    15
    16
    <span class="pln">sar </span><span class="pun">-</span><span class="pln">n DEV </span><span class="lit">2</span> <span class="lit">2</span><span class="com"><span class="hljs-comment"># 输出结果包括按字节数/包数统计的,当前/平均网络接口利用率。</span></span>
    <span class="typ"><span class="hljs-comment">Linux</span></span> <span class="lit"><span class="hljs-comment">2.6</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">32</span></span><span class="pun"><span class="hljs-comment">-</span></span><span class="lit"><span class="hljs-comment">573.12</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">1.el6.x86</span></span><em><span class="lit"><span class="hljs-comment">64</span></span> <span class="pun"><span class="hljs-comment">(</span></span><span class="pln"><span class="hljs-comment">centos6</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">6</span></span><span class="pun"><span class="hljs-comment">)</span></span>     <span class="lit"><span class="hljs-comment">06</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">23</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">2016</span></span><span class="pln"><span class="hljs-comment">     _x86_64</span></span></em>    <span class="pun"><span class="hljs-comment">(</span></span><span class="lit"><span class="hljs-comment">8</span></span><span class="pln"><span class="hljs-comment"> CPU</span></span><span class="pun"><span class="hljs-comment">)</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">34</span></span><span class="pln"><span class="hljs-comment"> AM     IFACE   rxpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    rxkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    txkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   rxcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s  rxmcst</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">36</span></span><span class="pln"><span class="hljs-comment"> AM        lo      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">36</span></span><span class="pln"><span class="hljs-comment"> AM      eth0      </span></span><span class="lit"><span class="hljs-comment">2.50</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.15</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">36</span></span><span class="pln"><span class="hljs-comment"> AM   docker0      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">36</span></span><span class="pln"><span class="hljs-comment"> AM     IFACE   rxpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    rxkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    txkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   rxcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s  rxmcst</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">38</span></span><span class="pln"><span class="hljs-comment"> AM        lo      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">38</span></span><span class="pln"><span class="hljs-comment"> AM      eth0      </span></span><span class="lit"><span class="hljs-comment">1.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.06</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">38</span></span><span class="pln"><span class="hljs-comment"> AM      eth1      </span></span><span class="lit"><span class="hljs-comment">2.00</span></span>      <span class="lit"><span class="hljs-comment">2.00</span></span>      <span class="lit"><span class="hljs-comment">0.17</span></span>      <span class="lit"><span class="hljs-comment">0.40</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="lit"><span class="hljs-comment">09</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">58</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">38</span></span><span class="pln"><span class="hljs-comment"> AM   docker0      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="typ"><span class="hljs-comment">Average</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">        IFACE   rxpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txpck</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    rxkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s    txkB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   rxcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s   txcmp</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s  rxmcst</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">s</span></span>
    <span class="typ"><span class="hljs-comment">Average</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">           lo      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="typ"><span class="hljs-comment">Average</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">         eth0      </span></span><span class="lit"><span class="hljs-comment">1.75</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.10</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="typ"><span class="hljs-comment">Average</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">         eth1      </span></span><span class="lit"><span class="hljs-comment">2.25</span></span>      <span class="lit"><span class="hljs-comment">1.75</span></span>      <span class="lit"><span class="hljs-comment">0.18</span></span>      <span class="lit"><span class="hljs-comment">0.29</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>
    <span class="typ"><span class="hljs-comment">Average</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment">      docker0      </span></span><span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>      <span class="lit"><span class="hljs-comment">0.00</span></span>

回显结果主要字段说明:

  •  IFACE:网络接口名称
  • rxpck/s、txpck/s:每秒收/发的数据包数量
  • rxkB/s、txkB/s:每秒收/发的字节数,以kB/s为单位
  • rxcmp/s、txcmp/s:每秒收/发的压缩过的数据包数量
  • rxmcst/s:每秒收到的多播数据包

iftop 的使用

iftop 命令常见用法如下:

  1. 1
    2
    <span class="pln">iftop </span><span class="pun">[-</span><span class="pln">i interface</span><span class="pun">]</span>
    <span class="pun">参数</span> <span class="pun">-</span><span class="pln">i </span><span class="pun">后跟的</span><span class="pln"> interface </span><span class="pun">表示网络接口名,比如</span><span class="pln"> eth0</span><span class="pun">、</span><span class="pln">eth1 </span><span class="pun">等等。如果不通过</span> <span class="pun">-</span><span class="pln">i </span><span class="pun">参数指定接口名,则默认检测第一块网卡的使用情况,对于</span><span class="pln"> ECS</span><span class="pun">,通常为内网网卡</span><span class="pln"> eth0</span><span class="pun">。</span>

示例:

使用 iftop -i eth1 指令可以查看 ECS 服务器公网网卡的带宽使用情况。示例输出如下:iftop 示意图

回显结果说明:

  • 第一行:带宽使用情况显示。
  • 中间部分为外部连接列表,即记录了哪些 IP 正在和本机的网络连接。
  • 中间部分靠右侧部分是实时流量信息,分别是该访问 IP 连接到本机 2 秒、10 秒和 40 秒的平均流量。
  • => 代表发送数据,<= 代表接收数据 。
  • 底部三行:
    • 第一列:TX 表示发送流量,RX 表示接收流量,TOTAL 表示总流量。
    • 第二列 cum:表示第一列各种情况的总流量。
    • 第三列 peak:表示第一列各种情况的流量峰值。
    • 第四列 rates:表示第一列各种情况 2 秒、10 秒、40 秒内的平均流量。

注意:iftop 的流量显示单位是Mb,这里的 b 是比特(bit),不是字节(byte)。而 ifstat 显示的单位是 KB 中的 B 是字节。1 byte = 8 bit。

另外,进入 iftop 界面后,可以通过按下相应的字母快捷按键,来对显示结果进行调整。常见操作命令如下(区分大小写):

  • h 切换是否显示帮助。
  • n 切换显示本机的 IP 或主机名。
  • s 切换是否显示本机的 host 信息。
  • d 切换是否显示远端目标主机的 host 信息。
  • t 切换显示格式为 2 行 /1 行 / 只显示发送流量 / 只显示接收流量。
  • N 切换显示端口号或端口服务名称。
  • S 切换是否显示本机的端口信息。
  • D 切换是否显示远端目标主机的端口信息。
  • p 切换是否显示端口信息。
  • P 切换暂停/继续显示。
  • b 切换是否显示平均流量图形条。
  • B 切换计算2秒或10秒或40秒内的平均流量。
  • T 切换是否显示每个连接的总流量。
  • l 打开屏幕过滤功能,输入要过滤的字符。比如输入相应 IP 地址,回车后,屏幕就只显示这个 IP 相关的流量信息。
  • L 切换显示画面上边的刻度;刻度不同,流量图形条会有变化。
  • j 或按 k 向上或向下滚动屏幕显示的连接记录。
  • 1 或 2 或 3 根据右侧显示的三列流量数据进行排序。
  • < 根据左边的本机名或 IP 排序。
  • > 根据远端目标主机的主机名或 IP 排序。
  • o 切换是否固定只显示当前的连接。
  • f 编辑过滤代码。
  • ! 调用 shell 命令。
  • q 退出。

nethogs 的使用

Nethogs 是一款开源的网络流量监控工具,可用于显示每个进程的带宽占用情况。这样可以更直观定位异常流量的来源。Nethogs 支持 IPv4 和 IPv6协议,支持本地网卡及 PPP 连接。

Nethogs工具的安装方法请参阅其官方帮助文档,本文不再详述。

安装完毕,直接输入 nethogs 启动工具即可。不带任何参数时,nethogs 默认监控 eth0。用户可以通过 ifconfig 等指令核实具体哪个网络接口(比如 eth1、eth0)对应公网网卡。

示例输出:

  1. 1
    2
    3
    4
    5
    6
    7
    8
    <span class="pln">nethogs eth1</span>
    <span class="com"><span class="hljs-comment"># 输出结果如下:</span></span>
    <span class="typ"><span class="hljs-comment">NetHogs</span></span><span class="pln"><span class="hljs-comment"> version </span></span><span class="lit"><span class="hljs-comment">0.8</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">0</span></span><span class="pln"><span class="hljs-comment">  PID USER     PROGRAM                                 DEV        SENT      RECEIVED   </span></span>
    <span class="lit"><span class="hljs-comment">9951</span></span><span class="pln"><span class="hljs-comment">  root     wget                                    eth1      </span></span><span class="lit"><span class="hljs-comment">7.253</span></span>     <span class="lit"><span class="hljs-comment">322.344</span></span><span class="pln"><span class="hljs-comment"> KB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">sec</span></span>
    <span class="lit"><span class="hljs-comment">9670</span></span><span class="pln"><span class="hljs-comment">  root     sshd</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="pln"><span class="hljs-comment"> root@pts</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">0</span></span><span class="pun"><span class="hljs-comment">,</span></span><span class="pln"><span class="hljs-comment">pts</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="lit"><span class="hljs-comment">1</span></span><span class="pln"><span class="hljs-comment">                  eth1      </span></span><span class="lit"><span class="hljs-comment">1.328</span></span>       <span class="lit"><span class="hljs-comment">0.199</span></span><span class="pln"><span class="hljs-comment"> KB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">sec</span></span>
    <span class="pun"><span class="hljs-comment">?</span></span><span class="pln"><span class="hljs-comment">     root     </span></span><span class="pun"><span class="hljs-comment">..</span></span><span class="lit"><span class="hljs-comment">16.2</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">226</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">33300</span></span><span class="pun"><span class="hljs-comment">-</span></span><span class="lit"><span class="hljs-comment">122.224</span></span><span class="pun"><span class="hljs-comment">.</span></span><span class="lit"><span class="hljs-comment">153.106</span></span><span class="pun"><span class="hljs-comment">:</span></span><span class="lit"><span class="hljs-comment">48167</span></span>             <span class="lit"><span class="hljs-comment">0.000</span></span>       <span class="lit"><span class="hljs-comment">0.000</span></span><span class="pln"><span class="hljs-comment"> KB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">sec</span></span>
    <span class="pun"><span class="hljs-comment">?</span></span><span class="pln"><span class="hljs-comment">     root     unknown TCP                                        </span></span><span class="lit"><span class="hljs-comment">0.000</span></span>       <span class="lit"><span class="hljs-comment">0.000</span></span><span class="pln"><span class="hljs-comment"> KB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">sec</span></span>
    <span class="pln"><span class="hljs-comment">  TOTAL                                                           </span></span><span class="lit"><span class="hljs-comment">8.581</span></span>     <span class="lit"><span class="hljs-comment">322.543</span></span><span class="pln"><span class="hljs-comment"> KB</span></span><span class="pun"><span class="hljs-comment">/</span></span><span class="pln"><span class="hljs-comment">sec </span></span>

回显结果说明:

  • PID 列表示相应流量关联程序的进程号。
  • USER 列表示相应进程的所属用户。
  • PROGRAM 列表示程序的具体执行路径。
  • DEV 列当前监控的网络接口名称。
  • Sent 列表示相应进程已经发送的数据流量。
  • Received 列表示程序已经接收的数据流量。

在 nethogs 监控界面,按下 s 可以按 Sent 列进行排序,按下 r 可以按 Received 列进行排序,按下 m 可以切换不同的统计单位显示 (kb/s, kb, b,mb)。

nethogs 默认的监控间隔是 1 秒,用户可以通过 -d 参数来设定监控间隔。例如设定监控间隔为5秒,可以输入:

  1. 1
    <span class="pln">nethogs </span><span class="pun">-</span><span class="pln">d </span><span class="lit">5</span>

发表评论

:?: :razz: :sad: :evil: :!: :smile: :oops: :grin: :eek: :shock: :???: :cool: :lol: :mad: :twisted: :roll: :wink: :idea: :arrow: :neutral: :cry: :mrgreen: