第1章　緒論....................................................................................................................1
1.1　系統(tǒng)性能 .............................................................................................................. 1
1.2　人員 ...................................................................................................................... 2
1.3　活動 ...................................................................................................................... 3
1.4　視角 ...................................................................................................................... 4
1.5　性能工程是充滿挑戰(zhàn)的 ...................................................................................... 5
1.5.1　主觀性 ...................................................................................................... 5
1.5.2　復(fù)雜性 ...................................................................................................... 5
1.5.3　多個原因 .................................................................................................. 6
1.5.4　多個性能問題 .......................................................................................... 6
1.6　延時 ...................................................................................................................... 6
1.7　可觀測性 .............................................................................................................. 7
1.7.1　計數(shù)器、統(tǒng)計數(shù)據(jù)和指標(biāo) ...................................................................... 8
1.7.2　剖析 ........................................................................................................ 10
1.7.3　跟蹤 ........................................................................................................ 11
1.8　實驗 .................................................................................................................... 13
1.9　云計算 ................................................................................................................ 14
1.10　方法 .................................................................................................................. 14
1.10.1 Linux 性能分析 60 秒 .......................................................................... 15
1.11　案例研究........................................................................................................... 15
1.11.1　緩慢的磁盤 .......................................................................................... 16
1.11.2　軟件變更 .............................................................................................. 18
XXXIV 目錄
1.11.3　更多閱讀 .............................................................................................. 19
1.12　參考資料 .......................................................................................................... 19
第2章　方法..................................................................................................................21
2.1　術(shù)語 .................................................................................................................... 22
2.2　模型 .................................................................................................................... 23
2.2.1　受測系統(tǒng) ................................................................................................ 23
2.2.2　排隊系統(tǒng) ................................................................................................ 23
2.3　概念 .................................................................................................................... 24
2.3.1　延時 ........................................................................................................ 24
2.3.2　時間量級 ................................................................................................ 25
2.3.3　權(quán)衡 ........................................................................................................ 26
2.3.4　調(diào)優(yōu)的影響 ............................................................................................ 27
2.3.5　合適的層級 ............................................................................................ 28
2.3.6　何時停止分析 ........................................................................................ 28
2.3.7　性能推薦的時間點 ................................................................................ 29
2.3.8　負載與架構(gòu) ............................................................................................ 30
2.3.9　擴展性 .................................................................................................... 30
2.3.10　指標(biāo) ...................................................................................................... 32
2.3.11　使用率 .................................................................................................. 33
2.3.12　飽和度 .................................................................................................. 34
2.3.13　剖析 ...................................................................................................... 35
2.3.14　緩存 ...................................................................................................... 35
2.3.15　已知的未知 .......................................................................................... 37
2.4　視角 .................................................................................................................... 37
2.4.1　資源分析 ................................................................................................ 38
2.4.2　工作負載分析 ........................................................................................ 39
2.5　方法 .................................................................................................................... 40
2.5.1　街燈訛方法 ............................................................................................ 42
2.5.2　隨機變動訛方法 .................................................................................... 42
2.5.3　責(zé)怪他人訛方法 .................................................................................... 43
2.5.4 Ad Hoc 核對清單法 ............................................................................... 43
2.5.5　問題陳述法 ............................................................................................ 44
2.5.6　科學(xué)法 .................................................................................................... 44
目錄 XXXV
2.5.7　診斷循環(huán) ................................................................................................ 46
2.5.8　工具法 .................................................................................................... 46
2.5.9 USE 方法 ................................................................................................ 46
2.5.10 RED 方法 ............................................................................................. 52
2.5.11　工作負載特征歸納 .............................................................................. 53
2.5.12　向下鉆取分析 ...................................................................................... 54
2.5.13　延時分析 .............................................................................................. 55
2.5.14 R 方法 ................................................................................................... 56
2.5.15　事件跟蹤 .............................................................................................. 56
2.5.16　基礎(chǔ)線統(tǒng)計 .......................................................................................... 58
2.5.17　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 59
2.5.18　緩存調(diào)優(yōu) .............................................................................................. 59
2.5.19　微基準(zhǔn)測試 .......................................................................................... 60
2.5.20　性能箴言 .............................................................................................. 60
2.6　建模 .................................................................................................................... 61
2.6.1　企業(yè)與云 ................................................................................................ 61
2.6.2　可視化識別 ............................................................................................ 62
2.6.3 Amdahl 擴展定律 .................................................................................. 63
2.6.4　通用擴展定律 ........................................................................................ 64
2.6.5　排隊理論 ................................................................................................ 65
2.7　容量規(guī)劃 ............................................................................................................ 68
2.7.1　資源極限 ................................................................................................ 69
2.7.2　因素分析 ................................................................................................ 70
2.7.3　擴展方案 ................................................................................................ 71
2.8　統(tǒng)計 .................................................................................................................... 72
2.8.1　量化性能收益 ........................................................................................ 72
2.8.2　平均值 .................................................................................................... 73
2.8.3　標(biāo)準(zhǔn)方差、百分位數(shù)、中位數(shù) ............................................................ 74
2.8.4　變異系數(shù) ................................................................................................ 75
2.8.5　多重模態(tài)分布 ........................................................................................ 75
2.8.6　異常值 .................................................................................................... 76
2.9　監(jiān)測 .................................................................................................................... 76
2.9.1　基于時間的規(guī)律 .................................................................................... 76
2.9.2　監(jiān)測產(chǎn)品 ................................................................................................ 78
XXXVI 目錄
2.9.3　自啟動以來的信息統(tǒng)計 ........................................................................ 78
2.10　可視化 .............................................................................................................. 78
2.10.1　線圖 ...................................................................................................... 79
2.10.2　散點圖 .................................................................................................. 80
2.10.3　熱圖 ...................................................................................................... 81
2.10.4　時間線圖 .............................................................................................. 82
2.10.5　表面圖 .................................................................................................. 83
2.10.6　可視化工具 .......................................................................................... 83
2.11　練習(xí)................................................................................................................... 84
2.12　參考資料 .......................................................................................................... 84
第3章　操作系統(tǒng)...........................................................................................................86
3.1　術(shù)語 .................................................................................................................... 87
3.2　背景 .................................................................................................................... 88
3.2.1　內(nèi)核 ........................................................................................................ 88
3.2.2　內(nèi)核態(tài)與用戶態(tài) .................................................................................... 90
3.2.3　系統(tǒng)調(diào)用 ................................................................................................ 91
3.2.4　中斷 ........................................................................................................ 92
3.2.5　時鐘和空閑 ............................................................................................ 96
3.2.6　進程 ........................................................................................................ 96
3.2.7　棧 ............................................................................................................ 99
3.2.8　虛擬內(nèi)存 .............................................................................................. 100
3.2.9　調(diào)度器 .................................................................................................. 102
3.2.10　文件系統(tǒng) ............................................................................................ 103
3.2.11　緩存 .................................................................................................... 105
3.2.12　網(wǎng)絡(luò) .................................................................................................... 106
3.2.13　設(shè)備驅(qū)動 ............................................................................................ 106
3.2.14　多處理器 ............................................................................................ 107
3.2.15　搶占 .................................................................................................... 107
3.2.16　資源管理 ............................................................................................ 107
3.2.17　可觀測性 ............................................................................................ 108
3.3　內(nèi)核 .................................................................................................................. 108
3.3.1 UNIX .................................................................................................... 109
3.3.2 BSD....................................................................................................... 110
目錄 XXXVII
3.3.3 Solaris ....................................................................................................111
3.4 Linux ..................................................................................................................111
3.4.1 Linux 內(nèi)核開發(fā) .................................................................................... 112
3.4.2 systemd ................................................................................................. 117
3.4.3 KPTI（meltdown） ............................................................................... 118
3.4.4 eBPF ..................................................................................................... 118
3.5　其他主題 .......................................................................................................... 119
3.5.1 PGO 內(nèi)核 ............................................................................................. 119
3.5.2 unikernel ............................................................................................... 120
3.5.3　微內(nèi)核和混合內(nèi)核 .............................................................................. 120
3.5.4　分布式操作系統(tǒng) .................................................................................. 121
3.6　內(nèi)核比較 .......................................................................................................... 121
3.7　練習(xí) .................................................................................................................. 122
3.8　參考資料 .......................................................................................................... 122
3.8.1　延伸閱讀 .............................................................................................. 125
第4章　觀測工具.........................................................................................................126
4.1　工具范圍 .......................................................................................................... 127
4.1.1　靜態(tài)性能工具 ...................................................................................... 127
4.1.2　危機處理工具 ...................................................................................... 128
4.2　工具類型 .......................................................................................................... 129
4.2.1　固定計數(shù)器 .......................................................................................... 130
4.2.2　剖析 ...................................................................................................... 131
4.2.3　跟蹤 ...................................................................................................... 132
4.2.4　監(jiān)測 ...................................................................................................... 133
4.3　監(jiān)測來源 .......................................................................................................... 135
4.3.1 /proc ...................................................................................................... 137
4.3.2 /sys ........................................................................................................ 140
4.3.3　延時核算 .............................................................................................. 142
4.3.4 netlink ................................................................................................... 143
4.3.5 tracepoint .............................................................................................. 143
4.3.6 kprobes .................................................................................................. 148
4.3.7 uprobes .................................................................................................. 151
4.3.8 USDT .................................................................................................... 152
XXXVIII 目錄
4.3.9　硬件計數(shù)器 .......................................................................................... 153
4.3.10　其他觀測源 ........................................................................................ 156
4.4 sar ...................................................................................................................... 158
4.4.1 sar(1) 的覆蓋范圍 ................................................................................ 158
4.4.2 sar(1) 監(jiān)測 ............................................................................................ 159
4.4.3 sar(1) 實時報告 .................................................................................... 162
4.4.4 sar(1) 文檔 ............................................................................................ 163
4.5　跟蹤工具 .......................................................................................................... 163
4.6　觀測工具的觀測 .............................................................................................. 164
4.7　練習(xí) .................................................................................................................. 165
4.8　參考資料 .......................................................................................................... 166
第5章　應(yīng)用程序.........................................................................................................168
5.1　應(yīng)用程序基礎(chǔ) .................................................................................................. 168
5.1.1　目標(biāo) ...................................................................................................... 170
5.1.2　常見情況的優(yōu)化 .................................................................................. 171
5.1.3　可觀測性 .............................................................................................. 171
5.1.4　大 O 標(biāo)記法 ......................................................................................... 172
5.2　應(yīng)用程序性能技術(shù) .......................................................................................... 173
5.2.1　選擇 I/O 尺寸 ....................................................................................... 173
5.2.2　緩存 ...................................................................................................... 173
5.2.3　緩沖區(qū) .................................................................................................. 174
5.2.4　輪詢 ...................................................................................................... 174
5.2.5　并發(fā)和并行 .......................................................................................... 174
5.2.6　非阻塞 I/O ............................................................................................ 178
5.2.7　處理器綁定 .......................................................................................... 179
5.2.8　性能箴言 .............................................................................................. 179
5.3　編程語言 .......................................................................................................... 179
5.3.1　編譯型語言 .......................................................................................... 180
5.3.2　解釋型語言 .......................................................................................... 182
5.3.3　虛擬機 .................................................................................................. 182
5.3.4　垃圾回收 .............................................................................................. 182
5.4　方法 .................................................................................................................. 183
5.4.1 CPU 剖析.............................................................................................. 184
目錄 XXXIX
5.4.2 off-CPU 分析 ........................................................................................ 186
5.4.3　系統(tǒng)調(diào)用分析 ...................................................................................... 189
5.4.4 USE 方法 .............................................................................................. 190
5.4.5　線程狀態(tài)分析 ...................................................................................... 191
5.4.6　鎖分析 .................................................................................................. 195
5.4.7　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 196
5.4.8　分布式跟蹤 .......................................................................................... 196
5.5　觀測工具 .......................................................................................................... 197
5.5.1 perf ........................................................................................................ 198
5.5.2 profile .................................................................................................... 201
5.5.3 offcputime ............................................................................................. 202
5.5.4 strace ..................................................................................................... 203
5.5.5 execsnoop ............................................................................................. 205
5.5.6 syscount ................................................................................................ 206
5.5.7 bpftrace ................................................................................................. 207
5.6　明白了 .............................................................................................................. 212
5.6.1　缺少符號 .............................................................................................. 212
5.6.2　缺少棧 .................................................................................................. 213
5.7　練習(xí) .................................................................................................................. 214
5.8　參考資料 .......................................................................................................... 216
第6章　CPU................................................................................................................218
6.1　術(shù)語 .................................................................................................................. 219
6.2　模型 .................................................................................................................. 220
6.2.1 CPU 架構(gòu).............................................................................................. 220
6.2.2 CPU 內(nèi)存緩存...................................................................................... 220
6.2.3 CPU 運行隊列...................................................................................... 221
6.3　概念 .................................................................................................................. 222
6.3.1　時鐘頻率 .............................................................................................. 222
6.3.2　指令 ...................................................................................................... 222
6.3.3　指令流水線 .......................................................................................... 223
6.3.4　指令寬度 .............................................................................................. 223
6.3.5　指令尺寸 .............................................................................................. 223
6.3.6 SMT ...................................................................................................... 224
XL 目錄
6.3.7 IPC 和 CPI ............................................................................................ 224
6.3.8　使用率 .................................................................................................. 225
6.3.9　用戶時間 / 內(nèi)核時間 ........................................................................... 225
6.3.10　飽和度 ................................................................................................ 225
6.3.11　搶占 .................................................................................................... 226
6.3.12　優(yōu)先級反轉(zhuǎn) ........................................................................................ 226
6.3.13　多進程和多線程 ................................................................................ 226
6.3.14　字長 .................................................................................................... 228
6.3.15　編譯器優(yōu)化 ........................................................................................ 228
6.4　架構(gòu) .................................................................................................................. 228
6.4.1　硬件 ...................................................................................................... 228
6.4.2　軟件 ...................................................................................................... 238
6.5　方法 .................................................................................................................. 242
6.5.1　工具法 .................................................................................................. 242
6.5.2 USE 方法 .............................................................................................. 243
6.5.3　負載特征歸納 ...................................................................................... 243
6.5.4　剖析 ...................................................................................................... 245
6.5.5　周期分析 .............................................................................................. 248
6.5.6　性能監(jiān)測 .............................................................................................. 248
6.5.7　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 249
6.5.8　優(yōu)先級調(diào)優(yōu) .......................................................................................... 249
6.5.9　資源控制 .............................................................................................. 250
6.5.10 CPU 綁定............................................................................................ 250
6.5.11　微基準(zhǔn)測試 ........................................................................................ 250
6.6　觀測工具 .......................................................................................................... 251
6.6.1 uptime ................................................................................................... 252
6.6.2 vmstat .................................................................................................... 255
6.6.3 mpstat .................................................................................................... 255
6.6.4 sar .......................................................................................................... 256
6.6.5 ps ........................................................................................................... 257
6.6.6 top ......................................................................................................... 258
6.6.7 pidstat .................................................................................................... 259
6.6.8 time 和 ptime ........................................................................................ 260
6.6.9 turbostat ................................................................................................ 261
目錄 XLI
6.6.10 showboost ........................................................................................... 262
6.6.11 pmcarch ............................................................................................... 263
6.6.12 tlbstat................................................................................................... 263
6.6.13 perf ...................................................................................................... 264
6.6.14 profile .................................................................................................. 274
6.6.15 cpudist ................................................................................................. 276
6.6.16 runqlat ................................................................................................. 277
6.6.17 runqlen ................................................................................................ 278
6.6.18 softirqs ................................................................................................ 279
6.6.19 hardirqs ............................................................................................... 280
6.6.20 bpftrace ............................................................................................... 280
6.6.21　其他工具 ............................................................................................ 283
6.7　可視化 .............................................................................................................. 285
6.7.1　使用率熱圖 .......................................................................................... 286
6.7.2　亞秒級偏移量熱圖 .............................................................................. 286
6.7.3　火焰圖 .................................................................................................. 287
6.7.4 FlameScope........................................................................................... 290
6.8　實驗 .................................................................................................................. 291
6.8.1 Ad Hoc .................................................................................................. 291
6.8.2 SysBench .............................................................................................. 291
6.9　調(diào)優(yōu) .................................................................................................................. 292
6.9.1　編譯器選項 .......................................................................................... 293
6.9.2　調(diào)度優(yōu)先級和調(diào)度類 .......................................................................... 293
6.9.3　調(diào)度器選項 .......................................................................................... 293
6.9.4　調(diào)節(jié)調(diào)速器 .......................................................................................... 294
6.9.5　能耗狀態(tài) .............................................................................................. 295
6.9.6 CPU 綁定.............................................................................................. 295
6.9.7　獨占 CPU 組......................................................................................... 295
6.9.8　資源控制 .............................................................................................. 296
6.9.9　安全啟動選項 ...................................................................................... 296
6.9.10　處理器選項（BIOS 調(diào)優(yōu)） ................................................................ 297
6.10　練習(xí) ................................................................................................................ 297
6.11　參考資料......................................................................................................... 298
XLII 目錄
第7章　內(nèi)存................................................................................................................301
7.1　術(shù)語 .................................................................................................................. 302
7.2　概念 .................................................................................................................. 303
7.2.1　虛擬內(nèi)存 .............................................................................................. 303
7.2.2　換頁 ...................................................................................................... 303
7.2.3　按需換頁 .............................................................................................. 305
7.2.4　過度提交 .............................................................................................. 306
7.2.5　進程交換 .............................................................................................. 306
7.2.6　文件系統(tǒng)緩存用量 .............................................................................. 306
7.2.7　使用率和飽和度 .................................................................................. 307
7.2.8　分配器 .................................................................................................. 307
7.2.9　共享內(nèi)存 .............................................................................................. 307
7.2.10　工作集大小 ........................................................................................ 308
7.2.11　字長 .................................................................................................... 308
7.3　架構(gòu) .................................................................................................................. 308
7.3.1　硬件 ...................................................................................................... 309
7.3.2　軟件 ...................................................................................................... 313
7.3.3　進程虛擬地址空間 .............................................................................. 317
7.4　方法 .................................................................................................................. 320
7.4.1　工具法 .................................................................................................. 321
7.4.2 USE 方法 .............................................................................................. 322
7.4.3　描述使用情況 ...................................................................................... 323
7.4.4　周期分析 .............................................................................................. 324
7.4.5　性能監(jiān)測 .............................................................................................. 324
7.4.6　泄漏檢測 .............................................................................................. 324
7.4.7　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 325
7.4.8　資源控制 .............................................................................................. 325
7.4.9　微基準(zhǔn)測試 .......................................................................................... 326
7.4.10　內(nèi)存收縮 ............................................................................................ 326
7.5　觀測工具 .......................................................................................................... 326
7.5.1 vmstat .................................................................................................... 327
7.5.2 PSI......................................................................................................... 328
7.5.3 swapon .................................................................................................. 329
目錄 XLIII
7.5.4 sar .......................................................................................................... 329
7.5.5 slabtop ................................................................................................... 331
7.5.6 numastat ................................................................................................ 332
7.5.7 ps ........................................................................................................... 332
7.5.8 top ......................................................................................................... 333
7.5.9 pmap ..................................................................................................... 334
7.5.10 perf ...................................................................................................... 336
7.5.11 drsnoop ................................................................................................ 339
7.5.12 wss ...................................................................................................... 340
7.5.13 bpftrace ............................................................................................... 341
7.5.14　其他工具 ............................................................................................ 345
7.6　調(diào)優(yōu) .................................................................................................................. 348
7.6.1　可調(diào)參數(shù) .............................................................................................. 348
7.6.2　多種頁面大小 ...................................................................................... 349
7.6.3　分配器 .................................................................................................. 350
7.6.4 NUMA 綁定 ......................................................................................... 350
7.6.5　資源控制 .............................................................................................. 351
7.7 　練習(xí) ................................................................................................................. 351
7.8　參考資料 .......................................................................................................... 353
第8章　文件系統(tǒng).........................................................................................................355
8.1　術(shù)語 .................................................................................................................. 356
8.2　模型 .................................................................................................................. 357
8.2.1　文件系統(tǒng)接口 ...................................................................................... 357
8.2.2　文件系統(tǒng)緩存 ...................................................................................... 357
8.2.3　二級緩存 .............................................................................................. 358
8.3　概念 .................................................................................................................. 358
8.3.1　文件系統(tǒng)延時 ...................................................................................... 358
8.3.2　緩存 ...................................................................................................... 359
8.3.3　隨機與順序 I/O .................................................................................... 359
8.3.4　預(yù)取 ...................................................................................................... 360
8.3.5　預(yù)讀 ...................................................................................................... 361
8.3.6　回寫緩存 .............................................................................................. 361
8.3.7　同步寫 .................................................................................................. 362
XLIV 目錄
8.3.8　裸 I/O 與直接 I/O ................................................................................. 362
8.3.9　非阻塞 I/O ............................................................................................ 363
8.3.10　內(nèi)存映射文件 .................................................................................... 363
8.3.11　元數(shù)據(jù) ................................................................................................ 363
8.3.12　邏輯 I/O 與物理 I/O ........................................................................... 364
8.3.13　操作并不平等 .................................................................................... 366
8.3.14　特殊的文件系統(tǒng) ................................................................................ 367
8.3.15　訪問時間戳 ........................................................................................ 367
8.3.16　容量 .................................................................................................... 367
8.4　架構(gòu) .................................................................................................................. 367
8.4.1　文件系統(tǒng) I/O 棧 ................................................................................... 368
8.4.2 VFS ....................................................................................................... 368
8.4.3　文件系統(tǒng)緩存 ...................................................................................... 369
8.4.4　文件系統(tǒng)特性 ...................................................................................... 371
8.4.5　文件系統(tǒng)種類 ...................................................................................... 373
8.4.6　卷和池 .................................................................................................. 378
8.5　方法 .................................................................................................................. 379
8.5.1　磁盤分析 .............................................................................................. 380
8.5.2　延時分析 .............................................................................................. 380
8.5.3　負載特征歸納 ...................................................................................... 382
8.5.4　性能監(jiān)測 .............................................................................................. 384
8.5.5　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 384
8.5.6　緩存調(diào)優(yōu) .............................................................................................. 385
8.5.7　負載分離 .............................................................................................. 385
8.5.8　微基準(zhǔn)測試 .......................................................................................... 385
8.6　觀測工具 .......................................................................................................... 387
8.6.1 mount .................................................................................................... 387
8.6.2 free ........................................................................................................ 388
8.6.3 top ......................................................................................................... 388
8.6.4 vmstat .................................................................................................... 389
8.6.5 sar .......................................................................................................... 389
8.6.6 slabtop ................................................................................................... 390
8.6.7 strace ..................................................................................................... 390
8.6.8 fatrace ................................................................................................... 391
目錄 XLV
8.6.9 LatencyTOP .......................................................................................... 392
8.6.10 opensnoop ........................................................................................... 392
8.6.11 filetop .................................................................................................. 393
8.6.12 cachestat .............................................................................................. 395
8.6.13 ext4dist（xfs、zfs、btrfs、nfs） ........................................................ 395
8.6.14 ext4slower（xfs、zfs、btrfs、nfs） ................................................... 397
8.6.15 bpftrace ............................................................................................... 398
8.6.16　其他工具 ............................................................................................ 404
8.6.17　可視化 ................................................................................................ 406
8.7　實驗 .................................................................................................................. 407
8.7.1 Ad Hoc .................................................................................................. 407
8.7.2　微基準(zhǔn)測試工具 .................................................................................. 407
8.7.3　緩存刷新 .............................................................................................. 409
8.8　調(diào)優(yōu) .................................................................................................................. 410
8.8.1　應(yīng)用程序調(diào)用 ...................................................................................... 410
8.8.2 ext4 ....................................................................................................... 411
8.8.3 ZFS ....................................................................................................... 413
8.9　練習(xí) .................................................................................................................. 415
8.10　參考資料 ........................................................................................................ 416
第9章　磁盤................................................................................................................418
9.1　術(shù)語 .................................................................................................................. 419
9.2　模型 .................................................................................................................. 419
9.2.1　簡單磁盤 .............................................................................................. 420
9.2.2　緩存磁盤 .............................................................................................. 420
9.2.3　控制器 .................................................................................................. 421
9.3　概念 .................................................................................................................. 421
9.3.1　測量時間 .............................................................................................. 421
9.3.2　時間尺度 .............................................................................................. 424
9.3.3　緩存 ...................................................................................................... 425
9.3.4　隨機 I/O 與連續(xù) I/O ............................................................................. 425
9.3.5　讀 / 寫比 ............................................................................................... 426
9.3.6 I/O 大小 ................................................................................................ 426
9.3.7 IOPS 并不平等 ..................................................................................... 427
XLVI 目錄
9.3.8　非數(shù)據(jù)傳輸磁盤命令 .......................................................................... 427
9.3.9　使用率 .................................................................................................. 427
9.3.10　飽和度 ................................................................................................ 428
9.3.11 I/O 等待 .............................................................................................. 428
9.3.12　同步與異步 ........................................................................................ 429
9.3.13　磁盤 I/O 與應(yīng)用程序 I/O ................................................................... 429
9.4　架構(gòu) .................................................................................................................. 430
9.4.1　磁盤類型 .............................................................................................. 430
9.4.2　接口 ...................................................................................................... 436
9.4.3　存儲類型 .............................................................................................. 438
9.4.4　操作系統(tǒng)磁盤 I/O 棧 ........................................................................... 441
9.5　方法 .................................................................................................................. 444
9.5.1　工具法 .................................................................................................. 444
9.5.2 USE 方法 .............................................................................................. 445
9.5.3　性能監(jiān)測 .............................................................................................. 446
9.5.4　負載特征歸納 ...................................................................................... 446
9.5.5　延時分析 .............................................................................................. 448
9.5.6　靜態(tài)性能調(diào)優(yōu) ...................................................................................... 449
9.5.7　緩存調(diào)優(yōu) .............................................................................................. 450
9.5.8　資源控制 .............................................................................................. 450
9.5.9　微基準(zhǔn)測試 .......................................................................................... 451
9.5.10　伸縮 .................................................................................................... 452
9.6　觀測工具 .......................................................................................................... 452
9.6.1 iostat ...................................................................................................... 453
9.6.2 sar .......................................................................................................... 458
9.6.3 PSI......................................................................................................... 458
9.6.4 pidstat .................................................................................................... 459
9.6.5 perf ........................................................................................................ 460
9.6.6 biolatency .............................................................................................. 462
9.6.7 biosnoop ................................................................................................ 465
9.6.8 iotop、biotop ........................................................................................ 467
9.6.9 biostacks ............................................................................................... 469
9.6.10 blktrace ............................................................................................... 470
9.6.11 bpftrace ............................................................................................... 474
目錄 XLVII
9.6.12 MegaCli .............................................................................................. 478
9.6.13 smartctl ............................................................................................... 479
9.6.14 SCSI 日志 ........................................................................................... 480
9.6.15　其他工具 ............................................................................................ 481
9.7　可視化 .............................................................................................................. 482
9.7.1　折線圖 .................................................................................................. 482
9.7.2　延時散點圖 .......................................................................................... 482
9.7.3　延時熱圖 .............................................................................................. 483
9.7.4　偏移量熱圖 .......................................................................................... 483
9.7.5　使用率熱圖 .......................................................................................... 484
9.8　實驗 .................................................................................................................. 484
9.8.1 Ad Hoc .................................................................................................. 485
9.8.2　自定義負載生成器 .............................................................................. 485
9.8.3　微基準(zhǔn)測試工具 .................................................................................. 485
9.8.4　隨機讀示例 .......................................................................................... 486
9.8.5 ioping .................................................................................................... 486
9.8.6 fio .......................................................................................................... 487
9.8.7 blkreplay ............................................................................................... 487
9.9　調(diào)優(yōu) .................................................................................................................. 488
9.9.1　操作系統(tǒng)可調(diào)參數(shù) .............................................................................. 488
9.9.2　磁盤設(shè)備可調(diào)參數(shù) .............................................................................. 489
9.9.3　磁盤控制器可調(diào)參數(shù) .......................................................................... 489
9.10　練習(xí) ................................................................................................................ 490
9.11　參考資料......................................................................................................... 491
第10章　網(wǎng)絡(luò)..............................................................................................................493
10.1　術(shù)語 ................................................................................................................ 494
10.2　模型 ................................................................................................................ 494
10.2.1　網(wǎng)絡(luò)接口 ............................................................................................ 494
10.2.2　控制器 ................................................................................................ 495
10.2.3　協(xié)議棧 ................................................................................................ 495
10.3　概念 ................................................................................................................ 496
10.3.1　網(wǎng)絡(luò)和路由 ........................................................................................ 496
10.3.2　協(xié)議 .................................................................................................... 497
XLVIII 目錄
10.3.3　封裝 .................................................................................................... 498
10.3.4　包的大小 ............................................................................................ 498
10.3.5　延時 .................................................................................................... 499
10.3.6　緩沖 .................................................................................................... 500
10.3.7　連接積壓隊列 .................................................................................... 501
10.3.8　接口協(xié)商 ............................................................................................ 501
10.3.9　避免阻塞 ............................................................................................ 501
10.3.10　使用率 .............................................................................................. 502
10.3.11　本地連接 .......................................................................................... 502
10.4　架構(gòu) ................................................................................................................ 502
10.4.1　協(xié)議 .................................................................................................... 503
10.4.2　硬件 .................................................................................................... 508
10.4.3　軟件 .................................................................................................... 510
10.5　方法 ................................................................................................................ 517
10.5.1　工具法 ................................................................................................ 518
10.5.2 USE 方法 ............................................................................................ 519
10.5.3　工作負載特征歸納 ............................................................................ 519
10.5.4　延時分析 ............................................................................................ 520
10.5.5　性能監(jiān)測 ............................................................................................ 522
10.5.6　數(shù)據(jù)包嗅探 ........................................................................................ 522
10.5.7 TCP 分析 ............................................................................................ 523
10.5.8　靜態(tài)性能調(diào)優(yōu) .................................................................................... 524
10.5.9　資源控制 ............................................................................................ 525
10.5.10　微基準(zhǔn)測試 ...................................................................................... 525
10.6　觀測工具 ........................................................................................................ 526
10.6.1 ss ......................................................................................................... 527
10.6.2 ip ......................................................................................................... 529
10.6.3 ifconfig ................................................................................................ 530
10.6.4 nstat ..................................................................................................... 530
10.6.5 netstat .................................................................................................. 532
10.6.6 sar ........................................................................................................ 535
10.6.7 nicstat .................................................................................................. 538
10.6.8 ethtool ................................................................................................. 539
10.6.9 tcplife .................................................................................................. 540
目錄 XLIX
10.6.10 tcptop ................................................................................................ 541
10.6.11 tcpretrans ........................................................................................... 542
10.6.12 bpftrace ............................................................................................. 543
10.6.13 tcpdump ............................................................................................ 551
10.6.14 Wireshark .......................................................................................... 552
10.6.15　其他工具 .......................................................................................... 553
10.7　實驗 ................................................................................................................ 554
10.7.1 ping ..................................................................................................... 555
10.7.2 traceroute ............................................................................................ 555
10.7.3 pathchar............................................................................................... 556
10.7.4 iperf ..................................................................................................... 557
10.7.5 netperf ................................................................................................. 558
10.7.6 tc ......................................................................................................... 558
10.7.7　其他工具 ............................................................................................ 559
10.8　調(diào)優(yōu) ................................................................................................................ 559
10.8.1　系統(tǒng)級可調(diào)參數(shù) ................................................................................ 560
10.8.2　套接字選項 ........................................................................................ 565
10.8.3　配置 .................................................................................................... 566
10.9　練習(xí) ................................................................................................................ 566
10.10　參考資料 ...................................................................................................... 567
第11章　云計算...........................................................................................................572
11.1　背景................................................................................................................. 573
11.1.1　實例類型 ............................................................................................ 574
11.1.2　可擴展的架構(gòu) .................................................................................... 574
11.1.3　容量規(guī)劃 ............................................................................................ 575
11.1.4　存儲 .................................................................................................... 577
11.1.5　多租戶 ................................................................................................ 578
11.1.6　編排（Kubernetes） ............................................................................ 579
11.2　硬件虛擬化..................................................................................................... 580
11.2.1　實現(xiàn) .................................................................................................... 582
11.2.2　系統(tǒng)開銷 ............................................................................................ 582
11.2.3　資源控制 ............................................................................................ 588
11.2.4　可觀測性 ............................................................................................ 590
L 目錄
11.3　操作系統(tǒng)虛擬化............................................................................................. 599
11.3.1　實現(xiàn)方式 ............................................................................................ 600
11.3.2　系統(tǒng)開銷 ............................................................................................ 603
11.3.3　資源控制 ............................................................................................ 606
11.3.4　可觀測性 ............................................................................................ 610
11.4　輕量虛擬化..................................................................................................... 623
11.4.1　實現(xiàn) .................................................................................................... 623
11.4.2　開銷 .................................................................................................... 624
11.4.3　資源控制 ............................................................................................ 624
11.4.4　可觀測性 ............................................................................................ 624
11.5　其他類型......................................................................................................... 626
11.6　比較................................................................................................................. 627
11.7 練習(xí)................................................................................................................... 628
11.8　參考資料......................................................................................................... 629
第12章　基準(zhǔn)測試.......................................................................................................632
12.1　背景 ................................................................................................................ 633
12.1.1　原因 .................................................................................................... 633
12.1.2　有效的基準(zhǔn)測試 ................................................................................ 634
12.1.3　基準(zhǔn)測試失敗 .................................................................................... 635
12.2 　基準(zhǔn)測試的類型 ........................................................................................... 642
12.2.1　微基準(zhǔn)測試 ........................................................................................ 642
12.2.2　模擬 .................................................................................................... 644
12.2.3　回放 .................................................................................................... 645
12.2.4　行業(yè)標(biāo)準(zhǔn) ............................................................................................ 645
12.3　方法 ................................................................................................................ 647
12.3.1　被動基準(zhǔn)測試 .................................................................................... 647
12.3.2　主動基準(zhǔn)測試 .................................................................................... 648
12.3.3 CPU 剖析............................................................................................ 651
12.3.4 USE 方法 ............................................................................................ 652
12.3.5　工作負載特征歸納 ............................................................................ 652
12.3.6　自定義基準(zhǔn)測試 ................................................................................ 652
12.3.7　逐漸增加負載 .................................................................................... 653
12.3.8　合理性檢查 ........................................................................................ 655
目錄 LI
12.3.9　統(tǒng)計分析 ............................................................................................ 656
12.3.10　基準(zhǔn)測試檢查清單 .......................................................................... 657
12.4　基準(zhǔn)測試問題 ................................................................................................ 657
12.5　練習(xí) ................................................................................................................ 659
12.6　參考資料 ........................................................................................................ 659
第13章　perf...............................................................................................................661
13.1　子命令概覽 .................................................................................................... 662
13.2　單行命令 ........................................................................................................ 664
13.3 perf事件 .......................................................................................................... 668
13.4　硬件事件 ........................................................................................................ 671
13.4.1　頻率采樣 ............................................................................................ 672
13.5　軟件事件 ........................................................................................................ 673
13.6 tracepoint事件 ................................................................................................ 674
13.7　探針事件 ........................................................................................................ 675
13.7.1 kprobe ................................................................................................. 675
13.7.2 uprobe ................................................................................................. 677
13.7.3 USDT 探針 ......................................................................................... 680
13.8 perf stat ........................................................................................................... 681
13.8.1　選項 .................................................................................................... 682
13.8.2　周期統(tǒng)計信息 .................................................................................... 683
13.8.3 CPU 均衡............................................................................................ 683
13.8.4　事件過濾器 ........................................................................................ 683
13.8.5　隱藏統(tǒng)計信息 .................................................................................... 684
13.9 perf record ....................................................................................................... 684
13.9.1　選項 .................................................................................................... 685
13.9.2 CPU 剖析............................................................................................ 685
13.9.3　棧遍歷 ................................................................................................ 686
13.10 perf report ..................................................................................................... 687
13.10.1 TUI .................................................................................................... 687
13.10.2 STDIO............................................................................................... 688
13.11 perf script ...................................................................................................... 689
13.11.1　火焰圖 .............................................................................................. 690
13.11.2　跟蹤腳本 .......................................................................................... 691
LII 目錄
13.12 perf trace ....................................................................................................... 691
13.12.1　內(nèi)核版本 .......................................................................................... 692
13.13　其他命令 ...................................................................................................... 692
13.14 perf文檔 ........................................................................................................ 694
13.15　參考資料 ...................................................................................................... 694
第14章　Ftrace............................................................................................................696
14.1　功能概述 ........................................................................................................ 697
14.2 tracefs（/sys） ............................................................................................... 699
14.2.1 tracefs 的內(nèi)容 ..................................................................................... 700
14.3 Ftrace函數(shù)剖析器 .......................................................................................... 702
14.4 Ftrace函數(shù)跟蹤 .............................................................................................. 703
14.4.1　使用 trace............................................................................................ 704
14.4.2　使用 trace_pipe ................................................................................... 706
14.4.3　選項 .................................................................................................... 706
14.5　跟蹤點 ............................................................................................................ 707
14.5.1　過濾器 ................................................................................................ 708
14.5.2　觸發(fā)器 ................................................................................................ 709
14.6 kprobes ............................................................................................................ 710
14.6.1　事件跟蹤 ............................................................................................ 710
14.6.2　參數(shù) .................................................................................................... 711
14.6.3　返回值 ................................................................................................ 712
14.6.4　過濾器和觸發(fā)器 ................................................................................ 712
14.6.5 kprobe 剖析 ........................................................................................ 713
14.7 uprobes ............................................................................................................ 713
14.7.1　事件跟蹤 ............................................................................................ 713
14.7.2　參數(shù)和返回值 .................................................................................... 714
14.7.3　過濾器和觸發(fā)器 ................................................................................ 714
14.7.4 uprobe 剖析 ........................................................................................ 715
14.8 Ftrace function_graph ..................................................................................... 715
14.8.1　圖表跟蹤 ............................................................................................ 715
14.8.2　選項 .................................................................................................... 717
14.9 Ftrace hwlat .................................................................................................... 717
14.10 Ftrace hist觸發(fā)器 .......................................................................................... 718
目錄 LIII
14.10.1　單關(guān)鍵字 .......................................................................................... 718
14.10.2　字段 .................................................................................................. 719
14.10.3　修飾器 .............................................................................................. 720
14.10.4 PID 過濾器 ....................................................................................... 720
14.10.5　多關(guān)鍵字 .......................................................................................... 721
14.10.6　棧蹤跡關(guān)鍵字 .................................................................................. 722
14.10.7　綜合事件 .......................................................................................... 723
14.11 trace-cmd....................................................................................................... 725
14.11.1　子命令概述 ...................................................................................... 726
14.11.2 trace-cmd 單行命令 ......................................................................... 727
14.11.3 trace-cmd 和 perf(1) 的比較 ............................................................ 729
14.11.4 trace-cmd function_graph ................................................................. 730
14.11.5 KernelShark ...................................................................................... 730
14.11.6 trace-cmd 文檔 ................................................................................. 731
14.12 perf ftrace ...................................................................................................... 731
14.13 perf-tools ....................................................................................................... 732
14.13.1　工具覆蓋 .......................................................................................... 733
14.13.2　單用途的工具 .................................................................................. 733
14.13.3　多用途工具 ...................................................................................... 735
14.13.4 perf-tools 單行命令 .......................................................................... 735
14.13.5　示例 .................................................................................................. 738
14.13.6 perf-tools 與 BCC/BPF 的對比........................................................ 738
14.13.7　文檔 .................................................................................................. 739
14.14 Ftrace文檔 .................................................................................................... 739
14.15　參考資料 ...................................................................................................... 739
第15章　BPF...............................................................................................................741
15.1 BCC ................................................................................................................ 743
15.1.1　安裝 .................................................................................................... 744
15.1.2　工具范圍 ............................................................................................ 744
15.1.3　單用途工具 ........................................................................................ 745
15.1.4　多用途工具 ........................................................................................ 747
15.1.5　單行命令 ............................................................................................ 747
15.1.6　多用途工具示例 ................................................................................ 749
LIV 目錄
15.1.7 BCC 與 bpftrace 的比較 .................................................................... 749
15.1.8　文檔 .................................................................................................... 750
15.2 bpftrace ........................................................................................................... 751
15.2.1　安裝 .................................................................................................... 752
15.2.2　工具 .................................................................................................... 752
15.2.3　單行命令 ............................................................................................ 753
15.2.4　編程 .................................................................................................... 756
15.2.5　參考 .................................................................................................... 764
15.2.6　文檔 .................................................................................................... 770
15.3　參考資料 ........................................................................................................ 771
第16章　案例研究.......................................................................................................773
16.1　無法解釋的收益 ............................................................................................ 773
16.1.1　問題陳述 ............................................................................................ 773
16.1.2　分析策略 ............................................................................................ 774
16.1.3　統(tǒng)計數(shù)據(jù) ............................................................................................ 774
16.1.4　配置 .................................................................................................... 776
16.1.5 PMC .................................................................................................... 778
16.1.6　軟件事件 ............................................................................................ 779
16.1.7　跟蹤 .................................................................................................... 780
16.1.8　結(jié)論 .................................................................................................... 782
16.2　其他信息 ........................................................................................................ 783
16.3　參考資料 ........................................................................................................ 783
附錄A　USE方法：Linux............................................................................................784
附錄B　sar總結(jié)...........................................................................................................788
附錄C　bpftrace單行命令............................................................................................790
附錄D　精選練習(xí)題答案..............................................................................................796
附錄E　系統(tǒng)性能名人錄..............................................................................................798