writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200 fio-2.1.3 Starting 1 process writefile: (groupid=0, jobs=1): err= 0: pid=2232: Tue Aug 2 13:54:40 2016 write: io=9216.0MB, bw=216792KB/s, iops=211, runt= 43531msec slat (usec): min=75, max=9826, avg=4347.19, stdev=2487.86 clat (msec): min=21, max=1013, avg=930.29, stdev=79.63 lat (msec): min=28, max=1013, avg=934.64, stdev=79.62 clat percentiles (msec): | 1.00th=[ 453], 5.00th=[ 898], 10.00th=[ 922], 20.00th=[ 930], | 30.00th=[ 930], 40.00th=[ 930], 50.00th=[ 938], 60.00th=[ 938], | 70.00th=[ 947], 80.00th=[ 955], 90.00th=[ 963], 95.00th=[ 996], | 99.00th=[ 1004], 99.50th=[ 1004], 99.90th=[ 1012], 99.95th=[ 1012], | 99.99th=[ 1012] bw (KB /s): min= 1113, max=230501, per=98.92%, avg=214449.59, stdev=24033.70 lat (msec) : 50=0.07%, 100=0.12%, 250=0.35%, 500=0.58%, 750=0.60% lat (msec) : 1000=95.97%, 2000=2.32% cpu : usr=8.08%, sys=6.23%, ctx=7146, majf=0, minf=28 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued : total=r=0/w=9216/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=9216.0MB, aggrb=216792KB/s, minb=216792KB/s, maxb=216792KB/s, mint=43531msec, maxt=43531msec Disk stats (read/write): xvdc: ios=122/221185, merge=0/0, ticks=36/6142416, in_queue=6142312, util=99.71% benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 ... benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 fio-2.1.3 Starting 4 processes benchmark: (groupid=0, jobs=4): err= 0: pid=2238: Tue Aug 2 13:55:10 2016 read : io=3878.5MB, bw=132345KB/s, iops=33086, runt= 30009msec slat (usec): min=4, max=10276, avg=110.38, stdev=539.72 clat (msec): min=1, max=77, avg=15.36, stdev= 4.61 lat (msec): min=1, max=77, avg=15.47, stdev= 4.65 clat percentiles (usec): | 1.00th=[ 7968], 5.00th=[ 9152], 10.00th=[10304], 20.00th=[11456], | 30.00th=[12736], 40.00th=[13888], 50.00th=[14784], 60.00th=[15680], | 70.00th=[17024], 80.00th=[18560], 90.00th=[21376], 95.00th=[23680], | 99.00th=[29312], 99.50th=[32384], 99.90th=[42240], 99.95th=[46848], | 99.99th=[56064] bw (KB /s): min=26796, max=39324, per=25.01%, avg=33098.56, stdev=2081.85 lat (msec) : 2=0.01%, 4=0.01%, 10=8.51%, 20=77.73%, 50=13.72% lat (msec) : 100=0.03% cpu : usr=7.24%, sys=10.09%, ctx=35880, majf=0, minf=635 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued : total=r=992882/w=0/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): READ: io=3878.5MB, aggrb=132344KB/s, minb=132344KB/s, maxb=132344KB/s, mint=30009msec, maxt=30009msec Disk stats (read/write): xvdc: ios=991083/0, merge=88/0, ticks=4658452/0, in_queue=4665924, util=99.84% benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 ... benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128 fio-2.1.3 Starting 4 processes benchmark: (groupid=0, jobs=4): err= 0: pid=2247: Tue Aug 2 13:55:40 2016 write: io=3778.1MB, bw=128958KB/s, iops=32239, runt= 30007msec slat (usec): min=5, max=21843, avg=113.56, stdev=545.84 clat (msec): min=1, max=77, avg=15.76, stdev= 4.92 lat (msec): min=1, max=77, avg=15.87, stdev= 4.97 clat percentiles (usec): | 1.00th=[ 8160], 5.00th=[ 9280], 10.00th=[10560], 20.00th=[11712], | 30.00th=[12992], 40.00th=[14144], 50.00th=[14912], 60.00th=[16064], | 70.00th=[17280], 80.00th=[19072], 90.00th=[22144], 95.00th=[24960], | 99.00th=[31104], 99.50th=[34560], 99.90th=[45312], 99.95th=[49408], | 99.99th=[59136] bw (KB /s): min=26658, max=38247, per=25.00%, avg=32240.77, stdev=2209.45 lat (msec) : 2=0.01%, 4=0.05%, 10=7.73%, 20=76.24%, 50=15.94% lat (msec) : 100=0.05% cpu : usr=7.30%, sys=9.94%, ctx=37070, majf=0, minf=117 IO depths : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0% submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0% complete : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1% issued : total=r=0/w=967411/d=0, short=r=0/w=0/d=0 Run status group 0 (all jobs): WRITE: io=3778.1MB, aggrb=128958KB/s, minb=128958KB/s, maxb=128958KB/s, mint=30007msec, maxt=30007msec Disk stats (read/write): xvdc: ios=70/964945, merge=0/97, ticks=20/4580432, in_queue=4589636, util=99.76%