writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process

writefile: (groupid=0, jobs=1): err= 0: pid=2089: Tue Aug  2 14:23:44 2016
  write: io=9216.0MB, bw=221374KB/s, iops=216, runt= 42630msec
    slat (usec): min=99, max=30341, avg=4254.62, stdev=2450.77
    clat (msec): min=19, max=1012, avg=910.13, stdev=81.77
     lat (msec): min=26, max=1012, avg=914.38, stdev=81.80
    clat percentiles (msec):
     |  1.00th=[  433],  5.00th=[  889], 10.00th=[  889], 20.00th=[  889],
     | 30.00th=[  898], 40.00th=[  906], 50.00th=[  906], 60.00th=[  914],
     | 70.00th=[  938], 80.00th=[  947], 90.00th=[  963], 95.00th=[  971],
     | 99.00th=[ 1004], 99.50th=[ 1012], 99.90th=[ 1012], 99.95th=[ 1012],
     | 99.99th=[ 1012]
    bw (KB  /s): min= 1094, max=232079, per=98.94%, avg=219023.07, stdev=25365.69
    lat (msec) : 20=0.01%, 50=0.05%, 100=0.13%, 250=0.36%, 500=0.61%
    lat (msec) : 750=0.61%, 1000=96.02%, 2000=2.21%
  cpu          : usr=8.17%, sys=6.31%, ctx=7201, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=221374KB/s, minb=221374KB/s, maxb=221374KB/s, mint=42630msec, maxt=42630msec

Disk stats (read/write):
  xvdc: ios=141/221185, merge=0/0, ticks=32/6019168, in_queue=6019036, util=99.71%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2167: Tue Aug  2 14:24:14 2016
  read : io=3916.1MB, bw=133665KB/s, iops=33416, runt= 30007msec
    slat (usec): min=4, max=12921, avg=109.18, stdev=506.71
    clat (msec): min=2, max=67, avg=15.20, stdev= 4.28
     lat (msec): min=2, max=67, avg=15.31, stdev= 4.32
    clat percentiles (usec):
     |  1.00th=[ 7968],  5.00th=[ 9152], 10.00th=[10688], 20.00th=[11712],
     | 30.00th=[12864], 40.00th=[13888], 50.00th=[14656], 60.00th=[15680],
     | 70.00th=[16768], 80.00th=[18048], 90.00th=[20352], 95.00th=[22912],
     | 99.00th=[28544], 99.50th=[31872], 99.90th=[40704], 99.95th=[44800],
     | 99.99th=[53504]
    bw (KB  /s): min=23232, max=39608, per=25.01%, avg=33426.29, stdev=2273.50
    lat (msec) : 4=0.01%, 10=7.50%, 20=81.67%, 50=10.80%, 100=0.02%
  cpu          : usr=7.42%, sys=10.41%, ctx=40948, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=1002723/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3916.1MB, aggrb=133665KB/s, minb=133665KB/s, maxb=133665KB/s, mint=30007msec, maxt=30007msec

Disk stats (read/write):
  xvdc: ios=999774/0, merge=115/0, ticks=4576052/0, in_queue=4576984, util=99.77%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2198: Tue Aug  2 14:24:45 2016
  write: io=3924.9MB, bw=133936KB/s, iops=33484, runt= 30007msec
    slat (usec): min=4, max=7262, avg=108.98, stdev=496.79
    clat (msec): min=1, max=71, avg=15.17, stdev= 4.23
     lat (msec): min=1, max=71, avg=15.28, stdev= 4.27
    clat percentiles (usec):
     |  1.00th=[ 8032],  5.00th=[ 9024], 10.00th=[10688], 20.00th=[11712],
     | 30.00th=[12864], 40.00th=[13888], 50.00th=[14656], 60.00th=[15680],
     | 70.00th=[16768], 80.00th=[18048], 90.00th=[20096], 95.00th=[22656],
     | 99.00th=[28032], 99.50th=[31104], 99.90th=[41728], 99.95th=[45312],
     | 99.99th=[54016]
    bw (KB  /s): min=28063, max=38143, per=25.00%, avg=33486.14, stdev=2017.12
    lat (msec) : 2=0.01%, 4=0.01%, 10=7.79%, 20=81.46%, 50=10.73%
    lat (msec) : 100=0.02%
  cpu          : usr=7.69%, sys=10.59%, ctx=41898, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=1004758/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3924.9MB, aggrb=133936KB/s, minb=133936KB/s, maxb=133936KB/s, mint=30007msec, maxt=30007msec

Disk stats (read/write):
  xvdc: ios=257/1001704, merge=0/113, ticks=68/4560216, in_queue=4560436, util=99.99%