writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process

writefile: (groupid=0, jobs=1): err= 0: pid=2139: Tue Aug  2 14:19:29 2016
  write: io=9216.0MB, bw=218383KB/s, iops=213, runt= 43214msec
    slat (usec): min=116, max=31168, avg=4320.30, stdev=2492.82
    clat (msec): min=22, max=1100, avg=922.01, stdev=85.18
     lat (msec): min=27, max=1100, avg=926.33, stdev=85.23
    clat percentiles (msec):
     |  1.00th=[  445],  5.00th=[  889], 10.00th=[  889], 20.00th=[  898],
     | 30.00th=[  906], 40.00th=[  914], 50.00th=[  922], 60.00th=[  938],
     | 70.00th=[  947], 80.00th=[  963], 90.00th=[  971], 95.00th=[  988],
     | 99.00th=[ 1090], 99.50th=[ 1090], 99.90th=[ 1090], 99.95th=[ 1106],
     | 99.99th=[ 1106]
    bw (KB  /s): min= 1014, max=232541, per=99.12%, avg=216464.55, stdev=25595.87
    lat (msec) : 50=0.07%, 100=0.12%, 250=0.36%, 500=0.60%, 750=0.61%
    lat (msec) : 1000=94.01%, 2000=4.24%
  cpu          : usr=7.97%, sys=6.22%, ctx=7279, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=218382KB/s, minb=218382KB/s, maxb=218382KB/s, mint=43214msec, maxt=43214msec

Disk stats (read/write):
  xvdc: ios=111/221185, merge=0/0, ticks=28/6096832, in_queue=6096708, util=99.71%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2178: Tue Aug  2 14:19:59 2016
  read : io=4031.4MB, bw=137572KB/s, iops=34393, runt= 30007msec
    slat (usec): min=4, max=7027, avg=105.88, stdev=480.95
    clat (msec): min=1, max=64, avg=14.77, stdev= 4.08
     lat (msec): min=1, max=64, avg=14.88, stdev= 4.12
    clat percentiles (usec):
     |  1.00th=[ 7776],  5.00th=[ 8896], 10.00th=[10432], 20.00th=[11456],
     | 30.00th=[12480], 40.00th=[13632], 50.00th=[14272], 60.00th=[15168],
     | 70.00th=[16320], 80.00th=[17536], 90.00th=[19584], 95.00th=[22144],
     | 99.00th=[27264], 99.50th=[30080], 99.90th=[39680], 99.95th=[43264],
     | 99.99th=[50432]
    bw (KB  /s): min=29856, max=40159, per=25.01%, avg=34408.48, stdev=1829.32
    lat (msec) : 2=0.01%, 4=0.02%, 10=8.38%, 20=82.83%, 50=8.77%
    lat (msec) : 100=0.01%
  cpu          : usr=7.66%, sys=10.77%, ctx=43165, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=1032034/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=4031.4MB, aggrb=137572KB/s, minb=137572KB/s, maxb=137572KB/s, mint=30007msec, maxt=30007msec

Disk stats (read/write):
  xvdc: ios=1029035/0, merge=114/0, ticks=4569464/0, in_queue=4571560, util=99.78%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2187: Tue Aug  2 14:20:29 2016
  write: io=3971.5MB, bw=135508KB/s, iops=33876, runt= 30011msec
    slat (usec): min=4, max=8064, avg=107.62, stdev=492.88
    clat (msec): min=1, max=70, avg=14.99, stdev= 4.07
     lat (msec): min=1, max=70, avg=15.10, stdev= 4.11
    clat percentiles (usec):
     |  1.00th=[ 8032],  5.00th=[ 9152], 10.00th=[10688], 20.00th=[11584],
     | 30.00th=[12736], 40.00th=[13760], 50.00th=[14528], 60.00th=[15424],
     | 70.00th=[16512], 80.00th=[17536], 90.00th=[19840], 95.00th=[22144],
     | 99.00th=[27520], 99.50th=[30848], 99.90th=[40192], 99.95th=[44288],
     | 99.99th=[52992]
    bw (KB  /s): min=28664, max=38352, per=25.01%, avg=33888.29, stdev=1714.00
    lat (msec) : 2=0.01%, 4=0.01%, 10=7.40%, 20=83.28%, 50=9.29%
    lat (msec) : 100=0.02%
  cpu          : usr=7.73%, sys=10.70%, ctx=42023, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=1016680/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3971.5MB, aggrb=135507KB/s, minb=135507KB/s, maxb=135507KB/s, mint=30011msec, maxt=30011msec

Disk stats (read/write):
  xvdc: ios=83/1013639, merge=0/98, ticks=24/4550108, in_queue=4550592, util=99.86%