writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process

writefile: (groupid=0, jobs=1): err= 0: pid=2265: Tue Aug  2 14:08:32 2016
  write: io=9216.0MB, bw=217477KB/s, iops=212, runt= 43394msec
    slat (usec): min=108, max=16888, avg=4335.87, stdev=2475.26
    clat (msec): min=23, max=1029, avg=926.26, stdev=79.00
     lat (msec): min=29, max=1031, avg=930.60, stdev=78.99
    clat percentiles (msec):
     |  1.00th=[  469],  5.00th=[  906], 10.00th=[  906], 20.00th=[  906],
     | 30.00th=[  914], 40.00th=[  930], 50.00th=[  938], 60.00th=[  938],
     | 70.00th=[  947], 80.00th=[  947], 90.00th=[  963], 95.00th=[  988],
     | 99.00th=[ 1020], 99.50th=[ 1020], 99.90th=[ 1029], 99.95th=[ 1029],
     | 99.99th=[ 1029]
    bw (KB  /s): min= 1017, max=227555, per=99.15%, avg=215620.32, stdev=24387.30
    lat (msec) : 50=0.07%, 100=0.11%, 250=0.33%, 500=0.56%, 750=0.56%
    lat (msec) : 1000=94.40%, 2000=3.97%
  cpu          : usr=7.98%, sys=6.27%, ctx=7513, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=217476KB/s, minb=217476KB/s, maxb=217476KB/s, mint=43394msec, maxt=43394msec

Disk stats (read/write):
  xvdc: ios=131/221185, merge=0/0, ticks=32/6124020, in_queue=6123920, util=99.67%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2271: Tue Aug  2 14:09:02 2016
  read : io=3957.5MB, bw=135036KB/s, iops=33758, runt= 30007msec
    slat (usec): min=4, max=7269, avg=108.14, stdev=492.50
    clat (msec): min=1, max=76, avg=15.05, stdev= 4.24
     lat (msec): min=1, max=76, avg=15.16, stdev= 4.28
    clat percentiles (usec):
     |  1.00th=[ 7904],  5.00th=[ 9024], 10.00th=[10432], 20.00th=[11584],
     | 30.00th=[12736], 40.00th=[13760], 50.00th=[14528], 60.00th=[15552],
     | 70.00th=[16512], 80.00th=[17792], 90.00th=[20096], 95.00th=[22656],
     | 99.00th=[28032], 99.50th=[30848], 99.90th=[40704], 99.95th=[44800],
     | 99.99th=[52992]
    bw (KB  /s): min=28670, max=39584, per=25.01%, avg=33777.27, stdev=1751.21
    lat (msec) : 2=0.01%, 4=0.01%, 10=8.31%, 20=81.32%, 50=10.35%
    lat (msec) : 100=0.02%
  cpu          : usr=7.48%, sys=10.47%, ctx=42413, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=1013003/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3957.5MB, aggrb=135035KB/s, minb=135035KB/s, maxb=135035KB/s, mint=30007msec, maxt=30007msec

Disk stats (read/write):
  xvdc: ios=1010338/0, merge=103/0, ticks=4574504/0, in_queue=4575524, util=99.79%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2280: Tue Aug  2 14:09:32 2016
  write: io=3857.2MB, bw=131618KB/s, iops=32904, runt= 30008msec
    slat (usec): min=4, max=7958, avg=111.32, stdev=505.73
    clat (msec): min=1, max=69, avg=15.44, stdev= 4.33
     lat (msec): min=1, max=69, avg=15.55, stdev= 4.37
    clat percentiles (usec):
     |  1.00th=[ 8096],  5.00th=[ 9280], 10.00th=[10816], 20.00th=[11840],
     | 30.00th=[13120], 40.00th=[14144], 50.00th=[14912], 60.00th=[15936],
     | 70.00th=[17024], 80.00th=[18304], 90.00th=[20608], 95.00th=[23168],
     | 99.00th=[28800], 99.50th=[31616], 99.90th=[41728], 99.95th=[45312],
     | 99.99th=[53504]
    bw (KB  /s): min=27880, max=38248, per=25.01%, avg=32915.42, stdev=1831.92
    lat (msec) : 2=0.01%, 4=0.01%, 10=7.15%, 20=80.79%, 50=12.04%
    lat (msec) : 100=0.02%
  cpu          : usr=7.41%, sys=10.14%, ctx=41652, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=987395/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3857.2MB, aggrb=131617KB/s, minb=131617KB/s, maxb=131617KB/s, mint=30008msec, maxt=30008msec

Disk stats (read/write):
  xvdc: ios=258/984533, merge=0/96, ticks=68/4555056, in_queue=4555664, util=100.00%