writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process

writefile: (groupid=0, jobs=1): err= 0: pid=2093: Tue Aug  2 14:05:40 2016
  write: io=9216.0MB, bw=220155KB/s, iops=214, runt= 42866msec
    slat (usec): min=99, max=35200, avg=4282.72, stdev=2488.07
    clat (msec): min=21, max=1110, avg=912.71, stdev=80.34
     lat (msec): min=27, max=1116, avg=916.99, stdev=80.36
    clat percentiles (msec):
     |  1.00th=[  465],  5.00th=[  881], 10.00th=[  889], 20.00th=[  889],
     | 30.00th=[  898], 40.00th=[  906], 50.00th=[  914], 60.00th=[  922],
     | 70.00th=[  938], 80.00th=[  955], 90.00th=[  963], 95.00th=[  979],
     | 99.00th=[ 1012], 99.50th=[ 1020], 99.90th=[ 1057], 99.95th=[ 1090],
     | 99.99th=[ 1106]
    bw (KB  /s): min=  935, max=232541, per=99.36%, avg=218739.94, stdev=25336.55
    lat (msec) : 50=0.07%, 100=0.12%, 250=0.33%, 500=0.56%, 750=0.56%
    lat (msec) : 1000=95.43%, 2000=2.93%
  cpu          : usr=8.04%, sys=6.16%, ctx=7283, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=220155KB/s, minb=220155KB/s, maxb=220155KB/s, mint=42866msec, maxt=42866msec

Disk stats (read/write):
  xvdc: ios=111/221185, merge=0/0, ticks=32/6059784, in_queue=6059704, util=99.71%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2188: Tue Aug  2 14:06:10 2016
  read : io=3868.4MB, bw=131999KB/s, iops=32999, runt= 30009msec
    slat (usec): min=4, max=9455, avg=110.83, stdev=516.19
    clat (msec): min=3, max=88, avg=15.39, stdev= 4.31
     lat (msec): min=3, max=88, avg=15.51, stdev= 4.35
    clat percentiles (usec):
     |  1.00th=[ 8096],  5.00th=[ 9280], 10.00th=[10816], 20.00th=[11840],
     | 30.00th=[13120], 40.00th=[14144], 50.00th=[14912], 60.00th=[15808],
     | 70.00th=[16768], 80.00th=[18048], 90.00th=[20352], 95.00th=[22912],
     | 99.00th=[29056], 99.50th=[32640], 99.90th=[42240], 99.95th=[45824],
     | 99.99th=[56064]
    bw (KB  /s): min=21840, max=37860, per=25.01%, avg=33010.68, stdev=2135.43
    lat (msec) : 4=0.01%, 10=6.69%, 20=82.15%, 50=11.13%, 100=0.03%
  cpu          : usr=7.26%, sys=10.12%, ctx=40437, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=990293/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3868.4MB, aggrb=131999KB/s, minb=131999KB/s, maxb=131999KB/s, mint=30009msec, maxt=30009msec

Disk stats (read/write):
  xvdc: ios=987536/0, merge=110/0, ticks=4571344/0, in_queue=4572484, util=99.78%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2243: Tue Aug  2 14:06:41 2016
  write: io=3862.5MB, bw=131795KB/s, iops=32948, runt= 30010msec
    slat (usec): min=4, max=8905, avg=111.12, stdev=506.67
    clat (msec): min=1, max=73, avg=15.42, stdev= 4.39
     lat (msec): min=1, max=73, avg=15.53, stdev= 4.43
    clat percentiles (usec):
     |  1.00th=[ 8096],  5.00th=[ 9152], 10.00th=[10688], 20.00th=[11840],
     | 30.00th=[12992], 40.00th=[14144], 50.00th=[14784], 60.00th=[15936],
     | 70.00th=[17024], 80.00th=[18304], 90.00th=[20608], 95.00th=[23168],
     | 99.00th=[29056], 99.50th=[32384], 99.90th=[42240], 99.95th=[45824],
     | 99.99th=[54016]
    bw (KB  /s): min=28734, max=38320, per=25.01%, avg=32962.02, stdev=1728.77
    lat (msec) : 2=0.01%, 4=0.01%, 10=7.63%, 20=80.14%, 50=12.21%
    lat (msec) : 100=0.02%
  cpu          : usr=7.42%, sys=10.09%, ctx=41550, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=988793/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3862.5MB, aggrb=131795KB/s, minb=131795KB/s, maxb=131795KB/s, mint=30010msec, maxt=30010msec

Disk stats (read/write):
  xvdc: ios=174/986032, merge=0/109, ticks=40/4558872, in_queue=4562904, util=99.93%