writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process

writefile: (groupid=0, jobs=1): err= 0: pid=2221: Tue Aug  2 14:25:42 2016
  write: io=9216.0MB, bw=225204KB/s, iops=219, runt= 41905msec
    slat (usec): min=112, max=12148, avg=4176.89, stdev=2383.06
    clat (msec): min=21, max=964, avg=895.37, stdev=76.16
     lat (msec): min=26, max=965, avg=899.55, stdev=76.15
    clat percentiles (msec):
     |  1.00th=[  433],  5.00th=[  889], 10.00th=[  889], 20.00th=[  889],
     | 30.00th=[  898], 40.00th=[  898], 50.00th=[  898], 60.00th=[  906],
     | 70.00th=[  906], 80.00th=[  914], 90.00th=[  922], 95.00th=[  955],
     | 99.00th=[  963], 99.50th=[  963], 99.90th=[  963], 99.95th=[  963],
     | 99.99th=[  963]
    bw (KB  /s): min= 1145, max=231619, per=98.90%, avg=222734.74, stdev=25199.35
    lat (msec) : 50=0.07%, 100=0.13%, 250=0.36%, 500=0.61%, 750=0.61%
    lat (msec) : 1000=98.23%
  cpu          : usr=8.28%, sys=6.45%, ctx=7278, majf=0, minf=28
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=225204KB/s, minb=225204KB/s, maxb=225204KB/s, mint=41905msec, maxt=41905msec

Disk stats (read/write):
  xvdc: ios=111/221185, merge=0/0, ticks=28/5920196, in_queue=5920080, util=99.69%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2227: Tue Aug  2 14:26:12 2016
  read : io=3974.9MB, bw=135639KB/s, iops=33909, runt= 30008msec
    slat (usec): min=4, max=7285, avg=107.42, stdev=488.21
    clat (msec): min=2, max=75, avg=14.98, stdev= 4.18
     lat (msec): min=2, max=75, avg=15.09, stdev= 4.22
    clat percentiles (usec):
     |  1.00th=[ 7904],  5.00th=[ 9024], 10.00th=[10560], 20.00th=[11456],
     | 30.00th=[12736], 40.00th=[13760], 50.00th=[14400], 60.00th=[15424],
     | 70.00th=[16512], 80.00th=[17792], 90.00th=[20096], 95.00th=[22400],
     | 99.00th=[27776], 99.50th=[30592], 99.90th=[40192], 99.95th=[43776],
     | 99.99th=[52480]
    bw (KB  /s): min=29285, max=38962, per=25.01%, avg=33922.61, stdev=1845.26
    lat (msec) : 4=0.01%, 10=8.18%, 20=81.90%, 50=9.90%, 100=0.01%
  cpu          : usr=7.63%, sys=10.67%, ctx=42636, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=1017563/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3974.9MB, aggrb=135638KB/s, minb=135638KB/s, maxb=135638KB/s, mint=30008msec, maxt=30008msec

Disk stats (read/write):
  xvdc: ios=1014683/0, merge=124/0, ticks=4584640/0, in_queue=4585640, util=99.78%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes

benchmark: (groupid=0, jobs=4): err= 0: pid=2236: Tue Aug  2 14:26:42 2016
  write: io=3932.4MB, bw=134187KB/s, iops=33546, runt= 30008msec
    slat (usec): min=5, max=9277, avg=108.70, stdev=497.56
    clat (msec): min=3, max=74, avg=15.14, stdev= 4.10
     lat (msec): min=3, max=74, avg=15.25, stdev= 4.14
    clat percentiles (usec):
     |  1.00th=[ 8096],  5.00th=[ 9280], 10.00th=[10816], 20.00th=[11712],
     | 30.00th=[12992], 40.00th=[14016], 50.00th=[14656], 60.00th=[15552],
     | 70.00th=[16768], 80.00th=[17792], 90.00th=[20096], 95.00th=[22400],
     | 99.00th=[27776], 99.50th=[30848], 99.90th=[40192], 99.95th=[44288],
     | 99.99th=[52992]
    bw (KB  /s): min=28279, max=37729, per=25.01%, avg=33565.98, stdev=1678.93
    lat (msec) : 4=0.01%, 10=7.15%, 20=82.76%, 50=10.07%, 100=0.02%
  cpu          : usr=7.72%, sys=10.66%, ctx=41683, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=1006672/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3932.4MB, aggrb=134187KB/s, minb=134187KB/s, maxb=134187KB/s, mint=30008msec, maxt=30008msec

Disk stats (read/write):
  xvdc: ios=262/1003771, merge=0/119, ticks=68/4566764, in_queue=4566812, util=100.00%