writefile: (g=0): rw=write, bs=1M-1M/1M-1M/1M-1M, ioengine=libaio, iodepth=200
fio-2.1.3
Starting 1 process
Jobs: 1 (f=1): [W] [97.7% done] [0KB/208.9MB/0KB /s] [0/208/0 iops] [eta 00m:01s]
writefile: (groupid=0, jobs=1): err= 0: pid=2757: Tue Aug  2 13:28:53 2016
  write: io=9216.0MB, bw=221905KB/s, iops=216, runt= 42528msec
    slat (usec): min=60, max=39800, avg=4241.54, stdev=2529.64
    clat (msec): min=26, max=1007, avg=907.45, stdev=76.65
     lat (msec): min=27, max=1012, avg=911.70, stdev=76.64
    clat percentiles (msec):
     |  1.00th=[  465],  5.00th=[  881], 10.00th=[  889], 20.00th=[  898],
     | 30.00th=[  898], 40.00th=[  906], 50.00th=[  914], 60.00th=[  922],
     | 70.00th=[  930], 80.00th=[  938], 90.00th=[  947], 95.00th=[  947],
     | 99.00th=[  955], 99.50th=[  955], 99.90th=[  971], 99.95th=[  988],
     | 99.99th=[ 1004]
    bw (KB  /s): min= 1035, max=233005, per=99.12%, avg=219947.01, stdev=25070.74
    lat (msec) : 50=0.07%, 100=0.11%, 250=0.37%, 500=0.54%, 750=0.59%
    lat (msec) : 1000=98.30%, 2000=0.03%
  cpu          : usr=8.30%, sys=4.51%, ctx=7067, majf=0, minf=29
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.2%, 32=0.3%, >=64=99.3%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=9216/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=9216.0MB, aggrb=221905KB/s, minb=221905KB/s, maxb=221905KB/s, mint=42528msec, maxt=42528msec

Disk stats (read/write):
  xvdc: ios=141/221185, merge=0/0, ticks=28/6009876, in_queue=6009756, util=99.37%
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randread, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes
Jobs: 4 (f=4): [rrrr] [100.0% done] [132.4MB/0KB/0KB /s] [33.9K/0/0 iops] [eta 00m:00s]
benchmark: (groupid=0, jobs=4): err= 0: pid=2763: Tue Aug  2 13:29:24 2016
  read : io=3977.4MB, bw=135725KB/s, iops=33931, runt= 30008msec
    slat (usec): min=4, max=24343, avg=107.44, stdev=502.89
    clat (msec): min=1, max=65, avg=14.97, stdev= 4.43
     lat (msec): min=1, max=65, avg=15.08, stdev= 4.48
    clat percentiles (usec):
     |  1.00th=[ 7584],  5.00th=[ 8768], 10.00th=[10304], 20.00th=[11328],
     | 30.00th=[12480], 40.00th=[13632], 50.00th=[14400], 60.00th=[15424],
     | 70.00th=[16512], 80.00th=[17792], 90.00th=[20352], 95.00th=[22912],
     | 99.00th=[28800], 99.50th=[31872], 99.90th=[41216], 99.95th=[45312],
     | 99.99th=[52480]
    bw (KB  /s): min=29008, max=38435, per=25.01%, avg=33943.62, stdev=1797.04
    lat (msec) : 2=0.02%, 4=0.15%, 10=8.93%, 20=79.99%, 50=10.89%
    lat (msec) : 100=0.02%
  cpu          : usr=7.52%, sys=10.55%, ctx=41619, majf=0, minf=631
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=1018212/w=0/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
   READ: io=3977.4MB, aggrb=135725KB/s, minb=135725KB/s, maxb=135725KB/s, mint=30008msec, maxt=30008msec

Disk stats (read/write):
  xvdc: ios=1015610/0, merge=113/0, ticks=4571140/0, in_queue=4572432, util=99.65%
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
...
benchmark: (g=0): rw=randwrite, bs=4K-4K/4K-4K/4K-4K, ioengine=libaio, iodepth=128
fio-2.1.3
Starting 4 processes
Jobs: 4 (f=4): [wwww] [100.0% done] [0KB/130.5MB/0KB /s] [0/33.4K/0 iops] [eta 00m:00s]
benchmark: (groupid=0, jobs=4): err= 0: pid=2772: Tue Aug  2 13:29:54 2016
  write: io=3942.7MB, bw=134539KB/s, iops=33634, runt= 30008msec
    slat (usec): min=5, max=17890, avg=108.37, stdev=505.23
    clat (msec): min=1, max=71, avg=15.10, stdev= 4.36
     lat (msec): min=1, max=71, avg=15.21, stdev= 4.40
    clat percentiles (usec):
     |  1.00th=[ 7776],  5.00th=[ 9024], 10.00th=[10432], 20.00th=[11584],
     | 30.00th=[12736], 40.00th=[13760], 50.00th=[14528], 60.00th=[15552],
     | 70.00th=[16768], 80.00th=[18048], 90.00th=[20352], 95.00th=[22912],
     | 99.00th=[28288], 99.50th=[31104], 99.90th=[40704], 99.95th=[44800],
     | 99.99th=[52992]
    bw (KB  /s): min=27377, max=39944, per=25.02%, avg=33655.58, stdev=2137.43
    lat (msec) : 2=0.01%, 4=0.13%, 10=8.44%, 20=79.96%, 50=11.43%
    lat (msec) : 100=0.02%
  cpu          : usr=7.72%, sys=10.64%, ctx=41172, majf=0, minf=117
  IO depths    : 1=0.1%, 2=0.1%, 4=0.1%, 8=0.1%, 16=0.1%, 32=0.1%, >=64=100.0%
     submit    : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
     complete  : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.1%
     issued    : total=r=0/w=1009313/d=0, short=r=0/w=0/d=0

Run status group 0 (all jobs):
  WRITE: io=3942.7MB, aggrb=134539KB/s, minb=134539KB/s, maxb=134539KB/s, mint=30008msec, maxt=30008msec

Disk stats (read/write):
  xvdc: ios=260/1006741, merge=0/98, ticks=60/4540912, in_queue=4543256, util=99.84%