btrfs存储配额实战

测试配额是否已实施
$ sudo btrfs qgroup show /home
ERROR: can’t list qgroups: quotas not enabled

$ sudo btrfs quota enable /home

g@g:/home$ sudo btrfs subvolume create gqqnbig
Create subvolume ‘./gqqnbig’
g@g:/home$ sudo btrfs subvolume list /home
ID 257 gen 9 top level 5 path gqqnbig
# 这里显示/home/gqqnbig这个目录（subvolume）的id是257，后面的gen、top level不用管。
g@g:/home$ sudo mkdir shared
g@g:/home$ sudo btrfs subvolume create shared/gqqnbig
Create subvolume ‘shared/gqqnbig’
# 再创建一个属于用户gqqnbig的目录（subvolume）
g@g:/home$ tree
.
├── g
├── gqqnbig
└── shared
└── gqqnbig

4 directories, 0 files
g@g:/home$ sudo btrfs subvolume list /home
ID 257 gen 16 top level 5 path gqqnbig
ID 258 gen 16 top level 5 path shared/gqqnbig
g@g:/home$ sudo btrfs qgroup limit 10M 1/1000 /home

测试性能

启用了quota
g@g:/home/gqqnbig$ fio –name TEST –eta-newline=5s –filename=fio-tempfile.dat –rw=write –size=500m –io_size=10g –blocksize=1024k –ioengine=libaio –fsync=10000 –iodepth=32 –direct=1 –numjobs=1 –runtime=60 –group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
TEST: Laying out IO file (1 file / 500MiB)
Jobs: 1 (f=1): [W(1)][87.5%][w=1137MiB/s][w=1137 IOPS][eta 00m:01s]
Jobs: 1 (f=1): [W(1)][100.0%][w=1195MiB/s][w=1195 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5262: Sat Mar 13 07:03:38 2021
write: IOPS=1231, BW=1231MiB/s (1291MB/s)(10.0GiB/8318msec); 0 zone resets
slat (usec): min=291, max=3799, avg=801.43, stdev=227.19
clat (usec): min=8, max=45831, avg=24340.30, stdev=4705.25
lat (usec): min=713, max=46938, avg=25143.09, stdev=4753.47
clat percentiles (usec):
| 1.00th=[ 3752], 5.00th=[19530], 10.00th=[22152], 20.00th=[22676],
| 30.00th=[23200], 40.00th=[23462], 50.00th=[23987], 60.00th=[24511],
| 70.00th=[25560], 80.00th=[26608], 90.00th=[28705], 95.00th=[30540],
| 99.00th=[39060], 99.50th=[40633], 99.90th=[42206], 99.95th=[44303],
| 99.99th=[45876]
bw ( MiB/s): min= 1115, max= 1340, per=99.64%, avg=1226.66, stdev=72.77, samples=16
iops : min= 1115, max= 1340, avg=1226.50, stdev=72.91, samples=16
lat (usec) : 10=0.01%, 100=0.09%, 250=0.10%, 500=0.01%, 750=0.02%
lat (usec) : 1000=0.08%
lat (msec) : 2=0.26%, 4=0.51%, 10=1.46%, 20=2.53%, 50=94.93%
fsync/fdatasync/sync_file_range:
sync (nsec): min=1137, max=1137, avg=1137.00, stdev= 0.00
sync percentiles (nsec):
| 1.00th=[ 1144], 5.00th=[ 1144], 10.00th=[ 1144], 20.00th=[ 1144],
| 30.00th=[ 1144], 40.00th=[ 1144], 50.00th=[ 1144], 60.00th=[ 1144],
| 70.00th=[ 1144], 80.00th=[ 1144], 90.00th=[ 1144], 95.00th=[ 1144],
| 99.00th=[ 1144], 99.50th=[ 1144], 99.90th=[ 1144], 99.95th=[ 1144],
| 99.99th=[ 1144]
cpu : usr=2.03%, sys=97.52%, ctx=447, majf=0, minf=12
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=1231MiB/s (1291MB/s), 1231MiB/s-1231MiB/s (1291MB/s-1291MB/s), io=10.0GiB (10.7GB), run=8318-8318msec

未启用quota
g@g:~$ fio –name TEST –eta-newline=5s –filename=fio-tempfile.dat –rw=write –size=500m –io_size=10g –blocksize=1024k –ioengine=libaio –fsync=10000 –iodepth=32 –direct=1 –numjobs=1 –runtime=60 –group_reporting
TEST: (g=0): rw=write, bs=(R) 1024KiB-1024KiB, (W) 1024KiB-1024KiB, (T) 1024KiB-1024KiB, ioengine=libaio, iodepth=32
fio-3.16
Starting 1 process
Jobs: 1 (f=1): [W(1)][36.8%][w=205MiB/s][w=205 IOPS][eta 00m:12s]
Jobs: 1 (f=1): [W(1)][50.0%][w=211MiB/s][w=210 IOPS][eta 00m:14s]
Jobs: 1 (f=1): [W(1)][83.3%][w=202MiB/s][w=202 IOPS][eta 00m:04s]
Jobs: 1 (f=1): [W(1)][92.9%][w=217MiB/s][w=217 IOPS][eta 00m:02s]
Jobs: 1 (f=1): [W(1)][100.0%][w=220MiB/s][w=219 IOPS][eta 00m:00s]
TEST: (groupid=0, jobs=1): err= 0: pid=5391: Sat Mar 13 07:08:06 2021
write: IOPS=358, BW=359MiB/s (376MB/s)(10.0GiB/28536msec); 0 zone resets
slat (usec): min=305, max=208754, avg=944.30, stdev=3418.22
clat (usec): min=47, max=361746, avg=87102.09, stdev=69447.47
lat (usec): min=707, max=362378, avg=88049.07, stdev=69570.53
clat percentiles (msec):
| 1.00th=[ 9], 5.00th=[ 23], 10.00th=[ 23], 20.00th=[ 24],
| 30.00th=[ 26], 40.00th=[ 27], 50.00th=[ 34], 60.00th=[ 140],
| 70.00th=[ 148], 80.00th=[ 153], 90.00th=[ 163], 95.00th=[ 174],
| 99.00th=[ 275], 99.50th=[ 300], 99.90th=[ 334], 99.95th=[ 342],
| 99.99th=[ 351]
bw ( KiB/s): min=155337, max=1337344, per=99.66%, avg=366208.84, stdev=345264.49, samples=57
iops : min= 151, max= 1306, avg=357.28, stdev=337.24, samples=57
lat (usec) : 50=0.01%, 100=0.06%, 250=0.03%, 750=0.02%, 1000=0.07%
lat (msec) : 2=0.09%, 4=0.23%, 10=0.71%, 20=1.74%, 50=49.01%
lat (msec) : 100=2.03%, 250=44.16%, 500=1.84%
fsync/fdatasync/sync_file_range:
sync (nsec): min=1146, max=1146, avg=1146.00, stdev= 0.00
sync percentiles (nsec):
| 1.00th=[ 1144], 5.00th=[ 1144], 10.00th=[ 1144], 20.00th=[ 1144],
| 30.00th=[ 1144], 40.00th=[ 1144], 50.00th=[ 1144], 60.00th=[ 1144],
| 70.00th=[ 1144], 80.00th=[ 1144], 90.00th=[ 1144], 95.00th=[ 1144],
| 99.00th=[ 1144], 99.50th=[ 1144], 99.90th=[ 1144], 99.95th=[ 1144],
| 99.99th=[ 1144]
cpu : usr=1.06%, sys=31.44%, ctx=3581, majf=0, minf=14
IO depths : 1=0.2%, 2=0.4%, 4=0.8%, 8=1.6%, 16=3.3%, 32=93.6%, >=64=0.0%
submit : 0=0.0%, 4=100.0%, 8=0.0%, 16=0.0%, 32=0.0%, 64=0.0%, >=64=0.0%
complete : 0=0.0%, 4=99.8%, 8=0.0%, 16=0.0%, 32=0.2%, 64=0.0%, >=64=0.0%
issued rwts: total=0,10240,0,1 short=0,0,0,0 dropped=0,0,0,0
latency : target=0, window=0, percentile=100.00%, depth=32

Run status group 0 (all jobs):
WRITE: bw=359MiB/s (376MB/s), 359MiB/s-359MiB/s (376MB/s-376MB/s), io=10.0GiB (10.7GB), run=28536-28536msec

有时删除subvolume后，该subvolume不会被立即删除。0/642是一个例子，它的size不为0，但在subvolume list不显示。但用-d选项可以看到。

20:33 $ sudo btrfs qgroup show -c /home
qgroupid rfer excl child
——– —- —- —–
0/5 334.69GiB 334.69GiB —
0/617 619.24MiB 619.24MiB —
0/618 10.05GiB 10.05GiB —
0/642 16.00KiB 16.00KiB —
1/1002 10.66GiB 10.66GiB 0/617,0/618
1/5007 16.00KiB 16.00KiB 0/642
20:34 $ sudo btrfs subvolume list /home
ID 617 gen 18339 top level 5 path qiqig
ID 618 gen 17905 top level 5 path shared/qiqig
20:36 $ sudo btrfs subvolume list -d /home
ID 642 gen 18311 top level 0 path DELETED