[lxc-devel] Report correct filesystem usage / limits on BTRFS subvolumes with quota

[Austin S. Hemmelgarn]

On 2018-08-13 22:49, Jeff Mahoney wrote:
> On 7/31/18 9:49 AM, Thomas Leister wrote:
>> Dear David,
>> hello everyone,
>>
>> during a recent project of mine involving LXD and BTRFS I found out that
>> quotas on BTRFS subvolumes are enforced, but file system usage and
>> limits set via quotas are not reported correctly in LXC containers.
>>
>> I've found this discussion regarding my problem:
>> https://github.com/lxc/lxd/issues/2180
>>
>> There was already a proposal to introduce subvolume quota support some
>> time ago:
>> https://marc.info/?l=linux-btrfs&m=147576434114415&w=2
>>
>> @David as I've seen your response on that topic on the mailing list,
>> maybe you can tell me if there are any plans to support correct
>> subvolume quota reporting e.g. for "df -h" calls from within a
>> container? Maybe there's already something on your / SUSE's roadmap? :-)
>>
>> As more and more container environments spin up these days, there might
>> be a growing demand on that :-) Personally I'd really appreciate if I
>> could read the current file system usage and limit from within a
>> container using BTRFS as storage backend.
> 
> In addition to Qu's deeper dive into the qgroup internals, the bigger
> issue is that statfs(), which df uses, only allows the kernel to report
> two numbers to describe space usage: total blocks and free blocks[1].
> 
> Currently, the only space tracking we have at a subvolume level is
> qgroups.  There are other trees than "file" (i.e. subvolume) trees in
> btrfs and those aren't accounted using qgroups.
Last I knew, ext4 and XFS didn't count metadata (other than xattrs) 
against quotas, so outside of accounting inlined files properly (I'm not 
sure if qgroups do so or not), we're about equivalent to them in terms 
of what we're counting.
> 
> So what should these numbers contain to describe to the user what their
> space usage looks like?  We can have subvolume qgroup limits, nested
> qgroup limits, disk capacity limits.  Df on btrfs is already confusing
> enough with the data/metadata allocation split combined with potentially
> different allocation policies for each.
This is _easy_ though.  You report whatever the lowest current limit is. 
  99% of the time if something or someone is calling `df`, they want to 
know how much space on that volume is currently available for usage. 
Just reporting whichever of the limits they're going to hit first 
handles that case perfectly, and more importantly, is completely 
unambiguous.
> 
> That's the biggest hurdle to reporting per-subvolume information via df.
>   It's a feature request that's been in my TODO inbox for a while.  I
> started in on it again this year and came to the conclusion that wiring
> qgroups into df would be easy[2] - but would ultimately raise more
> questions than it solved for the user.  That's pretty much what I'm
> seeing in this thread.
> 
> So, what I have mostly working is adding support for 'btrfs qgroup show'
> to output in JSON format so that tools can be easily written to use the
> numbers available to provide exactly what information the user wants
> (given the limit of what's available.)
> 
> -Jeff
> 
> [1] There's f_bavail too, but that's not really relevant and runs into
> the same issue as above in any case.
> [2] Well, wiring it into statfs would be easy.  Publishing vfsmounts for
> every subvolume so df actually calls it on unqualified 'df' executions
> is rather a bit more involved.
But probably worth it, because we could then expose in the mount options 
for each subvolume whether or not it's in a qgroup (and possibly which 
one if it is), which would help alleviate a decent percentage of the 
confusion.  Right now, it's a pain in the arse to figure out if a given 
subvolume has a quota or not, because _none_ of the conventional methods 
of determining if quotas are in effect work at all.