[lxc-users] zfs disk usage for published lxd images

[Tomasz Chmielewski]

I've been using btrfs quite a lot and it's great technology. There are 
some shortcomings though:

1) compression only really works with compress-force mount argument

On a system which only stores text logs (receiving remote rsyslog logs), 
I was gaining around 10% with compress=zlib mount argument - not that 
great for text files/logs. With compress-force=zlib, I'm getting over 
85% compress ratio (i.e. using just 165 GB of disk space to store 1.2 TB 
data). Maybe that's the consequence of receiving log streams, not sure 
(but, compress-force fixed bad compression ratio).


2) the first kernel where I'm not getting out-of-space issues is 4.6 
(which was released yesterday). If you're using a distribution kernel, 
you will probably be seeing out-of-space issues. Quite likely scenario 
to hit out-of-space with a kernel lower than 4.6 is to use a database 
(postgresql, mongo etc.) and to snapshot the volume. Ubuntu users can 
download kernel packages from 
http://kernel.ubuntu.com/~kernel-ppa/mainline/


3) had some really bad experiences with btrfs quotas stability in older 
kernels, and judging by the amount of changes in this area on 
linux-btrfs mailing list, I'd rather wait a few stable kernels than use 
it again


4) if you use databases - you should chattr +C database dir, otherwise, 
performance will suffer. Please remember that chattr +C does not have 
any effect on existing files, so you might need to stop your database, 
copy the files out, chattr +C the database dir, copy the files in


Other than that - works fine, snapshots are very useful.

It's hard to me to say what's "more stable" on Linux (btrfs or zfs); my 
bets would be btrfs getting more attention in the coming year, as it's 
getting its remaining bugs fixed.


Tomasz Chmielewski
http://wpkg.org




On 2016-05-16 20:20, Ron Kelley wrote:
> I tried ZFS on various linux/FreeBSD builds in the past and the
> performance was aweful.  It simply required too much RAM to perform
> properly.  This is why I went the BTRFS route.
> 
> Maybe I should look at ZFS again on Ubuntu 16.04...
> 
> 
> 
> On 5/16/2016 6:59 AM, Fajar A. Nugraha wrote:
>> On Mon, May 16, 2016 at 5:38 PM, Ron Kelley <rkelleyrtp at gmail.com> 
>> wrote:
>>> For what's worth, I use BTRFS, and it works great.
>> 
>> Btrfs also works in nested lxd, so if that's your primary use I highly
>> recommend btrfs.
>> 
>> Of course, you could also keep using zfs-backed containers, but
>> manually assign a zvol-formatted-as-btrfs for first-level-container's
>> /var/lib/lxd.
>> 
>>>  Container copies are almost instant.  I can use compression with 
>>> minimal overhead,
>> 
>> zfs and btrfs are almost identical in that aspect (snapshot/clone, and
>> lz4 vs lzop in compression time and ratio). However, lz4 (used in zfs)
>> is MUCH faster at decompression compared to lzop (used in btrfs),
>> while lzop uses less memory.
>> 
>>> use quotas to limit container disk space,
>> 
>> zfs does that too
>> 
>>> and can schedule a deduplication task via cron to save even more 
>>> space.
>> 
>> That is, indeed, only available in btrfs
>> 
> _______________________________________________
> lxc-users mailing list
> lxc-users at lists.linuxcontainers.org
> http://lists.linuxcontainers.org/listinfo/lxc-users