[Lxc-users] concurrent aptitude/dpkg runs in separate containers --> bork bork bork

Trent W. Buck twb at cybersource.com.au
Thu Feb 3 04:58:15 UTC 2011


Daniel Lezcano <daniel.lezcano at free.fr> writes:

> On 01/12/2011 07:39 AM, Trent W. Buck wrote:
>> Mike<debian at good-with-numbers.com>  writes:
>>
>>>> Trent W. Buck wrote:
>>>>> I can provision a new LXC container, which includes running a few
>>>>> "aptitude install foo" lines (inside the containers), and it Just Works.
>>>>> If I try to provision two containers at the same time, both containers
>>>>> appear to hang with a dpkg process in the D state[0].
>>>>>
>>>>> My config for each container looks like this:
>>>>> lxc.mount.entry  = /home       /srv/lxc/proud/home       none bind
>>> Doesn't aptitude write into the home directory that you're sharing
>>> across containers? locks ~/.aptitude/cache?
>> Possibly, but dhclient's running aptitude as uid 0, whose home is in
>> /root, which is not shared between hosts.
>>
>> In any case, IIRC I export HOME=`mktemp -d` before aptitude runs, as a
>> fugly workaround for etckeeper.
>>
>> I suppose if containers' /tmp are tmpfses, aptitude might be running
>> into some problem with locking on tmpfs, although I'm not aware of any
>> bogus operational semantics for tmpfs...
>
> Can you give, for each process in D state, the content of 
> /proc/<pid>/stack ?
> That may help to understand where the lock occurs in the kernel

I ran into it again today, accidentally.

I was recently reminded that dpkg doesn't mix well with ext4 and btrfs,
because it calls sync/fsync all the time.  Now, my containers are on
different ext4 LVs, but maybe >1 process calling f/sync all the time is
enough to slow them both down.

I'm being a bit more patient than last time, and I think they ARE
proceeding, just REALLY slowly.  Meanwhile aptitude consumes a 100% of a
core busy-waiting for a response from dpkg :-/

They look like this:

    $ ssh omega cat /proc/7713/stack
    Warning: Permanently added 'omega,192.168.155.22' (RSA) to the list of known hosts.
    [<ffffffff811669b7>] sync_inodes_sb+0x87/0xb0
    [<ffffffff8116b292>] __sync_filesystem+0x82/0x90
    [<ffffffff8116b379>] sync_filesystems+0xd9/0x130
    [<ffffffff8116b431>] sys_sync+0x21/0x40
    [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
    [<ffffffffffffffff>] 0xffffffffffffffff

    $ ssh omega cat /proc/5619/stack
    Warning: Permanently added 'omega,192.168.155.22' (RSA) to the list of known hosts.
    [<ffffffff81222865>] jbd2_log_wait_commit+0xc5/0x150
    [<ffffffff811d7a2c>] ext4_sync_file+0x13c/0x2e0
    [<ffffffff8116b051>] vfs_fsync_range+0xa1/0xe0
    [<ffffffff8116b0fd>] vfs_fsync+0x1d/0x20
    [<ffffffff8116b13e>] do_fsync+0x3e/0x60
    [<ffffffff8116b190>] sys_fsync+0x10/0x20
    [<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
    [<ffffffffffffffff>] 0xffffffffffffffff

I couldn't capture >1 pid from the same ps run before one or the other
vanished.  The ps output of D states looks like this:

    $ ssh omega lxc-ps auxf  | grep ' D '
               root       465  0.0  0.0      0     0 ?        D    Jan12   0:04  \_ [kdmflush]
               root       476  0.0  0.0      0     0 ?        D    Jan12   0:01  \_ [kdmflush]
               root       591  0.0  0.0      0     0 ?        D    Jan12   0:10  \_ [jbd2/dm-0-8]
               root      1437  0.0  0.0      0     0 ?        D    Jan12   0:11  \_ [jbd2/dm-3-8]
               root      2278  0.0  0.0      0     0 ?        D    Jan12   0:21  \_ [kdmflush]
               root      2293  0.0  0.0      0     0 ?        D    Jan12   1:06  \_ [jbd2/dm-21-8]
               root     26611  0.0  0.0      0     0 ?        D    15:27   0:00  \_ [kdmflush]
               root     26635  0.0  0.0      0     0 ?        D    15:27   0:00  \_ [jbd2/dm-6-8]
    template-amd64 root      4763  0.7  0.0  17708  6064 ?        D    15:47   0:00  |                   |                   \_ /usr/bin/dpkg --status-fd 34 --unpack --auto-deconfigure /srv/mirror/ubuntu/pool/main/u/udev/udev_151-12.3_amd64.deb /srv/mirror/ubuntu/pool/main/l/lvm2/dmsetup_1.02.39-1ubuntu4.1_amd64.deb /srv/mirror/ubuntu/pool/main/g/glib2.0/libglib2.0-0_2.24.1-0ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libu/libusb/libusb-0.1-4_0.1.12-14ubuntu0.2_amd64.deb /srv/mirror/ubuntu/pool/main/p/plymouth/plymouth_0.8.2-2ubuntu2.2_amd64.deb /srv/mirror/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.42-1ubuntu2.1_amd64.deb /srv/mirror/ubuntu/pool/main/p/plymouth/libplymouth2_0.8.2-2ubuntu2.2_amd64.deb /srv/mirror/ubuntu/pool/main/m/mountall/mountall_2.15.3_amd64.deb /srv/mirror/ubuntu/pool/main/u/upstart/upstart_0.6.5-8_amd64.deb /srv/mirror/ubuntu/pool/main/u/util-linux/bsdutils_2.17.2-0ubuntu1.10.04.2_amd64.deb
    greed      root      4325  0.7  0.1  14296  8880 ?        D    15:45   0:01      |                       \_ /usr/bin/dpkg --status-fd 35 --unpack --auto-deconfigure /srv/mirror/ubuntu/pool/main/x/xorg/x11-common_7.5+5ubuntu1_all.deb /srv/mirror/ubuntu/pool/main/f/freetype/libfreetype6_2.3.11-1ubuntu2.4_amd64.deb /srv/mirror/ubuntu/pool/main/t/ttf-dejavu/ttf-dejavu-core_2.30-2_all.deb /srv/mirror/ubuntu/pool/main/f/fontconfig/fontconfig-config_2.8.0-2ubuntu1_all.deb /srv/mirror/ubuntu/pool/main/f/fontconfig/libfontconfig1_2.8.0-2ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libf/libfontenc/libfontenc1_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxfixes/libxfixes3_4.0.4-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxdamage/libxdamage1_1.1.2-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxxf86vm/libxxf86vm1_1.1.0-2_amd64.deb /srv/mirror/ubuntu/pool/main/m/mesa/libgl1-mesa-glx_7.7.1-1ubuntu3_amd64.deb /srv/mirror/ubuntu/pool/main/libi/libice/libice6_1.0.6-1_amd64.deb /srv/mirror/ubuntu/pool/main/libs/libsm/libsm6_1.1.1-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxt/libxt6_1.0.7-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxmu/libxmu6_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxpm/libxpm4_3.5.8-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxaw/libxaw7_1.0.7-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxrender/libxrender1_0.9.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/x/xft/libxft2_2.1.14-1ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxi/libxi6_1.3-3_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1-2_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxtst/libxtst6_1.1.0-2_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxv/libxv1_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxxf86dga/libxxf86dga1_1.1.1-2_amd64.deb






More information about the lxc-users mailing list