[Lxc-users] concurrent aptitude/dpkg runs in separate containers --> bork bork bork
Trent W. Buck
twb at cybersource.com.au
Thu Feb 3 04:58:15 UTC 2011
Daniel Lezcano <daniel.lezcano at free.fr> writes:
> On 01/12/2011 07:39 AM, Trent W. Buck wrote:
>> Mike<debian at good-with-numbers.com> writes:
>>
>>>> Trent W. Buck wrote:
>>>>> I can provision a new LXC container, which includes running a few
>>>>> "aptitude install foo" lines (inside the containers), and it Just Works.
>>>>> If I try to provision two containers at the same time, both containers
>>>>> appear to hang with a dpkg process in the D state[0].
>>>>>
>>>>> My config for each container looks like this:
>>>>> lxc.mount.entry = /home /srv/lxc/proud/home none bind
>>> Doesn't aptitude write into the home directory that you're sharing
>>> across containers? locks ~/.aptitude/cache?
>> Possibly, but dhclient's running aptitude as uid 0, whose home is in
>> /root, which is not shared between hosts.
>>
>> In any case, IIRC I export HOME=`mktemp -d` before aptitude runs, as a
>> fugly workaround for etckeeper.
>>
>> I suppose if containers' /tmp are tmpfses, aptitude might be running
>> into some problem with locking on tmpfs, although I'm not aware of any
>> bogus operational semantics for tmpfs...
>
> Can you give, for each process in D state, the content of
> /proc/<pid>/stack ?
> That may help to understand where the lock occurs in the kernel
I ran into it again today, accidentally.
I was recently reminded that dpkg doesn't mix well with ext4 and btrfs,
because it calls sync/fsync all the time. Now, my containers are on
different ext4 LVs, but maybe >1 process calling f/sync all the time is
enough to slow them both down.
I'm being a bit more patient than last time, and I think they ARE
proceeding, just REALLY slowly. Meanwhile aptitude consumes a 100% of a
core busy-waiting for a response from dpkg :-/
They look like this:
$ ssh omega cat /proc/7713/stack
Warning: Permanently added 'omega,192.168.155.22' (RSA) to the list of known hosts.
[<ffffffff811669b7>] sync_inodes_sb+0x87/0xb0
[<ffffffff8116b292>] __sync_filesystem+0x82/0x90
[<ffffffff8116b379>] sync_filesystems+0xd9/0x130
[<ffffffff8116b431>] sys_sync+0x21/0x40
[<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
$ ssh omega cat /proc/5619/stack
Warning: Permanently added 'omega,192.168.155.22' (RSA) to the list of known hosts.
[<ffffffff81222865>] jbd2_log_wait_commit+0xc5/0x150
[<ffffffff811d7a2c>] ext4_sync_file+0x13c/0x2e0
[<ffffffff8116b051>] vfs_fsync_range+0xa1/0xe0
[<ffffffff8116b0fd>] vfs_fsync+0x1d/0x20
[<ffffffff8116b13e>] do_fsync+0x3e/0x60
[<ffffffff8116b190>] sys_fsync+0x10/0x20
[<ffffffff810121b2>] system_call_fastpath+0x16/0x1b
[<ffffffffffffffff>] 0xffffffffffffffff
I couldn't capture >1 pid from the same ps run before one or the other
vanished. The ps output of D states looks like this:
$ ssh omega lxc-ps auxf | grep ' D '
root 465 0.0 0.0 0 0 ? D Jan12 0:04 \_ [kdmflush]
root 476 0.0 0.0 0 0 ? D Jan12 0:01 \_ [kdmflush]
root 591 0.0 0.0 0 0 ? D Jan12 0:10 \_ [jbd2/dm-0-8]
root 1437 0.0 0.0 0 0 ? D Jan12 0:11 \_ [jbd2/dm-3-8]
root 2278 0.0 0.0 0 0 ? D Jan12 0:21 \_ [kdmflush]
root 2293 0.0 0.0 0 0 ? D Jan12 1:06 \_ [jbd2/dm-21-8]
root 26611 0.0 0.0 0 0 ? D 15:27 0:00 \_ [kdmflush]
root 26635 0.0 0.0 0 0 ? D 15:27 0:00 \_ [jbd2/dm-6-8]
template-amd64 root 4763 0.7 0.0 17708 6064 ? D 15:47 0:00 | | \_ /usr/bin/dpkg --status-fd 34 --unpack --auto-deconfigure /srv/mirror/ubuntu/pool/main/u/udev/udev_151-12.3_amd64.deb /srv/mirror/ubuntu/pool/main/l/lvm2/dmsetup_1.02.39-1ubuntu4.1_amd64.deb /srv/mirror/ubuntu/pool/main/g/glib2.0/libglib2.0-0_2.24.1-0ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libu/libusb/libusb-0.1-4_0.1.12-14ubuntu0.2_amd64.deb /srv/mirror/ubuntu/pool/main/p/plymouth/plymouth_0.8.2-2ubuntu2.2_amd64.deb /srv/mirror/ubuntu/pool/main/libp/libpng/libpng12-0_1.2.42-1ubuntu2.1_amd64.deb /srv/mirror/ubuntu/pool/main/p/plymouth/libplymouth2_0.8.2-2ubuntu2.2_amd64.deb /srv/mirror/ubuntu/pool/main/m/mountall/mountall_2.15.3_amd64.deb /srv/mirror/ubuntu/pool/main/u/upstart/upstart_0.6.5-8_amd64.deb /srv/mirror/ubuntu/pool/main/u/util-linux/bsdutils_2.17.2-0ubuntu1.10.04.2_amd64.deb
greed root 4325 0.7 0.1 14296 8880 ? D 15:45 0:01 | \_ /usr/bin/dpkg --status-fd 35 --unpack --auto-deconfigure /srv/mirror/ubuntu/pool/main/x/xorg/x11-common_7.5+5ubuntu1_all.deb /srv/mirror/ubuntu/pool/main/f/freetype/libfreetype6_2.3.11-1ubuntu2.4_amd64.deb /srv/mirror/ubuntu/pool/main/t/ttf-dejavu/ttf-dejavu-core_2.30-2_all.deb /srv/mirror/ubuntu/pool/main/f/fontconfig/fontconfig-config_2.8.0-2ubuntu1_all.deb /srv/mirror/ubuntu/pool/main/f/fontconfig/libfontconfig1_2.8.0-2ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libf/libfontenc/libfontenc1_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxfixes/libxfixes3_4.0.4-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxdamage/libxdamage1_1.1.2-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxxf86vm/libxxf86vm1_1.1.0-2_amd64.deb /srv/mirror/ubuntu/pool/main/m/mesa/libgl1-mesa-glx_7.7.1-1ubuntu3_amd64.deb /srv/mirror/ubuntu/pool/main/libi/libice/libice6_1.0.6-1_amd64.deb /srv/mirror/ubuntu/pool/main/libs/libsm/libsm6_1.1.1-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxt/libxt6_1.0.7-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxmu/libxmu6_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxpm/libxpm4_3.5.8-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxaw/libxaw7_1.0.7-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxrender/libxrender1_0.9.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/x/xft/libxft2_2.1.14-1ubuntu1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxi/libxi6_1.3-3_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxinerama/libxinerama1_1.1-2_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxtst/libxtst6_1.1.0-2_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxv/libxv1_1.0.5-1_amd64.deb /srv/mirror/ubuntu/pool/main/libx/libxxf86dga/libxxf86dga1_1.1.1-2_amd64.deb
More information about the lxc-users
mailing list