[lxc-users] zombie process blocks stopping of container

Tamas Papp tompos at martos.bme.hu
Sun Jun 8 15:47:38 UTC 2014


On 06/03/2014 05:34 PM, Tamas Papp wrote:
>
> On 06/03/2014 05:08 PM, Stéphane Graber wrote:
>> On Tue, Jun 03, 2014 at 04:56:03PM +0200, Tamas Papp wrote:
>>> On 06/03/2014 04:50 PM, Stéphane Graber wrote:
>>>> lxc-stop will send SIGPWR (or the equivalent signal) to the container,
>>>> wait 30s then SIGKILL init. lxc-stop -k will skip the SIGPWR step,
>>>> lxc-stop --nokill will skip the SIGKILL step.
>>>>
>>>> It's pretty odd that init after a kill -9 is still marked 
>>>> running... I'd
>>>> have expected it to either go away or get stuck in D state if
>>>> something's really wrong...
>>>>
>>>> Do you see anything relevant in the kernel log?
>>> Nothing. I was in hurry, so I restarted the whole machine, I cannot
>>> collect more information.
>>> Unfortunately I'm pretty sure it will be back soon, since this was
>>> not the first time.
>>> What do you suggest, what should I check, when I face it again?
>> So my hope would be for the kernel to report the task as hung which
>> causes a stacktrace to be dumped in dmesg. If not, then it's going to be
>> a bit harder to figure it out...
>>
>
> Is this valuable?
>
> [514047.425278] ---[ end trace 3d2c1319330f8514 ]---
> [514047.469086] kernel BUG at 
> /build/buildd/linux-3.13.0/mm/memory.c:3756!
> [514047.490011] invalid opcode: 0000 [#10] SMP
> [514047.510486] Modules linked in: joydev hid_generic usbhid hid 
> binfmt_misc veth vhost_net vhost macvtap macvlan xt_conntrack 
> ipt_REJECT ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle 
> ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
> nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables 
> x_tables bridge stp llc gpio_ich x86_pkg_temp_thermal intel_powerclamp 
> coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul 
> ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper 
> ablk_helper cryptd serio_raw sb_edac edac_core lpc_ich hpwdt hpilo 
> ioatdma ipmi_si mac_hid acpi_power_meter lp parport zfs(POF) 
> zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10 
> raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor 
> raid6_pq raid0 multipath linear igb i2c_algo_bit dca ahci ptp raid1 
> psmouse libahci pps_core hpsa
> [514047.750609] CPU: 2 PID: 25087 Comm: java Tainted: PF     D O 
> 3.13.0-27-generic #50-Ubuntu
> [514047.795328] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83 
> 12/20/2013
> [514047.818632] task: ffff88175a5fc7d0 ti: ffff88176180e000 task.ti: 
> ffff88176180e000
> [514047.865496] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>] 
> handle_mm_fault+0xe61/0xf10
> [514047.914325] RSP: 0018:ffff88176180fd98  EFLAGS: 00010246
> [514047.939354] RAX: 0000000000000100 RBX: 00000007ff41a730 RCX: 
> ffff88176180fb10
> [514047.989807] RDX: ffff88175a5fc7d0 RSI: 0000000000000000 RDI: 
> 8000000cf2a009e6
> [514048.040666] RBP: ffff88176180fe20 R08: 0000000000000000 R09: 
> 00000000000000a9
> [514048.092282] R10: 0000000000000001 R11: 0000000000000000 R12: 
> ffff881765ea5fd0
> [514048.145498] R13: ffff88176ac77080 R14: ffff8802711ee200 R15: 
> 0000000000000080
> [514048.199709] FS:  00007f83947f7700(0000) GS:ffff88103fc40000(0000) 
> knlGS:0000000000000000
> [514048.254465] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [514048.281837] CR2: 00007f6fdd6cec58 CR3: 0000000275364000 CR4: 
> 00000000001427e0
> [514048.335499] Stack:
> [514048.361406]  ffff88175a5fc7d0 0000000000000000 00007f839802f000 
> ffff8802711ee200
> [514048.413528]  0000000000000f54 0000000000000000 0000000000000000 
> ffffea001e3c7000
> [514048.465624]  800000078f1c0867 ffffea0043985670 ffffea00000000a9 
> ffff88176180fe00
> [514048.517663] Call Trace:
> [514048.542981]  [<ffffffff81725924>] __do_page_fault+0x184/0x560
> [514048.568584]  [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
> [514048.593936]  [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
> [514048.618792]  [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
> [514048.643188]  [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
> [514048.667032]  [<ffffffff81722188>] page_fault+0x28/0x30
> [514048.690535] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d 
> c8 e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9 
> 68 f3 ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8 
> 98 e3
> [514048.763043] RIP  [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
> [514048.786554]  RSP <ffff88176180fd98>
> [514048.809155] ------------[ cut here ]------------
> [514048.809343] ---[ end trace 3d2c1319330f8515 ]---
> [514048.856623] kernel BUG at 
> /build/buildd/linux-3.13.0/mm/memory.c:3756!
> [514048.879824] invalid opcode: 0000 [#11] SMP
> [514048.902099] Modules linked in: joydev hid_generic usbhid hid 
> binfmt_misc veth vhost_net vhost macvtap macvlan xt_conntrack 
> ipt_REJECT ip6table_filter ip6_tables xt_CHECKSUM iptable_mangle 
> ipt_MASQUERADE iptable_nat nf_conntrack_ipv4 nf_defrag_ipv4 
> nf_nat_ipv4 nf_nat nf_conntrack xt_tcpudp iptable_filter ip_tables 
> x_tables bridge stp llc gpio_ich x86_pkg_temp_thermal intel_powerclamp 
> coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul 
> ghash_clmulni_intel aesni_intel aes_x86_64 lrw gf128mul glue_helper 
> ablk_helper cryptd serio_raw sb_edac edac_core lpc_ich hpwdt hpilo 
> ioatdma ipmi_si mac_hid acpi_power_meter lp parport zfs(POF) 
> zunicode(POF) zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10 
> raid456 async_raid6_recov async_memcpy async_pq async_xor async_tx xor 
> raid6_pq raid0 multipath linear igb i2c_algo_bit dca ahci ptp raid1 
> psmouse libahci pps_core hpsa
> [514049.154915] CPU: 16 PID: 25089 Comm: java Tainted: PF     D O 
> 3.13.0-27-generic #50-Ubuntu
> [514049.200777] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83 
> 12/20/2013
> [514049.224243] task: ffff88175a5fafe0 ti: ffff88175a724000 task.ti: 
> ffff88175a724000
> [514049.271305] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>] 
> handle_mm_fault+0xe61/0xf10
> [514049.320379] RSP: 0000:ffff88175a725d98  EFLAGS: 00010246
> [514049.345393] RAX: 0000000000000100 RBX: 00000007ff412730 RCX: 
> ffff88175a725b10
> [514049.396064] RDX: ffff88175a5fafe0 RSI: 0000000000000000 RDI: 
> 8000000cf2a009e6
> [514049.446897] RBP: ffff88175a725e20 R08: 0000000000000000 R09: 
> 00000000000000a9
> [514049.498535] R10: 0000000000000001 R11: 0000000000000000 R12: 
> ffff881765ea5fd0
> [514049.551984] R13: ffff88176ac77080 R14: ffff8802711ee200 R15: 
> 0000000000000080
> [514049.606194] FS:  00007f83945f5700(0000) GS:ffff88103fd40000(0000) 
> knlGS:0000000000000000
> [514049.661165] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> [514049.688513] CR2: 00000007ff412730 CR3: 0000000275364000 CR4: 
> 00000000001427e0
> [514049.742387] Stack:
> [514049.768268]  0000000000000001 ffff88175a725db0 ffffffff8109a780 
> ffff88175a725dd0
> [514049.820366]  ffffffff810d7ad6 0000000000000001 ffffffff81f1f810 
> ffffea0025998400
> [514049.872439]  8000000966610867 ffffea0043985670 ffffea00000000a9 
> 00000001ffffffff
> [514049.924437] Call Trace:
> [514049.949730]  [<ffffffff8109a780>] ? wake_up_state+0x10/0x20
> [514049.975243]  [<ffffffff810d7ad6>] ? wake_futex+0x66/0x90
> [514050.000224]  [<ffffffff81725924>] __do_page_fault+0x184/0x560
> [514050.024857]  [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
> [514050.049169]  [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
> [514050.072946]  [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
> [514050.096475]  [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
> [514050.119657]  [<ffffffff81722188>] page_fault+0x28/0x30
> [514050.142278] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d 
> c8 e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9 
> 68 f3 ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8 
> 98 e3
> [514050.211451] RIP  [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
> [514050.233676]  RSP <ffff88175a725d98>
> [514050.255620] ---[ end trace 3d2c1319330f8516 ]---
> [516197.062287] init: lxc-instance (fisheye1) main process (4489) 
> killed by KILL signal


Now it happened again, with a different container.

I see a lot of stuck processes, this time even a 'ps x' gets stuck:

  1677 ?        Ss     0:07 /lib/systemd/systemd-logind
  1687 ?        S      0:00 upstart-file-bridge --daemon
  1695 ?        Ss     0:44 /usr/bin/perl -wT /usr/sbin/munin-node
  3022 ?        S      0:00 [kworker/u130:5]
  3035 tty4     Ss+    0:00 /sbin/getty -8 38400 tty4
  3039 tty5     Ss+    0:00 /sbin/getty -8 38400 tty5
  3052 tty2     Ss+    0:00 /sbin/getty -8 38400 tty2
  3058 tty3     Ss+    0:00 /sbin/getty -8 38400 tty3
  3064 tty6     Ss+    0:00 /sbin/getty -8 38400 tty6
  3066 ?        S      0:00 [kworker/11:0]
  3096 ?        Ss     0:00 /usr/sbin/sshd -D
  3112 ?        Ss     0:01 cron
  3115 ?        Ss     0:00 sshd: tompos [priv]
  3159 pts/0    S      0:00 sudo -i
  3166 ?        Ss     1:25 /usr/sbin/irqbalance
  3189 pts/0    S+     0:00 -bash
  3200 ?        Sl     0:00 /usr/sbin/libvirtd -d
  3494 ?        Ss     0:02 /usr/lib/postfix/master
  3553 ?        S<     0:00 [kworker/u132:2]
  3566 ?        Ss     0:00 lxc-start -n chemicalize
  3699 ?        Ss     0:02 [init]
  4557 ?        Ss     0:00 /sbin/mdadm --monitor --pid-file 
/run/mdadm/monitor.pid --daemonise --scan --syslog
  4629 tty1     Ss+    0:00 /sbin/getty -8 38400 tty1
  4693 ?        D      0:00 pidof /usr/sbin/apache2
  4865 ?        S      0:00 [kauditd]
  4889 ?        S      0:07 [vhost-4888]
  4892 ?        S<     0:00 [zil_clean/0]
  4895 ?        S      0:00 [kvm-pit/4888]
  4921 ?        S      4:38 [vhost-4920]
  4924 ?        S<     0:00 [zil_clean/0]
  4927 ?        S      0:00 [kvm-pit/4920]
  5092 ?        S<     0:00 [kworker/u133:0]
  5163 ?        S      0:00 [kworker/u130:6]
  5460 ?        S      0:00 [kworker/5:2]
  5466 ?        S      0:00 [kworker/1:2]
  5571 ?        S<     0:00 [kworker/u132:1]
  6007 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
  6010 ?        Z      0:00 [lsof] <defunct>
  6723 ?        S      0:01 [kworker/10:2]
  6725 ?        S      0:00 [kworker/7:1]
  6740 ?        S      0:00 [kworker/15:2]
  6750 ?        S      0:00 [kworker/16:0]
  7215 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
  7219 ?        Z      0:00 [lsof] <defunct>
  7938 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
  7942 ?        Z      0:00 [lsof] <defunct>
  7955 ?        S      0:00 [kworker/u129:0]
  8856 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
  8859 ?        Z      0:00 [lsof] <defunct>
  9477 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
  9480 ?        Z      0:00 [lsof] <defunct>
10601 ?        S      0:00 [kworker/3:0]
10750 ?        D      0:00 /usr/bin/lsof -w -l +d /var/lib/php5
10753 ?        Z      0:00 [lsof] <defunct>
11351 ?        D      0:00 ps axf
11575 ?        S      0:00 sudo -i
11599 ?        S      0:00 -bash
11636 ?        D      0:00 ps ax
12056 ?        S      0:00 sudo -i
12079 ?        S      0:00 -bash
12417 ?        S      1:21 [kworker/9:0]
12735 ?        S      0:00 [kworker/8:0]
13341 ?        S      0:00 [kworker/12:2]
13367 ?        S      0:00 [kworker/21:1]
13377 ?        S      0:00 [kworker/4:0]
13651 ?        S      0:00 [kworker/u129:1]
14237 ?        S      7:03 [kworker/10:1]
15450 ?        S      0:00 [kworker/6:0]
20520 ?        S      0:00 [kworker/u130:0]
21166 ?        S      0:00 [kworker/u129:3]
21324 ?        S      1:57 [kworker/6:2]
21617 ?        S      0:25 [kworker/21:2]
22163 ?        S      0:00 [kworker/u129:4]
23529 ?        D      0:00 pgrep -fa lxc-stop
23981 ?        S      0:00 sudo -i
24026 ?        S      0:00 -bash
24594 ?        D      0:00 ps axf
24740 ?        S      3:10 [kworker/16:1]
24796 ?        S      0:00 sudo -i
24825 ?        S      0:00 -bash
24869 ?        D      0:00 ps ax
25155 ?        Ss     0:00 sshd: tompos [priv]
25316 pts/11   S      0:00 sudo -i
25341 pts/11   S      0:00 -bash
25449 ?        Ss     0:00 /usr/lib/x86_64-linux-gnu/lxc/lxc-monitord 
/tank/lxc 5


Now I have a dmesg output, probably everything started, when this happened:

[423094.126247] ------------[ cut here ]------------
[423094.152443] kernel BUG at /build/buildd/linux-3.13.0/mm/memory.c:3756!
[423094.178742] invalid opcode: 0000 [#1] SMP
[423094.204554] Modules linked in: binfmt_misc vhost_net vhost macvtap 
macvlan ipt_MASQUERADE iptable_nat nf_nat_ipv4 nf_nat nf_conntrack_ipv4 
nf_defrag_ipv4 xt_conntrack nf_conntrack ipt_REJECT xt_CHECKSUM 
iptable_mangle xt_tcpudp veth ip6table_filter ip6_tables iptable_filter 
ip_tables x_tables bridge stp llc x86_pkg_temp_thermal intel_powerclamp 
coretemp kvm_intel kvm crct10dif_pclmul crc32_pclmul ghash_clmulni_intel 
gpio_ich aesni_intel aes_x86_64 lrw gf128mul sb_edac glue_helper 
serio_raw lp hpwdt ablk_helper lpc_ich hpilo cryptd ioatdma edac_core 
ipmi_si acpi_power_meter mac_hid parport zfs(POF) zunicode(POF) 
zavl(POF) zcommon(POF) znvpair(POF) spl(OF) raid10 raid456 
async_raid6_recov async_memcpy async_pq async_xor async_tx xor raid6_pq 
raid0 multipath linear igb i2c_algo_bit dca raid1 ahci ptp psmouse 
libahci pps_core hpsa
[423094.495398] CPU: 13 PID: 11939 Comm: java Tainted: PF          O 
3.13.0-27-generic #50-Ubuntu
[423094.548747] Hardware name: HP ProLiant SL210t Gen8/, BIOS P83 12/20/2013
[423094.574988] task: ffff881ca7ec47d0 ti: ffff88163f05a000 task.ti: 
ffff88163f05a000
[423094.627597] RIP: 0010:[<ffffffff811793d1>] [<ffffffff811793d1>] 
handle_mm_fault+0xe61/0xf10
[423094.678564] RSP: 0000:ffff88163f05bd98  EFLAGS: 00010246
[423094.703767] RAX: 0000000000000100 RBX: 00000007d0200000 RCX: 
ffff88163f05bb10
[423094.754679] RDX: ffff881ca7ec47d0 RSI: 0000000000000000 RDI: 
8000000571c009e6
[423094.807019] RBP: ffff88163f05be20 R08: 0000000000000000 R09: 
00000000000000a9
[423094.859710] R10: 0000000000000001 R11: 0000000000000000 R12: 
ffff88133954a408
[423094.912560] R13: ffff880320a12cc0 R14: ffff881219b4d400 R15: 
0000000000000080
[423094.966237] FS:  00007fc5f1825700(0000) GS:ffff88103fce0000(0000) 
knlGS:0000000000000000
[423095.020459] CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
[423095.047460] CR2: 00000007d0200000 CR3: 00000017cd0d9000 CR4: 
00000000001427e0
[423095.100677] Stack:
[423095.126512]  0000000000000001 ffff88163f05bdb0 ffffffff8109a780 
ffff88163f05bdd0
[423095.177861]  ffffffff810d7ad6 0000000000000001 ffffffff81f1ebb0 
ffff88163f05be78
[423095.229467]  ffffffff810d983d ffff88163f05be48 00000000000000a9 
00000001ffffffff
[423095.280978] Call Trace:
[423095.306138]  [<ffffffff8109a780>] ? wake_up_state+0x10/0x20
[423095.331531]  [<ffffffff810d7ad6>] ? wake_futex+0x66/0x90
[423095.356962]  [<ffffffff810d983d>] ? futex_wake_op+0x4ed/0x620
[423095.381663]  [<ffffffff81725924>] __do_page_fault+0x184/0x560
[423095.405532]  [<ffffffff811112ec>] ? acct_account_cputime+0x1c/0x20
[423095.429089]  [<ffffffff8109d76b>] ? account_user_time+0x8b/0xa0
[423095.452361]  [<ffffffff8109dd84>] ? vtime_account_user+0x54/0x60
[423095.475382]  [<ffffffff81725d1a>] do_page_fault+0x1a/0x70
[423095.498153]  [<ffffffff81722188>] page_fault+0x28/0x30
[423095.520058] Code: ff 48 89 d9 4c 89 e2 4c 89 ee 4c 89 f7 44 89 4d c8 
e8 34 c1 ff ff 85 c0 0f 85 94 f5 ff ff 49 8b 3c 24 44 8b 4d c8 e9 68 f3 
ff ff <0f> 0b be 8e 00 00 00 48 c7 c7 f8 39 a6 81 44 89 4d c8 e8 98 e3
[423095.586301] RIP  [<ffffffff811793d1>] handle_mm_fault+0xe61/0xf10
[423095.607798]  RSP <ffff88163f05bd98>
[423095.659448] ---[ end trace a2f5bece5536224a ]---


But I'm not sure, in syslog I can see only this:

Jun  8 13:44:17 v306 kernel: [423094.912560] R13: ffff880320a12cc0 R14: 
ffff881219b4d400 R15: 0000000000000080
Jun  8 13:44:17 v306 kernel: [423095.498153] [<ffffffff81722188>] 
page_fault+0x28/0x30


But I can see puppet activity in syslog after this point, so I'm not 
sure at all.




$ echo m /proc/sysrq-trigger

[436815.646569] SysRq : Show Memory
[436815.671215] Mem-Info:
[436815.695291] Node 0 DMA per-cpu:
[436815.720109] CPU    0: hi:    0, btch:   1 usd:   0
[436815.744586] CPU    1: hi:    0, btch:   1 usd:   0
[436815.768253] CPU    2: hi:    0, btch:   1 usd:   0
[436815.791281] CPU    3: hi:    0, btch:   1 usd:   0
[436815.813492] CPU    4: hi:    0, btch:   1 usd:   0
[436815.834925] CPU    5: hi:    0, btch:   1 usd:   0
[436815.855547] CPU    6: hi:    0, btch:   1 usd:   0
[436815.875366] CPU    7: hi:    0, btch:   1 usd:   0
[436815.895133] CPU    8: hi:    0, btch:   1 usd:   0
[436815.914074] CPU    9: hi:    0, btch:   1 usd:   0
[436815.932275] CPU   10: hi:    0, btch:   1 usd:   0
[436815.949823] CPU   11: hi:    0, btch:   1 usd:   0
[436815.967354] CPU   12: hi:    0, btch:   1 usd:   0
[436815.984089] CPU   13: hi:    0, btch:   1 usd:   0
[436816.000033] CPU   14: hi:    0, btch:   1 usd:   0
[436816.015874] CPU   15: hi:    0, btch:   1 usd:   0
[436816.031694] CPU   16: hi:    0, btch:   1 usd:   0
[436816.046706] CPU   17: hi:    0, btch:   1 usd:   0
[436816.060922] CPU   18: hi:    0, btch:   1 usd:   0
[436816.074508] CPU   19: hi:    0, btch:   1 usd:   0
[436816.087294] CPU   20: hi:    0, btch:   1 usd:   0
[436816.099308] CPU   21: hi:    0, btch:   1 usd:   0
[436816.110240] CPU   22: hi:    0, btch:   1 usd:   0
[436816.120043] CPU   23: hi:    0, btch:   1 usd:   0
[436816.128585] Node 0 DMA32 per-cpu:
[436816.136225] CPU    0: hi:  186, btch:  31 usd:  37
[436816.143444] CPU    1: hi:  186, btch:  31 usd: 174
[436816.149946] CPU    2: hi:  186, btch:  31 usd: 100
[436816.156165] CPU    3: hi:  186, btch:  31 usd:  53
[436816.162173] CPU    4: hi:  186, btch:  31 usd: 109
[436816.167750] CPU    5: hi:  186, btch:  31 usd: 173
[436816.173344] CPU    6: hi:  186, btch:  31 usd:   0
[436816.178987] CPU    7: hi:  186, btch:  31 usd:   0
[436816.184628] CPU    8: hi:  186, btch:  31 usd:   0
[436816.190250] CPU    9: hi:  186, btch:  31 usd:   0
[436816.195878] CPU   10: hi:  186, btch:  31 usd:   0
[436816.201506] CPU   11: hi:  186, btch:  31 usd:   0
[436816.207166] CPU   12: hi:  186, btch:  31 usd:  79
[436816.212843] CPU   13: hi:  186, btch:  31 usd: 164
[436816.218555] CPU   14: hi:  186, btch:  31 usd: 165
[436816.224265] CPU   15: hi:  186, btch:  31 usd: 156
[436816.229992] CPU   16: hi:  186, btch:  31 usd: 125
[436816.235737] CPU   17: hi:  186, btch:  31 usd:  71
[436816.241529] CPU   18: hi:  186, btch:  31 usd:   0
[436816.247343] CPU   19: hi:  186, btch:  31 usd:   0
[436816.253138] CPU   20: hi:  186, btch:  31 usd:   0
[436816.258967] CPU   21: hi:  186, btch:  31 usd:   0
[436816.264778] CPU   22: hi:  186, btch:  31 usd:   0
[436816.270588] CPU   23: hi:  186, btch:  31 usd:   0
[436816.276379] Node 0 Normal per-cpu:
[436816.282375] CPU    0: hi:  186, btch:  31 usd: 129
[436816.289083] CPU    1: hi:  186, btch:  31 usd:  77
[436816.295807] CPU    2: hi:  186, btch:  31 usd: 175
[436816.302594] CPU    3: hi:  186, btch:  31 usd:  64
[436816.308984] CPU    4: hi:  186, btch:  31 usd: 111
[436816.314912] CPU    5: hi:  186, btch:  31 usd: 161
[436816.320789] CPU    6: hi:  186, btch:  31 usd:   0
[436816.326713] CPU    7: hi:  186, btch:  31 usd:   0
[436816.332553] CPU    8: hi:  186, btch:  31 usd:   0
[436816.338343] CPU    9: hi:  186, btch:  31 usd:   0
[436816.344081] CPU   10: hi:  186, btch:  31 usd:   0
[436816.349794] CPU   11: hi:  186, btch:  31 usd:   0
[436816.355504] CPU   12: hi:  186, btch:  31 usd:  86
[436816.361230] CPU   13: hi:  186, btch:  31 usd: 128
[436816.367007] CPU   14: hi:  186, btch:  31 usd: 154
[436816.372806] CPU   15: hi:  186, btch:  31 usd:  74
[436816.378602] CPU   16: hi:  186, btch:  31 usd: 180
[436816.384414] CPU   17: hi:  186, btch:  31 usd: 168
[436816.390208] CPU   18: hi:  186, btch:  31 usd:   0
[436816.396054] CPU   19: hi:  186, btch:  31 usd:  25
[436816.401902] CPU   20: hi:  186, btch:  31 usd:   0
[436816.407797] CPU   21: hi:  186, btch:  31 usd:   0
[436816.413628] CPU   22: hi:  186, btch:  31 usd:   0
[436816.419424] CPU   23: hi:  186, btch:  31 usd:   0
[436816.425204] Node 1 Normal per-cpu:
[436816.431188] CPU    0: hi:  186, btch:  31 usd:   0
[436816.437898] CPU    1: hi:  186, btch:  31 usd:   0
[436816.444571] CPU    2: hi:  186, btch:  31 usd:   0
[436816.451260] CPU    3: hi:  186, btch:  31 usd:   0
[436816.457486] CPU    4: hi:  186, btch:  31 usd: 155
[436816.463282] CPU    5: hi:  186, btch:  31 usd:  56
[436816.469060] CPU    6: hi:  186, btch:  31 usd:  59
[436816.474803] CPU    7: hi:  186, btch:  31 usd:  55
[436816.480493] CPU    8: hi:  186, btch:  31 usd: 178
[436816.486202] CPU    9: hi:  186, btch:  31 usd: 115
[436816.491926] CPU   10: hi:  186, btch:  31 usd:  29
[436816.497670] CPU   11: hi:  186, btch:  31 usd:  39
[436816.503446] CPU   12: hi:  186, btch:  31 usd:   0
[436816.509287] CPU   13: hi:  186, btch:  31 usd:  18
[436816.515183] CPU   14: hi:  186, btch:  31 usd:  74
[436816.521080] CPU   15: hi:  186, btch:  31 usd:   0
[436816.527007] CPU   16: hi:  186, btch:  31 usd:   0
[436816.532882] CPU   17: hi:  186, btch:  31 usd:   0
[436816.538709] CPU   18: hi:  186, btch:  31 usd: 135
[436816.544554] CPU   19: hi:  186, btch:  31 usd: 164
[436816.550421] CPU   20: hi:  186, btch:  31 usd: 167
[436816.556303] CPU   21: hi:  186, btch:  31 usd: 123
[436816.562164] CPU   22: hi:  186, btch:  31 usd: 116
[436816.568012] CPU   23: hi:  186, btch:  31 usd:  50
[436816.573882] active_anon:2919029 inactive_anon:27295 isolated_anon:0
[436816.573882]  active_file:801196 inactive_file:1948997 isolated_file:0
[436816.573882]  unevictable:0 dirty:0 writeback:0 unstable:0
[436816.573882]  free:12852299 slab_reclaimable:283058 
slab_unreclaimable:276326
[436816.573882]  mapped:15018 shmem:557 pagetables:8924 bounce:0
[436816.573882]  free_cma:0
[436816.624593] Node 0 DMA free:15828kB min:8kB low:8kB high:12kB 
active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB 
unevictable:0kB isolated(anon):0kB isolated(file):0kB present:15976kB 
managed:15892kB mlocked:0kB dirty:0kB writeback:0kB mapped:0kB shmem:0kB 
slab_reclaimable:0kB slab_unreclaimable:0kB kernel_stack:0kB 
pagetables:0kB unstable:0kB bounce:0kB free_cma:0kB writeback_tmp:0kB 
pages_scanned:0 all_unreclaimable? yes
[436816.707576] lowmem_reserve[]: 0 1789 64251 64251
[436816.724738] Node 0 DMA32 free:1076484kB min:1252kB low:1564kB 
high:1876kB active_anon:136364kB inactive_anon:16360kB 
active_file:20572kB inactive_file:50928kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:1914544kB 
managed:1835596kB mlocked:0kB dirty:0kB writeback:0kB mapped:456kB 
shmem:28kB slab_reclaimable:13220kB slab_unreclaimable:11172kB 
kernel_stack:40kB pagetables:796kB unstable:0kB bounce:0kB free_cma:0kB 
writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436816.876588] lowmem_reserve[]: 0 0 62462 62462
[436816.902021] Node 0 Normal free:24898544kB min:43708kB low:54632kB 
high:65560kB active_anon:6981544kB inactive_anon:90080kB 
active_file:1865844kB inactive_file:5616424kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:65011712kB 
managed:63961116kB mlocked:16kB dirty:20kB writeback:0kB mapped:17608kB 
shmem:1472kB slab_reclaimable:429048kB slab_unreclaimable:418516kB 
kernel_stack:4760kB pagetables:20188kB unstable:0kB bounce:0kB 
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436817.114072] lowmem_reserve[]: 0 0 0 0
[436817.147814] Node 1 Normal free:25418340kB min:45140kB low:56424kB 
high:67708kB active_anon:4558208kB inactive_anon:2740kB 
active_file:1318368kB inactive_file:2128636kB unevictable:0kB 
isolated(anon):0kB isolated(file):0kB present:67108860kB 
managed:66055976kB mlocked:16kB dirty:0kB writeback:0kB mapped:42008kB 
shmem:728kB slab_reclaimable:689964kB slab_unreclaimable:675616kB 
kernel_stack:2640kB pagetables:14760kB unstable:0kB bounce:0kB 
free_cma:0kB writeback_tmp:0kB pages_scanned:0 all_unreclaimable? no
[436817.395923] lowmem_reserve[]: 0 0 0 0
[436817.431221] Node 0 DMA: 1*4kB (U) 0*8kB 1*16kB (U) 0*32kB 1*64kB (U) 
1*128kB (U) 1*256kB (U) 0*512kB 1*1024kB (U) 1*2048kB (R) 3*4096kB (M) = 
15828kB
[436817.502378] Node 0 DMA32: 4749*4kB (UEM) 4478*8kB (UEM) 2182*16kB 
(UEM) 1562*32kB (UEM) 965*64kB (UEM) 516*128kB (UM) 234*256kB (UM) 
97*512kB (UM) 71*1024kB (UEM) 118*2048kB (MR) 94*4096kB (M) = 1076484kB
[436817.608847] Node 0 Normal: 297772*4kB (UEM) 105932*8kB (UEM) 
30526*16kB (UEM) 11629*32kB (UEM) 5556*64kB (UEM) 3070*128kB (UM) 
2380*256kB (UEM) 1798*512kB (UEM) 1875*1024kB (UM) 3684*2048kB (MR) 
2504*4096kB (M) = 24898704kB
[436817.717184] Node 1 Normal: 234109*4kB (UEM) 129663*8kB (UEM) 
59754*16kB (UEM) 18947*32kB (UEM) 6832*64kB (UEM) 5715*128kB (UEM) 
4586*256kB (UEM) 4855*512kB (UEM) 3978*1024kB (UEM) 2618*2048kB (M) 
1860*4096kB (MR) = 25418348kB
[436817.825583] Node 0 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=2048kB
[436817.897389] Node 1 hugepages_total=0 hugepages_free=0 
hugepages_surp=0 hugepages_size=2048kB
[436817.969947] 2756671 total pagecache pages
[436818.005541] 5919 pages in swap cache
[436818.040308] Swap cache stats: add 91884, delete 85965, find 87132/88084
[436818.075267] Free swap  = 19451316kB
[436818.109350] Total swap = 19514748kB
[436818.142606] 33512773 pages RAM
[436818.175612] 0 pages HighMem/MovableOnly
[436818.208207] 263221 pages reserved



After 'echo t' the machine got instable and hanged up completely.


Can it be HW issue?

tamas


More information about the lxc-users mailing list