<div dir="ltr">Upgraded lxc<div><div>lxc-libs-1.0.3-1.el6.x86_64</div><div>lxc-1.0.3-1.el6.x86_64</div><div><br></div><div>CentOS release 6.3 (Final)<br></div><div><br></div><div><div>uname -a</div><div>Linux myhost 2.6.32-279.el6.x86_64 #1 SMP Fri Jun 22 12:19:21 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux</div>
</div><div><br></div><div><br></div><div>But the problem persists, have had couple of kernel panics.</div><div><br></div><div><div>------------[ cut here ]------------</div><div>kernel BUG at mm/slab.c:533!</div><div>invalid opcode: 0000 [#1] SMP</div>
<div>last sysfs file: /sys/devices/virtual/dmi/id/sys_vendor</div><div>CPU 0</div><div>Modules linked in: veth bridge stp llc ipv6 e1000e(U) sg microcode i2c_i801 iTCO_wdt iTCO_vendor_support shpchp i5000_edac edac_core i5k_amb ioatdma dca ext3 jbd mbcache sd_mod crc_t10dif aacraid pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]</div>
<div><br></div><div>Pid: 0, comm: swapper Tainted: G I--------------- 2.6.32-279.el6.x86_64 #1 Supermicro X7DVL/X7DVL</div><div>RIP: 0010:[<ffffffff81163f75>] [<ffffffff81163f75>] free_block+0x165/0x170</div>
<div>RSP: 0018:ffff8800282032d0 EFLAGS: 00010046</div><div>RAX: ffffea000a54e368 RBX: ffff88042fcf03c0 RCX: 0000000000000010</div><div>RDX: 0040000000000000 RSI: ffff8802f3bb6d40 RDI: ffff8802f3aeb000</div><div>RBP: ffff880028203320 R08: ffffea000e79b720 R09: 0000000000000000</div>
<div>R10: 0000000000000000 R11: 0000000080042000 R12: 000000000000000c</div><div>R13: ffff88042fea13a8 R14: 0000000000000002 R15: ffffea0000000000</div><div>FS: 0000000000000000(0000) GS:ffff880028200000(0000) knlGS:0000000000000000</div>
<div>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b</div><div>CR2: 00007fc077681000 CR3: 000000042216f000 CR4: 00000000000006f0</div><div>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div><div>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div>
<div>Process swapper (pid: 0, threadinfo ffffffff81a00000, task ffffffff81a8d020)</div><div>Stack:</div><div> ffff88042fc216c0 ffff8802f3bb6d40 000000000000100c ffff8802f3aeb000</div><div><d> ffff880028203360 ffff8802f3bc4000 ffff88042fea1380 0000000000000286</div>
<div><d> ffff88042fcf03c0 ffff88042fea1398 ffff880028203390 ffffffff81164500</div><div>Call Trace:</div><div> <IRQ></div><div> [<ffffffff81164500>] kfree+0x310/0x320</div><div> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210</div>
<div> [<ffffffff8142fef8>] skb_release_data+0xd8/0x110</div><div> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210</div><div> [<ffffffff8142fa2e>] __kfree_skb+0x1e/0xa0</div><div> [<ffffffff8142fb72>] kfree_skb+0x42/0x90</div>
<div> [<ffffffff8143c949>] enqueue_to_backlog+0x179/0x210</div><div> [<ffffffff8143fb20>] netif_rx+0xb0/0x160</div><div> [<ffffffff8143fe32>] dev_forward_skb+0x122/0x180</div><div> [<ffffffffa03446e6>] veth_xmit+0x86/0xe0 [veth]</div>
<div> [<ffffffff8143b0cc>] dev_hard_start_xmit+0x2bc/0x3f0</div><div> [<ffffffff81458c1a>] sch_direct_xmit+0x15a/0x1c0</div><div> [<ffffffff8143f878>] dev_queue_xmit+0x4f8/0x6f0</div><div> [<ffffffffa03276bc>] br_dev_queue_push_xmit+0x6c/0xa0 [bridge]</div>
<div> [<ffffffffa032d378>] br_nf_dev_queue_xmit+0x28/0xa0 [bridge]</div><div> [<ffffffffa032de10>] br_nf_post_routing+0x1d0/0x280 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div>
[<ffffffffa0327650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]</div>
<div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div><div> [<ffffffffa0327650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]</div><div> [<ffffffffa03276f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div>
[<ffffffffa0327733>] br_forward_finish+0x43/0x60 [bridge]</div><div> [<ffffffffa032d9b8>] br_nf_forward_finish+0x128/0x140 [bridge]</div><div> [<ffffffffa032eea8>] ? br_nf_forward_ip+0x318/0x3c0 [bridge]</div>
<div> [<ffffffffa032eea8>] br_nf_forward_ip+0x318/0x3c0 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div> [<ffffffffa03276f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div>
<div> [<ffffffffa03276f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div> [<ffffffffa0327750>] ? __br_forward+0x0/0xc0 [bridge]</div><div> [<ffffffffa03277c2>] __br_forward+0x72/0xc0 [bridge]</div><div>
[<ffffffffa0327601>] br_flood+0xc1/0xd0 [bridge]</div><div> [<ffffffffa0327625>] br_flood_forward+0x15/0x20 [bridge]</div><div> [<ffffffffa03287ae>] br_handle_frame_finish+0x27e/0x2a0 [bridge]</div><div>
[<ffffffffa032e318>] br_nf_pre_routing_finish+0x228/0x340 [bridge]</div><div> [<ffffffffa032e88f>] br_nf_pre_routing+0x45f/0x760 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div>
[<ffffffffa0328530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]</div>
<div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div><div> [<ffffffffa0328530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]</div><div> [<ffffffffa032895c>] br_handle_frame+0x18c/0x250 [bridge]</div><div>
[<ffffffff8143a839>] __netif_receive_skb+0x519/0x6f0</div><div> [<ffffffff8143ca38>] netif_receive_skb+0x58/0x60</div><div> [<ffffffff8143cbe4>] napi_gro_complete+0x84/0xe0</div><div> [<ffffffff8143ce0b>] dev_gro_receive+0x1cb/0x290</div>
<div> [<ffffffff8143cf4b>] __napi_gro_receive+0x7b/0x170</div><div> [<ffffffff8143f06f>] napi_gro_receive+0x2f/0x50</div><div> [<ffffffffa027233b>] e1000_receive_skb+0x5b/0x90 [e1000e]</div><div> [<ffffffffa0275601>] e1000_clean_rx_irq+0x241/0x4c0 [e1000e]</div>
<div> [<ffffffffa027cb8d>] e1000e_poll+0x8d/0x380 [e1000e]</div><div> [<ffffffff8143aaaa>] ? process_backlog+0x9a/0x100</div><div> [<ffffffff8143f193>] net_rx_action+0x103/0x2f0</div><div> [<ffffffff81073ec1>] __do_softirq+0xc1/0x1e0</div>
<div> [<ffffffff810db800>] ? handle_IRQ_event+0x60/0x170</div><div> [<ffffffff8100c24c>] call_softirq+0x1c/0x30</div><div> [<ffffffff8100de85>] do_softirq+0x65/0xa0</div><div> [<ffffffff81073ca5>] irq_exit+0x85/0x90</div>
<div> [<ffffffff81505af5>] do_IRQ+0x75/0xf0</div><div> [<ffffffff8100ba53>] ret_from_intr+0x0/0x11</div><div> <EOI></div><div> [<ffffffff81014877>] ? mwait_idle+0x77/0xd0</div><div> [<ffffffff8150338a>] ? atomic_notifier_call_chain+0x1a/0x20</div>
<div> [<ffffffff81009e06>] cpu_idle+0xb6/0x110</div><div> [<ffffffff814e433a>] rest_init+0x7a/0x80</div><div> [<ffffffff81c21f7b>] start_kernel+0x424/0x430</div><div> [<ffffffff81c2133a>] x86_64_start_reservations+0x125/0x129</div>
<div> [<ffffffff81c21438>] x86_64_start_kernel+0xfa/0x109</div><div>Code: 41 5c 41 5d 41 5e 41 5f c9 c3 0f 1f 40 00 48 8b 72 08 48 89 c7 e8 2c f0 11 00 e9 07 ff ff ff 48 8b 40 10 48 8b 10 e9 3e ff ff ff <0f> 0b eb fe 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89</div>
<div>RIP [<ffffffff81163f75>] free_block+0x165/0x170</div><div> RSP <ffff8800282032d0></div></div></div></div><div class="gmail_extra"><br clear="all"><div>ShiB.<br>while ( ! ( succeed = try() ) );</div>
<br><br><div class="gmail_quote">On Sat, May 3, 2014 at 8:14 PM, Michael H. Warfield <span dir="ltr"><<a href="mailto:mhw@wittsend.com" target="_blank">mhw@wittsend.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
On Sat, 2014-05-03 at 19:40 +0530, Shibashish wrote:<br>
> Hi,<br>
<br>
> My server with 4 lxc VM is kernel panicking often. On analyzing the<br>
> crash dump, it shows the following. I have 4 VMs, with 3G memory each,<br>
> memory+swap set at 4G in the croup settings.<br>
<br>
> lxc version: 0.9.0.alpha2<br>
<br>
You definitely need to upgrade that version of LXC.<br>
<br>
1) It's old.<br>
2) It's an alpha version.<br>
><br>
<br>
> I did a hardware swap, but the problem persists. Please let me know<br>
> what to do next. Should I upgrade kernel<br>
> to 2.6.32-431.11.2.el6.centos.plus ?<br>
<br>
I would most definitely update the entire system including and<br>
especially the kernel. Nothing that LXC does should cause a kernel<br>
panic.<br>
><br>
> KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/vmlinux<br>
> DUMPFILE: /var/crash/MailScanner has detected a possible fraud<br>
> attempt from "127.0.0" claiming to be MailScanner warning: numerical<br>
> links are often malicious: 127.0.0.1-2014-05-03-06:11:01/vmcore<br>
> [PARTIAL DUMP]<br>
> CPUS: 8<br>
> DATE: Sat May 3 06:09:12 2014<br>
> UPTIME: 07:56:53<br>
> LOAD AVERAGE: 0.07, 0.06, 0.01<br>
> TASKS: 651<br>
> NODENAME: myhost<br>
> RELEASE: 2.6.32-279.el6.x86_64<br>
> VERSION: #1 SMP Fri Jun 22 12:19:21 UTC 2012<br>
> MACHINE: x86_64 (1866 Mhz)<br>
> MEMORY: 16 GB<br>
> PANIC: "kernel BUG at mm/slab.c:533!"<br>
> PID: 0<br>
> COMMAND: "swapper"<br>
> TASK: ffff880426373540 (1 of 8) [THREAD_INFO:<br>
> ffff880426374000]<br>
> CPU: 7<br>
> STATE: TASK_RUNNING (PANIC)<br>
><br>
><br>
I'm not even sure how this relates to LXC. From what I'm seeing below,<br>
I do see functions in veth and br_*, which could be from an LXC<br>
container, so the fault is failing somewhere down through the bridging<br>
code and into e1000 NIC driver and interrupt handlers. That's a kernel<br>
fault of some sort, and really deep. Definitely upgrade that kernel.<br>
><br>
> ------------[ cut here ]------------<br>
> kernel BUG at mm/slab.c:533!<br>
> invalid opcode: 0000 [#1] SMP<br>
> last sysfs file: /sys/devices/system/cpu/online<br>
> CPU 7<br>
> Modules linked in: veth bridge stp llc ipv6 e1000e(U) sg microcode<br>
> i2c_i801 iTCO_wdt iTCO_vendor_support i5000_edac edac_core i5k_amb<br>
> ioatdma dca shpchp ext3 jbd mbcache sd_mod crc_t10dif aacraid<br>
> pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm<br>
> i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last<br>
> unloaded: scsi_wait_scan]<br>
><br>
><br>
> Pid: 0, comm: swapper Tainted: G I---------------<br>
> 2.6.32-279.el6.x86_64 #1 Supermicro X7DVL/X7DVL<br>
> RIP: 0010:[<ffffffff81163f75>] [<ffffffff81163f75>] free_block<br>
> +0x165/0x170<br>
> RSP: 0018:ffff8800283c32d0 EFLAGS: 00010046<br>
> RAX: ffffea0009fd5878 RBX: ffff88042fcf03c0 RCX: 0000000000000010<br>
> RDX: 0040000000000000 RSI: ffff8802bba2cec0 RDI: ffff8802daab9800<br>
> RBP: ffff8800283c3320 R08: ffffea0009d7b600 R09: 0000000000000000<br>
> R10: 0000000000000000 R11: 0000000080042000 R12: 000000000000000c<br>
> R13: ffff880426350aa8 R14: 0000000000000002 R15: ffffea0000000000<br>
> FS: 0000000000000000(0000) GS:ffff8800283c0000(0000)<br>
> knlGS:0000000000000000<br>
> CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b<br>
> CR2: 0000003fafe7b3f0 CR3: 00000004240a1000 CR4: 00000000000006e0<br>
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000<br>
> DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400<br>
> Process swapper (pid: 0, threadinfo ffff880426374000, task<br>
> ffff880426373540)<br>
> Stack:<br>
> ffff88042fc216c0 ffff8802bba2cec0 000000000000100c ffff8802daab9800<br>
> <d> ffff8800283c3360 ffff8802daabc800 ffff880426350a80<br>
> 0000000000000286<br>
> <d> ffff88042fcf03c0 ffff880426350a98 ffff8800283c3390<br>
> ffffffff81164500<br>
> Call Trace:<br>
> <IRQ><br>
> [<ffffffff81164500>] kfree+0x310/0x320<br>
> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210<br>
> [<ffffffff8142fef8>] skb_release_data+0xd8/0x110<br>
> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210<br>
> [<ffffffff8142fa2e>] __kfree_skb+0x1e/0xa0<br>
> [<ffffffff8142fb72>] kfree_skb+0x42/0x90<br>
> [<ffffffff8143c949>] enqueue_to_backlog+0x179/0x210<br>
> [<ffffffff8143fb20>] netif_rx+0xb0/0x160<br>
> [<ffffffff8143fe32>] dev_forward_skb+0x122/0x180<br>
> [<ffffffffa02396e6>] veth_xmit+0x86/0xe0 [veth]<br>
> [<ffffffff8143b0cc>] dev_hard_start_xmit+0x2bc/0x3f0<br>
> [<ffffffff81458c1a>] sch_direct_xmit+0x15a/0x1c0<br>
> [<ffffffff8143f878>] dev_queue_xmit+0x4f8/0x6f0<br>
> [<ffffffffa032c6bc>] br_dev_queue_push_xmit+0x6c/0xa0 [bridge]<br>
> [<ffffffffa0332378>] br_nf_dev_queue_xmit+0x28/0xa0 [bridge]<br>
> [<ffffffffa0332e10>] br_nf_post_routing+0x1d0/0x280 [bridge]<br>
> [<ffffffff814665e9>] nf_iterate+0x69/0xb0<br>
> [<ffffffffa032c650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]<br>
> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110<br>
> [<ffffffffa032c650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]<br>
> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]<br>
> [<ffffffffa032c733>] br_forward_finish+0x43/0x60 [bridge]<br>
> [<ffffffffa03329b8>] br_nf_forward_finish+0x128/0x140 [bridge]<br>
> [<ffffffffa0333ea8>] ? br_nf_forward_ip+0x318/0x3c0 [bridge]<br>
> [<ffffffffa0333ea8>] br_nf_forward_ip+0x318/0x3c0 [bridge]<br>
> [<ffffffff814665e9>] nf_iterate+0x69/0xb0<br>
> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]<br>
> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110<br>
> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]<br>
> [<ffffffffa032c750>] ? __br_forward+0x0/0xc0 [bridge]<br>
> [<ffffffffa032c7c2>] __br_forward+0x72/0xc0 [bridge]<br>
> [<ffffffffa032c601>] br_flood+0xc1/0xd0 [bridge]<br>
> [<ffffffffa032c625>] br_flood_forward+0x15/0x20 [bridge]<br>
> [<ffffffffa032d7ae>] br_handle_frame_finish+0x27e/0x2a0 [bridge]<br>
> [<ffffffffa0333318>] br_nf_pre_routing_finish+0x228/0x340 [bridge]<br>
> [<ffffffffa033388f>] br_nf_pre_routing+0x45f/0x760 [bridge]<br>
> [<ffffffff814665e9>] nf_iterate+0x69/0xb0<br>
> [<ffffffffa032d530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]<br>
> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110<br>
> [<ffffffffa032d530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]<br>
> [<ffffffffa032d95c>] br_handle_frame+0x18c/0x250 [bridge]<br>
> [<ffffffff8143a839>] __netif_receive_skb+0x519/0x6f0<br>
> [<ffffffff8143ca38>] netif_receive_skb+0x58/0x60<br>
> [<ffffffff8143cbe4>] napi_gro_complete+0x84/0xe0<br>
> [<ffffffff8143ce0b>] dev_gro_receive+0x1cb/0x290<br>
> [<ffffffff8143cf4b>] __napi_gro_receive+0x7b/0x170<br>
> [<ffffffff8143f06f>] napi_gro_receive+0x2f/0x50<br>
> [<ffffffffa027733b>] e1000_receive_skb+0x5b/0x90 [e1000e]<br>
> [<ffffffffa027a601>] e1000_clean_rx_irq+0x241/0x4c0 [e1000e]<br>
> [<ffffffffa0281b8d>] e1000e_poll+0x8d/0x380 [e1000e]<br>
> [<ffffffff8143aaaa>] ? process_backlog+0x9a/0x100<br>
> [<ffffffff8143f193>] net_rx_action+0x103/0x2f0<br>
> [<ffffffff81073ec1>] __do_softirq+0xc1/0x1e0<br>
> [<ffffffff810db800>] ? handle_IRQ_event+0x60/0x170<br>
> [<ffffffff8100c24c>] call_softirq+0x1c/0x30<br>
> [<ffffffff8100de85>] do_softirq+0x65/0xa0<br>
> [<ffffffff81073ca5>] irq_exit+0x85/0x90<br>
> [<ffffffff81505af5>] do_IRQ+0x75/0xf0<br>
> [<ffffffff8100ba53>] ret_from_intr+0x0/0x11<br>
> <EOI><br>
> [<ffffffff81014877>] ? mwait_idle+0x77/0xd0<br>
> [<ffffffff8150338a>] ? atomic_notifier_call_chain+0x1a/0x20<br>
> [<ffffffff81009e06>] cpu_idle+0xb6/0x110<br>
> [<ffffffff814f6cdf>] start_secondary+0x22a/0x26d<br>
> Code: 41 5c 41 5d 41 5e 41 5f c9 c3 0f 1f 40 00 48 8b 72 08 48 89 c7<br>
> e8 2c f0 11 00 e9 07 ff ff ff 48 8b 40 10 48 8b 10 e9 3e ff ff ff <0f><br>
> 0b eb fe 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89<br>
> RIP [<ffffffff81163f75>] free_block+0x165/0x170<br>
> RSP <ffff8800283c32d0><br>
><br>
><br>
><br>
><br>
> ShiB.<br>
> while ( ! ( succeed = try() ) );<br>
><br>
><br>
<br>
Regards,<br>
Mike<br>
<span class="HOEnZb"><font color="#888888">--<br>
Michael H. Warfield (AI4NB) | <a href="tel:%28770%29%20978-7061" value="+17709787061">(770) 978-7061</a> | mhw@WittsEnd.com<br>
/\/\|=mhw=|\/\/ | <a href="tel:%28678%29%20463-0932" value="+16784630932">(678) 463-0932</a> | <a href="http://www.wittsend.com/mhw/" target="_blank">http://www.wittsend.com/mhw/</a><br>
NIC whois: MHW9 | An optimist believes we live in the best of all<br>
PGP Key: 0x674627FF | possible worlds. A pessimist is sure of it!<br>
<br>
</font></span><br>_______________________________________________<br>
lxc-users mailing list<br>
<a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a><br>
<a href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.org/listinfo/lxc-users</a><br></blockquote></div><br></div>