<div dir="ltr">Hi,<br><div class="gmail_quote"><div dir="ltr"><div><br></div><div>My server with 4 lxc VM is kernel panicking often. On analyzing the crash dump, it shows the following. I have 4 VMs, with 3G memory each, memory+swap set at 4G in the croup settings.</div>
<div><br></div><div>lxc version: 0.9.0.alpha2<br></div><div><br></div><div>I did a hardware swap, but the problem persists. Please let me know what to do next. Should I upgrade kernel to 2.6.32-431.11.2.el6.centos.plus ?</div>
<div><br></div><div><div> KERNEL: /usr/lib/debug/lib/modules/2.6.32-279.el6.x86_64/vmlinux</div><div> DUMPFILE: /var/crash/<a href="http://127.0.0." target="_blank">127.0.0.</a><a href="tel:1-2014-05-03-06" value="+12014050306" target="_blank">1-2014-05-03-06</a>:11:01/vmcore [PARTIAL DUMP]</div>
<div> CPUS: 8</div><div> DATE: Sat May 3 06:09:12 2014</div>
<div> UPTIME: 07:56:53</div><div>LOAD AVERAGE: 0.07, 0.06, 0.01</div><div> TASKS: 651</div><div> NODENAME: myhost</div><div> RELEASE: 2.6.32-279.el6.x86_64</div><div> VERSION: #1 SMP Fri Jun 22 12:19:21 UTC 2012</div>
<div> MACHINE: x86_64 (1866 Mhz)</div><div> MEMORY: 16 GB</div><div> PANIC: "kernel BUG at mm/slab.c:533!"</div><div> PID: 0</div><div> COMMAND: "swapper"</div><div> TASK: ffff880426373540 (1 of 8) [THREAD_INFO: ffff880426374000]</div>
<div> CPU: 7</div><div> STATE: TASK_RUNNING (PANIC)</div><div><br></div></div><div><br></div><div>------------[ cut here ]------------</div><div>kernel BUG at mm/slab.c:533!</div><div>invalid opcode: 0000 [#1] SMP</div>
<div>last sysfs file: /sys/devices/system/cpu/online</div><div>CPU 7</div><div>Modules linked in: veth bridge stp llc ipv6 e1000e(U) sg microcode i2c_i801 iTCO_wdt iTCO_vendor_support i5000_edac edac_core i5k_amb ioatdma dca shpchp ext3 jbd mbcache sd_mod crc_t10dif aacraid pata_acpi ata_generic ata_piix radeon ttm drm_kms_helper drm i2c_algo_bit i2c_core dm_mirror dm_region_hash dm_log dm_mod [last unloaded: scsi_wait_scan]</div>
<div><br></div><div>Pid: 0, comm: swapper Tainted: G I--------------- 2.6.32-279.el6.x86_64 #1 Supermicro X7DVL/X7DVL</div><div>RIP: 0010:[<ffffffff81163f75>] [<ffffffff81163f75>] free_block+0x165/0x170</div>
<div>RSP: 0018:ffff8800283c32d0 EFLAGS: 00010046</div><div>RAX: ffffea0009fd5878 RBX: ffff88042fcf03c0 RCX: 0000000000000010</div><div>RDX: 0040000000000000 RSI: ffff8802bba2cec0 RDI: ffff8802daab9800</div><div>RBP: ffff8800283c3320 R08: ffffea0009d7b600 R09: 0000000000000000</div>
<div>R10: 0000000000000000 R11: 0000000080042000 R12: 000000000000000c</div><div>R13: ffff880426350aa8 R14: 0000000000000002 R15: ffffea0000000000</div><div>FS: 0000000000000000(0000) GS:ffff8800283c0000(0000) knlGS:0000000000000000</div>
<div>CS: 0010 DS: 0018 ES: 0018 CR0: 000000008005003b</div><div>CR2: 0000003fafe7b3f0 CR3: 00000004240a1000 CR4: 00000000000006e0</div><div>DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000</div><div>DR3: 0000000000000000 DR6: 00000000ffff0ff0 DR7: 0000000000000400</div>
<div>Process swapper (pid: 0, threadinfo ffff880426374000, task ffff880426373540)</div><div>Stack:</div><div> ffff88042fc216c0 ffff8802bba2cec0 000000000000100c ffff8802daab9800</div><div><d> ffff8800283c3360 ffff8802daabc800 ffff880426350a80 0000000000000286</div>
<div><d> ffff88042fcf03c0 ffff880426350a98 ffff8800283c3390 ffffffff81164500</div><div>Call Trace:</div><div> <IRQ></div><div> [<ffffffff81164500>] kfree+0x310/0x320</div><div> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210</div>
<div> [<ffffffff8142fef8>] skb_release_data+0xd8/0x110</div><div> [<ffffffff8143c949>] ? enqueue_to_backlog+0x179/0x210</div><div> [<ffffffff8142fa2e>] __kfree_skb+0x1e/0xa0</div><div> [<ffffffff8142fb72>] kfree_skb+0x42/0x90</div>
<div> [<ffffffff8143c949>] enqueue_to_backlog+0x179/0x210</div><div> [<ffffffff8143fb20>] netif_rx+0xb0/0x160</div><div> [<ffffffff8143fe32>] dev_forward_skb+0x122/0x180</div><div> [<ffffffffa02396e6>] veth_xmit+0x86/0xe0 [veth]</div>
<div> [<ffffffff8143b0cc>] dev_hard_start_xmit+0x2bc/0x3f0</div><div> [<ffffffff81458c1a>] sch_direct_xmit+0x15a/0x1c0</div><div> [<ffffffff8143f878>] dev_queue_xmit+0x4f8/0x6f0</div><div> [<ffffffffa032c6bc>] br_dev_queue_push_xmit+0x6c/0xa0 [bridge]</div>
<div> [<ffffffffa0332378>] br_nf_dev_queue_xmit+0x28/0xa0 [bridge]</div><div> [<ffffffffa0332e10>] br_nf_post_routing+0x1d0/0x280 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div>
[<ffffffffa032c650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]</div>
<div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div><div> [<ffffffffa032c650>] ? br_dev_queue_push_xmit+0x0/0xa0 [bridge]</div><div> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div>
[<ffffffffa032c733>] br_forward_finish+0x43/0x60 [bridge]</div><div> [<ffffffffa03329b8>] br_nf_forward_finish+0x128/0x140 [bridge]</div><div> [<ffffffffa0333ea8>] ? br_nf_forward_ip+0x318/0x3c0 [bridge]</div>
<div> [<ffffffffa0333ea8>] br_nf_forward_ip+0x318/0x3c0 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div>
<div> [<ffffffffa032c6f0>] ? br_forward_finish+0x0/0x60 [bridge]</div><div> [<ffffffffa032c750>] ? __br_forward+0x0/0xc0 [bridge]</div><div> [<ffffffffa032c7c2>] __br_forward+0x72/0xc0 [bridge]</div><div>
[<ffffffffa032c601>] br_flood+0xc1/0xd0 [bridge]</div><div> [<ffffffffa032c625>] br_flood_forward+0x15/0x20 [bridge]</div><div> [<ffffffffa032d7ae>] br_handle_frame_finish+0x27e/0x2a0 [bridge]</div><div>
[<ffffffffa0333318>] br_nf_pre_routing_finish+0x228/0x340 [bridge]</div><div> [<ffffffffa033388f>] br_nf_pre_routing+0x45f/0x760 [bridge]</div><div> [<ffffffff814665e9>] nf_iterate+0x69/0xb0</div><div>
[<ffffffffa032d530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]</div>
<div> [<ffffffff814667a4>] nf_hook_slow+0x74/0x110</div><div> [<ffffffffa032d530>] ? br_handle_frame_finish+0x0/0x2a0 [bridge]</div><div> [<ffffffffa032d95c>] br_handle_frame+0x18c/0x250 [bridge]</div><div>
[<ffffffff8143a839>] __netif_receive_skb+0x519/0x6f0</div><div> [<ffffffff8143ca38>] netif_receive_skb+0x58/0x60</div><div> [<ffffffff8143cbe4>] napi_gro_complete+0x84/0xe0</div><div> [<ffffffff8143ce0b>] dev_gro_receive+0x1cb/0x290</div>
<div> [<ffffffff8143cf4b>] __napi_gro_receive+0x7b/0x170</div><div> [<ffffffff8143f06f>] napi_gro_receive+0x2f/0x50</div><div> [<ffffffffa027733b>] e1000_receive_skb+0x5b/0x90 [e1000e]</div><div> [<ffffffffa027a601>] e1000_clean_rx_irq+0x241/0x4c0 [e1000e]</div>
<div> [<ffffffffa0281b8d>] e1000e_poll+0x8d/0x380 [e1000e]</div><div> [<ffffffff8143aaaa>] ? process_backlog+0x9a/0x100</div><div> [<ffffffff8143f193>] net_rx_action+0x103/0x2f0</div><div> [<ffffffff81073ec1>] __do_softirq+0xc1/0x1e0</div>
<div> [<ffffffff810db800>] ? handle_IRQ_event+0x60/0x170</div><div> [<ffffffff8100c24c>] call_softirq+0x1c/0x30</div><div> [<ffffffff8100de85>] do_softirq+0x65/0xa0</div><div> [<ffffffff81073ca5>] irq_exit+0x85/0x90</div>
<div> [<ffffffff81505af5>] do_IRQ+0x75/0xf0</div><div> [<ffffffff8100ba53>] ret_from_intr+0x0/0x11</div><div> <EOI></div><div> [<ffffffff81014877>] ? mwait_idle+0x77/0xd0</div><div> [<ffffffff8150338a>] ? atomic_notifier_call_chain+0x1a/0x20</div>
<div> [<ffffffff81009e06>] cpu_idle+0xb6/0x110</div><div> [<ffffffff814f6cdf>] start_secondary+0x22a/0x26d</div><div>Code: 41 5c 41 5d 41 5e 41 5f c9 c3 0f 1f 40 00 48 8b 72 08 48 89 c7 e8 2c f0 11 00 e9 07 ff ff ff 48 8b 40 10 48 8b 10 e9 3e ff ff ff <0f> 0b eb fe 0f 1f 80 00 00 00 00 55 48 89 e5 48 83 ec 30 48 89</div>
<div>RIP [<ffffffff81163f75>] free_block+0x165/0x170</div><div> RSP <ffff8800283c32d0></div><div><br></div><div><br></div><div>ShiB.<br>while ( ! ( succeed = try() ) );</div>
</div>
</div><br></div>