[Lxc-users] Running LXC on ZFS, never comes back online after reboot

Tue Aug 27 03:42:33 UTC 2013

Greetings LXC folks,

With LXC and ZFS on AWS, after I've created 1 or more containers, the
machine will never come back up after a reboot.

One "fix" I've found for this is to always explicity run `sudo zpool export
tank` before every system restart, but there are situations where this is
not feasible (unexpected reboots due to fat finger, power outages,
hurricanes, cosmic rays, etc..).

Precise steps to reproduce:

Using Ubuntu 12.04, run the following commands:
#################
    # Install ZFS and LXC
    sudo apt-add-repository -y ppa:zfs-native/stable
    sudo apt-add-repository -y ppa:ubuntu-lxc/daily
    sudo apt-get update
    sudo apt-get install -y lxc ubuntu-zfs
    # Note: /dev/xvdb is an EBS volume.
    sudo zpool create -o ashift=12 tank /dev/xvdb
    sudo zpool export tank
    sudo zpool import tank
    sudo zfs create -o compression=on tank/lxc
    sudo mv /var/lib/lxc{,.orig}
    sudo ln -s /tank/lxc /var/lib/lxc
    sudo lxc-create -t ubuntu -B zfs -n test
    sudo reboot # Or use AWS web interface to trigger a system restart.

--

Additionally, here is information from the boot log for one such failed
system - see especially the end, with "canonicalization errors":

e.g. filesystem 'tank/sendhub_v9' cannot be mounted at
'/var/lib/lxc/sendhub_v9/rootfs' due to canonicalization error 2.

More interesting portion of boot log:

mountall: swapon /dev/xvdc [454] terminated with status 255

mountall: Problem activating swap: /dev/xvdc

filesystem 'tank/base' cannot be mounted at '/var/lib/lxc/base/rootfs'
due to canonicalization error 2.

mountall: mount tank/base [509] terminated with status 2

mountall: Filesystem could not be mounted: tank/base

filesystem 'tank/base-playframework2' cannot be mounted at
'/var/lib/lxc/base-playframework2/rootfs' due to canonicalization
error 2.

mountall: mount tank/base-playframework2 [511] terminated with status 2

mountall: Filesystem could not be mounted: tank/base-playframework2

filesystem 'tank/base-python' cannot be mounted at
'/var/lib/lxc/base-python/rootfs' due to canonicalization error 2.

mountall: mount tank/base-python [513] terminated with status 2

mountall: Filesystem could not be mounted: tank/base-python

filesystem 'tank/sendhub' cannot be mounted at
'/var/lib/lxc/sendhub/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub [515] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub

filesystem 'tank/sendhub_v10' cannot be mounted at
'/var/lib/lxc/sendhub_v10/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v10 [517] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v10

filesystem 'tank/sendhub_v11' cannot be mounted at
'/var/lib/lxc/sendhub_v11/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v11 [519] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v11

filesystem 'tank/sendhub_v12' cannot be mounted at
'/var/lib/lxc/sendhub_v12/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v12 [521] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v12

filesystem 'tank/sendhub_v13' cannot be mounted at
'/var/lib/lxc/sendhub_v13/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v13 [523] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v13

filesystem 'tank/sendhub_v14' cannot be mounted at
'/var/lib/lxc/sendhub_v14/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v14 [525] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v14

filesystem 'tank/sendhub_v15' cannot be mounted at
'/var/lib/lxc/sendhub_v15/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v15 [527] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v15

filesystem 'tank/sendhub_v16' cannot be mounted at
'/var/lib/lxc/sendhub_v16/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v16 [529] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v16

filesystem 'tank/sendhub_v8' cannot be mounted at
'/var/lib/lxc/sendhub_v8/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v8 [531] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v8

filesystem 'tank/sendhub_v9' cannot be mounted at
'/var/lib/lxc/sendhub_v9/rootfs' due to canonicalization error 2.

mountall: mount tank/sendhub_v9 [533] terminated with status 2

mountall: Filesystem could not be mounted: tank/sendhub_v9

cloud-init start-local running: Tue, 27 Aug 2013 03:20:54 +0000. up 4.27 seconds

no instance data found in start-local

ci-info: lo    : 1 127.0.0.1       255.0.0.0       .

ci-info: eth0  : 1 10.182.150.160  255.255.255.192 22:00:0a:b6:96:a0

ci-info: route-0: 0.0.0.0         10.182.150.129  0.0.0.0         eth0   UG

ci-info: route-1: 10.182.150.128  0.0.0.0         255.255.255.192 eth0   U

cloud-init start running: Tue, 27 Aug 2013 03:20:55 +0000. up 5.19 seconds

found data source: DataSourceEc2

2013-08-27 03:20:56,111 - __init__.py[WARNING]: Unhandled
non-multipart userdata ''

An error occurred while mounting tank/base.

Press S to skip mounting or M for manual recovery

unable to connect to system bus: Failed to connect to socket
/var/run/dbus/system_bus_socket: No such file or directory

 * Asking all remaining processes to terminate...
    [80G
    [74G[ OK ]

 * All processes ended within 1 seconds....
    [80G
    [74G[ OK ]

 * Deconfiguring network interfaces...
    [80G
    [74G[ OK ]

 * Deactivating swap...
    [80G
    [74G[ OK ]

 * Unmounting local filesystems...
    [80G
    [74G[ OK ]

 * Will now restart

[171999.780573] Restarting system.

And here is a full copy of the boot log:

 Xen Minimal OS!
  start_info: 0x1890000(VA)
    nr_pages: 0x1e0000
  shared_inf: 0x7e1b7000(MA)
     pt_base: 0x1893000(VA)
nr_pt_frames: 0x11
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ffbd(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80ae0(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 18a7
    max_pfn: 1e0000
Mapping memory range 0x1c00000 - 0x1e0000000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 27a0000(27a0000)-1e0000000(1e0000000)
MM: done
Demand map pfns at 1e0001000-21e0001000.
Heap resides at 21e0002000-41e0002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x1e0001000.
Initialising scheduler
Thread "Idle": pointer: 0x21e0002010, stack: 0x36f0000
Initialising xenbus
Thread "xenstore": pointer: 0x21e00027c0, stack: 0x3700000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x21e0002f70, stack: 0x3710000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********

backend at /local/domain/0/backend/vbd/32/2049
Failed to read /local/domain/0/backend/vbd/32/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/32/2049/feature-flush-cache.
16777216 sectors of 512 bytes
**************************
vbd 2080 is hd1
******************* BLKFRONT for device/vbd/2080 **********

backend at /local/domain/0/backend/vbd/32/2080
Failed to read /local/domain/0/backend/vbd/32/2080/feature-barrier.
Failed to read /local/domain/0/backend/vbd/32/2080/feature-flush-cache.
880732160 sectors of 512 bytes
**************************
vbd 2096 is hd2
******************* BLKFRONT for device/vbd/2096 **********

backend at /local/domain/0/backend/vbd/32/2096
Failed to read /local/domain/0/backend/vbd/32/2096/feature-barrier.
Failed to read /local/domain/0/backend/vbd/32/2096/feature-flush-cache.
880732160 sectors of 512 bytes
**************************
vbd 2288 is hd3
******************* BLKFRONT for device/vbd/2288 **********

backend at /local/domain/0/backend/vbd/32/2288
Failed to read /local/domain/0/backend/vbd/32/2288/feature-barrier.
Failed to read /local/domain/0/backend/vbd/32/2288/feature-flush-cache.
16777216 sectors of 512 bytes
**************************

    [H
    [J  Booting 'Ubuntu 12.04.2 LTS, kernel 3.2.0-40-virtual'

root  (hd0)

 Filesystem type is ext2fs, using whole disk

kernel  /boot/vmlinuz-3.2.0-40-virtual root=LABEL=cloudimg-rootfs ro console=hv

c0

initrd  /boot/initrd.img-3.2.0-40-virtual

close blk: backend at /local/domain/0/backend/vbd/32/2049
close blk: backend at /local/domain/0/backend/vbd/32/2080
close blk: backend at /local/domain/0/backend/vbd/32/2096
close blk: backend at /local/domain/0/backend/vbd/32/2288
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.0-40-virtual (buildd at allspice) (gcc
version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #64-Ubuntu SMP Mon Mar
25 21:42:18 UTC 2013 (Ubuntu 3.2.0-40.64-virtual 3.2.40)
[    0.000000] Command line: root=LABEL=cloudimg-rootfs ro console=hvc0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Released 0 pages of unused memory
[    0.000000] Set 0 page(s) to 1-1 mapping
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 00000001e0800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x1e0800 max_arch_pfn = 0x400000000
[    0.000000] x2apic enabled by BIOS, switching to x2apic ops
[    0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[    0.000000] init_memory_mapping: 0000000100000000-00000001e0800000
[    0.000000] RAMDISK: 02057000 - 02c88000
[    0.000000] NUMA turned off
[    0.000000] Faking a node at 0000000000000000-00000001e0800000
[    0.000000] Initmem setup node 0 0000000000000000-00000001e0800000
[    0.000000]   NODE_DATA [00000001dfffb000 - 00000001dfffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x001e0800
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x001e0800
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[    0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[    0.000000] Allocating PCI resources starting at 1e0900000 (gap:
1e0900000:400000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64
nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff8801dfc00000 s82880
r8192 d23616 u1048576
[171370.202872] Built 1 zonelists in Node order, mobility grouping on.
 Total pages: 1935244
[171370.202876] Policy zone: Normal
[171370.202879] Kernel command line: root=LABEL=cloudimg-rootfs ro console=hvc0
[171370.203562] PID hash table entries: 4096 (order: 3, 32768 bytes)
[171370.216143] Checking aperture...
[171370.225421] No AGP bridge found
[171370.259516] Memory: 7613252k/7872512k available (6546k kernel
code, 448k absent, 258812k reserved, 6647k data, 920k init)
[171370.259634] SLUB: Genslabs=15, HWalign=64, Order=0-3,
MinObjects=0, CPUs=2, Nodes=1
[171370.259679] Hierarchical RCU implementation.
[171370.259681] 	RCU dyntick-idle grace-period acceleration is enabled.
[171370.259695] NR_IRQS:4352 nr_irqs:288 16
[171370.259831] Console: colour dummy device 80x25
[171370.259953] console [tty0] enabled
[171370.260072] console [hvc0] enabled
[171370.269781] allocated 63963136 bytes of page_cgroup
[171370.269799] please try 'cgroup_disable=memory' option if you don't
want memory cgroups
[171370.269866] installing Xen timer for CPU 0
[171370.269918] Detected 1795.672 MHz processor.
[171370.269930] Calibrating delay loop (skipped), value calculated
using timer frequency.. 3591.34 BogoMIPS (lpj=7182688)
[171370.269943] pid_max: default: 32768 minimum: 301
[171370.269992] Security Framework initialized
[171370.270012] AppArmor: AppArmor initialized
[171370.270017] Yama: becoming mindful.
[171370.272679] Dentry cache hash table entries: 1048576 (order: 11,
8388608 bytes)
[171370.277206] Inode-cache hash table entries: 524288 (order: 10,
4194304 bytes)
[171370.278125] Mount-cache hash table entries: 256
[171370.278359] Initializing cgroup subsys cpuacct
[171370.278371] Initializing cgroup subsys memory
[171370.278391] Initializing cgroup subsys devices
[171370.278399] Initializing cgroup subsys freezer
[171370.278406] Initializing cgroup subsys blkio
[171370.278424] Initializing cgroup subsys perf_event
[171370.278537] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[171370.278538] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[171370.278559] CPU: Physical Processor ID: 1
[171370.278565] CPU: Processor Core ID: 7
[171370.278709] SMP alternatives: switching to UP code
[171370.328421] ftrace: allocating 26549 entries in 105 pages
[171370.341088] cpu 0 spinlock event irq 17
[171370.341125] Performance Events: unsupported p6 CPU model 45 no PMU
driver, software events only.
[171370.341308] NMI watchdog disabled (cpu0): hardware events not enabled
[171370.341419] installing Xen timer for CPU 1
[171370.341441] cpu 1 spinlock event irq 23
[171370.341474] SMP alternatives: switching to SMP code
[171370.386364] NMI watchdog disabled (cpu1): hardware events not enabled
[171370.386427] Brought up 2 CPUs
[171370.388074] devtmpfs: initialized
[171370.389186] EVM: security.selinux
[171370.389193] EVM: security.SMACK64
[171370.389197] EVM: security.capability
[171370.390251] Grant table initialized
[171370.390330] print_constraints: dummy:
[171370.410206] RTC time: 165:165:165, date: 165/165/65
[171370.410257] NET: Registered protocol family 16
[171370.410467] Trying to unpack rootfs image as initramfs...
[171370.411442] PCI: setting up Xen PCI frontend stub
[171370.412634] bio: create slab <bio-0> at 0
[171370.412752] ACPI: Interpreter disabled.
[171370.412947] xen/balloon: Initialising balloon driver.
[171370.414083] xen-balloon: Initialising balloon driver.
[171370.414259] vgaarb: loaded
[171370.414389] i2c-core: driver [aat2870] using legacy suspend method
[171370.414396] i2c-core: driver [aat2870] using legacy resume method
[171370.414480] SCSI subsystem initialized
[171370.414601] usbcore: registered new interface driver usbfs
[171370.414625] usbcore: registered new interface driver hub
[171370.427649] Freeing initrd memory: 12484k freed
[171370.432117] usbcore: registered new device driver usb
[171370.432356] PCI: System does not support PCI
[171370.432364] PCI: System does not support PCI
[171370.432516] NetLabel: Initializing
[171370.432522] NetLabel:  domain hash size = 128
[171370.432526] NetLabel:  protocols = UNLABELED CIPSOv4
[171370.432545] NetLabel:  unlabeled traffic allowed by default
[171370.432653] Switching to clocksource xen
[171370.442432] AppArmor: AppArmor Filesystem Enabled
[171370.442460] pnp: PnP ACPI: disabled
[171370.446124] NET: Registered protocol family 2
[171370.446872] IP route cache hash table entries: 262144 (order: 9,
2097152 bytes)
[171370.449982] TCP established hash table entries: 524288 (order: 11,
8388608 bytes)
[171370.451863] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[171370.452039] TCP: Hash tables configured (established 524288 bind 65536)
[171370.458335] TCP reno registered
[171370.458387] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[171370.458466] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[171370.458584] NET: Registered protocol family 1
[171370.458604] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[171370.458612] Placing 64MB software IO TLB between ffff8800fb8f7000
- ffff8800ff8f7000
[171370.458620] software IO TLB at phys 0xfb8f7000 - 0xff8f7000
[171370.458692] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[171370.459139] audit: initializing netlink socket (disabled)
[171370.459161] type=2000 audit(1377573070.756:1): initialized
[171370.486532] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[171370.488484] VFS: Disk quotas dquot_6.5.2
[171370.488564] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[171370.489213] fuse init (API version 7.17)
[171370.489350] msgmni has been set to 14894
[171370.489937] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[171370.489999] io scheduler noop registered
[171370.490010] io scheduler deadline registered (default)
[171370.490062] io scheduler cfq registered
[171370.490161] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[171370.490194] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[171370.490776] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[171370.543547] Linux agpgart interface v0.103
[171370.545355] brd: module loaded
[171370.546338] loop: module loaded
[171370.548555] blkfront device/vbd/2049 num-ring-pages 1 nr_ents 32.
[171370.552723] blkfront device/vbd/2096 num-ring-pages 1 nr_ents 32.
[171370.553683] blkfront: xvda1: barrier or flush: disabled
[171370.556825] blkfront: xvdd: barrier or flush: disabled
[171370.558452] blkfront device/vbd/2080 num-ring-pages 1 nr_ents 32.
[171370.559675]  xvdd: unknown partition table
[171370.560493] Setting capacity to 880732160
[171370.560508] xvdd: detected capacity change from 0 to 450934865920
[171370.562133] blkfront device/vbd/2288 num-ring-pages 1 nr_ents 32.
[171370.563815] blkfront: xvdc: barrier or flush: disabled
[171370.564466] Fixed MDIO Bus: probed
[171370.564499] tun: Universal TUN/TAP device driver, 1.6
[171370.564505] tun: (C) 1999-2004 Max Krasnyansky <maxk at qualcomm.com>
[171370.564589] PPP generic driver version 2.4.2
[171370.564809] Initialising Xen virtual ethernet driver.
[171370.565081]  xvdc: unknown partition table
[171370.565356] Setting capacity to 880732160
[171370.565367] xvdc: detected capacity change from 0 to 450934865920
[171370.565893] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[171370.565918] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[171370.565933] uhci_hcd: USB Universal Host Controller Interface driver
[171370.565975] usbcore: registered new interface driver libusual
[171370.566003] i8042: PNP: No PS/2 controller found. Probing ports directly.
[171370.693405] blkfront: xvdp: barrier or flush: disabled
[171370.710175]  xvdp: unknown partition table
[171371.571729] i8042: No controller found
[171371.571905] mousedev: PS/2 mouse device common for all mice
[171371.611815] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[171371.611893] rtc_cmos: probe of rtc_cmos failed with error -38
[171371.612004] device-mapper: uevent: version 1.0.3
[171371.612105] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19)
initialised: dm-devel at redhat.com
[171371.612121] EFI Variables Facility v0.08 2004-May-17
[171371.612451] TCP cubic registered
[171371.612575] NET: Registered protocol family 10
[171371.613445] NET: Registered protocol family 17
[171371.613457] Registering the dns_resolver key type
[171371.613674] registered taskstats version 1
[171371.626426] XENBUS: Device with no driver: device/console/0
[171371.626449]   Magic number: 1:252:3141
[171371.626492] /build/buildd/linux-3.2.0/drivers/rtc/hctosys.c:
unable to open rtc device (rtc0)
[171371.626510] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[171371.626518] EDD information not available.
[171371.627091] Freeing unused kernel memory: 920k freed
[171371.627442] Write protecting the kernel read-only data: 12288k
[171371.634594] Freeing unused kernel memory: 1628k freed
[171371.635788] Freeing unused kernel memory: 1192k freed
Loading, please wait...
[171371.684238] udevd[86]: starting version 175
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[171371.975077] EXT4-fs (xvdp): INFO: recovery required on readonly filesystem
[171371.975092] EXT4-fs (xvdp): write access will be enabled during recovery
[171372.131245] EXT4-fs (xvdp): recovery complete
[171372.132441] EXT4-fs (xvdp): mounted filesystem with ordered data
mode. Opts: (null)
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[171373.374913] SPL: Loaded module v0.6.1-rc14
[171373.407580] zunicode: module license 'CDDL' taints kernel.
[171373.407597] Disabling lock debugging due to kernel taint
[171373.518929] SPL: using hostid 0x1d0aa6ed
[171373.520108] ZFS: Loaded module v0.6.1-rc14, ZFS pool version 5000,
ZFS filesystem version 5
[171373.670372] EXT4-fs (xvdp): re-mounted. Opts: (null)
swapon: /dev/xvdc: read swap header failed: Invalid argument

mountall: swapon /dev/xvdc [492] terminated with status 255

mountall: Problem activating swap: /dev/xvdc

cloud-init start-local running: Tue, 27 Aug 2013 03:11:15 +0000. up 4.53 seconds

no instance data found in start-local

ci-info: lo    : 1 127.0.0.1       255.0.0.0       .

ci-info: eth0  : 1 10.182.150.160  255.255.255.192 22:00:0a:b6:96:a0

ci-info: route-0: 0.0.0.0         10.182.150.129  0.0.0.0         eth0   UG

ci-info: route-1: 10.182.150.128  0.0.0.0         255.255.255.192 eth0   U

cloud-init start running: Tue, 27 Aug 2013 03:11:15 +0000. up 5.24 seconds

found data source: DataSourceEc2

2013-08-27 03:11:16,222 - __init__.py[WARNING]: Unhandled
non-multipart userdata ''

Skipping profile in /etc/apparmor.d/disable: usr.sbin.rsyslogd

 * Starting AppArmor profiles
    [80G
    [74G[ OK ]

 * Stopping System V initialisation compatibility
    [74G[ OK ]

 * Stopping Handle applying cloud-config
    [74G[ OK ]

 * Starting System V runlevel compatibility
    [74G[ OK ]

 * Starting save kernel messages
    [74G[ OK ]

 * Starting ACPI daemon
    [74G[ OK ]

 * Starting regular background program processing daemon
    [74G[ OK ]

 * Starting deferred execution scheduler
    [74G[ OK ]

 * Starting automatic crash report generation
    [74G[ OK ]

 * Starting CPU interrupts balancing daemon
    [74G[ OK ]

 * Starting NTP server ntpd
    [80G * Starting configure network device security
    [74G[ OK ]

    [74G[ OK ]

 * Stopping save kernel messages
    [74G[ OK ]

 * Starting configure network device
    [74G[ OK ]

landscape-client is not configured, please run landscape-config.

 * Starting lxc network
    [74G[ OK ]

 * Starting lxc
    [74G[ OK ]

 * Stopping System V runlevel compatibility
    [74G[ OK ]

 * Starting execute cloud user/final scripts
    [74G[ OK ]

cloud-init boot finished at Tue, 27 Aug 2013 03:11:17 +0000. Up 6.92 seconds
acpid: exiting

Tue Aug 27 03:20:08 UTC 2013: shutting down for reboot [up 538s].
Checking for running unattended-upgrades:
 * Stopping landscape-client daemon
    [80G
    [74G[
    [31mfail
    [39;49m]
 * Asking all remaining processes to terminate...
    [80G
    [74G[ OK ]
 * All processes ended within 1 seconds....
    [80G
    [74G[ OK ]
 * Deconfiguring network interfaces...
    [80G
    [74G[ OK ]
 * Deactivating swap...
    [80G
    [74G[ OK ]
 * Unmounting local filesystems...
    [80G
    [74G[ OK ]
 * Will now restart
[171910.582908] Restarting system.
Xen Minimal OS!
  start_info: 0x1890000(VA)
    nr_pages: 0x1e0000
  shared_inf: 0x7dcfe000(MA)
     pt_base: 0x1893000(VA)
nr_pt_frames: 0x11
    mfn_list: 0x990000(VA)
   mod_start: 0x0(VA)
     mod_len: 0
       flags: 0x0
    cmd_line: root=/dev/sda1 ro 4
  stack:      0x94f860-0x96f860
MM: Init
      _text: 0x0(VA)
     _etext: 0x5ffbd(VA)
   _erodata: 0x78000(VA)
     _edata: 0x80ae0(VA)
stack start: 0x94f860(VA)
       _end: 0x98fe68(VA)
  start_pfn: 18a7
    max_pfn: 1e0000
Mapping memory range 0x1c00000 - 0x1e0000000
setting 0x0-0x78000 readonly
skipped 0x1000
MM: Initialise page allocator for 27a0000(27a0000)-1e0000000(1e0000000)
MM: done
Demand map pfns at 1e0001000-21e0001000.
Heap resides at 21e0002000-41e0002000.
Initialising timer interface
Initialising console ... done.
gnttab_table mapped at 0x1e0001000.
Initialising scheduler
Thread "Idle": pointer: 0x21e0002010, stack: 0x36f0000
Initialising xenbus
Thread "xenstore": pointer: 0x21e00027c0, stack: 0x3700000
Dummy main: start_info=0x96f960
Thread "main": pointer: 0x21e0002f70, stack: 0x3710000
"main" "root=/dev/sda1" "ro" "4"
vbd 2049 is hd0
******************* BLKFRONT for device/vbd/2049 **********

backend at /local/domain/0/backend/vbd/33/2049
Failed to read /local/domain/0/backend/vbd/33/2049/feature-barrier.
Failed to read /local/domain/0/backend/vbd/33/2049/feature-flush-cache.
16777216 sectors of 512 bytes
**************************
vbd 2064 is hd1
******************* BLKFRONT for device/vbd/2064 **********

backend at /local/domain/0/backend/vbd/33/2064
Failed to read /local/domain/0/backend/vbd/33/2064/feature-barrier.
Failed to read /local/domain/0/backend/vbd/33/2064/feature-flush-cache.
209715200 sectors of 512 bytes
**************************
vbd 2080 is hd2
******************* BLKFRONT for device/vbd/2080 **********

backend at /local/domain/0/backend/vbd/33/2080
Failed to read /local/domain/0/backend/vbd/33/2080/feature-barrier.
Failed to read /local/domain/0/backend/vbd/33/2080/feature-flush-cache.
880732160 sectors of 512 bytes
**************************
vbd 2096 is hd3
******************* BLKFRONT for device/vbd/2096 **********

backend at /local/domain/0/backend/vbd/33/2096
Failed to read /local/domain/0/backend/vbd/33/2096/feature-barrier.
Failed to read /local/domain/0/backend/vbd/33/2096/feature-flush-cache.
880732160 sectors of 512 bytes
**************************
vbd 2288 is hd4
******************* BLKFRONT for device/vbd/2288 **********

backend at /local/domain/0/backend/vbd/33/2288
Failed to read /local/domain/0/backend/vbd/33/2288/feature-barrier.
Failed to read /local/domain/0/backend/vbd/33/2288/feature-flush-cache.
16777216 sectors of 512 bytes
**************************

    [H
    [J  Booting 'Ubuntu 12.04.2 LTS, kernel 3.2.0-40-virtual'

root  (hd0)

 Filesystem type is ext2fs, using whole disk

kernel  /boot/vmlinuz-3.2.0-40-virtual root=LABEL=cloudimg-rootfs ro console=hv

c0

initrd  /boot/initrd.img-3.2.0-40-virtual

close blk: backend at /local/domain/0/backend/vbd/33/2049
close blk: backend at /local/domain/0/backend/vbd/33/2064
close blk: backend at /local/domain/0/backend/vbd/33/2080
close blk: backend at /local/domain/0/backend/vbd/33/2096
close blk: backend at /local/domain/0/backend/vbd/33/2288
[    0.000000] Initializing cgroup subsys cpuset
[    0.000000] Initializing cgroup subsys cpu
[    0.000000] Linux version 3.2.0-40-virtual (buildd at allspice) (gcc
version 4.6.3 (Ubuntu/Linaro 4.6.3-1ubuntu5) ) #64-Ubuntu SMP Mon Mar
25 21:42:18 UTC 2013 (Ubuntu 3.2.0-40.64-virtual 3.2.40)
[    0.000000] Command line: root=LABEL=cloudimg-rootfs ro console=hvc0
[    0.000000] KERNEL supported cpus:
[    0.000000]   Intel GenuineIntel
[    0.000000]   AMD AuthenticAMD
[    0.000000]   Centaur CentaurHauls
[    0.000000] ACPI in unprivileged domain disabled
[    0.000000] Released 0 pages of unused memory
[    0.000000] Set 0 page(s) to 1-1 mapping
[    0.000000] BIOS-provided physical RAM map:
[    0.000000]  Xen: 0000000000000000 - 00000000000a0000 (usable)
[    0.000000]  Xen: 00000000000a0000 - 0000000000100000 (reserved)
[    0.000000]  Xen: 0000000000100000 - 00000001e0800000 (usable)
[    0.000000] NX (Execute Disable) protection: active
[    0.000000] DMI not present or invalid.
[    0.000000] No AGP bridge found
[    0.000000] last_pfn = 0x1e0800 max_arch_pfn = 0x400000000
[    0.000000] x2apic enabled by BIOS, switching to x2apic ops
[    0.000000] last_pfn = 0x100000 max_arch_pfn = 0x400000000
[    0.000000] init_memory_mapping: 0000000000000000-0000000100000000
[    0.000000] init_memory_mapping: 0000000100000000-00000001e0800000
[    0.000000] RAMDISK: 02057000 - 02c88000
[    0.000000] NUMA turned off
[    0.000000] Faking a node at 0000000000000000-00000001e0800000
[    0.000000] Initmem setup node 0 0000000000000000-00000001e0800000
[    0.000000]   NODE_DATA [00000001dfffb000 - 00000001dfffffff]
[    0.000000] Zone PFN ranges:
[    0.000000]   DMA      0x00000010 -> 0x00001000
[    0.000000]   DMA32    0x00001000 -> 0x00100000
[    0.000000]   Normal   0x00100000 -> 0x001e0800
[    0.000000] Movable zone start PFN for each node
[    0.000000] early_node_map[2] active PFN ranges
[    0.000000]     0: 0x00000010 -> 0x000000a0
[    0.000000]     0: 0x00000100 -> 0x001e0800
[    0.000000] SFI: Simple Firmware Interface v0.81 http://simplefirmware.org
[    0.000000] SMP: Allowing 2 CPUs, 0 hotplug CPUs
[    0.000000] PM: Registered nosave memory: 00000000000a0000 - 0000000000100000
[    0.000000] PCI: Warning: Cannot find a gap in the 32bit address range
[    0.000000] PCI: Unassigned devices with 32bit resource registers may break!
[    0.000000] Allocating PCI resources starting at 1e0900000 (gap:
1e0900000:400000)
[    0.000000] Booting paravirtualized kernel on Xen
[    0.000000] Xen version: 3.4.3.amazon (preserve-AD)
[    0.000000] setup_percpu: NR_CPUS:64 nr_cpumask_bits:64
nr_cpu_ids:2 nr_node_ids:1
[    0.000000] PERCPU: Embedded 28 pages/cpu @ffff8801dfc00000 s82880
r8192 d23616 u1048576
[171950.163048] Built 1 zonelists in Node order, mobility grouping on.
 Total pages: 1935244
[171950.163052] Policy zone: Normal
[171950.163055] Kernel command line: root=LABEL=cloudimg-rootfs ro console=hvc0
[171950.163738] PID hash table entries: 4096 (order: 3, 32768 bytes)
[171950.176337] Checking aperture...
[171950.185622] No AGP bridge found
[171950.219380] Memory: 7613252k/7872512k available (6546k kernel
code, 448k absent, 258812k reserved, 6647k data, 920k init)
[171950.219498] SLUB: Genslabs=15, HWalign=64, Order=0-3,
MinObjects=0, CPUs=2, Nodes=1
[171950.219543] Hierarchical RCU implementation.
[171950.219545] 	RCU dyntick-idle grace-period acceleration is enabled.
[171950.219559] NR_IRQS:4352 nr_irqs:288 16
[171950.219697] Console: colour dummy device 80x25
[171950.219819] console [tty0] enabled
[171950.219939] console [hvc0] enabled
[171950.229462] allocated 63963136 bytes of page_cgroup
[171950.229482] please try 'cgroup_disable=memory' option if you don't
want memory cgroups
[171950.229554] installing Xen timer for CPU 0
[171950.229603] Detected 1795.673 MHz processor.
[171950.229614] Calibrating delay loop (skipped), value calculated
using timer frequency.. 3591.34 BogoMIPS (lpj=7182692)
[171950.229627] pid_max: default: 32768 minimum: 301
[171950.229676] Security Framework initialized
[171950.229696] AppArmor: AppArmor initialized
[171950.229702] Yama: becoming mindful.
[171950.232378] Dentry cache hash table entries: 1048576 (order: 11,
8388608 bytes)
[171950.236915] Inode-cache hash table entries: 524288 (order: 10,
4194304 bytes)
[171950.237831] Mount-cache hash table entries: 256
[171950.238064] Initializing cgroup subsys cpuacct
[171950.238077] Initializing cgroup subsys memory
[171950.238097] Initializing cgroup subsys devices
[171950.238105] Initializing cgroup subsys freezer
[171950.238111] Initializing cgroup subsys blkio
[171950.238128] Initializing cgroup subsys perf_event
[171950.238243] ENERGY_PERF_BIAS: Set to 'normal', was 'performance'
[171950.238244] ENERGY_PERF_BIAS: View and update with x86_energy_perf_policy(8)
[171950.238263] CPU: Physical Processor ID: 0
[171950.238268] CPU: Processor Core ID: 5
[171950.238411] SMP alternatives: switching to UP code
[171950.288157] ftrace: allocating 26549 entries in 105 pages
[171950.300850] cpu 0 spinlock event irq 17
[171950.300886] Performance Events: unsupported p6 CPU model 45 no PMU
driver, software events only.
[171950.301061] NMI watchdog disabled (cpu0): hardware events not enabled
[171950.301166] installing Xen timer for CPU 1
[171950.301186] cpu 1 spinlock event irq 23
[171950.301217] SMP alternatives: switching to SMP code
[171950.346136] NMI watchdog disabled (cpu1): hardware events not enabled
[171950.346173] Brought up 2 CPUs
[171950.347820] devtmpfs: initialized
[171950.348909] EVM: security.selinux
[171950.348915] EVM: security.SMACK64
[171950.348920] EVM: security.capability
[171950.349961] Grant table initialized
[171950.350028] print_constraints: dummy:
[171950.369907] RTC time: 165:165:165, date: 165/165/65
[171950.369957] NET: Registered protocol family 16
[171950.370155] Trying to unpack rootfs image as initramfs...
[171950.371138] PCI: setting up Xen PCI frontend stub
[171950.372318] bio: create slab <bio-0> at 0
[171950.372431] ACPI: Interpreter disabled.
[171950.372617] xen/balloon: Initialising balloon driver.
[171950.373754] xen-balloon: Initialising balloon driver.
[171950.373924] vgaarb: loaded
[171950.374050] i2c-core: driver [aat2870] using legacy suspend method
[171950.374057] i2c-core: driver [aat2870] using legacy resume method
[171950.374137] SCSI subsystem initialized
[171950.374247] usbcore: registered new interface driver usbfs
[171950.374263] usbcore: registered new interface driver hub
[171950.374298] usbcore: registered new device driver usb
[171950.387271] Freeing initrd memory: 12484k freed
[171950.391046] PCI: System does not support PCI
[171950.391054] PCI: System does not support PCI
[171950.391195] NetLabel: Initializing
[171950.391201] NetLabel:  domain hash size = 128
[171950.391206] NetLabel:  protocols = UNLABELED CIPSOv4
[171950.391224] NetLabel:  unlabeled traffic allowed by default
[171950.391316] Switching to clocksource xen
[171950.400785] AppArmor: AppArmor Filesystem Enabled
[171950.400812] pnp: PnP ACPI: disabled
[171950.404438] NET: Registered protocol family 2
[171950.405176] IP route cache hash table entries: 262144 (order: 9,
2097152 bytes)
[171950.421455] TCP established hash table entries: 524288 (order: 11,
8388608 bytes)
[171950.423379] TCP bind hash table entries: 65536 (order: 8, 1048576 bytes)
[171950.423569] TCP: Hash tables configured (established 524288 bind 65536)
[171950.423582] TCP reno registered
[171950.423648] UDP hash table entries: 4096 (order: 5, 131072 bytes)
[171950.423738] UDP-Lite hash table entries: 4096 (order: 5, 131072 bytes)
[171950.423855] NET: Registered protocol family 1
[171950.423879] PCI-DMA: Using software bounce buffering for IO (SWIOTLB)
[171950.423891] Placing 64MB software IO TLB between ffff8800fb8f7000
- ffff8800ff8f7000
[171950.423902] software IO TLB at phys 0xfb8f7000 - 0xff8f7000
[171950.423975] platform rtc_cmos: registered platform RTC device (no
PNP device found)
[171950.424419] audit: initializing netlink socket (disabled)
[171950.424437] type=2000 audit(1377573650.743:1): initialized
[171950.451752] HugeTLB registered 2 MB page size, pre-allocated 0 pages
[171950.453710] VFS: Disk quotas dquot_6.5.2
[171950.453805] Dquot-cache hash table entries: 512 (order 0, 4096 bytes)
[171950.454453] fuse init (API version 7.17)
[171950.454588] msgmni has been set to 14894
[171950.455175] Block layer SCSI generic (bsg) driver version 0.4
loaded (major 253)
[171950.455240] io scheduler noop registered
[171950.455249] io scheduler deadline registered (default)
[171950.455298] io scheduler cfq registered
[171950.455398] pci_hotplug: PCI Hot Plug PCI Core version: 0.5
[171950.455429] pciehp: PCI Express Hot Plug Controller Driver version: 0.4
[171950.456003] Serial: 8250/16550 driver, 32 ports, IRQ sharing enabled
[171950.510213] Linux agpgart interface v0.103
[171950.512010] brd: module loaded
[171950.512973] loop: module loaded
[171950.515095] blkfront device/vbd/2049 num-ring-pages 1 nr_ents 32.
[171950.518799] blkfront device/vbd/2096 num-ring-pages 1 nr_ents 32.
[171950.519676] blkfront: xvda1: barrier or flush: disabled
[171950.522562] blkfront: xvdd: barrier or flush: disabled
[171950.524105] blkfront device/vbd/2080 num-ring-pages 1 nr_ents 32.
[171950.525233]  xvdd: unknown partition table
[171950.525796] Setting capacity to 880732160
[171950.525808] xvdd: detected capacity change from 0 to 450934865920
[171950.527542] blkfront device/vbd/2288 num-ring-pages 1 nr_ents 32.
[171950.529071] blkfront: xvdc: barrier or flush: disabled
[171950.530136]  xvdc: unknown partition table
[171950.530369] Setting capacity to 880732160
[171950.530384] xvdc: detected capacity change from 0 to 450934865920
[171950.531891] blkfront device/vbd/2064 num-ring-pages 1 nr_ents 32.
[171950.532873] Fixed MDIO Bus: probed
[171950.532904] tun: Universal TUN/TAP device driver, 1.6
[171950.532910] tun: (C) 1999-2004 Max Krasnyansky <maxk at qualcomm.com>
[171950.532978] PPP generic driver version 2.4.2
[171950.533048] Initialising Xen virtual ethernet driver.
[171950.533754] blkfront: xvdp: barrier or flush: disabled
[171950.534955] ehci_hcd: USB 2.0 'Enhanced' Host Controller (EHCI) Driver
[171950.534977] ohci_hcd: USB 1.1 'Open' Host Controller (OHCI) Driver
[171950.534995] uhci_hcd: USB Universal Host Controller Interface driver
[171950.535059] usbcore: registered new interface driver libusual
[171950.535095] i8042: PNP: No PS/2 controller found. Probing ports directly.
[171950.535959]  xvdp: unknown partition table
[171951.038453] blkfront: xvdb: barrier or flush: disabled
[171951.040542]  xvdb: unknown partition table
[171951.540377] Setting capacity to 209715200
[171951.540395] xvdb: detected capacity change from 0 to 107374182400
[171951.540966] i8042: No controller found
[171951.541142] mousedev: PS/2 mouse device common for all mice
[171951.581047] rtc_cmos rtc_cmos: rtc core: registered rtc_cmos as rtc0
[171951.581127] rtc_cmos: probe of rtc_cmos failed with error -38
[171951.581237] device-mapper: uevent: version 1.0.3
[171951.581341] device-mapper: ioctl: 4.22.0-ioctl (2011-10-19)
initialised: dm-devel at redhat.com
[171951.581356] EFI Variables Facility v0.08 2004-May-17
[171951.581682] TCP cubic registered
[171951.581808] NET: Registered protocol family 10
[171951.582654] NET: Registered protocol family 17
[171951.582666] Registering the dns_resolver key type
[171951.582870] registered taskstats version 1
[171951.595600] XENBUS: Device with no driver: device/console/0
[171951.595623]   Magic number: 1:252:3141
[171951.595668] /build/buildd/linux-3.2.0/drivers/rtc/hctosys.c:
unable to open rtc device (rtc0)
[171951.595686] BIOS EDD facility v0.16 2004-Jun-25, 0 devices found
[171951.595692] EDD information not available.
[171951.596279] Freeing unused kernel memory: 920k freed
[171951.596626] Write protecting the kernel read-only data: 12288k
[171951.603890] Freeing unused kernel memory: 1628k freed
[171951.605088] Freeing unused kernel memory: 1192k freed
Loading, please wait...
[171951.653649] udevd[86]: starting version 175
Begin: Loading essential drivers ... done.
Begin: Running /scripts/init-premount ... done.
Begin: Mounting root file system ... Begin: Running /scripts/local-top ... done.
Begin: Running /scripts/local-premount ... done.
[171951.956780] EXT4-fs (xvdp): mounted filesystem with ordered data
mode. Opts: (null)
Begin: Running /scripts/local-bottom ... done.
done.
Begin: Running /scripts/init-bottom ... done.
[171953.104898] SPL: Loaded module v0.6.1-rc14
[171953.115445] zunicode: module license 'CDDL' taints kernel.
[171953.115455] Disabling lock debugging due to kernel taint
[171953.421663] ZFS: Loaded module v0.6.1-rc14, ZFS pool version 5000,
ZFS filesystem version 5
[171953.434461] SPL: using hostid 0x1d0aa6ed
[171953.609044] EXT4-fs (xvdp): re-mounted. Opts: (null)
swapon: /dev/xvdc: read swap header failed: Invalid argument

mountall: swapon /dev/xvdc [454] terminated with status 255

mountall: Problem activating swap: /dev/xvdc