[Lxc-users] Write operation to /sys/bus/rbd/add fails inside docker container

[Omar Marquez]

Hi All,


We're trying to mount an rbd image inside of a linux container that has been created with docker (https://www.docker.io/). We seem to have access to the rbd kernel module from inside the container:

# lsmod | grep ceph
libceph               218854  1 rbd
libcrc32c              12603  3 xfs,libceph,dm_persistent_data

And we can query the pool for available rbds and create rbds from inside the container:

# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring create lxctest --size 51200
# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring ls
lxctest

But for some reason, we can't seem to map the device to the container:

# rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring map lxctest
rbd: add failed: (22) Invalid argument

I don't see anything particularly interesting in dmesg or messages on either the container or the host box. Any ideas on how to troubleshoot this?

….

An strace produces the below results:

#  strace rbd -p dockers --id dockers --keyring /etc/ceph/ceph.client.dockers.keyring map test

…
futex(0xb5637c, FUTEX_WAKE_OP_PRIVATE, 1, 1, 0xb56378, {FUTEX_OP_SET, 0, FUTEX_OP_CMP_GT, 1}) = 1
futex(0xb562f8, FUTEX_WAKE_PRIVATE, 1)  = 1
add_key(0x424408, 0x7fff82c4e210, 0x7fff82c4e140, 0x22, 0xfffffffe) = 607085216
stat("/sys/bus/rbd", {st_mode=S_IFDIR|0755, st_size=0, ...}) = 0
open("/sys/bus/rbd/add", O_WRONLY)      = 3
write(3, "10.198.41.6:6789<http://10.198.41.6:6789/>,10.198.41.8:678<http://10.198.41.8:678/>"..., 96) = -1 EINVAL (Invalid argument)
close(3)                                = 0
rt_sigaction(SIGINT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0
rt_sigaction(SIGQUIT, {SIG_IGN, [], SA_RESTORER, 0x7fbf8a7efa90}, {SIG_DFL, [], 0}, 8) = 0
rt_sigprocmask(SIG_BLOCK, [CHLD], [PIPE], 8) = 0
clone(child_stack=0, flags=CLONE_PARENT_SETTID|SIGCHLD, parent_tidptr=0x7fff82c4e040) = 22
wait4(22, [{WIFEXITED(s) && WEXITSTATUS(s) == 0}], 0, NULL) = 22
rt_sigaction(SIGINT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, NULL, 8) = 0
rt_sigaction(SIGQUIT, {SIG_DFL, [], SA_RESTORER, 0x7fbf8a7efa90}, NULL, 8) = 0
rt_sigprocmask(SIG_SETMASK, [PIPE], NULL, 8) = 0
write(2, "rbd: add failed: ", 17rbd: add failed: )       = 17
write(2, "(22) Invalid argument", 21(22) Invalid argument)   = 21
write(2, "\n", 1
)                       = 1
exit_group(1)                           = ?
+++ exited with 1 +++


The app is run inside the container with setuid = 0 and the container is able to mount all required filesystems …
Could it be that is not possible to write to /sys/bus/rbd/add inside an lxc container ?

Thx,
Om

________________________________

The information in this e-mail is intended only for the person or entity to which it is addressed.

It may contain confidential and /or privileged material. If someone other than the intended recipient should receive this e-mail, he / she shall not be entitled to read, disseminate, disclose or duplicate it.

If you receive this e-mail unintentionally, please inform us immediately by "reply" and then delete it from your system. Although this information has been compiled with great care, neither IMC Financial Markets & Asset Management nor any of its related entities shall accept any responsibility for any errors, omissions or other inaccuracies in this information or for the consequences thereof, nor shall it be bound in any way by the contents of this e-mail or its attachments. In the event of incomplete or incorrect transmission, please return the e-mail to the sender and permanently delete this message and any attachments.

Messages and attachments are scanned for all known viruses. Always scan attachments before opening them.
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.linuxcontainers.org/pipermail/lxc-users/attachments/20131018/92e19e35/attachment.html>