<div dir="ltr"><span id="gmail-docs-internal-guid-fb90a344-493a-4c49-762f-f891c279b2a2"><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;vertical-align:baseline;white-space:pre-wrap">Greetings,</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;vertical-align:baseline;white-space:pre-wrap">The TL;DR - we don’t fully support this today, but are cycling towards a resolution. I would love to have your thoughts/requirements added to the bug listed below.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">I've given this thread some thought and you're encountering an edge that we haven't thoroughly tested. We do have a desire to enable developers to properly model their workloads in kubernetes running on LXD just like they would on a cloud.  This thread was started by the conjure-up folks: </span><a href="https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/202" style="text-decoration:none"><span style="font-size:11pt;font-family:arial;background-color:transparent;text-decoration:underline;vertical-align:baseline;white-space:pre-wrap">https://github.com/juju-solutions/bundle-canonical-kubernetes/issues/202</span></a><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> that somewhat explores the initial thoughts on this work.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">I spent about an hour diving into the Ceph integration path we have already completed to see if it is a viable option, but my results were not successful, and reproduction seems to be order dependent. This is not an ideal solution.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">What I can say is that we are aware of this limitation, and would love to enable this. We’re looking towards a lighter weight solution (like gluster or nfs) for the initial enablement on local development setups.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">I’ll include those links and a bit of instruction from my Ceph hacking for further reading material just in case you feel like diving in and hacking on that vector:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Solving for RBD Mount/Format permissions denied</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/lxc/lxd/issues/2709">https://github.com/lxc/lxd/issues/2709</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Install ceph-common on the host</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">apt-get install ceph-common</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Next step would be to stand up CDK</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">conjure-up canonical-kubernetes</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Evaluate the nova-lxd lxd profile for kernel modules and escalated security on the container.. Yielding a less secure lxd container in this configuration, but more operability in this context:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/conjure-up/spells/blob/master/openstack-novalxd/steps/lxd-profile.yaml">https://github.com/conjure-up/spells/blob/master/openstack-novalxd/steps/lxd-profile.yaml</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Specifically, you're going to need to add some whitelisted modules, set the container to privileged, and add the rbd devices (min + major - found via lsblk /dev/rbd# - order dependent, as the device has to exist first)</span></p><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">(note, I'm using the snap so my commands will be prefixed with lxd. to scope the request to the snap bins, this may be divergent from your commands which will just be native lxc profile show, and so on)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap"> $ lxd.lxc profile show juju-storage-test</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">config:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  boot.autostart: "true"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  linux.kernel_modules: openvswitch,nbd,ip_tables,ip6_tables,netlink_diag,rbd</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  raw.lxc: |</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">    lxc.aa_profile=unconfined</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">    lxc.mount.auto=sys:rw</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  security.nesting: "true"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  security.privileged: "true"</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">description: ""</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">devices:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">  root:</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">    path: /</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">    type: disk</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">name: juju-storage-test</span></p><br><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Once the deployment has converged and it’s pulled down your credentials, you're ready to deploy Ceph and start enlisting OSD's (using the file storage type)</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"># note: you will indeed need 6 total lxd containers to run the ceph service. 3 mons for quorum, and 3 osd’s to ensure your cluster health. I tried with one and this failed.</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">juju deploy ceph-mon -n 3</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">juju deploy ceph-osd -n 3</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">juju add-relation ceph-mon ceph-osd</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">juju config ceph-osd osd-devices=/srv/ceph-osd use-direct-io=false</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Juju add-relation kubernetes-master ceph-mon</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Juju run-action kubernetes-master/0 create-rbd-pv name=testpv size=50</span></p><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:"ubuntu mono";background-color:transparent;vertical-align:baseline;white-space:pre-wrap">-- this is where things failed for me with regard to either unable to mount the RBD due to it thinks there’s a mounted filesystem on it (I presume watchers were to blame)</span></p><br><br><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;font-weight:700;vertical-align:baseline;white-space:pre-wrap">Cited sources for the answers:</span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Nova-lxd bundle configuration for ceph units</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/conjure-up/spells/blob/master/openstack-novalxd/bundle.yaml#L60-L76">https://github.com/conjure-up/spells/blob/master/openstack-novalxd/bundle.yaml#L60-L76</a></span></p><br><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap">Enable loopback storage support on the localhost provider for Juju (unreferenced)</span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><span style="font-size:11pt;font-family:arial;background-color:transparent;vertical-align:baseline;white-space:pre-wrap"><a href="https://github.com/juju/docs/issues/1665">https://github.com/juju/docs/issues/1665</a></span></p><p dir="ltr" style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">cholcomb, and icey on #juju on freenode (storage engineers)</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt">stokachu and lazypower on #juju on freenode (kubernetes engineers)</p><p style="line-height:1.38;margin-top:0pt;margin-bottom:0pt"><br></p><br></span></div>