<div dir="ltr"><div>btw, there is also is another "glitch". When I run that gradle script from
console, after it finishes, I need to reset terminal, otherwise I do not
see echo from keys while typing.<br><br>At the end is some parts from host gradle script, which is responsible for launching/managing lxc.<br></div><div>init function is used to initialize container and launch to start.<br>
</div><div>In my scenario we need 4 machines to perform single test.<br></div><div>1st is main and 3 other is running different software.<br></div><div><br></div><div></div><div>on main container there is also injected gradle script which actually performs tests.<br>
</div><div>main - run injected gradle script<br></div><div>2,3,4 - run some program<br></div><div><br></div><div>part of injected script on main:<br>task testx(dependsOn: ":ready") << {<br> module = module.replace("_sub_", "/")<br>
println "MODULE PROCESSING: " + module<br> JmxLauncher jmxc = new JmxLauncher(module: module, basePath: basePath, ips: ips, logDir: logDir)<br> jmxc.scan()<br> println ":*:exit"<br>}<br></div>
<div><br></div><div>Currently i wrote different function to terminate guests, which literally kills processes inside container and therefore container terminates, maybe not the best solution but as workaround works well:<br>
static def killContainer = { String cname -><br> println "Stopping $cname"<br> "cat ${cgroupMount()}/lxc/$cname/cgroup.procs".execute().in.text.eachLine {<br> println "killing processid: $it"<br>
"kill -9 $it".execute().waitFor()<br> }<br> }<br><br><br></div><div><br>build.gradle (main script on host) -><br>class LxcProcess extends Thread {<br> String mod, type, cmd, scmd, fcmd, cmdPrefix = ":*:"<br>
CmdListener listener<br> int retry = 1<br> boolean stdout = false, stderr = false<br> private Process proc<br> private boolean shutdown = false, finished = false<br> private final Object lock = new Object();<br>
private InputStream out, err<br><br> def isAlive = { Process p -><br> try {<br> p.exitValue()<br> false<br> } catch (Exception e) {<br> true<br> }<br> }<br>
<br> private def stopProc = {<br> //Process fproc = fcmd.execute()<br> //fproc.waitFor()<br> Fn.killContainer("$mod-$type")<br>/*<br> if (isAlive(proc)) {<br> println "Can't stop $container"<br>
}<br>*/<br> }<br><br> private def closeStreams() {<br> try {<br> if (out) out.close()<br> } catch (Exception e) {<br> }<br> try {<br> if (err) err.close()<br>
} catch (Exception e) {<br> }<br> }<br><br> private def startProc = {<br>// println "START PROC !!!!\n CMD: $cmd\n SCMD: $scmd\n FCMD: $fcmd"<br> closeStreams()<br> proc = scmd.execute()<br>
out = <a href="http://proc.in">proc.in</a><br> err = proc.err<br> String prefix = "$mod-$type"<br><br> if (stderr) {<br> new OutputReader(inputStream: err, prefix: prefix + " err:").start()<br>
}<br> if (stdout || listener) {<br> if (!listener) {<br> println "READER WITHOUT LISTENER"<br> new OutputReader(inputStream: out, prefix: prefix + " out:").start()<br>
} else {<br> println "READER WITH LISTENER"<br> new OutputReader(inputStream: out, prefix: prefix + " out:", listener: listener, output: stdout, cmdPrefix: cmdPrefix).start()<br>
}<br> }<br> }<br><br> public void run() {<br> scmd = "lxc-start -n $mod-$type -- $cmd"<br> fcmd = "lxc-stop -n $mod-$type -k"<br><br> while (!shutdown) {<br>
if (!proc)<br> startProc()<br> int exit = proc.waitFor()<br> println "EXIT CODE: $exit"<br> if (exit == 0)<br> shutdown = true<br> if (!shutdown) {<br>
synchronized (lock) { //sync with restart command to avoid duplicates<br> if (retry > 0) {<br> startProc()<br> retry--<br> } else<br>
shutdown = true<br> }<br> }<br> sleep(100)<br>// println "AFTER SLEEP: " + shutdown<br> }<br>// println "Shutdown: " + shutdown<br>
finished = true<br> }<br><br> def restart() {<br> synchronized (lock) {<br> stopProc()<br> proc = scmd.execute()<br> }<br> }<br><br> def finish() {<br> shutdown = true;<br>
stopProc()<br> while (!finished)<br> sleep(50)<br> }<br><br> def waitFor() {<br>// println "PROC NULL:" + proc == null<br> //proc.waitFor()<br> while (!finished)<br>
sleep(50)<br> }<br><br> public boolean isFinished() {<br> finished<br> }<br>}<br><br>class OutputReader extends Thread {<br> public InputStream inputStream<br> public String prefix<br> public CmdListener listener<br>
public boolean output = true<br> public cmdPrefix = ":*:"<br><br> public void run() {<br> BufferedReader reader = new BufferedReader(new InputStreamReader(inputStream))<br> String line<br>
try {<br> while ((line = reader.readLine()) != null) {<br> if (output) println(prefix ? prefix + ": " + line : line)<br> if (listener && line.startsWith(cmdPrefix)) {<br>
// println "EVENT: " + line.substring(cmdPrefix.length() - 1)<br> listener.event(line.substring(cmdPrefix.length()))<br> }<br> }<br> } catch (IOException e) {<br>
} finally {<br> try {<br> if (reader != null)<br> reader.close()<br> if (inputStream != null)<br> inputStream.close()<br> } catch (Exception e) {<br>
e.printStackTrace()<br> }<br> }<br> }<br>}<br><br>class Instance implements CmdListener {<br> Map<String, Container> containers<br> def mod<br> java.nio.file.Path rwPath, logDir<br>
private LxcProcess mgmtProc, routerProc, mpProc, mainProc<br> private boolean mainReady = false, destroyed = false<br><br> def init() {<br> if (destroyed) return<br> Fn.execShell "brctl addbr ${Fn.checksum(mod)}br1"<br>
Fn.execShell "ifconfig ${Fn.checksum(mod)}br1 up"<br> containers.each { it.value.init(mod) }<br> }<br><br> def launch() {<br> if (destroyed) return<br> containers["main"].start(mod, this) // we will receive events from main<br>
boolean isRunning = true<br> while (isRunning) {<br> if (!containers["main"].isFinished() && !mainReady) {<br> sleep(50)<br> } else if (containers["main"].isFinished()) {<br>
isRunning = false<br> } else if (mainReady) {<br> ["mp", "router", "mgmt"].each {<br> if (containers[it].isFinished()) {<br> isRunning = false<br>
}<br> }<br> }<br> sleep(50)<br> }<br> [_cnlist].each {<br> if (!containers[it].isFinished())<br> containers[it].finish()<br>
}<br> }<br><br> def destroy() {<br> containers.each { it.value.destroy() }<br> Fn.execShell "ifconfig ${Fn.checksum(mod)}br1 down"<br> Fn.execShell "brctl delbr ${Fn.checksum(mod)}br1"<br>
destroyed = true<br> }<br><br> def collectLogs() {<br> try {<br> def srcLogDir = rwPath.resolve("$mod-main/$mod-main/var/log")<br> def dstLogDir = logDir.resolve("$mod")<br>
println "Source log dir: " + srcLogDir.toString() + "\n Dst log dir: " + dstLogDir.toString()<br> copy {<br> from(srcLogDir.toString()) {<br> include '**/*'<br>
}<br> into dstLogDir.toString()<br> }<br> println "Collect 4"<br> } catch (Exception e) {<br> e.printStackTrace()<br> }<br> }<br><br>
def finish() {<br><br> }<br><br> def getLogs() {<br><br> }<br><br> @Override<br> void event(String cmdLine) {<br> String[] cmd = cmdLine.split(" ")<br> switch (cmd[0]) {<br> case "ready":<br>
containers["mgmt"].start(mod)<br> containers["router"].start(mod)<br> containers["mp"].start(mod)<br> mainReady = true;<br> break;<br>
case "start":<br> containers[cmd[1]].start(mod)<br> break<br> case "stop":<br> containers[cmd[1]].finish()<br> break<br>
case "restart":<br> containers[cmd[1]].finish()<br> containers[cmd[1]].start(mod)<br> break<br> case "exit":<br> containers["main"].finish()<br>
break<br> default:<br> println "UNKNOW COMMMAND: " + cmdLine<br> }<br> }<br>}<br><br>class Container {<br> String rwPath, runCmd, name, lxcBasePath = "/var/lib/lxc"<br>
String[] paths<br> private LxcProcess proc<br> Map bindings<br> static def templateEngine = new SimpleTemplateEngine()<br> static def templateText = new File('config.template').text<br> private boolean destroyed = false<br>
private def ant = new groovy.util.AntBuilder()<br> private def initMod<br><br> def init(def modName, CmdListener listener = null) {<br> def mainMountEntry = "lxc.mount.entry=/tmp2/writeable $lxcBasePath/$modName-$name/rootfs/slave none bind 0 0"<br>
if (destroyed)<br> return<br> initMod = modName<br> def cname = modName + "-" + name<br> def writeDir = "${rwPath}/${modName}-${name}"<br> ant.delete(dir: "$lxcBasePath/${cname}", quiet: true)<br>
ant.mkdir(dir: "$lxcBasePath/${cname}")<br> ant.mkdir(dir: "$lxcBasePath/${cname}/rootfs")<br> ant.delete(dir: writeDir, quiet: true)<br> ant.mkdir(dir: writeDir)<br> bindings["localBridge"] = "${Fn.checksum(modName)}br1"<br>
bindings["publicBridge"] = "pubbr0"<br> bindings["gw"] = "10.0.0.1"<br> Fn.execShell "mount -t aufs -o br=${writeDir}=rw:${paths.join(":")} -o udba=reval none ${lxcBasePath + "/${cname}/rootfs"}"<br>
def binding = ["privateIp": bindings["privateIp"], "localBridge": bindings["localBridge"], "publicIp": bindings["publicIp"],<br> "publicBridge": bindings["publicBridge"], "gw": bindings["gw"], "srvname": cname]<br>
//bindings for template<br> def template = templateEngine.createTemplate(templateText).make(binding)<br> if (name.equals("main")) {<br> ant.mkdir(dir: "$lxcBasePath/$modName-$name/rootfs/slave")<br>
Files.write(Paths.get(lxcBasePath + "/${cname}/config"), (template.toString() + "\n$mainMountEntry\n").getBytes())<br> } else<br> Files.write(Paths.get(lxcBasePath + "/${cname}/config"), template.toString().getBytes())<br>
ant.echo(file: "$lxcBasePath/${cname}/rootfs/etc/hosts", append: true, message: "${bindings["privateIp"]} $modName-$name\n")<br> proc = new LxcProcess(mod: modName, type: name, cmd: runCmd)<br>
}<br><br> def destroy() {<br> def item = "$initMod-$name"<br> "lxc-ls --active".execute().text.split(" ").grep { item.toString().equals(it.toString()) }.each {<br> Fn.execShell "lxc-stop -n $item -k"<br>
}<br> Fn.execShell "umount -f $lxcBasePath/${item}/rootfs"<br> Fn.deleteDir("$lxcBasePath/${item}")<br> Fn.deleteDir("/tmp2/writeable/${item}")<br>// ant.delete(dir: "/tmp2/writeable/${it}", quiet: true)<br>
destroyed = true<br> }<br><br> def start(String modName, CmdListener listener = null) {<br> if (destroyed)<br> return<br> proc.listener = listener<br> proc.start()<br> }<br>
<br> def finish() {<br> proc.finish()<br> }<br><br> def waitFor() {<br> proc.waitFor()<br> }<br><br> def isFinished() {<br> proc.isFinished() || proc.getState() == Thread.State.TERMINATED<br>
}<br>}<br><br><br></div>template:<br>lxc.network.type = veth<br>lxc.network.flags = up<br>lxc.network.ipv4 = $privateIp/24<br>lxc.network.link = $localBridge<br><br>lxc.network.type = veth<br>lxc.network.flags = up<br>
lxc.network.ipv4 = $publicIp/16<br>lxc.network.link = $publicBridge<br>lxc.network.ipv4.gateway = $gw<br><br>lxc.utsname = $srvname<br>lxc.devttydir = lxc<br>lxc.tty = 1<br>lxc.pts = 1024<br>lxc.rootfs = /var/lib/lxc/$srvname/rootfs<br>
<br>lxc.mount.auto = proc:mixed sys:ro<br><br>lxc.mount.entry=proc /var/lib/lxc/$srvname/proc proc nodev,noexec,nosuid 0 0<br>lxc.mount.entry=sys /var/lib/lxc/$srvname/sys sysfs defaults 0 0<br>lxc.mount.entry=shm /var/lib/lxc/$srvname/dev/shm tmpfs rw,nosuid,nodev,noexec,relatime 0 0<br>
lxc.mount.entry=tmp /var/lib/lxc/$srvname/tmp tmpfs rw,nosuid,nodev,noexec 0 0<br>lxc.mount.entry=run /var/lib/lxc/$srvname/run tmpfs rw,nosuid,nodev,relatime,mode=755 0 0<br><br>lxc.cgroup.devices.deny = a<br>lxc.cgroup.devices.allow = c 1:3 rwm<br>
lxc.cgroup.devices.allow = c 1:5 rwm<br>lxc.cgroup.devices.allow = c 5:1 rwm<br>lxc.cgroup.devices.allow = c 5:0 rwm<br>lxc.cgroup.devices.allow = c 4:0 rwm<br>lxc.cgroup.devices.allow = c 4:1 rwm<br>lxc.cgroup.devices.allow = c 1:9 rwm<br>
lxc.cgroup.devices.allow = c 1:8 rwm<br>lxc.cgroup.devices.allow = c 136:* rwm<br>lxc.cgroup.devices.allow = c 5:2 rwm<br>lxc.cgroup.devices.allow = c 254:0 rwm<br><br>#lxc.cgroup.memory.limit_in_bytes = 512M<br></div><div class="gmail_extra">
<br><br><div class="gmail_quote">On 17 March 2014 14:25, Serge Hallyn <span dir="ltr"><<a href="mailto:serge.hallyn@ubuntu.com" target="_blank">serge.hallyn@ubuntu.com</a>></span> wrote:<br><blockquote class="gmail_quote" style="margin:0 0 0 .8ex;border-left:1px #ccc solid;padding-left:1ex">
<div class="">Quoting Haralds Ulmanis (<a href="mailto:haralds@evilezh.net">haralds@evilezh.net</a>):<br>
> So, when i run from shell that gradle script - everything is fine and<br>
> perfect. When i run from jenkins - I can't stop/kill, get info from<br>
> containers.<br>
> LXC is version 1.0.0 with kernel 3.13.0<br>
> Any suggestions ?<br>
<br>
</div>Please show us the gradle script.<br>
_______________________________________________<br>
lxc-users mailing list<br>
<a href="mailto:lxc-users@lists.linuxcontainers.org">lxc-users@lists.linuxcontainers.org</a><br>
<a href="http://lists.linuxcontainers.org/listinfo/lxc-users" target="_blank">http://lists.linuxcontainers.org/listinfo/lxc-users</a><br>
</blockquote></div><br></div>