[lxc-devel] [lxd/master] VM: Detect broken/hung qemu process and reflect in LXD state output

[tomponline on Github]

A non-text attachment was scrubbed...
Name: not available
Type: text/x-mailbox
Size: 472 bytes
Desc: not available
URL: <http://lists.linuxcontainers.org/pipermail/lxc-devel/attachments/20201001/63f35f58/attachment.bin>
-------------- next part --------------
From c2ea7cef7d6cee39ad87acec8024f408f172202f Mon Sep 17 00:00:00 2001
From: Thomas Parrott <thomas.parrott at canonical.com>
Date: Thu, 1 Oct 2020 16:07:03 +0100
Subject: [PATCH 1/2] lxd/instance/drivers/driver/qemu: Updates IsRunning to
 not check for BROKEN state

As this can never ben returned from StatusCode.String() which is what vm.State() returns.

Signed-off-by: Thomas Parrott <thomas.parrott at canonical.com>
---
 lxd/instance/drivers/driver_qemu.go | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/lxd/instance/drivers/driver_qemu.go b/lxd/instance/drivers/driver_qemu.go
index b7c0360223..bbd4f1d899 100644
--- a/lxd/instance/drivers/driver_qemu.go
+++ b/lxd/instance/drivers/driver_qemu.go
@@ -4221,7 +4221,7 @@ func (vm *qemu) agentGetState() (*api.InstanceState, error) {
 // IsRunning returns whether or not the instance is running.
 func (vm *qemu) IsRunning() bool {
 	state := vm.State()
-	return state != "BROKEN" && state != "STOPPED"
+	return state != "STOPPED"
 }
 
 // IsFrozen returns whether the instance frozen or not.

From 5ad910fcdfcb63e2daab337d1fefa0ede0a6f6ff Mon Sep 17 00:00:00 2001
From: Thomas Parrott <thomas.parrott at canonical.com>
Date: Thu, 1 Oct 2020 16:08:24 +0100
Subject: [PATCH 2/2] lxd/instance/drivers/driver/qemu: Updates statusCode() to
 detect if monitor failure with running VM

If the QMP monitor cannot connect, but VM process mentioned in qemu.pid still exists, then return error state, as qemu has likely crashed/hung.

Signed-off-by: Thomas Parrott <thomas.parrott at canonical.com>
---
 lxd/instance/drivers/driver_qemu.go | 7 +++++++
 1 file changed, 7 insertions(+)

diff --git a/lxd/instance/drivers/driver_qemu.go b/lxd/instance/drivers/driver_qemu.go
index bbd4f1d899..3ced883459 100644
--- a/lxd/instance/drivers/driver_qemu.go
+++ b/lxd/instance/drivers/driver_qemu.go
@@ -4304,6 +4304,13 @@ func (vm *qemu) statusCode() api.StatusCode {
 	// Connect to the monitor.
 	monitor, err := qmp.Connect(vm.monitorPath(), qemuSerialChardevName, vm.getMonitorEventHandler())
 	if err != nil {
+		// If cannot connect to monitor, but qemu process in pid file still exists, then likely qemu
+		// has crashed/hung and this instance is in an error state.
+		pid, _ := vm.pid()
+		if pid > 0 && shared.PathExists(fmt.Sprintf("/proc/%d", pid)) {
+			return api.Error
+		}
+
 		// If we fail to connect, chances are the VM isn't running.
 		return api.Stopped
 	}