[lxc-devel] RFC on testing plan

[Serge Hallyn]

The lxc/src/tests/ directory generally has contained tests which helped
me to write new features.  They are at the same time too detailed and too
quickly written to be very maintainable.  I've also had a separate
testsuite at lp:~serge-hallyn/+junk/lxc-test, which contained more
ubuntu-specific (and generally more macro-) tests.

I'm going to work toward a more sane and more complete test layout.
Ideally all tests would be in the git tree, and run on a 'make
check-all', to prevent tests going stale and try to minimize
regressions slipping by under the radar.

Here's what I'm thinking:

1. API tests - much of this is in src/tests now, but I'd like to
consolidate them quite a bit.  These should be run more easily with a
'make check' rule.  I intend to add this (and fix the fallout from all
the tests which have recently broken).  The tests need to be run with
LD_LIBRARY_PATH set to lxc/src/lxc and liblxc.so.1 symlinked to
liblxc.so.  We probably want to put these under src/tests/microtests/ or
somesuch.

2. distro-specific tests.  This would test things like network setups,
LSM configuration, and (for now) container autostart.  This could be
a per-distro dir under src/tests/$distro/, and executed by
'make check-distro' calling src/tests/lxc-test-distro-test (which
detects current distro and runs the appropriate test).

3. template tests.  Given the direction we recently set - that we want
all templates to be runnable under all distros - I believe we should
run every template in turn.  This could sit under src/tests/templates/
and started by 'make check-templates' which calls src/tests/lxc-test-templates.

4. bindings tests.  Under src/tests/bindings/$binding (lua, go, python3)
and executed by 'make check-bindings'.

Feedback welcome.  While I wait for that I intend to start on cleaning
up and consolidating the existing testcases (basically start on (1)).

-serge