To get an overall status of your cluster, run the tests from the command line. There are two types of tests: testinfra tests verify the setup, and end-to-end tests that check the webservice availability.

Testinfra tests

Testinfra tests are split into two groups, lets call them blackbox and clearbox tests. The blackbox tests run on your provisioning machine and test the Stackspin cluster from the outside. For example, the certificate check will check if the Stackspin returns valid certificates for the provided services. The clearbox tests run on the Stackspin host and check i.e. if docker is installed in the right version etc. Our testinfra tests are a combination of blackbox and clearbox tests.

If you have installed the Stackspin python requirements inside a virtual environment then you need to activate it before testing:

$ . env/bin/activate

Then enter the test directory in the Git repository on your provisioning machine.

$ cd test

To run the test against your cluster, first export the CLUSTER_DIR environment variable with the location of your cluster config directory (replace stackspin.example.org with your cluster name): Then export the KUBECONFIG variable that lets kubectl talk to your cluster. Thirdly, export a variable that contains the password for the Prometheus HTTP endpoint.

$ export CLUSTER_DIR="${PWD}/../clusters/stackspin.example.org"
$ export KUBECONFIG=${CLUSTER_DIR}/kube_config_cluster.yml
$ export BASIC_AUTH_PW=$(kubectl get secret -n stackspin stackspin-prometheus-basic-auth --template '{{ .data.pass }}' | base64 -d)


Run all tests

$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'

Test all applications

This will check for:

  • The applications return proper certificates

  • All helm releases are successfully installed

  • All app pods are running and healthy (this test includes all optional applications)

These tests includes all optional applications and will fail for optional applications that are not installed.

$ pytest -s -m 'app' --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'

Tests a specific application

$ pytest -s -m 'app' --app="wordpress" --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'

Advanced usage of Testinfra tests

Specify host manually:

$ py.test -s --hosts='ssh://root@example.stackspin.net'

Run only tests tagged with prometheus:

$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus

Run cert test manually using the ansible inventory file:

$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs

Run cert test manually against a different cluster, not configured in any ansible inventory file, either by using pytest:

$ FQDN='example.stackspin.net' py.test -sv -m 'certs'

or directly:

$ FQDN='example.stackspin.net' pytest/test_certs.py

Running Testinfra tests with local gitlab-runner docker executor

Export the following environment variables like this:

$ export CI_REGISTRY_IMAGE='open.greenhost.net:4567/stackspin/stackspin'
$ export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_stackspin_ci)"
$ export COSMOS_API_TOKEN='…'



Known Issues

The default ssh backend for testinfra tests is paramiko, which does not work out of the box. It fails to connect to the host because the ed25519 hostkey is not verified. Therefore we need to force plain ssh:// with either connection=ssh or --hosts=ssh://…

End to end tests


We are migrating our end to end tests from Taiko to Cypress. You can track the progress in this issue. You need to check below which applications can be tested with which framework. We are sorry for the current inconvenience, we hope we’ll soon finish the migration.

End to end tests run in a browser to test if all the web services are up and running and functioning correctly.

Taiko tests

Used for testing these applications:

  • Dashboard

  • Wekan

  • Wordpress

  • Zulip

Install Taiko on your provisioning machine:

$ npm install -g taiko

Run Taiko tests

Taiko tests are integrated in the stackspin CLI command suite. To run all Taiko tests, run the following command in this repository:

$ python -m stackspin CLUSTERNAME test

To learn more about the test subcommand, run:

$ python -m stackspin CLUSTERNAME test --help

You can also only run a Taiko test for a specific application, i.e.:

$ python -m stackspin CLUSTERNAME test --taiko-tags nextcloud

Advanced Taiko tests

If you want to use Taiko without invoking the stackspin CLI, go to the test/taiko directory and run:

For nextcloud & onlyoffice tests:

$ export DOMAIN='stackspin.example.net'
$ export SSO_USERNAME='user1'
$ export SSO_USER_PW='...'
$ export TAIKO_TESTS='nextcloud'
$ taiko --observe taiko-tests.js

You can replace nextcloud with grafana or wordpress to test the other applications, or with all to test all applications.

Cypress tests

Cypress tests are not yet integrated in the stackspin CLI command suite. Used for testing these applications:

  • Grafana

  • Nextcloud

To install Cypress on your provisioning machine go to the test/ directory and run:

$  npm install -g cypress

Test Grafana

$ export CYPRESS_BASE_URL='https://grafana.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed

Test nextcloud

The cypress end to end test is not included in the main Stackspin repository, but in the Nextcloud helm chart repository.

$ git clone https://open.greenhost.net/stackspin/nextcloud.git
$ export CYPRESS_BASE_URL='https://files.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed