Testing
To get an overall status of your cluster, run the tests from the command line. There are two types of tests: testinfra tests verify the setup, and end-to-end tests that check the webservice availability.
Testinfra tests
Testinfra tests are split into two groups, lets call them blackbox and clearbox tests. The blackbox tests run on your provisioning machine and test the Stackspin cluster from the outside. For example, the certificate check will check if the Stackspin returns valid certificates for the provided services. The clearbox tests run on the Stackspin host and check i.e. if docker is installed in the right version etc. Our testinfra tests are a combination of blackbox and clearbox tests.
If you have installed the Stackspin python requirements inside a virtual environment then you need to activate it before testing:
$ . env/bin/activate
Then enter the test directory in the Git repository on your provisioning machine.
$ cd test
To run the test against your cluster, first export the CLUSTER_DIR
environment variable with the location of your cluster config directory
(replace stackspin.example.org with your cluster name):
Then export the KUBECONFIG variable that lets kubectl talk to your cluster.
Thirdly, export a variable that contains the password for the Prometheus HTTP endpoint.
$ export CLUSTER_DIR="${PWD}/../clusters/stackspin.example.org"
$ export KUBECONFIG=${CLUSTER_DIR}/kube_config_cluster.yml
$ export BASIC_AUTH_PW=$(kubectl get secret -n stackspin stackspin-prometheus-basic-auth --template '{{ .data.pass }}' | base64 -d)
$
Run all tests
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Test all applications
This will check for:
The applications return proper certificates
All helm releases are successfully installed
All app pods are running and healthy (this test includes all optional applications)
These tests includes all optional applications and will fail for optional applications that are not installed.
$ pytest -s -m 'app' --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Tests a specific application
$ pytest -s -m 'app' --app="wordpress" --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Advanced usage of Testinfra tests
Specify host manually:
$ py.test -s --hosts='ssh://root@example.stackspin.net'
Run only tests tagged with prometheus:
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
Run cert test manually using the ansible inventory file:
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
Run cert test manually against a different cluster, not configured in any ansible inventory file, either by using pytest:
$ FQDN='example.stackspin.net' py.test -sv -m 'certs'
or directly:
$ FQDN='example.stackspin.net' pytest/test_certs.py
Running Testinfra tests with local gitlab-runner docker executor
Export the following environment variables like this:
$ export CI_REGISTRY_IMAGE='open.greenhost.net:4567/stackspin/stackspin'
$ export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_stackspin_ci)"
$ export COSMOS_API_TOKEN='…'
then:
$ gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
Known Issues
The default ssh backend for testinfra tests is paramiko,
which does not work out of the box.
It fails to connect to the host because the ed25519 hostkey is not verified.
Therefore we need to force plain ssh:// with either connection=ssh or --hosts=ssh://…
End to end tests
Warning
We are migrating our end to end tests from Taiko to Cypress. You can track the progress in this issue. You need to check below which applications can be tested with which framework. We are sorry for the current inconvenience, we hope we’ll soon finish the migration.
End to end tests run in a browser to test if all the web services are up and running and functioning correctly.
Taiko tests
Used for testing these applications:
Dashboard
Wekan
Wordpress
Zulip
Install Taiko on your provisioning machine:
$ npm install -g taiko
Run Taiko tests
Taiko tests are integrated in the stackspin CLI command suite.
To run all Taiko tests, run the following command in this repository:
$ python -m stackspin CLUSTERNAME test
To learn more about the test subcommand, run:
$ python -m stackspin CLUSTERNAME test --help
You can also only run a Taiko test for a specific application, i.e.:
$ python -m stackspin CLUSTERNAME test --taiko-tags nextcloud
Advanced Taiko tests
If you want to use Taiko without invoking the stackspin CLI, go to the test/taiko directory and run:
For nextcloud & onlyoffice tests:
$ export DOMAIN='stackspin.example.net'
$ export SSO_USERNAME='user1'
$ export SSO_USER_PW='...'
$ export TAIKO_TESTS='nextcloud'
$ taiko --observe taiko-tests.js
You can replace nextcloud with grafana or wordpress to test the other applications, or with all to test all applications.
Cypress tests
Cypress tests are not yet integrated in the stackspin CLI command suite.
Used for testing these applications:
Grafana
Nextcloud
To install Cypress on your provisioning machine go to the test/ directory and run:
$ npm install -g cypress
Test Grafana
$ export CYPRESS_BASE_URL='https://grafana.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed
Test nextcloud
The cypress end to end test is not included in the main Stackspin repository, but in the Nextcloud helm chart repository.
$ git clone https://open.greenhost.net/stackspin/nextcloud.git
$ export CYPRESS_BASE_URL='https://files.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed