=======
Testing
=======
To get an overall status of your cluster,
run the tests from the command line.
There are two types of tests:
`testinfra `__ tests verify the setup,
and end-to-end tests that check the webservice availability.
Testinfra tests
===============
Testinfra tests are split into two groups, lets call them *blackbox* and
*clearbox* tests. The blackbox tests run on your provisioning machine and test
the Stackspin cluster from the outside. For example, the certificate check will check
if the Stackspin returns valid certificates for the provided services.
The clearbox tests run on the Stackspin host and check i.e. if docker is installed
in the right version etc. Our testinfra tests are a combination of blackbox and
clearbox tests.
If you have installed the Stackspin python requirements inside a :ref:`virtual environment `
then you need to activate it before testing:
.. code:: console
$ . env/bin/activate
Then enter the *test* directory in the Git repository
**on your provisioning machine**.
.. code:: console
$ cd test
To run the test against your cluster, first export the *CLUSTER_DIR*
environment variable with the location of your cluster config directory
(replace *stackspin.example.org* with your cluster name):
Then export the *KUBECONFIG* variable that lets ``kubectl`` talk to your cluster.
Thirdly, export a variable that contains the password for the Prometheus HTTP endpoint.
.. code:: console
$ export CLUSTER_DIR="${PWD}/../clusters/stackspin.example.org"
$ export KUBECONFIG=${CLUSTER_DIR}/kube_config_cluster.yml
$ export BASIC_AUTH_PW=$(kubectl get secret -n stackspin stackspin-prometheus-basic-auth --template '{{ .data.pass }}' | base64 -d)
$
Run all tests
-------------
.. code:: console
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Test all applications
---------------------
This will check for:
* The applications return proper certificates
* All helm releases are successfully installed
* All app pods are running and healthy (this test includes all optional
applications)
These tests includes all optional applications and will fail for optional
applications that are not installed.
.. code:: console
$ pytest -s -m 'app' --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Tests a specific application
----------------------------
.. code:: console
$ pytest -s -m 'app' --app="wordpress" --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'
Advanced usage of Testinfra tests
---------------------------------
Specify host manually:
.. code:: console
$ py.test -s --hosts='ssh://root@example.stackspin.net'
Run only tests tagged with *prometheus*:
.. code:: console
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus
Run cert test manually using the ansible inventory file:
.. code:: console
$ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs
Run cert test manually against a different cluster, not configured in any
ansible inventory file, either by using pytest:
.. code:: console
$ FQDN='example.stackspin.net' py.test -sv -m 'certs'
or directly:
.. code:: console
$ FQDN='example.stackspin.net' pytest/test_certs.py
Running Testinfra tests with local gitlab-runner docker executor
Export the following environment variables like this:
.. code:: console
$ export CI_REGISTRY_IMAGE='open.greenhost.net:4567/stackspin/stackspin'
$ export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_stackspin_ci)"
$ export COSMOS_API_TOKEN='…'
then:
.. code:: console
$ gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap
Known Issues
------------
The default ssh backend for testinfra tests is *paramiko*,
which does not work out of the box.
It fails to connect to the host because the *ed25519* hostkey is not verified.
Therefore we need to force plain *ssh://* with either ``connection=ssh`` or ``--hosts=ssh://…``
End to end tests
================
End to end tests run in a browser to test if all the web services are up and running
and functioning correctly.
Cypress tests
-------------
Cypress tests are *not yet* integrated in the ``stackspin`` CLI command suite.
Used for testing these applications:
* Grafana
* Hedgedoc
* Nextcloud
* Wekan
* Wordpress
* Zulip
To install `Cypress`_ on your provisioning machine
go to the *test/* directory and run:
.. code:: console
$ npm install -g cypress
Test Grafana
''''''''''''
.. code:: console
$ export CYPRESS_BASE_URL='https://grafana.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed
Test nextcloud
''''''''''''''
The cypress end to end test is not included in the main Stackspin repository,
but in the `Nextcloud helm chart repository `__.
.. code:: console
$ git clone https://open.greenhost.net/stackspin/nextcloud.git
$ export CYPRESS_BASE_URL='https://files.stackspin.example.net'
$ export CYPRESS_USE_SSO_LOGIN="true"
$ export CYPRESS_SSO_USER='admin'
$ export CYPRESS_SSO_PW='...'
$ cypress run --headed
.. _Cypress: https://go.cypress.io