=======
Testing
=======

To get an overall status of your cluster,
run the tests from the command line.
There are two types of tests:
`testinfra <https://testinfra.readthedocs.io/>`__ tests verify the setup,
and end-to-end tests that check the webservice availability.

Testinfra tests
===============

Testinfra tests are split into two groups, lets call them *blackbox* and
*clearbox* tests.  The blackbox tests run on your provisioning machine and test
the Stackspin cluster from the outside. For example, the certificate check will check
if the Stackspin returns valid certificates for the provided services.
The clearbox tests run on the Stackspin host and check i.e. if docker is installed
in the right version etc. Our testinfra tests are a combination of blackbox and
clearbox tests.

If you have installed the Stackspin python requirements inside a :ref:`virtual environment <installation/install_cli:Create a python virtual environment (Optional)>`
then you need to activate it before testing:

.. code:: console

   $ . env/bin/activate

Then enter the *test* directory in the Git repository
**on your provisioning machine**.

.. code:: console

   $ cd test

To run the test against your cluster, first export the *CLUSTER_DIR*
environment variable with the location of your cluster config directory
(replace *stackspin.example.org* with your cluster name):
Then export the *KUBECONFIG* variable that lets ``kubectl`` talk to your cluster.
Thirdly, export a variable that contains the password for the Prometheus HTTP endpoint.

.. code:: console

   $ export CLUSTER_DIR="${PWD}/../clusters/stackspin.example.org"
   $ export KUBECONFIG=${CLUSTER_DIR}/kube_config_cluster.yml
   $ export BASIC_AUTH_PW=$(kubectl get secret -n stackspin stackspin-prometheus-basic-auth --template '{{ .data.pass }}' | base64 -d)


   $

Run all tests
-------------

.. code:: console

   $ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'

Test all applications
---------------------

This will check for:

* The applications return proper certificates
* All helm releases are successfully installed
* All app pods are running and healthy (this test includes all optional
  applications)

These tests includes all optional applications and will fail for optional
applications that are not installed.

.. code:: console

   $ pytest -s -m 'app' --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'


Tests a specific application
----------------------------

.. code:: console

   $ pytest -s -m 'app' --app="wordpress" --connection=ansible --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*'

Advanced usage of Testinfra tests
---------------------------------

Specify host manually:

.. code:: console

   $ py.test -s --hosts='ssh://root@example.stackspin.net'

Run only tests tagged with *prometheus*:

.. code:: console

   $ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m prometheus

Run cert test manually using the ansible inventory file:

.. code:: console

   $ py.test -s --ansible-inventory=${CLUSTER_DIR}/inventory.yml --hosts='ansible://*' -m certs

Run cert test manually against a different cluster, not configured in any
ansible inventory file, either by using pytest:

.. code:: console

   $ FQDN='example.stackspin.net' py.test -sv -m 'certs'

or directly:

.. code:: console

   $ FQDN='example.stackspin.net' pytest/test_certs.py

Running Testinfra tests with local gitlab-runner docker executor

Export the following environment variables like this:

.. code:: console

   $ export CI_REGISTRY_IMAGE='open.greenhost.net:4567/stackspin/stackspin'
   $ export SSH_PRIVATE_KEY="$(cat ~/.ssh/id_ed25519_stackspin_ci)"
   $ export COSMOS_API_TOKEN='…'

then:

.. code:: console

   $ gitlab-runner exec docker --env CI_REGISTRY_IMAGE="$CI_REGISTRY_IMAGE" --env SSH_PRIVATE_KEY="$SSH_PRIVATE_KEY" --env COSMOS_API_TOKEN="$COSMOS_API_TOKEN" bootstrap


Known Issues
------------

The default ssh backend for testinfra tests is *paramiko*,
which does not work out of the box.
It fails to connect to the host because the *ed25519* hostkey is not verified.
Therefore we need to force plain *ssh://* with either ``connection=ssh`` or ``--hosts=ssh://…``

End to end tests
================

.. warning::

   We are migrating our end to end tests from `Taiko`_ to `Cypress`_.
   You can track the progress in `this issue <https://open.greenhost.net/stackspin/stackspin/-/issues/1345>`__.
   You need to check below which applications can be tested with which framework.
   We are sorry for the current inconvenience, we hope we'll soon finish the migration.

End to end tests run in a browser to test if all the web services are up and running
and functioning correctly.

Taiko tests
-----------

Used for testing these applications:

* Dashboard
* Wekan
* Wordpress
* Zulip

Install `Taiko`_ on your provisioning machine:

.. code:: console

   $ npm install -g taiko

Run Taiko tests
'''''''''''''''

Taiko tests are integrated in the ``stackspin`` CLI command suite.
To run all Taiko tests, run the following command in this repository:

.. code:: console

   $ python -m stackspin CLUSTERNAME test

To learn more about the ``test`` subcommand, run:

.. code:: console

   $ python -m stackspin CLUSTERNAME test --help

You can also only run a Taiko test for a specific application, i.e.:

.. code:: console

   $ python -m stackspin CLUSTERNAME test --taiko-tags nextcloud

Advanced Taiko tests
''''''''''''''''''''

If you want to use Taiko without invoking the stackspin CLI,
go to the *test/taiko* directory and run:

For nextcloud & onlyoffice tests:

.. code:: console

   $ export DOMAIN='stackspin.example.net'
   $ export SSO_USERNAME='user1'
   $ export SSO_USER_PW='...'
   $ export TAIKO_TESTS='nextcloud'
   $ taiko --observe taiko-tests.js

You can replace *nextcloud* with *grafana* or *wordpress*
to test the other applications,
or with *all* to test all applications.

Cypress tests
-------------

Cypress tests are *not yet* integrated in the ``stackspin`` CLI command suite.
Used for testing these applications:

* Grafana
* Nextcloud

To install `Cypress`_ on your provisioning machine
go to the *test/* directory and run:

.. code:: console

   $  npm install -g cypress

Test Grafana
''''''''''''

.. code:: console

   $ export CYPRESS_BASE_URL='https://grafana.stackspin.example.net'
   $ export CYPRESS_USE_SSO_LOGIN="true"
   $ export CYPRESS_SSO_USER='admin'
   $ export CYPRESS_SSO_PW='...'
   $ cypress run --headed

Test nextcloud
''''''''''''''

The cypress end to end test is not included in the main Stackspin repository,
but in the `Nextcloud helm chart repository <https://open.greenhost.net/stackspin/nextcloud>`__.

.. code:: console

   $ git clone https://open.greenhost.net/stackspin/nextcloud.git
   $ export CYPRESS_BASE_URL='https://files.stackspin.example.net'
   $ export CYPRESS_USE_SSO_LOGIN="true"
   $ export CYPRESS_SSO_USER='admin'
   $ export CYPRESS_SSO_PW='...'
   $ cypress run --headed


.. _Taiko: https://en.wikipedia.org/wiki/Single_sign-on
.. _Cypress: https://go.cypress.io