Logging

Logs from pods and containers can be read in different ways:

  • In the cluster filesystem at /var/log/pods/ or /var/logs/containers/.

  • Using kubectl logs

  • Querying aggregated logs with Grafana, see below.

Central log aggregation

We use Promtail, Loki and Grafana for easy access of aggregated logs. The Loki documentation is a good starting point how this setup works. There are two ways of viewing aggregated logs:

  • Via the Grafana web interface

  • Using the logcli command line tool

Viewing logs in Grafana

The Using Loki in Grafana gets you started with querying your cluster logs with Grafana. You will find the Loki Grafana integration on your cluster at https://grafana.stackspin.example.org/explore together with some generic query examples.

Please follow LogQL query examples for more LogQL query examples.

Query logs with logcli

Please refer to logcli for installing logcli on your Laptop. The create a port-forwarding to your cluster using the kubectl tool:

$ kubectl -n stackspin port-forward pod/loki-0 3100

In another terminal you can now use logcli to query loki like this:

$ logcli query '{app=~".+"}'

Please follow LogQL query examples for more LogQL query examples.

Search older messages (in this case the last week and limit the output to 2000 lines):

$ logcli query --since=168h --limit=2000 --forward '{app="helm-controller"}'

LogQL query examples

Please also refer to the LogQL documentation and the log queries documentation .

Query all aggregated logs (unfortunatly we can’t find a better way of doing this since LogQL always expects a stream label to get queried):

{app=~".+"}

Query all logs for a keyword:

{app=~".+"} |= "error"

Query all k8s apps for errors using a regular expression, case-insensitive:

{app=~".+"} |~ `(error|fail|exception|fatal)`

Flux

Flux is responsible for installing applications. It uses four controllers:

  • source-controller that tracks Helm and Git repositories like https://open.greenhost.net/stackspin/stackspin for updates.

  • kustomize-controller to deploy kustomizations that often install helmreleases.

  • helm-controller to deploy the helmreleases.

  • notification-controller that is responsible for inbound and outbound flux messages

Query all messages from the source-controller:

{app="source-controller"}

Query all messages from flux and helm-controller:

{app=~"(source-controller|helm-controller)"}

helm-controller messages containing wordpress:

'{app = "helm-controller"} |= "wordpress"'

helm-controller messages containing wordpress without unchanged events (to only show the installation messages):

'{app = "helm-controller"} |= "wordpress" != "unchanged"'

Filter out redundant helm-controller messages:

'{app="helm-controller"} !~ `(unchanged|event=refreshed|method=Sync|component=checkpoint)`'

Cert-manager

Cert manager is responsible for requesting Let’s Encrypt TLS certificates.

Query cert-manager messages containing chat:

'{app="cert-manager"} |= "chat"'

Hydra

Hydra is the single sign-on system.

Show only warnings and errors from hydra:

{container_name="hydra"} != "level=info"

Debug oauth2 single sign-on with zulip:

{container_name=~"(hydra|zulip)"}

Etc

Query kubernetes events processed by the eventrouter app containing warning:

'{app="eventrouter"} |~ "warning"'