Ship selected cluster logs to Loki

This how-to guides you to ship a (number of) specified logs to the central Loki for the purpose of monitoring.

Prerequisites

  • Write access to the commodore-defaults repository

  • Knowledge of which specific logs are required for your needs

If you are unsure which specific logs you need, you can query the cluster-local Loki. This shows you the available logs, and you can formulate queries that answer the specific questions you have.

Procedure

  1. Identify whether any of your logs can be pre-filtered.

    • To keep the number of logs as low as possible, you should only ship the logs that you need.

    • Pre-filtering is achieved via filtering by content and/or filtering by metadata.

    • logs are sent to Loki using OpenShifts LogForwarder.

  2. Add the filtering and forwarding configuration to the appropriate class in the Commodore defaults.

    For example, the configuration for VSHN Managed OpenShift is managed in distribution/openshift4/central-logging.yaml.

  3. Setup a datasource in Grafana. The datasource setup only needs to be done once for each Loki organization.

    Before doing this, check if there’s already a configured datasource for your Loki organization.

    The datasource configuration needs at least the following settings:

    • URL: http://vshn-appuio-loki-gateway.vshn-appuio-loki.svc.cluster.local

    • Header: X-Scope-OrgID: <name of the Loki organization where you ship the logs>

    • HTTP Method: POST

    You can setup the datasource in the Grafana configuration (vshn_appuio_grafana.helm_values.datasources) in the APPUiO Syn tenant repo.

  4. Once your change is rolled out, logs should start arriving in Loki. You can then query the logs in Grafana in the datasource for your Loki organization.