Monitor applications via STACKIT Observability

This section shows the process, how customer workloads on SKE can be monitored using the STACKIT Observability service.

Overview

Here is a high-level architecture diagram of the monitoring setup:

Prerequisites

• You have ordered the following services:
  • STACKIT Kubernetes Engine
  • STACKIT Observability
• You have basic knowledge of Prometheus, Grafana and Kubernetes.
• You have verified your Kubernetes cluster using the kubectl command. kubectl is a command line tool which allows you to run commands against Kubernetes clusters:

  kubectl get nodes
  
  NAME STATUS ROLES AGE VERSION
  shoot--<cluster-name>--<worker-pool>-<node-id> Ready <none> 19d <kubernetes-version>
  shoot--<cluster-name>--<worker-pool>-<node-id> Ready <none> 19d <kubernetes-version>

Steps

1. Deploy the Prometheus operator on your SKE cluster

curl -sL https://github.com/prometheus-operator/prometheus-operator/releases/download/v0.92.1/bundle.yaml | kubectl apply -f -

Pin the version to ensure reproducible behaviour.

Create credentials for the Observability Service via the STACKIT API or the STACKIT Portal. The credentials JSON contains the fields username, password, and urls.pushMetricsUrl.

Create a Kubernetes secret with your Observability credentials:

kubectl create secret generic observabilitysecret \
 --from-literal=username='<your_observability_username>'\
 --from-literal=password='<your_observability_password>'\
 -n default

Next you need to create a Prometheus instance in order to scrape your metrics and push them to Observability.

In the example below we created a PodMonitor in order to scrape the metrics of a pod. You could also create a ServiceMonitor to scrape metrics there.

Here you have to use your pushMetricsUrl which you got with your username and password.

apiVersion: v1
kind: ServiceAccount
metadata:
  name: prometheus
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: prometheus
rules:
- apiGroups: [""]
  resources:
  - nodes
  - nodes/metrics
  - services
  - endpoints
  - pods
  verbs: ["get", "list", "watch"]
- apiGroups: [""]
  resources:
  - configmaps
  verbs: ["get"]
- apiGroups:
  - networking.k8s.io
  resources:
  - ingresses
  verbs: ["get", "list", "watch"]
- nonResourceURLs: ["/metrics"]
  verbs: ["get"]
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: prometheus
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: prometheus
subjects:
- kind: ServiceAccount
  name: prometheus
  namespace: default
---
apiVersion: monitoring.coreos.com/v1
kind: Prometheus
metadata:
  name: prometheus
spec:
  serviceAccountName: prometheus
  podMonitorSelector:
    matchLabels:
      team: frontend
  resources:
    requests:
      memory: 400Mi
  enableAdminAPI: false
  remoteWrite:
  - url: "<your_pushMetrics_URL>"
    basicAuth:
      username:
        name: observabilitysecret
        key: username
      password:
        name: observabilitysecret
        key: password
---
apiVersion: v1
kind: Service
metadata:
  name: prometheus
spec:
  type: ClusterIP
  ports:
  - port: 80
    targetPort: http
    protocol: TCP
    name: http
  selector:
    prometheus: prometheus

2. Create a PodMonitor for your application

In order to monitor your application, you need to create a PodMonitor to scrape the metrics of your pod:

apiVersion: monitoring.coreos.com/v1
kind: PodMonitor
metadata:
  name: product-demo
  labels:
    team: frontend
spec:
  selector:
    matchLabels:
      app: product-demo
  podMetricsEndpoints:
  - port: metrics
  namespaceSelector:
    any: true

3. Visualize your metrics in Grafana

Get your grafanaUrl, grafanaAdminUsername and grafanaAdminPassword from the Observability credentials JSON you created in step 1. Open the grafanaUrl in browser and use your Grafana credentials. Click the Explore symbol:

Check your metrics using a simple query. Query up{job="product-demo"} to verify that the PodMonitor created in step 2 is scraping your application pods:

You can verify your metrics by comparing the IPs from your Kubernetes node details using below command:

kubectl get nodes -o wide

NAME STATUS ROLES AGE VERSION INTERNAL-IP EXTERNAL-IP OS-IMAGE KERNEL-VERSION CONTAINER-RUNTIME
shoot--<cluster-name>--<worker-pool>-<node-id> Ready <none> 4h8m <kubernetes-version> <internal-ip> <none> Flatcar Container Linux by Kinvolk <os-version> (Oklo)
shoot--<cluster-name>--<worker-pool>-<node-id> Ready <none> 20d <kubernetes-version> <internal-ip> <none> Flatcar Container Linux by Kinvolk <os-version> (Oklo)
shoot--<cluster-name>--<worker-pool>-<node-id> Ready <none> 20d <kubernetes-version> <internal-ip> <none> Flatcar Container Linux by Kinvolk <os-version> (Oklo)