Setup Monitoring Stack for Kubernetes Cluster

The monitoring stack is responsible for collecting, storing, and visualizing metrics.
A metric is a numerical value used to measure the performance or state of a system or process.

Prometheus collects and stores metrics as time series data, meaning each metric is stored with a timestamp. This allows us to track changes over time.

Grafana uses these metrics to create visualizations such as graphs and dashboards.

The standard monitoring stack in Kubernetes consists of Prometheus and Grafana.

Step 1: Check Helm Components

First install Helm if you haven't already from here.

helm ls -A : lists all Helm releases installed across all namespaces
helm repo list : list all added Helm repositories

The Traefik helm releases come by default with k3s.
So there are none that we have installed yet.

Step 2: Install Monitoring Stack

We will use the kube-prometheus-stack Helm chart, which simplifies deploying Prometheus and Grafana.

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts

helm repo list

helm repo update

helm install prometheus-stack prometheus-community/kube-prometheus-stack --namespace=monitoring --create-namespace

## list helm releases
helm ls -A

## Check the running pods
kubectl -n monitoring get pods

Let's look at what each of these pods does:

1. prometheus - Collects and stores metrics

2. alertmanager - Handles alerts based on defined rules

3. grafana - Visualizes metrics through dashboards

4. prom-operator - Enables custom resource definitions and controllers which simplify the usage of Prometheus and setting up monitoring

5. kube-state-metrics - Interacts with the kube api-server and generates metrics about the state of the Kubernetes objects

6. node-exporter - Collects node-level metrics

Step 3: Access Grafana

List services in the monitoring namespace:

k get svc -n monitoring

Look for the grafana service (prometheus-stack-grafana).

The default port number for Grafana is 3000.

To access Grafana locally, we need to port-forward:

k port-forward svc/prometheus-stack-grafana 3000:80 -n monitoring

We can access Grafana at http://localhost:3000

Step 4: Retrieve Grafana Password

Now to login, we need the default password and this is how we can get it's value:

Create a directory called monitoring (to store the values in a file) and then run this command:

helm show values prometheus-community/kube-prometheus-stack > prometheus-default-values.yaml

Open the file and search for "grafana" to locate its section, where the adminPassword is defined. Alternatively, you can search for "password" and identify the entry under the grafana section.

Now with these credentials we can login to Grafana.

Alternate way:

## Access Grafana local instance:

export POD_NAME=$(kubectl --namespace monitoring get pod -l "app.kubernetes.io/name=grafana,app.kubernetes.io/instance=prometheus-stack" -oname)
kubectl --namespace monitoring port-forward $POD_NAME 3000

## Get Grafana 'admin' user password by running:

kubectl --namespace monitoring get secrets prometheus-stack-grafana -o jsonpath="{.data.admin-password}" | base64 -d ; echo

## Get your grafana admin user password by running:

kubectl get secret --namespace monitoring -l app.kubernetes.io/component=admin-secret -o jsonpath="{.items[0].data.admin-password}" | base64 --decode ; echo

Step 5: Explore Dashboards

Now if we click on the dashboards, we can see a bunch of default dashboards available.

Let's check the Node Exporter / Nodes dashboard.
This dashboard gives us visual metrics about our cluster nodes like:
- CPU Usage
- Load Average
- Memory Usage
- Disk Space Usage etc.

Step 6: Update Grafana Password

The password that we chose earlier is the default password and we have to change it in order to secure our Grafana application.

Create a values.yaml file:

grafana:
  adminPassword: password1

Upgrade the Helm release:
Now we need to upgrade our helm chart with the new values from the monitoring directory that we created earlier.

helm upgrade prometheus-stack prometheus-community/kube-prometheus-stack -n monitoring --values values.yaml

Now let's log in again using the new password.

Step 7: Expose Grafana App

Port forwading isn't the right way to access our grafana instance. We need an to access it directly via an IP.

Let's create a new service of type Load Balancer.

Create service.yaml inside the monitoring directory:

apiVersion: v1
kind: Service
metadata:
  labels:
    app: grafana
  name: grafana-service
  namespace: monitoring
spec:
  selector:
    app.kubernetes.io/instance: prometheus-stack
    app.kubernetes.io/name: grafana
  ports:
    - protocol: TCP
      port: 3000
      targetPort: 3000
  type: LoadBalancer

Apply this manifest:

k apply -f service.yaml

k get svc -n monitoring

Now we can see that we have the grafana-service of type LoadBalancer.
We can see that it is assigned an External IP.
Now we can access our Grafana app at this IP at port 3000.
Open http://<External_IP>:3000