Integrate persistence Prometheus and Grafana on Kubernetes

The combination of Prometheus and Grafana is becoming a more and more common monitoring stack used by DevOps teams for storing and visualizing time series data. Prometheus acts as the storage backend and Grafana as the interface for analysis and visualization.

Prometheus collects metrics from monitored targets by scraping metrics from HTTP endpoints on these targets. Like any server running processes on a host machine, there are specific metrics that need to be monitored such as used memory and storage as well as general ones reporting on the status of the service. Conveniently, Prometheus exposes a wide variety of metrics that can be easily monitored. By adding Grafana as a visualization layer, we can easily set up a monitoring stack for our monitoring stack.

For running kubernetes, we have to setup minikube and kubectl and check whether they are up and running.

For deploying prometheus:

A persistent volume offers persistent storage that enables your data to remain intact, regardless of whether the containers to which the storage is connected or terminated. A persistent volume claim (PVC) is a request for storage, which is met by binding the PVC to a persistent volume (PV). So we will first create persistent storage for all logs.


Prometheus is configured via command-line flags and a configuration file. While the command-line flags configure immutable system parameters (such as storage locations, amount of data to keep on disk and in memory, etc.), the configuration file defines everything related to scraping jobs and their instances, as well as which rule files to load. In this we have used configuration file which contains scrape configuration and its targets.


The deployment file contains details for a ReplicaSet, including a PodTemplate to apply to all the pods in the set. The ReplicaSet data is contained in the first “spec” section of the file. We have passed the file path that is to be used as the configuration for the Prometheus using “args” as it is mentioned in the image arguments.


Now, the service will expose the pods to external network by using NodePort.


At last the Kustomization file

For deploying Grafana

First we will create a pvc to store setting changed.


Now we will create configmap to create data source in Grafana.


Now we will create deployment according to our requirements.


The service file will expose Grafana pods.

Finally the kustomization file

Default username in grafana is ‘admin’ and default password is also ‘admin’.

Here is my GitHub repository for the same

That’s all geeks. Any feedback and suggestions will be appreciated.

Devops Enthusiast