Introduction:
In the ever-evolving realm of Kubernetes, robust logging is crucial for maintaining visibility, troubleshooting issues, and ensuring seamless application performance. Helm, the Kubernetes package manager, streamlines the setup of logging, making it both efficient and scalable. This guide will walk you through the process of configuring logging in a Kubernetes production environment using Helm, complete with comprehensive documentation and practical code snippets.
Prerequisites:
Before we begin, make sure you have the following prerequisites:
A running Kubernetes cluster (local or cloud-based).
The
kubectlcommand-line tool installed and configured to communicate with your cluster.Helm installed on your cluster
A basic understanding of Kubernetes concepts such as Pods, Deployments, and Services.
A K8s Cluster running locally :
kato@master1:~$ kubectl get nodes
NAME STATUS ROLES AGE VERSION
master1 Ready control-plane,master 2d10h v1.23.10
node1 Ready worker 2d10h v1.23.10
node2 Ready worker 2d10h v1.23.10
Source Code for the Project
https://github.com/Gatete-Bruno/Logging
This repository contains Kubernetes YAMLs and Helm values files for setting up the ELK stack (Elasticsearch, Filebeat, Logstash, and Kibana) in a Kubernetes cluster.
Prerequisites
A running Kubernetes cluster
Helm installed on your local machine
Explanation of Manifests
Elasticsearch
elasticsearch/pv.yaml
This file defines three PersistentVolumes for Elasticsearch data storage:
apiVersion: v1
kind: PersistentVolume
metadata:
name: elk-pv-0
labels:
type: local
spec:
storageClassName: standard
capacity:
storage: 1Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/elk0"
apiVersion, kind: Specify the resource type.
metadata: Contains the name and labels for the PV.
spec: Defines storage class, capacity, access modes, and host path.
elasticsearch/values.yaml
This file contains custom Helm values for Elasticsearch:
# Permit co-located instances for solitary minikube virtual machines.
antiAffinity: "soft"
# Shrink default JVM heap.
esJavaOpts: "-Xmx1024m -Xms1024m"
# Allocate smaller chunks of memory per pod.
resources:
requests:
cpu: "100m"
memory: "512M"
limits:
cpu: "1000m"
memory: "4024M"
# Request smaller persistent volumes.
volumeClaimTemplate:
accessModes: [ "ReadWriteOnce" ]
storageClassName: "standard"
resources:
requests:
storage: 1Gi
antiAffinity: Allows soft anti-affinity to permit co-located instances.
esJavaOpts: JVM options for Elasticsearch.
resources: Specifies resource requests and limits.
volumeClaimTemplate: Defines storage class and size for the persistent volumes.
Filebeat
filebeat/filebeat.yaml
This file configures Filebeat deployment and daemonset:
daemonset:
enabled: true
filebeatConfig:
filebeat.yml: |
filebeat.inputs:
- type: container
paths:
- /var/log/containers/*.log
processors:
- add_kubernetes_metadata:
host: ${NODE_NAME}
matchers:
- logs_path:
logs_path: "/var/log/containers/"
output.elasticsearch:
host: '${NODE_NAME}'
hosts: '${ELASTICSEARCH_HOSTS:elasticsearch-master:9200}'
output.logstash:
hosts: ['logstash:9600']
daemonset: Enables Filebeat DaemonSet to collect logs from all nodes.
filebeatConfig: Configures Filebeat to read container logs and send them to Elasticsearch and Logstash.
filebeat/pv.yaml
Defines a PersistentVolume for Filebeat:
apiVersion: v1
kind: PersistentVolume
metadata:
name: filebeat-pv
labels:
type: local
spec:
storageClassName: standard
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/filebeat"
apacity: 3Gi of storage.
hostPath: Path on the host where the volume is stored.
Kibana
kibana/kibana-service.yaml
Configures a NodePort service for Kibana:
apiVersion: v1
kind: Service
metadata:
name: kibana-kibana
spec:
type: NodePort
ports:
- port: 5601
targetPort: 5601
nodePort: 30001 # Choose a port between 30000-32767
selector:
app: kibana
type: NodePort service to expose Kibana.
ports: Maps port 5601 for Kibana access.
kibana/values.yaml
Configures the Elasticsearch host for Kibana:
elasticsearchHosts: "http://elasticsearch-master:9200"
Logstash
logstash/logstash-http-svc.yaml
Defines a ClusterIP service for Logstash HTTP input:
kind: Service
apiVersion: v1
metadata:
name: logstash-http
spec:
selector:
app: logstash
ports:
- protocol: "TCP"
port: 9601
type: ClusterIP
selector: Selects pods with the label
app: logstash.ports: Defines port 9601 for Logstash HTTP input.
logstash/logstash-ingress.yaml
Configures an Ingress for Logstash:
apiVersion: networking.k8s.io/v1
kind: Ingress
metadata:
annotations:
kubernetes.io/ingress.class: nginx
nginx.ingress.kubernetes.io/configuration-snippet: |
more_set_headers "X-Frame-Options: Deny";
more_set_headers "X-Xss-Protection: 1; mode=block";
more_set_headers "X-Content-Type-Options: nosniff";
name: logstash-ingress
spec:
rules:
- host: logstash.democluster.com
http:
paths:
- backend:
service:
name: logstash-http
port:
number: 9601
path: /
pathType: ImplementationSpecific
annotations: Custom headers for security.
rules: Defines ingress rules for Logstash.
logstash/pv.yaml
Defines a PersistentVolume for Logstash:
apiVersion: v1
kind: PersistentVolume
metadata:
name: logstash-pv
labels:
type: local
spec:
storageClassName: standard
capacity:
storage: 3Gi
accessModes:
- ReadWriteOnce
hostPath:
path: "/data/logstash"
capacity: 3Gi of storage.
hostPath: Path on the host where the volume is stored.
logstash/pvc.yaml
Defines a PersistentVolumeClaim for Logstash:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: logstash-pvc
spec:
storageClassName: standard
volumeName: logstash-pv
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 1Gi
volumeName: Binds to the
logstash-pvvolume.requests: 1Gi of storage.
Installation Steps
Clone the Repository
First, clone the repository that contains the necessary YAML files and Helm values:
git clone https://github.com/Gatete-Bruno/Logging
Directory Structure
The repository is organized as follows:
├── README.md
├── elasticsearch
│ ├── pv.yaml
│ └── values.yaml
├── filebeat
│ ├── filebeat.yaml
│ ├── pv.yaml
│ └── values.yaml
├── kibana
│ ├── kibana-ingress.yaml
│ └── values.yaml
└── logstach
├── logstash-http-svc.yaml
├── logstash-ingress.yaml
├── pv.yaml
└── pvc.yaml
- Add the Helm Repo
helm repo add elastic https://helm.elastic.co
helm repo update
- Elasticsearch Setup Create and Apply Persistent Volumes
kubectl apply -f elasticsearch/pv.yaml
Install Elasticsearch
cd elasticsearch
helm install elasticsearch elastic/elasticsearch --version="7.17.3" -f values.yaml
cd ..
- Filebeat Setup Create and Apply Persistent Volumes
kubectl apply -f filebeat/pv.yaml
cd ..
Install Filebeat
cd filebeat
helm install filebeat elastic/filebeat --version="7.17.3" -f values.yaml
cd ..
- Kibana Setup Install Kibana
cd kibana
helm install kibana elastic/kibana --version="7.17.3" -f values.yaml
cd ..
Update Ingress Hostname Modify the ingress.yaml file to match your hostname. Apply Ingress
kubectl apply -f kibana/
cd ..
- Logstash Setup Apply Logstash YAMLs
cd logstash
kubectl apply -f .
cd ..
- Verify Installation Check the status of the pods to ensure they are all running:
kubectl get pods
If all pods are in the Running status, go to your Kibana host address to access the ELK stack.
All Pods running (for this demo we setup all Kubernetes Objects on default namespace)
In this case i opted to use a service instead of an ingress ,since we just testing out kibana therefore we can access it via an ip addrress:
In my case:
[192.168.1.60:30001/app/discover#/?_g=(filte..
All you need to do is now have data to Ingest and be able to access Logs using logstash
