In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is an option to use logstash as well instead of Fluentd but we are using Fluentd as it has better performance in the case of Kubernetes Logging. First of all, we will require a Kubernetes set up to perform these tasks. I'm using the GKE for my current setup but the same can be replicated on any of the Kubernetes setups. If you want to create a single master setup can follow this link or for multi-master set up the following link.
We will be creating an EFK setup with open-source Xpack enabled to have authentication and authorization enabled on Kibana plus authentication on part of Elasticsearch and Fluentd.
Let's start by creating a separate Namespace for our EKF as kube-logging.
This is an optional command which can be used to provide admin access to your Namespace if in case it's not able to get data from other Namespaces.
We will start by creating Elasticsearch setup first which will include Storage Class so that we can have persistence storage for our Elasticsearch nodes and can have data if in case any pods fail. In my case, as I'm working on GKE so provisioner is based upon that. If you're using a different setup than GKE use the respective provisioner.
As we are using the Xpack for our setup so we need to create a password for the elastic user which is superuser in our case. We will be using Kubernetes Secrets to store this password and it will be called on respective EFK services. We are using the password in base64 as Kubernetes support it in that way only.
Now let's deploy our Elasticsearch as Statefulset to have high availability of our cluster which includes 3 Master's with persistence storage and Xpack enabled.
Now let's create a service for our Eleasticsearch cluster so that it can be accessed via that.
Let's check how our Elasticsearch setup looks like.
Now let's deploy Kibana for our cluster so that we can visualize our data.
First, we need to create a ConnfigMap for our Kibana as it will work as a configuration file for the same. It will have the credentials which will be used to connect Elasticsearch. The path from which we want to access the Kibana in our case we will be example.com/kibana instead of accessing it from the root domain path. The rest of the security parameters are optional according to personal needs.
Let's deploy our Kibana with the following deployment.
Let's create a service for our Kibana deployment.
Let's check how our Kibana setup looks like.
Our Elasticsearch and Kibana are deployed successfully and the last thing which is left is Fluentd to collect logs from all the other pods in our Kubernetes cluster and push them to Elasticsearch.
We will create a ConfigMap for our Fluentd deployment so we can have better control over the parsing of logs and can pass the Elasticsearch credentials to it. You can download the latest copy of this config from the following link.
We are using the following ConfigMap with entries made to connect with our Elasticsearch cluster.
Let's deploy our Fluentd with the following DaemonSet. It also includes ClusterRole which is necessary to provide required permission to our Fluentd to be able to fetch logs from all pods within our Kubernetes Cluster. It includes credentials to connect to the Elasticsearch cluster and other required configuration.
Let's check how our Kibana setup looks like.
Cool, Now our setup is up and running. Let's check our Kibana to see how it turns up. I'm using port-forward just to check the Kibana dashboard.
Create an index pattern to access all the logs from Elasticseach as logstash-* in the management section.
Now let's go to Discover option to see our logs and how they come up.
As we have deployed Kibana with example.com/kibana so following Istio configuration can be used to do this route if you're using Istio and Ingress Gateway. If you want to set up Istio can follow the link.
If you want to test whether logs are coming from new deployments and all namespaces the following deployment can be used.
We will be creating an EFK setup with open-source Xpack enabled to have authentication and authorization enabled on Kibana plus authentication on part of Elasticsearch and Fluentd.
Let's start by creating a separate Namespace for our EKF as kube-logging.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat namespace-kube-logging.yaml
kind: Namespace
apiVersion: v1
metadata:
name: kube-logging
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f namespace-kube-logging.yaml
namespace/kube-logging created
This is an optional command which can be used to provide admin access to your Namespace if in case it's not able to get data from other Namespaces.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts --user=kube-logging
We will start by creating Elasticsearch setup first which will include Storage Class so that we can have persistence storage for our Elasticsearch nodes and can have data if in case any pods fail. In my case, as I'm working on GKE so provisioner is based upon that. If you're using a different setup than GKE use the respective provisioner.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_sc.yaml apiVersion: storage.k8s.io/v1 kind: StorageClass metadata: name: elasticsearch-sc provisioner: kubernetes.io/gce-pd parameters: type: pd-ssd fstype: ext4 replication-type: none
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_sc.yaml storageclass.storage.k8s.io/elasticsearch-sc created
As we are using the Xpack for our setup so we need to create a password for the elastic user which is superuser in our case. We will be using Kubernetes Secrets to store this password and it will be called on respective EFK services. We are using the password in base64 as Kubernetes support it in that way only.
ravi_gadgil@cloudshell:~/logging/efk_bk$ echo -n 'changeme' | base64 Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elastic_user_secret.yaml apiVersion: v1 kind: Secret metadata: name: elasticsearch-pw-elastic namespace: kube-logging type: Opaque data: password: Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elastic_user_secret.yaml secret/elasticsearch-pw-elastic created
Now let's deploy our Elasticsearch as Statefulset to have high availability of our cluster which includes 3 Master's with persistence storage and Xpack enabled.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elastic_user_secret.yaml
apiVersion: v1
kind: Secret
metadata:
name: elasticsearch-pw-elastic
namespace: kube-logging
type: Opaque
data:
password: Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elastic_user_secret.yaml
secret/elasticsearch-pw-elastic created
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
name: es-cluster
namespace: kube-logging
spec:
serviceName: elasticsearch
replicas: 3
selector:
matchLabels:
app: elasticsearch
template:
metadata:
labels:
app: elasticsearch
spec:
containers:
- name: elasticsearch
image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
resources:
limits:
cpu: 1000m
requests:
cpu: 100m
ports:
- containerPort: 9200
name: rest
protocol: TCP
- containerPort: 9300
name: inter-node
protocol: TCP
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
env:
- name: cluster.name
value: k8s-logs
- name: node.name
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: discovery.seed_hosts
value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
- name: cluster.initial_master_nodes
value: "es-cluster-0,es-cluster-1,es-cluster-2"
- name: ES_JAVA_OPTS
value: "-Xms512m -Xmx512m"
- name: xpack.security.enabled
value: "true"
- name: xpack.monitoring.collection.enabled
value: "true"
- name: ELASTIC_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-pw-elastic
key: password
initContainers:
- name: fix-permissions
image: busybox
command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
securityContext:
privileged: true
volumeMounts:
- name: data
mountPath: /usr/share/elasticsearch/data
- name: increase-vm-max-map
image: busybox
command: ["sysctl", "-w", "vm.max_map_count=262144"]
securityContext:
privileged: true
- name: increase-fd-ulimit
image: busybox
command: ["sh", "-c", "ulimit -n 65536"]
securityContext:
privileged: true
volumeClaimTemplates:
- metadata:
name: data
labels:
app: elasticsearch
spec:
accessModes: [ "ReadWriteOnce" ]
storageClassName: elasticsearch-sc
resources:
requests:
storage: 50Gi
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_statefulset.yaml statefulset.apps/es-cluster created
Now let's create a service for our Eleasticsearch cluster so that it can be accessed via that.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_svc.yaml
kind: Service
apiVersion: v1
metadata:
name: elasticsearch
namespace: kube-logging
labels:
app: elasticsearch
spec:
selector:
app: elasticsearch
clusterIP: None
ports:
- port: 9200
name: rest
- port: 9300
name: inter-node
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_svc.yaml service/elasticsearch created
Let's check how our Elasticsearch setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging NAME READY STATUS RESTARTS AGE pod/es-cluster-0 1/1 Running 0 20m pod/es-cluster-1 1/1 Running 0 19m pod/es-cluster-2 1/1 Running 0 19m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 36s NAME READY AGE statefulset.apps/es-cluster 3/3 20m
Now let's deploy Kibana for our cluster so that we can visualize our data.
First, we need to create a ConnfigMap for our Kibana as it will work as a configuration file for the same. It will have the credentials which will be used to connect Elasticsearch. The path from which we want to access the Kibana in our case we will be example.com/kibana instead of accessing it from the root domain path. The rest of the security parameters are optional according to personal needs.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana_configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
namespace: kube-logging
name: kibana-config
labels:
app: kibana
data:
kibana.yml: |-
server.host: 0.0.0.0
elasticsearch:
hosts: ${ELASTICSEARCH_HOSTS}
username: ${ELASTICSEARCH_USER}
password: ${ELASTICSEARCH_PASSWORD}
server.basePath: ${SERVER_BASEPATH}
server.rewriteBasePath: ${SERVER_REWRITEBASEPATH}
xpack.security.enabled: ${XPACK_SECURITY_ENABLED}
xpack.security.sessionTimeout: 3600000
xpack.security.encryptionKey: "REDACTEDbvhjdbvwkbvwdjnbkjvbsdvjskvjiwufwifuhiuwefbwkjcsdjckbdskj"
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana_configmap.yaml configmap/kibana-config created
Let's deploy our Kibana with the following deployment.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
namespace: kube-logging
name: kibana
labels:
app: kibana
spec:
replicas: 1
selector:
matchLabels:
app: kibana
template:
metadata:
labels:
app: kibana
spec:
containers:
- name: kibana
image: docker.elastic.co/kibana/kibana:7.6.2
ports:
- containerPort: 5601
name: webinterface
env:
- name: ELASTICSEARCH_HOSTS
value: "http://elasticsearch.kube-logging.svc.cluster.local:9200"
- name: SERVER_BASEPATH
value: "/kibana"
- name: SERVER_REWRITEBASEPATH
value: "true"
- name: XPACK_SECURITY_ENABLED
value: "true"
- name: ELASTICSEARCH_USER
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-pw-elastic
key: password
volumeMounts:
- name: config
mountPath: /usr/share/kibana/config/kibana.yml
readOnly: true
subPath: kibana.yml
volumes:
- name: config
configMap:
name: kibana-config
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana.yaml deployment.apps/kibana created
Let's create a service for our Kibana deployment.
dravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana_service.yaml
apiVersion: v1
kind: Service
metadata:
namespace: kube-logging
name: kibana
labels:
app: kibana
spec:
type: NodePort
ports:
- port: 5601
name: webinterface
targetPort: 5601
selector:
app: kibana
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana_service.yaml service/kibana created
Let's check how our Kibana setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging NAME READY STATUS RESTARTS AGE pod/es-cluster-0 1/1 Running 0 66m pod/es-cluster-1 1/1 Running 0 66m pod/es-cluster-2 1/1 Running 0 66m pod/kibana-5c4b9749f7-mmpgg 1/1 Running 0 4m39s NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 47m service/kibana NodePort 10.44.13.43 <none> 5601:32023/TCP 3m16s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana 1/1 1 1 4m39s NAME DESIRED CURRENT READY AGE replicaset.apps/kibana-5c4b9749f7 1 1 1 4m39s NAME READY AGE statefulset.apps/es-cluster 3/3 66m
Our Elasticsearch and Kibana are deployed successfully and the last thing which is left is Fluentd to collect logs from all the other pods in our Kubernetes cluster and push them to Elasticsearch.
We will create a ConfigMap for our Fluentd deployment so we can have better control over the parsing of logs and can pass the Elasticsearch credentials to it. You can download the latest copy of this config from the following link.
We are using the following ConfigMap with entries made to connect with our Elasticsearch cluster.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat fluentd-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
name: fluentd-config-v0.2.0
namespace: kube-logging
labels:
addonmanager.kubernetes.io/mode: Reconcile
data:
system.conf: |-
<system>
root_dir /tmp/fluentd-buffers/
</system>
containers.input.conf: |-
# This configuration file for Fluentd / td-agent is used
# to watch changes to Docker log files. The kubelet creates symlinks that
# capture the pod name, namespace, container name & Docker container ID
# to the docker logs for pods in the /var/log/containers directory on the host.
# If running this fluentd configuration in a Docker container, the /var/log
# directory should be mounted in the container.
#
# These logs are then submitted to Elasticsearch which assumes the
# installation of the fluent-plugin-elasticsearch & the
# fluent-plugin-kubernetes_metadata_filter plugins.
# See https://github.com/uken/fluent-plugin-elasticsearch &
# https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for
# more information about the plugins.
#
# Example
# =======
# A line in the Docker log file might look like this JSON:
#
# {"log":"2014/09/25 21:15:03 Got request with path wombat\n",
# "stream":"stderr",
# "time":"2014-09-25T21:15:03.499185026Z"}
#
# The time_format specification below makes sure we properly
# parse the time format produced by Docker. This will be
# submitted to Elasticsearch and should appear like:
# $ curl 'http://elasticsearch-logging:9200/_search?pretty'
# ...
# {
# "_index" : "logstash-2014.09.25",
# "_type" : "fluentd",
# "_id" : "VBrbor2QTuGpsQyTCdfzqA",
# "_score" : 1.0,
# "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n",
# "stream":"stderr","tag":"docker.container.all",
# "@timestamp":"2014-09-25T22:45:50+00:00"}
# },
# ...
#
# The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log
# record & add labels to the log record if properly configured. This enables users
# to filter & search logs on any metadata.
# For example a Docker container's logs might be in the directory:
#
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
#
# and in the file:
#
# 997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
#
# where 997599971ee6... is the Docker ID of the running container.
# The Kubernetes kubelet makes a symbolic link to this file on the host machine
# in the /var/log/containers directory which includes the pod name and the Kubernetes
# container name:
#
# synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
# ->
# /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
#
# The /var/log directory on the host is mapped to the /var/log directory in the container
# running this instance of Fluentd and we end up collecting the file:
#
# /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
#
# This results in the tag:
#
# var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
#
# The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name
# which are added to the log message as a kubernetes field object & the Docker container ID
# is also added under the docker field object.
# The final tag is:
#
# kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
#
# And the final log record look like:
#
# {
# "log":"2014/09/25 21:15:03 Got request with path wombat\n",
# "stream":"stderr",
# "time":"2014-09-25T21:15:03.499185026Z",
# "kubernetes": {
# "namespace": "default",
# "pod_name": "synthetic-logger-0.25lps-pod",
# "container_name": "synth-lgr"
# },
# "docker": {
# "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"
# }
# }
#
# This makes it easier for users to search for logs by pod name or by
# the name of the Kubernetes container regardless of how many times the
# Kubernetes pod has been restarted (resulting in a several Docker container IDs).
# Json Log Example:
# {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
# CRI Log Example:
# 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here
<source>
@id fluentd-containers.log
@type tail
path /var/log/containers/*.log
pos_file /var/log/es-containers.log.pos
tag raw.kubernetes.*
read_from_head true
<parse>
@type multi_format
<pattern>
format json
time_key time
time_format %Y-%m-%dT%H:%M:%S.%NZ
</pattern>
<pattern>
format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
time_format %Y-%m-%dT%H:%M:%S.%N%:z
</pattern>
</parse>
</source>
# Detect exceptions in the log output and forward them as one log entry.
<match raw.kubernetes.**>
@id raw.kubernetes
@type detect_exceptions
remove_tag_prefix raw
message log
stream stream
multiline_flush_interval 5
max_bytes 500000
max_lines 1000
</match>
# Concatenate multi-line logs
<filter **>
@id filter_concat
@type concat
key message
multiline_end_regexp /\n$/
separator ""
</filter>
# Enriches records with Kubernetes metadata
<filter kubernetes.**>
@id filter_kubernetes_metadata
@type kubernetes_metadata
</filter>
# Fixes json fields in Elasticsearch
<filter kubernetes.**>
@id filter_parser
@type parser
key_name log
reserve_data true
remove_key_name_field true
<parse>
@type multi_format
<pattern>
format json
</pattern>
<pattern>
format none
</pattern>
</parse>
</filter>
system.input.conf: |-
# Example:
# 2015-12-21 23:17:22,066 [salt.state ][INFO ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
<source>
@id minion
@type tail
format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
time_format %Y-%m-%d %H:%M:%S
path /var/log/salt/minion
pos_file /var/log/salt.pos
tag salt
</source>
# Example:
# Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
<source>
@id startupscript.log
@type tail
format syslog
path /var/log/startupscript.log
pos_file /var/log/es-startupscript.log.pos
tag startupscript
</source>
# Examples:
# time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
# time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
# TODO(random-liu): Remove this after cri container runtime rolls out.
<source>
@id docker.log
@type tail
format /^time="(?<time>[^"]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
path /var/log/docker.log
pos_file /var/log/es-docker.log.pos
tag docker
</source>
# Example:
# 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
<source>
@id etcd.log
@type tail
# Not parsing this, because it doesn't have anything particularly useful to
# parse out of it (like severities).
format none
path /var/log/etcd.log
pos_file /var/log/es-etcd.log.pos
tag etcd
</source>
# Multi-line parsing is required for all the kube logs because very large log
# statements, such as those that include entire object bodies, get split into
# multiple lines by glog.
# Example:
# I0204 07:32:30.020537 3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
<source>
@id kubelet.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kubelet.log
pos_file /var/log/es-kubelet.log.pos
tag kubelet
</source>
# Example:
# I1118 21:26:53.975789 6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
<source>
@id kube-proxy.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-proxy.log
pos_file /var/log/es-kube-proxy.log.pos
tag kube-proxy
</source>
# Example:
# I0204 07:00:19.604280 5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
<source>
@id kube-apiserver.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-apiserver.log
pos_file /var/log/es-kube-apiserver.log.pos
tag kube-apiserver
</source>
# Example:
# I0204 06:55:31.872680 5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
<source>
@id kube-controller-manager.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-controller-manager.log
pos_file /var/log/es-kube-controller-manager.log.pos
tag kube-controller-manager
</source>
# Example:
# W0204 06:49:18.239674 7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
<source>
@id kube-scheduler.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/kube-scheduler.log
pos_file /var/log/es-kube-scheduler.log.pos
tag kube-scheduler
</source>
# Example:
# I0603 15:31:05.793605 6 cluster_manager.go:230] Reading config from path /etc/gce.conf
<source>
@id glbc.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/glbc.log
pos_file /var/log/es-glbc.log.pos
tag glbc
</source>
# Example:
# I0603 15:31:05.793605 6 cluster_manager.go:230] Reading config from path /etc/gce.conf
<source>
@id cluster-autoscaler.log
@type tail
format multiline
multiline_flush_interval 5s
format_firstline /^\w\d{4}/
format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
time_format %m%d %H:%M:%S.%N
path /var/log/cluster-autoscaler.log
pos_file /var/log/es-cluster-autoscaler.log.pos
tag cluster-autoscaler
</source>
# Logs from systemd-journal for interesting services.
# TODO(random-liu): Remove this after cri container runtime rolls out.
<source>
@id journald-docker
@type systemd
matches [{ "_SYSTEMD_UNIT": "docker.service" }]
<storage>
@type local
persistent true
path /var/log/journald-docker.pos
</storage>
read_from_head true
tag docker
</source>
<source>
@id journald-container-runtime
@type systemd
matches [{ "_SYSTEMD_UNIT": "{{ fluentd_container_runtime_service }}.service" }]
<storage>
@type local
persistent true
path /var/log/journald-container-runtime.pos
</storage>
read_from_head true
tag container-runtime
</source>
<source>
@id journald-kubelet
@type systemd
matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
<storage>
@type local
persistent true
path /var/log/journald-kubelet.pos
</storage>
read_from_head true
tag kubelet
</source>
<source>
@id journald-node-problem-detector
@type systemd
matches [{ "_SYSTEMD_UNIT": "node-problem-detector.service" }]
<storage>
@type local
persistent true
path /var/log/journald-node-problem-detector.pos
</storage>
read_from_head true
tag node-problem-detector
</source>
<source>
@id kernel
@type systemd
matches [{ "_TRANSPORT": "kernel" }]
<storage>
@type local
persistent true
path /var/log/kernel.pos
</storage>
<entry>
fields_strip_underscores true
fields_lowercase true
</entry>
read_from_head true
tag kernel
</source>
forward.input.conf: |-
# Takes the messages sent over TCP
<source>
@id forward
@type forward
</source>
monitoring.conf: |-
# Prometheus Exporter Plugin
# input plugin that exports metrics
<source>
@id prometheus
@type prometheus
</source>
<source>
@id monitor_agent
@type monitor_agent
</source>
# input plugin that collects metrics from MonitorAgent
<source>
@id prometheus_monitor
@type prometheus_monitor
<labels>
host ${hostname}
</labels>
</source>
# input plugin that collects metrics for output plugin
<source>
@id prometheus_output_monitor
@type prometheus_output_monitor
<labels>
host ${hostname}
</labels>
</source>
# input plugin that collects metrics for in_tail plugin
<source>
@id prometheus_tail_monitor
@type prometheus_tail_monitor
<labels>
host ${hostname}
</labels>
</source>
output.conf: |-
<match **>
@id elasticsearch
@type elasticsearch
@log_level info
type_name _doc
include_tag_key true
host elasticsearch.kube-logging.svc.cluster.local
port 9200
user "#{ENV['ELASTICSEARCH_USER']}"
password "#{ENV['ELASTICSEARCH_PASSWORD']}"
logstash_format true
<buffer>
@type file
path /var/log/fluentd-buffers/kubernetes.system.buffer
flush_mode interval
retry_type exponential_backoff
flush_thread_count 2
flush_interval 5s
retry_forever
retry_max_interval 30
chunk_limit_size 2M
total_limit_size 500M
overflow_action block
</buffer>
</match>
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f fluentd-config.yaml configmap/fluentd-config-v0.2.0 created
Let's deploy our Fluentd with the following DaemonSet. It also includes ClusterRole which is necessary to provide required permission to our Fluentd to be able to fetch logs from all pods within our Kubernetes Cluster. It includes credentials to connect to the Elasticsearch cluster and other required configuration.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat fluentd.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
name: fluentd
namespace: kube-logging
labels:
app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: fluentd
labels:
app: fluentd
rules:
- apiGroups:
- ""
resources:
- pods
- namespaces
verbs:
- get
- list
- watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
name: fluentd
roleRef:
kind: ClusterRole
name: fluentd
apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
name: fluentd
namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: fluentd-v3.0.1
namespace: kube-logging
labels:
k8s-app: fluentd
version: v3.0.1
addonmanager.kubernetes.io/mode: Reconcile
spec:
selector:
matchLabels:
k8s-app: fluentd
version: v3.0.1
template:
metadata:
labels:
k8s-app: fluentd
version: v3.0.1
# This annotation ensures that fluentd does not get evicted if the node
# supports critical pod annotation based priority scheme.
# Note that this does not guarantee admission on the nodes (#40573).
annotations:
seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
spec:
serviceAccount: fluentd
serviceAccountName: fluentd
containers:
- name: fluentd
image: quay.io/fluentd_elasticsearch/fluentd:v3.0.1
env:
- name: FLUENT_ELASTICSEARCH_HOST
value: "elasticsearch.kube-logging.svc.cluster.local"
- name: FLUENT_ELASTICSEARCH_PORT
value: "9200"
- name: FLUENT_ELASTICSEARCH_SCHEME
value: "http"
- name: FLUENT_UID
value: "0"
- name: FLUENTD_ARGS
value: --no-supervisor -q
- name: ELASTICSEARCH_USER
value: "elastic"
- name: ELASTICSEARCH_PASSWORD
valueFrom:
secretKeyRef:
name: elasticsearch-pw-elastic
key: password
resources:
limits:
memory: 500Mi
requests:
cpu: 100m
memory: 200Mi
volumeMounts:
- name: varlog
mountPath: /var/log
- name: varlibdockercontainers
mountPath: /var/lib/docker/containers
readOnly: true
- name: config-volume
mountPath: /etc/fluent/config.d
terminationGracePeriodSeconds: 30
volumes:
- name: varlog
hostPath:
path: /var/log
- name: varlibdockercontainers
hostPath:
path: /var/lib/docker/containers
- name: config-volume
configMap:
name: fluentd-config-v0.2.0
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f fluentd.yaml serviceaccount/fluentd created clusterrole.rbac.authorization.k8s.io/fluentd created clusterrolebinding.rbac.authorization.k8s.io/fluentd created daemonset.apps/fluentd-v3.0.1 created
Let's check how our Kibana setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging NAME READY STATUS RESTARTS AGE pod/es-cluster-0 1/1 Running 0 117m pod/es-cluster-1 1/1 Running 0 117m pod/es-cluster-2 1/1 Running 0 117m pod/fluentd-v3.0.1-697gv 1/1 Running 0 2m15s pod/fluentd-v3.0.1-c5x76 1/1 Running 0 2m15s pod/fluentd-v3.0.1-js6xj 1/1 Running 0 2m16s pod/fluentd-v3.0.1-kfrpf 1/1 Running 0 2m16s pod/fluentd-v3.0.1-wtjgj 1/1 Running 0 2m15s pod/fluentd-v3.0.1-x5ws9 1/1 Running 0 2m16s pod/kibana-5c4b9749f7-mmpgg 1/1 Running 0 55m NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE service/elasticsearch ClusterIP None <none> 9200/TCP,9300/TCP 98m service/kibana NodePort 10.44.13.43 <none> 5601:32023/TCP 54m NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE daemonset.apps/fluentd-v3.0.1 6 6 6 6 6 <none> 2m16s NAME READY UP-TO-DATE AVAILABLE AGE deployment.apps/kibana 1/1 1 1 55m NAME DESIRED CURRENT READY AGE replicaset.apps/kibana-5c4b9749f7 1 1 1 55m NAME READY AGE statefulset.apps/es-cluster 3/3 117m
Cool, Now our setup is up and running. Let's check our Kibana to see how it turns up. I'm using port-forward just to check the Kibana dashboard.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl port-forward --namespace kube-logging service/kibana 5601:5601 & [1] 5494 ravi_gadgil@cloudshell:~/logging/efk_bk$ Forwarding from 127.0.0.1:5601 -> 5601
As we have defined our Kibana to open on /kibana URL so use that to access.
 |
| Kibana Dashboard |
Use credentials elastic/changeme to login to your Kibana dashboard if you have not changed the same in the Secret file.
 |
| Kibana Dashboard |
 |
| Kibana Dashboard |
 |
| Kibana Dashboard |
 |
| Kibana Dashboard |
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat istio-virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
name: default-gateway
namespace: istio-system
spec:
selector:
istio: ingressgateway
servers:
- hosts:
- '*'
port:
name: http
number: 80
protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
name: kibana
namespace: kube-logging
spec:
hosts:
- "example.com"
gateways:
- default-gateway.istio-system.svc.cluster.local
http:
- match:
- uri:
prefix: /kibana
route:
- destination:
host: kibana.kube-logging.svc.cluster.local
port:
number: 5601
If you want to test whether logs are coming from new deployments and all namespaces the following deployment can be used.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat counter.yaml
apiVersion: v1
kind: Pod
metadata:
name: counter
spec:
containers:
- name: count
image: busybox
args: [/bin/sh, -c,
'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']
