Thursday, May 7, 2020

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is an option to use logstash as well instead of Fluentd but we are using Fluentd as it has better performance in the case of Kubernetes Logging. First of all, we will require a Kubernetes set up to perform these tasks. I'm using the GKE for my current setup but the same can be replicated on any of the Kubernetes setups. If you want to create a single master setup can follow this link or for multi-master set up the following link.

We will be creating an EFK setup with open-source Xpack enabled to have authentication and authorization enabled on Kibana plus authentication on part of Elasticsearch and Fluentd.

Let's start by creating a separate Namespace for our EKF as kube-logging.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat namespace-kube-logging.yaml
kind: Namespace
apiVersion: v1
metadata:
  name: kube-logging
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f namespace-kube-logging.yaml
namespace/kube-logging created

This is an optional command which can be used to provide admin access to your Namespace if in case it's not able to get data from other Namespaces.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl create clusterrolebinding permissive-binding --clusterrole=cluster-admin --user=admin --user=kubelet --group=system:serviceaccounts --user=kube-logging

We will start by creating Elasticsearch setup first which will include Storage Class so that we can have persistence storage for our Elasticsearch nodes and can have data if in case any pods fail. In my case, as I'm working on GKE so provisioner is based upon that. If you're using a different setup than GKE use the respective provisioner.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_sc.yaml
apiVersion: storage.k8s.io/v1
kind: StorageClass
metadata:
  name: elasticsearch-sc
provisioner: kubernetes.io/gce-pd
parameters:
  type: pd-ssd
  fstype: ext4
  replication-type: none
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_sc.yaml
storageclass.storage.k8s.io/elasticsearch-sc created

As we are using the Xpack for our setup so we need to create a password for the elastic user which is superuser in our case. We will be using Kubernetes Secrets to store this password and it will be called on respective EFK services. We are using the password in base64 as Kubernetes support it in that way only.
ravi_gadgil@cloudshell:~/logging/efk_bk$ echo -n 'changeme' | base64
Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elastic_user_secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: elasticsearch-pw-elastic
  namespace: kube-logging
type: Opaque
data:
  password: Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elastic_user_secret.yaml
secret/elasticsearch-pw-elastic created

Now let's deploy our Elasticsearch as Statefulset to have high availability of our cluster which includes 3 Master's with persistence storage and Xpack enabled.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elastic_user_secret.yaml
apiVersion: v1
kind: Secret
metadata:
  name: elasticsearch-pw-elastic
  namespace: kube-logging
type: Opaque
data:
  password: Y2hhbmdlbWU=
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elastic_user_secret.yaml
secret/elasticsearch-pw-elastic created
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_statefulset.yaml
apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: es-cluster
  namespace: kube-logging
spec:
  serviceName: elasticsearch
  replicas: 3
  selector:
    matchLabels:
      app: elasticsearch
  template:
    metadata:
      labels:
        app: elasticsearch
    spec:
      containers:
      - name: elasticsearch
        image: docker.elastic.co/elasticsearch/elasticsearch:7.6.2
        resources:
            limits:
              cpu: 1000m
            requests:
              cpu: 100m
        ports:
        - containerPort: 9200
          name: rest
          protocol: TCP
        - containerPort: 9300
          name: inter-node
          protocol: TCP
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
        env:
          - name: cluster.name
            value: k8s-logs
          - name: node.name
            valueFrom:
              fieldRef:
                fieldPath: metadata.name
          - name: discovery.seed_hosts
            value: "es-cluster-0.elasticsearch,es-cluster-1.elasticsearch,es-cluster-2.elasticsearch"
          - name: cluster.initial_master_nodes
            value: "es-cluster-0,es-cluster-1,es-cluster-2"
          - name: ES_JAVA_OPTS
            value: "-Xms512m -Xmx512m"
          - name: xpack.security.enabled
            value: "true"
          - name: xpack.monitoring.collection.enabled
            value: "true"
          - name: ELASTIC_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elasticsearch-pw-elastic
                key: password
      initContainers:
      - name: fix-permissions
        image: busybox
        command: ["sh", "-c", "chown -R 1000:1000 /usr/share/elasticsearch/data"]
        securityContext:
          privileged: true
        volumeMounts:
        - name: data
          mountPath: /usr/share/elasticsearch/data
      - name: increase-vm-max-map
        image: busybox
        command: ["sysctl", "-w", "vm.max_map_count=262144"]
        securityContext:
          privileged: true
      - name: increase-fd-ulimit
        image: busybox
        command: ["sh", "-c", "ulimit -n 65536"]
        securityContext:
          privileged: true
  volumeClaimTemplates:
  - metadata:
      name: data
      labels:
        app: elasticsearch
    spec:
      accessModes: [ "ReadWriteOnce" ]
      storageClassName: elasticsearch-sc
      resources:
        requests:
          storage: 50Gi
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_statefulset.yaml
statefulset.apps/es-cluster created

Now let's create a service for our Eleasticsearch cluster so that it can be accessed via that.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat elasticsearch_svc.yaml
kind: Service
apiVersion: v1
metadata:
  name: elasticsearch
  namespace: kube-logging
  labels:
    app: elasticsearch
spec:
  selector:
    app: elasticsearch
  clusterIP: None
  ports:
    - port: 9200
      name: rest
    - port: 9300
      name: inter-node
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f elasticsearch_svc.yaml
service/elasticsearch created

Let's check how our Elasticsearch setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging
NAME               READY   STATUS    RESTARTS   AGE
pod/es-cluster-0   1/1     Running   0          20m
pod/es-cluster-1   1/1     Running   0          19m
pod/es-cluster-2   1/1     Running   0          19m

NAME                    TYPE        CLUSTER-IP   EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch   ClusterIP   None         <none>        9200/TCP,9300/TCP   36s

NAME                          READY   AGE
statefulset.apps/es-cluster   3/3     20m

Now let's deploy Kibana for our cluster so that we can visualize our data.
First, we need to create a ConnfigMap for our Kibana as it will work as a configuration file for the same. It will have the credentials which will be used to connect Elasticsearch. The path from which we want to access the Kibana in our case we will be example.com/kibana instead of accessing it from the root domain path. The rest of the security parameters are optional according to personal needs.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana_configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  namespace: kube-logging
  name: kibana-config
  labels:
    app: kibana
data:
  kibana.yml: |-
    server.host: 0.0.0.0
    elasticsearch:
      hosts: ${ELASTICSEARCH_HOSTS}
      username: ${ELASTICSEARCH_USER}
      password: ${ELASTICSEARCH_PASSWORD}
    server.basePath: ${SERVER_BASEPATH}
    server.rewriteBasePath: ${SERVER_REWRITEBASEPATH}
    xpack.security.enabled: ${XPACK_SECURITY_ENABLED}
    xpack.security.sessionTimeout: 3600000
    xpack.security.encryptionKey: "REDACTEDbvhjdbvwkbvwdjnbkjvbsdvjskvjiwufwifuhiuwefbwkjcsdjckbdskj"
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana_configmap.yaml
configmap/kibana-config created

Let's deploy our Kibana with the following deployment.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  namespace: kube-logging
  name: kibana
  labels:
    app: kibana
spec:
  replicas: 1
  selector:
    matchLabels:
      app: kibana
  template:
    metadata:
      labels:
        app: kibana
    spec:
      containers:
      - name: kibana
        image: docker.elastic.co/kibana/kibana:7.6.2
        ports:
        - containerPort: 5601
          name: webinterface
        env:
        - name: ELASTICSEARCH_HOSTS
          value: "http://elasticsearch.kube-logging.svc.cluster.local:9200"
        - name: SERVER_BASEPATH
          value: "/kibana"
        - name: SERVER_REWRITEBASEPATH
          value: "true"
        - name: XPACK_SECURITY_ENABLED
          value: "true"
        - name: ELASTICSEARCH_USER
          value: "elastic"
        - name: ELASTICSEARCH_PASSWORD
          valueFrom:
            secretKeyRef:
              name: elasticsearch-pw-elastic
              key: password
        volumeMounts:
        - name: config
          mountPath: /usr/share/kibana/config/kibana.yml
          readOnly: true
          subPath: kibana.yml
      volumes:
      - name: config
        configMap:
          name: kibana-config
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana.yaml
deployment.apps/kibana created

Let's create a service for our Kibana deployment.
dravi_gadgil@cloudshell:~/logging/efk_bk$ cat kibana_service.yaml
apiVersion: v1
kind: Service
metadata:
  namespace: kube-logging
  name: kibana
  labels:
    app: kibana
spec:
  type: NodePort
  ports:
  - port: 5601
    name: webinterface
    targetPort: 5601
  selector:
    app: kibana
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f kibana_service.yaml
service/kibana created

Let's check how our Kibana setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging
NAME                          READY   STATUS    RESTARTS   AGE
pod/es-cluster-0              1/1     Running   0          66m
pod/es-cluster-1              1/1     Running   0          66m
pod/es-cluster-2              1/1     Running   0          66m
pod/kibana-5c4b9749f7-mmpgg   1/1     Running   0          4m39s

NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch   ClusterIP   None          <none>        9200/TCP,9300/TCP   47m
service/kibana          NodePort    10.44.13.43   <none>        5601:32023/TCP      3m16s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana   1/1     1            1           4m39s

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-5c4b9749f7   1         1         1       4m39s

NAME                          READY   AGE
statefulset.apps/es-cluster   3/3     66m

Our Elasticsearch and Kibana are deployed successfully and the last thing which is left is Fluentd to collect logs from all the other pods in our Kubernetes cluster and push them to Elasticsearch.
We will create a ConfigMap for our Fluentd deployment so we can have better control over the parsing of logs and can pass the Elasticsearch credentials to it. You can download the latest copy of this config from the following link.

We are using the following ConfigMap with entries made to connect with our Elasticsearch cluster.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat fluentd-config.yaml
kind: ConfigMap
apiVersion: v1
metadata:
  name: fluentd-config-v0.2.0
  namespace: kube-logging
  labels:
    addonmanager.kubernetes.io/mode: Reconcile
data:
  system.conf: |-
    <system>
      root_dir /tmp/fluentd-buffers/
    </system>

  containers.input.conf: |-
    # This configuration file for Fluentd / td-agent is used
    # to watch changes to Docker log files. The kubelet creates symlinks that
    # capture the pod name, namespace, container name & Docker container ID
    # to the docker logs for pods in the /var/log/containers directory on the host.
    # If running this fluentd configuration in a Docker container, the /var/log
    # directory should be mounted in the container.
    #
    # These logs are then submitted to Elasticsearch which assumes the
    # installation of the fluent-plugin-elasticsearch & the
    # fluent-plugin-kubernetes_metadata_filter plugins.
    # See https://github.com/uken/fluent-plugin-elasticsearch &
    # https://github.com/fabric8io/fluent-plugin-kubernetes_metadata_filter for
    # more information about the plugins.
    #
    # Example
    # =======
    # A line in the Docker log file might look like this JSON:
    #
    # {"log":"2014/09/25 21:15:03 Got request with path wombat\n",
    #  "stream":"stderr",
    #   "time":"2014-09-25T21:15:03.499185026Z"}
    #
    # The time_format specification below makes sure we properly
    # parse the time format produced by Docker. This will be
    # submitted to Elasticsearch and should appear like:
    # $ curl 'http://elasticsearch-logging:9200/_search?pretty'
    # ...
    # {
    #      "_index" : "logstash-2014.09.25",
    #      "_type" : "fluentd",
    #      "_id" : "VBrbor2QTuGpsQyTCdfzqA",
    #      "_score" : 1.0,
    #      "_source":{"log":"2014/09/25 22:45:50 Got request with path wombat\n",
    #                 "stream":"stderr","tag":"docker.container.all",
    #                 "@timestamp":"2014-09-25T22:45:50+00:00"}
    #    },
    # ...
    #
    # The Kubernetes fluentd plugin is used to write the Kubernetes metadata to the log
    # record & add labels to the log record if properly configured. This enables users
    # to filter & search logs on any metadata.
    # For example a Docker container's logs might be in the directory:
    #
    #  /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b
    #
    # and in the file:
    #
    #  997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
    #
    # where 997599971ee6... is the Docker ID of the running container.
    # The Kubernetes kubelet makes a symbolic link to this file on the host machine
    # in the /var/log/containers directory which includes the pod name and the Kubernetes
    # container name:
    #
    #    synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #    ->
    #    /var/lib/docker/containers/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b/997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b-json.log
    #
    # The /var/log directory on the host is mapped to the /var/log directory in the container
    # running this instance of Fluentd and we end up collecting the file:
    #
    #   /var/log/containers/synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # This results in the tag:
    #
    #  var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # The Kubernetes fluentd plugin is used to extract the namespace, pod name & container name
    # which are added to the log message as a kubernetes field object & the Docker container ID
    # is also added under the docker field object.
    # The final tag is:
    #
    #   kubernetes.var.log.containers.synthetic-logger-0.25lps-pod_default_synth-lgr-997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b.log
    #
    # And the final log record look like:
    #
    # {
    #   "log":"2014/09/25 21:15:03 Got request with path wombat\n",
    #   "stream":"stderr",
    #   "time":"2014-09-25T21:15:03.499185026Z",
    #   "kubernetes": {
    #     "namespace": "default",
    #     "pod_name": "synthetic-logger-0.25lps-pod",
    #     "container_name": "synth-lgr"
    #   },
    #   "docker": {
    #     "container_id": "997599971ee6366d4a5920d25b79286ad45ff37a74494f262e3bc98d909d0a7b"
    #   }
    # }
    #
    # This makes it easier for users to search for logs by pod name or by
    # the name of the Kubernetes container regardless of how many times the
    # Kubernetes pod has been restarted (resulting in a several Docker container IDs).

    # Json Log Example:
    # {"log":"[info:2016-02-16T16:04:05.930-08:00] Some log text here\n","stream":"stdout","time":"2016-02-17T00:04:05.931087621Z"}
    # CRI Log Example:
    # 2016-02-17T00:04:05.931087621Z stdout F [info:2016-02-16T16:04:05.930-08:00] Some log text here
    <source>
      @id fluentd-containers.log
      @type tail
      path /var/log/containers/*.log
      pos_file /var/log/es-containers.log.pos
      tag raw.kubernetes.*
      read_from_head true
      <parse>
        @type multi_format
        <pattern>
          format json
          time_key time
          time_format %Y-%m-%dT%H:%M:%S.%NZ
        </pattern>
        <pattern>
          format /^(?<time>.+) (?<stream>stdout|stderr) [^ ]* (?<log>.*)$/
          time_format %Y-%m-%dT%H:%M:%S.%N%:z
        </pattern>
      </parse>
    </source>

    # Detect exceptions in the log output and forward them as one log entry.
    <match raw.kubernetes.**>
      @id raw.kubernetes
      @type detect_exceptions
      remove_tag_prefix raw
      message log
      stream stream
      multiline_flush_interval 5
      max_bytes 500000
      max_lines 1000
    </match>

    # Concatenate multi-line logs
    <filter **>
      @id filter_concat
      @type concat
      key message
      multiline_end_regexp /\n$/
      separator ""
    </filter>

    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @id filter_kubernetes_metadata
      @type kubernetes_metadata
    </filter>

    # Fixes json fields in Elasticsearch
    <filter kubernetes.**>
      @id filter_parser
      @type parser
      key_name log
      reserve_data true
      remove_key_name_field true
      <parse>
        @type multi_format
        <pattern>
          format json
        </pattern>
        <pattern>
          format none
        </pattern>
      </parse>
    </filter>

  system.input.conf: |-
    # Example:
    # 2015-12-21 23:17:22,066 [salt.state       ][INFO    ] Completed state [net.ipv4.ip_forward] at time 23:17:22.066081
    <source>
      @id minion
      @type tail
      format /^(?<time>[^ ]* [^ ,]*)[^\[]*\[[^\]]*\]\[(?<severity>[^ \]]*) *\] (?<message>.*)$/
      time_format %Y-%m-%d %H:%M:%S
      path /var/log/salt/minion
      pos_file /var/log/salt.pos
      tag salt
    </source>

    # Example:
    # Dec 21 23:17:22 gke-foo-1-1-4b5cbd14-node-4eoj startupscript: Finished running startup script /var/run/google.startup.script
    <source>
      @id startupscript.log
      @type tail
      format syslog
      path /var/log/startupscript.log
      pos_file /var/log/es-startupscript.log.pos
      tag startupscript
    </source>

    # Examples:
    # time="2016-02-04T06:51:03.053580605Z" level=info msg="GET /containers/json"
    # time="2016-02-04T07:53:57.505612354Z" level=error msg="HTTP Error" err="No such image: -f" statusCode=404
    # TODO(random-liu): Remove this after cri container runtime rolls out.
    <source>
      @id docker.log
      @type tail
      format /^time="(?<time>[^"]*)" level=(?<severity>[^ ]*) msg="(?<message>[^"]*)"( err="(?<error>[^"]*)")?( statusCode=($<status_code>\d+))?/
      path /var/log/docker.log
      pos_file /var/log/es-docker.log.pos
      tag docker
    </source>

    # Example:
    # 2016/02/04 06:52:38 filePurge: successfully removed file /var/etcd/data/member/wal/00000000000006d0-00000000010a23d1.wal
    <source>
      @id etcd.log
      @type tail
      # Not parsing this, because it doesn't have anything particularly useful to
      # parse out of it (like severities).
      format none
      path /var/log/etcd.log
      pos_file /var/log/es-etcd.log.pos
      tag etcd
    </source>

    # Multi-line parsing is required for all the kube logs because very large log
    # statements, such as those that include entire object bodies, get split into
    # multiple lines by glog.

    # Example:
    # I0204 07:32:30.020537    3368 server.go:1048] POST /stats/container/: (13.972191ms) 200 [[Go-http-client/1.1] 10.244.1.3:40537]
    <source>
      @id kubelet.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kubelet.log
      pos_file /var/log/es-kubelet.log.pos
      tag kubelet
    </source>

    # Example:
    # I1118 21:26:53.975789       6 proxier.go:1096] Port "nodePort for kube-system/default-http-backend:http" (:31429/tcp) was open before and is still needed
    <source>
      @id kube-proxy.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-proxy.log
      pos_file /var/log/es-kube-proxy.log.pos
      tag kube-proxy
    </source>

    # Example:
    # I0204 07:00:19.604280       5 handlers.go:131] GET /api/v1/nodes: (1.624207ms) 200 [[kube-controller-manager/v1.1.3 (linux/amd64) kubernetes/6a81b50] 127.0.0.1:38266]
    <source>
      @id kube-apiserver.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-apiserver.log
      pos_file /var/log/es-kube-apiserver.log.pos
      tag kube-apiserver
    </source>

    # Example:
    # I0204 06:55:31.872680       5 servicecontroller.go:277] LB already exists and doesn't need update for service kube-system/kube-ui
    <source>
      @id kube-controller-manager.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-controller-manager.log
      pos_file /var/log/es-kube-controller-manager.log.pos
      tag kube-controller-manager
    </source>

    # Example:
    # W0204 06:49:18.239674       7 reflector.go:245] pkg/scheduler/factory/factory.go:193: watch of *api.Service ended with: 401: The event in requested index is outdated and cleared (the requested history has been cleared [2578313/2577886]) [2579312]
    <source>
      @id kube-scheduler.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/kube-scheduler.log
      pos_file /var/log/es-kube-scheduler.log.pos
      tag kube-scheduler
    </source>

    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    <source>
      @id glbc.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/glbc.log
      pos_file /var/log/es-glbc.log.pos
      tag glbc
    </source>

    # Example:
    # I0603 15:31:05.793605       6 cluster_manager.go:230] Reading config from path /etc/gce.conf
    <source>
      @id cluster-autoscaler.log
      @type tail
      format multiline
      multiline_flush_interval 5s
      format_firstline /^\w\d{4}/
      format1 /^(?<severity>\w)(?<time>\d{4} [^\s]*)\s+(?<pid>\d+)\s+(?<source>[^ \]]+)\] (?<message>.*)/
      time_format %m%d %H:%M:%S.%N
      path /var/log/cluster-autoscaler.log
      pos_file /var/log/es-cluster-autoscaler.log.pos
      tag cluster-autoscaler
    </source>

    # Logs from systemd-journal for interesting services.
    # TODO(random-liu): Remove this after cri container runtime rolls out.
    <source>
      @id journald-docker
      @type systemd
      matches [{ "_SYSTEMD_UNIT": "docker.service" }]
      <storage>
        @type local
        persistent true
        path /var/log/journald-docker.pos
      </storage>
      read_from_head true
      tag docker
    </source>

    <source>
      @id journald-container-runtime
      @type systemd
      matches [{ "_SYSTEMD_UNIT": "{{ fluentd_container_runtime_service }}.service" }]
      <storage>
        @type local
        persistent true
        path /var/log/journald-container-runtime.pos
      </storage>
      read_from_head true
      tag container-runtime
    </source>

    <source>
      @id journald-kubelet
      @type systemd
      matches [{ "_SYSTEMD_UNIT": "kubelet.service" }]
      <storage>
        @type local
        persistent true
        path /var/log/journald-kubelet.pos
      </storage>
      read_from_head true
      tag kubelet
    </source>

    <source>
      @id journald-node-problem-detector
      @type systemd
      matches [{ "_SYSTEMD_UNIT": "node-problem-detector.service" }]
      <storage>
        @type local
        persistent true
        path /var/log/journald-node-problem-detector.pos
      </storage>
      read_from_head true
      tag node-problem-detector
    </source>

    <source>
      @id kernel
      @type systemd
      matches [{ "_TRANSPORT": "kernel" }]
      <storage>
        @type local
        persistent true
        path /var/log/kernel.pos
      </storage>
      <entry>
        fields_strip_underscores true
        fields_lowercase true
      </entry>
      read_from_head true
      tag kernel
    </source>

  forward.input.conf: |-
    # Takes the messages sent over TCP
    <source>
      @id forward
      @type forward
    </source>

  monitoring.conf: |-
    # Prometheus Exporter Plugin
    # input plugin that exports metrics
    <source>
      @id prometheus
      @type prometheus
    </source>

    <source>
      @id monitor_agent
      @type monitor_agent
    </source>

    # input plugin that collects metrics from MonitorAgent
    <source>
      @id prometheus_monitor
      @type prometheus_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # input plugin that collects metrics for output plugin
    <source>
      @id prometheus_output_monitor
      @type prometheus_output_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

    # input plugin that collects metrics for in_tail plugin
    <source>
      @id prometheus_tail_monitor
      @type prometheus_tail_monitor
      <labels>
        host ${hostname}
      </labels>
    </source>

  output.conf: |-
    <match **>
      @id elasticsearch
      @type elasticsearch
      @log_level info
      type_name _doc
      include_tag_key true
      host elasticsearch.kube-logging.svc.cluster.local
      port 9200
      user "#{ENV['ELASTICSEARCH_USER']}"
      password "#{ENV['ELASTICSEARCH_PASSWORD']}"
      logstash_format true
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size 2M
        total_limit_size 500M
        overflow_action block
      </buffer>
    </match>
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f fluentd-config.yaml
configmap/fluentd-config-v0.2.0 created

Let's deploy our Fluentd with the following DaemonSet. It also includes ClusterRole which is necessary to provide required permission to our Fluentd to be able to fetch logs from all pods within our Kubernetes Cluster. It includes credentials to connect to the Elasticsearch cluster and other required configuration.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat fluentd.yaml
apiVersion: v1
kind: ServiceAccount
metadata:
  name: fluentd
  namespace: kube-logging
  labels:
    app: fluentd
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: fluentd
  labels:
    app: fluentd
rules:
- apiGroups:
  - ""
  resources:
  - pods
  - namespaces
  verbs:
  - get
  - list
  - watch
---
kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: fluentd
roleRef:
  kind: ClusterRole
  name: fluentd
  apiGroup: rbac.authorization.k8s.io
subjects:
- kind: ServiceAccount
  name: fluentd
  namespace: kube-logging
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: fluentd-v3.0.1
  namespace: kube-logging
  labels:
    k8s-app: fluentd
    version: v3.0.1
    addonmanager.kubernetes.io/mode: Reconcile
spec:
  selector:
    matchLabels:
      k8s-app: fluentd
      version: v3.0.1
  template:
    metadata:
      labels:
        k8s-app: fluentd
        version: v3.0.1
      # This annotation ensures that fluentd does not get evicted if the node
      # supports critical pod annotation based priority scheme.
      # Note that this does not guarantee admission on the nodes (#40573).
      annotations:
        seccomp.security.alpha.kubernetes.io/pod: 'docker/default'
    spec:
      serviceAccount: fluentd
      serviceAccountName: fluentd
      containers:
      - name: fluentd
        image: quay.io/fluentd_elasticsearch/fluentd:v3.0.1
        env:
          - name:  FLUENT_ELASTICSEARCH_HOST
            value: "elasticsearch.kube-logging.svc.cluster.local"
          - name:  FLUENT_ELASTICSEARCH_PORT
            value: "9200"
          - name: FLUENT_ELASTICSEARCH_SCHEME
            value: "http"
          - name: FLUENT_UID
            value: "0"
          - name: FLUENTD_ARGS
            value: --no-supervisor -q
          - name: ELASTICSEARCH_USER
            value: "elastic"
          - name: ELASTICSEARCH_PASSWORD
            valueFrom:
              secretKeyRef:
                name: elasticsearch-pw-elastic
                key: password
        resources:
          limits:
            memory: 500Mi
          requests:
            cpu: 100m
            memory: 200Mi
        volumeMounts:
        - name: varlog
          mountPath: /var/log
        - name: varlibdockercontainers
          mountPath: /var/lib/docker/containers
          readOnly: true
        - name: config-volume
          mountPath: /etc/fluent/config.d
      terminationGracePeriodSeconds: 30
      volumes:
      - name: varlog
        hostPath:
          path: /var/log
      - name: varlibdockercontainers
        hostPath:
          path: /var/lib/docker/containers
      - name: config-volume
        configMap:
          name: fluentd-config-v0.2.0
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl apply -f fluentd.yaml
serviceaccount/fluentd created
clusterrole.rbac.authorization.k8s.io/fluentd created
clusterrolebinding.rbac.authorization.k8s.io/fluentd created
daemonset.apps/fluentd-v3.0.1 created

Let's check how our Kibana setup looks like.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl get all -n kube-logging
NAME                          READY   STATUS    RESTARTS   AGE
pod/es-cluster-0              1/1     Running   0          117m
pod/es-cluster-1              1/1     Running   0          117m
pod/es-cluster-2              1/1     Running   0          117m
pod/fluentd-v3.0.1-697gv      1/1     Running   0          2m15s
pod/fluentd-v3.0.1-c5x76      1/1     Running   0          2m15s
pod/fluentd-v3.0.1-js6xj      1/1     Running   0          2m16s
pod/fluentd-v3.0.1-kfrpf      1/1     Running   0          2m16s
pod/fluentd-v3.0.1-wtjgj      1/1     Running   0          2m15s
pod/fluentd-v3.0.1-x5ws9      1/1     Running   0          2m16s
pod/kibana-5c4b9749f7-mmpgg   1/1     Running   0          55m

NAME                    TYPE        CLUSTER-IP    EXTERNAL-IP   PORT(S)             AGE
service/elasticsearch   ClusterIP   None          <none>        9200/TCP,9300/TCP   98m
service/kibana          NodePort    10.44.13.43   <none>        5601:32023/TCP      54m

NAME                            DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
daemonset.apps/fluentd-v3.0.1   6         6         6       6            6           <none>          2m16s

NAME                     READY   UP-TO-DATE   AVAILABLE   AGE
deployment.apps/kibana   1/1     1            1           55m

NAME                                DESIRED   CURRENT   READY   AGE
replicaset.apps/kibana-5c4b9749f7   1         1         1       55m

NAME                          READY   AGE
statefulset.apps/es-cluster   3/3     117m

Cool, Now our setup is up and running. Let's check our Kibana to see how it turns up. I'm using port-forward just to check the Kibana dashboard.
ravi_gadgil@cloudshell:~/logging/efk_bk$ kubectl port-forward --namespace kube-logging service/kibana 5601:5601 &
[1] 5494
ravi_gadgil@cloudshell:~/logging/efk_bk$ Forwarding from 127.0.0.1:5601 -> 5601

As we have defined our Kibana to open on /kibana URL so use that to access.
Kibana DashBoard
Kibana Dashboard
Use credentials elastic/changeme to login to your Kibana dashboard if you have not changed the same in the Secret file.
Kibana Dashboard
Kibana Dashboard
Create an index pattern to access all the logs from Elasticseach as logstash-* in the management section.
Kibana Dashboard
Kibana Dashboard
Kibana Dashboard
Kibana Dashboard
Now let's go to Discover option to see our logs and how they come up.
Kibana Dashboard
Kibana Dashboard
As we have deployed Kibana with example.com/kibana so following Istio configuration can be used to do this route if you're using Istio and Ingress Gateway. If you want to set up Istio can follow the link.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat istio-virtualservice.yaml
apiVersion: networking.istio.io/v1alpha3
kind: Gateway
metadata:
  name: default-gateway
  namespace: istio-system
spec:
  selector:
    istio: ingressgateway
  servers:

  - hosts:
    - '*'
    port:
      name: http
      number: 80
      protocol: HTTP
---
apiVersion: networking.istio.io/v1alpha3
kind: VirtualService
metadata:
  name: kibana
  namespace: kube-logging
spec:
  hosts:
  - "example.com"
  gateways:
  - default-gateway.istio-system.svc.cluster.local
  http:
  - match:
    - uri:
        prefix: /kibana
    route:
    - destination:
        host: kibana.kube-logging.svc.cluster.local
        port:
          number: 5601

If you want to test whether logs are coming from new deployments and all namespaces the following deployment can be used.
ravi_gadgil@cloudshell:~/logging/efk_bk$ cat counter.yaml
apiVersion: v1
kind: Pod
metadata:
  name: counter
spec:
  containers:
  - name: count
    image: busybox
    args: [/bin/sh, -c,
            'i=0; while true; do echo "$i: $(date)"; i=$((i+1)); sleep 1; done']

No comments:

Post a Comment

Setup fully configurable EFK Elasticsearch Fluentd Kibana setup in Kubernetes

In the following setup, we will be creating a fully configurable Elasticsearch, Flunetd, Kibana setup better known as EKF setup. There is a...