Monitoring and Logging

Introduction

Here we describe how you can setup your logging and monitoring for kubernetes keeping in mind dashboard design also.

Logging

Logging mainly divide into three parts

1. Logs collector

   1. fluentd

   2. Elasticsearch Beats (like filebeat for logs)

   3. fluent bit (light version of fluentd written in C for performance and low resource utilization)

   4. Telegraf (part of influxdb project)

2. Logs storage

   1. Elasticsearch (best for text based searching)

   2. filesystem (if you won't interested in text based search)

   3. InfluxDB (influxDB also able to store logs, but I am not so much aware)

3. Logs dashboard

   1. Kibana (Kibana is well known ELK stack project and well works with elasticsearch database, but the problem is it uses lots of RAM, you can use logtrail plugin for showing logs in tail form based on some filters)

Here we are going to use Elasticsearch, fluent bit, and Kibana for logging solution for kubernetes.

Fluent bit/Fluentd

For collecting all logs you need to run fluent bit/fluentd as a DaemonSet and collect all files from folder /var/log/containers. You can also use helm chart for the deployment of this.

Here we can also create new index of each namespace of kubernetes, this easily the filtering little bit on logtrail plugin screen

  output.conf: |
    # Enriches records with Kubernetes metadata
    <filter kubernetes.**>
      @type kubernetes_metadata
      annotation_match [ ".+"]  # Get annotations
    </filter>
    
    <filter kubernetes.**>
      @type record_modifier
      remove_keys _dummy_
      <record>
        _dummy_ ${ record['stag']= tag; if record["kubernetes"].has_key?("annotations"); if record["kubernetes"]["annotations"].has_key?('fluentbit_io/parser'); record['stag'] = record["kubernetes"]["annotations"]['fluentbit_io/parser'] + "." + tag ; end ; end; nil}
        # _dummy_ ${ record["before"] = "before" ; nil }
      </record>
    </filter>
    
    match kubernetes.**>
      @type rewrite_tag_filter
      capitalize_regex_backreference yes
      @log_level info
      <rule>
        key stag
        pattern ^(.+)$
        tag modified.$1
      </rule>
    </match>

    <filter modified.**>
      @type record_modifier
      remove_keys stag
    </filter>

    
    <match **>
      @id elasticsearch
      @type elasticsearch_dynamic
      @log_level info
      include_tag_key true
      host "#{ENV['OUTPUT_HOST']}"
      port "#{ENV['OUTPUT_PORT']}"
      logstash_format true
      logstash_prefix namespace-${record['kubernetes']['namespace_name']}
      <buffer>
        @type file
        path /var/log/fluentd-buffers/kubernetes.system.buffer
        flush_mode interval
        retry_type exponential_backoff
        flush_thread_count 2
        flush_interval 5s
        retry_forever
        retry_max_interval 30
        chunk_limit_size "#{ENV['OUTPUT_BUFFER_CHUNK_LIMIT']}"
        queue_limit_length "#{ENV['OUTPUT_BUFFER_QUEUE_LIMIT']}"
        overflow_action block
      </buffer>
    </match>

  output.conf: |-
    <match **>
       type elasticsearch_dynamic
       log_level info
       include_tag_key true
       host elasticsearch-logging
       port 9200
       logstash_format true
       logstash_prefix namespace-${record['kubernetes']['namespace_name']}
       # Set the chunk limits.
       buffer_chunk_limit 2M
       buffer_queue_limit 8
       flush_interval 5s
       # Never wait longer than 5 minutes between retries.
       max_retry_wait 30
       # Disable the limit on the number of retries (retry forever).
       disable_retry_limit
       # Use multiple threads for processing.
       num_threads 2
    </match>

In above you can see that I am using logstash_prefix which is used generate dynamic index in elasticsearch, you can also use your own pattern, this will help you filter in settings of logtrail plugin

Use any standard way to install fluent bit/fluentd on your kubernetes, for e.g., use below helm charts

1. fluentd -> https://github.com/kubernetes/charts/tree/master/incubator/fluentd-elasticsearch
2. fluent bit -> https://github.com/kubernetes/charts/tree/master/stable/fluent-bit

Elasticsearch

Install elasticsearch on your own, either use helm chart (https://github.com/kubernetes/charts/tree/master/incubator/elasticsearch) and just use any VMs to install it, you can find many tutorial to configure and optimize it.

• Include all steps of installation of elasticsearch

Kibana

You can install default kibana either on Kubernetes or on any machine from where it can access elasticsearch

Logging tail in Kibana

You can use logtail(https://github.com/sivasamyk/logtrail) plugin of kibana for that, and use below logtrail.json

{
  "index_patterns" : [
    {
      "es": {
        "default_index": "logstash-local-influxdb-*",
        "allow_url_parameter": false
      },
      "tail_interval_in_seconds": 2,
      "es_index_time_offset_in_seconds": 0,
      "display_timezone": "local",
      "display_timestamp_format": "MMM DD HH:mm:ss",
      "max_buckets": 500,
      "nested_objects" : true,
      "default_time_range_in_days" : 0,
      "fields" : {
        "mapping" : {
            "timestamp" : "@timestamp",
            "display_timestamp" : "@timestamp",
            "hostname" : "kubernetes.pod_name",
            "program": "kubernetes.container_name",
            "message": "log"
        },
        "message_format": "ENV|{{{kubernetes.labels.environment}}} APP|{{{kubernetes.labels.app}}} : {{{log}}}"
      }
    }
    ]
}

In above you can just use various filters according to index patterns made my fluentd/fluent bit and also define your own mapping