Docker - ELK 7.6 : Filebeat

bogotobogo.com site search:

Note

Elastic Stack docker/kubernetes series:

Docker - ELK 7.6 : Elasticsearch

Docker - ELK 7.6 : Filebeat

Docker - ELK 7.6 : Logstash (All in One)

Docker - ELK 7.6 : Kibana

Docker - ELK 7.6 : Kibana II

Docker - ELK 7.6 : Elastic Stack with Docker Compose

Docker - Deploy Elastic Cloud on Kubernetes (ECK) via Elasticsearch operator on minikube

Docker - Deploy Elastic Stack via Helm on minikube

Filebeat

Filebeat is a lightweight shipper for forwarding and centralizing log data. Installed as an agent on our servers, Filebeat monitors the log files or locations that we specify, collects log events, and forwards them either to Elasticsearch or Logstash for indexing.

source: https://www.elastic.co/guide/en/beats/filebeat/current/how-filebeat-works.html

Filebeat is one of the best log file shippers out there today — it’s lightweight, supports SSL and TLS encryption, supports back pressure with a good built-in recovery mechanism (it records the last successful line indexed in the registry, so in case of network issues or interruptions in transmissions, remembers where it left off when re-establishing a connection). However, it cannot, in most cases, turn our logs into easy-to-analyze structured log messages using filters for log enhancements. That's the role played by Logstash. That's why we will almost always need to use Filebeat and Logstashin tandem. Filebeat (log files), and the other members of the Beats family (Packetbeat: network metrics), Metricbeat: server metrics), acts as a lightweight agent deployed on the edge host, pumping data into Logstash for aggregation, filtering and enrichment.

We can ship logs on hosts via Filebeat directly into Elasticsearch. Since Filebeat ships data in JSON format, Elasticsearch should be able to parse the timestamp and message fields without too much hassle. However, adding context to the log messages by parsing them up into separate fields, filtering out unwanted bits of data and enriching others — cannot be handled without Logstash.

When we start Filebeat, it starts one or more inputs that look in the locations we've specified for log data. For each log that Filebeat locates, Filebeat starts a harvester. Each harvester reads a single log for new content and sends the new log data to libbeat, which aggregates the events and sends the aggregated data to the output that we've configured for Filebeat.

Install Filebeat 7.6 with Docker

Filebeat is available as Docker images. The images use centos:7 as the base image.

A list of all published Docker images and tags is available at www.docker.elastic.co.

Pulling the image

Issue a docker pull command against the Elastic Docker registry:

$ docker pull docker.elastic.co/beats/filebeat:7.6.2

Filebeat setup

For the latest updates on working with Elastic stack and Filebeat, skip this and please check Docker - ELK 7.6 : Logstash on Centos 7.

As discussed earlier, the filebeat can directly ship logs to elasticsearch bypassing optional Logstash. So, we need to make sure that the "elasticsearch" and "kibana" containers are running with dokcer network (here, "mynetwork"):

$ docker network create mynetwork --driver bridge    

$ docker run -p 9200:9200 -p 9300:9300 --net=mynetwork --name elasticsearch -e "discovery.type=single-node" docker.elastic.co/elasticsearch/elasticsearch:7.6.2 

$ docker run --name kibana --net=mynetwork -p 5601:5601 kibana:7.6.2 

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
07d4880601e6        kibana:7.6.2                                          "/usr/local/bin/dumb…"   2 hours ago         Up 2 hours          0.0.0.0:5601->5601/tcp                           kibana
3602c16cd3b1        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   3 hours ago         Up 3 hours          0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch

Running Filebeat with the setup command will create the index pattern and load visualizations , dashboards, and machine learning jobs. Run this command:

$ docker run --net=mynetwork --name filebeat docker.elastic.co/beats/filebeat:7.6.2 setup -E setup.kibana.host=kibana:5601 -E output.elasticsearch.hosts=["elasticsearch:9200"]
Overwriting ILM policy is disabled. Set `setup.ilm.overwrite:true` for enabling.

Index setup finished.
Loading dashboards (Kibana must be running and reachable)
Loaded dashboards
Setting up ML using setup --machine-learning is going to be removed in 8.0.0. Please use the ML app instead.
See more: https://www.elastic.co/guide/en/elastic-stack-overview/current/xpack-ml.html
Loaded machine learning job configurations
Loaded Ingest pipelines

Now, we can check the beat is in Kibana:

Configure Filebeat on Docker

The Docker image provides several methods for configuring Filebeat. The conventional approach is to provide a configuration file via a volume mount, but it's also possible to create a custom image with our configuration included.

Download an example configuration file as a starting point:

$ curl -L -O https://raw.githubusercontent.com/elastic/beats/7.6/deploy/docker/filebeat.docker.yml

$ cat filebeat.docker.yml
filebeat.config:
  modules:
    path: ${path.config}/modules.d/*.yml
    reload.enabled: false

filebeat.autodiscover:
  providers:
    - type: docker
      hints.enabled: true

processors:
- add_cloud_metadata: ~

output.elasticsearch:
  hosts: '${ELASTICSEARCH_HOSTS:localhost:9200}'
  username: '${ELASTICSEARCH_USERNAME:}'
  password: '${ELASTICSEARCH_PASSWORD:}'

To test this works, we may want to restart the other containers ("elasticsearch" and "kibana"):

We'll configure Filebeat on Docker by providing the filebeat.docker.yml via a volume mount. With docker run, the volume mount can be specified like this:

$ docker run -d \
  --name=filebeat \
  --user=root \
  --network=mynetwork \
  --volume="$(pwd)/filebeat.docker.yml:/usr/share/filebeat/filebeat.yml:ro" \
  --volume="/var/lib/docker/containers:/var/lib/docker/containers:ro" \
  --volume="/var/run/docker.sock:/var/run/docker.sock:ro" \
  docker.elastic.co/beats/filebeat:7.6.2 filebeat -e -strict.perms=false \
  -E output.elasticsearch.hosts=["elasticsearch:9200"]

944f1aa052cf109332a31f8ab1685d8455299ba8fdd260ba14e3990fb6e995b9

$ docker ps
CONTAINER ID        IMAGE                                                 COMMAND                  CREATED             STATUS              PORTS                                            NAMES
944f1aa052cf        docker.elastic.co/beats/filebeat:7.6.2                "/usr/local/bin/dock…"   6 minutes ago       Up 6 minutes                                                         filebeat
49673f99c7d1        kibana:7.6.2                                          "/usr/local/bin/dumb…"   13 minutes ago      Up 13 minutes       0.0.0.0:5601->5601/tcp                           kibana
8fc7a98f6fc9        docker.elastic.co/elasticsearch/elasticsearch:7.6.2   "/usr/local/bin/dock…"   13 minutes ago      Up 13 minutes       0.0.0.0:9200->9200/tcp, 0.0.0.0:9300->9300/tcp   elasticsearch