INT 21h

• 🏠 ДОМОЙ
• Админство
• Програмство
• Электронство
• Микроскопство
• Мой софт
• 3D печать
• Авто
• Музей
• Часы

Filebeat to Opensearch

№ 11882 В разделе Sysadmin от March 7th, 2023,
В подшивках: ELK

Opensearch is a fork of Elasticsearch and some API functions not compatible with Filebeat and other *beats. To get it worked you need special Logstash version with opensearch plugin support in a middle.

Filebeat config

Usual docker compose configuration:

version: '3.3'
services:
  filebeat:
    image: elastic/filebeat:7.16.2
    command: ["-e", "--strict.perms=false"]
    environment:
      LOGSTASH_HOST: 10.2.113.39
      LOGSTASH_PORT: "5044"
    user: root
    restart: always
    volumes:
      - /srv/db/mysql-logs:/var/log/mysql:ro
      - /srv/db/filebeat.yml:/usr/share/filebeat/filebeat.yml
      - /srv/db/filebeat-modules.d:/usr/share/filebeat/modules.d

Inside filebeat.yml set output.logstash

filebeat.config.modules:
  path: ${path.config}/modules.d/*.yml

output.logstash:
  hosts: ['${LOGSTASH_HOST}:${LOGSTASH_PORT}']
  index: "service-db-test-%{+yyyy.MM.dd}"
setup:
  template:
    name: "service-db-test"
    pattern: "service-db-test"

Logstash config

Here magic begins. Get this special logstash version up.

version: "3"
services:
  logstash:
    image: opensearchproject/logstash-oss-with-opensearch-output-plugin:7.16.2
    volumes:
      - "./config:/usr/share/logstash/config:ro"
    ports:
      - "5044:5044/tcp"
      - "5044:5044/udp"
    restart: always

Logstash pipeline configuration looks same as usual, but with opensearch in output section:

input {
  beats {
    host => "0.0.0.0"
    port => "5044"
  }
}

output {
  if [service][type] == "mysql" {
    opensearch
    {
	hosts => ["https://10.3.27.105:9200"]
	index => "service-db-test-filebeat-%{+yyyy.MM.dd}"
	user => "mylogin"
	password => "mypassword"
	ssl => "true"
	ssl_certificate_verification => "false"
    }
    #stdout {}
  } else {
    opensearch
    {
	hosts => ["https://10.3.27.105:9200"]
	index => "stand-test-filebeat-%{+yyyy.MM.dd}"
	user => "mylogin"
	password => "mypassword"
	ssl => "true"
	ssl_certificate_verification => "false"
    }
    #stdout{}
  }
}

Now include this pipeline to logstash.yml: path.config: "./config/pipeline.conf"

Kibana

See it in Kibana.

Done. Now you can get a cup of tea.

Нет комментариев »