Origin 缘起

目前日志方案使用的是Graylog，方便是真方便。但是因为底层用的是Elasticsearch以及Java开发的原因。在对资源要求上的确是蛮高。目前生产使用的都是8Core/16GB+1TBDisk的机器。趁有空测试下新的日志系统，目前比较了几个新秀开源的方案，钟情于OpenObserve。官方说号称10倍性能与成本。

新秀VictoriaLogs也号称性能强悍,成本节约。有空也研究下

Practice 实践

因为之前使用Graylog，所以直接使用Graylog的GELF协议。这里起一个Vector来作为Agent代理，用于收集UDP GELF日志.

那么原来的架构就变成了

GELF--->Vector(Agent)-->OpenObserve
  • docker-compose.yml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34

version: '2.2'

services:
    vector:
      image: timberio/vector:0.42.0-debian
      container_name: vector-agent
      restart: always
      privileged: true
      ports:
      - 12201:12201/udp
      - 12514:12514/udp
      environment:
      - TZ=Asia/Shanghai
      volumes:
      - ./vt_data:/var/lib/vector:rw
      - ./vector.yaml:/etc/vector/vector.yaml:ro
      mem_limt: 1g
      links:
      - openobserver:ob

    openobserver:
      image: public.ecr.aws/zinclabs/openobserve:latest
      container_name: openobserver
      privileged: true
      restart: always
      ports:
      - 5080:5080
      environment:
      - TZ=Asia/Shanghai
      - ZO_ROOT_USER_EMAIL=root@example.com
      - ZO_ROOT_USER_PASSWORD=password
      - ZO_DATA_DIR=/data
      volumes:
      - ./oo_data:/data:rw
  • vector.yaml
 1
 2
 3
 4
 5
 6
 7
 8
 9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
61
62
63
64
65
66
67
68
69
70

data_dir: "/var/lib/vector"
  enabled: false
# address = "127.0.0.1:8686"

# Ingest data by tailing one or more files
sources:
  bytes_logs:
    type: socket
    address: 0.0.0.0:12205
    mode: udp
    receive_buffer_bytes: 181900
    decoding:
      codec: bytes
      lossylossy: true

  syslog_logs:
    type: syslog
    address: 0.0.0.0:12514
    mode: udp

# transforms的输入可以是transform的ID，也可以是sources的ID
transforms:
  syslog_json:
    type: remap
    inputs:
    - syslog_logs
    source: |
            . = parse_json!(.message)

  gelf_logs:
    type: remap
    inputs:
    - bytes_logs
    source: |
      .message = decode_gzip!(.message)
      . = parse_json!(.message)
      del(._forwarder)      


# Send structured data to a short-term storage
# Document refer to https://vector.dev/docs/reference/vrl/functions/
sinks:
  ob_default:
    type: http
    inputs:
    - gelf_logs
    uri: http://ob:5080/api/default/default/_json
    method: post
    compression: gzip
    auth:
      strategy: basic
      user: "root@example.com"
      password: "egZ9SbrICBMH9oXo"
    encoding:
      codec: json
      timestamp_format: "rfc3339"

  ob_syslog:
    type: http
    inputs:
    - syslog_json
    uri: http://ob:5080/api/default/syslog/_json
    method: post
    compression: gzip
    auth:
      strategy: basic
      user: "root@example.com"
      password: "egZ9SbrICBMH9oXo"
    encoding:
      codec: json

Tips

  • Vector的UDP协议，如果数据过大，可能会丢失。
  • Vector在处理过程中，.message代表原始数据
  • OpenObserver可以通过URI来实现不同数据流，会自动生成索引