Skip to main content

Beats Agents

The Beats family of open-source observability agents are popular in the Elastic and OpenSearch communities. filebeat ships log file data. winlogbeat ships Windows event logs. There are also dozens of community beats for shipping data using libbeat.

SparkLogs can receive data via the OpenSearch and Elasticsearch REST API for bulk indexing and can thus receive data from any beat that ships log data. Metrics beats are not yet supported.

Since SparkLogs is schemaless, no configuration or management of index templates is required. You can configure any beats agent to simply output data using the elasticsearch bulk indexing API to the SparkLogs cloud, and you're all set.

How to Use​

Follow these steps for each logical agent that will receive data from a beats agent:

1. Create agent and get config template​

In the app, click the Configure sidebar button:
Configure Sidebar Button
and then click the Agents tab.

As appropriate, create a new agent, or highlight an existing agent and click View API Key. In the dialog that shows the agent configuration template, click the Beats tab and copy the configuration template.

2. Customize configuration​

Copy the configuration template and customize it based on your needs. At a minimum, add additional inputs in the config as appropriate (e.g., for files, kernel logs, etc.).

Example Beats configuration template
...(other configurations like filebeat.inputs)...

output.elasticsearch:
  # The :443 are required. Use es6, es7, or es8 subdomain as
  # appropriate for compatibility with your version of the beats tools.
  hosts: ["https://es8.ingest-<REGION>.engine.sparklogs.app:443/"]
  username: "<AGENT-ID>"
  password: "<AGENT-ACCESS-TOKEN>"

  # Uncomment if you are doing more than 1 MB/sec of log data
  # preset: throughput

  # If you will be doing more than 4 MB/sec of log data, instead of
  # using the preset, uncomment this to set workers to the number
  # of Mbyte/sec that you need to keep up. For example for 8 Mbyte/sec:
  # workers: 8

  # You can let it auto-choose an index name, or specify an index name
  # index: "filebeat"

  # You should disable any pipeline, it's not needed or supported. Keep commented:
  # pipeline: "ingest/elastic"

3. Deploy Beats agents​

On each system that will ship data to SparkLogs for this agent, install the appropriate beats agent (e.g., filebeat, winlogbeat, or a community beat) with the appropriate configuration, and make sure it starts on system boot.