Skip to main content

Grafana Alloy Agent

Overview​

Grafana Alloy is an open source agent for observability signal processing, including log processing and forwarding. It is an opinionated, vendor-neutral distribution of the OpenTelemetry (OTel) Collector.

In Alloy you configure components that are each responsible for a unit of action (like collecting log data or adding a label to each log message), and then you compose these components into an observability pipeline. Configurations are organized into modules, which can be imported for reuse or used as the main component passed to alloy run.

Alloy supports collecting log data from a variety of sources, including files, systemd journal, syslog, Kubernetes, Docker, Kafka, Heroku, Windows event logs, Cloudflare, GELF over UDP, and more.

Alloy can ship logs to SparkLogs using the loki.write component that sends data using the Loki Push API.

Unlike Loki, which requires low cardinality of labels for good cost and performance, SparkLogs is schemaless and allows unlimited fields and values, and does not bog down with high cardinality of labels.

Although AutoExtract will automatically extract structured field data from your raw logs and is recommended in the typical case, you can use Loki components such as loki.process, which has structured parsing for docker, logfmt, CRI, and Windows event logs. Advanced capabilities include regexp parsing, data redaction, and GeoIP enrichment.

If you have multi-line log events (e.g., log lines with exception backtraces), consider using the stage.multiline block to join them together.

How to Use​

Follow these steps for each logical agent that will receive data from Alloy:

1. Create agent and get config template​

In the app, click the Configure sidebar button:
Configure Sidebar Button
and then click the Agents tab.

As appropriate, create a new agent, or highlight an existing agent and click View API Key. In the dialog that shows the agent configuration template, click the Grafana Alloy tab and copy the configuration template.

2. Customize configuration​

Copy the configuration template and customize it based on your needs. At a minimum, add additional inputs in the config as appropriate (e.g., for files, kernel logs, etc.).

Example Alloy configuration template

Make sure to get your configuration template from the app, as your ingestion endpoint can vary based on your provisioned region. This is an example of what it will look like:

loki.write "default" {
  endpoint {
    url = https://ingest-<REGION>.engine.sparklogs.app:443/loki/api/v1/push
    batch_size = "2MiB"
    remote_timeout = "60s"

    // Or get basic authentication based on environment variables, e.g., env("AGENT_ID")
    basic_auth {
      username = "<AGENT-ID>"
      password = "<AGENT-ACCESS-TOKEN>"
    }
  }

  // NOTE: duplicate this endpoint block multiple times within the same loki.write block
  // if greater performance is needed so that Alloy will concurrently send ingestion data
  // requests in parallel.
}

// Other component configurations go here, for example:
// local.file_match "var" {
//   path_targets = [{"__path__" = "/var/logs/**/*.log"}]
// }
// 
// loki.source.file "files" {
//   targets    = local.file_match.var.targets
//   forward_to = [loki.write.default.receiver]
// }

3. Deploy Alloy agents​

On each system that will ship data to SparkLogs for this agent, install the Grafana Alloy agent with the appropriate configuration, and make sure it starts on system boot.

Advanced Use Cases​

Multiline aggregation​

You may wish to have multiple lines of log text joined together into one log event. For example, if your application logs a long stack trace after an error message. While this isn't required, you may find it easier to explore and analyze your log data with multi-line log messages properly merged into a single log event.

Alloy has built-in support for multi-line parsing based on regular expression detection of new log lines.

Performance and clustering​

Large-scale deployments should consider using the clustering capabilities so that the load of processing and shipping logs can be scaled horizontally. For example, log collection from Kubernetes supports clustering.

Note that certain logging agents like Vector and Fluent Bit have built-in support for concurrency and may be able to achieve greater log forwarding throughput per system.

Buffering to disk​

By default, Alloy buffers received log data into memory and does not write log data to disk before sending to the cloud for ingestion. This can be a good strategy when reading log data from sources that are already durable (e.g., files). If you have more stringent durability requirements, Alloy provides experimental support for first writing received log data to disk in a WAL.