Skip to main content

Loki Push API

Overview​

Log data can be ingested via the Loki Push API over HTTPS. Support is provided for both POST /api/prom/push (deprecated) and POST /loki/api/v1/push. Both variants of the JSON format as well as the snappy-compressed protobuf format are supported.

Popular tools support shipping log data via the Loki Push API, including Grafana Alloy and Promtail.

SparkLogs overcomes many traditional pains of using Loki to ingest and query log data:

  • The schemaless design with support for infinite fields means there are no bottlenecks caused by high cardinality of labels.
  • SparkLogs scales from zero to petabytes in an instant; its serverless architecture allows it to handle extremely bursty or high throughput log volumes without load issues.
  • There are no scaling limitations to worry about where full-text indexes, storage, and querying of data are scale-out and fully managed.
  • Querying is fast even with billions of matching events with our indexing and caching engine, and you can explore and filter large result sets with ease.

With SparkLogs, you can keep your existing log aggregation and forwarding tools and simply point them to the SparkLogs cloud instead. Then enjoy logging without limits.

Implementation Details​

This API is implemented on the main ingest HTTPS endpoint on port 443 via the two REST endpoints:

  • POST /api/prom/push (deprecated)
  • POST /loki/api/v1/push

The full URI will depend on your region and look something like:
https://ingest-<REGION>.engine.sparklogs.app/loki/api/v1/push

The Content-Type header should be set to application/json when sending data in JSON format. In this case, the content can be compressed and the Content-Encoding header can be set to indicate the compression method used (gzip, deflate, or snappy are supported).

For other values of Content-Type header (or if it's not set at all), then the data should be a protobuf encoding of the log data that is compressed with snappy (block algorithm). Note that the Content-Encoding encoding header should not be set even though the submitted data is snappy-compressed.

JSON Format​

The new format has a JSON payload with one or more streams of data, with each stream containing values representing log events. Stream labels that apply to all events in a stream are encoded as a JSON object, and each log message can encode additional custom fields on a per log event basis.

{
  "streams": [
    {
      "stream": {
        "label1": "somevalue",
        "anotherlabel": "bar"
      },
      "values": [
          [ "<timestamp>", "<log message>"],
          [ "<timestamp>", "<log message>"],
          [ "<timestamp>", "<log message>", {"customfield": "hello", "another_field": "world"}]
      ]
    }
  ]
}

The old format takes the form:

{
  "streams": [
    {
      "labels": "<LogQL with label key-value pairs>",
      "entries": [
        {
          "ts": "<timestamp>",
          "line": "<log message>"
        }
      ]
    }
  ]
}

Labels at the stream level will be applied before labels and structured fields inside individual events. If individual events have fieldnames that conflict with the names of stream labels, then they will be renamed so as to not conflict.

A variety of timestamp formats are supported, including:

  • Number of seconds since the UNIX epoch
  • Number of nanoseconds since the UNIX epoch
  • RFC3339
  • RFC3339Nano
  • Any other timestamp format supported by SparkLogs

Authentication​

Authentication is required for every request. Credentials consist of an agent ID and an agent access token. These values can be obtained when an agent is first created, or by using the View API Key button if you have sufficient privileges.

Credentials can be passed either by HTTP Basic authentication or HTTP Bearer token authentication, and follows the same format as in the HTTPS+JSON API.

Performance​

Clients that submit large amounts of data to ingest MUST batch multiple log entries together so that under high volume multiple log events are sent with each HTTP request.

When possible aim to achieve a batch size of around 1MiB. Batches larger than 25MiB will fail. Use concurrent requests to submit data in parallel to increase throughput.