Skip to main content

HTTPS+JSON API

Overview​

Log data can be ingested by sending the data in JSON format with a POST request to our HTTPS endpoint. HTTP 1.1, HTTP/2, and HTTP/3 are all supported. TLS and authentication are required. Compressing data is encouraged but optional.

tip

SparkLogs is optimized to receive log data in batches roughly 1MiB in size, and to accept as many parallel requests as needed to ingest data quickly. Note that log batches larger than 25MiB will fail. Our configuration templates for popular log agents are tuned for optimal throughput.

JSON Payload Specification​

The JSON with the log data should be a UTF-8 encoded JSON array of either strings with the log event text or objects with stuctured fields of each log event (including the log message).

If the payload contains an array of JSON objects with structured log fields, these fields will be interpreted based on automatic standard field mapping.

If you have control over the field names of the event that you're submitting, we recommend either using the SparkLogs standard field names, or the standard field names defined in the OpenTelemetry Data Model or in the Elastic Common Schema.

The payload can optionally wrap the JSON array with the log events in a root JSON object, with a top level field of log, event, or meta containing the JSON array of events.

Example JSON Payload 1 - Simple array of text log messages
[
    "2024-09-06 20:35:01.000-0700 INFO start of request, action=create, count=15",
    "2024-09-06 20:35:02.000-0700 WARN record already exists, upserting...",
    "2024-09-06 20:35:03.000-0700 INFO end of request, elapsed_ms=512"
]
Example JSON Payload 2 - Array of log event objects
[
    {"message": "2024-09-06 20:35:01.000-0700 INFO start of request, action=create, count=15", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.123-0700"},
    {"message": "2024-09-06 20:35:02.000-0700 WARN record already exists, upserting...", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.124-0700"},
    {"message": "2024-09-06 20:35:03.000-0700 INFO end of request, elapsed_ms=512", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.126-0700"}
]
Example JSON Payload 3 - Top-level object with array of log event objects
{
    "count": 3,
    "log": [
        {"message": "2024-09-06 20:35:01.000-0700 INFO start of request, action=create, count=15", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.123-0700"},
        {"message": "2024-09-06 20:35:02.000-0700 WARN record already exists, upserting...", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.124-0700"},
        {"message": "2024-09-06 20:35:03.000-0700 INFO end of request, elapsed_ms=512", "source": "gameserver1", "env": "prod", "observedtimestamp": "2024-09-06 20:35:25.126-0700"}
    ]
}

Request Specification​

Our HTTPS endpoint allows data to be submitted via a POST request over HTTP 1.1, HTTP/2, or HTTP/3. Modern TLS 1.2 (or better) is required. The URI of the endpoint varies based on your workspace's assigned region. It has the form https://ingest-<REGION>.engine.sparklogs.app/ingest/v1.

The Content-Type header should be set to application/json; charset=utf-8. The JSON should be UTF-8 encoded.

Parameters​

Ingestion behavior can be customized by URI query parameters or through HTTP header values:

Query ParameterHTTP HeaderDescription
tzX-TimezoneThe IANA name (e.g., America/Denver) of the default time zone to use for timestamps that do not have an explicit time zone (e.g., Jan 1, 2024 5:06:07). Defaults to UTC. The format UTC[+-]HH:MM is also accepted. Note that you can override this on a per-log-event basis by passing a __agent_timezone field value for a given log event. This special field will not be ingested, but will override the default time zone for that particular log event.
no_auto_extractX-No-AutoExtractSet to true to disable AutoExtract. Defaults to false.
no_detect_categoryX-No-Detect-CategorySet to true to prevent AutoExtract from setting the category based on the message text. Defaults to false.
no_detect_severityX-No-Detect-SeveritySet to true to prevent AutoExtract from setting the log event severity based on the message text. Defaults to false.
no_detect_timestampX-No-Detect-TimestampSet to true to prevent AutoExtract from detecting the timestamp of a message from the message text. Note that timestamp detection is never performed if a given log event already had an explicit timestamp set. For a given log event, if there was no explicit timestamp, and no timestamp is detected in a log message, then it will fallback to observedtimestamp, or if that is not available, it uses the time of the previous event, and then falls back to the time of ingestion. Defaults to false.
no_remove_timestamp_from_messageX-No-Remove-Timestamp-From-MessageSet to true to prevent AutoExtract from removing the text of a detected timestamp from the message text. By default the text of detected timestamps are removed from the log message so that the timestamp does not appear twice in query results. Defaults to false.
no_unwrap_raw_json_in_messageX-No-Unwrap-Raw-Json-In-MessageSet to true to disable automatic JSON unwrapping.
auto_extract_errors_are_fatalX-AutoExtract-Errors-Are-FatalSet to true to fail the HTTP with status 400 if the log message fails the AutoExtract process.
prev_event_tX-Prev-Event-TAdvanced option that is normally not needed. The timestamp of the log event that immediately precedes the first event in this batch. Timestamp is the number of microseconds since the UNIX epoch. Defaults to 0 to indicate this is unknown.
no_auto_classifyX-No-AutoClassifySet to true to disable AutoClassify. Defaults to false.
pattern_max_lengthX-Pattern-Max-LengthGoverns the maximum length of the pattern produced by AutoClassify. Defaults to 128. Minimum 16.
severity_mapX-Severity-MapA custom severity mapping to use in addition to standard severity mappings.
no_dedupX-No-DedupSet to true to disable deduplication of event payloads larger than 16 KB. Defaults to false.

Authentication​

Authentication is required for every request. Credentials consist of an agent ID and an agent access token. These values can be obtained when an agent is first created, or by using the View API Key button if you have sufficient privileges.

Credentials can be passed either by HTTP Basic authentication or HTTP Bearer token authentication.

HTTP Basic Authentication​

Specify the agent ID for the username and the agent access token for the password.

HTTP basic authentication by setting the Authorization header Basic <credentials> where <credentials> is the Base64 encoding of the string <AGENT-ID>:<AGENT-ACCESS-TOKEN>.

HTTP Bearer Token Authentication​

To use this method, set the Authorization HTTP header to the value Bearer <AGENT-ID>:<AGENT-ACCESS-TOKEN>.

Payload Compression​

JSON log data is highly compressible, often getting 8:1 or better compression ratios, especially on appropriately sized batches of log data. Unless client CPU usage is extremely limited, we recommend compressing the payload.

To indicate the payload is compressed, set the Content-Encoding HTTP header appropriately. Supported encodings include gzip, deflate, zlib, snappy, and lz4/lz4-block.

For gzip and deflate we recommend using compression level 3. If CPU usage is a concern, a compression level of 1 is very CPU efficient yet is still usually very effective on log data.

The snappy algorithm is even more CPU efficient while still achieving excellent compression ratios. Both the snappy stream and block formats are supported, and the right format is automatically detected.

Another option is the lz4 compression encoding. This is even more CPU efficient than snappy while having lower compression ratios. Both the stream (frame) and block formats are supported. If the data is in LZ4 block format, then you must also specify the X-Original-Content-Length header to indicate the uncompressed size of the data.

Response​

The success or failure of the request will be indicated by the HTTP status code in the response.

A request with 1 MB of data will usually take less than 400ms to process, but in rare cases may take up to 10 seconds. We recommend having a timeout of 90 seconds, and retrying requests after network failures or certain retryable HTTP error responses (e.g., 429, 500).

Error responses:​

  • HTTP 400: some (portion) of the input is invalid and cannot be processed; you should skip this payload (or add it to a dead letter queue) as retrying will not help
  • HTTP 401: agent credentials are invalid
  • HTTP 403: agent credentials were valid, but access was denied to the target cloud resources; check private cloud setup
  • HTTP 413: the payload is too large to be processed in one request; split the payload into smaller parts and retry
  • HTTP 429: the service is busy processing requests; wait and retry the request; this should only be sent if abusive behavior is detected or there is a temporary service issue; in rare cases, this is returned when an unexpected error occurred while reading the request body
  • HTTP 500: there is a temporary service issue; wait and retry the request

For error responses, the body of the response will be JSON with the form:

{"error": "some error message", "error_code": "error_reason_string"}

Success response:​

On success, HTTP 200 will be returned with a JSON body as follows:

{
    "status": "some success message",
    "elapsed_ms": n,     // number of milliseconds it took to ingest the data
    "count": n,          // number of ingested log events
    "billable_bytes": n, // billable bytes accrued for ingesting this data
    "final_event_t": n   // timestamp of the last event in the batch; can be used as prev_event_t when submitting next batch as appropriate (usually not needed)
}

Performance​

Clients that submit large amounts of data to ingest MUST batch multiple log entries together so that under high volume multiple log events are sent with each HTTP request.

When possible aim to achieve a batch size of around 1MiB. Batches larger than 25MiB will fail.

A common architecture is to batch log events together in a buffer, and to either flush the buffer when a target buffer size is reached or a reasonable "batch timeout" expires since the first or last event in the batch (e.g., 10 seconds). This ensures that log events are still submitted in near real time and that at high volume each request contains many log events.

Using HTTP connection keep-alive (or more advanced features like connection multiplexing in HTTP/2) are supported but not required. For good performance, the most important criteria is using properly sized batches.

Event Ordering​

To achieve very high throughput, you may need to submit multiple ingest requests in parallel. Proper ordering of events can be achieved by ensuring each log event contains enough data to place the order of the log event independent of the log events around it. This can be achieved through a couple of strategies:

  • Timestamps in log messages extracted by AutoExtract, with a fallback to an observedtimestamp (e.g., each log event's created_time or ingested time):
    • This is the recommended method if your log events contain a timestamp with each log line.
    • AutoExtract will detect and extract this timestamp and use it as the log timestamp of each message.
    • If such a timestamp is not found, it will then fallback to the structured field that is considered to be an "observed timestamp" (the time when the client first saw the log event).
    • Our configuration templates for most log forwarding agents are configured to use this strategy.
  • An explicit timestamp field submitted with each log event:
    • In this strategy, the timestamp of each log event is explicitly through a timestamp field submitted with each log event.

Using a strategy like this ensures that each event will be displayed in query results in the proper logical order, regardless of the order in which it was ingested into the system as part of parallel batches.

Note that popular log forwarding agents like Vector or Fluent Bit will follow the above strategies to ensure proper ordering even with parallel batch submission. Our configuration templates for these agents use the observedtimestamp strategy, although you are free to configure them differently as appropriate.

Payload Deduplication​

In situations where the cloud finished processing a request successfully, but the client never received the response (perhaps it timed out just at the wrong moment), the client will normally retry the request. In traditional systems, this can lead to duplicate log events being ingested.

To automatically mitigate this problem, SparkLogs deduplicates identical payloads that are at least 16 KB in size. If a duplicate payload is detected, an HTTP 200 response is immediately returned, and with an additional field deduplicated set to true in the response body.

Note that individual log events are not deduplicated, only entire payloads.

For advanced situations only, this behavior can be disabled by setting the X-No-Dedup header to true (not recommended).