Skip to main content

Timestamp Management

Assigning accurate timestamps to each log event is critical for usable observability data. Events with inaccurate timestamps may lead to confusion, with the events appearing out of order, or appearing in the wrong time period.

This can be further complicated when log data may be delayed for minutes, hours, days, or even months, and could be ingested long after the event occurred. Having an explicit strategy for how timestamps are assigned will ensure you understand what to expect under different scenarios.

How Timestamps Are Assigned​

SparkLogs provides flexibility in how timestamps are assigned to each event so you can ensure accurate timestamps for your data. There are several options:

  • Source Timestamps: Assign timestamps to each event at the source before sending data.
  • AutoExtract Timestamps: Allow AutoExtract to automatically detect timestamps in log messages and assign them to the event.
  • Hybrid: Attempt to AutoExtract the timestamp first, and if none is detected fall back to the timestamp assigned at the source by your log ingestion agent.

Our template and example configurations follow the hybrid approach as it usually does the right thing with the least configuration.

If a timestamp cannot be assigned through any method, then the time of ingestion is used as the event timestamp.

If two events are received with the same timestamp, they are assigned an integer event_index representing the order in which they were received within the ingestion batch. Query results will display events in timestamp order and then sorted by event_index, ensuring that events should appear in the proper order.

Source Timestamps​

In this method, you can specify the timestamp of each event using field values in the input data. In each source event populate a field that maps to the timestamp field (see mappings). Common fields include @timestamp (Elastic), timestamp, time, ts, t, instant, eventtime, or date.

The field value can use any of the 100+ supported timestamp formats.

AutoExtract Timestamps​

AutoExtract can detect a timestamp for an event in two ways:

  • As a date+time string in an unstructured part of the textual log message.
  • As a value in a structured field that is extracted from the textual log message through AutoExtract, including in embedded JSON, key/value pairs, and more (details).

For a date+time string in the message text to be used as the event timestamp, the following rules apply:

  • The timestamp must be in one of the 100+ supported timestamp formats.
  • If a timezone is not specified in the timestamp, it will use the default time zone indicated with the ingestion request (normally the timezone of the forwarding agent submitting the request).
  • The timestamp must be in the first line of the textual message and must be in the first 60 characters of the message.
  • The timestamp must be close to the current time (timestamps towards the front of the message may be up to 30 days in the past, other timestamps must be within 24 hours of the current time).

When AutoExtract detects a timestamp in the message text, it will assign the parsed value into the standard timestamp field and then also remove the parsed timestamp from the message text. (This makes it easier to read the message text without the timestamp cluttering it up.) You can customize this behavior by setting the X-No-Remove-Timestamp-From-Message HTTP request header to true.

To detect timestamp values in structured fields extracted from the message text, AutoExtract will look for a field with one of the field names supported in the standard field mappings (see above) and that has a recognized timestamp format. If such a field is found, it will be used for the event timestamp, and the structured field value will also be stored as structured data with the message with its parsed field name.

Hybrid Approach​

In the hybrid approach, AutoExtract will first attempt to detect a timestamp, and if none is found, it will use a timestamp from the source event. To use this approach, set an observed_timestamp field in each source event (see agent configuration examples in ingestion tools). Several different fields are supported from common industry schemas, including observedtimestamp, created, and ingested.

Ingesting Really Old Data​

If an event's timestamp is too far in the future (more than 24 hours) or too far in the past (more than 30 days), then the event timestamp will be set to the time of ingestion (ingested_timestamp) and the original timestamp will be stored into original_timestamp. In this case, when querying this data, you can set the time range of the query to be for when the data was ingested, and then use LQL query expressions to further constrain the original_timestamp field value (for example original_timestamp between 2025-08-20T04:00:00 and 2025-08-23T08:30:00).

Querying for Lagging Events​

When querying for lagging events, you can use the original_timestamp field to filter for events that

When lagging data is ingested consistently, SparkLogs will show the source for these lagging events on the monitoring dashboard. On this dashboard you can copy the details of lagging data source. From this detailed report use the agent_id and source values to form LQL queries to find the specific lagging events:

On the Explore page, set the query time window to be around the time of the "last seen" time of the lagging event (obtained from the dashboard), add LQL to filter the agent_id and source appropriately to match, and then add an LQL expression to filter for any event that has an adjusted timestamp: original_timestamp! (field exists operator !). This will produce all lagging events ingested within the given time window for that agent and source.

For example:

agent_id="658aa664-215e-4611-8e93-d2ffedb922ef" source=ELEGANTDOLPHIN original_timestamp!