Field Type Detection

Overview

A key design principle of SparkLogs is to avoid configuration whenever possible. Rather than asking you to manually setup custom fields (specifying name and data type), the existence of data that contains a given field value is what defines that custom field.

Additionally, when data is ingested, each field value is analyzed, and a type is assigned.

Possible field types are:

• text: Stores strings of arbitrary length
• numeric: Floating point or integer numbers
• boolean: Yes/No or True/False values
• timestamp: Date/time with a timezone offset
• severity: A severity level from 1-24 that reflects the level of attention for an event (SparkLogs severity ladder; numeric scale inspired by OpenTelemetry SeverityNumber)
• facility: A facility level from 0-24 as defined in the syslog specification

Tip: Fields with the same name but different types

It's possible to have data across different log events for the same field name that will result in different types. The system will handle the situation gracefully, allowing you to query data any of these fields. Even though they have the same name, the type name is also part of the field name through a special type suffix (e.g., #s or #n[]). In LQL queries you can refer to fields with or without the special type suffix.

Numeric Fields

Numeric fields are stored as 64-bit floating point numbers. If you want to treat a numeric field as a 64-bit integer field in your query expressions, you can append the special #i type suffix, which will force a conversion to a 64-bit integer type during the query process.

Boolean Fields

Boolean fields are either true or false. AutoExtract will convert a field with a value of either true or false (case insensitive) to a boolean field.

Timestamp Fields

Date/time information is formatted in various ways. Our goal is to automatically detect and parse as many valid date/time formats as possible.

Extracted information will detect and treat valid date/time values that have spaces as one value, even if the date/time value is not quoted. This ensures that log text like finished=October 20, 2023 6:06pm will produce a timestamp value finished: 2023-10-20T18:06:00, rather than just finished: October.

If a timezone is not specified in the value itself, the vector.dev logging agent will specify a default timezone to use (defaults to the timezone of the machine running the vector.dev agent).

Over 100 timestamp formats are supported, including many variants of the following common formats:

2022-06-13T07:59:49Z              RFC3339
2022-06-13T07:59:49.123456 EST    RFC3339 with microseconds
02/Jan/2006:15:04:05 -0700        Apache
Sun Dec 04 04:47:44 2005          Apache
17/06/09 20:10:40                 Spark
2013-Feb-03                       Date only
Thu, 4 Jan 2018 17:53:36 +0000    Weekday with date/time and TZ offset
September 17, 2012 10:09pm        Long date with am/pm
1332151919                        UNIX timestamp in seconds
1384216367111222                  UNIX timestamp in microseconds
2014年04月08日                     Chinese

Refer to the extended examples of the excellent dateparse golang package for a full list.

Digit strings in the timestamp field

In the event's timestamp field, and in a field forced to a timestamp with #t, a string made only of digits is read by its shape:

• 4, 8 or 14 digits spell a date: 2022, 20220613 and 20220613075949. A 14-digit date can carry fractional seconds, such as 20220613075949.5. A 14-digit string that is not a valid date, such as 00001655107189, is not read.
• 4 digits, a dot and 1 or 2 digits are a year and month: 2022.05 is May 2022.
• Any other string of digits is a UNIX epoch time, read in the unit its value implies (seconds through nanoseconds), the same way as the JSON number with those digits. A fraction is kept down to microseconds: 1655107189.5 is half a second after 1655107189.
• A JSON number is always read as a UNIX epoch time, never as a date.

Known limits:

• In a query, a string of digits compared to a timestamp is read by its number of digits, so "0001655107189" there is not the same time as it is in the timestamp field.
• In a custom field, a string of digits in square brackets, such as "[1655107189]", can be detected as a timestamp.

Unix epoch field-name suffixes

A field name that ends with a known (case insensitive) suffix names a UNIX epoch unit: seconds, milliseconds, microseconds or nanoseconds. A number, or a string of digits, under such a name is read in that unit, with an optional fraction such as 1655107189.5. If the value is a plausible time in that unit, from 1973 to the year 4821 (2540 for nanoseconds), it is stored as a timestamp and the suffix is removed from the stored field name. Otherwise, for example a count of seconds under a milliseconds suffix, the value is detected as usual and the field keeps its full name. Any other string, such as 2022-06-13T07:59:49Z or a number with a leading dot or square brackets, is read the way the event's timestamp field reads it, and the suffix is removed when it is a time. A value that is not read as a time this way is never detected as a timestamp: it stays a number or text under its full field name. The OpenTelemetry event time fields time_unix_nano, timeUnixNano, observed_time_unix_nano and observedTimeUnixNano are an exception: as the event's own time, they accept a UNIX epoch time in any unit, judged by its size.

For example, in OpenTelemetry, the fieldname observedTimeUnixNano will be recognized as a timestamp if the value is a valid UNIX epoch timestamp in nanoseconds. The normalized time value is then stored to the field observedTime.

Snake-style suffixes use a leading underscore (a bare segment like unix_secs without _ is not treated as a hint):

• _unix_sec, _unix_secs, _unix_milli, _unix_millis, _unix_ms, _unix_micro, _unix_micros, _unix_us, _unix_nano, _unix_nanos, _unix_ns

Camel-style suffixes are fused trailing segments without a leading underscore, for example field names such as timeUnixNano or observedUnixMilli:

• unixsec, unixsecs, unixmilli, unixmillis, unixms, unixmicro, unixmicros, unixus, unixnano, unixnanos, unixns

If the suffix is the entire field name, the type hint still applies but the name does not change. If removing the suffix would conflict with a pre-existing field or a top-level name reserved by SparkLogs, the new field name is uniquified (for example time becomes time2).

Forcing a Type with a Field-Name Suffix

When you know a field's type, end its name with a type code. SparkLogs then reads the value only as that type and removes the code from the stored field name. The codes are the same ones you use in LQL queries.

Suffix	Stored as	Accepted values
#s	text	Any string. A number keeps its exact text, and a boolean becomes true or false.
#n	numeric	A number, or a string that holds a number, as numeric fields are detected.
#b	boolean	true or false, as a boolean or a string (case insensitive), as boolean fields are detected.
#t	timestamp	Any value the event's own timestamp field accepts (see timestamp fields and digit strings).
#i	text of digits	A string or number made only of the digits 0-9, of any length. Add #i to the field in a query to read it as a 64-bit integer; a value outside that range reads as null.

For example, this event:

{"system_info": {"bios_version#s": 7.40, "os_build#i": "26100", "bios_date#t": "2025-11-02"}}

is stored as system_info.bios_version with the text 7.40, system_info.os_build with the text 26100, and system_info.bios_date as a timestamp. Without the suffix, bios_version would be detected as the number 7.4.

• The suffix works at any depth, including fields of objects inside arrays. A flat dotted name such as system_info.bios_version#s is nested like any other dotted name.
• A value that cannot take the forced type, such as "abc" under #n, is stored as null under the name without the suffix. A null value stays null. If a field you suffixed shows nulls, check the value your sender produces.
• The suffix applies to single values. An object or array under a suffixed name keeps its full name, and its contents are detected as usual.
• Only the five lowercase codes at the very end of a name are recognized, and only the last one is removed: a#s#n is stored as the numeric field a#s. Names such as a#x, a#S or a#s.b keep their names and are detected as usual.
• Name conflicts follow the UNIX epoch suffix rules above: a suffix that is the entire field name keeps the name, and a stripped name that is already taken or is a top-level name reserved by SparkLogs (a standard field name, client_ip_location, an override such as category_override, or an internal field name such as __redacted) is uniquified (for example service#s becomes service2).
• A sender whose field names already end in one of these codes gets the new behavior: the field is stored under the shorter name with the forced type. Rename such a field to keep it detected as before.

Severity

Severity grades how much attention an event warrants: everyday chatter stays low; failures that need a ticket or always-surface investigation sit higher. SparkLogs uses a numeric scale from 1-24 (inspired by OpenTelemetry) and defines its own authoritative ladder of named bands with a carefully crafted meaning for each. Only the standard field severity can have the severity field type.

The SparkLogs Agent has deep data curation logic so event severity is a measure of how much attention a given event warrants. Third-party and uncurated logs still normalize through all the common name aliases (see below).

Name	Int	Meaning
Trace	1	Retained chatter
Debug	5	Diagnostic / down-capped forensic
Verbose	7	Extra detail below everyday Info
Info	9	Everyday operation
Display	10	Worth surfacing in UI, still everyday
Notice	11	Notable-normal
Warning	13	Degraded or failed-then-recovered
Minor	16	Real failure, weak blast / high flake
Error	17	Failed, unrecovered, bounded
Serious	18	Host- or workload-ticket grade
Severe	19	Infra / multi-workload threat
Critical	20	Always-surface demarcation
Fatal	21	Lost / no self-recovery
Emergency	24	System-unusable class

In addition to the severity names in the OpenTelemetry specification, many other common severity names are valid and mapped to appropriate severity levels internally. For example, all syslog severity level names are accepted. Common abbreviations (e.g., err or warn) are also accepted.

LQL query expressions also recognize valid severity values when filtering against the severity field. The following are example LQL query expressions showing this in action:

LQL

severity >= critical
severity in (serious, severe)
severity >= warning