Skip to main content

Special Value Extraction

Overview​

Certain types of values are commonly found in unstructured log messages that are not associated with any key/value pair, but are still valuable to capture into a more structured format, for easy filtering.

AutoExtract will detect and extract timestamps (x.ts[]), IP addresses (x.ips[]), bracketed values (x.b[]), parenthetical values (x.p[]), as well as any dynamic values that AutoClassify determines should not be part of the pattern text (x.var[] and x.num[]).

Timestamps​

Valid timestamps that are detected and not associated with a named value will be added to the special x.ts[] array field.

Querying this array field using any LQL operator will then match if any of the values in the array match. For example, this LQL query will match any event that has at least one timestamp that is between the two dates.

x.ts BETWEEN "2023-01-01" and "2023-12-31 23:59:59"

It's also possible to query if a specific member of the array meets a condition (e.g., x.ts[2]: ...).

IP addresses​

Valid IPv4 addresses (e.g., 192.168.0.1) are extracted and stored into the special x.ips[] array field.

Then you can find all log messages that reference a certain IP address easily. For example:

x.ips: 192.168.0.150

Bracketed and parenthetical values​

It's a common pattern for contextual information to be formatted within square brackets and/or parentheses within log messages. For example, process/thread IDs, environment names, source code locations, etc. These values may not be attached to a key/value pair, but are still useful to extract.

To solve for this common pattern, AutoExtract takes any value found within square brackets and appends it to the x.b[] array field (or appenda to x.p[] for parenthetical values). For example:

[PRODUCTION] - [pid:4112] [tid:23] Failed to process request (main.py:23)

will become:

x:
  b:
    - "PRODUCTION"
    - "pid:4112"
    - "tid:23"
  p:
    - "main.py:23"

Continuing this example, you could then filter for all log messages that contained a pid or tid value of 4112:

x.b: ":4112"