Overview

Motivation

SparkLogs is schemaless, so you can ingest structured data (any JSON) with unlimited custom fields.

In general, structured log data makes it easier to search/filter your logs, as you can set conditions on certain fields, and operators can be type-aware (e.g., numeric or timestamp comparisons for before or after a certain value).

However, configuring and maintaining parsing rules can be tedious and error-prone. SparkLogs makes it easy to have the best of both worlds.

Concept: Convention > Configuration

AutoExtract extracts structured data from unstructured or semi-structured log data, and doesn't require any configuration. AutoExtract often extracts the most relevant structured data from common log formats and is tuned for conventions that developers often follow when formatting their log messages (e.g., key/value pairs, logfmt, or embedded JSON values).

By learning the AutoExtract conventions, you can ensure your log data will become structured, and thus easier to search/filter/visualize.

AutoExtract automatically detects the data type of extracted fields (numeric, boolean, timestamp, or string), allowing you to avoid any setup work for custom fields.

AutoExtract also makes it easy to ingest fully structured data serialized in your text log data. It will extract detected JSON values, which can appear multiple times and can appear anywhere in your log message. You can mix in JSON data with key/value pair format (see example).