JSON Unwrapping

Concept

Certain logs may consist of data that is already structured in JSON format. Depending on how you capture and forward your logs to SparkLogs, this JSON may be sent in the message (text) field of the log message.

If the entire log text of a given event is a single JSON event, AutoExtract will "unwrap" the JSON in the log text and "promote" the fields in the JSON data as if they were part of the root log event. AutoExtract and AutoClassify then proceed as normal, first detecting which fields map to standard fields, then extracting custom fields and assigning a pattern.

JSON Unwrapping is automatically on and does not require configuration.

Example

For example, the Node.js bunyan logging library takes a single log message like Finished request target=/docs/example response=200 and generates a text log line that is a single JSON object like:

{"name":"myapp", "hostname":"myhost.internal", "pid":77, "msg":"Finished request target=/docs/example response=200"}

Without JSON Unwrapping

The captured log event is:

YAML

source: "myhost.internal"
message: '{"name":"myapp", "hostname":"myhost.internal", "pid":77, "msg":"Finished request target=/docs/example response=200"}'
x:
  name: "myapp"
  pid: 77
  msg: "Finished request target=/docs/example response=200"

JSON

{
  "source": "myhost.internal",
  "message": "{\"name\":\"myapp\", \"hostname\":\"myhost.internal\", \"pid\":77, \"msg\":\"Finished request target=/docs/example response=200\"}"
  "x": {
    "name": "myapp",
    "pid": 77,
    "msg": "Finished request target=/docs/example response=200"
  }
}

With JSON Unwrapping

The captured log event is:

YAML

source: "myhost.internal"
message: "Finished request target=/docs/example response=200"
pattern_hash: "Fr_458om"
pattern: "Finished request"
name: "myapp"
pid: 77
x:
  target: "/docs/example"
  resposne: 200

JSON

{
  "source": "myhost.internal",
  "message": "Finished request target=/docs/example response=200",
  "pattern_hash": "Fr_458om",
  "pattern": "Finished request",
  "name": "myapp",
  "pid": 77,
  "x": {
    "target": "/docs/example",
    "response": 200
  }
}

Things to note:

• The message text inside the JSON event (msg field) is automatically promoted to be the message text, rather than the message text being raw JSON.
• AutoClassify is able to automatically assign a pattern to the log event.
• Fields such as name and pid are promoted to root-level fields, rather than being treated as AutoExtract fields underneath the x field.
• With the log event message now properly set, AutoExtract is able to extract the target and response fields.

Disabling

If required, this behavior can be disabled by setting the X-No-Unwrap-Raw-Json-In-Message HTTP header to 1.