# The SparkLogs Platform

**SparkLogs** is a cloud-first log management platform that is limitless (petabyte-scale), easy (schemaless + auto-extract), affordable (ingest everything), and a joy to use (fast, interactive, and modern UX).
It is the platform underneath [IT fleet intelligence](https://sparklogs.com/docs/it-fleet-intelligence.md): the same engine that lets an AI cite fleet evidence also makes logs at any scale a pleasure to work with directly.

## Why we built it

SparkLogs was born out of a desire to solve the extreme pain of managing legacy observability systems at scale:

- Baby-sitting our own elastic clusters with 100+ TBs of logs
- Watching elastic clusters break during unexpected app load spikes, and then not having logs to discover why
- Grepping in-place logs simultaneously across countless machines
- Shipping files to NFS and manually grepping through it
- Waiting forever to visualize queries with **billions** of hits
- Paying a small fortune every month for a cloud log service (and then having to aggressively filter out data and reduce retention to keep costs down)
- Wrestling with field configs and manual parsing rules

## The data engine

We designed SparkLogs to eliminate these pain points:

- Ingestion is serverless and handles extreme volume bursts (100x baseline) with zero baby-sitting or configuration.
- There are no clusters, shards or nodes to size, patch or rebalance.
- Logs from every machine land in one place, so nobody greps hosts one at a time.
- The schemaless design fully supports unstructured and semi-structured events (freeform text and JSON data), with infinite custom fields and no cardinality limits.
- All signals are captured in a single platform with rich context, supported by [AutoExtract](https://sparklogs.com/docs/ingest/autoextract.md) to extract structured fields from unstructured text.
- 10x lower costs allow you to ingest everything, enrich with context, retain for longer, and query across all of it.
- The rich interactive UX and [Massive-Scale Adaptive Querying engine](https://sparklogs.com/docs/explore/adaptive-scale-querying.md) make it easy to do needle-in-haystack or broad queries across 100s of billions of events in seconds.
- Current and historical data is stored together with no extra platform costs for long-term retention and querying, consolidating your archive tier into one platform.

## Design Principles

### Ingestion should be "point and shoot"

- **Schemaless**:
  - Capture arbitrarily complex JSON data with each log event.
  - Fields don't have to be configured, just send data.
  - Infinite custom fields.
- **AutoExtract**: Stop wasting time manually configuring fields and complex parsing rules!
  - Our engine [automatically extracts](https://sparklogs.com/docs/ingest/autoextract.md) semi-structured (e.g., `key=value`) and JSON data embedded in your text log messages into custom fields.
  - IP addresses, timestamps, and bracketed values are automatically extracted and collected into custom arrays (`x.ips[]`, `x.ts[]`, `x.b[]`), for easy filtering later.
  - Field types (numeric, timestamp) are automatically detected based on content.
  - Automatic GeoIP lookups and foreign currency value conversion.
  - Instantly try it out in the [AutoExtract simulator](https://sparklogs.com/docs/ingest/autoextract/try-it-out.md)!
- **AutoClassify**: Automatic [categorization](https://sparklogs.com/docs/ingest/autoclassify.md) of your log messages, unlocking insights in patterns in your logs.
- **Minimal Configuration**: _Convention `>>` Configuration!_ A system should be smart and adapt to the way you work.
  - Standard fields (timestamp, severity, facility, source, app, message) are **automatically** detected and mapped from industry-standard log schemas: syslog, OpenTelemetry, Elastic Common Schema, vector, Windows Event Log, AWS CloudTrail, Google Cloud Logging, HEC (Splunk), zap, and log4j.
- **OpenTelemetry-Native**: First-class [OTLP/HTTP](https://sparklogs.com/docs/ingest/tools/otlp-http.md) ingestion means your applications ship logs **directly** to SparkLogs through the [OpenTelemetry SDK](https://sparklogs.com/docs/ingest/data-sources/languages/opentelemetry-sdks.md), paired with the logging library you already use: no vendor adapter, no custom transport.
  - Step-by-step setup guides for [Python](https://sparklogs.com/docs/ingest/data-sources/languages/python.md), [Node.js](https://sparklogs.com/docs/ingest/data-sources/languages/nodejs.md), [Java](https://sparklogs.com/docs/ingest/data-sources/languages/java.md), [.NET](https://sparklogs.com/docs/ingest/data-sources/languages/dotnet.md), [Go](https://sparklogs.com/docs/ingest/data-sources/languages/go.md), [Ruby](https://sparklogs.com/docs/ingest/data-sources/languages/ruby.md), [PHP](https://sparklogs.com/docs/ingest/data-sources/languages/php.md), and [Rust](https://sparklogs.com/docs/ingest/data-sources/languages/rust.md): batched, compressed, and production-ready by default.
  - Pivot fields (`source`, `service`, `app`) are auto-derived from your OTel `Resource` attributes, with no custom mappings to maintain.
  - Works with the [OpenTelemetry Collector](https://sparklogs.com/docs/ingest/tools/opentelemetry-collector.md), [Grafana Alloy](https://sparklogs.com/docs/ingest/tools/alloy.md), and any OTLP-compliant shipper.
  - Vendor-neutral by design: ship the same OTLP stream to SparkLogs today and to anything else tomorrow; your instrumentation stays portable.
- **Open**: Capture logs using your preferred agent, such as [Vector](https://sparklogs.com/docs/ingest/tools/vector.md), [Fluent Bit](https://sparklogs.com/docs/ingest/tools/fluentbit.md), [OpenTelemetry Collector](https://sparklogs.com/docs/ingest/tools/opentelemetry-collector.md), or any agent that can ship logs to an HTTPS endpoint.
  - Dozens of sources are available, including [files](https://vector.dev/docs/reference/configuration/sources/file/), [Kubernetes](https://vector.dev/docs/reference/configuration/sources/kubernetes_logs/), [journald](https://vector.dev/docs/reference/configuration/sources/journald/), [syslog](https://vector.dev/docs/reference/configuration/sources/syslog/), [Kafka](https://vector.dev/docs/reference/configuration/sources/kafka/), [Docker](https://vector.dev/docs/reference/configuration/sources/docker_logs/), [AWS SQS](https://vector.dev/docs/reference/configuration/sources/aws_sqs/), [GCP PubSub](https://vector.dev/docs/reference/configuration/sources/gcp_pubsub/), and [more](https://vector.dev/docs/reference/configuration/sources/).
  - Network syslog is easy to collect by forwarding through a [Vector](https://sparklogs.com/docs/ingest/tools/vector.md) or [Fluent Bit](https://sparklogs.com/docs/ingest/tools/fluentbit.md) agent.
- **Scalable**: Cloud-first in every way, our serverless design combines with the power of BigQuery to deliver lightning fast, petabyte-scale observability at an [unbelievably low cost](https://sparklogs.com/pricing).
- **Hierarchical**: Organize data captured from different environments (dev, QA, staging, production), geographic region, or different organizational units (business units, MSP clients) into hierarchies (e.g., `production/app1/europe`).
  - Enforce least-privilege access by restricting users' scope of access based on hierarchy.
  - Hierarchies can be rearranged at any time to fit your needs.

### Log search and analysis should be a joy

- **Robust**: Use [LQL](https://sparklogs.com/docs/explore/lightning-query-language.md), the Lightning Query Language, for SQL-like, type-aware data querying.
- **Visual**: Interactive histograms to effortlessly explore patterns across billions of events.
- **Powerful**: Navigate and browse huge result sets, even with billions of matches.
- **Fast**: Local 'zoom-in', filtering, search, and export for snappy investigations.
- **Modern**: A clean, realtime UX that works for you.
- **Anywhere, Anytime**: Access via the web, or native apps for MacOS, Windows, Linux, iOS, and Android.
- **Scalable**: Built on BigQuery with accelerated indexes: query petabytes in seconds!

### Enterprise ready

- **Role-based Access Control**: Enforce least-privilege access, and give each user or group only the scope of data and level of access they should have.
  A person or group can hold different roles on different parts of the organization tree, such as Administrator at one client and Analyst at another.
  See [Users, Roles and Groups](https://sparklogs.com/docs/workspace/users-and-roles.md).
- **Data Encryption**: All data is encrypted in-transit and [at rest](https://cloud.google.com/docs/security/encryption/default-encryption).
- **Data Security**: Our cloud operates on [Google-managed services](https://cloud.google.com/trust-center) within an [enterprise-foundations](https://cloud.google.com/architecture/security-foundations) deployment of the Google Cloud; our serverless design means that all infrastructure is continually updated and secured by Google; all changes to production are managed by IaC through a secured workflow. See the [Trust Center](https://sparklogs.com/security) for complete details.
- **Private Cloud**: Optionally store/process data in your own Google Cloud tenant for greater control, compliance, and cost savings.
- **Archiving and Replication**: Replicate all your data to your own cloud storage bucket (AWS S3, GCS, Azure Blob Storage, or any S3-compatible storage) for compliance, disaster recovery, or geographic distribution. Archived data is stored in a ready-to-query parquet data lake format. [Learn more](https://sparklogs.com/docs/archiving-replication.md).
- **SSO**: (coming soon) Option to sign in using Azure AD or Google credentials, standard in every plan.

---

Source: <https://sparklogs.com/docs/platform>
All SparkLogs documentation in Markdown: <https://sparklogs.com/llms.txt>
