Skip to main content

The SparkLogs Platform

SparkLogs is a cloud-first log management platform that is limitless (petabyte-scale), easy (schemaless + auto-extract), affordable (ingest everything), and a joy to use (fast, interactive, and modern UX). It is the platform underneath IT fleet intelligence: the same engine that lets an AI cite fleet evidence also makes logs at any scale a pleasure to work with directly.

Why we built it​

Isn't it sad that log aggregation, search, and visualization isn't a solved problem in 2026!?! Taming logs is a chore!

Over the years, we've tried it all:

  • Baby-sitting our own elastic clusters with 100+ TBs of logs
  • Watching elastic break during unexpected app load spikes, and then not having logs to discover why 😡
  • Grepping in-place logs simultaneously across countless machines
  • Shipping files to NFS and manually grepping through it
  • Waiting forever ⏳ to visualize queries with billions of hits
  • Paying a small fortune every month for a cloud log service (and then having to aggressively filter out data and reduce retention to keep costs down)
  • Wrestling with field configs and manual parsing rules

Sound familiar? We built SparkLogs so none of it is your problem again.

Observability 2.0​

SparkLogs is built from scratch for the "Observability 2.0" paradigm:

  • The schemaless design fully supports "wide events" with infinite custom fields and no cardinality limits.
  • All signals are captured in a single platform with rich context, supported by AutoExtract to extract structured fields from unstructured text.
  • 10x lower costs allow you to ingest everything, enrich with context, retain for longer, and query across all of it.
  • The rich interactive UX and Massive-Scale Adaptive Querying engine make it easy to do needle-in-haystack or broad queries across 100s of billions of events in seconds.
  • Current and historical data is stored together with no extra platform costs for long-term retention and querying, consolidating your archive tier into one platform.

Design Principles​

Ingestion should be "point and shoot"​

  • Schemaless:
    • Capture arbitrarily complex JSON data with each log event.
    • Fields don't have to be configured, just send data.
    • Infinite custom fields.
  • AutoExtract: Stop wasting time manually configuring fields and complex parsing rules!
    • Our engine automatically extracts semi-structured (e.g., key=value) and JSON data embedded in your text log messages into custom fields.
    • IP addresses, timestamps, and bracketed values are automatically extracted and collected into custom arrays (x.ips[], x.ts[], x.b[]), for easy filtering later.
    • Field types (numeric, timestamp) are automatically detected based on content.
    • Automatic GeoIP lookups and foreign currency value conversion.
    • Instantly try it out in the AutoExtract simulator!
  • AutoClassify: Automatic categorization of your log messages, unlocking insights in patterns in your logs.
  • Minimal Configuration: Convention >> Configuration! A system should be smart and adapt to the way you work.
    • Standard fields (timestamp, severity, facility, source, app, message) are automatically detected and mapped from industry-standard log schemas: syslog, OpenTelemetry, Elastic Common Schema, vector, Windows Event Log, AWS CloudTrail, Google Cloud Logging, HEC (Splunk), zap, and log4j.
  • OpenTelemetry-Native: First-class OTLP/HTTP ingestion means your applications ship logs directly to SparkLogs through the OpenTelemetry SDK, paired with the logging library you already use: no vendor adapter, no custom transport.
    • Step-by-step setup guides for Python, Node.js, Java, .NET, Go, Ruby, PHP, and Rust: batched, compressed, and production-ready by default.
    • Pivot fields (source, service, app) are auto-derived from your OTel Resource attributes, with no custom mappings to maintain.
    • Works with the OpenTelemetry Collector, Grafana Alloy, and any OTLP-compliant shipper.
    • Vendor-neutral by design: ship the same OTLP stream to SparkLogs today and to anything else tomorrow; your instrumentation stays portable.
  • Open: Capture logs using your preferred agent, such as Vector, Fluent Bit, OpenTelemetry Collector, or any agent that can ship logs to an HTTPS endpoint.
  • Scalable: Cloud-first in every way, our serverless design combines with the power of BigQuery to deliver lightning fast, petabyte-scale observability at an unbelievably low cost.
  • Hierarchical: Organize data captured from different environments (dev, QA, staging, production), geographic region, or different organizational units (business units, MSP clients) into hierarchies (e.g., production/app1/europe).
    • Enforce least-privilege access by restricting users' scope of access based on hierarchy.
    • Hierarchies can be rearranged at any time to fit your needs.

Log search and analysis should be a joy​

  • Robust: Use LQL, the Lightning Query Language, for SQL-like, type-aware data querying.
  • Visual: Interactive histograms to effortlessly explore patterns across billions of events.
  • Powerful: Navigate and browse huge result sets, even with billions of matches.
  • Fast: Local 'zoom-in', filtering, search, and export for snappy investigations.
  • Modern: A clean, realtime UX that works for you.
  • Anywhere, Anytime: Access via the web, or native apps for MacOS, Windows, Linux, iOS, and Android.
  • Scalable: Built on BigQuery with accelerated indexes: query petabytes in seconds!

Enterprise ready​

  • Role-based Access Control: Enforce least-privilege access, and give different users access to only the scope of data and level of access they should have.
  • Data Encryption: All data is encrypted in-transit and at rest.
  • Data Security: Our cloud operates on Google-managed services within an enterprise-foundations deployment of the Google Cloud; our serverless design means that all infrastructure is continually updated and secured by Google; all changes to production are managed by IaC through a secured workflow. See the Trust Center for complete details.
  • Private Cloud: Optionally store/process data in your own Google Cloud tenant for greater control, compliance, and cost savings.
  • Archiving and Replication: Replicate all your data to your own cloud storage bucket (AWS S3, GCS, Azure Blob Storage, or any S3-compatible storage) for compliance, disaster recovery, or geographic distribution. Archived data is stored in a ready-to-query parquet data lake format. Learn more.
  • SSO: 🚧🚧 Coming Soon! 🚧🚧 Option to login using Azure AD or Google credentials, standard in every plan.