Skip to main content

Paging and Sync

Large lists return a page at a time. The same calls can also return only what changed since your last run, for a nightly sync with a PSA for example.

Where paging is used​

Organizations, agents, ingest keys, registration tokens, members, invitations, groups and audit events page with a cursor. Roles, regions and replication targets are small and return everything in one response.

Page through a list​

A paged list takes limit. limit runs from 1 to 5,000, and a page holds 1,000 rows when you leave it out. Audit events are the exception: a page holds 500 by default. The response carries next_cursor. Pass that as cursor to get the next page, and stop when next_cursor is null.

curl "https://engine.sparklogs.app/app-api/workspaces/$WORKSPACE_ID/organizations?limit=5000" \
-H "Authorization: Bearer $SPARKLOGS_TOKEN"

Keep the same filters on every page. A cursor sent with different filters returns 400. limit and fields may change from page to page.

Each list's reference page states its order. Organizations, agents, ingest keys and registration tokens come in id order, and paging through a full list returns each row once. Audit events come newest first.

Fetch only what changed​

The lists of organizations and registration tokens take updated_since. With it, a run fetches only the rows that changed. No other list takes updated_since. For agents, filter the list by agent_status or collection_status instead, as Pull Agent Collection Health into a Dashboard shows. Every page of a list carries the same watermark. When you have read the last page, store the watermark. On the next run, pass the stored watermark as updated_since. The list returns the rows changed at or after that time, with a new watermark for the run after.

  1. First run: list with no updated_since, page to the end, store watermark.
  2. Every later run: list with updated_since set to the stored watermark, page to the end, store the new watermark.

The watermark is one minute earlier than the time the first page was read. The minute covers a change that was being saved at that moment. Each run with updated_since returns the changes from that time up to the new watermark, and the next run starts where this one ended. For example, a run at 02:00 returns a watermark of 01:59. The run at 03:00 passes it, gets the changes from 01:59 to 02:59, and stores 02:59.

A run with updated_since returns rows least recently changed first, by updated_at and then id. Apply each row by id, because a row can arrive more than once: a row that changes while you page can appear twice, and the first run with updated_since after a full list can return rows the full list already held.

An incremental sync has three limits:

  • Gaining access is not a change. When a token gains access to an organization, the organizations and registration tokens under it do not show up in the next sync.
  • Losing access leaves no marker. When a token loses access to an organization, its rows stop appearing. They are not marked as deleted.
  • Some changes do not move updated_at. A move higher in the tree changes the path of every organization, agent, ingest key and registration token below it without changing their updated_at. Deleting an organization is not a change to the registration tokens in it.

Because of these limits, run a full list (no updated_since) on a schedule, weekly for example, and treat it as the truth: replace what you stored with what it returns.

A sync in Python, storing its state between runs:

import os, requests

BASE = "https://engine.sparklogs.app/app-api"
HEADERS = {"Authorization": f"Bearer {os.environ['SPARKLOGS_TOKEN']}"}

def sync_organizations(workspace_id, state):
# state is {"watermark": None or a string, "organizations": {id: row}}
params = {"deleted": "include"}
full = not state["watermark"]
if not full:
params["updated_since"] = state["watermark"]
rows = {} if full else dict(state["organizations"])
while True:
response = requests.get(
f"{BASE}/workspaces/{workspace_id}/organizations",
headers=HEADERS, params=params, timeout=60)
response.raise_for_status()
page = response.json()
for row in page["organizations"]:
rows[row["id"]] = row # one row per id
if page["next_cursor"] is None:
break
params["cursor"] = page["next_cursor"] # same filters, plus the cursor
# replace the stored state only once the whole pass has succeeded
state["organizations"] = rows
state["watermark"] = page["watermark"]

The sample has no retries. See Retries for how to wrap the call. A failed pass leaves the stored state as it was, and the next run starts from the same watermark. Set state["watermark"] back to None to make the next run a full list.

Deleted objects​

Lists leave deleted objects out. Pass deleted=include to add them or deleted=only to list them alone. A deleted row has deleted_at set, and keeps appearing for as long as the token can reach it.

An agent also carries organization_deleted_at, set when its organization is deleted. deleted=only returns an agent when either field is set. A sync that should notice deletions passes deleted=include and reads both fields.

Choose fields​

A read takes fields, a comma-separated list of the top-level fields to return in place of the default set.

curl "https://engine.sparklogs.app/app-api/workspaces/$WORKSPACE_ID/organizations?fields=id,name,psa_client_id" \
-H "Authorization: Bearer $SPARKLOGS_TOKEN"
  • A field you did not select is absent from the response. A selected field with no value is null.
  • Some fields are sent only when named. Each call's reference lists them.
  • fields works together with updated_since, and the watermark comes back whatever you select.
  • Two fields on the agent reads are heavy: host_metadata and agent_data. Selecting one lowers the page size: limit can be at most 500, which is also its default.
  • A field that needs a further privilege returns 403, naming the privilege, when you select it without holding it.
  • A field or query parameter the call does not know returns 400. So does a parameter sent twice.