# Paging and Sync

Large lists return a page at a time.
The same calls can also return only what changed since your last run, for a nightly sync with a PSA for example.

## Where paging is used

Organizations, agents, ingest keys, registration tokens, members, invitations, groups and audit events page with a cursor.
Roles, regions and replication targets are small and return everything in one response.

## Page through a list

A paged list takes `limit`.
`limit` runs from 1 to 5,000, and a page holds 1,000 rows when you leave it out.
Audit events are the exception: a page holds 500 by default.
The response carries `next_cursor`.
Pass that as `cursor` to get the next page, and stop when `next_cursor` is null.

```bash
curl "https://engine.sparklogs.app/app-api/workspaces/$WORKSPACE_ID/organizations?limit=5000" \
  -H "Authorization: Bearer $SPARKLOGS_TOKEN"
```

Keep the same filters on every page.
A cursor sent with different filters returns 400.
`limit` and `fields` may change from page to page.

Each list's reference page states its order.
Organizations, agents, ingest keys and registration tokens come in `id` order, and paging through a full list returns each row once.
Audit events come newest first.

## Fetch only what changed

The lists of organizations and registration tokens take `updated_since`. With it, a run fetches only the rows that changed.
No other list takes `updated_since`.
For agents, filter the list by `agent_status` or `collection_status` instead, as [Pull Agent Collection Health into a Dashboard](https://sparklogs.com/docs/api/use-cases/agent-collection-health.md) shows.
Every page of a list carries the same `watermark`.
When you have read the last page, store the watermark.
On the next run, pass the stored watermark as `updated_since`.
The list returns the rows changed at or after that time, with a new watermark for the run after.

1. First run: list with no `updated_since`, page to the end, store `watermark`.
2. Every later run: list with `updated_since` set to the stored watermark, page to the end, store the new `watermark`.

The watermark is one minute earlier than the time the first page was read. The minute covers a change that was being saved at that moment.
Each run with `updated_since` returns the changes from that time up to the new watermark, and the next run starts where this one ended.
For example, a run at 02:00 returns a watermark of 01:59. The run at 03:00 passes it, gets the changes from 01:59 to 02:59, and stores 02:59.

A run with `updated_since` returns rows least recently changed first, by `updated_at` and then `id`.
Apply each row by `id`, because a row can arrive more than once: a row that changes while you page can appear twice, and the first run with `updated_since` after a full list can return rows the full list already held.

An incremental sync has three limits:

- **Gaining access is not a change.** When a token gains access to an organization, the organizations and registration tokens under it do not show up in the next sync.
- **Losing access leaves no marker.** When a token loses access to an organization, its rows stop appearing. They are not marked as deleted.
- **Some changes do not move `updated_at`.** A move higher in the tree changes the `path` of every organization, agent, ingest key and registration token below it without changing their `updated_at`. Deleting an organization is not a change to the registration tokens in it.

Because of these limits, run a full list (no `updated_since`) on a schedule, weekly for example, and treat it as the truth: replace what you stored with what it returns.

A sync in Python, storing its state between runs:

```python
import os, requests

BASE = "https://engine.sparklogs.app/app-api"
HEADERS = {"Authorization": f"Bearer {os.environ['SPARKLOGS_TOKEN']}"}

def sync_organizations(workspace_id, state):
    # state is {"watermark": None or a string, "organizations": {id: row}}
    params = {"deleted": "include"}
    full = not state["watermark"]
    if not full:
        params["updated_since"] = state["watermark"]
    rows = {} if full else dict(state["organizations"])
    while True:
        response = requests.get(
            f"{BASE}/workspaces/{workspace_id}/organizations",
            headers=HEADERS, params=params, timeout=60)
        response.raise_for_status()
        page = response.json()
        for row in page["organizations"]:
            rows[row["id"]] = row  # one row per id
        if page["next_cursor"] is None:
            break
        params["cursor"] = page["next_cursor"]  # same filters, plus the cursor
    # replace the stored state only once the whole pass has succeeded
    state["organizations"] = rows
    state["watermark"] = page["watermark"]
```

The sample has no retries. See [Retries](https://sparklogs.com/docs/api/retries.md) for how to wrap the call.
A failed pass leaves the stored state as it was, and the next run starts from the same watermark.
Set `state["watermark"]` back to `None` to make the next run a full list.

## Deleted objects

Lists leave deleted objects out.
Pass `deleted=include` to add them or `deleted=only` to list them alone.
A deleted row has `deleted_at` set, and keeps appearing for as long as the token can reach it.

An agent also carries `organization_deleted_at`, set when its organization is deleted.
`deleted=only` returns an agent when either field is set.
A sync that should notice deletions passes `deleted=include` and reads both fields.

## Choose fields

A read takes `fields`, a comma-separated list of the top-level fields to return in place of the default set.

```bash
curl "https://engine.sparklogs.app/app-api/workspaces/$WORKSPACE_ID/organizations?fields=id,name,psa_client_id" \
  -H "Authorization: Bearer $SPARKLOGS_TOKEN"
```

- A field you did not select is absent from the response. A selected field with no value is null.
- Some fields are sent only when named. Each call's reference lists them.
- `fields` works together with `updated_since`, and the `watermark` comes back whatever you select.
- Two fields on the agent reads are heavy: `host_metadata` and `agent_data`. Selecting one lowers the page size: `limit` can be at most 500, which is also its default.
- A field that needs a further privilege returns 403, naming the privilege, when you select it without holding it.
- A field or query parameter the call does not know returns 400. So does a parameter sent twice.

---

Source: <https://sparklogs.com/docs/api/paging-and-sync>
All SparkLogs documentation in Markdown: <https://sparklogs.com/llms.txt>
