Retries
Networks drop connections, and your automation has to decide whether to send a call again. The answer depends on whether the call changes anything, and on how it failed.
The API has no idempotency keys (a value sent with a request that lets a server recognize a repeat). It treats a retry as a second request. The rules below keep a retry from making the same change twice.
Reads
A read changes nothing. Send it again after any 429, 500, 503 or dropped connection.
For 429 and 503, wait the seconds in Retry-After first.
Every GET is a read.
So is one POST call: checking a client id for duplicates.
Writes the API refused
A write that returned 400, 401, 403, 404, 409, 413, 429 or 503 changed nothing. Fix the cause, or wait, and send it again.
This is true only when the response body is the API's own JSON, with an error_code.
A 503 or other 5xx with any other body did not come from the API. The write may or may not have happened: treat it as a dropped connection.
Writes with an unknown outcome
After a dropped connection or a 500, the write may or may not have taken effect. Read before you retry:
- Read the object the write was aimed at. For a create, list with a filter on the name you chose:
nameon organizations, groups and ingest keys,labelon registration tokens,emailon invitations. Names are not unique, so check every row the list returns. - If the change is there, continue. If not, send the write again.
For example, your automation asks to create the organization for PSA company 10057 and the connection drops.
It lists organizations with psa_client_id=10057.
One row means the create worked: use that id.
No rows means it did not: send the create again.
Every create in the example use cases follows a lookup on a value you chose, for this reason.
The lookup works only with a value that identifies one organization. Names are not unique. Two sites created with the same name and nothing else cannot be told apart. Give every organization you create a client id, even a site, and the lookup is exact.
Some calls behave differently:
| Call | After an unclear failure |
|---|---|
| Create a registration token or an ingest key | Do not create a second one. Find the object and reveal its secret, which can be read again at any time. |
| Replace an ingest key's secret | Do not rotate again: that would replace the secret a second time. Read the key's secret instead. |
| Rotate the token you called with | Never retry. If the response was lost, the token is locked out: create a new one in the app. |
| Send an invitation email | The email may already have gone. A retry sends another copy. |
| Create a group | If your retry returns 409 for a taken name, the first attempt may have succeeded and taken the name. List the groups to check. |
| Delete a grant, a group or a member | A 404 on your retry means the first attempt worked, as long as the id was right. These deletes cannot be undone. |
Writes that are safe to repeat
Some writes set a value and do not add one. Renaming a group to the name it already has, replacing a grant with the same role, or deleting something already deleted ends in the same state. Each call's reference page states whether the call is safe to repeat.
After a 409, read the object again
409 means the object changed since you read it, a name or email is taken, or the object's state does not allow the call, such as deleting an organization that still has agents. Your write was not applied. Read the object again and decide whether your change still makes sense before sending it.
A PowerShell function with the rules built in
Invoke-RestMethod throws on any 4xx or 5xx response. Read the status, the error body and Retry-After from the error.
This function does that.
It waits and retries a 429, and a 503 that carries the API's own error_code.
For a GET, it also retries a 500 and a 503 that did not come from the API. For any other method it stops with an error.
The error carries the status and the response body. Your code can use them to tell a refusal from a failure whose outcome is unknown.
It treats every POST as a write, including the one POST call that only reads, and does not retry a 500 on it.
function Invoke-SparkLogs {
param(
[Parameter(Mandatory)] [string] $Path,
[string] $Method = 'Get',
$Body,
[int] $MaxAttempts = 5
)
$request = @{
Method = $Method
Uri = "https://engine.sparklogs.app/app-api$Path"
Headers = @{ Authorization = "Bearer $env:SPARKLOGS_TOKEN" }
}
if ($null -ne $Body) {
$request.ContentType = 'application/json; charset=utf-8'
$request.Body = [Text.Encoding]::UTF8.GetBytes(($Body | ConvertTo-Json -Depth 10))
}
for ($attempt = 1; ; $attempt++) {
try {
return Invoke-RestMethod @request
} catch {
$response = $_.Exception.Response
if ($null -eq $response) { throw } # no response arrived: a write may have taken effect
$status = [int]$response.StatusCode
$problem = $null
try { $problem = $_.ErrorDetails.Message | ConvertFrom-Json } catch { }
$wait = if ($response.Headers -is [System.Net.WebHeaderCollection]) {
$response.Headers['Retry-After'] # Windows PowerShell 5.1
} else {
$response.Headers.RetryAfter.Delta.TotalSeconds # PowerShell 7
}
$again = ($status -eq 429 -and $problem.cause -ne 'invalid_token_attempts') -or
($status -eq 503 -and $problem.error_code) -or
($status -in 500, 503 -and $Method -eq 'Get')
if (-not $again -or $attempt -ge $MaxAttempts) {
$failure = [Exception]::new("SparkLogs returned $status $($problem.error_code): $($problem.error)")
$failure.Data['Status'] = $status
$failure.Data['Problem'] = $problem
throw $failure
}
if (-not $wait) { $wait = [Math]::Pow(2, $attempt) }
Start-Sleep -Seconds $wait
}
}
}
Use it like this:
$me = Invoke-SparkLogs -Path '/credential'
$workspace = $me.credential.workspace_id
$page = Invoke-SparkLogs -Path "/workspaces/$workspace/organizations"
$new = Invoke-SparkLogs -Method Post -Path "/workspaces/$workspace/organizations" -Body @{
parent_id = $parentId # the organization that holds your clients
name = 'Lakeside Clinic'
psa_client_id = '10042'
}
When a call fails, read the status from the error:
try {
Invoke-SparkLogs -Method Delete -Path "/workspaces/$workspace/groups/$groupId"
} catch {
$status = $_.Exception.Data['Status']
$fromApi = $_.Exception.Data['Problem'].error_code
if ($status -eq 404) { 'Already deleted.' }
elseif ($status -eq 500 -or -not $fromApi) { 'Unknown outcome: read the group before trying again.' }
else { throw }
}
$_.Exception.Data['Problem'] holds the response body, with error_code and cause.
The status is empty when no response arrived.
The body is sent as UTF-8, which keeps names with accents intact.
A value in the query string, such as a time for updated_since, has to be escaped: [uri]::EscapeDataString($watermark).
Wait between attempts
When the response has Retry-After, wait that long.
Otherwise wait 2 seconds, double the wait on each attempt, and stop after 5 attempts.
Report the last response, including error_code and incident_id if it has one.