Antares

Antares is an NGSI-LD Context Broker written in Rust: one native binary that stores entities, answers queries, keeps attribute history, delivers notifications over HTTP and MQTT, and federates with other brokers through Context Source Registrations. The same crates compile to WebAssembly and run the whole broker inside a web page.

What it implements

ETSI GS CIM 009 V1.9.1, the NGSI-LD API, clause by clause: the information model (4.x), the operations (5.x), the HTTP binding (6.x) and the MQTT notification binding (7). The Conformance chapter carries the ledger, one file per clause, and the ETSI test suite runs against every storage backend in CI. NGSI-LD 2.0 splits the same material into a core specification and an HTTP binding; Antares keeps the operation semantics and the binding in separate modules for that reason, and the Extending Antares chapter says what a new binding attaches to.

Three deployment shapes

Current state and history are chosen independently: ANTARES_STORE picks where entities live, ANTARES_TEMPORAL where their history goes, including none. Storage drivers has the ladder and the measured costs.

What the broker does not do

Authentication, rate limiting and quotas belong to the policy enforcement point in front of the broker (an API gateway), and so does every authorization decision: the broker ships no policy engine, only the seam one attaches to (ADR-0020). The broker validates, stores, serves, notifies and federates; it trusts the NGSILD-Tenant header it receives and enforces tenant isolation in the store. The Shared crates chapter shows how a gateway uses the broker's own parser, query engine and matcher to make its decisions with identical semantics.

Which chapter

Links: live conformance report, browser playground, source, ADRs.

Getting started

Every snippet on this page was executed against a broker built from this repository before being committed.

Install

Docker (multi-arch, amd64 + arm64):

docker run --rm -p 9090:9090 ghcr.io/joinedcontext/antares-broker:latest

From source (Rust toolchain per rust-toolchain.toml):

cargo run -p antares-broker        # serves http://0.0.0.0:9090

Release binary: attached to GitHub releases (from v0.1.1 on).

The default configuration needs zero infrastructure: in-memory store, in-process bus, all roles in one process. Check it is up:

curl -s localhost:9090/q/health    # {"status":"UP","store":"memory",...}

First entity

curl -i -X POST localhost:9090/ngsi-ld/v1/entities \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:TemperatureSensor:001",
    "type": "TemperatureSensor",
    "temperature": {"type": "Property", "value": 21.5, "unitCode": "CEL"},
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld"
  }'
# HTTP/1.1 201 Created
# Location: /ngsi-ld/v1/entities/urn:ngsi-ld:TemperatureSensor:001

curl -s 'localhost:9090/ngsi-ld/v1/entities?type=TemperatureSensor'

First subscription

Start something that shows incoming HTTP requests — nc -l 9491 is enough — then subscribe:

curl -i -X POST localhost:9090/ngsi-ld/v1/subscriptions \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:Subscription:demo",
    "type": "Subscription",
    "entities": [{"type": "TemperatureSensor"}],
    "watchedAttributes": ["temperature"],
    "notification": {"endpoint": {"uri": "http://127.0.0.1:9491/notify"}},
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld"
  }'
# HTTP/1.1 201 Created

Change the watched attribute — the fragment is application/ld+json (with Content-Type: application/json the @context would have to travel in the Link header instead; mixing both is rejected per CIM 009 clause 6.3.5):

curl -i -X PATCH \
  localhost:9090/ngsi-ld/v1/entities/urn:ngsi-ld:TemperatureSensor:001/attrs/temperature \
  -H 'Content-Type: application/ld+json' \
  -d '{"type": "Property", "value": 42.0,
       "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld"}'
# HTTP/1.1 204 No Content

The listener receives:

{"id": "urn:ngsi-ld:Notification:ca63483a-bc7e-4731-a36f-1168f455eb97", "type": "Notification",
 "subscriptionId": "urn:ngsi-ld:Subscription:demo", "notifiedAt": "2026-08-26T16:16:00.851Z",
 "data": [{"id": "urn:ngsi-ld:TemperatureSensor:001", "type": "TemperatureSensor",
           "temperature": {"type": "Property", "unitCode": "CEL", "value": 42.0}}]}

MQTT delivery works the same way with an mqtt[s]:// endpoint URI.

First federation pair

Two brokers, one Context Source Registration. Local processes talk over loopback, which the egress policy allows by default; on an internet-facing deployment set ANTARES_EGRESS_ALLOW_PRIVATE=false (see Configuration):

ANTARES_HTTP_PORT=9391 antares &   # broker A
ANTARES_HTTP_PORT=9392 antares &   # broker B

Create an entity only broker B knows:

curl -s -X POST localhost:9392/ngsi-ld/v1/entities \
  -H 'Content-Type: application/ld+json' \
  -d '{"id": "urn:ngsi-ld:ParkingSpot:B:042", "type": "ParkingSpot",
       "status": {"type": "Property", "value": "free"},
       "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld"}'

Tell broker A that broker B serves ParkingSpot entities:

curl -i -X POST localhost:9391/ngsi-ld/v1/csourceRegistrations \
  -H 'Content-Type: application/ld+json' \
  -d '{
    "id": "urn:ngsi-ld:ContextSourceRegistration:brokerB",
    "type": "ContextSourceRegistration",
    "information": [{"entities": [{"type": "ParkingSpot"}]}],
    "endpoint": "http://127.0.0.1:9392",
    "@context": "https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld"
  }'
# HTTP/1.1 201 Created

Query broker A — the result comes from broker B through the registration:

curl -s 'localhost:9391/ngsi-ld/v1/entities?type=ParkingSpot'
# [{"id":"urn:ngsi-ld:ParkingSpot:B:042","type":"ParkingSpot",
#   "status":{"type":"Property","value":"free"}}]

That is the whole federation model: registrations route queries (and writes, subscriptions, temporal queries) to the sources that declared the matching types and id patterns.

Next

• Subscriptions and notifications: formats, showChanges, periodic delivery, throttling, MQTT, retries.
• Temporal API: how history is recorded and queried.
• Federation: registration modes, loop protection, partial failures, context source subscriptions.
• Deployment and Operations when it leaves the laptop.

Deployment

Every shape on this page is exercised by CI: the compose stacks run the full ETSI suite in the matrix cells, the HA/rolling shape runs weekly (roll-weekly) and in the two -nats cells, and the k8s manifests encode the same constraints. Resource numbers are measured by the matrix run (1 Hz sampling of every broker process, tables in each run's CI summary).

Sizing (measured, full ETSI suite as workload)

The CI resource gate enforces 350 MiB during the suite. Postgres sizing follows standard PostgreSQL tuning; the weekly scale run publishes the measured resident set of both.

Sizing under load (measured, weekly scale run)

One broker process holding every role, 1 000 000 entities over 100 tenants, 10 000 subscriptions and 10 000 registrations, on eight cores:

The last row is a queue of unfinished requests, not a working set: the p99 is 34 s and one query in seven answers 5xx, while 50 queries/s answers every one with a p99 of 385 ms. Between those rows the broker's resident set grows by about 1.8 MiB per forwarded read in flight, and what bounds that number is ANTARES_MAX_CONNECTIONS — so a pod's memory limit and its connection ceiling are one decision, not two. The default ceiling of 10 000 admits far more than a small pod can hold: the reference manifests set 512 against a 1 GiB limit.

Single node, no database

# memory: state dies with the process — tests, demos
docker run --rm -p 9090:9090 ghcr.io/joinedcontext/antares-broker:latest

# file: durable via redb, fsync-before-ack; the data dir MUST be a volume
docker run --rm -p 9090:9090 \
  -e ANTARES_STORE=file -e ANTARES_DATA_DIR=/data \
  -v antares-data:/data \
  ghcr.io/joinedcontext/antares-broker:latest

file mode constraints (measured, documented in the README store table): queries run on in-memory maps (~19 KB RSS for a small entity of a few hundred bytes — comfortable to ~10k of those; the 1.5 KB entity Storage drivers measures costs 37.6 KB, so size from your own payload), one writer at ~3.1k fsynced writes/s, backup is stop-copy only (redb holds an exclusive lock). Beyond that, move to postgres.

Single node with PostgreSQL

docker compose -f compose-files/docker-compose.yml up

Broker + PostGIS. timescale differs only in the image and ANTARES_STORE; temporal data lands in a hypertable. Set ANTARES_REQUIRE_RLS=1 in shared-schema multi-tenant deployments so the broker refuses a DB role that bypasses Row-Level Security. Tenants are created implicitly by the first write; listing and purging them is described under operations.

HA: replicas behind a load balancer

docker compose -f compose-files/docker-compose-ha.yml up

Two broker replicas + haproxy + NATS JetStream + PostGIS — the rolling-update shape. The contract that makes rolls invisible:

1. stop_grace_period (30 s in the compose file) MUST exceed ANTARES_DRAIN_DELAY_MS + ANTARES_DRAIN_DEADLINE_SECS (default 2 s + 20 s), or docker stop turns the drain into a kill. Docker's own 10 s default does not.
2. Replicas of one logical broker share ANTARES_HOST_ALIAS — behind the LB they are one hop for federation loop detection.
3. /q/health answers 503 DRAINING during the drain window; the LB pulls the pod before the socket closes.

Role-split fleet (scale-out)

Five roles × two replicas from the same binary — only api pods serve HTTP; matcher/notifier/temporal/registry consume the JetStream streams:

STORE=postgres docker compose -f compose-files/docker-compose-etsi.yml \
  -f compose-files/docker-compose-roles.yml --profile db up -d
dev/roles-smoke.sh                                       # notify chain fires EXACTLY once
STORE=postgres ROLES_SPLIT=1 bash dev/rolling-update.sh  # roll all 10 in role-group order

This is the exact shape the postgres-nats/timescale-nats CI cells run the whole ETSI suite against — while the fleet rolls continuously.

Kubernetes

Reference manifests in deploy/k8s/ encode the constraints the store mode dictates instead of leaving them as deployment choices:

• broker-file.yaml is hard-coded Recreate: redb takes an exclusive file lock — a rolling update would deadlock on the volume.
• broker-postgres.yaml rolls normally; readiness = /q/ready (store ping + bus connected).
• nats.yaml is a 3-node JetStream StatefulSet (ANTARES_NATS_REPLICAS=3).
• postgres-cnpg.yaml uses the CloudNativePG operator for primary/replica failover; postgres-dev.yaml is a single labelled-dev pod.

Upgrades

Blue/green is the recommended path for major upgrades: deploy the new version empty, replay declarative state (entities/subscriptions/ registrations) through the standard API, verify, switch traffic — the broker's config-plane companion pattern. In-place minor upgrades follow the rolling contract above. The file store carries a format version and refuses a mismatched file rather than serving partial data.

Subscriptions and notifications

Every example below runs against the quickstart broker (examples/quickstart/compose.yml, or cargo run -p antares-broker) with its three TemperatureSensor entities from seed.sh. Notifications are received by a small HTTP server on port 9380 that answers 200 and prints each body. $U is http://localhost:9090/ngsi-ld/v1 and $CTX the core context URL used by seed.sh.

Create a subscription

curl -si -X POST $U/subscriptions -H 'Content-Type: application/ld+json' -d '{
  "id": "urn:ngsi-ld:Subscription:hot", "type": "Subscription",
  "entities": [{"type": "TemperatureSensor"}],
  "q": "temperature>30",
  "watchedAttributes": ["temperature"],
  "notification": {
    "attributes": ["temperature"],
    "endpoint": {"uri": "http://localhost:9380/notify", "accept": "application/json"}
  },
  "@context": "'$CTX'"}'

HTTP/1.1 201 Created
Location: /ngsi-ld/v1/subscriptions/urn:ngsi-ld:Subscription:hot

A change that matches sends one notification. After PATCH $U/entities/urn:ngsi-ld:TemperatureSensor:qs:1/attrs with temperature 31.7, the receiver gets:

{
  "id": "urn:ngsi-ld:Notification:e5fb42a7-f1c4-49d2-bbb8-bf2255448669",
  "type": "Notification",
  "subscriptionId": "urn:ngsi-ld:Subscription:hot",
  "notifiedAt": "2026-08-26T15:55:28.489Z",
  "data": [
    {"id": "urn:ngsi-ld:TemperatureSensor:qs:1", "type": "TemperatureSensor",
     "temperature": {"type": "Property", "unitCode": "CEL", "value": 31.7}}
  ]
}

With accept: application/json the @context travels in the Link header:

Content-Type: application/json
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"

With accept: application/ld+json it is a member of the body instead.

What triggers a notification

• entities (type, id, idPattern) and q, geoQ, scopeQ select the entities; watchedAttributes limits which attribute changes count; with no watchedAttributes every attribute of a matching entity counts.
• notificationTrigger defaults to ["attributeCreated", "attributeUpdated"], as the stored subscription above shows; add attributeDeleted, entityCreated, entityUpdated or entityDeleted to hear about those.
• An update that changes nothing sends nothing.
• timeInterval replaces change-driven delivery with a periodic one, see below. A subscription cannot carry both timeInterval and watchedAttributes.

Notification shape

notification.format picks the entity representation:

notification.attributes restricts the attributes in each entry. sysAttrs: true adds createdAt/modifiedAt at entity and attribute level. showChanges: true adds previousValue (or previousObject, previousLanguageMap) next to the new value; it requires a normalized format:

HTTP/1.1 400 Bad Request
{"detail":"showChanges cannot be true when format is keyValues (5.2.14)","title":"BadRequestData",...}

One patch of temperature to 29.0, seen by a showChanges + sysAttrs subscription and by a keyValues one:

{"id": "urn:ngsi-ld:TemperatureSensor:qs:1", "type": "TemperatureSensor",
 "createdAt": "2026-08-26T15:55:28.402Z", "modifiedAt": "2026-08-26T15:56:04.370Z",
 "temperature": {"type": "Property", "value": 29.0, "previousValue": 31.7, "unitCode": "CEL",
                 "createdAt": "2026-08-26T15:55:28.402Z", "modifiedAt": "2026-08-26T15:56:04.370Z"}}

{"id": "urn:ngsi-ld:TemperatureSensor:qs:1", "type": "TemperatureSensor",
 "location": {"type": "Point", "coordinates": [19.15, 48.73]}, "temperature": 29.0}

Grouped delivery

One request produces at most one notification per subscription, however many entities it touched. A batch upsert of two sensors reaches the keyValues subscription as one POST:

{
  "id": "urn:ngsi-ld:Notification:c06c6141-a3fc-4553-8369-ddeb69aa56a2",
  "type": "Notification",
  "subscriptionId": "urn:ngsi-ld:Subscription:kv",
  "notifiedAt": "2026-08-26T15:56:06.437Z",
  "data": [
    {"id": "urn:ngsi-ld:TemperatureSensor:qs:2", "type": "TemperatureSensor", "temperature": 27.5},
    {"id": "urn:ngsi-ld:TemperatureSensor:qs:3", "type": "TemperatureSensor", "temperature": 34.9}
  ]
}

and timesSent moves by one. A notification body is capped at 4 MiB, the same limit the broker accepts on inbound bodies. A grouped delivery over the cap is split at whole-entity boundaries into several notifications; a single entity larger than the cap is sent alone.

Periodic delivery: timeInterval

{"id": "urn:ngsi-ld:Subscription:tick", "type": "Subscription",
 "entities": [{"type": "TemperatureSensor"}], "timeInterval": 2,
 "notification": {"format": "keyValues",
   "endpoint": {"uri": "http://localhost:9380/tick", "accept": "application/json"}}}

Every 2 seconds the broker sends all matching entities, changed or not; five seconds after creation the receiver holds two notifications with all three sensors each. Any interval greater than zero is accepted, fractions included; the broker checks due subscriptions twice a second, so an interval below that is served at the tick rate. With NATS and several broker pods, one pod claims each tick, so an interval fires once per fleet.

Throttling

"throttling": 30 sends at most one notification per 30 seconds per subscription; changes inside the window are dropped, not queued. Three patches in a row produced one delivery and timesSent: 1. With several pods on NATS the window is kept per pod.

Delivery bookkeeping

GET $U/subscriptions/{id} shows the fields of CIM 009 5.8.6. After the first successful notification:

"notification": {
  "endpoint": {"accept": "application/json", "uri": "http://localhost:9380/notify"},
  "lastNotification": "2026-08-26T15:55:28.489Z",
  "lastSuccess": "2026-08-26T15:55:28.489Z",
  "status": "ok",
  "timesSent": 1
}

A subscription pointing at a closed port after one matching change:

"notification": {
  "endpoint": {"accept": "application/json", "uri": "http://localhost:9399/nobody"},
  "lastNotification": "2026-08-26T15:55:37.662Z",
  "lastFailure": "2026-08-26T15:55:37.663Z",
  "status": "failed",
  "timesFailed": 1,
  "timesSent": 1
}

timesSent counts notifications, not attempts: a delivery that is retried and then succeeds still adds one. status flips back to ok on the next successful delivery.

Retry and dead letters

The default is one attempt per notification. ANTARES_NOTIFY_ATTEMPTS, ANTARES_NOTIFY_BACKOFF_MS and ANTARES_NOTIFY_MAX_AGE_SECS turn on retries with exponential backoff on a separate task; a notification whose attempts or age run out becomes a dead letter, listed and replayable through /q/dead-letters and counted on /q/health as deadLetters. See Operations and the Admin API.

Egress

Delivery goes through the same egress policy as @context fetches and federation forwards: http, https, mqtt and mqtts only, redirects capped, DNS pinned, response size capped, and a per-host breaker that pauses a failing endpoint. Private and loopback destinations are allowed by default; ANTARES_EGRESS_ALLOW_PRIVATE=false denies them for an internet-facing deployment, and a refused delivery is booked as a failure. A scheme no notification binding serves is refused when the subscription is created — the endpoint is input data that does not meet the requirements of the operation (5.8.1.4, Table 5.5.2-1), so the error is BadRequestData:

{"detail":"no notification binding registered for endpoint scheme \"ftp\" (6.3.8)","status":400,"title":"BadRequestData",...}

MQTT endpoints

An endpoint URI of the form mqtt[s]://[user[:pass]@]host[:port]/topic delivers notifications as MQTT publishes (CIM 009 clause 7). The message is {"metadata": {...}, "body": <Notification>}; protocol parameters go in notifierInfo:

"endpoint": {
  "uri": "mqtt://localhost:1883/antares/hot",
  "accept": "application/json",
  "notifierInfo": [{"key": "MQTT-Version", "value": "mqtt5.0"}, {"key": "MQTT-QoS", "value": "1"}]
}

The broker validates the URI and the parameters at creation (201 above, with no MQTT broker running) and connects on the first delivery. Sessions are pooled per endpoint and credentials; the password never appears in error bodies or logs. The binding is the mqtt cargo feature, on by default.

Temporal API

The temporal API (CIM 009 clauses 5.6.11–5.6.16, 5.7.3, 5.7.4) serves the history of attribute values. Every example below runs against the quickstart broker seeded by examples/quickstart/seed.sh; $U is http://localhost:9090/ngsi-ld/v1 and $L the Link header carrying the core @context.

How history is recorded

Every write through the entity endpoints produces temporal events for the attribute instances it changed. The events are buffered for the request and drained to the temporal driver after the handler returns and before the response leaves, so a temporal read that follows a write sees it. An update that changes no value produces no event; ANTARES_TEMPORAL_RECORD narrows what enters history to observed instances or to nothing. A driver failure in the drain never changes the response already produced; it is counted as temporalDrainErrors on /q/health. The gates, the driver choice ANTARES_TEMPORAL and retention are described in Storage drivers.

Three patches of temperature with an observedAt each:

curl -X POST $U/entities/urn:ngsi-ld:TemperatureSensor:qs:1/attrs \
  -H 'Content-Type: application/json' -H "$L" \
  -d '{"temperature":{"type":"Property","value":24.1,"unitCode":"CEL","observedAt":"2026-08-26T09:00:00Z"}}'

The seed value 21.5 carried no observedAt, so it appears only under timeproperty=modifiedAt (below), never in the default observedAt history.

Querying

timerel is before, after or between around timeAt (and endTimeAt for between), applied to timeproperty (observedAt by default; createdAt, modifiedAt, deletedAt). Clause 4.11 fixes the bounds: after includes the timeAt instant, before excludes it, and between includes timeAt and excludes endTimeAt. attrs restricts the attributes.

curl -si "$U/temporal/entities/urn:ngsi-ld:TemperatureSensor:qs:1?timerel=after&timeAt=2026-08-26T08:30:00Z&attrs=temperature"

{"id": "urn:ngsi-ld:TemperatureSensor:qs:1", "type": "TemperatureSensor",
 "temperature": [
   {"type": "Property", "instanceId": "urn:ngsi-ld:Instance:6e0d857e-8845-57ee-9e5b-bbcc34d89a0b",
    "observedAt": "2026-08-26T09:00:00Z", "unitCode": "CEL", "value": 24.1},
   {"type": "Property", "instanceId": "urn:ngsi-ld:Instance:19536e56-b044-51cd-8658-2600fb344d98",
    "observedAt": "2026-08-26T10:00:00Z", "unitCode": "CEL", "value": 26.8}]}

Each instance carries an instanceId; it is the handle for the instance-level operations below. GET $U/temporal/entities?type=… queries several entities with the same parameters plus q, geoQ, scopeQ, id, idPattern and paging; POST $U/temporal/entityOperations/query takes the same query as a Query body with a temporalQ member:

curl -X POST $U/temporal/entityOperations/query -H 'Content-Type: application/json' -H "$L" -d '{
  "type": "Query", "entities": [{"type": "TemperatureSensor"}],
  "temporalQ": {"timerel": "after", "timeAt": "2026-08-25T10:00:00Z"},
  "attrs": ["temperature"], "q": "temperature>25"}'

Errors are BadRequestData 400 with the reason in detail:

{"detail":"invalid timerel \"since\"", ...}
{"detail":"timeAt must be a valid ISO 8601 DateTime (4.11)", ...}

Representation

format=temporalValues (or options=temporalValues) collapses each attribute to [value, time] pairs:

{"id": "urn:ngsi-ld:TemperatureSensor:qs:1", "type": "TemperatureSensor",
 "temperature": {"type": "Property",
   "values": [[22.4, "2026-08-26T08:00:00Z"], [24.1, "2026-08-26T09:00:00Z"], [26.8, "2026-08-26T10:00:00Z"]]}}

timeproperty=modifiedAt keys the history on the write time instead, so the seed value shows up:

"values": [[21.5, "2026-08-26T15:59:33.332Z"], [22.4, "2026-08-26T15:59:33.409Z"],
           [24.1, "2026-08-26T15:59:33.416Z"], [26.8, "2026-08-26T15:59:33.423Z"]]

lastN=2 keeps the two newest instances of each attribute, newest first:

"temperature": [
  {"type": "Property", "observedAt": "2026-08-26T10:00:00Z", "value": 26.8, ...},
  {"type": "Property", "observedAt": "2026-08-26T09:00:00Z", "value": 24.1, ...}]

lastN must be a positive integer; pick/omit and sysAttrs apply as on the entity endpoints.

Aggregation

aggrMethods (totalCount, distinctCount, sum, avg, min, max, stddev, sumsq) with aggrPeriodDuration returns one row per bucket as [value, bucketStart, bucketEnd], buckets anchored at timeAt:

curl "$U/temporal/entities/urn:ngsi-ld:TemperatureSensor:qs:1?timerel=between&timeAt=2026-08-26T08:00:00Z&endTimeAt=2026-08-26T11:00:00Z&attrs=temperature&aggrMethods=avg,max&aggrPeriodDuration=PT1H"

"temperature": {"type": "Property",
  "avg": [[22.4, "2026-08-26T08:00:00Z", "2026-08-26T09:00:00Z"],
          [24.1, "2026-08-26T09:00:00Z", "2026-08-26T10:00:00Z"],
          [26.8, "2026-08-26T10:00:00Z", "2026-08-26T11:00:00Z"]],
  "max": [[22.4, "2026-08-26T08:00:00Z", "2026-08-26T09:00:00Z"], ...]}

Without aggrPeriodDuration (or with PT0S) the whole range is one bucket, ending one second after the last instance:

"avg": [[24.433333333333334, "2026-08-26T08:00:00Z", "2026-08-26T10:00:01Z"]],
"sum": [[73.3, "2026-08-26T08:00:00Z", "2026-08-26T10:00:01Z"]],
"totalCount": [[3, "2026-08-26T08:00:00Z", "2026-08-26T10:00:01Z"]]

Which methods apply to which value type follows Tables 4.5.19.1-1 to -3; a method that does not apply to the attribute's values is a 400. A value carrying a Date, DateTime or Time datatype — written as a JSON-LD typed value, {"@type": "DateTime", "@value": "..."}, or as a string with valueType — is ordered by that datatype, so min and max apply to it and a Time also has an avg, returned as a Time. A plain JSON string is ordered lexicographically and has no average.

On postgres and timescale the aggregation runs in SQL when the query is exact there: no q, geoQ or scopeQ, the page pushed down, no omit, second-granular period, and only numeric or boolean values in the window. Any other shape, and every query on the memory and file drivers, reconstructs the instances and aggregates in the broker. The result is the same; the SQL path was measured at 5.7 s → 0.59 s for 50 entities with 150k instances.

Pagination

One response carries at most nine instances per attribute. Beyond that the broker cuts the whole entity at one instant, answers 206 Partial Content and names the window it served in Content-Range. Twelve hourly instances of a sensor:

HTTP/1.1 206 Partial Content
Content-Range: date-time 2026-08-25T00:00:00Z-2026-08-25T08:00:00Z/*
{"id": "urn:ngsi-ld:TemperatureSensor:qs:2", "type": "TemperatureSensor",
 "temperature": {"type": "Property", "values": [[20.0, "2026-08-25T00:00:00Z"], ..., [28.0, "2026-08-25T08:00:00Z"]]}}

Continue from the instant after the range end; the last page answers 200:

GET …?timerel=after&timeAt=2026-08-25T09:00:00Z&attrs=temperature&format=temporalValues
HTTP/1.1 200 OK
"values": [[29.0, "2026-08-25T09:00:00Z"], [30.0, "2026-08-25T10:00:00Z"], [31.0, "2026-08-25T11:00:00Z"]]

Every attribute is trimmed to the same boundary, so no instance of any attribute falls between two pages. Aggregated representations are computed over the whole evolution and are never cut. Entity-level paging of GET /temporal/entities uses limit/offset and Link rel="next" as the entity endpoints do.

Temporal entity maps

entityMap=true on a multi-entity query pins the matched id set for the following pages, so a client walking rel="next" links sees a stable set even while entities change:

HTTP/1.1 201 Created
Link: </ngsi-ld/v1/temporal/entities?entityMap=true&limit=1&offset=1&…>; rel="next";type="application/json"

The map lives under /temporal/entityMaps/{id} for one hour by default; a client may set expiresAt on the map, capped at 24 hours (6.4.3.2-1). It is stored in the current-state driver, so it survives a restart on file and postgres.

Writing and deleting history

All answer 204; a missing instance or entity is ResourceNotFound 404:

{"detail":"instance urn:ngsi-ld:Instance:cbcc1cff-46bb-4b7f-b82e-f42a60d75542 not found","status":404,"title":"ResourceNotFound", ...}

Purging a temporal entity removes its history only; GET $U/entities/{id} still answers 200 with the current state. Deleting the current-state entity mirrors a deletion instance into history, so the entity's last state remains queryable under timeproperty=deletedAt.

Instances added through the temporal endpoints are stored as sent. Instances the entity endpoints record with an observedAt are keyed on (entity, attribute, datasetId, observedAt), so a sensor re-sending the same measurement replaces the instance instead of duplicating it.

Retention and none

ANTARES_TEMPORAL_RETENTION_DAYS starts a sweep on the postgres or timescale half that drops instances older than the horizon; unset keeps everything. With ANTARES_TEMPORAL=none every temporal endpoint answers OperationNotSupported 422 (Table 6.3.2-1) and the entity endpoints record nothing:

HTTP/1.1 422 Unprocessable Entity
{"detail":"no temporal store is configured","status":422,"title":"OperationNotSupported","type":"https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported"}

/q/health names both halves: "store": "memory", "temporal": "none".

Federation

Antares implements the CIM 009 distributed-operations model (4.3.6, 5.9–5.11, 6.3.17, 6.3.18). Context Source Registrations (CSRs) are the routing table: a broker holding CSRs forwards matching requests to the registered sources, merges the answers and reports per-source problems without failing the request. The DistributedOperations and IOP suites (134 + 286 tests) cover this surface in every CI cell.

Every example below uses two brokers, broker-a on 9090 and broker-b on 9091, as in examples/federation/compose.yml:

cd examples/federation && docker compose up -d && ./run.sh

run.sh creates an entity on B, registers B at A and queries A:

entity created on B
CSR registered on A -> B
federated query via A:
[{"id":"urn:ngsi-ld:ParkingSpot:fed:042","type":"ParkingSpot","status":{"type":"Property","value":"free"}}]
OK: B's entity served by A

With two local binaries instead of compose, set ANTARES_HOST_ALIAS per process and run B_FROM_A=http://localhost:9091 ./run.sh.

Registrations

A CSR declares what a source holds (information[]: entity types, ids, idPattern, propertyNames, relationshipNames) and how to treat it (mode):

An exclusive registration names one entity id and its attributes (4.3.6.3); a type-only or pattern registration is refused:

{"detail":"an exclusive registration shall name an entity id — an id pattern or Entity type defining a group of entities is not supported (4.3.6.3)","status":400,"title":"BadRequestData", ...}

operations bounds what may be forwarded (default federationOps); contextSourceInfo key/value pairs travel as headers on every forward to that source (4.3.6.5); localOnly adds local=true to every forward (4.3.6.4). observationInterval and managementInterval gate temporal forwards to the sources whose window overlaps the query.

A contextSourceInfo value of urn:ngsi-ld:request is not sent as that string: it copies the same-named header off the request that triggered the forward, and sends nothing when the triggering request carried no such header (4.3.6.5, 6.3.19). {"key": "Authorization", "value": "urn:ngsi-ld:request"} is how a source behind the same identity provider is given the caller's credential, and it is the reason creating a registration is a privileged operation: the key names any header and the endpoint is whatever the registration says. Two groups of keys never reach the source as a raw header. Headers the registration's other members already decide take precedence and cannot be overridden (6.3.19), which is NGSILD-Tenant, and so do the ones the binding sets itself: Host, Via, Link, Connection, Content-Type, Content-Length. The accept, contentType, jsonldContext and ngsildConformance keys are the ones 4.3.6.6 and 4.3.6.8 give their own meaning; the forward acts on them, and passing them through raw would corrupt the negotiation they steer.

A contextSourceInfo value goes on the wire and nowhere else. The broker names the registration, and the endpoint with its userinfo stripped, in the log line, the warning, the error body and the dead letter that a failed forward produces; it never writes the pairs themselves, so a bearer token handed over for one connection is not later readable from an operator's log or from /q/dead-letters. The registration document itself still carries the value, because 5.9.4 serves the registration back as it was written.

The registered @context

"jsonldContext" names a @context the Context Source reads its terms in, and the forward is recompacted into it: the payload, the term-bearing query parameters (attrs, type, geoproperty) and — for the resources that name one Attribute in the path, /entities/{id}/attrs/{name} and its temporal and value variants — the path segment itself. The segment is not one of the two things 4.3.6.6 lists, but it carries a term the Context Source expands with the @context the forward advertises, so a request that switched the context and left the segment alone would write to a different Attribute.

The whole request travels in one @context or none of it does. A payload the broker cannot express in the registered context — a batch array, a body that does not re-expand — makes the forward fall back to the request's own context, logged as a warning, because advertising the registered context over terms that are not in it is how a peer writes Attributes nobody named.

Timeout and cooldown

management.timeout (5.2.34) bounds one forward in milliseconds; the broker caps it at 8 seconds, and a registration without it gets the cap. management.cooldown keeps a source that failed out of the fan-out for that many milliseconds; inside the window the forward is answered as a timeout without contacting the source. A registration with "management": {"timeout": 500, "cooldown": 10000} pointing at a closed port answers in 7 ms:

HTTP/1.1 200 OK
Ngsild-Warning: 199 broker-b "no response was received from the registration endpoint within the timeout period"

Forward history

Every forward that reaches the wire is booked on the registration it was made for, in the five read-only members of Table 5.2.9-2:

{
  "id": "urn:ngsi-ld:ContextSourceRegistration:weather",
  "type": "ContextSourceRegistration",
  "endpoint": "http://source-b:8080/ngsi-ld/v1",
  "information": [{"entities": [{"type": "WeatherObserved"}]}],
  "timesSent": 412,
  "timesFailed": 3,
  "lastSuccess": "2026-05-05T11:02:44.118Z",
  "lastFailure": "2026-05-04T22:17:09.640Z",
  "status": "ok"
}

timesSent counts every attempt, failures included; timesFailed counts the ones the table calls failures, which in the HTTP binding is any response code other than 2xx, a timeout and a refused connection. status names the last attempt alone, so a source that has recovered reads "ok" however large timesFailed is. A member appears when it first has something to say: a registration that has never failed carries no timesFailed and no lastFailure.

Three outcomes are deliberately not counted, because the operation never left this broker: a destination the egress policy refuses, a source the circuit breaker is holding open, and a registration inside its own management.cooldown window. The breaker only opens after failures that were attempted and booked, so a source that has gone away reads "failed" before the first forward is suppressed.

They are read-only. A create or update that carries any of them has that member dropped, not refused, which is what 5.2.9 asks for.

Same source, several registrations

Registrations naming the same source (same endpoint, mode, tenant, contextSourceAlias, contextSourceInfo and localOnly) fold into one forwarded request whose attribute and entity scopes are the union. A different contextSourceAlias behind the same endpoint is a different source (5.2.9) and is contacted separately.

Distributed reads

A read that matches CSRs fans out concurrently, ANTARES_FED_FANOUT at a time (default 8), each forward bounded by the registration timeout and ANTARES_MAX_FED_RESPONSE_BYTES. The forwarded request is narrowed to what the registration declares (4.3.6.1): a registration with propertyNames: ["status"] receives attrs=status even when the client asked for attrs=status,owner, and any owner the source returns anyway is dropped from the merge. The forward carries the Via hop and the client's Link context, never the client's NGSILD-Tenant (4.14); the registration's own tenant member names the tenant to address at the source:

GET /ngsi-ld/v1/entities?options=sysAttrs&type=ParkingSpot&attrs=status
Accept: application/json
Via: 1.1 broker-a
Link: <https://uri.etsi.org/ngsi-ld/v1/ngsi-ld-core-context-v1.9.jsonld>; rel="http://www.w3.org/ns/json-ld#context"; type="application/ld+json"

Entity halves from several sources merge per 4.5.5 before pagination.

Partial failures

A source that fails does not fail the request. Each problem becomes one NGSILD-Warning header (6.3.17, Table 6.3.17-1) naming the broker that saw it:

A 404 from a source is a miss, not a warning. Warnings a peer returns travel back to the client next to the broker's own, up to eight per source (maxPeerWarnings in /q/health): the list is written by the source but sent by this broker, and eight carries a real cascade without letting one source outgrow the fan-out or bury what the broker has to say about the others. A source that sends more has the rest dropped and a line written to the log. Beyond the registration cooldown, a per-host breaker pauses an endpoint whose forwards keep timing out; a source that answers, even with an error, is never paused.

Distributed writes

Writes forward by registration mode and the registered operations. An exclusive or redirect registration whose operations exclude the write is an error of type Conflict (409), because the data lives only there; an inclusive one that excludes it is skipped. Batch operations return per-entity success and error arrays with the remote results folded in; a partial failure across inclusive sources is 207.

Loop protection

Every forward carries a Via hop (6.3.18) whose pseudonym is this broker's alias for the tenant (ADR-0011): ANTARES_HOST_ALIAS for the default tenant, {alias}~{tenant} otherwise. /info/sourceIdentity answers the same value per tenant, and that is what a peer stores as the registration's contextSourceAlias:

GET /ngsi-ld/v1/info/sourceIdentity
{"id":"urn:ngsi-ld:ContextSourceIdentity:broker-a","type":"ContextSourceIdentity","contextSourceAlias":"broker-a", ...}

GET /ngsi-ld/v1/info/sourceIdentity   NGSILD-Tenant: odpady
{"id":"urn:ngsi-ld:ContextSourceIdentity:broker-a~odpady","type":"ContextSourceIdentity","contextSourceAlias":"broker-a~odpady", ...}

Two rules follow from the inbound chain:

• A registration whose contextSourceAlias already appears in the chain is not a matching registration (Table 6.3.18-2); the request has been there.
• A request whose chain names this broker itself runs locally without re-forwarding. When the only matching registration for a write is a single exclusive or redirect source, the loop closes on data that lives nowhere else and the answer is 508 (6.3.17):

POST /ngsi-ld/v1/entities/urn:ngsi-ld:Loop:1/attrs   Via: 1.1 broker-b
HTTP/1.1 508 Loop Detected
{"detail":"the Via chain already contains this broker","status":508,"title":"Loop Detected", ...}

A chain longer than 32 hops is treated as a loop whatever it names. Replicas of one logical broker behind a load balancer share one alias on purpose; they are one hop. Changing an alias breaks every peer's loop detection, so treat it as a published identifier.

Context source subscriptions

POST /csourceSubscriptions (5.11) watches the registrations instead of the entities. A subscription on ParkingSpot receives the matching registrations at creation and each later change with a triggerReason:

{"id": "urn:ngsi-ld:ContextSourceNotification:bafc4692-…", "type": "ContextSourceNotification",
 "subscriptionId": "urn:ngsi-ld:Subscription:csr-watch", "notifiedAt": "2026-08-26T16:02:43.945Z",
 "triggerReason": "newlyMatching",
 "data": [{"id": "urn:ngsi-ld:ContextSourceRegistration:broker-c", "type": "ContextSourceRegistration",
           "endpoint": "http://localhost:9092", "information": [{"entities": [{"type": ["ParkingSpot"]}]}]}]}

csf filters registrations by their Context Source Properties (4.9).

Distributed subscriptions

An entity subscription whose scope matches CSRs is reduced per source and created at the remote broker (5.8); the remote notifies back to POST {ANTARES_PUBLIC_URL}/ex/v1/remote-notify, so set that variable whenever the default http://{host_alias}:{port} is not routable from peers. That endpoint is the one non-standard route a federated deployment must leave reachable from its context sources: CIM 009 defines no path for it (5.8.1.4 says only that the copy carries "the notification endpoint of the local Broker", and 5.2.15 makes a notification endpoint any URI), so it lives outside the /ngsi-ld prefix ETSI owns rather than inside it, and it is versioned on its own (ADR-0019). Other brokers place it elsewhere: Orion-LD and coraine serve POST /ngsi-ld/ex/v1/notifications/{subId}, Scorpio POST /remotenotify/{id}.

What arrives there is routed by the mapping alone. The forwarded copy carries a broker-generated subscription id, never the subscriber's own, so a context source learns nothing about the subscriber and cannot address any other subscription; the tenant comes from the stored mapping and not from the request, and the notified entities are re-filtered against the local subscription's own selector before delivery. Reduced copies follow the local subscription's lifecycle (update, delete); the registration's csf gates which sources take part. Inbound notifications from peers are matched against local subscriptions like local changes.

Pagination without amplification

The first distributed query can build an EntityMap (5.14): entity id to contributing registrations. Later pages contact only the sources that hold the page's entities instead of re-broadcasting the query. Maps expire (expiresAt, one hour by default) and are honoured on retrieve and temporal paths.

A federated map is merged from the maps the Context Sources return, so what a source sends is held to Table 5.2.39-2 before it becomes part of a document this broker stores under its own id and serves. A key that is not an Entity id is dropped, per key rather than per source, and so is the @none a source uses for what it holds locally — that marker is about the source, and this broker has no Entity id to record it under. A source's own map id reaches linkedMaps only if it is a valid URI, because it travels back out as the NGSILD-EntityMap header of every later forwarded page; a source without a usable one simply re-runs its query when the page arrives. One source contributes at most as many entries as the broker's own page ceiling (5.5.9, 1 000 by default), which is the ceiling the local half of the map already carries.

Tenancy across the federation

The client's NGSILD-Tenant never propagates to forwards (4.14); a CSR addresses a specific tenant of a remote source through its tenant member, and the ~-suffixed alias keeps each (source, tenant) pair distinct in loop detection. A registration pointing back at the same broker for another tenant is a legitimate federation shape, not a loop.

The five-broker stack

The IOP worked example, five brokers and no Docker:

dev/run-five.sh    # ports 9090..9094, aliases antares1..antares5

Each broker gets ANTARES_PUBLIC_URL=http://localhost:PORT; this is the stack the 286-test IOP tree runs against in CI.

Antares operations runbook

Everything here re-states what the code, manifests and workflows already enforce — no claim without a test behind it (the workflow proving each claim is named inline).

Deploy

Docker (single node): see the README quickstart — memory needs zero infrastructure; file needs a mounted volume; postgres/timescale need ANTARES_DATABASE_URL. Compose stacks in compose-files/ (docker-compose.yml one broker + PostGIS; docker-compose-ha.yml adds a second broker1 replica + haproxy + NATS — the rolling-update shape; docker-compose-roles.yml is the TRUE role split: 5 roles × 2 replicas = 10 broker containers — api×2 behind haproxy, matcher/notifier/temporal/ registry ×2 as ops-only worker pods — one shared PG, ANTARES_BUS=nats; these stacks are what CI's ETSI cells run).

Kubernetes (reference manifests, deploy/k8s/): namespace.yaml, nats.yaml (3-replica JetStream), postgres-dev.yaml (dev-only single PG; production uses a CNPG cluster, postgres-cnpg.yaml is lint-only), broker-postgres.yaml (an antares-api Deployment with ANTARES_ROLES=api and an antares-worker Deployment with matcher/notifier/temporal/registry), broker-file.yaml (single-replica file mode, Recreate strategy), networkpolicy.yaml (deny-by-default ingress in the namespace, one allow per flow the other manifests use; egress stays open because notification endpoints, Context Sources and @context URLs are client data, gated by ANTARES_EGRESS_ALLOW_PRIVATE and the scheme allowlist rather than by a CIDR list). Both broker pod specs set enableServiceLinks: false (kubelet's injected ANTARES_* service-link vars would otherwise trip the unknown-config check; the broker also exempts those exact shapes). Proven by k8s-smoke.yml's k8s-manifests kind smoke (dispatch): apply + every rollout status green.

• A worker pod serves the /q admin surface and nothing else — health, readiness, metrics, the tenant calls and the dead-letter admin. The NGSI-LD API exists only on pods with the api role.
• ANTARES_BUS=nats requires a shared store (postgres/timescale) and refuses to boot otherwise; bus=local requires all roles in one process.
• Production gates: set ANTARES_REQUIRE_RLS=1 (refuses a DB role that bypasses RLS) and require auth on NATS (the broker logs a loud warning on an unauthenticated JetStream).

Health, readiness, metrics

The NATS-outage contract (proven by nats_e2e::nats_outage_flips_health_and_recovers): during an outage the API keeps serving (writes land in the transactional outbox), /q/ready goes 503, and on reconnect the outbox drains — the outage-time notifications arrive, none lost.

Sizing the connection pool

ANTARES_PG_POOL (default 20) is how many PostgreSQL connections one broker process may hold. It is a ceiling on concurrent database work, not a throughput dial, and the measured runs say the dial does very little: on a 16-physical-core box pool 100 is 8 % faster than pool 20 at eight allotted cores and 10 % slower at 1 024 concurrent query clients, and both pools hold the same 1 000 rps write knee and fail at the same 1 500 (performance). Raise it to buy concurrency the database can actually serve, never to buy speed.

Size it from the database, not from the broker:

ANTARES_PG_POOL  <=  (max_connections - superuser_reserved_connections - other clients)
                     / number of broker replicas

A pool larger than the server's share does not fail at startup — it fails later, at the first burst, as FATAL: sorry, too many clients already from whichever client asks last. Leave headroom for the migration job, the replicas rolling during an update, and anything else on the same database.

Three signals say the pool is the constraint:

When the pool has nothing to give inside its acquire timeout, the request is answered 503 with a Retry-After header and no body. The operation was never attempted, so the client may retry the same request unchanged; a batch that had already written part of its array reports the failure per item instead, and never as a 503 the client would retry into duplicates. This is an Antares decision, not a CIM 009 requirement: Table 6.3.2-1 has no error type for an overloaded server, and clause 6.3.2 requires the HTTP binding's own status codes beside it.

Observability

Three signals, one switch (ANTARES_TELEMETRY):

Log records exported while a request span is open carry that span's trace id, so a collector joins the three signals per request.

Tenants

Tenants come to exist implicitly (CIM 009 5.5.10): the first create operation carrying an NGSILD-Tenant header creates the tenant, and the default tenant always exists. The NGSI-LD API has no operation to list or remove tenants; the admin surface has both.

The path names the tenant; an NGSILD-Tenant header on these calls is ignored. Like the rest of /q/*, the routes sit outside /ngsi-ld/v1 and belong behind the gateway. What the gateway owns instead: who may create a tenant, quotas and rate limits per tenant, authentication.

Reserved tenant names

The broker keeps part of its own state under tenants it names itself (ADR-0012), and a client can never name one of them:

A request whose NGSILD-Tenant is one of these answers 400 BadRequestData on every operation, reads included, so such a tenant can be neither read, written nor created from the API, whether or not the broker has minted it yet. /q/tenants/{tenant} and the ?tenant= of the admin calls refuse the same names, and GET /q/tenants never lists them.

Do not start a tenant name with snap-. The match is literal, case-sensitive and anchored at the start: snapshots-team, city-snap-1 and SNAP-index are ordinary tenants.

The content of a snapshot is read the way CIM 009 6.3.22 defines: send the normal query under your own tenant and add NGSILD-Snapshot: <snapshot id>. The broker resolves the header to its internal tenant after the reserved-name check. A notification from a subscription created inside a snapshot carries the owner tenant and the NGSILD-Snapshot header, never the internal name.

Notification delivery

CIM 009 5.8.6 books every notification once: timesSent moves by one, lastNotification is stamped, then either lastSuccess or lastFailure plus status: "failed". The broker sends once by default. Retries are an operator choice (ANTARES_NOTIFY_ATTEMPTS, ANTARES_NOTIFY_BACKOFF_MS, ANTARES_NOTIFY_MAX_AGE_SECS, configuration) and are transport under that one notification: the first attempt is booked as above the moment it resolves, the retries run on their own task (a slow endpoint never delays another subscription), a retry that succeeds sets lastSuccess and status: "ok" without touching timesSent or timesFailed, and an exhausted policy leaves a dead letter: the exact request (endpoint, headers, payload) plus the attempt history, stored under the subscription's tenant in every store mode.

/q/health reports deadLetters, the letters this process wrote since start; the letters themselves are rows, so they survive restarts on the file, postgres and timescale stores and a tenant purge removes them with the rest of the tenant. Egress-policy refusals (private ranges, blocked schemes) are never retried and never dead-lettered: a policy verdict is not a transport failure.

The notifications in flight at one moment are bounded broker-wide, and one tenant may hold only a share of that bound. A subscription belongs to one tenant, and a delivery to an endpoint that accepts the connection and never answers holds its slot until endpoint.timeout expires — up to 30 seconds. Without the per-tenant share, one tenant pointing enough subscriptions at dead endpoints would hold every slot for that long and nothing would leave the broker for anybody else. The share is a fraction of the bound rather than an equal split of it, so a tenant delivering alone still runs several notifications at once.

Backup and restore, per store mode

postgres. A custom-format dump restores with --clean, so the same command works on an empty and on a populated database:

pg_dump  -U postgres -Fc antares > antares.dump
pg_restore -U postgres -d antares --clean --if-exists antares.dump

Drill on a database holding four entities: SELECT count(*) FROM entities answers 4, DELETE FROM entities brings it to 0, pg_restore exits 0 and the count is 4 again. Stop the brokers before restoring; they cache nothing, but a write during the restore lands in a table that is about to be replaced.

timescale. The same tools; wrap the restore in TimescaleDB's SELECT timescaledb_pre_restore(); and SELECT timescaledb_post_restore(); so the hypertable catalog is restored with the data.

file. redb holds an exclusive lock, so a copy of a running broker's file can tear. Stop, copy, restart:

kill -TERM $(pidof antares)          # drains, then exits
cp -r "$ANTARES_DATA_DIR" /backup/antares-$(date +%F)

Drill: an entity created with an observedAt, broker stopped with SIGTERM, directory copied, a broker started on the copy answers GET /entities/urn:ngsi-ld:Vehicle:f:1 and its temporal history. A second broker on a directory that is already open fails at startup:

Error: "open …/antares.redb: Database already open. Cannot acquire lock."

Background jobs

One sweep loop per process, every ANTARES_SWEEP_SECS (default 900):

• Expired entities (expiresAt, 4.22) are deleted.
• Registrations, snapshots and entity maps (5.14; one hour by default, a client-set expiresAt capped at 24 hours) carry their own expiry and are deleted by the same loop. A read never deletes one: it refuses the expired document and leaves the row for the sweep, so a broker pointed at a read replica serves GET without writing.
• With ANTARES_TEMPORAL_RETENTION_DAYS set, attribute instances older than the horizon are pruned from the postgres or timescale temporal half, wherever that half lives (a file store with postgres history runs the job too). Drill with ANTARES_TEMPORAL_RETENTION_DAYS=30 ANTARES_SWEEP_SECS=2: an entity with one instance observed seven months back and one observed this hour shows both before the sweep and only the recent one after it.

The outbox drainer (ANTARES_OUTBOX_DRAIN) is the other loop; it hands committed changes to the matcher and can be moved to a dedicated process.

A change whose document exceeds the bus message ceiling (256 KB) travels as a claim-check reference, and its outbox row is kept instead of deleted: that row holds the bodies the message dropped, and the matcher reads them back by the event's sequence number. Such rows carry a published_at stamp, sit out of the drain's page, and the maintenance pass frees them 24 hours later. A matcher lagging further behind than that resolves nothing — the change is logged and counted on antares_claim_check_unresolved_total. A non-zero counter means the consumer side, not the store, needs attention.

Egress breaker

Every broker-initiated request (notification, forward, @context fetch) passes the egress policy: scheme allowlist, private-range rule (ANTARES_EGRESS_ALLOW_PRIVATE), redirect cap, DNS pinning and response size caps. On top of it a per-destination breaker tracks timeouts: five consecutive timeouts trip the destination open; while open, one probe per 30 seconds is admitted (half-open) and a success closes it again. A destination that answers, even with an error status, never trips; only silence does. At most 4096 destinations are tracked. A refused or tripped delivery is booked on the subscription as a failure (lastFailure, status: failed) and, with a delivery policy configured, is not retried.

An @context fetch is the one broker-initiated request that gets a second attempt: a connection carrying no response at all — refused, or dropped before a status line — is sent once more before the client is answered LdContextNotAvailable. A timeout, a redirect-cap breach and any response that did arrive are answers rather than accidents, so none of them is repeated, and the two attempts share one fetch deadline.

Drain

On SIGTERM the broker flips /q/health to 503, keeps serving for ANTARES_DRAIN_DELAY_MS (default 2000) so the load balancer notices, then waits up to ANTARES_DRAIN_DEADLINE_SECS (default 20) for in-flight requests before exiting. Set the container's stop grace period above the sum. The rolling-update section below relies on exactly this sequence.

Bulk load (postgres, timescale)

dev/bulk-load.sh loads entities straight into the entities table for initial loads and migrations. It bypasses the broker: no notification fires, no history is recorded, and the secondary indexes are dropped for the duration, so run it against a database no broker is serving.

DATABASE_URL=postgres://postgres:postgres@localhost:5432/antares \
  dev/bulk-load.sh vehicles.ndjson            # tenant "default"
DATABASE_URL=… dev/bulk-load.sh vehicles.ndjson odpady

Input is NDJSON, one entity per line in the store's internal form: attribute names as expanded IRIs, each attribute an array of instances, id/type/scope short. type and scope may be a string or an array; the loader stores both as arrays, which is the shape the query evaluator reads. A line may name its own tenant as a prefix separated by the byte 0x02 (t42<0x02>{"id":…}), which lets one file load many tenants; a bare JSON line lands in the tenant given as the argument. The file may be a FIFO, so a generator can stream straight into the loader:

python3 dev/perf/gen.py --entities 1000000 --tenants 100 > /tmp/e.fifo &
DATABASE_URL=… dev/bulk-load.sh /tmp/e.fifo

{"id":"urn:ngsi-ld:Vehicle:bulk:1","type":"https://uri.etsi.org/ngsi-ld/default-context/Vehicle","https://uri.etsi.org/ngsi-ld/default-context/speed":[{"type":"Property","value":42}],"https://uri.etsi.org/ngsi-ld/location":[{"type":"GeoProperty","value":{"type":"Point","coordinates":[19.15,48.73]}}]}
{"id":"urn:ngsi-ld:Vehicle:bulk:2","type":"https://uri.etsi.org/ngsi-ld/default-context/Vehicle","scope":"/city/east","https://uri.etsi.org/ngsi-ld/default-context/speed":[{"type":"Property","value":7}]}

The script, step by step:

1. \copy the file into an UNLOGGED staging table; the jsonb cast is the only parser the payload meets.
2. Drop the five secondary indexes (i_entities_location, i_entities_jsonb, i_entities_types, i_entities_loc_ambiguous, i_entities_expires); the primary key stays because the insert needs it.
3. Derive the columns the store derives on write: types, scopes, created_at/modified_at (the document's values, else now()), expires_at, and location from the default GeoProperty when it has exactly one instance holding a GeoJSON geometry; any other shape with the GeoProperty present sets location_ambiguous, and geo queries then judge that row in the broker instead of the index.
4. INSERT … ON CONFLICT (tenant_id, id) DO NOTHING: a row that already exists is left as it is, the loader never overwrites API-written data.
5. Rebuild the five indexes and ANALYZE entities.

Three lines offered, one of them an id the API had already created:

COPY 3
DROP INDEX
INSERT 0 2
DROP TABLE
CREATE INDEX
…
ANALYZE
bulk load done: 3 lines offered into tenant 'default'

Verify through the broker once it is back up; every query kind must see the loaded rows next to the API-written ones:

curl "$U/entities?type=Vehicle&q=speed>20&options=keyValues"
curl "$U/entities?type=Vehicle&georel=near;maxDistance==1000&geometry=Point&coordinates=[19.15,48.73]&options=keyValues"
curl "$U/entities?type=Vehicle&scopeQ=/city/%23&options=keyValues"

or in SQL, before restarting the broker:

SELECT id, types[1], scopes, ST_AsText(location), location_ambiguous
FROM entities WHERE tenant_id = 'default' ORDER BY id;

Rolling update

dev/rolling-update.sh — one instance at a time against the HA compose stack: SIGTERM → /q/health flips 503 → haproxy ejects within 400 ms → in-flight requests finish → recreate on the current image → wait healthy + rise window before the next instance. Preconditions and env are documented in the script header. file mode cannot roll (redb allows one process per volume): use a Recreate strategy there, as broker-file.yaml does.

Role fleet: ROLES_SPLIT=1 dev/rolling-update.sh rolls all 10 pods of the role-split stack in role-group order — the same-group peer must be healthy before its twin goes down, so no role ever has 0 live pods (api pods gate on /q/health + the LB rise window; workers on /q/ready). Measured: full roll ≈ 43 s (the api pod pays the ~21 s drain, workers ~2 s each), 52/52 LB requests answered 200 across the whole roll.

Proven: the roll-weekly workflow (Tue 04:17 UTC + dispatch) runs the FULL ETSI suite through the LB while the replicas roll in a loop — the suite has no retries, so any red TP is a real drain bug. The per-push postgres-nats/timescale-nats matrix cells do the same over the 10-container role fleet. On k8s the same contract holds via the readiness probe + terminationGracePeriodSeconds exceeding drain delay + deadline.

State reset (test/staging discipline)

API-level delete PAIRED with DB truncate — dev/reset-broker.sh plus the suite's clean_db.sh; never raw-SQL-truncate or container-restart alone. Federation/temporal state is only truly cleared by a volume-wiping teardown. After a reset, restart the broker before measured runs (in-VM subscription maps survive an external clean).

Upgrades

Minor versions roll in place under the rolling-update contract above. Major upgrades go blue/green: deploy the new version EMPTY, replay declarative state (entities, subscriptions, registrations) through the standard NGSI-LD API from your configuration source of truth, verify with smoke queries, switch traffic. Because the broker is vanilla CIM 009, the replay needs no Antares-specific tooling — any GitOps/city-as-code plane that speaks the standard API can drive it (this is requirement CC-50/51 of the companion configuration-plane spec). Temporal history is NOT part of the replay — restore it from database backup (per-store recipes above). The file store carries a format version: a downgraded or corrupted file is refused at startup rather than partially served.

Rolling a minor version in place is proven on every release tag rather than asserted: upgrade-path builds the previous tag's binary and this one, lets the old binary write an entity, its history and a subscription through the standard API, then points the new binary at that same file and postgres store and requires all three back — the entity at its last value, the history intact, and the stored subscription firing on a write the new binary accepts. Run it against any two binaries with dev/upgrade-path.sh OLD NEW (ANTARES_TEST_DATABASE_URL adds the postgres half).

Where the proofs run

Browser & WebAssembly

The same broker crates compile to wasm32-unknown-unknown and run inside a web page — an NGSI-LD broker with zero installation. Current artifact: 4.05 MB raw, 1.55 MB gzipped (budgets: 8 MB / 3 MB, the build fails over budget). Try it: https://antaresbroker.joinedcontext.com/demo/.

Two ways to use it in a page

• Service Worker: the worker intercepts fetch and answers /ngsi-ld/v1/* for the whole origin — existing NGSI-LD client code works unchanged against the page's own URL.
• In-page API: await broker.fetch(request) with the browser's own Request/Response objects — a caller cannot tell it from a network broker.

Stores in the browser: memory, or persistent via AntaresBroker.persistentWithHandle(...) over an OPFS sync-access handle (the browser's origin-private file system) — the same redb format as the native file store.

What a page cannot do (structural, not missing features)

• No inbound sockets and CORS: other systems cannot call into a page, so inbound federation and external HTTP notification callbacks are out of reach. Outbound notifications and forwards still leave via fetch.
• No MQTT, NATS, Postgres, or role-split — those need an OS process.
• Content-Length is a forbidden browser header: the wasm seam stamps it from the buffered body, since a page can never send it (CIM 009 6.3.4 is enforced on the wire truth by whatever fronts the broker — in the Node tier, the shim).

The Node tier

www/node-shim.mjs serves the SAME .wasm bytes behind a real TCP port (Node ≥ 18) — this is how the browser artifact is conformance-tested: the wasm-file CI cell runs the serial ETSI suites + IOP against five dockerized shims over the redb file store. Per-shim env: ANTARES_STORE (memory/file), ANTARES_FILE (redb path), ANTARES_HOST_ALIAS, and globalThis.ANTARES_PUBLIC_URL for distributed subscriptions (wasm has no process env — the shim wires these through JS globals).

Build it

./dev/install-wasm-tools.sh   # wasm-bindgen (lockfile-matched) + wasm-opt
./dev/wasm-build.sh           # → www/pkg; fails if over the size budgets
node www/node-shim.mjs 9090   # the artifact behind a TCP port
./dev/wasm-test.sh            # Node smoke + headless-Chromium page test

www/index.html is the playground: create entities, subscribe, watch notifications arrive in-page — including a loopback federation demo where one browser tab hosts multiple context spaces federating through CSRs.

Configuration reference

All configuration is environment variables — no config file. Unknown ANTARES_STORE/ANTARES_BUS values are fatal at startup, never silently defaulted. This table is checked against the source by dev/check-env-docs.sh (CI fails when a variable exists in code but not here).

Core

Store backends

NATS scale-out

Federation & egress hardening

Notification delivery

Lifecycle & observability

Compile-time bounds (no variable sets them; spec-shaped rejections): URI 8 KiB → 414, JSON depth 64 → 400. The body cap is ANTARES_MAX_BODY_BYTES above. Every bound in force is reported live by GET /q/health under limits.

Node-shim (wasm tier) extras: ANTARES_FILE (redb path per shim) — see the browser guide.

Admin API

This chapter is every route the broker serves OUTSIDE /ngsi-ld/v1. There are three groups and no others: /q/ is the operator surface, /ex/v1/ is the peer-facing wire between brokers, and /x/ is whatever a deployment mounted. Everything else the broker answers is CIM 009, and the conformance ledger owns it.

/q/ carries no NGSI-LD semantics and belongs behind the gateway. A worker pod (ANTARES_ROLES without api) serves only these routes. Errors use the same problem-details shape as the NGSI-LD API.

The same routes as an OpenAPI document, with every status code and response schema: docs/openapi/antares-admin.yaml, rendered at https://antaresbroker.joinedcontext.com/openapi/admin.html (see the API reference).

GET /q/health

200 with the process view, 503 while the instance drains (a roll in progress). A memory-store broker answers:

{
  "status": "UP",
  "store": "memory",
  "temporal": "memory",
  "version": "0.1.0",
  "commit": "3432674",
  "notificationSchemes": ["http", "https", "mqtt", "mqtts"],
  "deadLetters": 0,
  "changesDropped": 0,
  "taskPanics": 0,
  "temporalDrainErrors": 0,
  "policy": { "engine": "allow-all", "timeoutMs": 250 },
  "surfaces": { "admin": { "prefix": "/q", "routes": 8 } },
  "limits": {
    "changeQueue": 1024,
    "deliveryWidth": 64,
    "deliveryWidthPerTenant": 8,
    "maxBatchItems": 1000,
    "maxBodyBytes": 4194304,
    "maxContextFetches": 32,
    "maxFedFanout": 8,
    "maxFedInflight": 256,
    "maxFedResponseBytes": 16777216,
    "maxFoldDocs": 100000,
    "maxGeoVertices": 1024,
    "maxInProcessCallDepth": 8,
    "maxJoinLevel": 10,
    "maxJsonDepth": 64,
    "maxPeerWarnings": 8,
    "maxQLinkLookups": 512,
    "maxQNodes": 512,
    "maxRegexCache": 1024,
    "maxRegexCacheBytes": 67108864,
    "maxRegexProgramBytes": 262144,
    "maxTrackedDestinations": 4096,
    "maxUriBytes": 8192,
    "rejectedBodyTooDeep": 0,
    "rejectedBodyTooLarge": 0,
    "rejectedUriTooLong": 0
  },
  "memory": { "allocatedBytes": 5671592, "residentBytes": 24051712 }
}

GET /q/ready

200 {"status":"READY","store":true} when the instance is not draining, the store answers (SELECT 1 on Postgres) and, under bus=nats, the bus is connected; otherwise 503 {"status":"NOT_READY", …} with the failing member false. Point the readiness probe here and the liveness probe at /q/health: a restart does not cure a lost database.

GET /q/metrics

Prometheus text with the antares_ prefix:

Every _seconds metric is a true histogram, bucketed at 5 ms, 10 ms, 25 ms, 50 ms, 100 ms, 250 ms, 500 ms, 1 s, 2.5 s, 5 s, 10 s, 30 s and 60 s. The bounds are what histogram_quantile() can resolve, and the top ones exist because service time reaches tens of seconds once the accept path saturates.

Tenants

GET /q/tenants

Answers the names, sorted, and nothing else:

GET /q/tenants
["default","acme","zoo"]

Names only on purpose. A deployment runs up to 10 000 tenants (ADR-0001), and a list carrying per-kind counts would cost a count per kind per tenant on one request. 200 always; the route takes no parameters. Pick a name, then read its detail:

The list is the customer accounts. The broker mints tenants of its own — one per Snapshot, plus the two indexes — and none of them appears here or is addressable through the routes below.

GET /q/tenants/

GET /q/tenants/acme
{"tenant":"acme","counts":{"entities":0,"subscriptions":0,"csourceSubscriptions":0,
  "registrations":0,"snapshots":0,"entityMaps":0,"distSubs":0,"attrInstances":0}}

200, 404 for a tenant that does not exist, 400 for a name outside the tenant grammar or one of the broker's internal names. createdAt is present on Postgres, where the tenants table records it. The default tenant always exists (5.5.10) and is always readable, even when empty.

DELETE /q/tenants/

Purges the tenant: 204 when done, 404 for an unknown tenant ({"title":"ResourceNotFound","detail":"tenant nope"}), 409 while a distributed subscription of it still holds a copy at a Context Source, 400 for a name outside the tenant grammar. The default tenant is emptied and keeps existing. The path names the tenant; an NGSILD-Tenant header is ignored. Background in operations.

Dead letters

A letter carries the subscription id, tenant, endpoint, headers, body, attempt count, first and last error and timestamps. What produces one and why an egress refusal never does: notification delivery.

GET /q/dead-letters

?tenant=&subscription=&limit= — letters of one tenant (the default tenant when tenant is absent), newest first, limit 100 by default; 400 for a limit that is not a positive integer or a tenant outside the grammar. Endpoint userinfo, receiverInfo, notifierInfo and the rendered headers of an older letter are shown blanked; the stored letter keeps them so a replay still authenticates.

POST /q/dead-letters/{id}/replay

?tenant= — one attempt through the same binding under the egress policy of the moment: 204 and the letter is gone, 502 with the failure text and the letter kept, 404 when the tenant holds no such letter.

DELETE /q/dead-letters/

?tenant= — 204, or 404 when the tenant holds no such letter.

The peer-facing wire

POST /ex/v1/remote-notify

Where a Context Source posts a notification for a distributed subscription this broker created (5.8.1.4). It is the one non-standard route that cannot be firewalled off with /q/: every Context Source a subscription copy was forwarded to has to reach it, and ANTARES_PUBLIC_URL is what that copy advertises. It sits outside the /ngsi-ld prefix ETSI owns, and v1 versions the broker-to-broker wire independently of the NGSI-LD API version (ADR-0019).

The body is an NGSI-LD Notification. Its subscriptionId is the routing key, and it is a broker-generated UUID rather than the subscriber's own Subscription id, so a Context Source learns nothing about the subscriber and cannot address any other subscription. That key resolves through the stored mapping to the tenant and the local subscription; the request's NGSILD-Tenant header is not read on this route at all.

200 when the notification was accepted — including when the mapping resolves but nothing was left to deliver. 400 for a body that is not JSON, a body without subscriptionId, or a data array carrying more Entities than ANTARES_MAX_BATCH_ITEMS (the cap is applied before any store touch: one notification drives one local retrieve and one federated fan-out per Entity, so an uncapped array is an amplification lever). 404 when no distributed subscription maps the key. Like /q/, this route carries the body limit and the bounds wall itself — a peer-facing write path must not be the one route where the documented caps do not apply.

Deployment surfaces under /x

/x, and any path below it, belongs to the deployment. A surface registered there (extending) mounts its own routes and documents them itself; the broker only guarantees the ground rules. A prefix outside /q, /x and below-/x is refused at startup, as is one that overlaps a surface already mounted, so a deployment route can never shadow a spec resource or race another surface for a path. ANTARES_API_SURFACES names which surfaces are mounted, and /q/health reports each one's prefix and route count under surfaces.

The shipped binary mounts admin and nothing else. The reference plugin (examples/plugin-example, off by default) mounts /x/example and is the worked example of both a surface and a façade.

Storage drivers

The broker holds storage behind two object-safe traits in the dependency-free antares-store crate (crates/antares-store/src/lib.rs, ADR-0013): the core names the traits and no backend.

AppState carries one Arc<dyn CurrentStateDriver> and one Arc<dyn TemporalDriver>. Both traits keep generic mutate<T, E> ergonomics through the *Ext extension traits over a boxed mutate_boxed. Dynamic dispatch runs a handful of times per request against a database round trip or a JSON-LD expansion, so it costs nothing measurable.

The store ladder

memory → file → postgres → timescale (ADR-0004): one binary, one configuration value. The mem/ and pg/ folders under crates/antares-sql/src/store/ are the two implementations; any.rs is the AnyStore dispatcher that implements both traits. The folder README (crates/antares-sql/src/store/README.md) lists the steps for adding a backend.

file is not a second store: the redb shadow holds one table per resource family (entities, subscriptions, csource_registrations, csource_subscriptions, temporal_entities, jsonld_contexts, snapshots, entity_map_docs, dist_subs, dead_letters, plus meta for the format version), keyed tenant\0id, value the expanded JSON. Every commit is Durability::Immediate inside the store's write-critical section, so the fsync completes before the HTTP answer leaves. A failed commit aborts the process rather than acknowledge a write the file does not hold. Boot rebuilds the maps from the file and refuses a format-version mismatch. redb keeps an exclusive lock: one broker per volume, stop-copy backup, Recreate-only rollouts. Measured on the dev box with 1.5 KB entities: about 3,100 fsynced writes per second, commit p50 0.21 ms, p99 0.85 ms.

Choosing the temporal driver

ANTARES_TEMPORAL names the history backend and defaults to following ANTARES_STORE. Any pairing works: file current state with timescale history, postgres current state with memory history. A backend different from the store builds a second store instance used only for history; a Postgres half runs its own maintenance and retention job wherever the current state lives, and /q/health reports both halves (store, temporal).

ANTARES_TEMPORAL=none installs the NoTemporal driver: supported() is false, the recorder and the bookkeeping paths become no-ops, and every client-facing temporal read answers OperationNotSupported with status 422 (CIM 009 Table 6.3.2-1):

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported",
  "title": "OperationNotSupported",
  "status": 422,
  "detail": "no temporal store is configured"
}

What enters history

History is fed after the response, from a queue of TemporalEvents the entity endpoints produce. An event enters history only when every gate admits it (crates/antares-api/src/history.rs):

1. Value change, in the producer: an attribute instance whose value did not change never becomes an event.
2. ANTARES_TEMPORAL_RECORD: all (the default) admits everything; observed keeps only instances carrying observedAt, so a write without one updates current state and leaves no history; none records nothing from the entity endpoints while the temporal API itself still stores what it is given.

observed is a narrowing, not a tidying: an Entity created through the Core API with no observedAt has no temporal evolution at all under it, and a temporal query over such an Entity answers empty rather than answering with its values. That is why all is the default — five of the ETSI temporal conformance tests create their fixtures without an observedAt and expect to read the history back. Choose observed only where every producer stamps its own observation times.

A drain that fails leaves the client's 2xx standing and increments temporalDrainErrors on /q/health. ANTARES_TEMPORAL_RETENTION_DAYS prunes attribute instances older than the window from the maintenance job on the temporal half; unset keeps everything. The migration sets no retention on purpose: a schema that silently drops data is the wrong surprise.

Measured storage cost

Measured on a 3.9 KB expanded entity (1.5 KB compact), and on attribute instances of the same shape:

Two consequences shape the design. The in-memory Value costs nine times its text, so memory-mode capacity is bounded by RAM, not by the file. And expanded JSON-LD costs 4.8 times the compact text but only 1.8 times on disk, because Postgres compression absorbs the repeated IRIs; storing the compact form would buy nothing.

Migrations

crates/antares-sql/migrations/, applied by the first process to start unless ANTARES_MIGRATE=0 hands the job to an init container.

Conformance

Antares implements ETSI GS CIM 009 V1.9.1. Conformance is tracked in two places that check each other: the clause ledger in docs/spec/, and the ETSI Robot Framework suite vendored at ngsi-ld-test-suite/, run in a matrix of store modes on every push.

Every claim on this page is a claim about the broker as it ships: the built-in allow-all policy engine, no addon crate, no feature this repository leaves off by default. A deployment that attaches its own policy engine (Policy engines) narrows what its callers are answered with, and the conformance statement no longer describes what those callers see.

The ledger

docs/spec/ holds one file per clause of CIM 009, 947 files, each carrying the clause text, the PDF pages it came from, and three hand-maintained fields: status, evidence (code and test anchors) and notes. The robot: list is generated: python3 dev/spec.py robot scans the suite for [Tags] in the clause form (5_6_6) and writes the matching TP names into the clause file.

Current counts (python3 dev/spec.py status):

947 sections
  implemented       477
  informative       468
  partial             2
  robot-tagged       193

A partial names its gap in notes: and is the honest status for a clause whose normative surface is not fully closed; not-implemented is empty. python3 dev/spec.py check fails on a malformed file, a stale robot: list or a count in this chapter that no longer matches the ledger; dev/spec.py gaps lists leaf clauses without a TP.

The suite

The suite directory holds 671 .robot files under TP/NGSI-LD/{CommonBehaviours, ContextInformation, ContextSource, DistributedOperations, jsonldContext}. Most carry ETSI's own numbering (002_01, D018_01); the rest are additions written here for normative surface the official set leaves untested, either clause-numbered (566_01 for 5.6.6, 5510_01 for 5.5.10, 4233_01 for 4.23.3) or slotted into the ETSI family they extend, and all following the same conventions and tagged with their clause so the ledger picks them up. Every file expands to test cases; one full run of a native cell is 1822 test cases:

Behaviour Antares defines for itself, where CIM 009 is silent, does not go in TP/ — that directory is run against other brokers in interoperability campaigns, so every file in it has to assert a SHALL the spec text carries. Those tests live in ngsi-ld-test-suite/AntaresSpecificTests/ instead, and each says in its own documentation that it is an Antares decision rather than a CIM 009 requirement, with the reason the behaviour exists.

The matrix

The same 1822 cases run once per native cell; wasm-file runs 1810 of them, the twelve MQTT cases having no broker socket to run against in the browser build. Every push gates on the quick preset; the full preset runs twice a week, on v* tags and on dispatch, and is what the report page and the badges render.

A native cell passes at 1822/1822, wasm-file at 1810/1810. dev/etsi-matrix-summary.py folds the per-cell results into one table and lists every failure across the matrix; a release requires that list to be empty.

Running a suite locally

One store mode per run, the one the change touches:

dev/etsi-local.sh                                  # workspace tests + memory cell
STORE=timescale dev/etsi-local.sh                  # one cell, all suites
STORE=all dev/etsi-local.sh                        # the quick trio, serially
STORE=postgres STOP_ON_ERROR=1 dev/etsi-pipeline.sh    # halt at the first red TP
STORE=file SUITES=Consumption,Subscription SKIP_BUILD=1 dev/etsi-pipeline.sh

dev/etsi-pipeline.sh knobs: STORE, STOP_ON_ERROR (default 1, CI sets 0), SKIP_BUILD (reuse the local image), SUITES (comma list), MQTT=1 (include the 058_* MQTT cases), KEEP_UP=1 (leave the stack running), MEM_LIMIT_MB (per-broker peak-RSS gate), ROLES_SPLIT=1 ROLL_DURING_RUN=1 (reproduce a -nats cell), WASM=1 WASM_DOCKER=1 STORE=file (reproduce the wasm cell). Results land in results/$STORE with Robot's own log.html per suite.

For a single clause during development, one broker without Docker is enough. resources/variables.py carries the suite's own compose addresses (scorpio1 for the broker, 172.28.0.18 for the notification and context-source mocks), which the runners rewrite and a bare robot does not, so the recipe overrides them itself:

cargo build -q -p antares-broker -j 2
ANTARES_HTTP_PORT=9377 ./target/debug/antares &
cd ngsi-ld-test-suite && robot --variable url:http://localhost:9377/ngsi-ld/v1 \
  --variable temporal_api_url:http://localhost:9377/ngsi-ld/v1 \
  --variable notification_server_host:127.0.0.1 \
  --variable context_source_host:127.0.0.1 \
  --variable context_server_host:127.0.0.1 \
  --outputdir /tmp/robot-566 TP/NGSI-LD/ContextInformation/Provision/Entities/DeleteEntity/566_01.robot

Suite and spec defects

A red TP is proven against the clause text before any broker change. When the text says the TP or the spec is wrong, the finding goes to docs/upstream/etsi-raises.md as a ready-to-file issue, and the fork carries the fix. The current list:

Filing is manual; each entry carries the clause quotation and the proposed fix so it can be pasted into the ETSI tracker as is.

Spec-statement coverage

The ledger says implemented or partial per clause; this table says which of those clauses no Robot test exercises at all. python3 dev/spec.py statements counts the SHALL statements in each leaf clause's text against the TPs tagged with its number (or its operation's number) and the code/test anchors its evidence cites. It adds no tests; it names where the next ones belong.

310 leaf clauses carry 1661 SHALL statements; 117 of them have no Robot TP (338 SHALLs), 63 cite no code/test anchor.

The fifteen untested clauses with the most SHALL statements:

A SHALL count is a proxy: one sentence can carry several rules, and a clause's unit tests (the anchors column) may assert what no TP does. The counts are a snapshot to regenerate after ledger or suite changes, not a gate.

Performance

Two scheduled runs on dev produce every performance number this project publishes; nothing in this chapter is typed in by hand. Both rent a dedicated-vCPU machine at Hetzner for the run, register it as an ephemeral GitHub runner, and delete it afterwards (perf-janitor sweeps a leaked server or volume by its expiry label). Results land under /reports/perf/latest/ with the raw CSVs next to the tables.

scale-weekly sizes the database to the box it rents: shared_buffers a quarter of RAM, maintenance_work_mem a sixteenth, the shared memory segment an eighth, and the broker's connection pool eight per core with the eight-core value pinned to the 100 it was measured at. A number from one server type therefore does not compare with a number from another; the box is named next to every table.

The shapes (perf-weekly)

Every script lives in dev/perf/ and runs on a laptop the same way it runs in CI; k6 is the only tool it needs.

The load generator shares the machine with the broker, as in every published broker table; the numbers describe that shape and nothing else, and quadrupling the concurrency shows the queue, not the broker. Reproduce them on your own box before quoting them:

cargo build --release -p antares-broker
dev/perf/startup.sh && dev/perf/shapes.sh && dev/perf/core-scale.sh && dev/perf/saturate.sh
python3 dev/perf/report.py results/perf     # index.html + perf.json

The measured ceiling

Two perf-weekly dispatches. The first, on ccx33 (8 logical, 4 physical cores, three passes) at commit b55d554, could isolate only the 1- and 2-core steps: pinning the broker to 4 cores needs 8 physical, and to 8 needs 16. The second, on ccx53 (32 logical, 16 physical cores, two passes) at commit 41610be, runs the whole ladder, and the tables below are that run. store=postgres in both; the load generator shares the machine, so every row is broker plus generator.

Core scaling, one row per store and pool: the broker pinned to N physical cores (SMT siblings excluded), the generator on the rest, query shape at 50 concurrent clients. cores used is the broker's CPU time over the window against the cores it was allotted.

The same rows as CPU spent per request (cores used over req/s), which is what the efficiency column is measuring underneath:

The saturation knee, whole box, open model:

Throughput per shape at 64, 256 and 1 024 concurrent clients, whole box:

The run predates ADR-0022: the Postgres driver still parked a thread per in-flight store call, which is what the four-figure peak threads column records. The knees and the per-request cost are what that shape delivered.

What the run says, in the order it matters:

• Scaling is close to linear to four cores (98 %, 92 % on memory; 91 %, 85 % and 90 %, 88 % on the two pools) and loses half of that at eight.
• The eighth core is used, not idle: 7.64 of 8 on memory, 7.03 and 6.47 on the two pools. What changes is the price of a request, which is flat from one core to four and then rises by about 80 % on all three stores. A cost that appears identically in the in-process store, which never parks a thread on a socket, is not the storage driver: it is contention above the store, in the path the three shapes share.
• The blocking pool is nowhere near its ceiling. The Postgres write shape parks 4 038 live OS threads at its knee against a ceiling of 11 024 (ANTARES_MAX_CONNECTIONS plus 1 024, main.rs), and the query shapes park 64 to 77. Nothing deadlocks and no shape reaches the cap.
• Pool size barely moves anything on this box. Pool 100 is 8 % faster at eight cores and 10 % slower at 1 024 concurrent query clients; both pools hold the same 1 000 rps write knee and fail at the same 1 500. The ccx33 run's "a larger pool is worse" reads as an artefact of four physical cores, not a property of the pool.
• The write path is what bends first. Both Postgres pools hold 1 000 rps and fail at 1 500 while every query shape holds the harness ceiling of 5 000 rps; the in-memory write path holds 5 000 rps on 0.47 cores.

The update shape

The ladder above is the query shape. perf-weekly run 33797374897, same ccx53 box, runs it again against the update shape — the write path with the notification pipeline behind it — once per store:

Updates do not merely scale worse than queries: past two cores they scale backwards, and eight cores serve fewer requests per second than one while burning 6.78 of them. Something serializes and the cores spend their time arriving at it.

Not the store's map lock, which is what a single-tenant ladder would blame and what tenant sharding would answer: this ladder drives one tenant, so sharding by tenant would leave every request on the same shard and change nothing. What the write path does that the query path does not is hand the worker's queue to another thread — block_in_place — for every document it writes, so the store can commit without stalling an async worker. In file mode that commit is an fsync and the hop is the point. In memory mode there is no commit: the write is a lock and a map insert, and the hop is the whole cost, paid once per write and paid more the more cores there are to hand work between. The hop is now taken only where something blocks; the paths that hold the write section for a whole scan (the 4.22 sweep, the two purges) still take it in either mode.

The exit criterion

perf-weekly run 33683839528 sets what the runtime work has to hold. A change to the request runtime, the drivers or the store keeps all of it:

• Efficiency at eight allotted physical cores, query at c50: at least 51 % on memory and 43 % on either Postgres pool.
• CPU per request at eight cores: no more than twice its one-core value on any store. The run itself sits at 1.97 (memory), 1.89 (pool 20) and 1.87 (pool 100).
• Saturation, whole box: the knee at 5 000 rps or better for every query shape and 1 000 rps or better for the Postgres write shape, p99 at the knee within 3.0 ms.
• Live OS threads at the write knee: 4 038 or fewer, against the 11 024 ceiling.

Deployment scenarios

dev/perf/scenarios.sh starts one or more brokers from the release binary (dev/perf/fleet.sh), seeds each world over the API and asks one question per scenario. MODE=check runs the conformance assertions of dev/perf/scenario-check.py against a memory-store fleet in seconds; MODE=load runs the same assertions first and then a k6 rate ladder whose every number comes from k6's summary or the sink's counters. A verdict is computed from those numbers and its note names the failing assertion. The fleet notifies a single-process sink.py of its own on port 9810: the load rig's sink on 9800 runs multi-process, and its front door only folds the workers' counters.

MODE=check STORE=memory   ./dev/perf/scenarios.sh              # all nine
MODE=load  STORE=postgres ./dev/perf/scenarios.sh loop fan-in  # a subset

The nine worlds, and what each one asks:

• S1 hot-entity: one broker, 1 000 vehicles. Concurrent partial updates on one entity (5.6.3, distinct datasetIds) versus the same load spread over all of them: are updates lost, and what does the contention cost in p99?
• S2 noisy-tenant: one broker, tenants quiet and loud. The quiet tenant's GET p99 is measured alone and under a write flood on loud (4.14 isolation, in time as well as in data).
• S3 slow-subscriber: one broker, ten fast endpoints and one that answers after 500 ms. Does the slow one hold the fast ones back, given ANTARES_DELIVERY_WIDTH_PER_TENANT?
• S4 fan-in: one broker, 50 subscriptions matching one entity. How many notifications per second does one update stream fan out to, and are all of them delivered?
• S5 hub-sources: a hub and two source brokers, registered with the tenant member (a forward never carries the client's tenant). Federated query and retrieve through the hub (4.3.6.1, 5.7.2): complete, merged, 404 for an absent id.
• S6 collision: the same entity id in the hub and a source. 4.5.5 merge of non-colliding attributes, 4.3.6.2 local data over an auxiliary source, 5.9.2.4 409 for an exclusive or redirect registration that overlaps data already held.
• S7 loop: brokers A and B registered to each other. A query terminates with each side's data once (6.3.18 Via); a write whose chain already names the receiver runs locally under an inclusive registration and answers 508 (6.3.17) when the only matching source is a redirect. An id-only write carries no type, so every type-only registration matches it: the 508 case needs a tenant whose only registration is the redirect.
• S8 distributed-subscription: a subscription at the hub, the entity at a source. 5.8.1.4: the hub plants a reduced copy at the source (registration operations includes federationOps) and a change at the source reaches the hub's subscriber, counted against the accepted updates.
• S9 ha-pair: two broker pods on one PostgreSQL database and one NATS JetStream bus (NATS_URL, PG_URL_BASE; skipped without them). Writes alternate between the pods; notifications are counted against accepted writes, so a duplicate or a loss shows as a mismatch.

The tables land in scenarios/ of the run artifact; report.py folds them into index.html and pdf.py gives each scenario a page (world, question, what CIM 009 requires, verdict, measured numbers).

The design targets (scale-weekly)

The README's target table is a design contract; this run is where each row gets its measured column. SCALE scales every count linearly, so the same rig runs at 0.0001 against a laptop's Postgres and at 1.0 on the rented box:

dev/perf/sink.py is the other end of every subscription and registration: it counts notifications and answers forwarded queries with an empty list, so the fan-out over 100,000 registrations costs the broker the matching and the HTTP round trips and nothing else.

Every run is capped at one hour: the server's TTL, the job timeout and the box's own shutdown timer agree on it. Scale 0.01 (1,000,000 entities, 100 tenants, 1,000 subscriptions, 1,000 registrations) fits with margin and is what the schedule runs; 1.0 does not fit in an hour and is a deliberate dispatch on a bigger box with the TTL raised in the workflow. Bulk load bypasses the broker (no notifications, no history), which is the documented path for initial loads in Operations.

The measured run

scale-weekly run 33863032274 on a ccx33, SCALE=0.01 with the subscription and registration counts raised to 10,000 each: 1,000,000 entities over 100 tenants. The load took 367 s for the entities (one COPY stream), 8 s for the subscriptions and 159 s for the registrations.

Subscriptions firing, over 10,000 subscriptions on 101 tenants:

Against run 33835261405, the same shape on the same server type before the per-drain @context memo landed:

Federated queries, over 10,000 registrations, every source the sink:

Resident set and CPU over the whole run, 1 085 samples about 1.3 s apart:

What the run says:

• Delivery is exact to 200 rps of writes: all 193 406 due notifications arrive, nothing is dropped, and the sink goes quiet in the same second the writes stop. The run before the memo delivered 99.9 % of the same shape, dropped 14 changes and took four seconds to drain. Write latency at that rate falls with it, 69.6 ms to 28.1 ms at the 99th percentile, because the matcher drain is no longer competing with the write path for a core.
• At 500 rps both runs are past the knee the run itself reports, and the newer one is further past it: the drop count barely moves (25 359 to 26 398) but what survives is delivered more slowly (848 to 561 POSTs/s). Cheaper matching feeds the delivery stage faster, so the queue behind it fills sooner. The limit line reads 200 rps in both runs; what changed is that 200 rps is now met exactly instead of nearly.
• The registry narrowing holds. Each tenant carries 100 registrations and a type=Vehicle query reaches 34 of them — the three Vehicle-typed classes of the eight in csr.md — so the index decides the fan-out and the forward path is not a broadcast. The ratio is flat from 50 to 200 rps.
• The distributed path returns no 5xx, no 4xx and no NGSILD-Warning over 9 376 queries: every one of the 421 216 source calls was answered and folded in.
• Where the broker bends, it is not out of CPU. At the Postgres query knee it holds 3 000 rps on 2.56 of 8 cores and at the write knee 1 000 rps on 1.64; on the federated path it sits at 2.2 cores while Postgres takes 5.1 and the host runs out at 7.9. The component that saturates is the database or the machine, never the broker alone.
• A federated query costs about eight times the Postgres work of a direct query of the same shape (42.5 against 5.5 mcores). The fan-out is 34 source calls and a registry match over 10,000 rows; the direct row is one local read, so the two are not the same unit of work and the ratio is the price of federation, not a regression against a like-for-like baseline. The broker side of that comparison is not available from this run. The sampler matched broker processes against the whole of the first broker's argv[0], and the shapes, saturate and startup stages start their own brokers from the same binary by a different path form, so it counted none of them and reported the idle first process instead — zero cores through windows serving 912 req/s. saturate.md disagrees with rss.csv for the same window (2.56 cores against zero) because that stage measures the process it started itself. The sampler now matches on the last two path components, so a later run carries a broker CPU column for every stage; the rows above are left as they were measured.

How each measurement works

No rig script is a unit test. Every one starts a real broker, drives it over HTTP, and folds what came back into one Markdown table. The pieces are the same in all of them:

  dev/perf/<script>.sh
        │
        ├── starts ──► antares  (release binary, one store)  :9090
        │                 │
        │                 └── store: memory | file | postgres (docker)
        │
        ├── drives ───► k6 (a dev/perf/k6-*.js scenario)  or  python3
        │                 └── constant-arrival-rate: offered load, not
        │                     closed-loop, so a slow broker shows up as a
        │                     growing queue instead of a slower client
        │
        ├── receives ─► dev/perf/sink.py  :9800
        │                 ├── POST /…      one notification, counted
        │                 └── GET  /csr/k  one forwarded query, answered []
        │
        └── samples ──► dev/perf/rss.sh    1 Hz → rss.csv
                          reads /proc for broker, Postgres, k6, sink, host

$OUT/phase is the thread that ties them together: each script writes the stage name into it before it starts, and rss.sh copies that string into every sample it takes, so any row of rss.csv can be attributed to the stage that caused it.

startup.sh — how long a cold broker takes to answer

Runs the binary, polls GET /q/health until the first 200, records the elapsed time and reads VmRSS out of /proc/<pid>/status immediately after. Five times per store, median reported. Nothing else is running, so the number is the process itself: binary load, config parse, store open, listener bind.

shapes.sh — throughput per request shape

Starts one broker, seeds 100 five-attribute entities through the API, then runs k6-shapes.js closed-loop at a fixed number of clients: a list query at 50 and 200, a single retrieve at 50. Three runs of five seconds, median reported, p99 from the same runs. Closed-loop on purpose — this table answers "how fast is one shape", not "where does it break".

core-scale.sh — does the broker use the cores it is given?

Pins the broker to 1, 2, 4 and 8 physical cores with taskset (SMT siblings excluded) and keeps k6 on the cores left over, refusing any step where the two would share one. The in-memory store on purpose: the question is whether the broker's own work parallelises, so nothing waits on a database.

saturate.sh — where the knee is

Open model. k6-saturate.js raises the arrival rate by 500 rps every stage and keeps going until p99 passes P99_MS or the error rate passes ERR. The knee is the last stage that held both. A run that never fails a stage reports none reached, which means the ladder is shorter than the box — the answer is more stages, not a bigger conclusion.

load.sh — building the dataset

The only stage that does not go through k6:

load.sh
  ├─ sink.py 9800 8        one front door, eight worker processes behind it
  │                        (one CPython process tops out near 5 000 req/s)
  ├─ gen.py | bulk-load.sh entities, one COPY stream straight into Postgres,
  │                        bypassing the broker: no notifications, no history
  ├─ api-load.py subscriptions   through the API, eight filter classes
  └─ api-load.py registrations   through the API, eight CSR classes

Entities bypass the broker because a hundred million of them through the API would measure the loader. Subscriptions and registrations do not: they have to pass validation and land in the matcher's index, which is what the later stages exercise.

fire.sh — do the subscriptions fire, and up to what rate

k6-fire.js                broker :9090                    sink.py :9800
    │                          │                                 │
    │ PATCH …/attrs (update)   │                                 │
    │ DELETE …/entities/{id}   │                                 │
    ├─────────────────────────►│                                 │
    │              204         │ 1. write the change              │
    │◄─────────────────────────┤ 2. match it against this         │
    │                          │    tenant's subscriptions        │
    │                          │ 3. queue it (changeQueue, 1024)  │
    │                          │ 4. deliver, deliveryWidth (64)   │
    │                          │    in flight, 8 per tenant       │
    │                          │                                  │
    │                          │  POST /  {"data": [entity, …]}   │
    │                          ├─────────────────────────────────►│
    │                          │                          204     │
    │                          │◄─────────────────────────────────┤
    │                          │                                  │
    └── k6 stops ──────────────┴── fire.sh polls /stats until the │
                                   count stops moving for 5 s     │

The count due is not measured, it is derived: k6-fire.js knows which subscription classes each write should trigger and evaluates the same rule api-load.py used to create them, so delivered / due is a real ratio and not a guess. quiet after is how long the sink kept receiving once the stream stopped — the drain. dropped by broker comes from the broker's own changesDropped counter, read from /q/health before and after, so a delivery gap can be charged to the queue, to the delivery policy or to the receiver rather than left ambiguous.

The limit is the last rate that delivered 99 % with no failed operation, and the ladder stops at the first rate that misses it.

fed.sh — federated queries over the registrations

The registrations point at sink.py, which answers every forwarded query with an empty array. That is deliberate: an empty answer costs the broker the index lookup, the fan-out and the HTTP round trips and nothing else, so the row measures the federation machinery rather than a source.

k6-fed.js                    broker :9090                        sink.py :9800
    │                             │                                      │
    │ GET /entities?type=Vehicle  │                                      │
    │ NGSILD-Tenant: t42          │                                      │
    ├────────────────────────────►│                                      │
    │                             │ 1. expand the query against @context │
    │                             │                                      │
    │                             │ 2. csource_index lookup (5.12): one  │
    │                             │    SQL query, narrowing on entity     │
    │                             │    type and entity id alone. It may  │
    │                             │    only REMOVE registrations the     │
    │                             │    matcher would reject anyway, so   │
    │                             │    a NULL dimension always survives  │
    │                             │                                      │
    │                             │ 3. the matcher decides the rest in   │
    │                             │    Rust: geoQ, scopeQ, csf, the      │
    │                             │    intervals, the idPattern regex,   │
    │                             │    the Via chain                     │
    │                             │                                      │
    │                             │ 4. fold registrations naming the     │
    │                             │    same source into one request      │
    │                             │    (5.2.9: same endpoint, mode,      │
    │                             │    tenant, alias, headers, localOnly)│
    │                             │                                      │
    │                             │ 5. narrow each forward to what its   │
    │                             │    registration declares (4.3.6.1)   │
    │                             │                                      │
    │                             │ 6. fan out, ANTARES_FED_FANOUT (8)   │
    │                             │    in flight, each bounded by the    │
    │                             │    registration's timeout            │
    │                             │                                      │
    │                             │  GET /csr/17/entities?type=…&attrs=… │
    │                             │  Via: 1.1 broker-a                   │
    │                             ├─────────────────────────────────────►│
    │                             │                        200  []       │
    │                             │◄─────────────────────────────────────┤
    │                             │      × N per query (`calls per       │
    │                             │        query` in the table)          │
    │                             │                                      │
    │                             │ 7. book the outcome on each          │
    │                             │    registration: timesSent,          │
    │                             │    timesFailed, lastSuccess,         │
    │                             │    lastFailure, status               │
    │                             │    (Table 5.2.9-2)                   │
    │                             │                                      │
    │                             │ 8. merge the halves (4.5.5),         │
    │                             │    paginate, answer                  │
    │       200 + entity list     │                                      │
    │◄────────────────────────────┤                                      │

calls per query is the number that matters: it is the fan-out the registry narrowing left behind, and it multiplies everything downstream. A query that reaches 34 sources costs 34 HTTP round trips, 34 timeout budgets and 34 bookkeeping writes.

Step 7 is where the federated path meets the database, once per source call rather than once per query:

one forwarded request
      │
      └─► note_forward (federation.rs)
             └─► CurrentStateDriver::record_forward
                    └─► record_forward_via_mutate (antares-store)
                           └─► PgDocStore::mutate(Kind::Registration)
                                  ├─ SELECT … FOR UPDATE   the row lock
                                  ├─ UPDATE csource_registrations
                                  └─ csource_index: rebuilt only when the
                                     write moved a member the index is
                                     built from — the counters are not

The failed (conn/4xx/5xx) and with a source warning columns are split because they are different faults: a failed query is the broker not answering, while a warning is the broker answering after a source did not (6.3.17), which is the documented outcome and not an error.

rss.sh — the CPU and memory column on every other table

One background sampler for the whole run. It resolves the broker by the last two components of its argv[0], the Postgres container by name, and k6, the sink and mosquitto by their own names, then writes RSS and CPU for each at 1 Hz along with whole-host busy cores. Every fire.md and fed.md row folds the samples that fall inside its own window, which is why a saturation claim can be checked against the phase that made it.

report.py — the artefacts

Folds every table into index.html, perf.json and invokes dev/perf/pdf.py to build report.pdf (the narrated report explaining what was stored, who called, and the conditions of each measured number), next to the raw CSVs, so a later run can be diffed against this one without rerunning anything.

Measuring delivery without the rented runner

fire.sh needs k6, so until now the notification pipeline could only be measured by dispatching scale-weekly. dev/perf/deliver.py drives the same path with the standard library alone, against a memory-store broker and sink.py:

ANTARES_STORE=memory ANTARES_EGRESS_ALLOW_PRIVATE=true antares &
python3 dev/perf/sink.py 9800 8 &
python3 dev/perf/deliver.py --seed --tenants 10 --entities 500
python3 dev/perf/api-load.py subscriptions --count 240 --tenants 10 \
  --sink-workers 8 --broker http://127.0.0.1:9090 --sink http://127.0.0.1:9800
python3 dev/perf/deliver.py --tenants 10 --entities 500 --rate 2000

The number to read is matches_per_second. One match is one (subscription, entity) pair: it is what the matcher evaluates and what a notification carries, so it is the unit both halves of the pipeline scale with. changes_per_second divides that by how many subscriptions each change fires, so it moves whenever the subscription set changes even though the pipeline is doing identical work.

What it measures, on twelve cores with a release build, at an offered 2 000 changes per second over ten tenants and 500 entities:

Matches per second rise with the subscription count, because a change is evaluated against more subscriptions and fires more of them. The broker carries the whole offered change rate up to 480 subscriptions over ten tenants, and drops 0.15 per cent of it at 960.

Holding the subscription count at 240 and raising the offered rate instead:

The last row measures the harness rather than the broker: deliver.py runs its writers as threads in one interpreter, and it cannot produce the 6 000 changes per second it was asked for. The broker absorbed every change offered at every rate, dropped none of them, and held 2.74 of twelve cores, so the table gives a floor for the delivery bound rather than the bound itself. Reaching that bound needs the writers in separate processes.

ANTARES_DELIVERY_WIDTH is not what governs this. Over a hundred-tenant shape at 1 000 changes per second, widths of 8, 64, 256 and 1 024 deliver 7 026, 7 028, 7 026 and 7 028 matches per second, none of them dropping a change: the offered load is absorbed whole at every width, so the knob has nothing to arbitrate.

The same ladder against PostgreSQL, which is what a deployment runs (synchronous_commit=off, 1 GiB of shared buffers, a pool of 50, the database sharing the box with the broker, the harness and the sink):

A change costs an entity update and a temporal-history insert here rather than a map write, so the knee arrives near 1 671 changes per second against the memory store's 7 800. Two things about that knee matter more than its position. The broker holds 1.39 cores at it while PostgreSQL holds 2.29 on average and peaks at 4.12, so the broker is not the component that runs out — which is why a criterion written as broker core utilization at saturation cannot be met on a rig that shares one box. And goodput goes backwards past the knee: offering 4 000 changes per second delivers less than offering 2 000, with broker CPU falling as it does, because nothing sheds write load before it drives the database past what it can commit. Neither is a property of the delivery path; both are the write path underneath it.

Setting up the rented runner (two repository secrets)

perf-weekly and scale-weekly rent a Hetzner Cloud server for the run (a ccx33 and a ccx53; the design targets add a volume), register it as an ephemeral GitHub runner, and delete it afterwards; perf-janitor sweeps anything past its expiry label. Until the two secrets below exist, both workflows stop at "Create server" with an empty token, so the weekly runs stay red and no perf report is published.

1. HCLOUD_TOKEN — Hetzner Cloud API token

1. Sign in at https://console.hetzner.cloud/.
2. Create a project of its own for this (for example antares-perf) so the token cannot touch anything else, and set a spending alert on the project (Project → Billing → Alerts).
3. In that project open Security → API tokens → Generate API token. Name it github-actions, permission Read & Write (the workflow creates and deletes servers and volumes and reads the price list).
4. Copy the token once; Hetzner never shows it again.

2. RUNNER_PAT — GitHub token that can register a runner

The workflow asks the GitHub API for a runner registration token (POST /repos/<owner>/<repo>/actions/runners/registration-token), which the default GITHUB_TOKEN is not allowed to do.

1. GitHub → Settings (your account) → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token.
2. Resource owner: the account that owns this repository. Repository access: Only select repositories → this repository.
3. Repository permissions: Administration: Read and write. Nothing else.
4. Expiration: one year is the maximum; put the renewal date in your calendar, the workflow fails with HTTP 401 when it lapses.
5. Copy the token once.

A classic token with the repo scope works too, but grants far more than the workflow needs.

3. Store both as repository secrets

Repository → Settings → Secrets and variables → Actions → New repository secret, twice:

The names must match exactly; the workflows read them as ${{ secrets.HCLOUD_TOKEN }} and ${{ secrets.RUNNER_PAT }}.

4. Limits on a fresh Hetzner project

A new project starts with small quotas. perf-weekly asks for a ccx33 (8 dedicated cores) and scale-weekly for a ccx53 (32); scale-weekly at 1.0 would add a 500 GB volume, and Hetzner answers dedicated core limit exceeded / volumes size limit exceeded until the limits are raised: Project → Limits → request more dedicated cores and volume storage (a short form, usually approved within a day). The 0.01 dry run needs the core limit only; it shrinks the volume to 10 GB.

5. First run

Actions → scale-weekly → Run workflow with scale = 0.01 (about an hour, a few euros); read the step summary for the cost line and the tables. Then Actions → perf-weekly → Run workflow. Both schedules take over from there (Saturday 03:17 UTC and Sunday 02:17 UTC), and pages folds the newest bundles into /reports/perf/latest/.

If you would rather not rent hardware, disable the two workflows (Actions → workflow → ⋯ → Disable workflow) so the weekly runs stop going red.

What a façade costs

A façade for another standard answers by driving this broker's own NGSI-LD router in process (Façades for another standard). The seam's own cost is a JSON round trip: the inner answer is serialized to bytes, parsed, and re-serialized into the façade's envelope. Everything else about the request happens exactly once.

Measured with the reference façade (GET /x/example/things?kind=Vehicle) against the NGSI-LD request it wraps (GET /entities?type=Vehicle&options=keyValues), both through the same router in the same process, 200 calls each, medians of five repetitions, release build:

The comparison is in process on purpose: the number is about a serialize and a parse, and a socket between the two halves would measure the socket. dev/perf/shapes.sh runs the same pair end to end against a built binary (the facade shape, skipped unless the binary was built with the reference plugin); the table above is the per-call figure.

What it decides: the seam has almost no fixed cost — an empty answer's round trip does not clear the noise — and what it does cost is proportional to the answer, about 0.7 µs per Entity on this shape. A typed operations layer, one where a façade reached the handlers through Rust types instead of through JSON, would save exactly that and nothing else. Sixty-six microseconds on a hundred-Entity page is not a reason to build and maintain a second, typed API surface beside the HTTP one, so that box stays closed until a façade measures this as its ceiling rather than as its rounding error.

What the numbers are not

They are one machine, one request shape, one week. A different instance type invalidates the whole history, which is why both workflows pin one. No regression gate exists until the noise profile has ten runs on that instance type; until then the runs report, and the report is the evidence.

Coverage

Two jobs measure which broker code no test executes; both publish rather than gate, except for the unit floor.

A zero-count line in the merged view means no Rust test and no ETSI test procedure in any store mode ever ran it. dev/coverage-attribution.py splits the merged profile into lines only the unit tests reach, lines only the Robot suite reaches, and lines both reach, so a clause whose only witness is a unit test shows up as such (the spec-statement table is the clause-level view of the same question).

The merged table's function columns count source functions, not compiled ones. An lcov tracefile names functions mangled, so one generic appears once per instantiation and one closure once per test binary that linked it — and a binary that never linked an instantiation records it as a miss. Counted that way the workspace has about twice as many functions as it has, half of them permanently uncovered, and the merged figure cannot be compared with the floors above. The table therefore keys a function by the start line of its FN record, which reproduces what cargo llvm-cov --summary-only reports to within a point. dev/coverage-attribution.py --selftest pins that, and workspace.yml runs it.

Reading the uncovered lines

Uncovered code falls into two kinds, and they call for different work:

• Reachable but untested: a request shape no test sends yet (an optional URL parameter, a rarely combined pair of options, a tenant header on an admin route). The fix is a test, usually a Robot TP with the clause tag so the ledger picks it up.
• Needs fault injection: the error arms behind a store that fails mid-transaction, a peer that answers with a truncated body, a notification endpoint that hangs. No request from the outside reaches them on a healthy stack; they need a failing dependency (the nats_e2e and mqtt_notify tests do this for their subsystems) or a mock that misbehaves (federation tests use one).

The weekly ratchet fails the job on a drop of more than one point against the published run, so a change that removes a test's reach is visible before the badge moves.

Reproducing locally

dev/etsi-coverage.sh memory        # one store mode, same script CI runs
cargo llvm-cov --workspace --html   # the unit half only, target/llvm-cov/html

Both need cargo-llvm-cov and the llvm-tools-preview component.

Shared crates

The broker is built from crates a gateway, a PEP or a proxy can depend on without pulling in the broker: same parsing, same validation, same query semantics. Scorpio and Stellio each keep one query AST with two evaluation backends so the query path and the notification path cannot drift; here a gateway is the third consumer of that same AST.

The backends behind those traits live one folder each under crates/antares-sql/src/store/; its README.md is the procedure for adding one.

CI builds and tests each of them standalone and fails on any dependency path back into the broker, the API crate or a storage backend (shared-crates job in workspace.yml). The same job checks that antares-api itself names no storage backend: a consumer composes its two drivers and hands them to AppState::with_drivers; the built-in memory and file store behind AppState::new is a dev-dependency the crate's own tests enable.

antares-api — what a host binary compiles against

antares-api is not one of the five: it is the broker's HTTP surface, and the only crates that depend on it are hosts that run that surface — antares-broker, antares-wasm and examples/plugin-example. Its API is small on purpose, because everything a host needs it reaches through the router or through AppState.

AppState::call is how a surface reaches the NGSI-LD API without a socket: it serves one request through this broker's own router, carrying the caller's NGSILD-Tenant, NGSILD-Snapshot, Link and policy subject headers into it. There is no second data path — a façade for another standard is a translation in front of the same handlers, so negotiation, the bounds wall, tenancy, the policy seam, history and notifications apply to its callers exactly as they do to an NGSI-LD client.

At the crate root: router(state) builds the NGSI-LD router, ops_router(state) the operational one behind Admin::PATHS, and wire(&mut state) installs the notification pipeline that a read-only host never has to pay for. spawn and background_tasks() are the crate's task bookkeeping, Admin names the operational paths, ApiSurface is the trait another standard's surface implements, and GIT_HASH is what /q/health reports. AppState and TemporalRecord come from state; DeliveryPolicy, page_sink and scope_matches are re-exported from the crates that own them so a host names one dependency instead of three.

Eleven modules stay public because a host or a surface reaches into them:

Everything else is pub(crate). There is no ratchet on this: once an item is crate-visible the compiler's dead-code lint is the gate, and it fires the moment an item loses its last caller. A helper that only a test reaches is pub behind the test-kit feature, so a release build does not carry it and the lint keeps seeing the crate as the shipped binary sees it.

Stability

All five are workspace-shared, not published: publish = false, path dependencies, one version for the whole workspace. Publishing (crates.io or a git tag consumers pin) is deferred until a consumer outside this repository exists — cutting a public API before that would freeze surfaces that are still moving with the conformance work. When that happens the public surface of each crate is reviewed like an API change and the crate gets its own semver line; until then breaking changes inside the workspace are ordinary commits.

The PEP boundary

Authentication, rate limiting and request transforms stay in the gateway in front of the broker, and the broker ships no policy engine (SECURITY.md states the same boundary). It does carry a policy seam — one trait, one built-in allow-all engine, every other engine an addon crate outside the workspace (ADR-0020) — for the three decisions a gateway cannot make from outside: narrowing the query the store runs, filtering one subscription's notification, and filtering a federated result before it is rendered. These crates are how a gateway does that job with broker-identical semantics — rewrite the incoming q= with an authorization predicate (gateway_filter), refuse a payload the broker would refuse before it costs a hop (gateway_expand), or answer "would this change notify subscription X" without a broker (would_notify). Stellio's entity-level authorization (a permission CTE injected into the main query) is the reference for doing the same on the SQL side with antares-ql::sql.

API reference

Three generated references are published next to this book:

• NGSI-LD HTTP API — ETSI's own OpenAPI document (docs/openapi/ngsi-ld-api.yaml, vendored at a pinned tag), rendered with ReDoc: https://antaresbroker.joinedcontext.com/openapi/ngsi-ld.html
• Operational API — the broker's own OpenAPI document for every route outside /ngsi-ld/v1 (docs/openapi/antares-admin.yaml: /q/ and the peer wire /ex/v1/), rendered the same way: https://antaresbroker.joinedcontext.com/openapi/admin.html
• Rust crates — rustdoc for every workspace crate, the entry point for the shared crates: https://antaresbroker.joinedcontext.com/api/

The broker serves the NGSI-LD API under /ngsi-ld/v1; the prose for the routes under /q/ is the Admin API chapter. A unit test in antares-api holds the operational document to the router: every path and method it lists is mounted, and every mounted one is listed.

Extending Antares

Antares grows in three layers. Each one has a fixed seam in the code; an extension attaches to a seam, it never adds one. The rule behind every seam: a boundary is crossed once per request or once per drained batch, never once per attribute or per matched subscription. That keeps the cost of an extension at one dynamic call per request instead of one marshalling round per value.

Cargo features

Every optional capability is one cargo feature plus a registration in antares-broker. Removing a feature never touches a core crate.

The browser artifact (antares-wasm) is the one build with postgres off: it drives the same router over the memory store and the OPFS shadow.

No other capability gets a flag: the native binary serves every store mode behind one ANTARES_STORE value, so postgres stays compiled in (the browser build is the deployment that sheds it), and the NATS bus, the telemetry stack and the admin routes are runtime switches that allocate nothing until enabled. A flag earns its place with a measured saving, and cargo build -p antares-broker --no-default-features is checked in CI so the smallest build keeps compiling.

Layer 1: component drivers

The core knows two storage traits, CurrentStateDriver and TemporalDriver (crates/antares-store/src/lib.rs), and nothing about redb, sqlx or TimescaleDB. Arc<dyn Trait> is the plugin interface. The Storage drivers chapter describes what each backend persists; this section describes how one is selected and added.

The registry

build_drivers in crates/antares-broker/src/main.rs is the registry: a match from the configured names to constructors. The shelf it selects from is a list of NAMES, not an enumeration — store_shelf() chains the built-in StoreMode values with whatever backends were compiled in from outside the workspace, so adding one never edits a core crate.

• ANTARES_STORE picks the current-state backend through build_store, one arm per backend.
• ANTARES_TEMPORAL picks the history backend. Absent, or the same name as the store, means one shared instance serves both seams. none installs NoTemporal: the recorder produces nothing and temporal reads answer OperationNotSupported 422. Any other backend name builds a second store used only through its temporal half.
• Every Postgres half, primary or temporal-only, gets its own maintenance loop (partitions, retention).

Both traits carry the same two lifecycle methods. version_info() answers what the driver runs on — engine, server version, extensions — from state captured at startup, never by querying on the call: /q/health is polled, and it prints the answer as storeInfo and temporalInfo. close() is the drain: the shutdown path closes both seams, because a deployment whose ANTARES_TEMPORAL names a second backend has two pools to close. One instance can serve both seams, so it is closed through each of them and close() must be idempotent.

An unknown name is fatal at startup and lists what the binary was built with:

ANTARES_TEMPORAL: unknown backend "mongo"; built with memory|file|postgres|timescale (temporal also: none)

/q/health reports both choices as store and temporal.

Notification sinks

NotificationSink (crates/antares-notifier/src/lib.rs) is the delivery seam. A sink declares the URI schemes it serves, validates its own endpoints at subscription creation (parse_endpoint), and delivers one prepared notification (deliver). SinkRegistry keys sinks by scheme and is the only way a binding is chosen: a subscription naming a scheme no sink serves is rejected when it is created, with BadRequestData (400), and a stored row that names one is dropped rather than delivered through some other binding. HTTP is always registered; MQTT registers behind the mqtt feature. Add one with AppState::with_sink. A WebSocket binding would be a sink registration plus a router merge, with no change to a core crate.

The egress policy — allowlist, private-range and metadata-address deny, per-destination circuit breaker — runs in the caller before deliver, so a sink cannot step around the verdict on the endpoint as written. What that check cannot do is judge a name it does not resolve: under the default ANTARES_EGRESS_ALLOW_PRIVATE=true an endpoint host given as a name passes it, and the addresses the name stands for are judged where they are dialled. A sink that opens its own socket therefore owes that filter — EgressPolicy::ip_is_metadata and ip_is_private over the resolved answer, before connecting, as the MQTT binding does in checked_addr and every reqwest client does through PolicyResolver. A sink that opens no socket says so by returning false from network(); every binding shipped here returns the default true, and a unit test holds that.

API surfaces

ApiSurface (crates/antares-api/src/surface.rs) is the routing seam: a name, a prefix, an axum::Router<AppState> and a version_info() object that /q/health prints under surfaces. The broker's own operational routes are one such surface, admin, mounted at /q.

A surface may only claim a reserved prefix — /q, /x, or a path below /x. The NGSI-LD API root belongs to the spec: a surface that could shadow a spec resource would make conformance a function of deployment configuration, so AppState::with_surface refuses any other prefix, and refuses a prefix that overlaps one already mounted rather than leaving the winner to route-matching order. Both refusals are startup errors.

ANTARES_API_SURFACES names the selection, comma-separated; absent means admin. An unknown name is fatal at startup and lists the shelf, the same way an unknown backend does. A selection that leaves admin out serves no /q at all, health and readiness included, so an empty selection is refused outright.

Adding one is a struct implementing the trait plus an entry in SURFACE_SHELF (crates/antares-broker/src/main.rs); no core crate changes.

Policy engines

PolicyEngine (crates/antares-api/src/policy.rs) is the authorization seam, and it is the narrowest of the five on purpose. Authentication, rate limiting and request transforms belong to the gateway in front of the broker (SECURITY.md says so, and ADR-0020 records why); what a gateway cannot do from outside is narrow the query the store runs, project one subscription's notification, and filter a federated result before it is rendered. Those three are what an engine is for.

An engine answers two questions. decide is asked once per operation and returns Allow, Deny(reason) — a 403 carrying urn:antares:error:AccessDenied, this broker's own URN because Table 6.3.2-1 names no access-denied error — or Filter, the narrowing the answer is built under. pre_notify is asked once per notification, just before it is sent, and returns Deliver, Drop (no delivery attempt at all: timesSent and lastNotification do not move) or a Filter projecting the entities it carries.

The Operation an engine is given names the clause, the ids, types and attributes the request selects, its q, scopeQ and geo query, and a write's body — every name already expanded, so an engine never has to carry a JSON-LD context. A batch is one operation carrying every Entity id its array names, asked once before any item is written, so a rule written over ids reaches the same verdict whichever surface the caller used. The same holds for the resources that are not Entities: a Subscription, a Context Source Registration, a Snapshot, an EntityMap or a stored @context is named by its own id, whether the request carries it in the path or chose it in the body of a create. The Subject is the tenant plus the request headers ANTARES_POLICY_SUBJECT_HEADERS names, and it stays in this process: the seam strips those headers from every forwarded request and keeps them out of notifications, logs and dead letters.

A Filter is only honoured where the broker can serve less than it was asked for: retrieve and query Entities, the two temporal reads, and the EntityMap creations behind them (policy::FILTERABLE). Return one on any other operation — a create, a delete, a subscription, a registration — and the seam answers 403 rather than dropping it; decide those with Allow or Deny.

A Filter may only narrow. Its q and scopeQ are conjoined into the condition the store already had — a scopeQ by the 4.19 distribution, so a caller filtering by Scope of its own is served the Scopes both select and never a Scope the engine ruled out — and pick/omit project members out of every document the answer carries, the Entities join= reaches over a Relationship included; there is no member it can add and no row it can widen the answer to. A pick/omit name is read in the core @context, never in the caller's, so a request cannot rebind the term a rule names and step out from under it; write the rule as an IRI and it is used as written, write it as a short name and it means what the default @context says. Set restricted and a narrowed read answers Antares-Results-Restricted: true, so an operator can tell a short answer from an empty one. Three limits are the seam's, not the engine's: an operation over everything the tenant holds — delete-by-type, purge, the whole-tenant snapshot clauses — is Allow or Deny and never a Filter, because doing it to less than it says is a data-loss bug; a Filter on a notification carrying a q is a Drop, because the broker cannot re-run a query there and must not report a narrowing it did not apply; and an engine that panics or overruns ANTARES_POLICY_TIMEOUT_MS is a Deny. Fail closed is the whole posture: a deployment that wires in a broken engine loses service, never its access rules.

ANTARES_POLICY names the engine, allow-all by default. That built-in engine decides nothing, and it is the one the shipped image and every CI gate run — conformance is asserted against it, never against an addon. Selecting it attaches no engine at all, so a broker on the default pays a branch per operation instead of a boxed decision and a timer; /q/health still reports allow-all, because that is the decision it takes. An unknown name is fatal at startup and lists the shelf the binary was built with, so a typo cannot quietly serve every request wide open.

That an addon stays out of a release is a gate, not a convention: dev/check-no-addon.sh runs on every ci, before the release binary build and before the image build, and refuses a dependency tree naming a crate under examples/ or any engine or façade crate; k8s-smoke then asserts the deployed image reports allow-all and no surface beyond admin.

Adding one is a struct implementing the trait plus an entry in POLICY_SHELF (crates/antares-broker/src/main.rs); no core crate changes. Hold it to antares_api::policy::run_policy_contract (behind the test-kit feature) from the crate's own tests: it asserts through the seam the three things an engine can get wrong — it stops answering, it hands back an answer the seam has to override, or it puts a member into an answer that was not there.

examples/plugin-example/src/policy.rs is the worked one. Its rules are a JSON document named by ANTARES_POLICY_RULES, one entry per tenant:

{
  "acme": {
    "denyTypes": ["Secret"],
    "omit": ["price"],
    "q": "speed<100"
  }
}

A tenant with no entry is unrestricted. denyTypes refuses any operation naming those Entity types — the request's own selector and a write's body type, matched against the short name or the expanded IRI; omit drops those Attributes from every document served and every notification sent; q is conjoined into every query that tenant runs. Rules are read once, at startup, and rules the engine cannot read make it refuse everything rather than allow everything. examples/plugin-example/tests/policy.rs runs the contract and then answers real requests through the router; ngsi-ld-test-suite/AntaresSpecificTests/policy_engine.robot does the same against a running broker, and .github/workflows/examples.yml runs the conformance suite on allow-all and that folder on the engine, in one job.

Façades for another standard

A SensorThings, OGC API, WFS or OData surface is an ApiSurface like any other, with one extra rule: it answers by driving this broker's NGSI-LD API in process, through AppState::call, and never by reaching the store. That is the whole design. Every façade request becomes an NGSI-LD request, so negotiation, the bounds wall, tenancy, the policy seam, history and notifications happen once, in the code that already implements CIM 009 — and there is no second data path to keep in step with the first.

async fn things(State(st): State<AppState>, headers: HeaderMap) -> Response {
    let req = Request::get("/ngsi-ld/v1/entities?type=Device&options=keyValues")
        .body(Body::empty())?;
    let resp = st.call(&headers, req).await;   // the broker answers
    // ...translate the answer into the standard's own shape
}

examples/plugin-example/src/surface.rs is the worked one, and examples/plugin-example/tests/facade.rs is the contract it is held to.

The tenant rule. A façade maps its own notion of a caller — a path segment, a subdomain, a header its standard defines — to NGSILD-Tenant, and to nothing else. AppState::call copies NGSILD-Tenant, NGSILD-Snapshot, Link and the policy subject headers from the outer request when the inner one does not set them, so a façade that has no tenant notion of its own inherits the caller's and cannot get it wrong. A façade that DOES map its own sets the header on the inner request, and the copy stands aside. What a façade must never do is derive a tenant from anything the broker did not validate: NGSILD-Tenant is checked once, in negotiate, and that check is the whole of 6.3.14.

The depth rule. A façade may call the broker, and a façade may be what the broker's route reaches, so a translation can legitimately sit several frames deep. What it may not do is close the circle: a route that translates into a request its own surface serves would call itself for as long as the stack lasts. AppState::call counts the frames one task is already inside and answers the ninth with InternalError (maxInProcessCallDepth in /q/health). The count is per task, so work a handler spawns starts a new chain — a notification is not inside the request that caused it.

The representation to ask for. Most of a mapping is already an option on the NGSI-LD request, and asking for the right one is the difference between a translation and a rewrite:

The error table. A façade keeps the broker's status — it is the verdict on the operation, and the façade has no better one — and re-renders the body in its own vocabulary. A caller of a SensorThings API is not expecting Table 6.3.2-1 ProblemDetails, and a façade that passed them through would be telling its clients to parse a second error model. What each type means to a translation:

6.3.4's own statuses (411, 413, 414, 415, 406) carry no body at all, so a façade fills the message from the reason phrase rather than from a payload that is not there.

The paging map. The broker pages with limit/offset and answers with RFC 8288 Link headers, rel="next" and rel="prev", carrying the response media type; count=true adds NGSILD-Results-Count (6.3.10, 6.3.13). Each standard renders the same two facts — where the next page is, and how many there are — its own way:

The façade rewrites the broker's Link into its own form rather than re-deriving the offsets: the broker already knows whether there IS a next page, and a façade that recomputed it would page differently from the API it fronts.

The write rule. A façade writes through the NGSI-LD write resources — POST /entities, the /entityOperations/* batch endpoints, the temporal resources — and never through AppState::store. A store call skips expansion, validation, the policy seam, the history recording layer and the change hook that feeds subscriptions: an Entity written that way is in the database, absent from every notification, and absent from history. The batch endpoints are the right target for a façade that receives many records at once (an STA POST /Observations array), because one batch request is one pass through that machinery instead of N.

The four mappings

These are recorded, not built. Each becomes its own crate behind its own off-by-default feature the day a deployment names the client that needs it; until then the seam above is the deliverable, and this is the design work already done so that day is a translation exercise rather than a research one. What each mapping shows is how little of it is new code: almost every line is a query parameter the NGSI-LD API already has.

SensorThings 1.1 (OGC 18-088). The entity model maps onto Smart Data Models types: Thing and Sensor are Entities (Device), ObservedProperty is the Attribute name, Datastream is the pairing of the two — a Device and one of its Attributes — and Location and FeatureOfInterest are the Entity's location GeoProperty. Observation is the one that decides the shape of the whole façade: an Observation is an Attribute instance at a point in time, which is history, not current state. So Observations?$filter=phenomenonTime ge <t> is GET /temporal/entities?timerel=after&timeAt=<t>, and the temporal resources (6.18-6.22) carry the bulk of the API rather than /entities. The query options are near-identities: $filter is a subset of q, $expand is join=inline, $select is attrs, and $top/$skip are limit/offset. $orderby is the one that needs care — NGSI-LD orders temporal results by observedAt, so an $orderby on anything else is either refused or paid for in the façade.

OGC API — Features Core (OGC 17-069r4). The nearest to free of the four. collections is GET /types (6.25), a collection's items is GET /entities?type=<type> with Accept: application/geo+json, which the broker already renders as a FeatureCollection (6.3.15). bbox is georel=within with a Polygon geometry; datetime is the temporal API again, timerel=between for an interval and timerel=after/before for an open one. limit is limit, numberMatched comes from count=true → NGSILD-Results-Count (6.3.13), and numberReturned is the length of what was answered. Only CRS84 is offered, which is what IETF RFC 7946 fixes and what the broker stores.

WFS 2.0 (OGC 09-025r2). An XML rendering over the OGC API Features mapping above, for a client that cannot move — never a second mapping. GetCapabilities is built from /types, DescribeFeatureType from /types/{type} (6.26), and GetFeature is the Features items request with the answer serialized as GML instead of GeoJSON. A Filter element is translated into the same q subset the STA $filter uses. Nothing in a WFS façade may reach the broker except through the Features translation: two mappings for one data model is how they drift.

OData 4.01. There is no separate mapping. SensorThings' query options ARE an OData subset, so the parser that serves $filter, $orderby, $select, $top, $skip and $count for STA is the same one an OData façade uses; what differs is the entity model in front of it. An OData façade is therefore the STA crate with a different naming layer, and the day one is asked for, that is the shape to build.

How to add a storage backend

examples/plugin-example is the worked answer: a crate outside crates/ that implements CurrentStateDriver, TemporalDriver, one ApiSurface, one NotificationSink and one PolicyEngine, and reaches a running broker through one cargo feature. Read it first — it is short, and it is built and tested with the workspace so it cannot drift from the seams it demonstrates.

A backend from outside the workspace:

1. A crate depending on antares-store (the two driver traits) and, if it also brings a surface or a binding, antares-api and antares-notifier. Nothing depends on it in return.
2. Implement CurrentStateDriver and TemporalDriver for one type. Methods the backend does not support keep the trait defaults, which return an unsupported error instead of panicking. A driver may over-return from query_entities — answering decided: false hands every predicate back to the API — but it may never drop a matching row and never cross a tenant.
3. Hold it to the driver contract. antares-store's test-kit feature exports run_current_state_contract and run_temporal_contract (crates/antares-store/src/contract.rs): the rules antares-api writes against and no backend decides for itself — a missing row answers None, a mutate never inserts (ADR-0005, ETSI 047_06), a rejected mutate commits nothing, batch results align with the input ids, upsert and batch_upsert answer opposite polarities, a query never drops a matching row and never crosses a tenant, and a stored @context answers only the tenant that stored it — a Cached copy of a public document belongs to none and every tenant reaches it (ADR-0021). Call both from the crate's own tests, the way examples/plugin-example/tests/contract.rs does. A driver whose calls block on an async runtime needs a multi-threaded runtime context.
4. Register it in antares-broker: an optional dependency, one feature that turns it on, and the name in the three shelves it belongs to — store_shelf() for the backend, SURFACE_SHELF for a surface, AppState::with_sink for a binding, POLICY_SHELF for a policy engine. Each is one #[cfg(feature = …)] line. Name the environment variables the backend reads in its doc comment; dev/check-env-docs.sh requires a row for each in docs/src/configuration.md.
5. Prove it against the conformance suite, not only against the contract: run the broker with ANTARES_STORE=<name> and put the ETSI suite through it. .github/workflows/examples.yml does exactly that for the reference plugin.

A backend that wants to be a built-in instead — one of the arms the shipped binary carries — takes the same first three steps and then joins the store ladder: an arm in AnyStore (crates/antares-sql/src/store/any.rs), a value in StoreMode (crates/antares-store/src/lib.rs), an arm in build_builtin, a background job next to the expiry sweep if it needs one, a row per pairing in crates/antares-broker/tests/store_combos.rs, and a cell in .github/workflows/etsi-matrix.yml. The full preset runs seven cells today: memory, file, postgres, timescale, postgres-nats, timescale-nats and wasm-file; every cell must pass the whole suite before the backend is part of a release. The API test suite runs once per built-in store — AppState::new composes a fresh store per state from ANTARES_TEST_STORE, and workspace.yml runs cargo nextest -p antares-api under each value; that harness reaches only backends antares-api can construct, which is why an outside driver proves itself through the suite in step 5.

SQL assembled at runtime stays inside crates/antares-sql/src/. workspace.yml fails the build on AssertSqlSafe or an sqlx query built with format! anywhere else under crates/ or examples/. Integration tests under a tests/ directory are exempt: a test that creates its own scratch database has no bind-parameter alternative. Everything a request supplies reaches the database as a bind parameter, and a new backend keeps that property.

Layer 2: lifecycle hooks

Five phases exist. Extensions attach to a phase; they never define one.

Failure policy follows the hook's role. An observer (metrics, audit) fails open: a broken observer loses its own data and the request completes. A gate fails closed: a broken gate refuses, it never waves a request through. A failed temporal drain is the worked example: the write keeps its 2xx, the failure is logged, counted in antares_temporal_drain_errors_total and shown as temporalDrainErrors on /q/health.

The history gate chain shows the granularity rule. GATES is an ordered list of fn(&AppState, &TemporalEvent) -> bool; every event of a request passes the chain once, in the drain, and the surviving events reach the temporal driver in one event_list call. Adding a gate is one entry in that list. Producers and drivers do not change.

Which hooks are active, and their settings, are data: they can be reloaded at runtime from the stores the broker already has. Hook code is a cargo feature. Nothing in the typed NGSI-LD path dispatches through a generic plugin chain, so conformance stays a property of the binary, not of a deployment's configuration.

HTTP-level concerns that belong to a gateway (authentication, rate limiting, request transforms) stay in the gateway in front of the broker. The shared crates give a gateway the broker's own parsing, expansion and matching for that job; see Shared crates. Authorization is the one concern that is split: the broker ships no policy engine, but it carries the seam an engine attaches to, because a query has to be narrowed before the store answers it and a notification has to be filtered on its way out — neither is visible from in front of the broker (ADR-0020).

Layer 3: dynamic loading

Not built. Rust has no stable ABI, so a loadable driver needs a C ABI with a version check that turns a mismatch into a link error, and native modules cannot be sandboxed. When a third party who cannot recompile Antares needs a driver, the shape is either #[repr(C)] vtables over the same two traits or a WebAssembly component driver for untrusted code. Until then the traits stay the only coupling, which keeps that loader the small half of the work.

Antares in the NGSI-LD ecosystem

A compliant peer, not a fork

NGSI-LD (ETSI GS CIM 009) is the context-management standard behind FIWARE-style smart-city platforms, and its whole point is that brokers are interchangeable: the same entities, queries, subscriptions and federation registrations work against any compliant implementation. Antares is a from-scratch Rust implementation of that standard — it shares no code with Orion-LD, Scorpio or Stellio, follows their naming tradition, and federates with them over the standard distributed- operations API. A deployment can mix brokers per site and migrate between them by replaying declarative state; conformance is the contract, and Antares publishes its evidence continuously on the ETSI conformance report page.

Where Antares fits best

• Resource-constrained and edge deployments — a ~35 MiB broker reaches places a JVM stack does not: industrial gateways, in-vehicle units, one-per-site municipal boxes (deployment).
• The browser and offline-first tooling — the wasm build is an NGSI-LD broker with zero installation: training environments, demos that need no backend, per-user sandboxes, edge UIs that keep working disconnected (wasm).
• High-density multi-tenancy — one shared schema with Row-Level Security and a 10,000-tenant design target makes per-user or per-department context spaces cheap, instead of one broker per tenant.

The configuration plane

Antares deliberately stays a vanilla data-plane engine: no YAML bootstrap, no vendor config API. Declarative city configuration — entities, subscriptions, registrations, pipelines as Git-versioned manifests, reconciled through the standard API — is a companion-project concern (the "city-as-code" pattern described in operations → upgrades). That split is what keeps the broker upgradable and replaceable, and it works with any compliant broker, not just Antares.

Standards posture

Implementation is spec-first against CIM 009 V1.9.1 (one ledger file per clause under docs/spec/); suspected defects in the official test suite are proven from the spec text and raised upstream at ETSI rather than worked around. Smart Data Models payloads work as-is — see the smart-city example dataset in the repository.

Architecture Decision Records

One file per irreversible decision, numbered, never rewritten.

Index

Format

Nygard's fields — Title, Status, Context, Decision, Consequences — plus one borrowed from MADR:

• Confirmation: how compliance with the decision can be checked — a named test, a CI job, a grep, or an explicit "manual review only". Every new ADR names its own fitness check; a decision nobody can verify drifts silently.

Shorter ADRs fold Context/Decision/Consequences into prose sections, as the existing files do; the five concerns must all be answerable from the text either way.

Immutability policy

Append-with-status. An accepted ADR's body is frozen; when a decision changes, a NEW ADR supersedes it and the old one's Status line gains superseded by ADR-00XX (see ADR-0005/ADR-0013 and the reversal recorded in ADR-0007). Never edit an old ADR's Decision to match new reality — the record of what was believed, and when it stopped being true, is the point of keeping them.