Storage drivers

The broker holds storage behind two object-safe traits in the dependency-free antares-store crate (crates/antares-store/src/lib.rs, ADR-0013): the core names the traits and no backend.

AppState carries one Arc<dyn CurrentStateDriver> and one Arc<dyn TemporalDriver>. Both traits keep generic mutate<T, E> ergonomics through the *Ext extension traits over a boxed mutate_boxed. Dynamic dispatch runs a handful of times per request against a database round trip or a JSON-LD expansion, so it costs nothing measurable.

The store ladder

memory → file → postgres → timescale (ADR-0004): one binary, one configuration value. The mem/ and pg/ folders under crates/antares-sql/src/store/ are the two implementations; any.rs is the AnyStore dispatcher that implements both traits. The folder README (crates/antares-sql/src/store/README.md) lists the steps for adding a backend.

file is not a second store: the redb shadow holds one table per resource family (entities, subscriptions, csource_registrations, csource_subscriptions, temporal_entities, jsonld_contexts, snapshots, entity_map_docs, dist_subs, dead_letters, plus meta for the format version), keyed tenant\0id, value the expanded JSON. Every commit is Durability::Immediate inside the store's write-critical section, so the fsync completes before the HTTP answer leaves. A failed commit aborts the process rather than acknowledge a write the file does not hold. Boot rebuilds the maps from the file and refuses a format-version mismatch. redb keeps an exclusive lock: one broker per volume, stop-copy backup, Recreate-only rollouts. Measured on the dev box with 1.5 KB entities: about 3,100 fsynced writes per second, commit p50 0.21 ms, p99 0.85 ms.

Choosing the temporal driver

ANTARES_TEMPORAL names the history backend and defaults to following ANTARES_STORE. Any pairing works: file current state with timescale history, postgres current state with memory history. A backend different from the store builds a second store instance used only for history; a Postgres half runs its own maintenance and retention job wherever the current state lives, and /q/health reports both halves (store, temporal).

ANTARES_TEMPORAL=none installs the NoTemporal driver: supported() is false, the recorder and the bookkeeping paths become no-ops, and every client-facing temporal read answers OperationNotSupported with status 422 (CIM 009 Table 6.3.2-1):

{
  "type": "https://uri.etsi.org/ngsi-ld/errors/OperationNotSupported",
  "title": "OperationNotSupported",
  "status": 422,
  "detail": "no temporal store is configured"
}

What enters history

History is fed after the response, from a queue of TemporalEvents the entity endpoints produce. An event enters history only when every gate admits it (crates/antares-api/src/history.rs):

1. Value change, in the producer: an attribute instance whose value did not change never becomes an event.
2. ANTARES_TEMPORAL_RECORD: all (the default) admits everything; observed keeps only instances carrying observedAt, so a write without one updates current state and leaves no history; none records nothing from the entity endpoints while the temporal API itself still stores what it is given; ruled admits what all does, but only for an Entity a retention rule opts in with a history duration.
3. New observation, in the Postgres and TimescaleDB stores (ANTARES_TEMPORAL_DEDUP, default on): an instance with an observedAt is one row per (Attribute, datasetId, observedAt), and a new value at the same instant corrects that row in place; an instance without one is written only when its value differs from the newest stored instance. Gate 1 compares with the current entity; this one compares with history, so an upsert that recreates an Attribute with an unchanged value adds no row. Unchanged values are not repeated in the temporal representation (CIM 009 4.5: a static Property is one instance). off writes a row per admitted event.

observed is a narrowing, not a tidying: an Entity created through the Core API with no observedAt has no temporal evolution at all under it, and a temporal query over such an Entity answers empty rather than answering with its values. That is why all is the default — five of the ETSI temporal conformance tests create their fixtures without an observedAt and expect to read the history back. Choose observed only where every producer stamps its own observation times.

A drain that fails leaves the client's 2xx standing and increments temporalDrainErrors on /q/health. ANTARES_TEMPORAL_RETENTION_DAYS prunes attribute instances older than the window from the maintenance job on the temporal half; unset keeps everything. The migration sets no retention on purpose: a schema that silently drops data is the wrong surprise.

Measured storage cost

Measured on a 3.9 KB expanded entity (1.5 KB compact), and on attribute instances of the same shape:

Two consequences shape the design. The in-memory Value costs nine times its text, so memory-mode capacity is bounded by RAM, not by the file. And expanded JSON-LD costs 4.8 times the compact text but only 1.8 times on disk, because Postgres compression absorbs the repeated IRIs; storing the compact form would buy nothing.

The instanceId lives in its column

An instance's instanceId is also the instance_id column of its history row. A new row leaves it out of data (about 70 B of a 245 B document for a bike count). Rows written before keep it there and read back as stored: a row whose data lacks instanceId is one written since. The reads aggregate each attribute's instance_id column beside its documents, in the same order, and the store puts the member back into each instance before the answer leaves it, so the API answers stay byte-identical. Building the member into each row in SQL instead cost Postgres 10-35 % more. Copies into attr_versions keep whole documents, as before.

createdAt, modifiedAt, observedAt and type stay in data. Rebuilding the two broker stamps from created_at and modified_at would save another ~75 B a row, but printing two timestamps per row cost Postgres 20-45 % more on a whole-history read of 240,000 rows, whatever the encoding (to_char, timestamp, epoch). The API echoes a client's observedAt spelling, and no column holds type.

A rollback to a release that still kept it in data serves the rows written since then without instanceId in temporal answers (the column still holds it, nothing is lost); rolling forward again restores the answers.

Integer history keys: the design

Migration 0020 built the tables, the dictionary lookup, every read over both parents and the maintenance of both. The writes route by the cut-over, which stays NULL: the keyed write still costs ~11 % more PostgreSQL CPU than the text one, so the migration that sets it ships once the ids are bound once per transaction. A history row keys on three texts: the Entity id (~60 B), the full expanded Attribute IRI (~70 B) and the broker's urn:ngsi-ld:Instance:<uuid> (57 B). Each is stored twice, in the heap and in u_attr_instances_time. The plan replaces them with an entity_no bigint and an attr_no integer from a per-Tenant dictionary of full expanded IRIs, and with an instance_no uuid.

The columns cannot change type in place: ALTER TYPE rewrites every partition of a ~110 GB history under an exclusive lock, and the partitions of one parent must all have its columns. Two shapes were measured on one synthetic week (2,000,000 rows: 2,000 docking stations, 4 Attributes, 250 instances each, identical data of 336 B, PostgreSQL 17, VACUUM ANALYZEd, bytes per row):

Reads, warm, both alternated twice: every row of the week through one index scan per (Entity, Attribute), 4.5-5.9 s today, 4.4-5.1 s for B; the newest 10 per (Entity, Attribute), 355-389 ms today, 276-287 ms for B.

Chosen: shape B, because it needs no work on the existing partitions. C needs a unique index on the new columns of every existing partition, built one CREATE INDEX CONCURRENTLY a partition as 0017 does, which reads the whole 110 GB for an index that holds nothing there. C also turns the three text columns nullable on the partitions that keep them, and its CHECK that a row holds exactly one key shape would scan every partition too. B's new parent carries NOT NULL keys from its first row.

The shape, as it is to be built:

• iri_ids (tenant_id, id bigint identity, iri): primary key (tenant_id, id), unique (tenant_id, iri), under the same per-command RLS policies as attr_instances (0005). Every lookup and every join carries the Tenant, so an id never names another Tenant's IRI.
• attr_rows: the columns of attr_instances with entity_no, attr_no and instance_no in place of the three texts, partitioned (or a hypertable) the same way, with the unique key (tenant_id, entity_no, attr_no, observed_at, instance_no).
• The cut-over is the Monday after the migration, stored in a one-row table the broker reads at boot. A row whose observed_at falls before it goes to attr_instances in the text shape, a late write included, so one instance has one conflict key. The partition job creates the weeks from the cut-over in attr_rows only. Retention drops weeks of both parents; once attr_instances holds no week, the store stops reading it.
• instance_no is the uuid of the broker's own urn:ngsi-ld:Instance:<uuid> in its canonical lowercase form. Any other id a temporal write brings (5.6.11.4) gets its uuid v5 as the key and stays in data as instanceId, which the read path already returns as stored. A v5 collision could only merge two instances that one writer created on one Attribute at one observedAt.
• The store keeps, per Tenant, the set of IRIs a committed iri_ids row holds. A write sends INSERT ... ON CONFLICT DO NOTHING RETURNING iri only for the IRIs missing there, in IRI order (a batch append sends its whole batch's first sights in one statement before any Entity), inside its own transaction, so RLS checks the Tenant. An IRI enters the set only once a later write finds it stored, because the transaction that inserted it may still roll back. A warm process sends no extra statement. A batch append sends first sights only for the Entities it finds in entities (locked FOR KEY SHARE until commit), so a skipped append leaves no dictionary row behind.
• The set is a two-generation LRU of 100 000 (Tenant, IRI) pairs per generation: about 30 MB per process when full. A pair read from the older generation moves to the newer one; when the newer one fills, the older one is dropped. A pair aged out costs its next write one INSERT ... ON CONFLICT DO NOTHING that finds the row (measured: 0.02 to 0.05 ms in PostgreSQL, 12 buffer hits, plus one round trip), after which it is known again.
• Chosen: the ids themselves are bound in SQL by iri_no in every statement, reads and writes alike, and never cached in memory. A cached id outlives a rolled-back insert or another pod's tenant purge and would key a row by an id that names nothing, which loses the row silently; iri_no is one probe of iri_ids' unique index per statement (per row for attr_no). A stale set entry instead fails the write with 23502 and clears that Tenant's entries. Options: ids in memory bound as parameters (the design above, no probe, silent loss on a stale id).
• Each history statement reads both parents as UNION ALL, one arm per parent, each arm through its own unique index. An ordered read with a LIMIT (the newest instance a write compares with, lastN's N-th newest) limits each arm itself: PostgreSQL merges two arms in index order only then, because an arm's equality on (tenant_id, entity_no, attr_no) does not make those leading index columns redundant in the parent's ordering, and it sorts every row of the Attribute otherwise. Copies into attr_versions keep whole documents, as now.
• An arm binds its ids with iri_no(tenant, iri), a STABLE invoker's-rights PL/pgSQL function the planner evaluates once per index scan. attr_rows_of(tenant, entity) and attr_rows_at(tenant, entity, attribute) return attr_rows in the row type of attr_instances; the planner inlines them, so the caller's predicates reach the btree. An instance id is the broker's urn:ngsi-ld:Instance:<uuid> unless data holds another instanceId.
• A release older than the change reads only attr_instances and would miss every row since the cut-over, so the migration that sets the cut-over is breaking_ (migrations.rollback.toml). 0020 leaves it NULL and is compatible: until it is set every row goes to attr_instances. That later migration sets it to date_trunc('week', now()) + 1 week, the partition job's own week boundary, once. The store reads it on its first write; a row whose observedAt (or the stamp that stands in for it) cannot be parsed stays in the text shape.

The keys save about 390 B a row. Getting a row under 300 B also needs the data side; today only TimescaleDB's columnstore gets there (120 B).

Short history

A type whose retention rule keeps history of PT1H or less (or, for a type without a history rule, the Tenant's * rule) writes its history rows UNLOGGED. Migration 0023 makes attr_rows a parent partitioned by the column short: attr_rows_long (the range-partitioned table 0020 built, logged) holds short = false, attr_rows_short (UNLOGGED, not range-partitioned) holds short = true. Every read and delete names attr_rows, so the planner merges both children in index order (one Merge Append, no extra arm) and the retention sweep, the 4.22 reap and a Tenant's delete reach both.

• The store learns the rule set on every reload of the retention documents (TemporalDriver::set_short_history). A short write goes to attr_rows_short whatever the cut-over; an Entity with its own historyRetention never does, and a type with a longer rule among its types keeps the Entity logged.
• A rule that grows past PT1H sends new rows to the logged side (or to attr_instances before the cut-over); the rows already written stay in attr_rows_short until the sweep drops them by the rule. A rule that shrinks to PT1H or less does the opposite. Across that one change, an observation already stored on the other side is recorded once more.
• Durability, the trade: an UNLOGGED table writes no WAL. A Postgres crash (not a clean restart) empties attr_rows_short, and a streaming replica or a failover target holds none of it, nor does a WAL-based backup or PITR. What is lost is at most the trail the sweep drops within the hour anyway; current state (entities) and every longer history stay logged. A clean broker or Postgres restart keeps the rows.
• Under TimescaleDB a hypertable can be neither unlogged nor a partition: attr_rows only gains the short column and every row stays logged.

Migrations

crates/antares-sql/migrations/, applied by the first process to start unless ANTARES_MIGRATE=0 hands the job to an init container.