Performance

Two scheduled runs on dev produce every performance number this project publishes; nothing in this chapter is typed in by hand. Both rent a dedicated-vCPU machine at Hetzner for the run, register it as an ephemeral GitHub runner, and delete it afterwards (perf-janitor sweeps a leaked server or volume by its expiry label). Results land under /reports/perf/latest/ with the raw CSVs next to the tables.

scale-weekly sizes the database to the box it rents: shared_buffers a quarter of RAM, maintenance_work_mem a sixteenth, the shared memory segment an eighth, and the broker's connection pool eight per core with the eight-core value pinned to the 100 it was measured at. A number from one server type therefore does not compare with a number from another; the box is named next to every table.

The shapes (perf-weekly)

Every script lives in dev/perf/ and runs on a laptop the same way it runs in CI; k6 is the only tool it needs.

The load generator shares the machine with the broker, as in every published broker table; the numbers describe that shape and nothing else, and quadrupling the concurrency shows the queue, not the broker. Reproduce them on your own box before quoting them:

cargo build --release -p antares-broker
dev/perf/startup.sh && dev/perf/shapes.sh && dev/perf/core-scale.sh && dev/perf/saturate.sh
python3 dev/perf/report.py results/perf     # index.html + perf.json

The measured ceiling

Two perf-weekly dispatches. The first, on ccx33 (8 logical, 4 physical cores, three passes) at commit b55d554, could isolate only the 1- and 2-core steps: pinning the broker to 4 cores needs 8 physical, and to 8 needs 16. The second, on ccx53 (32 logical, 16 physical cores, two passes) at commit 41610be, runs the whole ladder, and the tables below are that run. store=postgres in both; the load generator shares the machine, so every row is broker plus generator.

Core scaling, one row per store and pool: the broker pinned to N physical cores (SMT siblings excluded), the generator on the rest, query shape at 50 concurrent clients. cores used is the broker's CPU time over the window against the cores it was allotted.

The same rows as CPU spent per request (cores used over req/s), which is what the efficiency column is measuring underneath:

The saturation knee, whole box, open model:

Throughput per shape at 64, 256 and 1 024 concurrent clients, whole box:

The run predates ADR-0022: the Postgres driver still parked a thread per in-flight store call, which is what the four-figure peak threads column records. The knees and the per-request cost are what that shape delivered.

What the run says, in the order it matters:

• Scaling is close to linear to four cores (98 %, 92 % on memory; 91 %, 85 % and 90 %, 88 % on the two pools) and loses half of that at eight.
• The eighth core is used, not idle: 7.64 of 8 on memory, 7.03 and 6.47 on the two pools. What changes is the price of a request, which is flat from one core to four and then rises by about 80 % on all three stores. A cost that appears identically in the in-process store, which never parks a thread on a socket, is not the storage driver: it is contention above the store, in the path the three shapes share.
• The blocking pool is nowhere near its ceiling. The Postgres write shape parks 4 038 live OS threads at its knee against a ceiling of 11 024 (ANTARES_MAX_CONNECTIONS plus 1 024, main.rs), and the query shapes park 64 to 77. Nothing deadlocks and no shape reaches the cap.
• Pool size barely moves anything on this box. Pool 100 is 8 % faster at eight cores and 10 % slower at 1 024 concurrent query clients; both pools hold the same 1 000 rps write knee and fail at the same 1 500. The ccx33 run's "a larger pool is worse" reads as an artefact of four physical cores, not a property of the pool.
• The write path is what bends first. Both Postgres pools hold 1 000 rps and fail at 1 500 while every query shape holds the harness ceiling of 5 000 rps; the in-memory write path holds 5 000 rps on 0.47 cores.

The update shape

The ladder above is the query shape. perf-weekly run 33797374897, same ccx53 box, runs it again against the update shape — the write path with the notification pipeline behind it — once per store:

Updates do not merely scale worse than queries: past two cores they scale backwards, and eight cores serve fewer requests per second than one while burning 6.78 of them. Something serializes and the cores spend their time arriving at it.

Not the store's map lock, which is what a single-tenant ladder would blame and what tenant sharding would answer: this ladder drives one tenant, so sharding by tenant would leave every request on the same shard and change nothing. What the write path does that the query path does not is hand the worker's queue to another thread — block_in_place — for every document it writes, so the store can commit without stalling an async worker. In file mode that commit is an fsync and the hop is the point. In memory mode there is no commit: the write is a lock and a map insert, and the hop is the whole cost, paid once per write and paid more the more cores there are to hand work between. The hop is now taken only where something blocks; the paths that hold the write section for a whole scan (the 4.22 sweep, the two purges) still take it in either mode.

The exit criterion

perf-weekly run 33683839528 sets what the runtime work has to hold. A change to the request runtime, the drivers or the store keeps all of it:

• Efficiency at eight allotted physical cores, query at c50: at least 51 % on memory and 43 % on either Postgres pool.
• CPU per request at eight cores: no more than twice its one-core value on any store. The run itself sits at 1.97 (memory), 1.89 (pool 20) and 1.87 (pool 100).
• Saturation, whole box: the knee at 5 000 rps or better for every query shape and 1 000 rps or better for the Postgres write shape, p99 at the knee within 3.0 ms.
• Live OS threads at the write knee: 4 038 or fewer, against the 11 024 ceiling.

Deployment scenarios

dev/perf/scenarios.sh starts one or more brokers from the release binary (dev/perf/fleet.sh), seeds each world over the API and asks one question per scenario. MODE=check runs the conformance assertions of dev/perf/scenario-check.py against a memory-store fleet in seconds; MODE=load runs the same assertions first and then a k6 rate ladder whose every number comes from k6's summary or the sink's counters. A verdict is computed from those numbers and its note names the failing assertion. The fleet notifies a single-process sink.py of its own on port 9810: the load rig's sink on 9800 runs multi-process, and its front door only folds the workers' counters.

MODE=check STORE=memory   ./dev/perf/scenarios.sh              # all nine
MODE=load  STORE=postgres ./dev/perf/scenarios.sh loop fan-in  # a subset

The nine worlds, and what each one asks:

• S1 hot-entity: one broker, 1 000 vehicles. Concurrent partial updates on one entity (5.6.3, distinct datasetIds) versus the same load spread over all of them: are updates lost, and what does the contention cost in p99?
• S2 noisy-tenant: one broker, tenants quiet and loud. The quiet tenant's GET p99 is measured alone and under a write flood on loud (4.14 isolation, in time as well as in data).
• S3 slow-subscriber: one broker, ten fast endpoints and one that answers after 500 ms. Does the slow one hold the fast ones back, given ANTARES_DELIVERY_WIDTH_PER_TENANT?
• S4 fan-in: one broker, 50 subscriptions matching one entity. How many notifications per second does one update stream fan out to, and are all of them delivered?
• S5 hub-sources: a hub and two source brokers, registered with the tenant member (a forward never carries the client's tenant). Federated query and retrieve through the hub (4.3.6.1, 5.7.2): complete, merged, 404 for an absent id.
• S6 collision: the same entity id in the hub and a source. 4.5.5 merge of non-colliding attributes, 4.3.6.2 local data over an auxiliary source, 5.9.2.4 409 for an exclusive or redirect registration that overlaps data already held.
• S7 loop: brokers A and B registered to each other. A query terminates with each side's data once (6.3.18 Via); a write whose chain already names the receiver runs locally under an inclusive registration and answers 508 (6.3.17) when the only matching source is a redirect. An id-only write carries no type, so every type-only registration matches it: the 508 case needs a tenant whose only registration is the redirect.
• S8 distributed-subscription: a subscription at the hub, the entity at a source. 5.8.1.4: the hub plants a reduced copy at the source (registration operations includes federationOps) and a change at the source reaches the hub's subscriber, counted against the accepted updates.
• S9 ha-pair: two broker pods on one PostgreSQL database and one NATS JetStream bus (NATS_URL, PG_URL_BASE; skipped without them). Writes alternate between the pods; notifications are counted against accepted writes, so a duplicate or a loss shows as a mismatch.

The tables land in scenarios/ of the run artifact; report.py folds them into index.html and pdf.py gives each scenario a page (world, question, what CIM 009 requires, verdict, measured numbers).

The design targets (scale-weekly)

The README's target table is a design contract; this run is where each row gets its measured column. SCALE scales every count linearly, so the same rig runs at 0.0001 against a laptop's Postgres and at 1.0 on the rented box:

dev/perf/sink.py is the other end of every subscription and registration: it counts notifications and answers forwarded queries with an empty list, so the fan-out over 100,000 registrations costs the broker the matching and the HTTP round trips and nothing else.

Every run is capped at one hour: the server's TTL, the job timeout and the box's own shutdown timer agree on it. Scale 0.01 (1,000,000 entities, 100 tenants, 1,000 subscriptions, 1,000 registrations) fits with margin and is what the schedule runs; 1.0 does not fit in an hour and is a deliberate dispatch on a bigger box with the TTL raised in the workflow. Bulk load bypasses the broker (no notifications, no history), which is the documented path for initial loads in Operations.

The measured run

scale-weekly run 33863032274 on a ccx33, SCALE=0.01 with the subscription and registration counts raised to 10,000 each: 1,000,000 entities over 100 tenants. The load took 367 s for the entities (one COPY stream), 8 s for the subscriptions and 159 s for the registrations.

Subscriptions firing, over 10,000 subscriptions on 101 tenants:

Against run 33835261405, the same shape on the same server type before the per-drain @context memo landed:

Federated queries, over 10,000 registrations, every source the sink:

Resident set and CPU over the whole run, 1 085 samples about 1.3 s apart:

What the run says:

• Delivery is exact to 200 rps of writes: all 193 406 due notifications arrive, nothing is dropped, and the sink goes quiet in the same second the writes stop. The run before the memo delivered 99.9 % of the same shape, dropped 14 changes and took four seconds to drain. Write latency at that rate falls with it, 69.6 ms to 28.1 ms at the 99th percentile, because the matcher drain is no longer competing with the write path for a core.
• At 500 rps both runs are past the knee the run itself reports, and the newer one is further past it: the drop count barely moves (25 359 to 26 398) but what survives is delivered more slowly (848 to 561 POSTs/s). Cheaper matching feeds the delivery stage faster, so the queue behind it fills sooner. The limit line reads 200 rps in both runs; what changed is that 200 rps is now met exactly instead of nearly.
• The registry narrowing holds. Each tenant carries 100 registrations and a type=Vehicle query reaches 34 of them — the three Vehicle-typed classes of the eight in csr.md — so the index decides the fan-out and the forward path is not a broadcast. The ratio is flat from 50 to 200 rps.
• The distributed path returns no 5xx, no 4xx and no NGSILD-Warning over 9 376 queries: every one of the 421 216 source calls was answered and folded in.
• Where the broker bends, it is not out of CPU. At the Postgres query knee it holds 3 000 rps on 2.56 of 8 cores and at the write knee 1 000 rps on 1.64; on the federated path it sits at 2.2 cores while Postgres takes 5.1 and the host runs out at 7.9. The component that saturates is the database or the machine, never the broker alone.
• A federated query costs about eight times the Postgres work of a direct query of the same shape (42.5 against 5.5 mcores). The fan-out is 34 source calls and a registry match over 10,000 rows; the direct row is one local read, so the two are not the same unit of work and the ratio is the price of federation, not a regression against a like-for-like baseline. The broker side of that comparison is not available from this run. The sampler matched broker processes against the whole of the first broker's argv[0], and the shapes, saturate and startup stages start their own brokers from the same binary by a different path form, so it counted none of them and reported the idle first process instead — zero cores through windows serving 912 req/s. saturate.md disagrees with rss.csv for the same window (2.56 cores against zero) because that stage measures the process it started itself. The sampler now matches on the last two path components, so a later run carries a broker CPU column for every stage; the rows above are left as they were measured.

How each measurement works

No rig script is a unit test. Every one starts a real broker, drives it over HTTP, and folds what came back into one Markdown table. The pieces are the same in all of them:

  dev/perf/<script>.sh
        │
        ├── starts ──► antares  (release binary, one store)  :9090
        │                 │
        │                 └── store: memory | file | postgres (docker)
        │
        ├── drives ───► k6 (a dev/perf/k6-*.js scenario)  or  python3
        │                 └── constant-arrival-rate: offered load, not
        │                     closed-loop, so a slow broker shows up as a
        │                     growing queue instead of a slower client
        │
        ├── receives ─► dev/perf/sink.py  :9800
        │                 ├── POST /…      one notification, counted
        │                 └── GET  /csr/k  one forwarded query, answered []
        │
        └── samples ──► dev/perf/rss.sh    1 Hz → rss.csv
                          reads /proc for broker, Postgres, k6, sink, host

$OUT/phase is the thread that ties them together: each script writes the stage name into it before it starts, and rss.sh copies that string into every sample it takes, so any row of rss.csv can be attributed to the stage that caused it.

startup.sh — how long a cold broker takes to answer

Runs the binary, polls GET /q/health until the first 200, records the elapsed time and reads VmRSS out of /proc/<pid>/status immediately after. Five times per store, median reported. Nothing else is running, so the number is the process itself: binary load, config parse, store open, listener bind.

shapes.sh — throughput per request shape

Starts one broker, seeds 100 five-attribute entities through the API, then runs k6-shapes.js closed-loop at a fixed number of clients: a list query at 50 and 200, a single retrieve at 50. Three runs of five seconds, median reported, p99 from the same runs. Closed-loop on purpose — this table answers "how fast is one shape", not "where does it break".

core-scale.sh — does the broker use the cores it is given?

Pins the broker to 1, 2, 4 and 8 physical cores with taskset (SMT siblings excluded) and keeps k6 on the cores left over, refusing any step where the two would share one. The in-memory store on purpose: the question is whether the broker's own work parallelises, so nothing waits on a database.

saturate.sh — where the knee is

Open model. k6-saturate.js raises the arrival rate by 500 rps every stage and keeps going until p99 passes P99_MS or the error rate passes ERR. The knee is the last stage that held both. A run that never fails a stage reports none reached, which means the ladder is shorter than the box — the answer is more stages, not a bigger conclusion.

load.sh — building the dataset

The only stage that does not go through k6:

load.sh
  ├─ sink.py 9800 8        one front door, eight worker processes behind it
  │                        (one CPython process tops out near 5 000 req/s)
  ├─ gen.py | bulk-load.sh entities, one COPY stream straight into Postgres,
  │                        bypassing the broker: no notifications, no history
  ├─ api-load.py subscriptions   through the API, eight filter classes
  └─ api-load.py registrations   through the API, eight CSR classes

Entities bypass the broker because a hundred million of them through the API would measure the loader. Subscriptions and registrations do not: they have to pass validation and land in the matcher's index, which is what the later stages exercise.

fire.sh — do the subscriptions fire, and up to what rate

k6-fire.js                broker :9090                    sink.py :9800
    │                          │                                 │
    │ PATCH …/attrs (update)   │                                 │
    │ DELETE …/entities/{id}   │                                 │
    ├─────────────────────────►│                                 │
    │              204         │ 1. write the change              │
    │◄─────────────────────────┤ 2. match it against this         │
    │                          │    tenant's subscriptions        │
    │                          │ 3. queue it (changeQueue, 1024)  │
    │                          │ 4. deliver, deliveryWidth (64)   │
    │                          │    in flight, 8 per tenant       │
    │                          │                                  │
    │                          │  POST /  {"data": [entity, …]}   │
    │                          ├─────────────────────────────────►│
    │                          │                          204     │
    │                          │◄─────────────────────────────────┤
    │                          │                                  │
    └── k6 stops ──────────────┴── fire.sh polls /stats until the │
                                   count stops moving for 5 s     │

The count due is not measured, it is derived: k6-fire.js knows which subscription classes each write should trigger and evaluates the same rule api-load.py used to create them, so delivered / due is a real ratio and not a guess. quiet after is how long the sink kept receiving once the stream stopped — the drain. dropped by broker comes from the broker's own changesDropped counter, read from /q/health before and after, so a delivery gap can be charged to the queue, to the delivery policy or to the receiver rather than left ambiguous.

The limit is the last rate that delivered 99 % with no failed operation, and the ladder stops at the first rate that misses it.

fed.sh — federated queries over the registrations

The registrations point at sink.py, which answers every forwarded query with an empty array. That is deliberate: an empty answer costs the broker the index lookup, the fan-out and the HTTP round trips and nothing else, so the row measures the federation machinery rather than a source.

k6-fed.js                    broker :9090                        sink.py :9800
    │                             │                                      │
    │ GET /entities?type=Vehicle  │                                      │
    │ NGSILD-Tenant: t42          │                                      │
    ├────────────────────────────►│                                      │
    │                             │ 1. expand the query against @context │
    │                             │                                      │
    │                             │ 2. csource_index lookup (5.12): one  │
    │                             │    SQL query, narrowing on entity     │
    │                             │    type and entity id alone. It may  │
    │                             │    only REMOVE registrations the     │
    │                             │    matcher would reject anyway, so   │
    │                             │    a NULL dimension always survives  │
    │                             │                                      │
    │                             │ 3. the matcher decides the rest in   │
    │                             │    Rust: geoQ, scopeQ, csf, the      │
    │                             │    intervals, the idPattern regex,   │
    │                             │    the Via chain                     │
    │                             │                                      │
    │                             │ 4. fold registrations naming the     │
    │                             │    same source into one request      │
    │                             │    (5.2.9: same endpoint, mode,      │
    │                             │    tenant, alias, headers, localOnly)│
    │                             │                                      │
    │                             │ 5. narrow each forward to what its   │
    │                             │    registration declares (4.3.6.1)   │
    │                             │                                      │
    │                             │ 6. fan out, ANTARES_FED_FANOUT (8)   │
    │                             │    in flight, each bounded by the    │
    │                             │    registration's timeout            │
    │                             │                                      │
    │                             │  GET /csr/17/entities?type=…&attrs=… │
    │                             │  Via: 1.1 broker-a                   │
    │                             ├─────────────────────────────────────►│
    │                             │                        200  []       │
    │                             │◄─────────────────────────────────────┤
    │                             │      × N per query (`calls per       │
    │                             │        query` in the table)          │
    │                             │                                      │
    │                             │ 7. book the outcome on each          │
    │                             │    registration: timesSent,          │
    │                             │    timesFailed, lastSuccess,         │
    │                             │    lastFailure, status               │
    │                             │    (Table 5.2.9-2)                   │
    │                             │                                      │
    │                             │ 8. merge the halves (4.5.5),         │
    │                             │    paginate, answer                  │
    │       200 + entity list     │                                      │
    │◄────────────────────────────┤                                      │

calls per query is the number that matters: it is the fan-out the registry narrowing left behind, and it multiplies everything downstream. A query that reaches 34 sources costs 34 HTTP round trips, 34 timeout budgets and 34 bookkeeping writes.

Step 7 is where the federated path meets the database, once per source call rather than once per query:

one forwarded request
      │
      └─► note_forward (federation.rs)
             └─► CurrentStateDriver::record_forward
                    └─► record_forward_via_mutate (antares-store)
                           └─► PgDocStore::mutate(Kind::Registration)
                                  ├─ SELECT … FOR UPDATE   the row lock
                                  ├─ UPDATE csource_registrations
                                  └─ csource_index: rebuilt only when the
                                     write moved a member the index is
                                     built from — the counters are not

The failed (conn/4xx/5xx) and with a source warning columns are split because they are different faults: a failed query is the broker not answering, while a warning is the broker answering after a source did not (6.3.17), which is the documented outcome and not an error.

rss.sh — the CPU and memory column on every other table

One background sampler for the whole run. It resolves the broker by the last two components of its argv[0], the Postgres container by name, and k6, the sink and mosquitto by their own names, then writes RSS and CPU for each at 1 Hz along with whole-host busy cores. Every fire.md and fed.md row folds the samples that fall inside its own window, which is why a saturation claim can be checked against the phase that made it.

report.py — the artefacts

Folds every table into index.html, perf.json and invokes dev/perf/pdf.py to build report.pdf (the narrated report explaining what was stored, who called, and the conditions of each measured number), next to the raw CSVs, so a later run can be diffed against this one without rerunning anything.

Measuring delivery without the rented runner

fire.sh needs k6, so until now the notification pipeline could only be measured by dispatching scale-weekly. dev/perf/deliver.py drives the same path with the standard library alone, against a memory-store broker and sink.py:

ANTARES_STORE=memory ANTARES_EGRESS_ALLOW_PRIVATE=true antares &
python3 dev/perf/sink.py 9800 8 &
python3 dev/perf/deliver.py --seed --tenants 10 --entities 500
python3 dev/perf/api-load.py subscriptions --count 240 --tenants 10 \
  --sink-workers 8 --broker http://127.0.0.1:9090 --sink http://127.0.0.1:9800
python3 dev/perf/deliver.py --tenants 10 --entities 500 --rate 2000

The number to read is matches_per_second. One match is one (subscription, entity) pair: it is what the matcher evaluates and what a notification carries, so it is the unit both halves of the pipeline scale with. changes_per_second divides that by how many subscriptions each change fires, so it moves whenever the subscription set changes even though the pipeline is doing identical work.

What it measures, on twelve cores with a release build, at an offered 2 000 changes per second over ten tenants and 500 entities:

Matches per second rise with the subscription count, because a change is evaluated against more subscriptions and fires more of them. The broker carries the whole offered change rate up to 480 subscriptions over ten tenants, and drops 0.15 per cent of it at 960.

Holding the subscription count at 240 and raising the offered rate instead:

The last row measures the harness rather than the broker: deliver.py runs its writers as threads in one interpreter, and it cannot produce the 6 000 changes per second it was asked for. The broker absorbed every change offered at every rate, dropped none of them, and held 2.74 of twelve cores, so the table gives a floor for the delivery bound rather than the bound itself. Reaching that bound needs the writers in separate processes.

ANTARES_DELIVERY_WIDTH is not what governs this. Over a hundred-tenant shape at 1 000 changes per second, widths of 8, 64, 256 and 1 024 deliver 7 026, 7 028, 7 026 and 7 028 matches per second, none of them dropping a change: the offered load is absorbed whole at every width, so the knob has nothing to arbitrate.

The same ladder against PostgreSQL, which is what a deployment runs (synchronous_commit=off, 1 GiB of shared buffers, a pool of 50, the database sharing the box with the broker, the harness and the sink):

A change costs an entity update and a temporal-history insert here rather than a map write, so the knee arrives near 1 671 changes per second against the memory store's 7 800. Two things about that knee matter more than its position. The broker holds 1.39 cores at it while PostgreSQL holds 2.29 on average and peaks at 4.12, so the broker is not the component that runs out — which is why a criterion written as broker core utilization at saturation cannot be met on a rig that shares one box. And goodput goes backwards past the knee: offering 4 000 changes per second delivers less than offering 2 000, with broker CPU falling as it does, because nothing sheds write load before it drives the database past what it can commit. Neither is a property of the delivery path; both are the write path underneath it.

Setting up the rented runner (two repository secrets)

perf-weekly and scale-weekly rent a Hetzner Cloud server for the run (a ccx33 and a ccx53; the design targets add a volume), register it as an ephemeral GitHub runner, and delete it afterwards; perf-janitor sweeps anything past its expiry label. Until the two secrets below exist, both workflows stop at "Create server" with an empty token, so the weekly runs stay red and no perf report is published.

1. HCLOUD_TOKEN — Hetzner Cloud API token

1. Sign in at https://console.hetzner.cloud/.
2. Create a project of its own for this (for example antares-perf) so the token cannot touch anything else, and set a spending alert on the project (Project → Billing → Alerts).
3. In that project open Security → API tokens → Generate API token. Name it github-actions, permission Read & Write (the workflow creates and deletes servers and volumes and reads the price list).
4. Copy the token once; Hetzner never shows it again.

2. RUNNER_PAT — GitHub token that can register a runner

The workflow asks the GitHub API for a runner registration token (POST /repos/<owner>/<repo>/actions/runners/registration-token), which the default GITHUB_TOKEN is not allowed to do.

1. GitHub → Settings (your account) → Developer settings → Personal access tokens → Fine-grained tokens → Generate new token.
2. Resource owner: the account that owns this repository. Repository access: Only select repositories → this repository.
3. Repository permissions: Administration: Read and write. Nothing else.
4. Expiration: one year is the maximum; put the renewal date in your calendar, the workflow fails with HTTP 401 when it lapses.
5. Copy the token once.

A classic token with the repo scope works too, but grants far more than the workflow needs.

3. Store both as repository secrets

Repository → Settings → Secrets and variables → Actions → New repository secret, twice:

The names must match exactly; the workflows read them as ${{ secrets.HCLOUD_TOKEN }} and ${{ secrets.RUNNER_PAT }}.

4. Limits on a fresh Hetzner project

A new project starts with small quotas. perf-weekly asks for a ccx33 (8 dedicated cores) and scale-weekly for a ccx53 (32); scale-weekly at 1.0 would add a 500 GB volume, and Hetzner answers dedicated core limit exceeded / volumes size limit exceeded until the limits are raised: Project → Limits → request more dedicated cores and volume storage (a short form, usually approved within a day). The 0.01 dry run needs the core limit only; it shrinks the volume to 10 GB.

5. First run

Actions → scale-weekly → Run workflow with scale = 0.01 (about an hour, a few euros); read the step summary for the cost line and the tables. Then Actions → perf-weekly → Run workflow. Both schedules take over from there (Saturday 03:17 UTC and Sunday 02:17 UTC), and pages folds the newest bundles into /reports/perf/latest/.

If you would rather not rent hardware, disable the two workflows (Actions → workflow → ⋯ → Disable workflow) so the weekly runs stop going red.

What a façade costs

A façade for another standard answers by driving this broker's own NGSI-LD router in process (Façades for another standard). The seam's own cost is a JSON round trip: the inner answer is serialized to bytes, parsed, and re-serialized into the façade's envelope. Everything else about the request happens exactly once.

Measured with the reference façade (GET /x/example/things?kind=Vehicle) against the NGSI-LD request it wraps (GET /entities?type=Vehicle&options=keyValues), both through the same router in the same process, 200 calls each, medians of five repetitions, release build:

The comparison is in process on purpose: the number is about a serialize and a parse, and a socket between the two halves would measure the socket. dev/perf/shapes.sh runs the same pair end to end against a built binary (the facade shape, skipped unless the binary was built with the reference plugin); the table above is the per-call figure.

What it decides: the seam has almost no fixed cost — an empty answer's round trip does not clear the noise — and what it does cost is proportional to the answer, about 0.7 µs per Entity on this shape. A typed operations layer, one where a façade reached the handlers through Rust types instead of through JSON, would save exactly that and nothing else. Sixty-six microseconds on a hundred-Entity page is not a reason to build and maintain a second, typed API surface beside the HTTP one, so that box stays closed until a façade measures this as its ceiling rather than as its rounding error.

What the numbers are not

They are one machine, one request shape, one week. A different instance type invalidates the whole history, which is why both workflows pin one. No regression gate exists until the noise profile has ten runs on that instance type; until then the runs report, and the report is the evidence.