Public benchmarks

Read this before quoting any number from this section. Every benchmark class below is scored separately and is never combined into a single figure. A blended score would hide exactly the classes that are unimplemented, unrun, or measuring somebody else’s system. Unrun classes are published as NOT RUN rather than omitted — a page that lists only what went well flatters by silence.

This page is an index, not a ledger. Numbers produced by Quipu’s own runners are published here and re-derivable from this repository. Numbers produced elsewhere in the stack stay in the repository that produced them and are linked with the commit that published them — they are not copied into a table here, because a copied number rots silently while its source moves on.

Extraction → ingress (Text2KGBench)

Caboodle publishes a pinned Text2KGBench run at evaluations/text2kgbench/results/2026-09-03/report.json (commit 0a1b169). Read it there; this page deliberately does not restate its score table.

What it is a measurement of, in the report’s own words (evaluation_scope):

separate boundary measurements; upstream artifact is not graph-extract output

That sentence is the whole reason this section exists rather than a row of F1 numbers on Quipu’s trust page:

• The extraction half is a frozen_upstream_replay of the dataset’s own Vicuna-13B baseline responses, hash-pinned and re-scored. It measures a third-party model on 25 cases (selection.method: first_n_in_gold_file_order, so a fixed prefix, not a random sample). It is not a measurement of Quipu, of Caboodle, or of graph-extract, and quoting its F1 as one would be wrong in both directions: it is neither our credit nor our fault.
• The ingress half is ours and is the number this project can stand behind: 127 input triples materialised to 635 quads, all conforming (ingress.write.conforms: true, count: 635), through a governed Camayoc RML write into a disposable store, with the mapping and source both hash-pinned.

So the honest one-line summary is: the pipeline ingests a third party’s extractions into a governed graph without dropping or mangling any of them. The quality of the extractions themselves is the upstream baseline’s, and improving on it is a separate, unrun benchmark.

Bulk ingest (WatDiv)

This is NOT the Oxigraph comparison. It measures one thing: how fast Quipu loads a pinned third-party dataset into a fresh store. No other engine is involved, and nothing here says anything about query latency. The comparison class below remains NOT RUN.

The measurement

Quipu ingested a 10,916,457-triple WatDiv dataset in 2,848.9 s — 3,831.8 live facts per second — into a 3,227,811,840-byte store (295.7 bytes per fact). Release build, single process, --chunk 50000, on an idle-to-moderately-loaded 20-core host with 66 GB RAM, store on ext4. The storage medium is part of the number and is worth ~45x — see below.

The population appears in the same sentence as the rate deliberately: WatDiv’s “10M” archive contains 10,916,457 triples, not 10,000,000, and a rate quoted “at 10M” would be wrong by 9% before anyone checked anything else.

Why the fact count exceeds the triple count by exactly 3: a declared ingest writes three completion markers (declared count, source digest, completion) into the graph. That identity is the load’s own anti-vacuity check — a silently truncated load cannot produce it.

Throughput is a before/after delta of live facts read from the store, never the loader’s parse count. The two differ: the parse count reports triples the parser saw, and a re-ingest of identical content parses everything and writes nothing.

What this number does NOT support

• It is not a per-triple constant. Rate varies ~10x within a single run (below), so a figure taken from part of a load is not the load’s rate. Only end-to-end figures are quoted here.
• It does not extrapolate. See the refuted hypothesis below.
• It is not a comparison. Quipu and Oxigraph share the SPARQL parser and RDF data model (spargebra, oxrdf), so any future comparison measures storage and evaluation layers, never independent engines, and must say so.

The device this was measured on, and why load average could not see it

The rate above is an ext4-on-a-shared-device number. The store lived on the host’s root filesystem, which is also the working disk for roughly twenty concurrent agent sessions. Measured 2026-09-16 on the same host, same binary, same pinned 1M artifact, changing only the store’s device:

That is a 21.6x difference from the device alone, with the engine, the dataset and the host held fixed. So the published figure characterises Quipu ingesting while sharing a disk, not Quipu’s ingest ceiling, and a reader reproducing it on a quiet volume should expect to beat it substantially.

Why the contention check below does not cover this. The next section correlates rate against host load average and finds essentially none (-0.09). That result stands — and it cannot speak to this, because load average measures CPU runnable work, not disk saturation. The two come apart routinely: during the measurement above the host sat at a load average around 4 while the root device was 78 % busy. A store starved for I/O on an otherwise unbusy machine is invisible to the instrument that section used.

This does not explain the within-run variation described below, and it is not offered as an explanation — it is an untested candidate that the published correlation was structurally unable to detect. Stated so that the absence of a load-average correlation is not read as the absence of contention.

Rate is NOT constant within a load, and the obvious explanation is wrong

Instrumented per committed chunk, the first quarter of a 10,916,457-triple load runs about 10x slower than the rest (first-quartile median ~1,099 facts/s; later quartiles at or above the instrument’s resolution). Correlation with host load average is -0.09 across 2.1-8.7 — that is, essentially none — and with position in the run +0.49, over 161 commit intervals.

The obvious reading is that the transition happens after a fixed number of facts. That is refuted. A 108,997,714-triple load of the same dataset family, same binary, same chunk size, was 14x slower at the same absolute commit count (commits 41-55: 1,263 facts/s, against 18,333 facts/s at those commits in the smaller load). Whatever causes the speed-up, it is not “N facts ingested”.

Two explanations remain untested and are recorded rather than chosen: a proportional effect (the transition at some fraction of the dataset) and working-set residency (the smaller store is 3.2 GB and caches readily; the larger is 32.3 GB). They are not equivalent — the first says the cost never amortises, the second says it amortises whenever the store fits in memory.

⚠ Every rate here is a STORAGE figure — the medium is worth ~45x

Measured on this host, same source prefix, same binary, same --chunk 50000, differing only in where the store file sits:

So the published 3,831.8 facts/s is a property of the disk at least as much as of Quipu, and the same load in memory is roughly 45x faster. Roughly 98% of wall time on ext4 is durability rather than work.

This is why every figure on this page names its host shape, build profile and storage medium together. A throughput number quoted without all three is not reproducible and not comparable: a reader on different storage will not come close, and will have no way to know why.

It also bounds what a comparison against another engine could mean here. Two engines measured on this host would be measured mostly on its disk.

Rate is not monotonic: it halves between 4.4M and 6.8M facts

A second, larger load was instrumented per committed chunk at one-second resolution and stopped deliberately at 8,100,000 of 108,997,714 triples, because the shape had become the result. Rolling ten-commit rate:

The rate rises to a peak at ~4.4M facts and then halves, and it does so while the host gets QUIETER (median load 5.33 → 3.94 across the final bands). No figure from this load is published as a result: it is an incomplete run, its ledger row is marked valid_result: false, and a rate quoted from 7% of a dataset invites a division nobody should perform.

Three explanations were tested and all three are refuted:

1. A fixed number of facts ingested. The 10,916,457-triple load was ~14× faster at the same absolute commit count. Two scales were required to test this; one cannot.
2. Host load. Correlation of rate with one-minute load average is +0.068 across a 5.2× range (2.47–12.91) over 162 commit intervals — that is, none.
3. IRI cardinality, the most mechanistic candidate: interning cost rising as distinct IRIs accumulate. Measured directly — the 108,997,714-triple dataset carries 284,093 distinct subjects in its first 2,000,000 triples against 396,970 for the 10,916,457-triple one. It has fewer, so interning predicts the opposite of what was observed.

Why the smaller dataset is faster at matched fact count is unexplained, and is published as unexplained. Three candidates are dead; proposing a fourth after seeing the data would not be a finding.

Re-deriving it

python3 benchmark/public/watdiv_ingest.py --scale 10M \
  --archive <watdiv.10M.tar.bz2> --quipu <release quipu> \
  --db <scratch>.db --output benchmark/public/results/watdiv-ingest.jsonl \
  --pins benchmark/public/results/watdiv-pins.tsv

The archive is fetched once from the published WatDiv site; the runner pins its digest on first sight and verifies it afterwards, aborting on a mismatch rather than benchmarking bytes nobody pinned. The source is streamed from the archive and never unpacked — at the 100M scale the extracted form is ~15.6 GB, which would double the footprint of a run designed to leave nothing behind.

Guards that decide whether a row may be quoted, each covered by a test in benchmark/public/test_watdiv_ingest.py:

• a non-zero exit or a contended host marks the row valid_result: false with the reason named — the row is still written, because an unlabelled fast number is the hazard, not a labelled slow one;
• an unreadable store reads as UNKNOWN rather than a zero baseline, which would otherwise inflate the delta by whatever the store already held;
• an archive with no .nt member is refused rather than silently benchmarking the first file it finds.

WatDiv 1M diagnostic checkpoint

MEASURED, and CONTROL-INVALID. This is published because the rule at the top of this page applies to invalid results exactly as it applies to unrun ones: a class that was attempted and failed its own admission controls is published as such, never quietly dropped. Do not quote any number in this section as a performance result.

Receipts: docs/design/persistence-evidence/watdiv-1m-20260914/ — 140 per-request observations in verified-requests.json, plus cgroup, count-validation and provenance records.

Why the controls are invalid

The host was outside the protocol’s admission envelope for the whole run: root disk began at 94% and rose to 96% against an 80% ceiling, swap was occupied, other workloads were active, the ten-minute thermal/frequency admission was not performed, and a temperature sensor read 86 °C during two arms. A run that violates a control is invalid, not noisy — so these figures cannot support a ranking, a throughput claim, or an admission-green claim.

The completed ingest also ran on tmpfs, after a disk attempt was terminated at 150.386 s with 50,000 facts committed. A tmpfs figure is not a persistence figure.

What was observed

All 20 top-level WatDiv v0.6 templates were included; C2, C3 and F3 timed out in both arms and S7 timed out at concurrency four, leaving 17 templates with any successful result. Successful counts and scalar-row hashes were consistent across those 17. The tmpfs ingest completed in 26.789 s over 22 transactions, producing a 338,128,896-byte database.

Scope boundary — read before quoting anything above

• No comparison was run. There is no Oxigraph arm and no other engine here. Nothing in this section is a between-engine statement of any kind.
• 1M is a scale name, not a denominator. The pinned artifact is 152,195,750 bytes, SHA-256 c158998c66e11b33bc56cf7fa3cbc9e69c1c36bf9bdd1bab447d8a64e2d8da75, and its 1,091,718 parsed triples contain 13,033 identical duplicate lines — 1,078,685 distinct source triples plus three ingest metadata facts give 1,078,688 live facts.
• The generator was unseeded, so the ARTIFACT is the pin, never the process. Regenerating at the same scale factor does not reproduce this dataset.
• RSS includes file mappings and must not be read as private heap size. Concurrent requests overlap, so their memory samples are aggregate process observations and are not attributable to a single request.
• No correctness oracle was run. Zero-row responses and matching hashes do not establish semantic correctness.
• No percentiles, and no long-tail or cache-cold claims. Each template received one serial warmup and one measured wave; this is not the peer protocol’s thousands of warm repetitions.

Performance (WatDiv/LUBM)

NOT RUN. No WatDiv or LUBM query latency figures exist against Oxigraph or anything else, and none should be quoted from anywhere until a pinned runner produces them here. The bulk-ingest section above is a different class and is not a substitute: a load rate says nothing about how fast either engine answers a query.

This row exists so the absence is visible. The rule for this section is that a class with no result is published as NOT RUN and kept in the list, because the alternative — leaving it out until it looks good — is how a benchmark page stops being evidence and becomes marketing.

A comparison is PENDING, and it is not this row. A separate-process memory comparison against a RocksDB-backed Oxigraph on the same pinned dataset is scoped and not yet run. It is a memory residency measurement, not query latency, so it will not satisfy this row when it lands — it will earn its own. The diagnostic checkpoint above is likewise not a substitute: it has no second engine in it at all.

Note also that the in-process oxi_compare harness in this repository cannot fill this row. It deliberately shares the parser and data model between arms, so what it measures is storage and evaluation, never engine versus engine — and with one process there is no separate resident set to compare.

The rules this section is held to

Inherited from the benchmark programme, and stated here so a future page cannot quietly drop one:

1. Classes stay separately scored. Never blended into one compliance percentage.
2. Every number comes from a version-pinned, checked-in runner that exits non-zero on regression. just conformance-check enforces this for the conformance page: the committed ledger and the rendered page must agree.
3. Unsupported and unrun cases stay in the denominator, each with a named reason. NOT RUN is published as NOT RUN.
4. Cross-repository results are indexed and linked, never copied. A copy is a number with no owner: it cannot be re-derived from the page that shows it, and it goes stale without anyone editing it.
5. A result carries the boundary of what it measured. The extraction section above is the worked example — the same figures, published without their scope, would assert something about Quipu that nobody measured.