Sharing & Federation

The primitive: a Quipu store can hand its knowledge to another store, and compose another store’s knowledge, without either one having to trust the other by default. Every step is explicit, hash-verified, and labelled with where it came from — so you never absorb someone else’s knowledge by accident.

That is the whole idea. The rest of this page is how it works, and what proves it.

Try the receiving half in your browser — Quipu releases can include a knowledge pack of this repository, and that page imports it with Quipu compiled to WebAssembly. You watch the manifest get verified, the bundled shapes get adopted, and the graph get staged and promoted, and then you query it — and then edit it and export a pack that declares this one as its parent, which is the lineage half of the same story. It is the transcript below, with you as the receiver.

Follow the contributor constellation walkthrough to explore the vision, a design decision, and the code it governs on a phone.

One run, end to end

This transcript is not illustrative output. It is examples/sharing-demo/expected.txt, included here verbatim. just sharing-demo creates two fresh stores and checks that a new run still matches it; the required CI Build job runs the same check.

The demo loads an explicit identifier-policy catalogue before its outward share. Its policy lives in a separate named graph, so it does not become widget data in the shared ROOT graph.

1. SHARE A             scope=root facts=2 shapes=1
2. IMPORT B            outcome=quarantined admitted=0 quarantined=2 blocker=off_vocabulary
3. ADOPT + REIMPORT    outcome=staged admitted=2 quarantined=0 promotion_eligible=true
4. PROMOTE B           outcome=promoted triples=2
5. COMPOSE B           root_names=shared
6. DIVERGE             A_adds=a-only B_adds=b-only
7. STATUS              diverged=true ours_added=1 theirs_added=1 conflicts=0
8. RECONNECT           outcome=merged asserted=1 retracted=0 provenance_parents=2
9. CONVERGED B         root_names=a-only,b-only,shared
10. BOUNDARY           provider federation unions labelled query results; it does not merge store histories

The receiver first quarantines the intact bundle because the type is not in its local vocabulary. It then adopts the bundled shape deliberately, re-imports, and promotes with a named actor. After A and B make independent additions, status proves both histories moved and merge records two provenance parents. The demo intentionally stops at the current boundary: provider federation unions labelled query results but does not merge store histories. The broader evaluation and claims belong to the arXiv submission source for the shape-aware merge paper, not to this ten-line walkthrough.

Every claim below names the command, symbol, or verbatim message that backs it, so you can check it rather than believe it. Citations are file plus symbol, not line numbers, because line numbers rot and a page that cites them stops being true without anyone editing it. Quoted messages are exact — grep for them.

Two things most graph databases do, Quipu deliberately does not. It does not merge on receipt, and it does not take a peer’s word for how trustworthy that peer is.

flowchart LR
  subgraph A["Store A — producer"]
    AR[("ROOT")]
  end
  subgraph B["Store B — consumer"]
    BQ[["quarantine graph<br/>urn:quipu:import:quarantine:…"]]
    BR[("ROOT")]
  end
  AR -->|"quipu share --output dir/"| S["share bundle<br/>export.nt · shapes.ttl · manifest.json"]
  S -->|"quipu import dir/<br/>hashes verified"| BQ
  BQ -->|"quipu import promote &lt;id&gt;<br/>an operator's explicit act"| BR
  BR -.->|"quipu status dir/"| D{{"diverged?"}}
  D -.->|"quipu merge dir/<br/>conflict ⇒ exit 2"| BR

Prepare an outward share

Outward sharing checks the outgoing bytes against the store’s block-tier identifier-policy catalogue. Load your reviewed policy before the first share; loading only your data’s SHACL shapes does not supply these rules.

For a minimal example, save this shape as identifier-policy.shapes.ttl:

@prefix sh: <http://www.w3.org/ns/shacl#> .
@prefix aegis: <http://aegis.gastown.local/ontology/> .

<urn:demo:IdentifierPolicyShape> a sh:NodeShape ;
    sh:targetClass aegis:InternalIdentifierPattern ;
    sh:nodeKind sh:IRI .

Save its rule as identifier-policy.ttl:

@prefix aegis: <http://aegis.gastown.local/ontology/> .
@prefix rdfs: <http://www.w3.org/2000/01/rdf-schema#> .

# The demo's explicit identifier policy. Stored in its own named graph so the
# shared payload and its quarantine/admission transcript remain unchanged.
<urn:demo:private-identifiers> a aegis:InternalIdentifierPattern ;
    rdfs:label "Demo private identifiers" ;
    aegis:regex "private[.]example" ;
    aegis:enforcementTier "block" .

This demonstration rule only detects private.example. For real publication, replace it with a reviewed catalogue covering the identifiers your organization needs to exclude. The checker evaluates the supplied rules; a non-empty catalogue does not establish that your policy covers every private identifier.

Load the policy into the same database that will produce the share:

quipu shapes load identifier-policy identifier-policy.shapes.ttl --db my.db
quipu knot identifier-policy.ttl --graph urn:quipu:identifier-policy --db my.db
quipu share --output graph-share --db my.db

Also load the shapes governing your application data before ingesting it. The catalogue graph is separate from ROOT, the default share scope; its location does not change which application facts travel. Each rule’s type, label, regular expression and block tier must be together in ROOT or one named graph. Fragments split across graphs do not form a usable rule.

The CLI exits 2 if there is no block-tier catalogue, 1 if a rule matches, and 0 after checking a clean payload. A refusal creates no partial output directory. --no-shapes does not bypass the catalogue check. Use --destination internal only for an intended internal transfer; it skips the outward check and stamps the resulting artifact accordingly.

Release publication rollout

Native releases can ship before the trusted repository-share producer is enabled. While the repository variable TRUSTED_REPOSITORY_SHARE_ENABLED is unset or not true, the release summary explicitly records that no new repository qpack was published. The release may therefore have no graph artifact for the explorer.

Enable that variable only after the managed trusted producer is installed and its scheduled execution has been observed. Once enabled, the release job requires the qpack, checksum and source/binary provenance receipt; missing or invalid artifacts fail the job. The first enabled release must separately prove published artifact delivery. Hosted runners never receive the private policy catalogue.

What a share is

A share is a directory you can commit to git, attach to an email, or publish as a release asset. quipu share --output <dir> writes it deterministically (src/cli_pack.rs, cmd_share), and it holds exactly three files — the same three quipu import reads back (cmd_import):

The manifest’s parent_share field (src/share.rs, ShareOptions) is what makes a share a link in a chain rather than a loose dump: it names the share this one descends from, which is what later lets quipu merge find a common base.

Shares carry their shapes on purpose. A receiving store is never asked to guess what the sender meant by a predicate — it gets the constraints alongside the facts.

Receiving: verify, quarantine, promote

Import is two verbs, and the split is the point.

quipu import ./their-share --source https://example.org/share --actor alice
quipu import promote <share-id> --actor alice

quipu import verifies before storing anything. It recomputes the payload hash and refuses a mismatch (src/share_import.rs, verify_share):

share graph hash mismatch: manifest=… actual=…

What survives verification lands in a quarantine graph named urn:quipu:import:quarantine:<hash> (staging_graph) — present in the store, queryable, and not part of ROOT. The result reports an ImportCounts split of admitted versus quarantined triples.

quipu import promote is the separate, actor-attributed step that moves a staged share into ROOT (cmd_import, the promote arm). Nothing reaches your ROOT because a file arrived; it reaches ROOT because a named person ran the second command.

Identity across stores

Two stores will call the same thing by different names. Quipu handles that in two places, and today one of them is stronger than the other.

quipu knot writes real owl:sameAs edges, so the claim “these two IRIs are one entity” is itself a fact in the graph — visible, queryable, and retractable like any other fact. That is the model the project is built around.

Import proposes; it does not merge. On an exact canonical-name match (src/share_import.rs, resolve_and_rewrite: score == 1.0 && matched_on == "canonical_name:exact"), the incoming foreign IRI is kept. The match is reported as an exact_merges entry carrying its score and matched_on, so a caller can accept the exact hits in one action — but nothing is applied to the triples, and the foreign identity survives the import intact.

Sub-exact matches are reported the same way, as candidates for review. The difference between exact and sub-exact is now the confidence attached to a proposal, not whether it is applied behind you.

accept_exact restores the older behaviour explicitly: exact matches are applied as IRI rewrites at import time. It defaults to false. Use it only where you control the naming in both stores, and note that a rewrite is still not recorded as an owl:sameAs — it remains the one path on which identity is not queryable and not retractable.

Accepting a proposal is knot’s job, and that is the point of the split: an accepted alignment becomes an owl:sameAs fact — visible, queryable, retractable — rather than a silent edit to someone else’s identifiers.

What travels: facts, graphs, whole repositories

A share is the portable graph artifact. Its scope may be a slice of facts, a whole graph, or a repository graph; a release .qpack.tar.gz is a deterministic archive of the same text bundle, not a SQLite database. The bundle contains a canonical RDF payload, SHACL shapes, and JSON plus PROV-O/DCAT/SPDX RDF manifests (src/share.rs: share_payload, manifest_turtle).

Import accepts a directory, archive, or HTTP(S) release artifact. URL and archive inputs are bounded, verified, and loaded into a fresh in-memory store without a user-visible download (src/share_transport.rs: read_reference, import_in_memory). Verification is unchanged because bytes arrived over the network: graph, shapes, and envelope hashes must agree before the store is opened.

For a modified copy, --since writes a parent-bound delta using the deliberately restricted, interoperable part of SPARQL 1.1 Update:

quipu share --out child --since <parent-share>
quipu import delta <parent-share> child

The delta contains only ground DELETE DATA and INSERT DATA. Its manifest names and hashes the immediate parent and the materialized result; the importer rejects a wrong parent, a changed update, any other Update operation, or a result digest mismatch (src/share_delta.rs: write_delta, materialize). These are local artifact writes and reads: they do not send a delta to a remote store or grant ROOT admission.

The older SQLite pack commands remain an internal/archive compatibility surface:

quipu pack <graph-iri> --out <file>     # export a graph as an attachable pack
quipu pack --verify <file>              # check one before trusting it
quipu unpack <file> [--into <graph-iri>]

(src/cli_pack.rs: cmd_pack, cmd_unpack.) --verify exists so that “is this pack intact?” is answerable before you load it rather than after. SQLite packs are no longer the published repository interchange artifact.

For archives, quipu graph freeze|thaw|list is the deep freeze surface (src/cli_graph.rs), producing read-only full-history graphs. See Graph Kinds & Deep Freeze and Knowledge Packs.

Querying across stores

A FederatedProvider composes members behind one query (src/provider/mod.rs), and two properties are what make its answers honest.

Trust is declared locally. A member’s DeclaredLabel carries only trust and freshness — in the source’s words, “the axes an operator can honestly declare about a peer” — and it is declared by the local operator, never read from the member itself (src/provider/label.rs). A remote that declares nothing does not quietly pass a configured floor: an undeclared value fails a configured trust or freshness floor, which the source describes as “fail-safe at enforcement, honest at reporting”. And because durability, policy and kind cannot be honestly declared about a remote, a remote member degrades a dataset’s coverage on those axes to partial — “the conservative reading, not an omission”.

A merge is never silent. FederatedQuery returns one ProviderOutcome per member (src/provider/mod.rs), so a partial answer is reported as a partial answer instead of arriving as a short one. And when a share has diverged, quipu merge performs a genuine three-way reconnect against the common base located through parent_share (src/share_merge.rs, locate_base — which refuses outright with “incoming share has no parent_share; three-way merge has no base”). Conflicts are detected against the SHACL cardinalities (max_counts); on conflict Quipu keeps the base value, emits a DecisionRecord, and the CLI exits 2 (src/cli_pack.rs, cmd_merge). It would rather stop and tell you than pick a winner.

Federation and SPARQL SERVICE

Quipu supports SPARQL SERVICE, restricted to endpoints the operator has configured. Queries are parsed by spargebra (src/sparql/mod.rs) and the SERVICE pattern is evaluated in src/sparql/pattern.rs under the remote feature — which server implies and the shipped full build includes (Cargo.toml).

It is narrower than SPARQL 1.1’s open federation, by design:

• Variable endpoints are refused — “variable SERVICE endpoints are refused; use a configured endpoint IRI”. A query cannot compute its target at runtime.
• Unconfigured hosts are unreachable — with no configured remotes an endpoint fails with “SERVICE endpoint ‘…’ is unavailable: no configured remotes”, or returns the seed row under SILENT.
• Every returned row is labelled — _provider always, plus _trust and _freshness where you declared them. A federated answer cannot arrive anonymous.

The pinned W3C federated-query ledger scores all seven approved SERVICE cases. Quipu implements SERVICE as a query-planned remote subquery path using the same operator-configured declarations and labels as RemoteProvider; it is not the GraphProvider whole-query fanout path, and it is not open federation. The variable-endpoint case is a deliberate policy deviation because query data cannot widen the operator’s remote allowlist. See SPARQL 1.1 conformance for the measured score and named verdicts.

The whole stack speaks it

A primitive is only first-class if the tools around it use it, rather than reaching past it. The same share format and the same verify → quarantine → promote discipline appear at every layer:

The MCP layer is worth a second look, because the discipline is in the tool contract, not merely in the documentation. quipu_import’s own description reads “Stage a v1 share bundle into a per-source named graph. Never promotes.”, and promotion is a separate tool. An agent working through MCP therefore gets the same two-step admission an operator gets at the CLI — without having to know to ask for it, and without a way to skip it by accident.

Bobbin carries the canonical fixture set — export.nt, manifest.json, shapes.ttl and the import request/response pair — so “does this bundle conform?” is a test in another repository rather than a claim made in this one.

Built vs designed

Everything described above is built and reachable from the CLI today. The gaps are stated here rather than left to be inferred from careful wording.

Producing the repository release share

The repository release producer runs on a trusted host that can reach the live identifier-policy authority. Configure QUIPU_POLICY_SERVER there; use QUIPU_POLICY_TOKEN_FILE when authentication is required. Keep both the authority configuration and catalogue on that host. A missing, unreachable, empty, truncated, or malformed catalogue refuses production. There is no fixture or internal-destination fallback for a public release.

just contributor pack <output> uses this same boundary. The producer projects policy into a temporary copy of the index, scrubs the outward share, and proves its import into a fresh receiver before making the output directory available. The source index retains no policy projection. Failure leaves no share output.

For a release, check out the exact release tag in a clean source clone and use its checksum-verified native binary. On the trusted host:

bash scripts/publish-repository-share.sh --prepare "$TAG" \
  "$QUIPU_BIN" "$BOBBIN_BIN" "$SOURCE_REPO" "$NEW_OUTPUT"

--prepare creates the text qpack archive, checksum and provenance receipt locally. --publish additionally uploads those three explicit files to the existing release. It refuses to replace existing assets. The receipt binds the archive to the release tag, source revision and native binary hash; it includes no policy content or authority address.

The hosted release job uploads native binaries first. A separately managed trusted-host trigger must then run the producer. The hosted job waits for its three assets and verifies the receipt against its own source and binary. It fails if the producer cannot deliver within the deadline. Provision and prove that trigger before enabling this workflow; a local --prepare pass alone does not establish release automation.

These receipt checks establish byte and build correspondence under the release uploader’s authority. They are not a cryptographic producer attestation and do not upgrade an import from transport to attested; those tiers are described below. Neither the catalogue nor the producer’s private logs are release assets.

What an import proves about its producer

Hash verification proves a share is intact. It says nothing about who made it. Those are different claims, and since aegis-tadzdf every import reports which one it reached.

A first import from an unknown producer reports claimed, and that is correct rather than a failure. Reaching attested requires someone to decide that this key is that producer, using quipu attest register with a key obtained some other way.

Importing a share never registers its producer. That is the design, not an omission: a key that vouches for the bundle it arrived in vouches for nothing, and an attacker replacing the whole bundle would replace the key along with it. The governance plane states the same rule for itself — quipu never self-registers.

Automated callers should require attested. Accepting claimed is defensible, but it should be a caller’s deliberate choice rather than the effect of a tier that merely does not read as failure.

Command reference: CLI — sharing.

Import and attestation metrics

The server’s /metrics endpoint exposes process-local counters at the import and signature-verification decisions:

Import outcomes are staged, quarantined, unchanged, and error. Tiers are transport, claimed, attested, and unverified. A failure before trust has been established uses unverified, even if the request supplied an envelope. A failure after verification retains the verified tier. Malformed HTTP JSON and HTTP authentication refusals do not enter the library import path; see the HTTP request counters for those failures. An import is staging, not ROOT promotion.

Verification results are ok, badsig, replay, revoked, unbound, skew, invalid, and error. invalid covers malformed envelopes and binding/domain mismatches; error covers inability to consult protected registry/replay state. skew includes expired or not-yet-valid registered sessions. No unsigned import increments verification success. A claimed signature can verify successfully without proving the producer’s identity: use the import tier for that claim. The counters also cover library/CLI callers within their own processes; only the server’s own observations appear in its scrape.

Labels never contain keys, signatures, nonces, session identities, or error text. Series appear after observation; family declarations exist before the first attempt. Counters reset with the process, so compare increments within one process_start_time_seconds interval or use reset-aware Prometheus functions. A zero or absent verification-success series is not evidence that an import has proved authorship; verify a real operation and its corresponding counter delta.