Discovery Infrastructure

1. Overview

Finding, retrieving, and combining knowledge across the internet depends overwhelmingly on a handful of companies that each run their own index. This page explores a different shape: an open, federated way to find, retrieve, overlay, enrich, and integrate knowledge across many independently run sources, without any single company owning the index. It describes how several existing technologies could fit together toward that, and where genuinely new work would be needed.

Why does it matter who runs the index? Whoever runs an index decides what it covers, how results are ordered, which sources can be reached and on what terms, and whether access continues. Everyone who finds things through it, and every source that wants to be found through it, depends on those decisions. OpenHaven calls this kind of dependence on a single controlling entity capture risk: the risk that something people rely on becomes substantially controlled by one party, in ways that undermine its open or community-governed character. It is structural information, not a verdict on any company. The Centralized Platform Risks page shows what the highest level of it looks like for familiar platforms.

This is a general-purpose capability. It is not tied to any one community, subject area, or kind of data; the goal is to index and make available a wide range of data sources, information, and websites. For the situations this kind of infrastructure is meant to serve, and the wider case for decentralized architecture, see Why Decentralized Architecture.

The proposal builds on a curated directory of 60+ ecosystem maps, data sources, automation tools, and categorization schemas, compiled through ecosystem networking and mapping research. The data source analysis assesses the maps and schemas in that directory as candidate sources, including how each could be read.

The vision rests on five commitments:

1. Native sources are indexed automatically. The result is a partial cache plus a partial, federated “map of maps”: pathways to find relevant data, overlay it, enrich it, and integrate it, reusing existing technology wherever possible.
2. The system is federated, with synchronization across instances. A platform makes this practical, but it rests on protocols. At least one genuinely new protocol is probably needed, because the synchronization of index entries, cached knowledge, curation provenance, and trust between independent instances has no clean off-the-shelf standard.
3. Sources do not have to publish into it. Per-source “drivers” read each source in its native form and interpret it. This is the defining design commitment (see Section 3).
4. Semantic-web standards, AD4M, and Murmurations are likely parts of the solution, not the whole of it. ActivityPods may offer a building block for the federated platform layer, or that layer may need to be built differently.
5. The eventual specification may live outside OpenHaven. It could be a standalone effort that others adopt, extend, or steward.

However the architecture turns out, the work divides into five stages. Drivers carry out the first three for a single source; each instance carries out the last two and shares the results with other instances (Section 6).

• Ingest: pulling data from external sources through APIs, scraping, manual export, and protocol-native harvesting.
• Normalize: deduplicating and mapping heterogeneous data into a common shape.
• Enrich: applying categorization schemas, inferring relationships, and tagging geography.
• Store: persisting the normalized and enriched knowledge in whatever form each instance chooses.
• Serve: exposing it through search, traversal, and export functions that support coordination, technology mapping, and community engagement.

Section 8 describes how drivers could be developed against these stages.

2. Status of this page

Please read this page as a set of proposals.

• Nothing described here has been built or deployed. There is no reference implementation.
• No architecture has been selected. Section 5 presents three candidate architectures for comparison; none is recommended over the others.
• The two rough specs summarized in Section 7 are draft proposals, not standards, and have not been reviewed by anyone outside OpenHaven.
• Naming a technology here does not mean its authors are partners, have been consulted, or endorse anything on this page. Every description below comes from each project’s own public materials, in the tense that matches what that material says.

3. The core design question: how does knowledge get in?

The technologies in this space differ most in how knowledge enters a shared system. There are three broad approaches:

The proposal on this page follows the middle column. Nothing in the prior art reviewed below offers reaching into arbitrary, non-cooperating sources as a solved problem, and that is where the largest gap is. The approaches are not mutually exclusive at the edges: knowledge that has been harvested and normalized could also be published into other systems downstream, and a publish-into system could itself be treated as one more source to harvest.¹

4. Building blocks

Each technology below is described by what it already does, then by what this vision would add. Facts were checked against each project’s own public materials in October 2026.

4.1 Murmurations

Murmurations is a protocol in which a participant (a “node”) hosts a structured profile about itself at a URL of its choosing, validated against shared JSON Schemas, and asks for it to be recorded in a network index. Aggregators then query the index and download the profiles to build maps and directories. Its documentation describes the software as open source under GPLv3.

Its own roadmap page states that, as of December 2025, further development of Murmurations is paused while the maintainers work on other projects.² The architecture documentation mentions IPFS-based replication only as a possible future enhancement.

What the vision adds. Murmurations assumes sources publish a profile. The vision targets sources that never will. Its index-and-aggregator pattern remains a useful reference for the query layer once a driver has turned a source into a common shape.

4.2 KOI-net

KOI-net is a draft protocol for referencing and synchronizing knowledge objects between nodes. The published specification is labelled “Draft v1.0.” It defines:

• resource identifiers (RIDs); a node’s identifier is bound to a hash of its public key;
• three event types, NEW, UPDATE, and FORGET, which the specification describes as messages rather than commands, so each node decides how to respond;
• directed “edges” between nodes, delivered by webhook or polling, established through a proposal and approval handshake;
• full and partial node roles; and
• ECDSA P-256 keys and signed message envelopes.

The specification and a Python reference implementation are published under the Dynamical Systems Group organization on GitHub; the implementation is MIT-licensed.

What the vision adds. KOI-net specifies synchronization between nodes that already hold knowledge. It does not specify how an arbitrary external source becomes a knowledge object in the first place; that driver layer is the vision’s own contribution. KOI-net is the closest existing candidate for the instance-to-instance synchronization layer, and the first draft spec in Section 7 is framed as a possible profile of it.

4.3 AD4M and Coasys

AD4M, developed by Coasys, is licensed under the Cryptographic Autonomy License 1.0 and is under active development. Its documentation describes:

• Languages: pluggable protocols, each a small piece of code that defines how data is stored, retrieved, addressed, and (for shared data) synchronized. A Language runs in a sandbox, is identified by a hash of its code, and gives every piece of data an address of the form <language-hash>://<address>. A Language implements only the capabilities it needs, such as serving content or answering queries. Languages are the closest existing analogue to the “drivers” in this vision;
• Perspectives: an agent’s own knowledge graph of links (source, predicate, target), stored locally and queryable with SPARQL;
• Neighbourhoods: Perspectives shared among agents and synchronized by a “link language.” The default link language syncs over Holochain with a separate network for each Neighbourhood, and the documentation presents link languages that wrap other systems (a company database, a legacy groupware tool, or a federated protocol such as ActivityPub or Matrix) as integration scenarios. The README also describes an alternative link language that syncs through a self-hostable server over HTTP and WebSocket; and
• Multi-user hosting: a single AD4M node can serve many people, each with their own identity and isolated data, much as an email provider hosts mailboxes. The documentation says users can export their data and move to another host or to their own node.

The Language interface (version 1.0) is specified in WebAssembly Interface Types together with a prose specification, so there is a written specification for that layer.

What the vision adds. Languages adapt protocols and storage systems that are already known. The vision’s drivers must interpret uncooperative sources such as a blog, a PDF, or an out-of-date wiki, which is a harder, more extraction-dependent job. Adopting AD4M would also mean following its implementation as the source of truth rather than an independent standard, and installing a Language is a trust decision: the documentation notes that a Language can today reach any host the runtime can reach, with tighter scoping planned.

4.4 Geo, GRC-20, and Hypergraph

GRC-20 is a knowledge-graph standard, currently a draft (version 0.1.0, created November 2024) and released under CC0. Its unit is a space, a logical grouping that combines a knowledge graph with a governance or access-control process. Personal spaces can stay private or local; public spaces must be on-chain with public governance. Data is expressed as triples (entity, attribute, value) and relations, written through versioned edits or imported from another space. Geo Genesis is related Geo-ecosystem tooling. Hypergraph is a local-first TypeScript framework for building applications on GRC-20; its repository labels it a developer preview and not for production use. Its documentation describes data held in spaces owned by people or groups rather than by an application, with private data encrypted on the client and public data readable through an indexer for GRC-20 spaces.

What the vision adds. Geo is a publish-into design: data enters by authoring or importing inside a space someone already governs, never by interpreting a non-cooperating source. It is nonetheless a plausible normalization target (its triple shape resembles what drivers would emit), a plausible source to harvest in its own right, and a possible downstream endpoint for publishing curated results.

4.5 ActivityPods

ActivityPods combines ActivityPub (the W3C federation protocol) with Solid data pods. Each user gets one pod with a profile, an inbox, an outbox, and a follower list. Its homepage describes it as a “framework to create decentralized social apps” and lists three applications as being in production; it reports funding from NLnet’s NGI0 programmes.

What the vision adds. ActivityPods is a federation transport for social applications. Its materials do not discuss search or discovery either way, so whether ActivityPub-style federation could carry index, cache, or provenance data is an open question rather than something the project has addressed.

4.6 The semantic-web stack

RDF, JSON-LD, SHACL, and SPARQL are established W3C standards, and Solid builds on them. They offer a shared model for the common shape that drivers would emit (RDF serialized as JSON-LD), a way to check that shape (SHACL), and a way to query across many endpoints (SPARQL federation). The risk here is not the standards but the operating cost of running federated queries at scale.

5. How the pieces could compose

The three candidate architectures below use the same layer breakdown: harvest/driver, normalization target, storage, federated index (the “map of maps”), query routing, instance synchronization, and trust and curation.

Candidate A: a single-instance reference pipeline

This candidate is a five-layer pipeline run by one operator.

Almost everything in this candidate can be built from mature tools. What it does not provide is the federation layer, which is exactly the part this page is concerned with. Built as specified, it would give a good centralized knowledge platform, not a federated one.

Candidate B: KOI-net-node-centric

Each instance runs a KOI-net node. Drivers (new work) turn external sources into addressable knowledge objects, and NEW/UPDATE/FORGET events carry changes across negotiated edges to subscribing nodes. Each node decides for itself what to cache, index, and trust.

The main risk is that KOI-net is still a draft from one small group, so adopting it is a bet on a protocol that is not yet widely implemented. The upside is that early adopters could help shape it.

Candidate C: AD4M-centric

Each instance is an AD4M node, or a multi-user host serving many people. Drivers are Languages that wrap sources, what they find is held as links in Perspectives, and a community’s shared index is a Neighbourhood synchronized by a link language.

Candidate C would use software that exists today, and AD4M’s multi-user hosting is a working model for instances run on behalf of others. The costs are that it ties the design to one project’s implementation and, by default, to Holochain; that installing a Language is a trust decision; that Languages for uncooperative sources would still have to be written; and that it offers no way to route a query across Neighbourhoods.

No architecture is selected. None of the three is recommended over the others. Candidate B is notable because it already has a written protocol draft for the hardest piece, and Candidate C because it exists as software today.

6. Many independent instances, one network

The infrastructure this page describes would not be one system owned and run by one organization. A single operator running all the compute and storage would recreate the problem described in the Overview. The aim is many independently operated instances that synchronize with each other, so that anyone can use whichever instance they choose and reach, in principle, much the same body of indexed knowledge.

What a searcher would experience

• Choose any instance. A person, an organization, or an application would query whichever instance they trust, or the one their community runs, and get results drawn from what that instance holds and what it has received from others.
• See where results came from. Each result would carry its provenance: which source, which driver, which instance interpreted it, when, and whether a person reviewed it (Spec A). Where instances interpret a source differently, both interpretations can be shown rather than silently merged.
• Keep working when one instance fails. If an instance goes offline, changes its policies, or is run by someone a searcher does not trust, other instances would hold much of the same knowledge, and moving to one of them should not mean starting over. AD4M’s multi-user hosting is a documented analogue: hosts serve many users, each user keeps their own identity and data, and users can export their data and move to another host.
• Expect partial coverage. Each instance holds a partial cache: it learns what exists from broadcast manifests and fetches full contents on demand (Spec A). Results would be broad but not guaranteed complete, and this page does not claim otherwise.

How it could work in practice

• Who might run instances. Communities that want their own sources indexed, institutions or research groups covering a subject, libraries and archives, commercial hosts offering it as a service, and individuals. Different operators would choose different drivers and curation rules.
• What each operator takes on. Running drivers and keeping them working as sources change; storage for its cache and index; synchronization with the instances it chooses to connect to; moderation and curation of what it publishes; and responsibility for what it caches under the law of its jurisdiction (see the open question on legal limits).
• How instances connect. Each connection would be negotiated between two operators (in KOI-net terms, an edge established by proposal and approval), who decide what to share and whom to trust. A new instance would join by connecting to one or more existing ones, and any operator could drop a misbehaving instance locally, without a central decision.
• What remains unsolved. How instances are discovered in the first place (AD4M’s own hosting index is a central registry, which shows both the need and the risk); how a query reaches the right instances (see query routing in the open questions); how to keep a few large operators from becoming the new center; and who pays for running instances.

7. Rough specs

Two draft specs came out of this work. Each is deliberately rough, is marked “Draft proposal,” and is written so it could be lifted into its own repository. They are not published yet; this page summarizes them.

Spec A: Instance Sync and Provenance. Proposes that inter-instance synchronization be defined as a profile of KOI-net rather than a new transport, subject to confirming that KOI-net’s drafts allow new object types (an open question). Its central ideas:

• a subject key derived only from a source’s canonical locator, so every instance names the same source identically, kept separate from a per-instance derivation (one instance’s interpretation of that source at one time);
• the existing NEW/UPDATE/FORGET events mapped onto the derivation lifecycle, with manifests broadcast widely and full contents fetched on demand, so each instance holds a partial cache;
• mandatory provenance on every object (where it came from, when, by which driver and method, and whether a human reviewed it), aligned with W3C PROV-O;
• signed attestations (verified, disputed, superseded) as ordinary synchronized objects; and
• local trust: the spec defines how trust signals are represented and verified, not whom to trust, and conflicting interpretations of one source coexist rather than being merged.

Spec B: Driver Contract and Normalized Knowledge Object. Defines what a driver receives and returns and the shape of what it emits. Its central ideas:

• RDF serialized as JSON-LD, checked with SHACL, as the common target, with GRC-20 and Murmurations as export formats rather than the canonical form (GRC-20 is still a draft, and the mapping is not lossless);
• every assertion carries a confidence value, its basis, and a pointer into the source showing where it came from;
• explicit result statuses so that “found nothing” is never confused with “could not read” or “the source changed”; and
• firm limits on language-model-assisted extraction: no assertion without a quoted source span, validation before emission, and no silent gap-filling.

8. How drivers could be developed

A driver reads one source in its native form and emits knowledge objects in the shape Spec B defines. The work is similar from source to source, and it follows the first three stages from the Overview: ingest, normalize, and enrich. The instance then stores and serves the result. A minimal driver needs only the first two stages; enrichment can be added later.

Ingest: reading the source

• Access methods. Sources are reached through protocol-native APIs (such as the Murmurations index, Hylo’s GraphQL API, or the P2P Foundation wiki’s MediaWiki API), scheduled scrapes of structured pages, public exports (such as Kumu JSON), manual or batch imports (CSV or notes-vault exports), and geographic data APIs. The data source analysis lists the access method for each candidate source.
• Keep the raw result. Retrieved data should land in a simple staging area before anything is transformed, so that a failed normalization can be re-run without fetching the source again. Spec B’s result statuses keep “could not read” distinct from “found nothing.”
• Orchestration. Scheduling and chaining these steps can use a self-hosted workflow engine (for example n8n), a cloud automation service (for example Make), or custom scripts on a scheduler. The trade-off is operating effort against per-operation cost and flexibility.

Normalize: a common shape

• Target. The driver maps what it reads onto the normalized knowledge object in Spec B. An entity vocabulary such as person, organization, project, technology, framework, place, and media is a workable starting point for what the objects describe.
• Entity resolution. The same entity often appears under different names (“Life Itself” and “Life-Itself”). A two-pass approach works well: fuzzy string matching (for example with RapidFuzz) to find candidates, then embedding similarity (for example with sentence-transformers) to catch differently named duplicates, with borderline cases going to a human review queue. A confirmed merge keeps the source records linked to the canonical record, so provenance survives. Because independent instances may resolve differently, merges are local and conflicting interpretations coexist.
• Model-assisted resolution. For ambiguous pairs, a language model can be asked whether two records describe the same entity. It is costly at scale and must follow Spec B’s limits: no assertion without a quoted source span, and validation before emission.

Enrich: adding meaning

• Categorization. A categorization schema can be applied as a batch tagging job: define the schema’s categories, classify each relevant entity against them, and store each tag together with the schema that produced it. The data source analysis lists candidate schemas.
• Relationship inference. Beyond relationships a source declares, a driver can infer co-affiliation (shared organizations), topical proximity (nearby embeddings), geographic co-location, and schema-based affinity (shared tags across schemas). Inferred relationships should be marked as inferred, never presented as asserted by the source.
• Embeddings. Embedding each record supports semantic search and clustering, using a managed service (for example Nomic Atlas) or local models.
• Geography. Entities with locations can be tagged with bioregion and administrative boundaries, using a bioregion taxonomy such as One Earth’s, geocoding through OpenStreetMap Nominatim, and point-in-polygon lookups.

What each instance does with the result

• Store. Each instance chooses its storage. A graph store fits relationship-heavy data (examples: Neo4j, SurrealDB, Apache AGE), and a vector store handles similarity search (examples: Qdrant, pgvector, Chroma). AD4M Perspectives are the agent-centric alternative described in Candidate C.
• Serve. The functions an instance would expose are entity discovery and search; relationship traversal to a chosen depth; category and tag lookup; semantic similarity; geographic queries; a change feed for new, updated, and inferred entities; and export in standard formats (Murmurations profiles, CSV, JSON-LD), so that what instances learn can flow back out to the wider ecosystem.

9. Open questions

These are the questions the draft specs cannot settle alone.

• What do “scannable structured summarizations” contain, precisely enough to validate across independent instances that never agreed on a format?
• How much driver maintenance can be automated? Drivers break when sources change. How much repair can language models safely do, and where must a human re-review?
• Is the new synchronization layer a new protocol or a profile of KOI-net? That depends on what KOI-net’s draft permits.
• How does query routing work across a network of instances? Does every node poll every subscribed edge, or is there a smarter discovery mechanism?
• How are duplicates and conflicts handled when two independent drivers harvest the same source differently?
• How does trust get represented and verified between nodes that have never met, without becoming a popularity contest or an easy target for fake identities?
• What are the legal limits of caching and re-serving harvested content, and of handling personal data in it? These differ by jurisdiction and are not resolved here.

10. Credits

The ecosystem-mapping research this proposal grew out of was originally sponsored by the Institute of Applied Metatheory. The Collaborative Technology Alliance and DWeb communities offered helpful insights and research that contributed to this report.

11. References

Standards and specifications

• W3C. RDF 1.1 Concepts and Abstract Syntax. w3.org/TR/rdf11-concepts
• W3C. JSON-LD 1.1. w3.org/TR/json-ld11
• W3C. Shapes Constraint Language (SHACL). w3.org/TR/shacl
• W3C. SPARQL 1.1 Query Language. w3.org/TR/sparql11-query
• W3C. PROV-O: The PROV Ontology. w3.org/TR/prov-o
• W3C. Web Annotation Data Model. w3.org/TR/annotation-model
• W3C. ActivityPub. w3.org/TR/activitypub

Projects and protocols

• Murmurations. Documentation. docs.murmurations.network (architecture: about/architecture; roadmap: about/roadmap)
• KOI-net Protocol Specification, Draft v1.0. dynamicalsystemsgroup.github.io/koi-net-spec; reference implementation: github.com/DynamicalSystemsGroup/koi-net
• AD4M. Coasys. Documentation: docs.ad4m.dev (Languages, Perspectives, Neighbourhoods, and Multi-User Mode); source: github.com/coasys/ad4m; coasys.org/adam
• GRC-20 Knowledge Graph (Draft, v0.1.0). The Graph improvement proposals. github.com/graphprotocol/graph-improvement-proposals
• Hypergraph. github.com/graphprotocol/hypergraph
• ActivityPods. activitypods.org
• Solid Project. solidproject.org

Tools named in Section 8

• n8n (workflow orchestration). n8n.io
• Make (cloud workflow automation). make.com
• RapidFuzz (fuzzy string matching). github.com/maxbachmann/RapidFuzz
• sentence-transformers (local embeddings). sbert.net
• Nomic Atlas (embeddings and clustering). atlas.nomic.ai
• One Earth. Bioregion navigator. oneearth.org/navigator
• OpenStreetMap Nominatim (geocoding). nominatim.openstreetmap.org
• Neo4j. neo4j.com; SurrealDB. surrealdb.com; Apache AGE. age.apache.org
• Qdrant. qdrant.tech; pgvector. github.com/pgvector/pgvector; Chroma. trychroma.com

Footnotes

1. A third mode, consent-based federation, is a different convergence mechanism again: independent domains stay sovereign and connect only through agreed bridges. It is included in the comparison because it is a distinct design, not because any project is proposed or endorsed here. Its main limit for this vision is that it never reaches sources that do not take part. ↩

2. The roadmap page was read in October 2026. Whether development resumes, and in what form, is the maintainers’ decision. The Murmurations documentation uses “the index” in the singular while the project’s main site refers to “indexes”; this page therefore describes the pattern (an index plus aggregators that query it) without asserting how many independent indexes exist. ↩