RAG ingestion with per-chunk provenance

Table of Contents

For retrieval pipelines over large, changing corpora: compress each chunk at ingest time and store the receipt with the chunk.

What you get

Every stored chunk carries:

  • integrity.raw_sha256 / integrity.output_sha256 - the byte pair linking the stored chunk to its source text.
  • headline.ratio - how much was squeezed at ingest.
  • quality.fidelity_score (verified mode) - measured retention for this chunk.

When a downstream answer is wrong, you can now triage: did retrieval miss the chunk, or did ingestion degrade the chunk that held the answer? The receipt pair answers the second question - re-run the source chunk (determinism makes the replay byte-identical) and compare against what is stored.

Honest limits

  • Ingest-time salience is query-independent. The answer-bearing sentence can be dull and get dropped. This is a documented failure mode, not a hypothetical; pair the receipt with a retrieval-quality evaluation.
  • The receipt proves what the transform did to the chunk bytes. It does not prove the chunk is semantically sufficient for queries you have not asked yet.
  • Keep the source text (or its hash-chain anchor) somewhere. The receipt references hashes, not storage.

Mode: extractive for volume, verified where the loss question matters.