A reference pipeline from sources to retrieval to grounded generation, including control points where governance is enforced.
Key Takeaways
- • The context pipeline is a product with quality gates.
- • Treat documents as untrusted input.
- • Retrieval quality is measurable (freshness, coverage).
End-to-end context flow
Logical flow from ingestion to grounded generation with quality gates.
Sources and ingestion
Goal: bring sources into a controlled pipeline with ownership and metadata.
- → Keep a “source registry” as the system of record.
- → Block sources without owner or classification.
- → Separate ingestion from indexing so you can re-index safely.
Sources that fail quality gates should be quarantined (held for review) rather than silently dropped or ingested with bad metadata.
Preprocessing and chunking
Chunking is an architecture decision, not a mechanical step. Attach metadata (classification, owner, version) to every chunk.
Prompt injection defense:
- • Treat all document text as untrusted.
- • Strip or flag “instruction-like” patterns (e.g., “ignore previous instructions”).
- • Never allow content to override policy or tools.
Indexing
Use hybrid retrieval by combining embeddings for semantic matching with keyword search for precision and filtering.
- • Support filtering by classification and domain
- • Support versioning and deprecation
- • Support deletion and retention enforcement
Retrieval & Ranking
Retrieval should be a controlled process consisting of:
- 1. Policy filter: Remove chunks the user is not permitted to access.
- 2. Relevance: Perform semantic and lexical ranking.
- 3. Diversity: Ensure results avoid near-duplicates.
Grounding and citations
The model must answer using retrieved sources, or explicitly say it cannot. Prefer quoting small snippets over summarizing when accuracy matters.
GCP mapping
Illustrative. Each layer maps to equivalent services on AWS, Azure, or any cloud.
Failure modes
- ! Chunks lose metadata, so access control fails.
- ! No source versioning, causing non-reproducible outputs.
- ! Index becomes stale because refresh is not automated.
- ! Documents contain injection text that affects tool use.
- ! Retrieval returns irrelevant context and model hallucinations.
Checklist
- □ Every chunk has classification, owner, version, and source pointer.
- □ Hybrid retrieval exists with policy filtering.
- □ There is a quarantine flow for failed quality gates.
- □ Source versioning is implemented and queryable.
- □ Grounding rules are enforced with citations.