Knowledge Base

A knowledge base turns documents into chunks that Agents and workflows can retrieve. The resource belongs to a team, picks an embedding model at creation (plus an optional rerank model), and ingests content from several sources.

Deep-dive retrieval and ingestion pipeline

┌───────────────────────────┐     ┌────────────────────────────────────────────────────────┐
│ Upload document           │ ──► │ MarkItDown multi-format parser                         │
│ (PDF, DOCX, XLSX, HTML...)│     │ • Base64 embedded media extraction -> asset storage    │
└───────────────────────────┘     │ • Structural flattening and XSS cleaning (Bleach)      │
                                  └────────────────────────────────────────────────────────┘
                                                              │
                                                              ▼
                                  ┌────────────────────────────────────────────────────────┐
                                  │ CJK-optimized two-stage splitting engine               │
                                  │ • Escaped-newline hard split (\n\n, \n)                │
                                  │ • Zero-overlap recursive punctuation split +           │
                                  │   exact character-level trailing overlap               │
                                  └────────────────────────────────────────────────────────┘
                                                              │
                                         ┌────────────────────┴────────────────────┐
                                         ▼                                         ▼
                          ┌──────────────────────────────┐          ┌──────────────────────────────┐
                          │ PostgreSQL 17 + pg_search    │          │ Qdrant vector engine         │
                          │ • BM25 Tantivy lexical index │          │ • Dense vectors ({prefix}_{dim})│
                          └──────────────────────────────┘          └──────────────────────────────┘
                                         │                                         │
                                         └────────────────────┬────────────────────┘
                                                              ▼
                                  ┌────────────────────────────────────────────────────────┐
                                  │ Weighted Reciprocal Rank Fusion (RRF, k=60)            │
                                  │ • Single-channel fallback and shared vector caching    │
                                  └────────────────────────────────────────────────────────┘
                                                              │
                                                              ▼
                                  ┌────────────────────────────────────────────────────────┐
                                  │ Optional Cross-Encoder neural rerank                   │
                                  └────────────────────────────────────────────────────────┘
                                                              │
                                                              ▼
                                  ┌────────────────────────────────────────────────────────┐
                                  │ Context assembly and token budget protection           │
                                  │ • Per-document chunk cap and total budget control      │
                                  └────────────────────────────────────────────────────────┘

1. Document parsing and embedded media extraction:
   • Multi-format support: MarkItDown parses PDF, DOCX, PPTX, XLSX, HTML, CSV, TXT, MD, and JSON.
   • Automatic media extraction (replace_embedded_media_data_uris): base64 images embedded in a document are extracted into separate stored assets (/knowledge-bases/{kb_id}/documents/{doc_id}/media/{filename}) and replaced with lightweight URLs so vectors and token space do not balloon.
   • Structural flattening and cleaning: CSV becomes a Markdown table, JSON is flattened into dotted-path keys, and bleach strips dangerous tags.
2. CJK-aware splitting:
   • Custom separators pass through _decode_separator_escapes, which turns \n, \r, \t, and \\ back into real control characters — that is why typing \n\n in the form splits on newlines.
   • Chunks are split with zero internal overlap using a punctuation hierarchy (\n\n, \n, 。, ！, ？, . , ! , ? , ；, ; , ，, , , space, character), then exact character-level trailing slices of preceding chunks are prepended, guaranteeing deterministic token boundaries.
   • Tables and code blocks are preserved structurally: small tables stay intact, while large tables split row by row with the header repeated in every sub-chunk so character slicing cannot break the meaning.
3. Dual-engine hybrid retrieval and weighted RRF:
   • Lexical retrieval: PostgreSQL pg_search (BM25 index).
   • Dense vectors: the Qdrant vector database (collections partitioned by {prefix}_{dimension}).
   • Weighted RRF fusion: computes weighted reciprocal-rank fusion scores (defaults k = 60, dense_weight = 1.0, lexical_weight = 1.0). When one channel fails, retrieval degrades to single-channel recall.
4. Neural rerank (cross-encoder): reranking is on by default (rerank_enabled = true); the top rerank_candidate_k (default 10, range 1-100) RRF candidates go to the cross-encoder for rescoring and are filtered by the optional rerank_score_threshold (0-1, no threshold by default).
5. Context assembly and budget protection: bounded by context_token_budget (total token budget) and max_chunks_per_document (per-document chunk cap) so one long file cannot starve other retrieved results.

Chunking and embedding

• chunk_size defaults to 1000 characters (minimum 100, form maximum 2000) and chunk_overlap to 100 characters (minimum 0); both are measured in characters, not tokens, and token estimates use 4 characters ≈ 1 token.
• When separator is empty, the built-in priority separator list applies; a custom separator replaces the first primary separator while the remaining levels act as fallbacks.
• A knowledge base must pick an embedding model and dimension at creation: embedding_model_id cannot be changed afterwards, and the dimension is recorded when the first document is processed. Every later document and query must match that dimension or fail with a dimension-mismatch error. Changing the signature or dimension requires an explicit reprocess/rechunk or a new knowledge base.

Retrieval and ranking

Team visibility and cross-team sharing

• A knowledge base belongs to a team: team members access it by role, and a private knowledge base is visible only to its creator.
• Cross-team sharing is read-only: KnowledgeBaseSharePermission has exactly one value, read_only, so the team it is shared with can list and search it but cannot upload documents, change settings, or delete it. Write operations still require membership in the owning team (and a team admin where needed).
• Sharing constraints: a private knowledge base cannot be shared (private_kb_cannot_be_shared), it cannot be shared with the owning team itself, and a duplicate share returns kb_already_shared.
• On the list endpoint, include_shared defaults to true (knowledge bases shared with the current team are listed too) and own_only defaults to false; combining them switches from "including shared" to "this team only".

Retrieval lab

The Retrieval Lab runs single-configuration tests or A/B comparisons, showing each result's rank and score across the vector, full-text, fusion, and rerank stages, and can apply a local preset to the production knowledge base settings. When comparing chunking strategies, confirm the split result in the preview first, then rechunk processed documents.

Related content: Import documents, Chunking, Retrieval and quality tuning, Knowledge base settings reference.