io.github.fieldcure/rag

FieldCure MCP RAG Server

A Model Context Protocol (MCP) server for indexing and searching local document collections. Supports DOCX, HWPX, PDF (with OCR), Excel, PowerPoint, and audio (Whisper transcription, Windows-only), with hybrid keyword + semantic search optimized for Korean and English.

Built with C# and the official MCP C# SDK.

Commands

fieldcure-mcp-rag
├── serve         --base-path <path>                         # Multi-KB MCP search server (stdio)
├── exec          --path <kb-path> [--force] [--partial ...]  # Headless indexing for a single KB
├── exec-queue    --queue-file <path> [--sweep-all]           # Process deferred indexing queue
├── prune-orphans --base-path <path>                         # Delete orphan KB folders
└── smoke-ocr     --pdf <scanned.pdf>                        # Self-test: OCR a scanned PDF (Windows)

• serve — read-only MCP server serving all knowledge bases under the base path. Single process handles multiple KBs via kb_id parameter. Can run while exec is indexing (SQLite WAL).
• exec — scans source folders, chunks documents, contextualizes with AI, embeds, stores in SQLite. --partial re-runs only downstream stages when models change, preserving OCR output.
• exec-queue — sequential orchestrator consuming a deferred indexing queue. One entry at a time, no GPU contention. --sweep-all processes deferred entries too (used at app shutdown).
• prune-orphans — deletes orphan KB folders (GUID-named, no config.json). Protected folders (., _ prefix, -backup-) are never touched.
• smoke-ocr — diagnostic mode. Loads a scanned PDF through the OCR fallback parser, prints recognized text to stdout, and exits 0 on a non-empty result. Surfaces DllNotFoundException / BadImageFormatException distinctly so a missing or arch-mismatched native is immediately visible. Useful for verifying that the OCR native path is wired correctly on a given host (notably win-arm64 dnx installs).

Features

Search

• Hybrid BM25 + vector search with Reciprocal Rank Fusion (RRF)
• BM25-only fallback when no embedding provider is configured
• Korean-optimized chunking (sentence boundary, decimal protection, parenthesis-aware)
• SIMD-accelerated cosine similarity via System.Numerics.Vector
• FTS5 trigram index for substring and CJK-friendly keyword matching

Indexing

• Incremental indexing with SHA256 change detection
• AI-powered chunk contextualization with bilingual keyword enrichment (see Chunk Contextualization)
• 2-commit pipeline preserves expensive upstream work across embedding failures (see How Indexing Works)
• Math equation extraction from DOCX/HWPX as [math: LaTeX] blocks
• PDF with OCR fallback (Tesseract eng+kor) for scanned pages
• Audio transcription (.mp3, .wav, .m4a, .ogg, .flac, .webm) via Whisper.net — Windows-only. Model size (Tiny→Large) is auto-selected from detected GPU/RAM/cores at startup; each transcript chunk records audio.model_size and audio.transcribed_at for future reindex auditing
• Cross-process indexing lock with stale PID auto-cleanup
• Orphan cleanup for deleted files

Queue Orchestrator

• All indexing requests flow through start_reindex MCP tool — no direct exec spawn
• Scope merge rules: full ⊃ contextualization ⊃ embedding (duplicate requests upgrade, not duplicate)
• PID-based orchestrator lock with reuse defense (orchestrator.lock)
• Logical KB deletion (config.json removal) + prune-orphans physical cleanup
• Deferred indexing for app-shutdown batch processing (--sweep-all)

Operations

• Multi-KB serve: single process serves all knowledge bases under a base path, lazy-loaded per KB
• SQLite WAL mode allows search during indexing
• Graceful shutdown via cancel file
• Per-KB config.json with provider configuration

Integration

• Ollama native — embedding via /api/embed, contextualization via /api/chat with keep_alive and num_ctx support. Requires Ollama 0.4.0+.
• OpenAI-compatible — embedding via /v1/embeddings, contextualization via /v1/chat/completions. Works with OpenAI, Azure OpenAI, Groq, LM Studio, Together AI.
• Gemini native — embedding via /v1beta/models/{model}:embedContent with task_type asymmetric retrieval (RETRIEVAL_DOCUMENT / RETRIEVAL_QUERY) and Matryoshka dimension truncation (768 / 1536 / 3072). gemini-embedding-2, multilingual, 8k token input.
• Anthropic — contextualization via /v1/messages.
• API keys via environment variables — OPENAI_API_KEY, ANTHROPIC_API_KEY, etc. Batch indexing commands (exec, exec-queue) are env-var-only. Interactive MCP search can fall back to MCP elicitation when the client supports it.
• Standard MCP stdio transport (JSON-RPC over stdin/stdout)

Chunk Contextualization

Standard RAG chunking loses context — a sentence about "the protocol" becomes ambiguous when ripped from its surrounding paragraphs. This server addresses that with Unified Chunk Contextualization: a single LLM call per chunk that produces both contextual framing and bilingual (Korean + English) keywords in one pass.

The result is stored alongside the original chunk text:

• Original text is preserved for accurate retrieval display
• Contextualized text is what gets embedded and indexed in BM25
• Bilingual keywords enable cross-lingual search — a Korean query can retrieve English documents and vice versa

This is enabled by setting contextualizer in config.json. It can be disabled (set provider/model to empty) if you prefer raw chunk indexing.

How Indexing Works

The exec command runs a 5-stage pipeline per file:

1. Extract — text from document (DOCX, PDF OCR, audio transcription, etc.)
2. Chunk — split into ~1000 char windows
3. Contextualize — LLM enrichment (optional, see above)
4. Embed — vector embedding via API
5. Persist — save to SQLite

For large files, Stage 1 alone can take 20+ minutes — OCR on a 596-page scanned PDF, or Whisper transcription of a multi-hour audio recording. The first audio file in any KB also pays a one-time ggml model download (cached under {UserProfile}/.fieldcure/whisper-models/). To prevent expensive upstream work from being lost when later stages fail, the pipeline uses a 2-commit model:

Stages 1-3 (Extract → Chunk → Contextualize)
        ↓
[Commit 1] chunks saved as PendingEmbedding
        ↓
Stage 4 (Embed)
   ├─ success → [Commit 2a] promote chunks to Indexed
   └─ failure → chunks remain PendingEmbedding (retry next exec)

Why this matters: A 25-minute OCR result is persisted on disk before any embedding API call. If Stage 4 fails (network error, rate limit, token limit, process crash, even power loss), the chunks survive. The next exec hash-skips the file (no OCR re-run) and the deferred retry pass attempts only Stage 4.

Per-Chunk Failure Isolation (Binary Split)

If a single chunk in a file exceeds the embedding model's token limit (e.g., a math-dense page in a textbook), the binary split algorithm isolates that one chunk:

EmbedBatch([0..1249])         → 400 "input[846] too long"
  ├─ EmbedBatch([0..624])     → OK (promote 625)
  └─ EmbedBatch([625..1249])  → 400
      ├─ EmbedBatch([625..937])  → 400
      │   ... (binary search narrows toward chunk 846)
      │   └─ EmbedBatch([846..846]) → 400 (mark chunk 846 Failed)
      └─ EmbedBatch([938..1249]) → OK (promote 312)

Result: 1249 chunks indexed, only chunk 846 marked Failed. The file's status becomes Degraded — partially searchable instead of completely missing.

Deferred Retry Pass

Each exec ends with a retry pass over any chunks left in PendingEmbedding state from previous runs:

• Reads enriched text from DB — no OCR or contextualization re-run
• Calls the embedding API only — typically seconds, not minutes
• Up to 3 retries per chunk; on exhaustion, the chunk is marked Failed
• Auth errors (401/403) flag the provider as unavailable and skip the rest of the pass

File States

Status	Meaning	Hash-skip behavior
Ready	Fully indexed	Skip if hash matches
Degraded	Some chunks failed (binary-split isolated)	Skip if hash matches
PartiallyDeferred	Chunks pending embedding retry	Main loop skips; deferred pass picks up
Failed	Extraction or repeated embedding failure	Skip; requires --force to retry
NeedsAction	User intervention required	Skip with separate counter

Schema Versioning

Each KB DB carries a PRAGMA user_version tag. The exec command migrates older schemas automatically as part of InitializeSchema(). The serve command opens DBs read-only and never triggers migration — older-schema KBs continue to serve search queries correctly while their new-feature columns remain unused.

Installation

dotnet tool (recommended)

dotnet tool install -g FieldCure.Mcp.Rag

From source

git clone https://github.com/fieldcure/fieldcure-mcp-rag.git
cd fieldcure-mcp-rag
dotnet build

Requirements

• .NET 8.0 Runtime or later
• OCR: Windows x64 only — Tesseract OCR for scanned PDFs loads lazily on first use (Windows only). On other platforms, PDFs with embedded text work normally; scanned pages without a text layer are silently skipped.
• An embedding provider (Ollama, OpenAI, etc.) — optional, BM25 search works without it
• Ollama 0.4.0 or later (if using Ollama for embedding or contextualization)

Quick Start

Index a folder and search it without any embedding setup (BM25 only):

# 1. Install
dotnet tool install -g FieldCure.Mcp.Rag

# 2. Create a minimal config
$kbPath = "$env:LOCALAPPDATA\FieldCure\Mcp.Rag\demo"
New-Item -ItemType Directory -Force -Path $kbPath
@'
{
  "id": "demo",
  "name": "Demo KB",
  "sourcePaths": ["C:\\my-docs"]
}
'@ | Set-Content "$kbPath\config.json"

# 3. Index
fieldcure-mcp-rag exec --path $kbPath

# 4. Start the search server
fieldcure-mcp-rag serve --base-path "$env:LOCALAPPDATA\FieldCure\Mcp.Rag"

For full retrieval quality with semantic search and contextualization, add embedding and contextualizer blocks to config.json — see Usage below.

Usage

1. Create a knowledge base folder

%LOCALAPPDATA%\FieldCure\Mcp.Rag\{kb-id}\config.json

{
  "id": "my-kb-001",
  "name": "Project Docs",
  "created": "2026-04-03T00:00:00Z",
  "sourcePaths": ["C:\\Users\\me\\Documents\\project-docs"],
  "contextualizer": {
    "provider": "anthropic",
    "model": "claude-haiku-4-5-20251001",
    "apiKeyPreset": "Claude"
  },
  "embedding": {
    "provider": "openai",
    "model": "text-embedding-3-small",
    "apiKeyPreset": "OpenAI"
  }
}

API keys are resolved from environment variables: apiKeyPreset: "OpenAI" → OPENAI_API_KEY, "Claude" → ANTHROPIC_API_KEY, "Gemini" (or "Google") → GEMINI_API_KEY.

Gemini embedding example — asymmetric retrieval with 1536-dim Matryoshka truncation (50% storage of full 3072 with identical MTEB score):

"embedding": {
  "provider": "gemini",
  "model": "gemini-embedding-2",
  "apiKeyPreset": "Gemini",
  "dimension": 1536
}

Dimension	MTEB	Storage	Use case
768	67.99	25%	Storage-constrained
1536	68.17	50%	Recommended default
3072	68.17	100%	Maximum quality (pre-normalized)
In serve mode, search_documents can also prompt via MCP elicitation when the client supports it. In exec and exec-queue, missing keys must be provided via environment variables.

2. Index documents

fieldcure-mcp-rag exec --path "C:\Users\me\AppData\Local\FieldCure\Mcp.Rag\my-kb-001"

3. Start MCP search server

fieldcure-mcp-rag serve --base-path "C:\Users\me\AppData\Local\FieldCure\Mcp.Rag"

A single serve process handles all knowledge bases under the base path. Tools accept a kb_id parameter to target a specific KB.

Claude Desktop

Add to claude_desktop_config.json:

{
  "mcpServers": {
    "rag": {
      "command": "fieldcure-mcp-rag",
      "args": ["serve", "--base-path", "C:\\Users\\me\\AppData\\Local\\FieldCure\\Mcp.Rag"],
      "env": {
        "OPENAI_API_KEY": "sk-...",
        "ANTHROPIC_API_KEY": "sk-ant-..."
      }
    }
  }
}

config.json Reference

Field	Description
id	Knowledge base identifier
name	Display name
sourcePaths	List of folders to index (multiple supported)
contextualizer.provider	"anthropic", "openai", "ollama", or empty to disable
embedding.provider	"openai", "ollama", "gemini", or empty to disable
embedding.dimension	Output dimension. 0 = provider default. Gemini supports MRL truncation: 768 / 1536 / 3072.
contextualizer.model	Model ID, or empty to disable contextualization
contextualizer.apiKeyPreset	Maps to env var: "OpenAI" → OPENAI_API_KEY, "Claude" → ANTHROPIC_API_KEY
contextualizer.baseUrl	API base URL override (null = provider default)
embedding.*	Same structure as contextualizer
embedding.maxChunkChars	Max chars per chunk before pre-split (default: 4000)
embedding.batchSize	Max chunks per embedding API call (default: auto from provider table)
embedding.keepAlive	Ollama only: VRAM retention duration (default: "5m")
embedding.numCtx	Ollama only: context window tokens (default: 8192). Contextualizer only.
systemPrompt	Custom system prompt for contextualization (null = built-in default)

Tools

All tools (except list_knowledge_bases) require a kb_id parameter to specify the target knowledge base.

Tool	Description
list_knowledge_bases	List all available KBs with status (file/chunk counts, indexing status)
search_documents	Hybrid BM25 + vector search with RRF. Supports search_mode: auto, bm25, vector
get_document_chunk	Retrieve full content of a specific chunk by ID
start_reindex	Queue an indexing request. Scope merge, force/deferred flags, orchestrator auto-spawn
cancel_reindex	Remove a pending (not-yet-started) queue entry
get_index_info	Index metadata, queue state (status/position/deferred/last_error), contextualization health
check_changes	Dry-run filesystem scan. Lightweight, no API calls

Search Modes

search_mode	Behavior
auto	Hybrid when embedding available, else BM25. Recommended
bm25	Keyword-only (FTS5). No embedding call
vector	Semantic-only. Errors if no embedding provider

Supported Formats

Document formats are provided by FieldCure.DocumentParsers:

• DOCX — Microsoft Word (with math equation extraction)
• HWPX — Korean standard document (OWPML, with math equation extraction)
• XLSX — Excel spreadsheets
• PPTX — PowerPoint presentations
• PDF — PDF text extraction with ## Page N headers; OCR fallback for scanned pages (Tesseract, eng+kor)
• TXT, MD — Plain text / Markdown

Project Structure

src/FieldCure.Mcp.Rag/
├── Program.cs                     # CLI entry (exec | exec-queue | serve | prune-orphans)
├── MultiKbContext.cs              # Multi-KB manager (lazy load, Classify, lazy unload)
├── ExecQueueRunner.cs             # Deferred queue orchestrator
├── OrphanCleanupRunner.cs         # prune-orphans CLI
├── Configuration/
│   ├── RagConfig.cs               # config.json model (KeepAlive, NumCtx fields)
│   └── OllamaDefaults.cs          # Shared defaults (KeepAlive="5m", NumCtx=8192)
├── Indexing/
│   ├── IndexingEngine.cs          # 5-stage pipeline (2-commit model)
│   └── EmbeddingBatchSplitter.cs  # Binary-split per-chunk failure isolation
├── Contextualization/
│   ├── IChunkContextualizer.cs
│   ├── OpenAiChunkContextualizer.cs   # /v1/chat/completions
│   ├── OllamaChunkContextualizer.cs   # /api/chat (keep_alive + num_ctx)
│   ├── AnthropicChunkContextualizer.cs
│   └── NullChunkContextualizer.cs
├── Embedding/
│   ├── IEmbeddingProvider.cs
│   ├── OpenAiCompatibleEmbeddingProvider.cs  # /v1/embeddings
│   ├── OllamaEmbeddingProvider.cs            # /api/embed (keep_alive)
│   ├── NullEmbeddingProvider.cs
│   └── EmbeddingBatchSizes.cs
├── Storage/
│   └── SqliteVectorStore.cs       # SQLite + FTS5 + SIMD cosine similarity
├── Search/
│   ├── HybridSearcher.cs          # BM25 + Vector → RRF
│   └── RrfFusion.cs
├── Chunking/
│   ├── TextChunker.cs
│   └── ChunkLimits.cs
└── Tools/
    ├── ListKnowledgeBasesTool.cs
    ├── SearchDocumentsTool.cs
    ├── GetDocumentChunkTool.cs
    ├── StartReindexTool.cs        # Queue entry point + orchestrator spawn
    ├── CancelReindexTool.cs       # Remove pending queue entry
    ├── GetIndexInfoTool.cs        # Includes queue state
    └── CheckChangesTool.cs

Data Storage

Knowledge base data is stored at %LOCALAPPDATA%\FieldCure\Mcp.Rag\{kb-id}\:

• config.json — knowledge base configuration
• rag.db — SQLite database (chunks, embeddings, FTS5 index, file hashes, indexing lock)

Queue and lock files at %LOCALAPPDATA%\FieldCure\Mcp.Rag\:

• .deferred-queue.json — pending indexing requests
• orchestrator.lock — PID lock for the queue orchestrator

Development

# Build
dotnet build

# Test
dotnet test

# Pack as dotnet tool
dotnet pack src/FieldCure.Mcp.Rag -c Release

See Also

Part of the AssistStudio ecosystem.

License

MIT