divan/docs/upstream/SEMANTIC_SEARCH_SETUP.md

Upstream GanjoorService doc, kept for reference; Diwan's own setup is in the root README.

Semantic Search — Developer Setup Guide

Everything needed to get "find a poem about..." search running locally, plus the real problems encountered building it and their actual fixes — not a theoretical list, the specific things that actually went wrong during development, in enough detail to recognize them again.

If you just want to use the published embeddings data, ganjoor-embeddings is the place to look. This document is about running the .NET side of the feature (the code in this repo) against that data.

Table of contents

• What this feature actually is
• Prerequisites
• Getting the model
• Getting the embeddings data
• Configuration
• Running it locally
• Architecture, briefly
• Known issues and their actual fixes

What this feature actually is

A user types a description of what they're looking for ("a poem about the world's unfaithfulness") instead of exact keywords. Their query gets embedded into a 1024-dimensional vector using the same model that embedded every poem's AI-written summary offline; the poems whose vectors are closest (cosine similarity) come back as results. See ganjoor-embeddings's docs for how the poem-side embeddings were generated — this guide covers the query-time side, which lives in this repo.

Prerequisites

• .NET 10 SDK — check with dotnet --version. An older SDK can't build a net10.0 target at all; you'll get a clear build error, not a subtle one, if this is wrong.
• Windows, if you're testing against IIS the way production runs — the ONNX Runtime native binaries differ per OS/architecture, and at least one real bug (see below) only reproduced on Windows, not macOS, despite identical code.
• The base model + tokenizer files, and the embeddings data — see the next two sections.

Getting the model

onnx-community/Qwen3-Embedding-0.6B-ONNX on Hugging Face. You need, together in one folder:

• model.onnx + model.onnx_data (fp32), or model_quantized.onnx (int8 — what production actually uses; ~2.9x faster with negligible accuracy cost, per the model's own published benchmarks)
• vocab.json
• merges.txt

from huggingface_hub import snapshot_download
snapshot_download("onnx-community/Qwen3-Embedding-0.6B-ONNX", local_dir="./ganjoor-model")

Watch the folder structure — this repo's download layout puts model*.onnx files inside an onnx/ subfolder, but vocab.json/merges.txt/tokenizer.json in the parent folder. RMuseum's config needs ModelPath pointing into onnx/ and VocabPath/MergesPath pointing at the parent — don't assume everything landed in the same directory. If you get a "file not found" for the tokenizer files, this is almost certainly why.

Getting the embeddings data

Either download the published data (embeddings.f32 + embeddings-index.json) from ganjoor-embeddings's releases or Hugging Face, or generate it yourself using that repo's scripts. Either way you should end up with those two files in one folder — verify_embeddings.py in that repo will confirm the data is intact before you trust it.

Configuration

appsettings.json:

"SemanticSearch": {
  "Enabled": "True",
  "EmbeddingsDirectory": "C:\\path\\to\\embeddings-data",
  "ModelPath": "C:\\path\\to\\ganjoor-model\\onnx\\model_quantized.onnx",
  "VocabPath": "C:\\path\\to\\ganjoor-model\\vocab.json",
  "MergesPath": "C:\\path\\to\\ganjoor-model\\merges.txt",
  "Dimension": "1024"
}

Enabled defaults to False if the key is missing — deliberately fail-closed, so an instance that shouldn't load a ~1GB+ model into memory (see "Architecture" below) never does so by accident. You need "True" explicitly to actually use this locally.

Running it locally

dotnet build
dotnet run --project RMuseum

Confirm the endpoint works directly before assuming the UI is the problem if something looks off:

curl -X POST http://localhost:5000/api/ganjoor/search/semantic \
  -H "Content-Type: application/json" \
  -d '{"query": "a poem about the world'"'"'s unfaithfulness", "topK": 5}'

Architecture, briefly

• EmbeddingIndex — loads embeddings.f32/embeddings-index.json into memory once, does brute-force cosine similarity search (fast enough at this corpus size — no vector database needed).
• QueryEmbedder — embeds a single query using the same ONNX model + tokenizer setup as the offline generation pipeline. Must match it exactly — see "Known issues" below for what "exactly" turned out to mean in practice.
• LazySemanticSearchResources / LazyQueryScopeIndex — defer loading until first real use and never throw. This exists because an earlier version's eager, throwing DI factories meant a load failure prevented GanjoorController itself from being constructed — a 503 on every endpoint under /api/ganjoor, not just this feature, in actual production. Any future resource this feature needs to load lazily should follow this same pattern, not a throwing factory.
• SemanticSearchController — deliberately its own controller, not a method on GanjoorController, for the same isolation reason.
• Production runs this on a physically separate domain/app pool (ganjgah.ir, distinct from api.ganjoor.net) behind the SemanticSearch:Enabled flag — so a problem with this one feature (a native crash, a memory issue) can't take down the main site. GanjooRazor's APIRoot.SemanticSearchUrl points at whichever domain should actually serve this.

Known issues and their actual fixes

"Could not start git" / a tool that was just installed isn't found

If you install something (Git, an SDK) while Visual Studio/IIS Express is already running, the already-running process inherited its environment before the install — it won't see the update until fully restarted. A brand-new terminal window works because it's a fresh process; the already-open IDE doesn't. Fully close and reopen the IDE, not just recycle the current session.

pip install fails with "externally-managed-environment" (macOS)

Modern macOS Python (via Homebrew) blocks direct pip install. Use a virtual environment:

python3 -m venv venv && source venv/bin/activate

huggingface-cli prints a deprecation warning and does nothing

It's been replaced by hf. Use hf auth login instead of huggingface-cli login — this is not optional, the old command doesn't actually authenticate anymore despite appearing to run.

Hugging Face download/upload fails with a CAS/xet connection error

HF's newer Xet transfer backend can be flaky on some networks, especially slow/unstable ones. Force the older plain HTTP path:

export HF_HUB_DISABLE_XET=1

The ONNX model needs more than just input_ids/attention_mask

This specific model export is KV-cache-enabled (built for autoregressive generation), not a plain single-pass embedding graph — it also requires position_ids and a past_key_values.N.key/ .value pair per layer, even for one uncached pass. Always run with --inspect-only (in the ganjoor-embeddings generation script) or the equivalent model introspection before assuming tensor names/shapes — this project's assumptions were wrong on the first attempt, and the model itself told us the real signature faster than searching documentation did.

BpeTokenizer.Create(vocabStream, mergesStream) silently produces zero tokens for Persian

The simple two-argument overload doesn't do byte-level pre-tokenization — it works for the ASCII examples in Microsoft's own docs and silently fails (not errors — just produces empty output) for non-Latin scripts. The fix, found via reflection against the actual installed package (not documentation, which was stale for this):

var bpeOptions = new BpeOptions(vocabPath, mergesPath) { ByteLevel = true };
var tokenizer = BpeTokenizer.Create(bpeOptions);

If you hit a similarly unclear Microsoft.ML.Tokenizers API question, reflecting the actual installed assembly's real method signatures beats searching — see the pattern in ganjoor-embeddings' VERIFICATION.md for exactly how.

new Tokenizer(new Bpe(...)) doesn't compile ("cannot create an instance of abstract type")

An older code example (a 2022 ML.NET blog post) used a construction pattern that no longer applies to the current Microsoft.ML.Tokenizers package — Tokenizer is abstract now. Use BpeTokenizer.Create(...) (above), not this older pattern.

Query embeddings don't match document embeddings even with "correct" tokenization

Compare token ids directly — Python (tokenizers.Tokenizer.from_file(...), the same tokenizer that generated the corpus) vs. whatever the new consumer's language produces, on several real query strings, not just one. The specific gap found this way: Python's tokenizer always appends one extra token (id 151643, Qwen's end-of-sequence marker) that a from-vocab-files C# construction doesn't add automatically. Since pooling happens on the last token, missing this token means pooling from a different position than every document embedding used — not a subtle difference, a real one. Add it manually if your tokenizer setup doesn't include it:

var fullTokenIds = new List<int>(tokenIds) { 151643 };

Everything runs without error but the whole IIS site crashes when the endpoint is actually called

A native crash (Windows Event Viewer: w3wp.exe faulting in a native DLL, exception code c0000005 — an access violation) can't be caught by any C# try/catch — the process dies outright. Test any new native-backed code path (ONNX Runtime, or anything similar) as a disposable local console app first, on the same OS/architecture the real deployment uses, before it goes anywhere near a shared or production environment. In this project's case, the actual root cause turned out to be an outdated Visual C++ Redistributable on the server — install the latest from https://aka.ms/vs/17/release/vc_redist.x64.exe. If that doesn't resolve a similar crash, also check the IIS Application Pool's "Enable 32-Bit Applications" setting — a 32-bit worker process trying to load a 64-bit-only native library fails exactly the same way.

After a crash, the site stays down even after the underlying cause is fixed

IIS's Rapid-Fail Protection automatically disables an entire Application Pool after repeated crashes in a short window, and won't restart it on its own. IIS Manager → Application Pools → find it → Start — this is usually all that's needed, not evidence of a deeper unresolved problem.

Running multiple IIS worker processes multiplies this feature's memory use

Each worker process is a fully separate .NET process — "singleton" only means one instance per process. EmbeddingIndex (hundreds of MB) and QueryEmbedder (a loaded ONNX model) both get reloaded, in full, per worker process. 8 worker processes means 8 independent copies. If this feature is enabled on an app pool running more than one worker process, do the math on whether that's actually intended before assuming a load failure is the interesting problem.

dotnet run fails with Interop.Sys.GetCwd() / "unable to find the specified file"

Not a code problem — the shell's current directory got deleted/replaced out from under it (typically from re-extracting a zip over a folder you're currently cd'd into). cd to an absolute path to get a fresh reference, then retry.