From 60b1625bbb21f2e1a78fc2b4316bc521fb1879ca Mon Sep 17 00:00:00 2001 From: Hamid Reza Mohammadi Date: Fri, 11 Sep 2026 08:30:48 +0330 Subject: [PATCH] semantic search dev guide --- SEMANTIC_SEARCH_SETUP.md | 227 +++++++++++++++++++++++++++++++++++++++ 1 file changed, 227 insertions(+) create mode 100644 SEMANTIC_SEARCH_SETUP.md diff --git a/SEMANTIC_SEARCH_SETUP.md b/SEMANTIC_SEARCH_SETUP.md new file mode 100644 index 00000000..fe696b7d --- /dev/null +++ b/SEMANTIC_SEARCH_SETUP.md @@ -0,0 +1,227 @@ +# Semantic Search — Developer Setup Guide + +Everything needed to get "find a poem about..." search running locally, plus the real problems +encountered building it and their actual fixes — not a theoretical list, the specific things that +actually went wrong during development, in enough detail to recognize them again. + +If you just want to *use* the published embeddings data, [`ganjoor-embeddings`](https://github.com/ganjoor/ganjoor-embeddings) +is the place to look. This document is about running the .NET side of the feature (the code in +this repo) against that data. + +## Table of contents + +- [What this feature actually is](#what-this-feature-actually-is) +- [Prerequisites](#prerequisites) +- [Getting the model](#getting-the-model) +- [Getting the embeddings data](#getting-the-embeddings-data) +- [Configuration](#configuration) +- [Running it locally](#running-it-locally) +- [Architecture, briefly](#architecture-briefly) +- [Known issues and their actual fixes](#known-issues-and-their-actual-fixes) + +## What this feature actually is + +A user types a description of what they're looking for ("a poem about the world's +unfaithfulness") instead of exact keywords. Their query gets embedded into a 1024-dimensional +vector using the same model that embedded every poem's AI-written summary offline; the poems +whose vectors are closest (cosine similarity) come back as results. See +[`ganjoor-embeddings`](https://github.com/ganjoor/ganjoor-embeddings)'s docs for how the poem-side +embeddings were generated — this guide covers the query-time side, which lives in this repo. + +## Prerequisites + +- **.NET 10 SDK** — check with `dotnet --version`. An older SDK can't build a `net10.0` target at + all; you'll get a clear build error, not a subtle one, if this is wrong. +- **Windows**, if you're testing against IIS the way production runs — the ONNX Runtime native + binaries differ per OS/architecture, and at least one real bug (see below) only reproduced on + Windows, not macOS, despite identical code. +- The base model + tokenizer files, and the embeddings data — see the next two sections. + +## Getting the model + +**[`onnx-community/Qwen3-Embedding-0.6B-ONNX`](https://huggingface.co/onnx-community/Qwen3-Embedding-0.6B-ONNX)** +on Hugging Face. You need, together in one folder: + +- `model.onnx` + `model.onnx_data` (fp32), **or** `model_quantized.onnx` (int8 — what production + actually uses; ~2.9x faster with negligible accuracy cost, per the model's own published + benchmarks) +- `vocab.json` +- `merges.txt` + +```python +from huggingface_hub import snapshot_download +snapshot_download("onnx-community/Qwen3-Embedding-0.6B-ONNX", local_dir="./ganjoor-model") +``` + +**Watch the folder structure** — this repo's download layout puts `model*.onnx` files inside an +`onnx/` subfolder, but `vocab.json`/`merges.txt`/`tokenizer.json` in the *parent* folder. RMuseum's +config needs `ModelPath` pointing into `onnx/` and `VocabPath`/`MergesPath` pointing at the parent +— don't assume everything landed in the same directory. If you get a "file not found" for the +tokenizer files, this is almost certainly why. + +## Getting the embeddings data + +Either download the published data (`embeddings.f32` + `embeddings-index.json`) from +[`ganjoor-embeddings`'s releases](https://github.com/ganjoor/ganjoor-embeddings/releases) or +[Hugging Face](https://huggingface.co/datasets/ganjoor/ganjoor-poem-embeddings), or generate it +yourself using that repo's scripts. Either way you should end up with those two files in one +folder — `verify_embeddings.py` in that repo will confirm the data is intact before you trust it. + +## Configuration + +`appsettings.json`: + +```json +"SemanticSearch": { + "Enabled": "True", + "EmbeddingsDirectory": "C:\\path\\to\\embeddings-data", + "ModelPath": "C:\\path\\to\\ganjoor-model\\onnx\\model_quantized.onnx", + "VocabPath": "C:\\path\\to\\ganjoor-model\\vocab.json", + "MergesPath": "C:\\path\\to\\ganjoor-model\\merges.txt", + "Dimension": "1024" +} +``` + +**`Enabled` defaults to `False` if the key is missing** — deliberately fail-closed, so an instance +that shouldn't load a ~1GB+ model into memory (see "Architecture" below) never does so by +accident. You need `"True"` explicitly to actually use this locally. + +## Running it locally + +```bash +dotnet build +dotnet run --project RMuseum +``` + +Confirm the endpoint works directly before assuming the UI is the problem if something looks off: + +```bash +curl -X POST http://localhost:5000/api/ganjoor/search/semantic \ + -H "Content-Type: application/json" \ + -d '{"query": "a poem about the world'"'"'s unfaithfulness", "topK": 5}' +``` + +## Architecture, briefly + +- **`EmbeddingIndex`** — loads `embeddings.f32`/`embeddings-index.json` into memory once, does + brute-force cosine similarity search (fast enough at this corpus size — no vector database + needed). +- **`QueryEmbedder`** — embeds a single query using the same ONNX model + tokenizer setup as the + offline generation pipeline. Must match it exactly — see "Known issues" below for what "exactly" + turned out to mean in practice. +- **`LazySemanticSearchResources`** / **`LazyQueryScopeIndex`** — defer loading until first real + use and never throw. This exists because an earlier version's eager, throwing DI factories meant + a load failure prevented `GanjoorController` itself from being constructed — a 503 on *every* + endpoint under `/api/ganjoor`, not just this feature, in actual production. Any future resource + this feature needs to load lazily should follow this same pattern, not a throwing factory. +- **`SemanticSearchController`** — deliberately its own controller, not a method on + `GanjoorController`, for the same isolation reason. +- **Production runs this on a physically separate domain/app pool** (`ganjgah.ir`, distinct from + `api.ganjoor.net`) behind the `SemanticSearch:Enabled` flag — so a problem with this one feature + (a native crash, a memory issue) can't take down the main site. `GanjooRazor`'s + `APIRoot.SemanticSearchUrl` points at whichever domain should actually serve this. + +## Known issues and their actual fixes + +### "Could not start git" / a tool that was just installed isn't found + +If you install something (Git, an SDK) while Visual Studio/IIS Express is already running, the +already-running process inherited its environment *before* the install — it won't see the update +until fully restarted. A brand-new terminal window works because it's a fresh process; the +already-open IDE doesn't. **Fully close and reopen the IDE**, not just recycle the current session. + +### `pip install` fails with "externally-managed-environment" (macOS) + +Modern macOS Python (via Homebrew) blocks direct `pip install`. Use a virtual environment: +```bash +python3 -m venv venv && source venv/bin/activate +``` + +### `huggingface-cli` prints a deprecation warning and does nothing + +It's been replaced by `hf`. Use `hf auth login` instead of `huggingface-cli login` — this is not +optional, the old command doesn't actually authenticate anymore despite appearing to run. + +### Hugging Face download/upload fails with a `CAS`/`xet` connection error + +HF's newer Xet transfer backend can be flaky on some networks, especially slow/unstable ones. Force +the older plain HTTP path: +```bash +export HF_HUB_DISABLE_XET=1 +``` + +### The ONNX model needs more than just `input_ids`/`attention_mask` + +This specific model export is KV-cache-enabled (built for autoregressive generation), not a plain +single-pass embedding graph — it also requires `position_ids` and a `past_key_values.N.key`/ +`.value` pair **per layer**, even for one uncached pass. **Always run with `--inspect-only`** (in +the `ganjoor-embeddings` generation script) or the equivalent model introspection before assuming +tensor names/shapes — this project's assumptions were wrong on the first attempt, and the model +itself told us the real signature faster than searching documentation did. + +### `BpeTokenizer.Create(vocabStream, mergesStream)` silently produces zero tokens for Persian + +The simple two-argument overload doesn't do byte-level pre-tokenization — it works for the ASCII +examples in Microsoft's own docs and silently fails (not errors — just produces empty output) for +non-Latin scripts. The fix, found via reflection against the actual installed package (not +documentation, which was stale for this): + +```csharp +var bpeOptions = new BpeOptions(vocabPath, mergesPath) { ByteLevel = true }; +var tokenizer = BpeTokenizer.Create(bpeOptions); +``` + +If you hit a similarly unclear `Microsoft.ML.Tokenizers` API question, reflecting the actual +installed assembly's real method signatures beats searching — see the pattern in +`ganjoor-embeddings`' `VERIFICATION.md` for exactly how. + +### `new Tokenizer(new Bpe(...))` doesn't compile ("cannot create an instance of abstract type") + +An older code example (a 2022 ML.NET blog post) used a construction pattern that no longer applies +to the current `Microsoft.ML.Tokenizers` package — `Tokenizer` is abstract now. Use +`BpeTokenizer.Create(...)` (above), not this older pattern. + +### Query embeddings don't match document embeddings even with "correct" tokenization + +Compare token ids directly — Python (`tokenizers.Tokenizer.from_file(...)`, the same tokenizer +that generated the corpus) vs. whatever the new consumer's language produces, on several real +query strings, not just one. The specific gap found this way: Python's tokenizer always appends +one extra token (id `151643`, Qwen's end-of-sequence marker) that a from-vocab-files C# +construction doesn't add automatically. Since pooling happens on the *last* token, missing this +token means pooling from a different position than every document embedding used — not a subtle +difference, a real one. Add it manually if your tokenizer setup doesn't include it: +```csharp +var fullTokenIds = new List(tokenIds) { 151643 }; +``` + +### Everything runs without error but the whole IIS site crashes when the endpoint is actually called + +A native crash (Windows Event Viewer: `w3wp.exe` faulting in a native DLL, exception code +`c0000005` — an access violation) can't be caught by any C# `try`/`catch` — the process dies +outright. **Test any new native-backed code path (ONNX Runtime, or anything similar) as a +disposable local console app first**, on the same OS/architecture the real deployment uses, before +it goes anywhere near a shared or production environment. In this project's case, the actual root +cause turned out to be an outdated Visual C++ Redistributable on the server — install the latest +from `https://aka.ms/vs/17/release/vc_redist.x64.exe`. If that doesn't resolve a similar crash, +also check the IIS Application Pool's "Enable 32-Bit Applications" setting — a 32-bit worker +process trying to load a 64-bit-only native library fails exactly the same way. + +### After a crash, the site stays down even after the underlying cause is fixed + +IIS's Rapid-Fail Protection automatically disables an entire Application Pool after repeated +crashes in a short window, and won't restart it on its own. **IIS Manager → Application Pools → +find it → Start** — this is usually all that's needed, not evidence of a deeper unresolved problem. + +### Running multiple IIS worker processes multiplies this feature's memory use + +Each worker process is a fully separate .NET process — "singleton" only means one instance *per +process*. `EmbeddingIndex` (hundreds of MB) and `QueryEmbedder` (a loaded ONNX model) both get +reloaded, in full, per worker process. 8 worker processes means 8 independent copies. If this +feature is enabled on an app pool running more than one worker process, do the math on whether +that's actually intended before assuming a load failure is the interesting problem. + +### `dotnet run` fails with `Interop.Sys.GetCwd()` / "unable to find the specified file" + +Not a code problem — the shell's current directory got deleted/replaced out from under it +(typically from re-extracting a zip over a folder you're currently `cd`'d into). `cd` to an +absolute path to get a fresh reference, then retry.