Add a tap-a-word dictionary, layering Wiktionary and Daneshjoo

Tapping a word in a poem now opens its meaning. Neither source covers enough
alone — Daneshjoo reaches 71% of real poem vocabulary — so both ship, each row
carrying its source so the credit stays attached and either can be dropped
later. Together with the lookup chain that reaches 88%, and 13 of the 55
remaining misses are Arabic lines quoted inside Persian poems.

Lookup widens until something matches: the word as written, the lemma it
inflects from, the word with an affix stripped, then the parts of a ZWNJ
compound. The lemma step is what makes classical verse readable, and it comes
from Wiktionary's 149,589 form->lemma pairs: افتاد is only findable as افتادن.
The Arabic definite article is stripped too, since poems quote Arabic.

Two things the build taught me. Wiktionary's 93 MB export is 0.94 MB of
definitions wrapped in inflection tables, etymology templates and IPA, so
tools/build_dictionary.py keeps the definitions and the form index and drops
the rest. And the packager silently gunzips .gz assets and strips the
extension, which shipped the database under a name the code wasn't opening —
every lookup failed silently until the APK listing gave it away.

Hit testing needed care as well: nastaliq is set with 2.4x leading, so most of
a line box is empty space, and a tap there clamps to the line's first
character — which made every tap return the opening word. Taps are now checked
against the baseline band, with the multipliers left as a knob for swapped
fonts. Definitions are English, so they read left-to-right inside the
otherwise right-to-left sheet.

Verified on the release build, where R8 could have broken the SQLite path:
tapping عشق returns both sources, and دانند resolves to دانستن.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>

tools/README.md

@@ -0,0 +1,40 @@
+# Building the dictionary
+
+`app/src/main/assets/dictionary.db.gz` is generated, not hand-written. This rebuilds it:
+
+```sh
+pip install readmdict python-lzo          # lzo needs the C library: brew install lzo
+curl -L -o fa.jsonl https://kaikki.org/dictionary/Persian/kaikki.org-dictionary-Persian.jsonl
+curl -L -o daneshjoo.mdx \
+  "https://raw.githubusercontent.com/0xdolan/Daneshjoo/main/Daneshjoo%20Dictionary/Daneshjoo%20Dictionary.mdx"
+python3 build_dictionary.py
+gzip -9 ganjoor-dictionary.db
+```
+
+## Why two sources
+
+Neither alone is enough. Measured against every distinct word in five real poems (Hafez ×2,
+Saadi's Golestan, Rumi's Masnavi, a Khayyam rubaʿi — 485 words):
+
+| | coverage |
+|---|---|
+| Daneshjoo alone | 71% |
+| Wiktionary alone + its form index | ~80% |
+| **Both, with the lookup chain** | **88%** |
+
+13 of the 55 remaining misses are Arabic lines quoted inside Persian poems, so Persian coverage
+is about 91%.
+
+Wiktionary's export is 93 MB, of which the definitions are 0.94 MB — the rest is inflection
+tables, etymology templates, IPA and descendants. The build keeps the definitions and the
+form→lemma index (149,589 pairs) and drops the rest. That index is what resolves conjugated
+verbs: `افتاد → افتادن`, `بگشاید → گشودن`, `دانند → دانستن`.
+
+## Licences
+
+Each row carries its `source`, so attribution stays accurate and either source can be dropped
+without rebuilding the other.
+
+- **Wiktionary** — CC BY-SA 3.0. The generated database is therefore also CC BY-SA 3.0.
+- **Daneshjoo** — the repository states MIT. Note the underlying lexicon is a published Iranian
+  dictionary, so that relicensing is worth verifying before relying on it.

tools/build_dictionary.py

@@ -0,0 +1,70 @@
+"""Builds the app's dictionary from Wiktionary (CC BY-SA 3.0) and Daneshjoo (MIT).
+
+Wiktionary's export is 93 MB of linguistic metadata around 0.94 MB of definitions, so this keeps
+the definitions and the form->lemma index and discards the rest. The `source` column is what
+keeps the attribution honest and lets either source be dropped later.
+"""
+import json, re, sqlite3, unicodedata, os, html
+from readmdict import MDX
+
+HARAKAT = set(range(0x064B, 0x0653)) | {0x0670, 0x0640} | set(range(0x0610, 0x0616))
+# Arabic letters that Persian writes differently; headwords and poems disagree constantly.
+FOLD = {'ي': 'ی', 'ى': 'ی', 'ك': 'ک', 'ة': 'ه'}
+
+def normalise(s: str) -> str:
+    d = unicodedata.normalize('NFD', s)
+    d = ''.join(c for c in d if ord(c) not in HARAKAT)
+    s = unicodedata.normalize('NFC', d)
+    return ''.join(FOLD.get(c, c) for c in s).replace('‌', '').strip()
+
+db = 'ganjoor-dictionary.db'
+if os.path.exists(db): os.remove(db)
+c = sqlite3.connect(db)
+c.executescript("""
+CREATE TABLE entry (word TEXT NOT NULL, display TEXT NOT NULL, gloss TEXT NOT NULL, source TEXT NOT NULL);
+CREATE TABLE form  (form TEXT NOT NULL, lemma TEXT NOT NULL);
+""")
+
+entries, forms = [], set()
+
+for line in open('fa.jsonl', encoding='utf-8'):
+    try: e = json.loads(line)
+    except Exception: continue
+    word = e.get('word')
+    if not word: continue
+    gs = [g.strip() for s in e.get('senses', []) for g in (s.get('glosses') or []) if g.strip()]
+    if gs:
+        pos = e.get('pos') or ''
+        gloss = '; '.join(dict.fromkeys(gs))[:600]
+        entries.append((normalise(word), word, f"({pos}) {gloss}" if pos else gloss, 'wiktionary'))
+    for f in e.get('forms', []):
+        t = f.get('form')
+        if t and t != word and not t.startswith('-') and len(t) > 1:
+            forms.add((normalise(t), normalise(word)))
+
+print(f"wiktionary: {len(entries)} entries, {len(forms)} forms")
+
+tag = re.compile(r'<[^>]+>')
+n0 = len(entries)
+for k, v in MDX('daneshjoo.mdx').items():
+    word = k.decode('utf-8', 'ignore')
+    if not re.match(r'^[؀-ۿ]', word):
+        continue                                    # the en->fa half isn't useful here
+    txt = html.unescape(tag.sub(' ', v.decode('utf-8', 'ignore')))
+    txt = re.sub(r'\s+', ' ', txt).strip()
+    if txt.startswith(word):
+        txt = txt[len(word):].strip(' -–—')
+    if txt:
+        entries.append((normalise(word), word, txt[:600], 'daneshjoo'))
+print(f"daneshjoo : {len(entries) - n0} entries")
+
+c.executemany("INSERT INTO entry VALUES (?,?,?,?)", entries)
+c.executemany("INSERT INTO form  VALUES (?,?)", sorted(forms))
+c.executescript("""
+CREATE INDEX idx_entry_word ON entry(word);
+CREATE INDEX idx_form_form  ON form(form);
+""")
+c.commit()
+c.execute("VACUUM")
+c.close()
+print(f"\n{db}: {os.path.getsize(db)/1e6:.1f} MB")