E-books: upload with its own permission, review, and viewers for PDF, EPUB, text and archive.org #67

anas · 2026-10-09T12:52:42Z

anas commented

2026-10-09 12:52:42 +00:00

Owner request: a section ای بکس on each poet's page with uploaded books, served with a document viewer.

Who

A new grant content type, ای بکس (ebooks), with the action create, for chosen poets or all poets.
The ای بک شامل کریں button shows only for L2/L1 moderators with that grant for the poet, and for admins (super moderators).
Anyone with the grant may upload any book (owner decision). Reviewers confirm it: an upload is a revision (ebook) through L2 → L1 → admin.
Until it is approved, the book is not listed and its file is served only to moderators; signed out, it returns 404.

What

Files: PDF, EPUB and TXT, plus DOCX, which is turned into text with mammoth (BSD-2). Up to 100 MB. The kind is read from the bytes, not the name.
archive.org links (or a bare item id).
Details: title, source, copyright/permission and a note. They can be edited in review; the file itself cannot.

Viewing (/ebook/<id>)

PDF in the browser's own viewer.
EPUB with epub.js (BSD-2), with next/previous page buttons and arrow keys.
Text as text.
archive.org items in archive.org's embedded reader (archive.org/embed/<id>); they are not copied to our storage.
Each page also has a download link and the source/licence line.

Storage and indexing (owner direction)

File store: files live in DIVAN_FILES_DIR (default ../divan-files) under their SHA-256, ab/cd/<sha>.<ext>. A file uploaded twice is kept once, and the folder spreads evenly.
Elastic storage: on the server, point it at storage that can grow, such as an attached block-storage volume that can be resized later, or an S3-compatible bucket mounted with rclone. The README explains this. There is no S3 code yet; the folder setting is the seam.
Uploads: streamed to disk with a size check rather than held in memory by the API.
Database: an ebooks table with an index on poet (published only) and a trigram index on the normalised title plus the text of text books. Search shows matching e-books as their own group on the first page.
Public record: approved details, not the file, are committed to divan-data as divan/<poet>/ebooks/<id>.txt.

Tested

npm test: 42 pass. The new e-books test covers:
- the details text and archive.org ids;
- permission (403 without a grant or for another poet);
- the PDF, EPUB and DOCX→text kinds; a bad file returns 415; the same file is stored once;
- a title is required;
- an unpublished file is hidden from the public but open to a reviewer;
- L2 → L1 → admin publish, after which it is listed, the file is served as application/pdf with the same bytes, and the divan-data record and commit are written;
- the search index;
- a text book returns its text;
- archive.org link checks.
Headless Chrome:
- a moderator without the grant sees no button;
- with the grant, uploaded a real PDF (viewer iframe, file 200 application/pdf), a real EPUB (epub.js rendered the Urdu chapter) and an archive.org link (embedded reader);
- signed out, the unpublished file returns 404.
Nothing was published from the browser. The test books, files and accounts were removed.

Local setup: apply db/schema.sql; run npm install in api and web for mammoth and epubjs.

Later

Range requests for very large PDFs (they load whole for now).
Removing a rejected upload's file from storage.
Text search inside PDFs and EPUBs (only text books are indexed).

🤖 Generated with Claude Code

Owner request: a section **ای بکس** on each poet's page with uploaded books, served with a document viewer. **Who** - A new grant content type, **ای بکس** (`ebooks`), with the action **create**, for chosen poets or all poets. - The **ای بک شامل کریں** button shows only for L2/L1 moderators with that grant for the poet, and for admins (super moderators). - Anyone with the grant may upload any book (owner decision). Reviewers confirm it: an upload is a revision (`ebook`) through L2 → L1 → admin. - Until it is approved, the book is not listed and its file is served only to moderators; signed out, it returns 404. **What** - Files: **PDF**, **EPUB** and **TXT**, plus **DOCX**, which is turned into text with mammoth (BSD-2). Up to **100 MB**. The kind is read from the bytes, not the name. - **archive.org** links (or a bare item id). - Details: title, source, copyright/permission and a note. They can be edited in review; the file itself cannot. **Viewing** (`/ebook/<id>`) - PDF in the browser's own viewer. - EPUB with epub.js (BSD-2), with next/previous page buttons and arrow keys. - Text as text. - archive.org items in archive.org's embedded reader (`archive.org/embed/<id>`); they are not copied to our storage. - Each page also has a download link and the source/licence line. **Storage and indexing** (owner direction) - **File store:** files live in `DIVAN_FILES_DIR` (default `../divan-files`) under their SHA-256, `ab/cd/<sha>.<ext>`. A file uploaded twice is kept once, and the folder spreads evenly. - **Elastic storage:** on the server, point it at storage that can grow, such as an attached block-storage volume that can be resized later, or an S3-compatible bucket mounted with rclone. The README explains this. There is no S3 code yet; the folder setting is the seam. - **Uploads:** streamed to disk with a size check rather than held in memory by the API. - **Database:** an `ebooks` table with an index on poet (published only) and a **trigram index** on the normalised title plus the text of text books. Search shows matching e-books as their own group on the first page. - **Public record:** approved details, not the file, are committed to divan-data as `divan/<poet>/ebooks/<id>.txt`. **Tested** - `npm test`: 42 pass. The new e-books test covers: - the details text and archive.org ids; - permission (403 without a grant or for another poet); - the PDF, EPUB and DOCX→text kinds; a bad file returns 415; the same file is stored once; - a title is required; - an unpublished file is hidden from the public but open to a reviewer; - L2 → L1 → admin publish, after which it is listed, the file is served as application/pdf with the same bytes, and the divan-data record and commit are written; - the search index; - a text book returns its text; - archive.org link checks. - Headless Chrome: - a moderator without the grant sees no button; - with the grant, uploaded a real PDF (viewer iframe, file 200 application/pdf), a real EPUB (epub.js rendered the Urdu chapter) and an archive.org link (embedded reader); - signed out, the unpublished file returns 404. - Nothing was published from the browser. The test books, files and accounts were removed. **Local setup:** apply `db/schema.sql`; run `npm install` in api and web for mammoth and epubjs. **Later** - Range requests for very large PDFs (they load whole for now). - Removing a rejected upload's file from storage. - Text search inside PDFs and EPUBs (only text books are indexed). 🤖 Generated with [Claude Code](https://claude.com/claude-code)

anas added 1 commit 2026-10-09 12:52:43 +00:00

E-books: a poet's books as PDF, EPUB, text (DOCX made text) or archive.org items, uploaded with their own permission and approved like any change 09336ead0e

- ebooks permission (create, per poet); the upload button only for L2/L1 moderators with it (and admins).
- Files kept by SHA-256 in DIVAN_FILES_DIR (ab/cd/<sha>.<ext>; one copy per file), up to 100 MB, kind read from
  the bytes; details in PostgreSQL indexed by poet and trigram (title, text of text books); search shows e-books.
- An upload is a revision (entity 'ebook') through L2 -> L1 -> admin; unpublished files only for moderators;
  approved details committed to divan-data (divan/<poet>/ebooks/<id>.txt).
- ای بکس on the poet page; /ebook/<id>: PDF in the browser viewer, EPUB with epub.js, text, archive.org embed.
- README: e-books and the file store (attach growable block storage or an S3 bucket via rclone).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

anas merged commit 6f96548f7a into main

2026-10-09 12:54:28 +00:00

anas referenced this issue from a commit

2026-10-09 12:54:30 +00:00

Merge pull request 'E-books: upload with its own permission, review, and viewers for PDF, EPUB, text and archive.org' (#67) from feature/ebooks into main

anas referenced this pull request

2026-10-09 12:57:19 +00:00

Fix: e-books in the moderation queue and log had no title or link #68

anas referenced this pull request

2026-10-09 13:16:30 +00:00

E-books: a section for every poet (tabs, covers), right-to-left PDF reader, page bookmarks, book details #70

Sign in to join this conversation.