anas/divan
1
0
Fork 0
Code Issues 12 Pull Requests Actions Packages Projects Releases Wiki Activity

robots.txt, sitemap.xml and an RSS feed of new content #92

Merged
anas merged 2 commits from feature/robots-sitemap-rss into main 2026-10-09 19:55:29 +00:00
Conversation 0 Commits 2 Files Changed 9 +172
anas commented 2026-10-09 19:54:38 +00:00
Owner
Copy Link

Replaces #91 (Gitea kept a stale conflict status after it was reopened; the branch merges cleanly into main, with the footer and icons from #88 kept alongside). Closes #87; the robots and sitemap part of #83 (owner request to bring them forward). Page descriptions and share previews stay in #83.

/robots.txt

  • Search engines may read the reading pages, not accounts, moderation, admin, search results, the API, e-book files or /health.
  • It points to the sitemap.
  • DIVAN_NOINDEX=1 (the private preview) closes everything.

/sitemap.xml (from GET /api/sitemap)

  • Every poet and their e-books page, books/chapters, works, tags and e-books, plus home, about, tags, privacy and terms: 12,499 addresses.
  • A work's lastmod is when its latest Divan version was published.
  • One file (the limit is 50,000); cached for an hour.

/rss.xml (RSS 2.0, from GET /api/feed)

  • The latest 50 published changes, newest first: نیا ورژن of a work (title, poet, version number, its opening lines, summary), نئی ای بک, نئی ترتیب, ٹیگ.
  • Each item has the editor's public name (dc:creator) and a link.
  • Announced in every page's head (<link rel=alternate>) so browsers and feed readers find it; نئی تبدیلیاں in the footer.

Addresses: feed and sitemap links use SITE_URL if set, else the address the request came to.

Tested

  • New feeds.test.ts: all public pages present; a published version gives the work its date and appears first in the feed with kind, version, summary, author and opening lines; the limit is capped at 100.
  • On a test server: robots in both modes; the sitemap is valid XML with 12,499 addresses; the RSS is valid XML whose first item is the published e-book حدائقِ بخشش; the head link is present.
  • npm test passes.

The deploy repo sets SITE_URL and DIVAN_NOINDEX.

🤖 Generated with Claude Code

Replaces #91 (Gitea kept a stale conflict status after it was reopened; the branch merges cleanly into main, with the footer and icons from #88 kept alongside). Closes #87; the robots and sitemap part of #83 (owner request to bring them forward). Page descriptions and share previews stay in #83. **`/robots.txt`** - Search engines may read the reading pages, not accounts, moderation, admin, search results, the API, e-book files or `/health`. - It points to the sitemap. - `DIVAN_NOINDEX=1` (the private preview) closes everything. **`/sitemap.xml`** (from `GET /api/sitemap`) - Every poet and their e-books page, books/chapters, works, tags and e-books, plus home, about, tags, privacy and terms: **12,499 addresses**. - A work's `lastmod` is when its latest Divan version was published. - One file (the limit is 50,000); cached for an hour. **`/rss.xml`** (RSS 2.0, from `GET /api/feed`) - The latest 50 published changes, newest first: **نیا ورژن** of a work (title, poet, version number, its opening lines, summary), **نئی ای بک**, **نئی ترتیب**, **ٹیگ**. - Each item has the editor's public name (`dc:creator`) and a link. - Announced in every page's head (`<link rel=alternate>`) so browsers and feed readers find it; **نئی تبدیلیاں** in the footer. **Addresses:** feed and sitemap links use `SITE_URL` if set, else the address the request came to. **Tested** - New `feeds.test.ts`: all public pages present; a published version gives the work its date and appears first in the feed with kind, version, summary, author and opening lines; the limit is capped at 100. - On a test server: robots in both modes; the sitemap is valid XML with 12,499 addresses; the RSS is valid XML whose first item is the published e-book حدائقِ بخشش; the head link is present. - `npm test` passes. The deploy repo sets `SITE_URL` and `DIVAN_NOINDEX`. 🤖 Generated with [Claude Code](https://claude.com/claude-code)
anas added 2 commits 2026-10-09 19:54:38 +00:00 
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
# Conflicts:
#	web/src/components/Icon.astro
#	web/src/layouts/Base.astro
anas merged commit 9f61da990d into main 2026-10-09 19:55:29 +00:00
Sign in to join this conversation.
Reviewers
No reviewers
Labels
Clear labels   
area/api

   
area/data

   
area/infra

   
area/site

   
decision

   
deferred

   
i18n/urdu

   
search

   
v2
No Label
area/api
area/data
area/infra
area/site
decision
deferred
i18n/urdu
search
v2
Milestone
Clear milestone
No items
No Milestone
Projects
Clear projects
No project
Assignees
Clear assignees
No Assignees
1 Participants
Notifications
Due Date
The due date is invalid or out of range. Please use the format 'yyyy-mm-dd'.

No due date set.

Dependencies

No dependencies set.

Reference: anas/divan#92
No description provided.
Delete Branch "feature/robots-sitemap-rss"

Deleting a branch is permanent. Although the deleted branch may continue to exist for a short time before it actually gets removed, it CANNOT be undone in most cases. Continue?