Files
arsam e9cd12dda6 Import remaining project documentation and reference notes
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
2026-09-18 23:33:04 +03:30

24 KiB

DEPLOY.md - Native Debian Deployment Guide

IMPORTANT — Phase 12 (Web Push Notifications): A new PushSubscription table and the web-push backend package were added. Before starting the service, run:

cd backend
npm ci                    # installs web-push
npm run prisma:generate
npm run prisma:apply      # creates PushSubscription table
npm run build

Then generate VAPID keys once and add them to /etc/teknav/backend.env (see Section 6). Without VAPID keys the push routes return 503 and push delivery is silently skipped — all other features continue to work normally.

This is the complete production runbook for deploying Teknav natively on Debian, without Docker.

Production model:

  • Public site: https://www.teknav.ir
  • Frontend: Vite build served by nginx from /opt/teknav/current/dist
  • Backend: Fastify systemd service on 127.0.0.1:3010
  • Database: local PostgreSQL
  • Cache/session/SSE bus: local Redis
  • Uploads: /var/lib/teknav/uploads
  • Backups: /var/backups/teknav
  • Env file: /etc/teknav/backend.env

The deployed surface now also includes SEO topic hubs, glossary pages, Q&A, jobs, courses, premium membership, sponsored article labels, verified expert badges, 2FA, article corrections, first-party analytics/error capture, Zarinpal payment flow, and web push notifications (Phase 12).

Run these commands as a sudo-capable Linux user unless a command explicitly uses sudo -u teknav.

0. Local Development (Windows/Tuned)

If you are running locally and commands like prisma or tsx are not in your path, use the following "Tuned" commands from the root directory:

Update Database Seed (2026 Content)

cd backend
.\node_modules\.bin\tsx prisma\seed.ts

Force Clear Browser Cache (Articles)

If articles are missing or outdated, I have added a DATA_VERSION to the frontend. Simply refresh the page or clear your localStorage in the browser console:

localStorage.clear(); window.location.reload();

1. Base OS

Before server setup, point DNS to the server public IP:

www.teknav.ir  A     SERVER_IPV4
teknav.ir      A     SERVER_IPV4
www.teknav.ir  AAAA  SERVER_IPV6   optional
teknav.ir      AAAA  SERVER_IPV6   optional

Verify DNS from your workstation:

dig +short www.teknav.ir
dig +short teknav.ir

1. Base OS

Use Debian 12/13. Update the server:

sudo apt update
sudo apt full-upgrade -y
sudo reboot

Reconnect, then install base packages:

sudo apt update
sudo apt install -y \
  ca-certificates \
  curl \
  gnupg \
  git \
  rsync \
  build-essential \
  nginx \
  postgresql \
  postgresql-contrib \
  redis-server \
  certbot \
  python3-certbot-nginx \
  logrotate \
  ufw \
  unzip

Install Node.js 20 from NodeSource:

curl -fsSL https://deb.nodesource.com/setup_20.x | sudo -E bash -
sudo apt install -y nodejs
node -v
npm -v

Optional but useful:

sudo timedatectl set-timezone Asia/Tehran
timedatectl

2. System User and Directories

Create the non-login service user:

sudo useradd --system --home /opt/teknav --shell /usr/sbin/nologin teknav || true

Create required directories:

sudo mkdir -p \
  /opt/teknav/current \
  /opt/teknav/releases \
  /etc/teknav \
  /var/lib/teknav/uploads \
  /var/backups/teknav

Set ownership and permissions:

sudo chown -R teknav:teknav /opt/teknav /var/lib/teknav /var/backups/teknav
sudo chown root:teknav /etc/teknav
sudo chmod 755 /opt/teknav
sudo chmod 750 /opt/teknav/current /opt/teknav/releases /var/lib/teknav /var/backups/teknav /etc/teknav

3. PostgreSQL

Enable PostgreSQL:

sudo systemctl enable --now postgresql
sudo systemctl status postgresql --no-pager

Create a strong database password:

openssl rand -base64 32

Create the database user and database. Replace CHANGE_ME_STRONG_DB_PASSWORD first:

sudo -u postgres psql

Inside psql:

CREATE USER teknav WITH PASSWORD 'CHANGE_ME_STRONG_DB_PASSWORD';
CREATE DATABASE teknav OWNER teknav;
\q

Confirm local access:

psql "postgresql://teknav:CHANGE_ME_STRONG_DB_PASSWORD@127.0.0.1:5432/teknav?schema=public" -c "SELECT 1;"

Keep PostgreSQL local-only. Confirm it is not listening publicly:

sudo ss -ltnp | grep 5432

4. Redis

Enable Redis:

sudo systemctl enable --now redis-server
sudo systemctl status redis-server --no-pager
redis-cli ping

Expected:

PONG

Keep Redis local-only. Confirm:

sudo ss -ltnp | grep 6379

5. Upload Project Files

Choose one deployment source.

Option A, clone from Git:

sudo -u teknav git clone REPLACE_WITH_REPO_URL /opt/teknav/current
cd /opt/teknav/current

Option B, sync from your local machine to the server:

rsync -az --delete \
  --exclude node_modules \
  --exclude backend/node_modules \
  --exclude dist \
  --exclude .git \
  --exclude '*.zip' \
  --exclude 'backend/query_engine*.node' \
  --exclude 'backend/query_engine*.node.gz' \
  --exclude 'backend/schema-engine*' \
  --exclude 'backend/schema-engine*.gz' \
  ./ USER@SERVER_IP:/tmp/teknav-upload/

ssh USER@SERVER_IP
sudo rsync -a --delete /tmp/teknav-upload/ /opt/teknav/current/
sudo chown -R teknav:teknav /opt/teknav/current
cd /opt/teknav/current

Confirm key files exist:

test -f package.json
test -f backend/package.json
test -f backend/prisma/schema.prisma
test -f nginx/nginx.conf.example
test -f deploy/systemd/teknav-backend.service

6. Environment File

Create the production env file:

cd /opt/teknav/current
sudo cp .env.prod.example /etc/teknav/backend.env
sudo chown root:teknav /etc/teknav/backend.env
sudo chmod 640 /etc/teknav/backend.env
sudo nano /etc/teknav/backend.env

Set at least these values:

DATABASE_URL="postgresql://teknav:CHANGE_ME_STRONG_DB_PASSWORD@127.0.0.1:5432/teknav?schema=public"
REDIS_URL="redis://127.0.0.1:6379"
SESSION_SECRET=CHANGE_ME_48_BYTE_RANDOM_SECRET
COOKIE_DOMAIN=.teknav.ir
CORS_ORIGINS=https://www.teknav.ir,https://teknav.ir
OAUTH_CALLBACK_BASE=https://www.teknav.ir
NODE_ENV=production
PORT=3010
UPLOAD_DIR=/var/lib/teknav/uploads
BACKUP_DIR=/var/backups/teknav
LOG_LEVEL=info

Generate SESSION_SECRET:

openssl rand -base64 48

Set SMTP when email verification, password reset, campaign sending, and the weekly digest should work. If SMTP_HOST is empty, email jobs log what they would send and do not deliver mail:

SMTP_HOST=smtp.example.com
SMTP_PORT=587
SMTP_USER=...
SMTP_PASS=...
SMTP_FROM=no-reply@teknav.ir

Set SMS OTP through Kavenegar:

SMS_PROVIDER=kavenegar
SMS_API_KEY=...
SMS_SENDER=

Web push notifications (Phase 12). Generate VAPID keys once on the server:

node -e "const wp = require('web-push'); const k = wp.generateVAPIDKeys(); console.log(JSON.stringify(k,null,2))"

Then add to /etc/teknav/backend.env:

VAPID_PUBLIC_KEY=<paste_generated_public_key>
VAPID_PRIVATE_KEY=<paste_generated_private_key>
VAPID_SUBJECT=mailto:admin@teknav.ir

If these are not set, the push endpoints return 503 and sendPushToUser is a no-op — no crash, no side effects. Set them before announcing push to users.

Zarinpal payment gateway (required for membership subscriptions; until set, subscribe endpoint returns payment_unavailable):

ZARINPAL_MERCHANT_ID=xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx
ZARINPAL_SANDBOX=0
MEMBERSHIP_PRICE_IRR=5000000
MEMBERSHIP_DURATION_DAYS=365

File storage — local disk is the default and requires no env vars. Uploads go to UPLOAD_DIR (default /var/lib/teknav/uploads). S3-compatible storage is a future option wired but not yet active:

# Leave unset (or set to "local") to use local disk — recommended for now
# STORAGE_BACKEND=local

# S3 migration (Phase 12+): uncomment and fill in when needed
# STORAGE_BACKEND=s3
# S3_ENDPOINT=https://storage.example.ir
# S3_BUCKET=teknav-uploads
# S3_ACCESS_KEY=...
# S3_SECRET_KEY=...

OAuth callback URLs, if Google/GitHub login is enabled:

https://www.teknav.ir/api/v1/auth/oauth/google/callback
https://www.teknav.ir/api/v1/auth/oauth/github/callback

Validate env file readability for the service user:

sudo -u teknav test -r /etc/teknav/backend.env

Run the static production preflight before dependency installation/build steps:

sudo -u teknav npm run preflight:prod

The current expected warning is that no Prisma migrations directory exists yet; production schema application uses the documented prisma:apply fallback.

7. Install Dependencies

Use the committed lockfiles and install dependencies on the Linux host. Do not copy node_modules, generated Prisma engines, or any Windows-generated Prisma binaries from a workstation.

cd /opt/teknav/current
sudo -u teknav npm ci

Install backend dependencies:

cd /opt/teknav/current/backend
sudo -u teknav npm ci

If the server has enough memory and native image conversion is required, ensure optional sharp installed successfully:

cd /opt/teknav/current/backend
sudo -u teknav npm ls sharp || true

8. Prisma Generate, Schema Apply, and Seed

Generate Prisma client:

cd /opt/teknav/current/backend
sudo -u teknav npm run prisma:generate

This must run on the Linux server after npm ci. Prisma downloads/generates platform-specific engines for the target host; Windows-generated query_engine*.node and schema-engine* files are local artifacts and must not be deployed.

Apply schema:

sudo -u teknav npm run prisma:apply

prisma:apply is intentional. It uses migrations if they exist and falls back to prisma db push --skip-generate while this repo has no full migrations directory.

Seed production data:

sudo -u teknav npm run seed

Important:

  • Operator-facing seed credentials are in root ACCOUNTS.md.
  • ACCOUNTS.md is gitignored and nginx-blocked.
  • Every seeded user gets a URL-safe username and can log in with email or @username.
  • Rotate all seeded passwords after first login.
  • After rotation, remove resetPasswordOnSeed for accounts that must not be reset on future seeds.

9. Generate SEO Assets and Build

Generate crawler artifacts:

cd /opt/teknav/current
sudo -u teknav npm run seo:sitemap

Generate local Open Graph images:

sudo -u teknav npm run og:images

Build frontend:

sudo -u teknav npm run preflight:prod
sudo -u teknav npm run build

Build backend:

cd /opt/teknav/current/backend
sudo -u teknav npm run build

Confirm build output:

test -f /opt/teknav/current/dist/index.html
test -f /opt/teknav/current/public/sitemap.xml
test -f /opt/teknav/current/public/robots.txt
test -f /opt/teknav/current/public/feed.xml
test -f /opt/teknav/current/public/feeds/topic-ai.xml
test -f /opt/teknav/current/backend/dist/src/server.js

10. systemd Services and Timers

Install units:

cd /opt/teknav/current
sudo cp deploy/systemd/*.service /etc/systemd/system/
sudo cp deploy/systemd/*.timer /etc/systemd/system/
sudo cp deploy/logrotate/teknav /etc/logrotate.d/teknav
sudo chmod 644 /etc/systemd/system/teknav-*.service /etc/systemd/system/teknav-*.timer
sudo chmod 644 /etc/logrotate.d/teknav
sudo chmod +x scripts/systemd-backup.sh
sudo systemctl daemon-reload

Start backend:

sudo systemctl enable --now teknav-backend.service
sudo systemctl status teknav-backend.service --no-pager

Start scheduled publishing and backups:

sudo systemctl enable --now teknav-publish-due.timer
sudo systemctl enable --now teknav-weekly-digest.timer
sudo systemctl enable --now teknav-backup.timer
sudo systemctl list-timers 'teknav-*'

Useful service commands:

sudo journalctl -u teknav-backend -f
sudo systemctl restart teknav-backend
sudo systemctl start teknav-migrate
sudo journalctl -u teknav-migrate -n 100 --no-pager
sudo systemctl start teknav-publish-due
sudo systemctl start teknav-weekly-digest
sudo systemctl start teknav-backup

Local backend health before nginx:

curl -fsS http://127.0.0.1:3010/api/v1/health
curl -fsS http://127.0.0.1:3010/api/v1/ready

Expected:

{"ok":true}

/api/v1/health is minimal public health. /api/v1/ready returns 503 if PostgreSQL or Redis is unavailable. /api/v1/admin/health is the authenticated diagnostics endpoint with runtime feature flags.

11. nginx Before TLS

The repository nginx template contains placeholder certificate paths for an already-provisioned certificate. For a fresh Certbot install, create a temporary HTTP-only nginx config first:

sudo tee /etc/nginx/sites-available/teknav >/dev/null <<'NGINX'
server {
    listen 80;
    listen [::]:80;
    server_name teknav.ir www.teknav.ir;

    root /opt/teknav/current/dist;
    index index.html;
    charset utf-8;

    location = /ACCOUNTS.md {
        deny all;
        return 404;
    }

    location /api/ {
        proxy_pass http://127.0.0.1:3010;
        proxy_http_version 1.1;
        proxy_set_header Host $host;
        proxy_set_header X-Real-IP $remote_addr;
        proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
        proxy_set_header X-Forwarded-Proto $scheme;
        proxy_set_header Connection "";
    }

    location /uploads/ {
        alias /var/lib/teknav/uploads/;
        try_files $uri =404;
    }

    location / {
        try_files $uri $uri/ /index.html;
    }
}
NGINX

Enable site:

sudo ln -sf /etc/nginx/sites-available/teknav /etc/nginx/sites-enabled/teknav
sudo rm -f /etc/nginx/sites-enabled/default
sudo nginx -t
sudo systemctl enable --now nginx
sudo systemctl reload nginx

Confirm HTTP works:

curl -I http://www.teknav.ir/
curl -fsS http://www.teknav.ir/api/v1/health

12. TLS with Certbot

Issue certificates:

sudo certbot --nginx -d www.teknav.ir -d teknav.ir

Choose redirect to HTTPS when Certbot asks.

Test renewal:

sudo certbot renew --dry-run

13. Final nginx Production Config

Install the production nginx template:

cd /opt/teknav/current
sudo cp nginx/nginx.conf.example /etc/nginx/sites-available/teknav

Edit certificate paths in /etc/nginx/sites-available/teknav:

sudo nano /etc/nginx/sites-available/teknav

Replace:

ssl_certificate     /etc/nginx/ssl/_.teknav.ir-fullchain.pem;
ssl_certificate_key /etc/nginx/ssl/_.teknav.ir-privateKey.pem;

With Certbot paths:

ssl_certificate     /etc/letsencrypt/live/www.teknav.ir/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/www.teknav.ir/privkey.pem;

Validate and reload:

sudo nginx -t
sudo systemctl reload nginx

The production template does all of this:

  • Redirects apex teknav.ir to https://www.teknav.ir.
  • Serves the SPA from dist/.
  • Proxies /api/ to Fastify.
  • Disables buffering/compression for SSE paths.
  • Serves uploads from /var/lib/teknav/uploads.
  • Blocks dotfiles and /ACCOUNTS.md.
  • Enables gzip for JS, CSS, JSON, XML, RSS, fonts, and SVG.
  • Caches immutable static assets.
  • Keeps crawler files public: /robots.txt, /sitemap.xml, /feed.xml.

After the site is stable for a few days, increase HSTS in nginx from one week to one year:

add_header Strict-Transport-Security "max-age=31536000; includeSubDomains" always;

Then:

sudo nginx -t
sudo systemctl reload nginx

14. Firewall

Allow only SSH, HTTP, and HTTPS:

sudo ufw allow OpenSSH
sudo ufw allow 80/tcp
sudo ufw allow 443/tcp
sudo ufw enable
sudo ufw status verbose

Confirm private services are not public:

sudo ss -ltnp | grep -E ':(3010|5432|6379)'

Expected listeners should be local only.

15. Production Validation

Run these checks after nginx and TLS are live:

curl -fsS https://www.teknav.ir/api/v1/health
curl -fsS https://www.teknav.ir/api/v1/ready
curl -I https://www.teknav.ir/
curl -I https://www.teknav.ir/articles
curl -I https://www.teknav.ir/topics/ai
curl -I https://www.teknav.ir/newsletter
curl -I https://www.teknav.ir/tag/AI
curl -I https://www.teknav.ir/sitemap.xml
curl -I https://www.teknav.ir/robots.txt
curl -I https://www.teknav.ir/feed.xml
curl -I https://www.teknav.ir/feeds/topic-ai.xml
curl -I https://www.teknav.ir/ACCOUNTS.md

Expected:

  • /api/v1/health returns minimal ok: true.
  • /api/v1/ready returns 200 while PostgreSQL and Redis are healthy; it returns 503 if either dependency is unavailable.
  • /, /articles, /topics/ai, /newsletter, and a known /tag/... page return 200.
  • /robots.txt, /sitemap.xml, /feed.xml, and topic/author feeds under /feeds/ return 200.
  • /ACCOUNTS.md returns 403 or 404.

Run the production smoke script:

cd /opt/teknav/current
sudo -u teknav npm run preflight:prod
sudo -u teknav TEKNAV_SMOKE_URL=https://www.teknav.ir npm run smoke:prod

Optional custom article smoke list:

sudo -u teknav TEKNAV_SMOKE_URL=https://www.teknav.ir TEKNAV_SMOKE_ARTICLES=/article/agentic-ai-production,/article/ai-clean-data npm run smoke:prod

16. Functional Validation Checklist

Validate on Linux/staging or production-like infrastructure:

cd /opt/teknav/current/backend
sudo -u teknav npm run prisma:generate
sudo -u teknav npm run prisma:apply
sudo -u teknav npm run seed
cd /opt/teknav/current
sudo -u teknav npm run preflight:prod
sudo -u teknav npm run seo:sitemap
sudo -u teknav npm run og:images
sudo -u teknav npm run build
cd /opt/teknav/current/backend
sudo -u teknav npm run build
sudo -u teknav TEST_REQUIRE_SERVER=1 TEST_BASE_URL=http://127.0.0.1:3010 npm test
cd /opt/teknav/current
sudo -u teknav TEKNAV_SMOKE_URL=https://www.teknav.ir npm run smoke:prod

Manually verify:

  • Login by email.
  • Login by @username.
  • Profile edit rules for reader, writer, editor, reviewer, and admin.
  • Password change with current password.
  • Comments, likes, and flags.
  • Reading history and reading lists.
  • Writer follows.
  • Topic follow/unfollow on /topics/ai.
  • Writer panel submission: draft or review only, no direct public publish.
  • Reviewer/editor approval publishes only after Phase 10 quality checks pass.
  • GET /api/v1/admin/panel-metrics returns role-aware dashboard metrics for admin, editor, writer, and reviewer accounts.
  • Article share buttons record first-party share_clicked events.
  • Notifications and SSE.
  • Newsletter subscribe/unsubscribe/campaign send and public archive.
  • Article series pages.
  • Topic hub pages.
  • Admin analytics, SEO audit, and review workflow.

17. Updates

For a normal update:

cd /opt/teknav/current
sudo -u teknav git pull
sudo -u teknav npm ci
cd /opt/teknav/current/backend
sudo -u teknav npm ci
sudo -u teknav npm run prisma:generate
sudo -u teknav npm run prisma:apply
cd /opt/teknav/current
sudo -u teknav npm run seo:sitemap
sudo -u teknav npm run og:images
sudo -u teknav npm run build
cd /opt/teknav/current/backend
sudo -u teknav npm run build
sudo systemctl restart teknav-backend
sudo nginx -t
sudo systemctl reload nginx
cd /opt/teknav/current
sudo -u teknav TEKNAV_SMOKE_URL=https://www.teknav.ir npm run smoke:prod

Run npm run seo:sitemap after changing public routes, article slugs, article dates, categories, authors, topic hubs, newsletter archive routes, series, or crawler files.

Run npm run og:images after changing article Open Graph metadata.

18. Safer Release Directory Flow

For lower-risk deployments, deploy to a timestamped release and atomically switch current:

RELEASE="$(date +%Y%m%d-%H%M%S)"
sudo mkdir -p "/opt/teknav/releases/$RELEASE"
sudo rsync -a /tmp/teknav-upload/ "/opt/teknav/releases/$RELEASE/"
sudo chown -R teknav:teknav "/opt/teknav/releases/$RELEASE"
cd "/opt/teknav/releases/$RELEASE"
sudo -u teknav npm ci
cd backend
sudo -u teknav npm ci
sudo -u teknav npm run prisma:generate
sudo -u teknav npm run prisma:apply
cd ..
sudo -u teknav npm run seo:sitemap
sudo -u teknav npm run og:images
sudo -u teknav npm run build
cd backend
sudo -u teknav npm run build
cd ..
sudo ln -sfn "/opt/teknav/releases/$RELEASE" /opt/teknav/current
sudo systemctl restart teknav-backend
sudo nginx -t
sudo systemctl reload nginx

Rollback to the previous release:

ls -1 /opt/teknav/releases
sudo ln -sfn /opt/teknav/releases/PREVIOUS_RELEASE /opt/teknav/current
sudo systemctl restart teknav-backend
sudo nginx -t
sudo systemctl reload nginx

Only roll back code after a schema change if the older code is compatible with the newer schema.

19. Backups

Run backup manually:

sudo systemctl start teknav-backup
sudo journalctl -u teknav-backup -n 100 --no-pager
ls -lh /var/backups/teknav

Timers:

sudo systemctl list-timers 'teknav-*'

Default retention is 14 days. Override in /etc/teknav/backend.env:

BACKUP_RETENTION_DAYS=30

20. Restore

Stop backend before a full restore:

sudo systemctl stop teknav-backend

Restore a backup:

set -a
. /etc/teknav/backend.env
set +a
gunzip -c /var/backups/teknav/teknav-YYYYMMDD-HHMMSS.sql.gz | psql "$DATABASE_URL"

Restart and verify:

sudo systemctl start teknav-backend
curl -fsS https://www.teknav.ir/api/v1/health

21. Logs and Operations

Backend logs:

sudo journalctl -u teknav-backend -f
sudo journalctl -u teknav-backend -n 200 --no-pager

nginx logs:

sudo tail -f /var/log/nginx/access.log
sudo tail -f /var/log/nginx/error.log

Database:

sudo -u postgres psql -d teknav

Redis:

redis-cli info server
redis-cli info memory

Disk:

df -h
du -sh /var/lib/teknav/uploads /var/backups/teknav /opt/teknav/current

22. Security Notes

  • Never deploy Docker for this production model unless the architecture changes.
  • Never put ACCOUNTS.md under public/ or dist/.
  • Keep /etc/teknav/backend.env mode 640 and owned by root:teknav.
  • Rotate seeded passwords after first login.
  • Keep PostgreSQL, Redis, and Fastify bound to localhost.
  • Do not add remote analytics scripts, remote fonts, remote images, or CDN assets.
  • Keep nginx gzip enabled for static assets and JSON.
  • Keep SSE paths unbuffered:
    • /api/v1/auth/notifications/stream
    • /api/v1/admin/stream
  • VAPID_PRIVATE_KEY must stay in /etc/teknav/backend.env only. The public key is exposed via /api/v1/auth/push/vapid-key (authenticated) — never embed it in nginx config or any public file.

23. Troubleshooting

Backend does not start:

sudo systemctl status teknav-backend --no-pager
sudo journalctl -u teknav-backend -n 200 --no-pager

Schema apply fails:

cd /opt/teknav/current/backend
sudo -u teknav npm run prisma:generate
sudo -u teknav npm run prisma:apply

Frontend 404s on refresh:

sudo nginx -T | grep -A20 "location /"

Confirm try_files $uri $uri/ /index.html; exists.

API returns 502:

curl -fsS http://127.0.0.1:3010/api/v1/health
sudo journalctl -u teknav-backend -n 100 --no-pager

Uploads return 404:

grep UPLOAD_DIR /etc/teknav/backend.env
sudo ls -lah /var/lib/teknav/uploads
sudo chown -R teknav:teknav /var/lib/teknav/uploads
sudo nginx -T | grep -A12 "location /uploads/"

SSE notifications or admin live widgets do not update:

redis-cli ping
sudo nginx -T | grep -A20 "admin/stream"
sudo nginx -T | grep -A20 "notifications/stream"

Certbot renewal fails:

sudo certbot renew --dry-run
sudo nginx -t
sudo journalctl -u nginx -n 100 --no-pager

SMTP fails:

grep '^SMTP_' /etc/teknav/backend.env
sudo journalctl -u teknav-backend -n 200 --no-pager | grep -i smtp

Kavenegar OTP fails:

grep '^SMS_' /etc/teknav/backend.env
sudo journalctl -u teknav-backend -n 200 --no-pager | grep -i otp

Web push not delivering:

# Confirm VAPID keys are set
grep '^VAPID_' /etc/teknav/backend.env

# Check for 410 stale-subscription cleanup in logs
sudo journalctl -u teknav-backend -n 200 --no-pager | grep -i push

# Verify PushSubscription table exists
sudo -u postgres psql -d teknav -c '\dt "PushSubscription"'

# Verify at least one subscription row exists for a test user
sudo -u postgres psql -d teknav -c 'SELECT "userId", "endpoint" FROM "PushSubscription" LIMIT 5;'

If the table is missing, re-run cd backend && npm run prisma:apply.

Crawler files missing:

cd /opt/teknav/current
sudo -u teknav npm run seo:sitemap
ls -lh public/sitemap.xml public/robots.txt public/feed.xml
sudo nginx -t
sudo systemctl reload nginx