Import documentation: CLAUDE.md

Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.

CLAUDE.md

@@ -0,0 +1,125 @@
+# CLAUDE.md
+
+This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
+
+## What this is
+
+A single-file Telegram bot (`bot.py`) that receives files (and URLs) and uploads them to Google Drive. Uses **Pyrogram** (MTProto, v2+) for direct Telegram communication and **rclone** (with `gdrive:` remote) for Drive uploads. All async, all in one file.
+
+## Environment setup
+
+```bash
+source venv/bin/activate
+pip install -r requirements.txt
+```
+
+Required `.env` variables: `BOT_TOKEN`, `API_ID`, `API_HASH`. The rclone remote (`gdrive:`) must be authenticated separately via `rclone config`.
+
+## Running the bot
+
+**Production: runs as a systemd service** (`/etc/systemd/system/tg-gdrive-bot.service`, enabled on boot, `Restart=always` replaces the old `run_bot.sh` watchdog). Manage it with:
+
+```bash
+systemctl status tg-gdrive-bot      # state
+systemctl restart tg-gdrive-bot     # restart (or use the in-bot /restart)
+systemctl stop tg-gdrive-bot        # stop (do this before any manual run, or the session conflicts)
+journalctl -u tg-gdrive-bot -f      # follow logs (also appended to bot.log for the in-bot /logs)
+```
+
+The unit uses the venv interpreter (`venv/bin/python -u bot.py`) and a PATH that includes `/usr/local/bin` (for `deno`). stdout/stderr append to `bot.log` so the in-bot `/logs` command keeps working. `run_bot.sh` is the legacy nohup watchdog — superseded by systemd; don't run both.
+
+Manual run (debugging only — stop the service first):
+
+```bash
+systemctl stop tg-gdrive-bot
+source venv/bin/activate
+python bot.py
+```
+
+No real tests exist beyond the stub `test_local_api.py`.
+
+## Bot commands
+
+| Command | Auth required | Description |
+|---------|---------------|-------------|
+| `/start` | No | Upload instructions + system info |
+| `/help` | No | Full command reference |
+| `/list` | Yes | Last 10 files in Drive |
+| `/listall` | Yes | All files (paginated) |
+| `/search <q>` | Yes | Search files by name |
+| `/share <name>` | Yes | Make file public + share link |
+| `/status` | Yes | Uptime, RAM, CPU, disk |
+| `/logs` | Yes | Last 50 log lines |
+| `/adduser @user` | Yes | Grant full access |
+| `/removeuser @user` | Yes | Revoke full access |
+| `/setquota @user <size>` | Yes | Set daily upload quota |
+| `/clearquota @user` | Yes | Remove quota |
+| `/quotas` | Yes | Overview all quotas |
+| `/myquota` | No | Check own quota |
+| `/restart` | Yes | Graceful restart via `os.execv` |
+
+## Architecture
+
+All logic in `bot.py` (~1368 lines). Key components:
+
+- **`handle_file`** (`@app.on_message(filters.media)`) — receives any media type (document, photo, video, audio, voice, sticker, video_note, animation). Routes to one of two upload strategies based on file size:
+  - **`_download_then_upload`** (< 2 GB) — downloads to temp dir via `client.download_media` with retry logic (3 attempts, 5/15/30s backoff) and live progress updates, then `rclone copy` to Drive. Cleans up temp files on success.
+  - **`_stream_to_drive`** (≥ 2 GB) — streams directly through RAM via `client.stream_media` → `rclone rcat` stdin, with retry logic and speed tracking. Zero disk usage.
+- **`handle_url`** (`@app.on_message(filters.text)`) — detects URLs in text messages, downloads via `rclone copyurl` (RAM-only, no disk), saves to Drive. Pulse animation during transfer.
+- **`classify_to_folder`** — routes files into Drive subfolders by extension/MIME (Videos, Music, Photos, Software, Archives, Models, Docs, etc.).
+- **`resolve_filename`** — uses caption as filename override if provided; falls back to `file_name` attribute or generates one with appropriate extension.
+- **`send_links`** — after upload, queries Drive via `rclone lsjson` for the file ID, then sends direct-download + wget links.
+
+### Auth system
+
+- **`AUTHORIZED_USERS`** set persisted in `authorized_users.json`. Anyone in this set has unlimited upload access.
+- **Quota system** in `quotas.json` schema: `{"quotas": {"username": bytes}, "usage": {"username": {"YYYY-MM-DD": bytes}}}`. Non-auth users can upload if they have a quota with remaining bytes today. Quotas auto-reset at midnight UTC.
+
+### Upload methods
+
+| Size | Method | Disk usage | CLI tool |
+|------|--------|------------|----------|
+| < 2 GB | Download then upload | Temp file on disk | `rclone copy` |
+| ≥ 2 GB | RAM stream | None | `rclone rcat` |
+| URLs (any size) | RAM stream | None | `rclone copyurl` |
+| Media-site URLs | yt-dlp to temp dir, then upload | Temp file on disk | `yt-dlp` + `rclone copy` |
+
+The 2 GB threshold is `STREAM_THRESHOLD` at line 53.
+
+### yt-dlp media downloads
+
+URLs whose domain matches `YTDLP_DOMAINS` (YouTube, Instagram, TikTok, Twitter/X, Facebook, Reddit, Pinterest, Tumblr, LinkedIn, Aparat) or `NSFW_DOMAINS` (PornHub, XVideos, xHamster, YouPorn, RedTube, SpankBang) are routed by `_process_url` to `_ytdlp_download` instead of `rclone copyurl`. Files are downloaded with yt-dlp (venv package; ffmpeg merges streams) into a temp dir, then uploaded to `gdrive:MEDIA-DLS/<category>/` — category from `classify_to_folder` (Videos, Music, …), NSFW domains always go to `MEDIA-DLS/NSFW`. Adding the word `mp3` or `audio` after the link extracts audio (`-x --audio-format mp3`). Playlists are capped at 50 items; runtime capped at 3 h (livestreams). If `cookies.txt` exists in the bot dir it is passed via `--cookies` — required for Instagram and for YouTube (this server's IP is bot-flagged by YouTube). `curl_cffi` is installed for browser impersonation (needed by e.g. SpankBang). Keep yt-dlp updated (`venv/bin/pip install -U yt-dlp`) — extractors break when sites change.
+
+### Inline file-action buttons
+
+Every `send_links` message carries an inline keyboard (🌍 Share · ℹ️ Info · 📂 Move · 🗑 Delete with confirm step · 📥 Send to Telegram). Buttons reference an 8-hex-char token stored in `FILE_ACTIONS` (persisted to `file_actions.json`, capped at 500 entries, survives restarts). `handle_buttons` (`@app.on_callback_query()`) dispatches on the `action:token[:dest]` callback-data format and runs rclone (`link`, `moveto`, `deletefile`, `lsjson`). Authorized users can act on any file; quota users only on files they uploaded (`owner` field).
+
+### Torrents (aria2c)
+
+Magnet links (detected in `handle_url` via `MAGNET_RE`) and `.torrent` files (detected in `handle_file` by extension/mime) route to `process_torrent`. A minimal bencode decoder (`_bdecode` / `torrent_files`) reads the file list **without** sanitizing names (they must match what aria2 writes to disk and are used verbatim as Drive paths). Magnets first fetch metadata only (`_aria2_fetch_metadata`). Download strategy by total size: **< 8 GB** downloads the whole torrent then uploads each file; **≥ 8 GB** (`TORRENT_STREAM_THRESHOLD`) uses per-file mode (`--select-file=N`, download→upload→delete per file) to keep peak disk low. There is no RAM-stream path — BitTorrent is out-of-order, so torrents always hit disk. Work dir is `dl_tmp/` on `/dev/sda1` (NOT `/tmp`, which is RAM-backed tmpfs, only 3.9 GB). Pre-checks: free disk (`shutil.disk_usage`, needs biggest-file or total + 2 GB headroom) and quota. Files upload to `gdrive:Torrents/<name>/…` preserving structure. `aria2c` and the seeding-disabled flags live in `ARIA2_COMMON`; progress parsed by `ARIA2_PROGRESS_RE`.
+
+### Telegram delivery (Drive → Telegram)
+
+`/get <query>` and the 📥 button both call `deliver_to_telegram`: look up the file (`get_drive_entry` / `_find_drive_file`), refuse if over `TG_SEND_LIMIT` (~2 GB Telegram bot cap, offers `/share` instead), rclone-copy to `dl_tmp/` with progress (parsed from `--stats-one-line`), then `send_video`/`send_audio`/`send_document` by extension with an upload-progress callback. This is the only path that sends files *out* of Drive.
+
+### yt-dlp auto-updater
+
+`ytdlp_update_loop` (started in `_main()` alongside `app.start()`/`idle()`) runs `pip install -U yt-dlp` in the venv every 24 h and logs version changes. Subprocesses pick up the new version immediately.
+
+Spotify links (`SPOTIFY_DOMAINS`) are routed to `_spotify_download`, which runs **spotdl** (Spotify metadata → audio downloaded from YouTube Music) and uploads results to `MEDIA-DLS/Music`. Because the audio comes from YouTube, this also requires `cookies.txt` on this server. Deno is installed at `/root/.config/spotdl/deno` (symlinked to `/usr/local/bin/deno`) — yt-dlp needs it for YouTube JS challenges. Both yt-dlp and spotdl uploads share `_upload_media_files`.
+
+### Key tech stack
+
+- **Pyrogram** v2+ — async MTProto client (`Client`), `filters.media` / `filters.command()` handlers
+- **rclone** — Drive operations: `copy`, `rcat`, `copyurl`, `lsjson`, `link`
+- **`asyncio.create_subprocess_exec`** — all rclone calls are non-blocking subprocesses
+- **`psutil`** — RAM/CPU/disk stats for `/status`
+- **`python-dotenv`** — loads `.env`
+
+### Key constraints
+
+- All handlers are `async def` — Pyrogram is fully async.
+- rclone subprocesses use `asyncio.create_subprocess_exec` — do not replace with `subprocess.run`.
+- Temp files only deleted on successful upload (manual retry on failure).
+- Filenames cleaned via `sanitize_filename` (NFKC normalization, emoji removal, illegal char stripping).
+- The bot can also handle files sent by non-auth users if they have a daily quota assigned.

run_bot.sh

@@ -0,0 +1,19 @@
+#!/usr/bin/env bash
+# run_bot.sh — starts bot.py and auto-restarts it on crash
+set -euo pipefail
+
+DIR="$(cd "$(dirname "$0")" && pwd)"
+VENV="$DIR/venv/bin/activate"
+LOG="$DIR/bot.log"
+RETRY_DELAY=5   # seconds to wait before restarting after a crash
+
+echo "[run_bot] Starting watchdog. Logs → $LOG" | tee -a "$LOG"
+
+while true; do
+    echo "[run_bot] $(date -u '+%Y-%m-%d %H:%M:%S UTC') — launching bot.py" | tee -a "$LOG"
+    source "$VENV"
+    python -u "$DIR/bot.py" >> "$LOG" 2>&1 || true
+    EXIT=$?
+    echo "[run_bot] $(date -u '+%Y-%m-%d %H:%M:%S UTC') — bot.py exited (code $EXIT), restarting in ${RETRY_DELAY}s..." | tee -a "$LOG"
+    sleep "$RETRY_DELAY"
+done

send_pfp.py

@@ -0,0 +1,29 @@
+import asyncio, os
+from dotenv import load_dotenv
+from pyrogram import Client
+
+load_dotenv()
+app = Client("gdrive_bot", api_id=int(os.getenv("API_ID")),
+             api_hash=os.getenv("API_HASH"), bot_token=os.getenv("BOT_TOKEN"))
+
+TARGET = "coaxys"
+CAPTION = (
+    "Here's your new profile picture. \U0001F5BC\n\n"
+    "To set it: open @BotFather → /mybots → this bot → "
+    "Edit Bot → Edit Botpic → send this image."
+)
+
+async def main():
+    async with app:
+        me = await app.get_me()
+        print("Bot:", me.username)
+        try:
+            await app.send_photo(TARGET, "bot_pfp.jpg", caption=CAPTION)
+            # also send as a file so the original (uncompressed) is available for BotFather
+            await app.send_document(TARGET, "bot_pfp.jpg",
+                                    caption="Original file (use this for the sharpest result).")
+            print("RESULT: sent to", TARGET)
+        except Exception as e:
+            print("RESULT: FAILED ->", type(e).__name__, str(e))
+
+asyncio.run(main())

set_pfp.py

@@ -0,0 +1,19 @@
+import asyncio, os
+from dotenv import load_dotenv
+from pyrogram import Client
+
+load_dotenv()
+app = Client("gdrive_bot", api_id=int(os.getenv("API_ID")),
+             api_hash=os.getenv("API_HASH"), bot_token=os.getenv("BOT_TOKEN"))
+
+async def main():
+    async with app:
+        me = await app.get_me()
+        print("Bot:", me.username, "| id:", me.id)
+        try:
+            await app.set_profile_photo(photo="bot_pfp.jpg")
+            print("RESULT: set_profile_photo OK")
+        except Exception as e:
+            print("RESULT: FAILED ->", type(e).__name__, str(e))
+
+asyncio.run(main())

sort_drive.py

@@ -0,0 +1,74 @@
+#!/usr/bin/env python3
+"""
+One-shot script: reads the GDrive UH folder, creates type folders,
+and moves each file into the right one.
+"""
+import json, subprocess, sys
+
+REMOTE = "gdrive:"
+ROOT_ID = "1fEOd5m5psVYeN0hK6G_UJx_XwmXZcZ6l"
+BASE = ["rclone", "--drive-root-folder-id", ROOT_ID]
+
+def rclone(*args):
+    cmd = BASE + list(args)
+    print(f"  $ {' '.join(cmd)}")
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print(f"  ERROR: {r.stderr.strip()}")
+    return r.returncode == 0
+
+def classify(name: str, mime: str) -> str:
+    ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
+    if mime.startswith("video/") or ext in ("mp4", "mkv", "avi", "mov", "webm", "flv"):
+        return "Videos"
+    if mime.startswith("audio/") or ext in ("mp3", "flac", "wav", "ogg", "m4a", "aac"):
+        return "Music"
+    if ext in ("exe", "msi", "deb", "rpm", "dmg", "appimage"):
+        return "Software"
+    if ext in ("zip", "rar", "7z", "tar", "gz", "bz2", "xz", "zst"):
+        return "Archives"
+    if ext in ("gguf", "bin", "safetensors", "pt", "ckpt", "pth"):
+        return "Models"
+    if ext in ("pdf", "doc", "docx", "txt", "md", "csv", "xlsx", "pptx"):
+        return "Docs"
+    return "Other"
+
+# --- list files ---
+print("Listing files...")
+r = subprocess.run(BASE + ["lsjson", REMOTE], capture_output=True, text=True)
+files = json.loads(r.stdout)
+
+# group (skip directories)
+groups: dict[str, list] = {}
+for f in files:
+    if f["IsDir"]:
+        print(f"  [skip dir] {f['Name']}")
+        continue
+    folder = classify(f["Name"], f["MimeType"])
+    groups.setdefault(folder, []).append(f["Name"])
+
+# print plan
+print("\n=== Move plan ===")
+for folder, names in sorted(groups.items()):
+    print(f"\n📁 {folder}/ ({len(names)} files)")
+    for n in names:
+        print(f"   {n}")
+
+print("\n=== Executing ===")
+created: set = set()
+ok = err = 0
+for folder, names in sorted(groups.items()):
+    if folder not in created:
+        print(f"\nCreating folder: {folder}/")
+        rclone("mkdir", f"{REMOTE}{folder}")
+        created.add(folder)
+    for name in names:
+        dest = f"{REMOTE}{folder}/{name}"
+        src  = f"{REMOTE}{name}"
+        print(f"\nMoving → {folder}/{name[:60]}")
+        if rclone("moveto", src, dest):
+            ok += 1
+        else:
+            err += 1
+
+print(f"\n=== Done: {ok} moved, {err} errors ===")

sort_root.py

@@ -0,0 +1,68 @@
+#!/usr/bin/env python3
+"""Sort the gdrive root (parent of UH) into type folders."""
+import json, subprocess
+
+REMOTE = "gdrive:"
+BASE = ["rclone"]
+
+SKIP_DIRS = {"Google AI Studio", "Photos", "UH", "Untitled folder",
+             "Models", "Videos", "Archives", "Docs", "Config", "ZIM"}
+
+def rclone(*args):
+    cmd = BASE + list(args)
+    print(f"  $ {' '.join(str(a) for a in cmd)}")
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print(f"  ERROR: {r.stderr.strip()}")
+    return r.returncode == 0
+
+def classify(name: str) -> str:
+    ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
+    if ext == "gguf":
+        return "Models"
+    if ext == "zim":
+        return "ZIM"
+    if ext in ("mp4", "mkv", "avi", "mov", "webm"):
+        return "Videos"
+    if ext in ("zip", "rar", "7z", "tar", "gz", "bz2"):
+        return "Archives"
+    if ext in ("conf", "ovpn", "cfg", "ini", "toml", "yaml", "yml"):
+        return "Config"
+    if ext in ("docx", "doc", "pdf", "txt", "md", "csv", "xlsx"):
+        return "Docs"
+    return "Other"
+
+print("Listing root...")
+r = subprocess.run(BASE + ["lsjson", REMOTE], capture_output=True, text=True)
+files = json.loads(r.stdout)
+
+groups: dict[str, list] = {}
+for f in files:
+    if f["IsDir"]:
+        print(f"  [skip dir] {f['Name']}")
+        continue
+    folder = classify(f["Name"])
+    groups.setdefault(folder, []).append(f["Name"])
+
+print("\n=== Move plan ===")
+for folder, names in sorted(groups.items()):
+    print(f"\n📁 {folder}/ ({len(names)} files)")
+    for n in names:
+        print(f"   {n}")
+
+print("\n=== Executing ===")
+created: set = set()
+ok = err = 0
+for folder, names in sorted(groups.items()):
+    if folder not in created:
+        print(f"\nCreating folder: {folder}/")
+        rclone("mkdir", f"{REMOTE}{folder}")
+        created.add(folder)
+    for name in names:
+        print(f"\nMoving → {folder}/{name[:70]}")
+        if rclone("moveto", f"{REMOTE}{name}", f"{REMOTE}{folder}/{name}"):
+            ok += 1
+        else:
+            err += 1
+
+print(f"\n=== Done: {ok} moved, {err} errors ===")

sort_root_full.py

@@ -0,0 +1,89 @@
+#!/usr/bin/env python3
+"""Full sort of gdrive root — classifies every loose file and moves it to the right folder."""
+import json, subprocess, sys
+
+REMOTE = "gdrive:"
+BASE   = ["rclone"]
+# Folders that should never be touched (already organised or system folders)
+KEEP_DIRS = {"Google AI Studio", "Photos", "Untitled folder",
+             "Videos", "Music", "Software", "Archives", "Bin",
+             "Docs", "Config", "Models", "ZIM", "SPH"}
+
+def run(*args):
+    cmd = list(args)
+    r = subprocess.run(cmd, capture_output=True, text=True)
+    if r.returncode != 0:
+        print(f"    ERROR: {r.stderr.strip()[:120]}")
+    return r.returncode == 0
+
+def classify(name: str, mime: str) -> str:
+    ext = name.rsplit(".", 1)[-1].lower() if "." in name else ""
+    if mime.startswith("video/") or ext in ("mp4","mkv","avi","mov","webm","flv"):
+        return "Videos"
+    if mime.startswith("audio/") or ext in ("mp3","flac","wav","ogg","m4a","aac"):
+        return "Music"
+    if mime.startswith("image/") or ext in ("jpg","jpeg","png","gif","webp","bmp","heic","heif"):
+        return "Photos"
+    if ext in ("exe","msi","deb","rpm","dmg","appimage"):
+        return "Software"
+    if ext in ("zip","rar","7z","tar","gz","bz2","xz","zst"):
+        return "Archives"
+    if ext == "gguf":
+        return "Models"
+    if ext in ("safetensors","pt","ckpt","pth"):
+        return "Models"
+    if ext == "zim":
+        return "ZIM"
+    if ext in ("conf","ovpn","cfg","ini","toml"):
+        return "Config"
+    if ext in ("pdf","doc","docx","txt","md","csv","xlsx","pptx","json"):
+        return "Docs"
+    if ext == "bin":
+        return "Bin"
+    # Google Docs / Sheets / Slides (no extension, mime-based)
+    gdoc_mimes = ("application/vnd.google-apps.document",
+                  "application/vnd.google-apps.spreadsheet",
+                  "application/vnd.google-apps.presentation",
+                  "application/vnd.google-apps.form",
+                  "application/vnd.openxmlformats")
+    if any(mime.startswith(m) for m in gdoc_mimes):
+        return "Docs"
+    return "Other"
+
+print("Listing root (depth 0)...")
+r = subprocess.run(
+    BASE + ["lsjson", REMOTE, "--max-depth", "0"],
+    capture_output=True, text=True
+)
+items = json.loads(r.stdout)
+
+loose_files = [f for f in items if not f["IsDir"]]
+dirs        = [f["Name"] for f in items if f["IsDir"]]
+
+print(f"Found {len(loose_files)} loose files, {len(dirs)} folders\n")
+
+groups: dict[str, list] = {}
+for f in loose_files:
+    folder = classify(f["Name"], f["MimeType"])
+    groups.setdefault(folder, []).append(f["Name"])
+
+print("=== Move plan ===")
+for folder, names in sorted(groups.items()):
+    print(f"  {folder}/  ← {len(names)} files")
+
+print("\n=== Executing ===")
+created: set = set()
+ok = err = 0
+
+for folder, names in sorted(groups.items()):
+    if folder not in created:
+        run("rclone", "mkdir", f"{REMOTE}{folder}")
+        created.add(folder)
+    for name in names:
+        print(f"  → {folder}/{name[:65]}")
+        if run("rclone", "moveto", f"{REMOTE}{name}", f"{REMOTE}{folder}/{name}"):
+            ok += 1
+        else:
+            err += 1
+
+print(f"\n=== Done: {ok} moved, {err} errors ===")

test_local_api.py

@@ -0,0 +1,4 @@
+from telegram.ext import Application
+import os
+
+print("PTB local API config available")