Import documentation: CLAUDE.md
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
This commit is contained in:
arsam
@@ -0,0 +1,125 @@
|
|||||||
|
# CLAUDE.md
|
||||||
|
|
||||||
|
This file provides guidance to Claude Code (claude.ai/code) when working with code in this repository.
|
||||||
|
|
||||||
|
## What this is
|
||||||
|
|
||||||
|
A single-file Telegram bot (`bot.py`) that receives files (and URLs) and uploads them to Google Drive. Uses **Pyrogram** (MTProto, v2+) for direct Telegram communication and **rclone** (with `gdrive:` remote) for Drive uploads. All async, all in one file.
|
||||||
|
|
||||||
|
## Environment setup
|
||||||
|
|
||||||
|
```bash
|
||||||
|
source venv/bin/activate
|
||||||
|
pip install -r requirements.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
Required `.env` variables: `BOT_TOKEN`, `API_ID`, `API_HASH`. The rclone remote (`gdrive:`) must be authenticated separately via `rclone config`.
|
||||||
|
|
||||||
|
## Running the bot
|
||||||
|
|
||||||
|
**Production: runs as a systemd service** (`/etc/systemd/system/tg-gdrive-bot.service`, enabled on boot, `Restart=always` replaces the old `run_bot.sh` watchdog). Manage it with:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl status tg-gdrive-bot # state
|
||||||
|
systemctl restart tg-gdrive-bot # restart (or use the in-bot /restart)
|
||||||
|
systemctl stop tg-gdrive-bot # stop (do this before any manual run, or the session conflicts)
|
||||||
|
journalctl -u tg-gdrive-bot -f # follow logs (also appended to bot.log for the in-bot /logs)
|
||||||
|
```
|
||||||
|
|
||||||
|
The unit uses the venv interpreter (`venv/bin/python -u bot.py`) and a PATH that includes `/usr/local/bin` (for `deno`). stdout/stderr append to `bot.log` so the in-bot `/logs` command keeps working. `run_bot.sh` is the legacy nohup watchdog — superseded by systemd; don't run both.
|
||||||
|
|
||||||
|
Manual run (debugging only — stop the service first):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
systemctl stop tg-gdrive-bot
|
||||||
|
source venv/bin/activate
|
||||||
|
python bot.py
|
||||||
|
```
|
||||||
|
|
||||||
|
No real tests exist beyond the stub `test_local_api.py`.
|
||||||
|
|
||||||
|
## Bot commands
|
||||||
|
|
||||||
|
| Command | Auth required | Description |
|
||||||
|
|---------|---------------|-------------|
|
||||||
|
| `/start` | No | Upload instructions + system info |
|
||||||
|
| `/help` | No | Full command reference |
|
||||||
|
| `/list` | Yes | Last 10 files in Drive |
|
||||||
|
| `/listall` | Yes | All files (paginated) |
|
||||||
|
| `/search <q>` | Yes | Search files by name |
|
||||||
|
| `/share <name>` | Yes | Make file public + share link |
|
||||||
|
| `/status` | Yes | Uptime, RAM, CPU, disk |
|
||||||
|
| `/logs` | Yes | Last 50 log lines |
|
||||||
|
| `/adduser @user` | Yes | Grant full access |
|
||||||
|
| `/removeuser @user` | Yes | Revoke full access |
|
||||||
|
| `/setquota @user <size>` | Yes | Set daily upload quota |
|
||||||
|
| `/clearquota @user` | Yes | Remove quota |
|
||||||
|
| `/quotas` | Yes | Overview all quotas |
|
||||||
|
| `/myquota` | No | Check own quota |
|
||||||
|
| `/restart` | Yes | Graceful restart via `os.execv` |
|
||||||
|
|
||||||
|
## Architecture
|
||||||
|
|
||||||
|
All logic in `bot.py` (~1368 lines). Key components:
|
||||||
|
|
||||||
|
- **`handle_file`** (`@app.on_message(filters.media)`) — receives any media type (document, photo, video, audio, voice, sticker, video_note, animation). Routes to one of two upload strategies based on file size:
|
||||||
|
- **`_download_then_upload`** (< 2 GB) — downloads to temp dir via `client.download_media` with retry logic (3 attempts, 5/15/30s backoff) and live progress updates, then `rclone copy` to Drive. Cleans up temp files on success.
|
||||||
|
- **`_stream_to_drive`** (≥ 2 GB) — streams directly through RAM via `client.stream_media` → `rclone rcat` stdin, with retry logic and speed tracking. Zero disk usage.
|
||||||
|
- **`handle_url`** (`@app.on_message(filters.text)`) — detects URLs in text messages, downloads via `rclone copyurl` (RAM-only, no disk), saves to Drive. Pulse animation during transfer.
|
||||||
|
- **`classify_to_folder`** — routes files into Drive subfolders by extension/MIME (Videos, Music, Photos, Software, Archives, Models, Docs, etc.).
|
||||||
|
- **`resolve_filename`** — uses caption as filename override if provided; falls back to `file_name` attribute or generates one with appropriate extension.
|
||||||
|
- **`send_links`** — after upload, queries Drive via `rclone lsjson` for the file ID, then sends direct-download + wget links.
|
||||||
|
|
||||||
|
### Auth system
|
||||||
|
|
||||||
|
- **`AUTHORIZED_USERS`** set persisted in `authorized_users.json`. Anyone in this set has unlimited upload access.
|
||||||
|
- **Quota system** in `quotas.json` schema: `{"quotas": {"username": bytes}, "usage": {"username": {"YYYY-MM-DD": bytes}}}`. Non-auth users can upload if they have a quota with remaining bytes today. Quotas auto-reset at midnight UTC.
|
||||||
|
|
||||||
|
### Upload methods
|
||||||
|
|
||||||
|
| Size | Method | Disk usage | CLI tool |
|
||||||
|
|------|--------|------------|----------|
|
||||||
|
| < 2 GB | Download then upload | Temp file on disk | `rclone copy` |
|
||||||
|
| ≥ 2 GB | RAM stream | None | `rclone rcat` |
|
||||||
|
| URLs (any size) | RAM stream | None | `rclone copyurl` |
|
||||||
|
| Media-site URLs | yt-dlp to temp dir, then upload | Temp file on disk | `yt-dlp` + `rclone copy` |
|
||||||
|
|
||||||
|
The 2 GB threshold is `STREAM_THRESHOLD` at line 53.
|
||||||
|
|
||||||
|
### yt-dlp media downloads
|
||||||
|
|
||||||
|
URLs whose domain matches `YTDLP_DOMAINS` (YouTube, Instagram, TikTok, Twitter/X, Facebook, Reddit, Pinterest, Tumblr, LinkedIn, Aparat) or `NSFW_DOMAINS` (PornHub, XVideos, xHamster, YouPorn, RedTube, SpankBang) are routed by `_process_url` to `_ytdlp_download` instead of `rclone copyurl`. Files are downloaded with yt-dlp (venv package; ffmpeg merges streams) into a temp dir, then uploaded to `gdrive:MEDIA-DLS/<category>/` — category from `classify_to_folder` (Videos, Music, …), NSFW domains always go to `MEDIA-DLS/NSFW`. Adding the word `mp3` or `audio` after the link extracts audio (`-x --audio-format mp3`). Playlists are capped at 50 items; runtime capped at 3 h (livestreams). If `cookies.txt` exists in the bot dir it is passed via `--cookies` — required for Instagram and for YouTube (this server's IP is bot-flagged by YouTube). `curl_cffi` is installed for browser impersonation (needed by e.g. SpankBang). Keep yt-dlp updated (`venv/bin/pip install -U yt-dlp`) — extractors break when sites change.
|
||||||
|
|
||||||
|
### Inline file-action buttons
|
||||||
|
|
||||||
|
Every `send_links` message carries an inline keyboard (🌍 Share · ℹ️ Info · 📂 Move · 🗑 Delete with confirm step · 📥 Send to Telegram). Buttons reference an 8-hex-char token stored in `FILE_ACTIONS` (persisted to `file_actions.json`, capped at 500 entries, survives restarts). `handle_buttons` (`@app.on_callback_query()`) dispatches on the `action:token[:dest]` callback-data format and runs rclone (`link`, `moveto`, `deletefile`, `lsjson`). Authorized users can act on any file; quota users only on files they uploaded (`owner` field).
|
||||||
|
|
||||||
|
### Torrents (aria2c)
|
||||||
|
|
||||||
|
Magnet links (detected in `handle_url` via `MAGNET_RE`) and `.torrent` files (detected in `handle_file` by extension/mime) route to `process_torrent`. A minimal bencode decoder (`_bdecode` / `torrent_files`) reads the file list **without** sanitizing names (they must match what aria2 writes to disk and are used verbatim as Drive paths). Magnets first fetch metadata only (`_aria2_fetch_metadata`). Download strategy by total size: **< 8 GB** downloads the whole torrent then uploads each file; **≥ 8 GB** (`TORRENT_STREAM_THRESHOLD`) uses per-file mode (`--select-file=N`, download→upload→delete per file) to keep peak disk low. There is no RAM-stream path — BitTorrent is out-of-order, so torrents always hit disk. Work dir is `dl_tmp/` on `/dev/sda1` (NOT `/tmp`, which is RAM-backed tmpfs, only 3.9 GB). Pre-checks: free disk (`shutil.disk_usage`, needs biggest-file or total + 2 GB headroom) and quota. Files upload to `gdrive:Torrents/<name>/…` preserving structure. `aria2c` and the seeding-disabled flags live in `ARIA2_COMMON`; progress parsed by `ARIA2_PROGRESS_RE`.
|
||||||
|
|
||||||
|
### Telegram delivery (Drive → Telegram)
|
||||||
|
|
||||||
|
`/get <query>` and the 📥 button both call `deliver_to_telegram`: look up the file (`get_drive_entry` / `_find_drive_file`), refuse if over `TG_SEND_LIMIT` (~2 GB Telegram bot cap, offers `/share` instead), rclone-copy to `dl_tmp/` with progress (parsed from `--stats-one-line`), then `send_video`/`send_audio`/`send_document` by extension with an upload-progress callback. This is the only path that sends files *out* of Drive.
|
||||||
|
|
||||||
|
### yt-dlp auto-updater
|
||||||
|
|
||||||
|
`ytdlp_update_loop` (started in `_main()` alongside `app.start()`/`idle()`) runs `pip install -U yt-dlp` in the venv every 24 h and logs version changes. Subprocesses pick up the new version immediately.
|
||||||
|
|
||||||
|
Spotify links (`SPOTIFY_DOMAINS`) are routed to `_spotify_download`, which runs **spotdl** (Spotify metadata → audio downloaded from YouTube Music) and uploads results to `MEDIA-DLS/Music`. Because the audio comes from YouTube, this also requires `cookies.txt` on this server. Deno is installed at `/root/.config/spotdl/deno` (symlinked to `/usr/local/bin/deno`) — yt-dlp needs it for YouTube JS challenges. Both yt-dlp and spotdl uploads share `_upload_media_files`.
|
||||||
|
|
||||||
|
### Key tech stack
|
||||||
|
|
||||||
|
- **Pyrogram** v2+ — async MTProto client (`Client`), `filters.media` / `filters.command()` handlers
|
||||||
|
- **rclone** — Drive operations: `copy`, `rcat`, `copyurl`, `lsjson`, `link`
|
||||||
|
- **`asyncio.create_subprocess_exec`** — all rclone calls are non-blocking subprocesses
|
||||||
|
- **`psutil`** — RAM/CPU/disk stats for `/status`
|
||||||
|
- **`python-dotenv`** — loads `.env`
|
||||||
|
|
||||||
|
### Key constraints
|
||||||
|
|
||||||
|
- All handlers are `async def` — Pyrogram is fully async.
|
||||||
|
- rclone subprocesses use `asyncio.create_subprocess_exec` — do not replace with `subprocess.run`.
|
||||||
|
- Temp files only deleted on successful upload (manual retry on failure).
|
||||||
|
- Filenames cleaned via `sanitize_filename` (NFKC normalization, emoji removal, illegal char stripping).
|
||||||
|
- The bot can also handle files sent by non-auth users if they have a daily quota assigned.
|
||||||
Reference in New Issue
Block a user