Axon/CLAUDE.md

MAIN PROMPT: "Build me a private, self-hosted AI search API called "Axon" — a better alternative to Tavily. It should be a FastAPI backend that I can run on my Linux server.

## Stack
- Python 3.11+
- FastAPI
- trafilatura (content extraction)
- httpx (async HTTP)
- sentence-transformers with BAAI/bge-reranker-v2-m3 (reranking)
- redis (caching)
- pydantic v2 (schemas)
- uvicorn (server)

## Search Sources
Implement all three and merge results, deduplicating by URL:
1. Brave Search API (primary) — via BRAVE_API_KEY env var
2. DuckDuckGo — using the duckduckgo-search Python package, no key needed
3. Serper.dev (fallback/secondary) — via SERPER_API_KEY env var, only called if Brave returns fewer than 5 results

## Project Structure
axon/
├── main.py
├── config.py
├── models.py
├── routers/
│   ├── search.py
│   ├── extract.py
│   └── research.py
├── services/
│   ├── sources/
│   │   ├── brave.py
│   │   ├── ddg.py
│   │   └── serper.py
│   ├── scraper.py
│   ├── reranker.py
│   ├── synthesizer.py
│   └── cache.py
├── middleware/
│   └── auth.py
├── .env.example
├── requirements.txt
└── README.md

## Endpoints

### POST /search
Main search endpoint.
Request body:
{
  "query": "string",
  "max_results": 10,          // default 10, max 20
  "search_depth": "basic" | "advanced",   // basic = no scraping, advanced = full scrape + rerank
  "include_answer": false,    // if true, synthesize an answer via LLM
  "include_raw_content": false,
  "domains_whitelist": [],    // only return results from these domains
  "domains_blacklist": [],    // never return results from these domains
  "topic": "general" | "news",
  "time_range": "day" | "week" | "month" | "year" | null
}

Response:
{
  "query": "string",
  "answer": "string | null",
  "results": [
    {
      "title": "string",
      "url": "string",
      "content": "string",      // snippet or extracted text
      "score": 0.95,            // relevance score 0-1
      "published_date": "string | null",
      "favicon": "string | null"
    }
  ],
  "response_time": 1.23,
  "credits_used": 1
}

### POST /extract
Extract clean content from a URL or list of URLs.
Request body:
{
  "urls": ["https://example.com"],
  "include_images": false
}

Response:
{
  "results": [
    {
      "url": "string",
      "title": "string",
      "content": "string",
      "success": true,
      "error": null
    }
  ]
}

### POST /research
Deep research endpoint. Given a topic, auto-generate 3-5 sub-queries, run /search on each, deduplicate, rerank everything together, synthesize a final report.
Request body:
{
  "query": "string",
  "depth": "basic" | "deep",   // basic = 3 sub-queries, deep = 5 sub-queries
  "output_format": "markdown" | "json"
}

Response:
{
  "query": "string",
  "report": "string",          // markdown or JSON report
  "sources": [...],            // all sources used
  "sub_queries": [...],        // the generated sub-queries
  "response_time": 12.4
}

### GET /health
Returns server status, redis connection status, reranker model load status.

## Core Logic Details

### Search Flow (advanced depth)
1. Query all enabled sources concurrently with asyncio.gather
2. Merge and deduplicate results by URL
3. For each result URL, scrape full content using trafilatura (async, with 5s timeout per URL, max 10 concurrent scrapes using asyncio.Semaphore)
4. Chunk content to max 512 tokens per chunk
5. Run bge-reranker-v2-m3 to score all chunks against the original query
6. Take the top-scoring chunk per URL as that result's content
7. Sort results by score descending
8. If include_answer is true, send top 5 results as context to LLM via LITELLM_BASE_URL + LITELLM_API_KEY and get a synthesized answer

### Search Flow (basic depth)
1. Same as above steps 1-2 (multi-source fetch + dedup)
2. Skip scraping — use the snippet returned by the search APIs directly
3. Still run reranker on snippets
4. Return results

### Caching
- Use Redis to cache search results for 10 minutes (key = hash of query + params)
- Cache extracted content per URL for 1 hour
- If REDIS_URL is not set, fall back to an in-memory dict cache

### Auth
- API key auth via X-API-Key header
- Keys stored in API_KEYS env var as a comma-separated list
- If API_KEYS is not set, skip auth entirely (dev mode)

### LLM Synthesis
- Call LLM via LITELLM_BASE_URL + LITELLM_API_KEY (standard OpenAI-compatible format)
- Model set via SYNTHESIS_MODEL env var, default "claude-sonnet-4-6"
- System prompt: "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."

## Config (.env.example)
BRAVE_API_KEY=
SERPER_API_KEY=
REDIS_URL=redis://localhost:6379
API_KEYS=your-secret-key-here
LITELLM_BASE_URL=http://localhost:4000
LITELLM_API_KEY=
SYNTHESIS_MODEL=claude-sonnet-4-6
RERANKER_MODEL=BAAI/bge-reranker-v2-m3
PORT=8000
MAX_CONCURRENT_SCRAPES=10
SCRAPE_TIMEOUT=5

## Error Handling
- Every endpoint must return proper HTTP status codes
- Scraping failures for individual URLs should be silently skipped, not crash the request
- Source API failures (Brave down, etc.) should be caught and logged, fall through to remaining sources
- Return 503 only if ALL sources fail

## Requirements
- All I/O must be fully async
- The reranker model should be loaded once at startup into memory, not per-request
- Log all requests with query, source count, scrape count, rerank score range, and response time
- Add CORS middleware allowing all origins
- Write a clean README with setup instructions and example curl commands for all endpoints"

# Axon Search API — Complete Project Reference

## What This Is

**Axon** is a self-hosted, private AI search API — built as a better alternative to Tavily. It is a FastAPI backend meant to run on a Linux server. It aggregates results from multiple search engines, scrapes full page content, reranks with a local ML model, and optionally synthesizes an AI-written answer using any OpenAI-compatible LLM endpoint.

**The goal:** `POST /search` with a query → get back ranked, cleaned, optionally AI-answered results. No rate limits you don't control. No external service owning your queries.

---

## How It Is Being Built

The project is being implemented via **Subagent-Driven Development** (one subagent per task, reviewed after each). The implementation plan lives at:

```
docs/superpowers/plans/2026-06-18-axon-search-api.md
```

Progress ledger (survives context compression):
```
.git/sdd/progress.md
```

---

## Overall Architecture

```
Request
  │
  ▼
APIKeyMiddleware          ← checks X-API-Key header
  │
  ├── POST /search        ← routers/search.py
  │     │
  │     ├── get_cache()                  check Redis / in-memory
  │     ├── search_brave() ──┐
  │     ├── search_ddg()   ──┼── asyncio.gather (concurrent)
  │     └── search_serper() ─┘  (serper only if brave < 5 results)
  │           │
  │           ├── deduplicate by URL
  │           ├── filter whitelist/blacklist
  │           ├── scrape_urls() [if advanced depth]
  │           ├── rerank_results()   ← CrossEncoder scores all chunks
  │           ├── synthesize_answer() [if include_answer=True]
  │           └── store in cache, return
  │
  ├── POST /extract       ← routers/extract.py  [NOT BUILT YET]
  │     └── per-URL: httpx fetch → trafilatura extract → cache 1h
  │
  ├── POST /research      ← routers/research.py [NOT BUILT YET]
  │     ├── generate_sub_queries() via LLM
  │     ├── run /search for each sub-query
  │     ├── deduplicate all results
  │     └── synthesize_report()
  │
  └── GET /health         ← main.py             [NOT BUILT YET]
        └── returns redis status + reranker load status
```

---

## Current Build Status

### ✅ DONE — 10 of 14 Tasks

All of the following have been implemented, reviewed by a subagent, and fixed where needed.

---

#### Task 1 — Project Scaffold ✅
**Commits:** `556b771..176e389`

Files created:
- `requirements.txt` — all pip dependencies
- `.env.example` — all configurable env vars with defaults
- `.gitignore`
- `axon/__init__.py`, `axon/middleware/__init__.py`, `axon/routers/__init__.py`, `axon/services/__init__.py`, `axon/services/sources/__init__.py`
- `tests/__init__.py`
- `axon/config.py` — `Settings` class and `settings` singleton

`axon/config.py` — the full thing, verbatim:
```python
from typing import Optional
from pydantic_settings import BaseSettings

class Settings(BaseSettings):
    brave_api_key: Optional[str] = None
    serper_api_key: Optional[str] = None
    redis_url: Optional[str] = "redis://localhost:6379"
    api_keys: Optional[str] = None
    litellm_base_url: str = "http://localhost:4000"
    litellm_api_key: Optional[str] = None
    synthesis_model: str = "claude-sonnet-4-6"
    reranker_model: str = "BAAI/bge-reranker-v2-m3"
    port: int = 8000
    max_concurrent_scrapes: int = 10
    scrape_timeout: int = 5

    @property
    def api_key_list(self) -> list[str]:
        if not self.api_keys:
            return []
        return [k.strip() for k in self.api_keys.split(",") if k.strip()]

    model_config = {"env_file": ".env", "extra": "ignore"}

settings = Settings()
```

---

#### Task 2 — Pydantic Models ✅
**Commits:** `176e389..b8e6a51`  
**File:** `axon/models.py`

All 8 schemas. Key ones:

```python
class SearchRequest(BaseModel):
    query: str
    max_results: int = Field(default=10, ge=1, le=20)
    search_depth: Literal["basic", "advanced"] = "basic"
    include_answer: bool = False
    include_raw_content: bool = False
    domains_whitelist: list[str] = []
    domains_blacklist: list[str] = []
    topic: Literal["general", "news"] = "general"
    time_range: Optional[Literal["day", "week", "month", "year"]] = None

class SearchResult(BaseModel):
    title: str
    url: str
    content: str
    score: float = 0.5
    published_date: Optional[str] = None
    favicon: Optional[str] = None

class SearchResponse(BaseModel):
    query: str
    answer: Optional[str] = None
    results: list[SearchResult]
    response_time: float
    credits_used: int = 1

class ExtractRequest(BaseModel):
    urls: list[str]
    include_images: bool = False

class ExtractResult(BaseModel):
    url: str
    title: Optional[str] = None
    content: str = ""
    success: bool
    error: Optional[str] = None

class ExtractResponse(BaseModel):
    results: list[ExtractResult]

class ResearchRequest(BaseModel):
    query: str
    depth: Literal["basic", "deep"] = "basic"
    output_format: Literal["markdown", "json"] = "markdown"

class ResearchResponse(BaseModel):
    query: str
    report: str
    sources: list[SearchResult]
    sub_queries: list[str]
    response_time: float
```

---

#### Task 3 — Cache Service ✅
**Commits:** `b8e6a51..200085c`  
**File:** `axon/services/cache.py`

Two backends with identical async interface:
- `InMemoryCache` — dict with TTL timestamps. Used when Redis is unavailable or `REDIS_URL` unset.
- `RedisCache` — wraps `redis.asyncio` client. Uses `setex` for TTL.

Key functions:
```python
async def get_cache() -> InMemoryCache | RedisCache:
    # Singleton. Tries Redis on first call; falls back to in-memory on failure.

def make_cache_key(prefix: str, **kwargs) -> str:
    # SHA-256 of JSON-serialized kwargs. Returns "prefix:<hex>"
    # Example: make_cache_key("search", query="hello") → "search:abc123..."
```

TTLs used by callers (not enforced here):
- Search results: 600 seconds (10 minutes)
- Extracted content: 3600 seconds (1 hour)

---

#### Task 4 — Brave Search Source ✅
**Commits:** `200085c..5da0eb9`  
**File:** `axon/services/sources/brave.py`

```python
async def search_brave(
    query: str,
    count: int = 10,
    topic: str = "general",
    time_range: str | None = None,
) -> list[SearchResult]:
```

- Returns `[]` immediately if `settings.brave_api_key` is falsy
- Returns `[]` on any HTTP/network error (never raises)
- `topic="news"` → hits `/res/v1/news/search`; else `/res/v1/web/search`
- Freshness param map: `day→pd`, `week→pw`, `month→pm`, `year→py`
- Header: `X-Subscription-Token`
- Parses `description` field (falls back to `extra_snippets[0]`) for content
- Provides `favicon` from `meta_url.favicon`

---

#### Task 5 — DDG + Serper Sources ✅
**Commits:** `5da0eb9..5a13863`  
**Files:** `axon/services/sources/ddg.py`, `axon/services/sources/serper.py`

**DDG:**
```python
async def search_ddg(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
```
- Uses `duckduckgo-search` package via `asyncio.to_thread()` (blocking call)
- Text results: url = `item["href"]`, content = `item["body"]`
- News results: url = `item["url"]`, content = `item["body"]`
- Timelimit map: `day→d`, `week→w`, `month→m`, `year→y`
- No API key needed

**Serper:**
```python
async def search_serper(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
```
- Returns `[]` if `settings.serper_api_key` is falsy
- `topic="news"` → `https://google.serper.dev/news`; else `/search`
- TBS map: `day→qdr:d`, `week→qdr:w`, `month→qdr:m`, `year→qdr:y`
- Header: `X-API-KEY`
- Parses `organic` or `news` key in response
- `count` capped at 10 for Serper (their limit per call)

---

#### Task 6 — Scraper Service ✅ (+ SSRF Fix)
**Commits:** `5a13863..aaa7ffa`, then SSRF fix at `42216c4`  
**File:** `axon/services/scraper.py`

```python
async def scrape_url(url: str, timeout: int = 5) -> str | None:
    # Validates URL first (SSRF check), then fetches, then extracts with trafilatura
    # Returns None on any failure

async def scrape_urls(
    urls: list[str],
    max_concurrent: int = 10,
    timeout: int = 5,
) -> dict[str, str | None]:
    # Concurrent batch scrape with asyncio.Semaphore
    # Individual failures become None values in the dict
```

**SSRF Protection** (added mid-build after automated security review flagged it):
- `_is_safe_url(url)` runs before every fetch
- Checks scheme is `http` or `https` only
- Resolves hostname via `socket.getaddrinfo`, rejects any result that is:
  - Loopback (127.x.x.x, ::1)
  - Private (10.x, 172.16-31.x, 192.168.x)
  - Link-local (169.254.x.x — blocks AWS metadata at 169.254.169.254)
  - Multicast, reserved, or unspecified
- `follow_redirects=False` (redirects to internal IPs are a common SSRF bypass)
- trafilatura called via `asyncio.to_thread()` (CPU-bound extraction)

---

#### Task 7 — Reranker Service ✅
**Commits:** `aaa7ffa..acc7a3c`  
**File:** `axon/services/reranker.py`

```python
_model = None  # loaded at startup, stays in memory

def load_reranker() -> None:
    # Loads BAAI/bge-reranker-v2-m3 CrossEncoder into _model
    # ~1GB download on first run, cached by HuggingFace after that

def get_reranker() -> "CrossEncoder | None":
    return _model

def chunk_text(text: str, max_tokens: int = 512) -> list[str]:
    # Splits into chunks of ~384 words (512 tokens × 0.75 words/token)
    # Empty string → [""]

async def rerank_results(
    query: str,
    contents: list[tuple[str, str]],  # (url, text) pairs
) -> list[tuple[str, str, float]]:
    # For each URL: chunks text, scores all chunks vs query
    # Picks best chunk per URL (argmax)
    # Sigmoid-normalizes raw CrossEncoder logits to [0, 1]
    # Returns sorted list of (url, best_chunk, score) — descending
    # model.predict() runs via asyncio.to_thread() (blocking/CPU)
```

Score normalization: `sigmoid(x) = 1 / (1 + exp(-x))` — maps CrossEncoder logits to 0–1 range.

---

#### Task 8 — Synthesizer Service ✅
**Commits:** `acc7a3c..6d9a057`  
**File:** `axon/services/synthesizer.py`

All LLM calls go to `settings.litellm_base_url + "/chat/completions"` (standard OpenAI format). `Authorization: Bearer <key>` header is only set if `settings.litellm_api_key` is non-empty.

System prompt used for all synthesis:
> "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."

```python
async def synthesize_answer(query: str, results: list[SearchResult]) -> str:
    # Uses top 5 results as context
    # Returns "No search results available..." if results is empty
    # Returns "Answer synthesis failed: <error>" on LLM error

async def generate_sub_queries(query: str, count: int = 3) -> list[str]:
    # Asks LLM to output exactly `count` queries, one per line
    # On LLM error: returns [query, f"{query} overview", f"{query} recent developments"][:count]

async def synthesize_report(query, results, output_format="markdown") -> str:
    # Uses top 20 results as context
    # markdown format → instructs "## Summary, ## Key Findings, ## Conclusion"
    # json format → instructs keys: summary, key_findings, conclusions
    # Returns "Report synthesis failed: <error>" on LLM error
    # 60s timeout (longer than answer synthesis at 30s)
```

---

#### Task 9 — Auth Middleware ✅
**Commits:** `6d9a057..bc49aa1`  
**File:** `axon/middleware/auth.py`

```python
_EXEMPT = {"/health", "/docs", "/openapi.json", "/redoc"}

class APIKeyMiddleware(BaseHTTPMiddleware):
    # If settings.api_key_list is empty → pass everything through (dev mode)
    # If path in _EXEMPT → pass through regardless
    # Otherwise → require X-API-Key header matching one of the keys
    # Invalid/missing → 401 {"detail": "Invalid or missing API key"}
```

`api_key_list` is a `@property` on `Settings` that computes from `settings.api_keys` (a raw comma-separated string). When monkeypatching in tests, patch `settings.api_keys` (the underlying field), not `api_key_list` (the property).

---

#### Task 10 — Search Router ✅ (commit done, review pending)
**Commit:** `4193bba`  
**File:** `axon/routers/search.py`

Exports `deduplicate_results()` — used by the research router (Task 12).

Full search flow, in order:
1. Build cache key from all request params → check cache; return immediately if hit
2. `search_brave` + `search_ddg` via `asyncio.gather` (concurrent)
3. If `len(brave_results) < 5`: also call `search_serper`
4. Merge all: `brave + ddg + serper`
5. If all empty: raise `HTTPException(503)`
6. `deduplicate_results()` → keep first URL occurrence
7. `_filter()` → apply `domains_whitelist` / `domains_blacklist`
8. Slice to `max_results`
9. If `search_depth == "advanced"`: `scrape_urls()` → replace result.content with scraped text where available
10. `rerank_results(query, [(url, content), ...])` → updates scores
11. Sort by score descending
12. If not `include_raw_content`: truncate content to 500 chars
13. If `include_answer`: `synthesize_answer()`
14. Log: `query, source_count, scrape_count, [min_score, max_score], response_time`
15. Store response in cache (TTL=600), return

---

### ❌ NOT DONE — Tasks 11–14

These four files do not exist yet:

---

#### Task 11 — Extract Router ❌
**File to create:** `axon/routers/extract.py`  
**Endpoint:** `POST /extract`

What it needs to do:
- Accept `ExtractRequest(urls: list[str], include_images: bool)`
- For each URL: check cache (key=`"extract:<sha256 of url>"`, TTL=3600), fetch if miss
- Fetch via httpx, extract text with `trafilatura.extract()` via `asyncio.to_thread()`
- Also try `trafilatura.extract_metadata()` to get title
- Individual URL failures: `ExtractResult(success=False, error="...")`
- Return `ExtractResponse(results=[...])`
- All URLs processed concurrently via `asyncio.gather`
- Cache successful extractions for 1 hour

---

#### Task 12 — Research Router ❌
**File to create:** `axon/routers/research.py`  
**Endpoint:** `POST /research`

What it needs to do:
- Accept `ResearchRequest(query, depth, output_format)`
- `depth="basic"` → 3 sub-queries; `depth="deep"` → 5 sub-queries
- Call `generate_sub_queries(query, count)` from synthesizer
- Run `search()` from `routers/search.py` for each sub-query (with `search_depth="advanced"`, `include_raw_content=True`)
- Sub-query searches run concurrently via `asyncio.gather(..., return_exceptions=True)`
- Failed sub-query searches are skipped (logged, not crashed)
- Merge all results → `deduplicate_results()`
- Call `synthesize_report(query, all_results, output_format)`
- Return `ResearchResponse(query, report, sources, sub_queries, response_time)`

**Import note:** `from axon.routers.search import search, deduplicate_results` — calling the Python function directly, not making an HTTP request.

---

#### Task 13 — Main App + Health ❌
**File to create:** `axon/main.py`

What it needs to do:
```python
from contextlib import asynccontextmanager
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware

@asynccontextmanager
async def lifespan(app):
    load_reranker()   # loads BAAI/bge-reranker-v2-m3 once at startup
    yield

app = FastAPI(title="Axon Search API", version="1.0.0", lifespan=lifespan)

# CORS: allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]
# Auth: add APIKeyMiddleware
# Routers: include search.router, extract.router, research.router

@app.get("/health")
async def health():
    cache = await get_cache()
    return {
        "status": "ok",
        "redis": "connected" | "in-memory fallback",
        "reranker": "loaded" | "not loaded",
    }
```

To run the server:
```bash
uvicorn axon.main:app --host 0.0.0.0 --port 8000
```

---

#### Task 14 — README ❌
**File to create:** `README.md`

Must include: setup steps, env var table, curl examples for all 4 endpoints (search, extract, research, health), source fallback logic explanation, performance notes.

---

## File Tree — Current State

```
/srv/Axon/
├── .env.example
├── .gitignore
├── CLAUDE.md                        ← this file
├── requirements.txt
├── docs/
│   └── superpowers/plans/
│       └── 2026-06-18-axon-search-api.md
├── axon/
│   ├── __init__.py
│   ├── config.py                    ✅
│   ├── models.py                    ✅
│   ├── main.py                      ❌ MISSING
│   ├── middleware/
│   │   ├── __init__.py
│   │   └── auth.py                  ✅
│   ├── routers/
│   │   ├── __init__.py
│   │   ├── search.py                ✅
│   │   ├── extract.py               ❌ MISSING
│   │   └── research.py              ❌ MISSING
│   └── services/
│       ├── __init__.py
│       ├── cache.py                 ✅
│       ├── reranker.py              ✅
│       ├── scraper.py               ✅ (includes SSRF protection)
│       ├── synthesizer.py           ✅
│       └── sources/
│           ├── __init__.py
│           ├── brave.py             ✅
│           ├── ddg.py               ✅
│           └── serper.py            ✅
└── tests/
    ├── __init__.py
    ├── test_config.py               ✅  3 tests
    ├── test_models.py               ✅  6 tests
    ├── test_cache.py                ✅  7 tests
    ├── test_sources.py              ✅ 10 tests (brave + ddg + serper)
    ├── test_scraper.py              ✅  4 tests
    ├── test_reranker.py             ✅  7 tests
    ├── test_synthesizer.py          ✅  4 tests
    ├── test_auth.py                 ✅  5 tests
    ├── test_search_router.py        ✅  4 tests
    ├── test_extract_router.py       ❌ MISSING
    └── test_research_router.py      ❌ MISSING (covered in test_search_router.py per plan)
```

**Total tests passing right now: 50**

---

## Environment & Running Tests

```bash
# The venv is at /srv/Axon/venv (Python 3.14)
source /srv/Axon/venv/bin/activate

# Run all tests
python -m pytest tests/ -v

# Run a specific file
python -m pytest tests/test_search_router.py -v

# pytest-asyncio version: 1.4.0 (strict mode)
# ALL async tests must be decorated with @pytest.mark.asyncio
```

Known warnings (not errors):
- Starlette currently emits a `TestClient` deprecation warning about its `httpx` integration.

---

## Environment Variables (from .env.example)

| Variable | Required | Default | Purpose |
|---|---|---|---|
| `BRAVE_API_KEY` | No | — | Enables Brave search source |
| `SERPER_API_KEY` | No | — | Enables Serper fallback source |
| `REDIS_URL` | No | `redis://localhost:6379` | Cache backend; omit for in-memory |
| `API_KEYS` | No | — | Comma-separated auth keys; unset = no auth |
| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible LLM endpoint |
| `LITELLM_API_KEY` | No | — | Bearer token for LLM endpoint |
| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model used for answer/report synthesis |
| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | HuggingFace CrossEncoder model |
| `PORT` | No | `8000` | uvicorn listen port |
| `MAX_CONCURRENT_SCRAPES` | No | `10` | asyncio.Semaphore limit for scraping |
| `SCRAPE_TIMEOUT` | No | `5` | Per-URL timeout in seconds |

---

## Coding Rules (enforced across all tasks)

1. **Python 3.11+ types only** — `str | None` not `Optional[str]`, `list[T]` not `List[T]`, `dict[K,V]` not `Dict[K,V]`. No imports from `typing` for these.
2. **Pydantic v2** — `model_dump()` and `model_dump_json()` only. Never `.dict()` or `.json()`.
3. **All I/O is async** — `httpx.AsyncClient`, `redis.asyncio`, `asyncio.to_thread()` for blocking calls.
4. **Never raise from service functions** — sources, scraper, synthesizer all catch all exceptions and return empty/None/error strings.
5. **Tests use `@pytest.mark.asyncio`** — no anyio markers.
6. **No `Optional` imports** — this kept appearing in subagent output and was fixed each time.

---

## Gotchas & Lessons Learned During Build

- **`@pytest.mark.anyio` vs `asyncio`** — subagents kept using `anyio`. Every test file was checked for this. Always `@pytest.mark.asyncio`.
- **`Optional[str]` vs `str | None`** — the plan's own code examples had `Optional` in them (for illustration). Subagents copied it. Had to fix in Tasks 3, 6.
- **`importlib.reload` in tests** — one subagent used it to reset module state for monkeypatching. Fragile and banned. Replaced with direct `monkeypatch.setattr()`.
- **SSRF** — automated security review flagged `scraper.py` after Task 6's commit. Fixed in Task 7's commit window by adding `_is_safe_url()` with full IP blocklist.
- **Review of auth tests** — reviewer incorrectly flagged monkeypatching `settings.api_keys` as wrong (said it should patch `api_key_list`). But `api_key_list` is a `@property` — you can't patch it directly. Patching the underlying field is correct. Confirmed all 5 tests pass.
- **Venv path** — the project uses `/srv/Axon/venv`. pytest-asyncio there is version 1.4.0. Must `source venv/bin/activate` before running any Python commands.

---

## Current Final State

Tasks 11-14 are now complete:
1. `axon/routers/extract.py` implements `POST /extract` with per-URL caching, trafilatura extraction, metadata title extraction, concurrent processing, and SSRF protection via the scraper URL safety check.
2. `axon/routers/research.py` implements `POST /research` by generating sub-queries, running advanced searches concurrently, skipping failed sub-searches, deduplicating sources, and synthesizing a final report.
3. `axon/main.py` creates the runnable FastAPI app with CORS, API key middleware, all routers, lifespan reranker loading, and `GET /health`.
4. `README.md` documents setup, configuration, endpoint curl examples, source fallback behavior, caching, and performance notes.

Additional cleanup:
- `axon/config.py` and `axon/models.py` now use `str | None` style types instead of `Optional[...]`.
- `axon/services/cache.py` now uses timezone-aware UTC datetimes, removing the Python 3.14 `datetime.utcnow()` warning.
- `axon/services/sources/ddg.py` now prefers the current `ddgs` package and falls back to `duckduckgo_search`; `ddgs>=9.14.0` is in `requirements.txt`.
- `axon/services/reranker.py` serializes model prediction and caps reranking to 4 chunks per document. This prevents `/research` from generating unbounded cross-encoder work on huge pages.
- `axon/routers/research.py` runs sub-searches sequentially with 5 results per sub-query. This avoids OOM on the current server while preserving deep multi-query research behavior.
- `axon/middleware/rate_limit.py` adds per-key/IP rate limiting.
- `axon/middleware/request_context.py` adds `X-Request-ID` and records lightweight metrics.
- `axon/routers/crawl.py` and `axon/routers/map.py` add Tavily-style bounded same-site crawl/map endpoints.
- `axon/services/scraper.py` now has reusable safe fetch, extraction, URL normalization, and same-site link discovery helpers with retries.
- `axon/services/index.py` adds a private SQLite FTS index; `POST /ingest`, `POST /index/search`, and normal `/search` can use private indexed content.
- `axon/services/jobs.py` and `axon/routers/jobs.py` add in-process background jobs for research/crawl/map.
- `axon/routers/tavily.py` adds `/tavily/search`, `/tavily/extract`, `/tavily/crawl`, and `/tavily/map` compatibility routes.
- `axon/routers/validate.py` and citation helpers add basic citation coverage validation.
- Browser extraction fallback is optional via `ENABLE_BROWSER_FALLBACK=true` if Playwright is installed.

Systemd deployment completed:
- Unit file: `/etc/systemd/system/axon.service`
- Enabled: yes
- Runtime env file: `/srv/Axon/.env` (ignored by git, mode `600`)
- LiteLLM model configured: `AR/openai/gpt-oss-120b:free`
- Service command: `/srv/Axon/venv/bin/uvicorn axon.main:app --host 0.0.0.0 --port 8000`

Live endpoint verification completed through systemd:
- `GET /health` -> 200, Redis connected, reranker loaded
- `GET /openapi.json` -> 200, paths include `/extract`, `/health`, `/metrics`, `/crawl`, `/map`, `/research`, `/search`
- `POST /extract` with `https://example.com` -> 200
- `POST /search` basic with answer synthesis -> 200 using LiteLLM
- `POST /search` advanced -> 200, completed in about 28 seconds
- `POST /research` basic markdown -> 200, 15 sources, completed in about 163 seconds

Operational note:
- The BGE reranker is memory-heavy. Before reranker chunk capping, `/research` OOM-killed the service at roughly 4.2 GB. After capping and sequential research searches, the tested `/research` run peaked around 2 GB and completed successfully.

The git branch is `master`. The project is runnable with:

```bash
source /srv/Axon/venv/bin/activate
uvicorn axon.main:app --host 0.0.0.0 --port 8000
```

To run a full test suite health check right now:
```bash
source /srv/Axon/venv/bin/activate && python -m pytest tests/ -v
```
Expected: **61 passed**.