Import documentation: README.md

Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.

CLAUDE.md

@@ -0,0 +1,816 @@
+MAIN PROMPT: "Build me a private, self-hosted AI search API called "Axon" — a better alternative to Tavily. It should be a FastAPI backend that I can run on my Linux server.
+
+## Stack
+- Python 3.11+
+- FastAPI
+- trafilatura (content extraction)
+- httpx (async HTTP)
+- sentence-transformers with BAAI/bge-reranker-v2-m3 (reranking)
+- redis (caching)
+- pydantic v2 (schemas)
+- uvicorn (server)
+
+## Search Sources
+Implement all three and merge results, deduplicating by URL:
+1. Brave Search API (primary) — via BRAVE_API_KEY env var
+2. DuckDuckGo — using the duckduckgo-search Python package, no key needed
+3. Serper.dev (fallback/secondary) — via SERPER_API_KEY env var, only called if Brave returns fewer than 5 results
+
+## Project Structure
+axon/
+├── main.py
+├── config.py
+├── models.py
+├── routers/
+│   ├── search.py
+│   ├── extract.py
+│   └── research.py
+├── services/
+│   ├── sources/
+│   │   ├── brave.py
+│   │   ├── ddg.py
+│   │   └── serper.py
+│   ├── scraper.py
+│   ├── reranker.py
+│   ├── synthesizer.py
+│   └── cache.py
+├── middleware/
+│   └── auth.py
+├── .env.example
+├── requirements.txt
+└── README.md
+
+## Endpoints
+
+### POST /search
+Main search endpoint.
+Request body:
+{
+  "query": "string",
+  "max_results": 10,          // default 10, max 20
+  "search_depth": "basic" | "advanced",   // basic = no scraping, advanced = full scrape + rerank
+  "include_answer": false,    // if true, synthesize an answer via LLM
+  "include_raw_content": false,
+  "domains_whitelist": [],    // only return results from these domains
+  "domains_blacklist": [],    // never return results from these domains
+  "topic": "general" | "news",
+  "time_range": "day" | "week" | "month" | "year" | null
+}
+
+Response:
+{
+  "query": "string",
+  "answer": "string | null",
+  "results": [
+    {
+      "title": "string",
+      "url": "string",
+      "content": "string",      // snippet or extracted text
+      "score": 0.95,            // relevance score 0-1
+      "published_date": "string | null",
+      "favicon": "string | null"
+    }
+  ],
+  "response_time": 1.23,
+  "credits_used": 1
+}
+
+### POST /extract
+Extract clean content from a URL or list of URLs.
+Request body:
+{
+  "urls": ["https://example.com"],
+  "include_images": false
+}
+
+Response:
+{
+  "results": [
+    {
+      "url": "string",
+      "title": "string",
+      "content": "string",
+      "success": true,
+      "error": null
+    }
+  ]
+}
+
+### POST /research
+Deep research endpoint. Given a topic, auto-generate 3-5 sub-queries, run /search on each, deduplicate, rerank everything together, synthesize a final report.
+Request body:
+{
+  "query": "string",
+  "depth": "basic" | "deep",   // basic = 3 sub-queries, deep = 5 sub-queries
+  "output_format": "markdown" | "json"
+}
+
+Response:
+{
+  "query": "string",
+  "report": "string",          // markdown or JSON report
+  "sources": [...],            // all sources used
+  "sub_queries": [...],        // the generated sub-queries
+  "response_time": 12.4
+}
+
+### GET /health
+Returns server status, redis connection status, reranker model load status.
+
+## Core Logic Details
+
+### Search Flow (advanced depth)
+1. Query all enabled sources concurrently with asyncio.gather
+2. Merge and deduplicate results by URL
+3. For each result URL, scrape full content using trafilatura (async, with 5s timeout per URL, max 10 concurrent scrapes using asyncio.Semaphore)
+4. Chunk content to max 512 tokens per chunk
+5. Run bge-reranker-v2-m3 to score all chunks against the original query
+6. Take the top-scoring chunk per URL as that result's content
+7. Sort results by score descending
+8. If include_answer is true, send top 5 results as context to LLM via LITELLM_BASE_URL + LITELLM_API_KEY and get a synthesized answer
+
+### Search Flow (basic depth)
+1. Same as above steps 1-2 (multi-source fetch + dedup)
+2. Skip scraping — use the snippet returned by the search APIs directly
+3. Still run reranker on snippets
+4. Return results
+
+### Caching
+- Use Redis to cache search results for 10 minutes (key = hash of query + params)
+- Cache extracted content per URL for 1 hour
+- If REDIS_URL is not set, fall back to an in-memory dict cache
+
+### Auth
+- API key auth via X-API-Key header
+- Keys stored in API_KEYS env var as a comma-separated list
+- If API_KEYS is not set, skip auth entirely (dev mode)
+
+### LLM Synthesis
+- Call LLM via LITELLM_BASE_URL + LITELLM_API_KEY (standard OpenAI-compatible format)
+- Model set via SYNTHESIS_MODEL env var, default "claude-sonnet-4-6"
+- System prompt: "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."
+
+## Config (.env.example)
+BRAVE_API_KEY=
+SERPER_API_KEY=
+REDIS_URL=redis://localhost:6379
+API_KEYS=your-secret-key-here
+LITELLM_BASE_URL=http://localhost:4000
+LITELLM_API_KEY=
+SYNTHESIS_MODEL=claude-sonnet-4-6
+RERANKER_MODEL=BAAI/bge-reranker-v2-m3
+PORT=8000
+MAX_CONCURRENT_SCRAPES=10
+SCRAPE_TIMEOUT=5
+
+## Error Handling
+- Every endpoint must return proper HTTP status codes
+- Scraping failures for individual URLs should be silently skipped, not crash the request
+- Source API failures (Brave down, etc.) should be caught and logged, fall through to remaining sources
+- Return 503 only if ALL sources fail
+
+## Requirements
+- All I/O must be fully async
+- The reranker model should be loaded once at startup into memory, not per-request
+- Log all requests with query, source count, scrape count, rerank score range, and response time
+- Add CORS middleware allowing all origins
+- Write a clean README with setup instructions and example curl commands for all endpoints"
+
+# Axon Search API — Complete Project Reference
+
+## What This Is
+
+**Axon** is a self-hosted, private AI search API — built as a better alternative to Tavily. It is a FastAPI backend meant to run on a Linux server. It aggregates results from multiple search engines, scrapes full page content, reranks with a local ML model, and optionally synthesizes an AI-written answer using any OpenAI-compatible LLM endpoint.
+
+**The goal:** `POST /search` with a query → get back ranked, cleaned, optionally AI-answered results. No rate limits you don't control. No external service owning your queries.
+
+---
+
+## How It Is Being Built
+
+The project is being implemented via **Subagent-Driven Development** (one subagent per task, reviewed after each). The implementation plan lives at:
+
+```
+docs/superpowers/plans/2026-06-18-axon-search-api.md
+```
+
+Progress ledger (survives context compression):
+```
+.git/sdd/progress.md
+```
+
+---
+
+## Overall Architecture
+
+```
+Request
+  │
+  ▼
+APIKeyMiddleware          ← checks X-API-Key header
+  │
+  ├── POST /search        ← routers/search.py
+  │     │
+  │     ├── get_cache()                  check Redis / in-memory
+  │     ├── search_brave() ──┐
+  │     ├── search_ddg()   ──┼── asyncio.gather (concurrent)
+  │     └── search_serper() ─┘  (serper only if brave < 5 results)
+  │           │
+  │           ├── deduplicate by URL
+  │           ├── filter whitelist/blacklist
+  │           ├── scrape_urls() [if advanced depth]
+  │           ├── rerank_results()   ← CrossEncoder scores all chunks
+  │           ├── synthesize_answer() [if include_answer=True]
+  │           └── store in cache, return
+  │
+  ├── POST /extract       ← routers/extract.py  [NOT BUILT YET]
+  │     └── per-URL: httpx fetch → trafilatura extract → cache 1h
+  │
+  ├── POST /research      ← routers/research.py [NOT BUILT YET]
+  │     ├── generate_sub_queries() via LLM
+  │     ├── run /search for each sub-query
+  │     ├── deduplicate all results
+  │     └── synthesize_report()
+  │
+  └── GET /health         ← main.py             [NOT BUILT YET]
+        └── returns redis status + reranker load status
+```
+
+---
+
+## Current Build Status
+
+### ✅ DONE — 10 of 14 Tasks
+
+All of the following have been implemented, reviewed by a subagent, and fixed where needed.
+
+---
+
+#### Task 1 — Project Scaffold ✅
+**Commits:** `556b771..176e389`
+
+Files created:
+- `requirements.txt` — all pip dependencies
+- `.env.example` — all configurable env vars with defaults
+- `.gitignore`
+- `axon/__init__.py`, `axon/middleware/__init__.py`, `axon/routers/__init__.py`, `axon/services/__init__.py`, `axon/services/sources/__init__.py`
+- `tests/__init__.py`
+- `axon/config.py` — `Settings` class and `settings` singleton
+
+`axon/config.py` — the full thing, verbatim:
+```python
+from typing import Optional
+from pydantic_settings import BaseSettings
+
+class Settings(BaseSettings):
+    brave_api_key: Optional[str] = None
+    serper_api_key: Optional[str] = None
+    redis_url: Optional[str] = "redis://localhost:6379"
+    api_keys: Optional[str] = None
+    litellm_base_url: str = "http://localhost:4000"
+    litellm_api_key: Optional[str] = None
+    synthesis_model: str = "claude-sonnet-4-6"
+    reranker_model: str = "BAAI/bge-reranker-v2-m3"
+    port: int = 8000
+    max_concurrent_scrapes: int = 10
+    scrape_timeout: int = 5
+
+    @property
+    def api_key_list(self) -> list[str]:
+        if not self.api_keys:
+            return []
+        return [k.strip() for k in self.api_keys.split(",") if k.strip()]
+
+    model_config = {"env_file": ".env", "extra": "ignore"}
+
+settings = Settings()
+```
+
+---
+
+#### Task 2 — Pydantic Models ✅
+**Commits:** `176e389..b8e6a51`  
+**File:** `axon/models.py`
+
+All 8 schemas. Key ones:
+
+```python
+class SearchRequest(BaseModel):
+    query: str
+    max_results: int = Field(default=10, ge=1, le=20)
+    search_depth: Literal["basic", "advanced"] = "basic"
+    include_answer: bool = False
+    include_raw_content: bool = False
+    domains_whitelist: list[str] = []
+    domains_blacklist: list[str] = []
+    topic: Literal["general", "news"] = "general"
+    time_range: Optional[Literal["day", "week", "month", "year"]] = None
+
+class SearchResult(BaseModel):
+    title: str
+    url: str
+    content: str
+    score: float = 0.5
+    published_date: Optional[str] = None
+    favicon: Optional[str] = None
+
+class SearchResponse(BaseModel):
+    query: str
+    answer: Optional[str] = None
+    results: list[SearchResult]
+    response_time: float
+    credits_used: int = 1
+
+class ExtractRequest(BaseModel):
+    urls: list[str]
+    include_images: bool = False
+
+class ExtractResult(BaseModel):
+    url: str
+    title: Optional[str] = None
+    content: str = ""
+    success: bool
+    error: Optional[str] = None
+
+class ExtractResponse(BaseModel):
+    results: list[ExtractResult]
+
+class ResearchRequest(BaseModel):
+    query: str
+    depth: Literal["basic", "deep"] = "basic"
+    output_format: Literal["markdown", "json"] = "markdown"
+
+class ResearchResponse(BaseModel):
+    query: str
+    report: str
+    sources: list[SearchResult]
+    sub_queries: list[str]
+    response_time: float
+```
+
+---
+
+#### Task 3 — Cache Service ✅
+**Commits:** `b8e6a51..200085c`  
+**File:** `axon/services/cache.py`
+
+Two backends with identical async interface:
+- `InMemoryCache` — dict with TTL timestamps. Used when Redis is unavailable or `REDIS_URL` unset.
+- `RedisCache` — wraps `redis.asyncio` client. Uses `setex` for TTL.
+
+Key functions:
+```python
+async def get_cache() -> InMemoryCache | RedisCache:
+    # Singleton. Tries Redis on first call; falls back to in-memory on failure.
+
+def make_cache_key(prefix: str, **kwargs) -> str:
+    # SHA-256 of JSON-serialized kwargs. Returns "prefix:<hex>"
+    # Example: make_cache_key("search", query="hello") → "search:abc123..."
+```
+
+TTLs used by callers (not enforced here):
+- Search results: 600 seconds (10 minutes)
+- Extracted content: 3600 seconds (1 hour)
+
+---
+
+#### Task 4 — Brave Search Source ✅
+**Commits:** `200085c..5da0eb9`  
+**File:** `axon/services/sources/brave.py`
+
+```python
+async def search_brave(
+    query: str,
+    count: int = 10,
+    topic: str = "general",
+    time_range: str | None = None,
+) -> list[SearchResult]:
+```
+
+- Returns `[]` immediately if `settings.brave_api_key` is falsy
+- Returns `[]` on any HTTP/network error (never raises)
+- `topic="news"` → hits `/res/v1/news/search`; else `/res/v1/web/search`
+- Freshness param map: `day→pd`, `week→pw`, `month→pm`, `year→py`
+- Header: `X-Subscription-Token`
+- Parses `description` field (falls back to `extra_snippets[0]`) for content
+- Provides `favicon` from `meta_url.favicon`
+
+---
+
+#### Task 5 — DDG + Serper Sources ✅
+**Commits:** `5da0eb9..5a13863`  
+**Files:** `axon/services/sources/ddg.py`, `axon/services/sources/serper.py`
+
+**DDG:**
+```python
+async def search_ddg(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
+```
+- Uses `duckduckgo-search` package via `asyncio.to_thread()` (blocking call)
+- Text results: url = `item["href"]`, content = `item["body"]`
+- News results: url = `item["url"]`, content = `item["body"]`
+- Timelimit map: `day→d`, `week→w`, `month→m`, `year→y`
+- No API key needed
+
+**Serper:**
+```python
+async def search_serper(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
+```
+- Returns `[]` if `settings.serper_api_key` is falsy
+- `topic="news"` → `https://google.serper.dev/news`; else `/search`
+- TBS map: `day→qdr:d`, `week→qdr:w`, `month→qdr:m`, `year→qdr:y`
+- Header: `X-API-KEY`
+- Parses `organic` or `news` key in response
+- `count` capped at 10 for Serper (their limit per call)
+
+---
+
+#### Task 6 — Scraper Service ✅ (+ SSRF Fix)
+**Commits:** `5a13863..aaa7ffa`, then SSRF fix at `42216c4`  
+**File:** `axon/services/scraper.py`
+
+```python
+async def scrape_url(url: str, timeout: int = 5) -> str | None:
+    # Validates URL first (SSRF check), then fetches, then extracts with trafilatura
+    # Returns None on any failure
+
+async def scrape_urls(
+    urls: list[str],
+    max_concurrent: int = 10,
+    timeout: int = 5,
+) -> dict[str, str | None]:
+    # Concurrent batch scrape with asyncio.Semaphore
+    # Individual failures become None values in the dict
+```
+
+**SSRF Protection** (added mid-build after automated security review flagged it):
+- `_is_safe_url(url)` runs before every fetch
+- Checks scheme is `http` or `https` only
+- Resolves hostname via `socket.getaddrinfo`, rejects any result that is:
+  - Loopback (127.x.x.x, ::1)
+  - Private (10.x, 172.16-31.x, 192.168.x)
+  - Link-local (169.254.x.x — blocks AWS metadata at 169.254.169.254)
+  - Multicast, reserved, or unspecified
+- `follow_redirects=False` (redirects to internal IPs are a common SSRF bypass)
+- trafilatura called via `asyncio.to_thread()` (CPU-bound extraction)
+
+---
+
+#### Task 7 — Reranker Service ✅
+**Commits:** `aaa7ffa..acc7a3c`  
+**File:** `axon/services/reranker.py`
+
+```python
+_model = None  # loaded at startup, stays in memory
+
+def load_reranker() -> None:
+    # Loads BAAI/bge-reranker-v2-m3 CrossEncoder into _model
+    # ~1GB download on first run, cached by HuggingFace after that
+
+def get_reranker() -> "CrossEncoder | None":
+    return _model
+
+def chunk_text(text: str, max_tokens: int = 512) -> list[str]:
+    # Splits into chunks of ~384 words (512 tokens × 0.75 words/token)
+    # Empty string → [""]
+
+async def rerank_results(
+    query: str,
+    contents: list[tuple[str, str]],  # (url, text) pairs
+) -> list[tuple[str, str, float]]:
+    # For each URL: chunks text, scores all chunks vs query
+    # Picks best chunk per URL (argmax)
+    # Sigmoid-normalizes raw CrossEncoder logits to [0, 1]
+    # Returns sorted list of (url, best_chunk, score) — descending
+    # model.predict() runs via asyncio.to_thread() (blocking/CPU)
+```
+
+Score normalization: `sigmoid(x) = 1 / (1 + exp(-x))` — maps CrossEncoder logits to 0–1 range.
+
+---
+
+#### Task 8 — Synthesizer Service ✅
+**Commits:** `acc7a3c..6d9a057`  
+**File:** `axon/services/synthesizer.py`
+
+All LLM calls go to `settings.litellm_base_url + "/chat/completions"` (standard OpenAI format). `Authorization: Bearer <key>` header is only set if `settings.litellm_api_key` is non-empty.
+
+System prompt used for all synthesis:
+> "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."
+
+```python
+async def synthesize_answer(query: str, results: list[SearchResult]) -> str:
+    # Uses top 5 results as context
+    # Returns "No search results available..." if results is empty
+    # Returns "Answer synthesis failed: <error>" on LLM error
+
+async def generate_sub_queries(query: str, count: int = 3) -> list[str]:
+    # Asks LLM to output exactly `count` queries, one per line
+    # On LLM error: returns [query, f"{query} overview", f"{query} recent developments"][:count]
+
+async def synthesize_report(query, results, output_format="markdown") -> str:
+    # Uses top 20 results as context
+    # markdown format → instructs "## Summary, ## Key Findings, ## Conclusion"
+    # json format → instructs keys: summary, key_findings, conclusions
+    # Returns "Report synthesis failed: <error>" on LLM error
+    # 60s timeout (longer than answer synthesis at 30s)
+```
+
+---
+
+#### Task 9 — Auth Middleware ✅
+**Commits:** `6d9a057..bc49aa1`  
+**File:** `axon/middleware/auth.py`
+
+```python
+_EXEMPT = {"/health", "/docs", "/openapi.json", "/redoc"}
+
+class APIKeyMiddleware(BaseHTTPMiddleware):
+    # If settings.api_key_list is empty → pass everything through (dev mode)
+    # If path in _EXEMPT → pass through regardless
+    # Otherwise → require X-API-Key header matching one of the keys
+    # Invalid/missing → 401 {"detail": "Invalid or missing API key"}
+```
+
+`api_key_list` is a `@property` on `Settings` that computes from `settings.api_keys` (a raw comma-separated string). When monkeypatching in tests, patch `settings.api_keys` (the underlying field), not `api_key_list` (the property).
+
+---
+
+#### Task 10 — Search Router ✅ (commit done, review pending)
+**Commit:** `4193bba`  
+**File:** `axon/routers/search.py`
+
+Exports `deduplicate_results()` — used by the research router (Task 12).
+
+Full search flow, in order:
+1. Build cache key from all request params → check cache; return immediately if hit
+2. `search_brave` + `search_ddg` via `asyncio.gather` (concurrent)
+3. If `len(brave_results) < 5`: also call `search_serper`
+4. Merge all: `brave + ddg + serper`
+5. If all empty: raise `HTTPException(503)`
+6. `deduplicate_results()` → keep first URL occurrence
+7. `_filter()` → apply `domains_whitelist` / `domains_blacklist`
+8. Slice to `max_results`
+9. If `search_depth == "advanced"`: `scrape_urls()` → replace result.content with scraped text where available
+10. `rerank_results(query, [(url, content), ...])` → updates scores
+11. Sort by score descending
+12. If not `include_raw_content`: truncate content to 500 chars
+13. If `include_answer`: `synthesize_answer()`
+14. Log: `query, source_count, scrape_count, [min_score, max_score], response_time`
+15. Store response in cache (TTL=600), return
+
+---
+
+### ❌ NOT DONE — Tasks 11–14
+
+These four files do not exist yet:
+
+---
+
+#### Task 11 — Extract Router ❌
+**File to create:** `axon/routers/extract.py`  
+**Endpoint:** `POST /extract`
+
+What it needs to do:
+- Accept `ExtractRequest(urls: list[str], include_images: bool)`
+- For each URL: check cache (key=`"extract:<sha256 of url>"`, TTL=3600), fetch if miss
+- Fetch via httpx, extract text with `trafilatura.extract()` via `asyncio.to_thread()`
+- Also try `trafilatura.extract_metadata()` to get title
+- Individual URL failures: `ExtractResult(success=False, error="...")`
+- Return `ExtractResponse(results=[...])`
+- All URLs processed concurrently via `asyncio.gather`
+- Cache successful extractions for 1 hour
+
+---
+
+#### Task 12 — Research Router ❌
+**File to create:** `axon/routers/research.py`  
+**Endpoint:** `POST /research`
+
+What it needs to do:
+- Accept `ResearchRequest(query, depth, output_format)`
+- `depth="basic"` → 3 sub-queries; `depth="deep"` → 5 sub-queries
+- Call `generate_sub_queries(query, count)` from synthesizer
+- Run `search()` from `routers/search.py` for each sub-query (with `search_depth="advanced"`, `include_raw_content=True`)
+- Sub-query searches run concurrently via `asyncio.gather(..., return_exceptions=True)`
+- Failed sub-query searches are skipped (logged, not crashed)
+- Merge all results → `deduplicate_results()`
+- Call `synthesize_report(query, all_results, output_format)`
+- Return `ResearchResponse(query, report, sources, sub_queries, response_time)`
+
+**Import note:** `from axon.routers.search import search, deduplicate_results` — calling the Python function directly, not making an HTTP request.
+
+---
+
+#### Task 13 — Main App + Health ❌
+**File to create:** `axon/main.py`
+
+What it needs to do:
+```python
+from contextlib import asynccontextmanager
+from fastapi import FastAPI
+from fastapi.middleware.cors import CORSMiddleware
+
+@asynccontextmanager
+async def lifespan(app):
+    load_reranker()   # loads BAAI/bge-reranker-v2-m3 once at startup
+    yield
+
+app = FastAPI(title="Axon Search API", version="1.0.0", lifespan=lifespan)
+
+# CORS: allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]
+# Auth: add APIKeyMiddleware
+# Routers: include search.router, extract.router, research.router
+
+@app.get("/health")
+async def health():
+    cache = await get_cache()
+    return {
+        "status": "ok",
+        "redis": "connected" | "in-memory fallback",
+        "reranker": "loaded" | "not loaded",
+    }
+```
+
+To run the server:
+```bash
+uvicorn axon.main:app --host 0.0.0.0 --port 8000
+```
+
+---
+
+#### Task 14 — README ❌
+**File to create:** `README.md`
+
+Must include: setup steps, env var table, curl examples for all 4 endpoints (search, extract, research, health), source fallback logic explanation, performance notes.
+
+---
+
+## File Tree — Current State
+
+```
+/srv/Axon/
+├── .env.example
+├── .gitignore
+├── CLAUDE.md                        ← this file
+├── requirements.txt
+├── docs/
+│   └── superpowers/plans/
+│       └── 2026-06-18-axon-search-api.md
+├── axon/
+│   ├── __init__.py
+│   ├── config.py                    ✅
+│   ├── models.py                    ✅
+│   ├── main.py                      ❌ MISSING
+│   ├── middleware/
+│   │   ├── __init__.py
+│   │   └── auth.py                  ✅
+│   ├── routers/
+│   │   ├── __init__.py
+│   │   ├── search.py                ✅
+│   │   ├── extract.py               ❌ MISSING
+│   │   └── research.py              ❌ MISSING
+│   └── services/
+│       ├── __init__.py
+│       ├── cache.py                 ✅
+│       ├── reranker.py              ✅
+│       ├── scraper.py               ✅ (includes SSRF protection)
+│       ├── synthesizer.py           ✅
+│       └── sources/
+│           ├── __init__.py
+│           ├── brave.py             ✅
+│           ├── ddg.py               ✅
+│           └── serper.py            ✅
+└── tests/
+    ├── __init__.py
+    ├── test_config.py               ✅  3 tests
+    ├── test_models.py               ✅  6 tests
+    ├── test_cache.py                ✅  7 tests
+    ├── test_sources.py              ✅ 10 tests (brave + ddg + serper)
+    ├── test_scraper.py              ✅  4 tests
+    ├── test_reranker.py             ✅  7 tests
+    ├── test_synthesizer.py          ✅  4 tests
+    ├── test_auth.py                 ✅  5 tests
+    ├── test_search_router.py        ✅  4 tests
+    ├── test_extract_router.py       ❌ MISSING
+    └── test_research_router.py      ❌ MISSING (covered in test_search_router.py per plan)
+```
+
+**Total tests passing right now: 50**
+
+---
+
+## Environment & Running Tests
+
+```bash
+# The venv is at /srv/Axon/venv (Python 3.14)
+source /srv/Axon/venv/bin/activate
+
+# Run all tests
+python -m pytest tests/ -v
+
+# Run a specific file
+python -m pytest tests/test_search_router.py -v
+
+# pytest-asyncio version: 1.4.0 (strict mode)
+# ALL async tests must be decorated with @pytest.mark.asyncio
+```
+
+Known warnings (not errors):
+- Starlette currently emits a `TestClient` deprecation warning about its `httpx` integration.
+
+---
+
+## Environment Variables (from .env.example)
+
+| Variable | Required | Default | Purpose |
+|---|---|---|---|
+| `BRAVE_API_KEY` | No | — | Enables Brave search source |
+| `SERPER_API_KEY` | No | — | Enables Serper fallback source |
+| `REDIS_URL` | No | `redis://localhost:6379` | Cache backend; omit for in-memory |
+| `API_KEYS` | No | — | Comma-separated auth keys; unset = no auth |
+| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible LLM endpoint |
+| `LITELLM_API_KEY` | No | — | Bearer token for LLM endpoint |
+| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model used for answer/report synthesis |
+| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | HuggingFace CrossEncoder model |
+| `PORT` | No | `8000` | uvicorn listen port |
+| `MAX_CONCURRENT_SCRAPES` | No | `10` | asyncio.Semaphore limit for scraping |
+| `SCRAPE_TIMEOUT` | No | `5` | Per-URL timeout in seconds |
+
+---
+
+## Coding Rules (enforced across all tasks)
+
+1. **Python 3.11+ types only** — `str | None` not `Optional[str]`, `list[T]` not `List[T]`, `dict[K,V]` not `Dict[K,V]`. No imports from `typing` for these.
+2. **Pydantic v2** — `model_dump()` and `model_dump_json()` only. Never `.dict()` or `.json()`.
+3. **All I/O is async** — `httpx.AsyncClient`, `redis.asyncio`, `asyncio.to_thread()` for blocking calls.
+4. **Never raise from service functions** — sources, scraper, synthesizer all catch all exceptions and return empty/None/error strings.
+5. **Tests use `@pytest.mark.asyncio`** — no anyio markers.
+6. **No `Optional` imports** — this kept appearing in subagent output and was fixed each time.
+
+---
+
+## Gotchas & Lessons Learned During Build
+
+- **`@pytest.mark.anyio` vs `asyncio`** — subagents kept using `anyio`. Every test file was checked for this. Always `@pytest.mark.asyncio`.
+- **`Optional[str]` vs `str | None`** — the plan's own code examples had `Optional` in them (for illustration). Subagents copied it. Had to fix in Tasks 3, 6.
+- **`importlib.reload` in tests** — one subagent used it to reset module state for monkeypatching. Fragile and banned. Replaced with direct `monkeypatch.setattr()`.
+- **SSRF** — automated security review flagged `scraper.py` after Task 6's commit. Fixed in Task 7's commit window by adding `_is_safe_url()` with full IP blocklist.
+- **Review of auth tests** — reviewer incorrectly flagged monkeypatching `settings.api_keys` as wrong (said it should patch `api_key_list`). But `api_key_list` is a `@property` — you can't patch it directly. Patching the underlying field is correct. Confirmed all 5 tests pass.
+- **Venv path** — the project uses `/srv/Axon/venv`. pytest-asyncio there is version 1.4.0. Must `source venv/bin/activate` before running any Python commands.
+
+---
+
+## Current Final State
+
+Tasks 11-14 are now complete:
+1. `axon/routers/extract.py` implements `POST /extract` with per-URL caching, trafilatura extraction, metadata title extraction, concurrent processing, and SSRF protection via the scraper URL safety check.
+2. `axon/routers/research.py` implements `POST /research` by generating sub-queries, running advanced searches concurrently, skipping failed sub-searches, deduplicating sources, and synthesizing a final report.
+3. `axon/main.py` creates the runnable FastAPI app with CORS, API key middleware, all routers, lifespan reranker loading, and `GET /health`.
+4. `README.md` documents setup, configuration, endpoint curl examples, source fallback behavior, caching, and performance notes.
+
+Additional cleanup:
+- `axon/config.py` and `axon/models.py` now use `str | None` style types instead of `Optional[...]`.
+- `axon/services/cache.py` now uses timezone-aware UTC datetimes, removing the Python 3.14 `datetime.utcnow()` warning.
+- `axon/services/sources/ddg.py` now prefers the current `ddgs` package and falls back to `duckduckgo_search`; `ddgs>=9.14.0` is in `requirements.txt`.
+- `axon/services/reranker.py` serializes model prediction and caps reranking to 4 chunks per document. This prevents `/research` from generating unbounded cross-encoder work on huge pages.
+- `axon/routers/research.py` runs sub-searches sequentially with 5 results per sub-query. This avoids OOM on the current server while preserving deep multi-query research behavior.
+- `axon/middleware/rate_limit.py` adds per-key/IP rate limiting.
+- `axon/middleware/request_context.py` adds `X-Request-ID` and records lightweight metrics.
+- `axon/routers/crawl.py` and `axon/routers/map.py` add Tavily-style bounded same-site crawl/map endpoints.
+- `axon/services/scraper.py` now has reusable safe fetch, extraction, URL normalization, and same-site link discovery helpers with retries.
+- `axon/services/index.py` adds a private SQLite FTS index; `POST /ingest`, `POST /index/search`, and normal `/search` can use private indexed content.
+- `axon/services/jobs.py` and `axon/routers/jobs.py` add in-process background jobs for research/crawl/map.
+- `axon/routers/tavily.py` adds `/tavily/search`, `/tavily/extract`, `/tavily/crawl`, and `/tavily/map` compatibility routes.
+- `axon/routers/validate.py` and citation helpers add basic citation coverage validation.
+- Browser extraction fallback is optional via `ENABLE_BROWSER_FALLBACK=true` if Playwright is installed.
+
+Systemd deployment completed:
+- Unit file: `/etc/systemd/system/axon.service`
+- Enabled: yes
+- Runtime env file: `/srv/Axon/.env` (ignored by git, mode `600`)
+- LiteLLM model configured: `AR/openai/gpt-oss-120b:free`
+- Service command: `/srv/Axon/venv/bin/uvicorn axon.main:app --host 0.0.0.0 --port 8000`
+
+Live endpoint verification completed through systemd:
+- `GET /health` -> 200, Redis connected, reranker loaded
+- `GET /openapi.json` -> 200, paths include `/extract`, `/health`, `/metrics`, `/crawl`, `/map`, `/research`, `/search`
+- `POST /extract` with `https://example.com` -> 200
+- `POST /search` basic with answer synthesis -> 200 using LiteLLM
+- `POST /search` advanced -> 200, completed in about 28 seconds
+- `POST /research` basic markdown -> 200, 15 sources, completed in about 163 seconds
+
+Operational note:
+- The BGE reranker is memory-heavy. Before reranker chunk capping, `/research` OOM-killed the service at roughly 4.2 GB. After capping and sequential research searches, the tested `/research` run peaked around 2 GB and completed successfully.
+
+The git branch is `master`. The project is runnable with:
+
+```bash
+source /srv/Axon/venv/bin/activate
+uvicorn axon.main:app --host 0.0.0.0 --port 8000
+```
+
+To run a full test suite health check right now:
+```bash
+source /srv/Axon/venv/bin/activate && python -m pytest tests/ -v
+```
+Expected: **61 passed**.

README.md

@@ -0,0 +1,330 @@
+# Axon Search API
+
+Axon is a private, self-hosted AI search API for Linux servers. It aggregates search results from Brave Search, DuckDuckGo, and Serper.dev, optionally scrapes full page content, reranks results with a local BGE cross-encoder, and can synthesize answers or research reports through any OpenAI-compatible LLM endpoint.
+
+## Features
+
+- `POST /search` for multi-source search with optional scraping, reranking, and answer synthesis
+- `POST /extract` for clean content extraction from one or more URLs
+- `POST /research` for multi-query research reports
+- `POST /crawl` for same-site traversal with extraction
+- `POST /map` for same-site URL discovery
+- `GET /health` for cache and reranker status
+- `GET /metrics` for lightweight request counters and latency
+- `POST /ingest` and `POST /index/search` for private local document retrieval
+- `/research/jobs`, `/crawl/jobs`, `/map/jobs`, and `GET /jobs/{id}` for async long-running work
+- `/tavily/*` compatibility routes for easier migration
+- `POST /validate/citations` for basic citation coverage checks
+- API key auth via `X-API-Key`
+- Per-key/IP rate limiting
+- Redis cache with in-memory fallback
+- CORS enabled for all origins
+- SSRF protection for URL scraping and extraction
+
+## Setup
+
+Axon requires Python 3.11 or newer.
+
+```bash
+python -m venv venv
+source venv/bin/activate
+pip install -r requirements.txt
+cp .env.example .env
+```
+
+Edit `.env` with any API keys you want to enable. At least DuckDuckGo works without a key, but Brave and Serper improve coverage and reliability.
+
+Start the server:
+
+```bash
+uvicorn axon.main:app --host 0.0.0.0 --port 8000
+```
+
+The reranker model is loaded at startup. The first run may download the configured HuggingFace model and can take longer than later starts.
+
+## Configuration
+
+| Variable | Required | Default | Purpose |
+|---|---:|---|---|
+| `BRAVE_API_KEY` | No | empty | Enables Brave Search results |
+| `SERPER_API_KEY` | No | empty | Enables Serper.dev fallback results |
+| `REDIS_URL` | No | `redis://localhost:6379` | Redis cache URL; falls back to memory if unavailable |
+| `API_KEYS` | No | empty | Comma-separated accepted API keys; empty disables auth |
+| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible chat completions base URL |
+| `LITELLM_API_KEY` | No | empty | Bearer token for synthesis requests |
+| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model sent to the LLM endpoint |
+| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | Sentence Transformers cross-encoder model |
+| `PORT` | No | `8000` | Preferred server port |
+| `MAX_CONCURRENT_SCRAPES` | No | `10` | Concurrent scrape limit for advanced search |
+| `SCRAPE_TIMEOUT` | No | `5` | Per-URL scrape timeout in seconds |
+| `EXTRACT_TIMEOUT` | No | `10` | Per-URL extraction timeout in seconds |
+| `RATE_LIMIT_PER_MINUTE` | No | `60` | Per API key or client IP limit; `0` disables |
+| `CRAWL_MAX_PAGES` | No | `50` | Server-side cap for crawl/map pages |
+| `CRAWL_MAX_DEPTH` | No | `3` | Server-side cap for crawl/map depth |
+| `INDEX_DB_PATH` | No | `axon_index.sqlite3` | SQLite FTS database path for private indexed docs |
+| `ENABLE_BROWSER_FALLBACK` | No | `false` | Enables optional Playwright rendering fallback when installed |
+
+## Authentication
+
+If `API_KEYS` is unset or empty, Axon runs in development mode and does not require authentication.
+
+If `API_KEYS` is set, send one configured key with each protected request:
+
+```bash
+curl -H "X-API-Key: your-secret-key" http://localhost:8000/health
+```
+
+`/health`, `/docs`, `/openapi.json`, and `/redoc` are exempt.
+
+## Endpoints
+
+### Health
+
+```bash
+curl http://localhost:8000/health
+```
+
+Example response:
+
+```json
+{
+  "status": "ok",
+  "redis": "in-memory fallback",
+  "reranker": "loaded"
+}
+```
+
+### Metrics
+
+```bash
+curl http://localhost:8000/metrics
+```
+
+Example response:
+
+```json
+{
+  "requests_total": 42,
+  "requests_by_path": {
+    "/search": 10,
+    "/extract": 3
+  },
+  "errors_total": 0,
+  "average_latency_ms": 1234.5
+}
+```
+
+### Search
+
+Basic search uses search API snippets and reranks them:
+
+```bash
+curl -X POST http://localhost:8000/search \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "query": "latest PostgreSQL indexing improvements",
+    "max_results": 10,
+    "search_depth": "basic",
+    "include_answer": false
+  }'
+```
+
+Advanced search scrapes result pages before reranking:
+
+```bash
+curl -X POST http://localhost:8000/search \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "query": "private AI search APIs",
+    "max_results": 10,
+    "search_depth": "advanced",
+    "include_answer": true,
+    "include_raw_content": false,
+    "include_private": true,
+    "domains_blacklist": ["reddit.com"],
+    "topic": "general",
+    "time_range": "month"
+  }'
+```
+
+### Extract
+
+```bash
+curl -X POST http://localhost:8000/extract \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "urls": [
+      "https://example.com/article"
+    ],
+    "include_images": false
+  }'
+```
+
+Each URL is processed independently. Failed URLs return `success: false` with an error while the rest of the request continues.
+
+### Research
+
+```bash
+curl -X POST http://localhost:8000/research \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "query": "state of open source vector databases",
+    "depth": "deep",
+    "search_depth": "basic",
+    "max_results_per_query": 5,
+    "output_format": "markdown"
+  }'
+```
+
+`depth: "basic"` generates 3 sub-queries. `depth: "deep"` generates 5 sub-queries. Sources are deduplicated by URL, and the final report is synthesized from the combined results. Use `search_depth: "basic"` for faster research, or `advanced` when scraped page content is worth the extra latency.
+
+For long-running work, create a background job:
+
+```bash
+curl -X POST http://localhost:8000/research/jobs \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{"query": "OpenAI", "search_depth": "basic"}'
+
+curl http://localhost:8000/jobs/<job_id> \
+  -H "X-API-Key: your-secret-key"
+```
+
+### Crawl
+
+```bash
+curl -X POST http://localhost:8000/crawl \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "url": "https://docs.example.com",
+    "max_depth": 1,
+    "max_pages": 20,
+    "include_raw_content": false,
+    "instructions": "python sdk"
+  }'
+```
+
+`/crawl` stays on the same hostname as the start URL, extracts each matching page, and returns discovered links. It is intentionally bounded by `CRAWL_MAX_PAGES` and `CRAWL_MAX_DEPTH`.
+
+### Map
+
+```bash
+curl -X POST http://localhost:8000/map \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "url": "https://docs.example.com",
+    "max_depth": 2,
+    "max_pages": 50,
+    "instructions": "api reference"
+  }'
+```
+
+`/map` returns same-site URLs without extracting full content.
+
+### Private Index
+
+Ingest private documents:
+
+```bash
+curl -X POST http://localhost:8000/ingest \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "documents": [
+      {
+        "id": "runbook-1",
+        "title": "Production Runbook",
+        "content": "Internal deployment and rollback notes.",
+        "url": "private://runbook-1"
+      }
+    ]
+  }'
+```
+
+Search only the private index:
+
+```bash
+curl -X POST http://localhost:8000/index/search \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{"query": "rollback notes"}'
+```
+
+Normal `/search` includes private index hits by default. Set `"include_private": false` to use only web sources.
+
+### Tavily-Compatible Routes
+
+Axon exposes compatibility routes under `/tavily`:
+
+```bash
+curl -X POST http://localhost:8000/tavily/search \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{"query": "OpenAI", "max_results": 5}'
+```
+
+Available compatibility routes:
+- `/tavily/search`
+- `/tavily/extract`
+- `/tavily/crawl`
+- `/tavily/map`
+
+### Citation Validation
+
+```bash
+curl -X POST http://localhost:8000/validate/citations \
+  -H "Content-Type: application/json" \
+  -H "X-API-Key: your-secret-key" \
+  -d '{
+    "report": "OpenAI was founded in 2015 [1].",
+    "sources": [
+      {"title": "Source", "url": "https://example.com", "content": "OpenAI was founded in 2015."}
+    ]
+  }'
+```
+
+## Search Source Logic
+
+Axon fetches Brave and DuckDuckGo concurrently. Serper.dev is called only when Brave returns fewer than 5 results. All results are merged and deduplicated by URL before filtering, scraping, reranking, and response truncation.
+
+If all enabled sources return no results, `/search` returns HTTP 503. Individual source failures are logged and treated as empty result sets.
+
+DuckDuckGo uses the current `ddgs` package when installed and falls back to `duckduckgo-search` for compatibility.
+
+## Caching
+
+- Search responses are cached for 10 minutes.
+- Successful extracted URL content is cached for 1 hour.
+- Redis is used when `REDIS_URL` is configured and reachable.
+- If Redis is unavailable, Axon logs a warning and uses in-memory cache for the current process.
+
+## Performance Notes
+
+Advanced search is slower than basic search because it fetches and extracts each result page. Tune `MAX_CONCURRENT_SCRAPES` and `SCRAPE_TIMEOUT` for your server and network limits.
+
+The reranker model is loaded once during FastAPI startup and reused for requests. Use a smaller cross-encoder model if startup time, memory use, or CPU latency is too high for your deployment.
+
+Reranking is serialized through a process-local lock and capped to a small number of chunks per document. This keeps the self-hosted service stable on modest servers, at the cost of some recall on very long pages.
+
+Optional browser rendering requires installing Playwright and browser assets separately, then setting `ENABLE_BROWSER_FALLBACK=true`.
+
+## Development
+
+Run tests:
+
+```bash
+source venv/bin/activate
+python -m pytest tests/ -v
+```
+
+Run a focused test file:
+
+```bash
+python -m pytest tests/test_search_router.py -v
+```

tests/test_rate_limit.py

@@ -0,0 +1,34 @@
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+
+def test_rate_limit_rejects_after_limit():
+    from axon.middleware.rate_limit import RateLimitMiddleware
+
+    app = FastAPI()
+
+    @app.get("/data")
+    async def data():
+        return {"ok": True}
+
+    app.add_middleware(RateLimitMiddleware, limit_per_minute=1)
+    client = TestClient(app)
+
+    assert client.get("/data").status_code == 200
+    assert client.get("/data").status_code == 429
+
+
+def test_rate_limit_exempts_health():
+    from axon.middleware.rate_limit import RateLimitMiddleware
+
+    app = FastAPI()
+
+    @app.get("/health")
+    async def health():
+        return {"ok": True}
+
+    app.add_middleware(RateLimitMiddleware, limit_per_minute=1)
+    client = TestClient(app)
+
+    assert client.get("/health").status_code == 200
+    assert client.get("/health").status_code == 200

tests/test_reranker.py

@@ -0,0 +1,74 @@
+import pytest
+
+
+def test_chunk_text_single_short_text():
+    from axon.services.reranker import chunk_text
+    chunks = chunk_text("hello world", max_tokens=512)
+    assert chunks == ["hello world"]
+
+
+def test_chunk_text_splits_long_text():
+    from axon.services.reranker import chunk_text
+    # 384 words per chunk at max_tokens=512 (512 * 0.75)
+    text = " ".join([f"word{i}" for i in range(500)])
+    chunks = chunk_text(text, max_tokens=512)
+    assert len(chunks) == 2
+    assert chunks[0].startswith("word0")
+
+
+def test_chunk_text_empty_string():
+    from axon.services.reranker import chunk_text
+    chunks = chunk_text("", max_tokens=512)
+    assert chunks == [""]
+
+
+def test_chunk_text_exact_boundary():
+    from axon.services.reranker import chunk_text
+    # 384 words should be exactly 1 chunk
+    text = " ".join(["x"] * 384)
+    chunks = chunk_text(text, max_tokens=512)
+    assert len(chunks) == 1
+
+
+@pytest.mark.asyncio
+async def test_rerank_results_sorts_by_score(monkeypatch):
+    import numpy as np
+
+    call_count = [0]
+
+    class FakeModel:
+        def predict(self, pairs):
+            # First pair gets low score, second gets high score
+            return np.array([0.1, 0.9])
+
+    monkeypatch.setattr("axon.services.reranker._model", FakeModel())
+
+    from axon.services.reranker import rerank_results
+    contents = [("https://low.com", "low relevance text"), ("https://high.com", "high relevance text")]
+    ranked = await rerank_results("query", contents)
+    assert ranked[0][0] == "https://high.com"
+    assert ranked[1][0] == "https://low.com"
+    assert ranked[0][2] > ranked[1][2]
+
+
+@pytest.mark.asyncio
+async def test_rerank_results_empty_input():
+    from axon.services.reranker import rerank_results
+    result = await rerank_results("query", [])
+    assert result == []
+
+
+@pytest.mark.asyncio
+async def test_rerank_results_handles_empty_content(monkeypatch):
+    import numpy as np
+
+    class FakeModel:
+        def predict(self, pairs):
+            return np.array([0.5] * len(pairs))
+
+    monkeypatch.setattr("axon.services.reranker._model", FakeModel())
+
+    from axon.services.reranker import rerank_results
+    contents = [("https://a.com", ""), ("https://b.com", "some text")]
+    ranked = await rerank_results("query", contents)
+    assert len(ranked) == 2

tests/test_research_router.py

@@ -0,0 +1,92 @@
+import pytest
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+from axon.models import SearchResponse, SearchResult
+
+
+def _result(url: str, title: str = "Title") -> SearchResult:
+    return SearchResult(title=title, url=url, content="content", score=0.9)
+
+
+@pytest.fixture
+def client(monkeypatch):
+    async def fake_generate(query, count=3):
+        return [f"{query} angle {i}" for i in range(count)]
+
+    async def fake_search(request):
+        return SearchResponse(
+            query=request.query,
+            results=[
+                _result("https://example.com/shared", "Shared"),
+                _result(f"https://example.com/{request.query.replace(' ', '-')}"),
+            ],
+            response_time=0.1,
+        )
+
+    async def fake_report(query, results, output_format="markdown"):
+        return f"report:{query}:{len(results)}:{output_format}"
+
+    monkeypatch.setattr("axon.routers.research.generate_sub_queries", fake_generate)
+    monkeypatch.setattr("axon.routers.research.search", fake_search)
+    monkeypatch.setattr("axon.routers.research.synthesize_report", fake_report)
+
+    from axon.routers.research import router
+
+    app = FastAPI()
+    app.include_router(router)
+    return TestClient(app)
+
+
+def test_research_basic_runs_three_sub_queries(client):
+    resp = client.post("/research", json={"query": "axon"})
+    assert resp.status_code == 200
+    data = resp.json()
+    assert len(data["sub_queries"]) == 3
+    assert data["report"] == "report:axon:4:markdown"
+    assert len(data["sources"]) == 4
+
+
+def test_research_deep_runs_five_sub_queries(client):
+    resp = client.post("/research", json={"query": "axon", "depth": "deep"})
+    assert resp.status_code == 200
+    data = resp.json()
+    assert len(data["sub_queries"]) == 5
+    assert len(data["sources"]) == 6
+
+
+def test_research_passes_json_output_format(client):
+    resp = client.post("/research", json={"query": "axon", "output_format": "json"})
+    assert resp.status_code == 200
+    assert resp.json()["report"].endswith(":json")
+
+
+def test_research_skips_failed_sub_search(monkeypatch):
+    async def fake_generate(query, count=3):
+        return ["one", "two", "three"]
+
+    async def fake_search(request):
+        if request.query == "two":
+            raise RuntimeError("source failed")
+        return SearchResponse(
+            query=request.query,
+            results=[_result(f"https://example.com/{request.query}")],
+            response_time=0.1,
+        )
+
+    async def fake_report(query, results, output_format="markdown"):
+        return f"count:{len(results)}"
+
+    monkeypatch.setattr("axon.routers.research.generate_sub_queries", fake_generate)
+    monkeypatch.setattr("axon.routers.research.search", fake_search)
+    monkeypatch.setattr("axon.routers.research.synthesize_report", fake_report)
+
+    from axon.routers.research import router
+
+    app = FastAPI()
+    app.include_router(router)
+    client = TestClient(app)
+
+    resp = client.post("/research", json={"query": "axon"})
+    assert resp.status_code == 200
+    assert resp.json()["report"] == "count:2"

tests/test_scraper.py

@@ -0,0 +1,69 @@
+import pytest
+
+
+@pytest.mark.asyncio
+async def test_scrape_url_returns_none_on_connection_error(monkeypatch):
+    import httpx
+
+    class FailClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def get(self, *args, **kwargs):
+            raise httpx.ConnectError("refused")
+
+    monkeypatch.setattr("axon.services.scraper.httpx.AsyncClient", FailClient)
+
+    from axon.services.scraper import scrape_url
+    result = await scrape_url("https://unreachable.example.com")
+    assert result is None
+
+
+@pytest.mark.asyncio
+async def test_scrape_url_returns_none_on_4xx(monkeypatch):
+    import httpx
+
+    class BadResponse:
+        def raise_for_status(self):
+            raise httpx.HTTPStatusError("404", request=None, response=None)
+        @property
+        def text(self): return ""
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def get(self, *args, **kwargs): return BadResponse()
+
+    monkeypatch.setattr("axon.services.scraper.httpx.AsyncClient", FakeClient)
+
+    from axon.services.scraper import scrape_url
+    result = await scrape_url("https://example.com/404")
+    assert result is None
+
+
+@pytest.mark.asyncio
+async def test_scrape_urls_returns_dict_keyed_by_url(monkeypatch):
+    async def fake_scrape(url, timeout=5):
+        return f"content of {url}"
+
+    monkeypatch.setattr("axon.services.scraper.scrape_url", fake_scrape)
+
+    from axon.services.scraper import scrape_urls
+    urls = ["https://a.com", "https://b.com"]
+    result = await scrape_urls(urls, max_concurrent=2, timeout=5)
+    assert result["https://a.com"] == "content of https://a.com"
+    assert result["https://b.com"] == "content of https://b.com"
+
+
+@pytest.mark.asyncio
+async def test_scrape_urls_skips_individual_failures(monkeypatch):
+    async def fake_scrape(url, timeout=5):
+        if "bad" in url:
+            return None
+        return "good content"
+
+    monkeypatch.setattr("axon.services.scraper.scrape_url", fake_scrape)
+
+    from axon.services.scraper import scrape_urls
+    result = await scrape_urls(["https://good.com", "https://bad.com"])
+    assert result["https://good.com"] == "good content"
+    assert result["https://bad.com"] is None

tests/test_search_router.py

@@ -0,0 +1,125 @@
+import pytest
+from fastapi.testclient import TestClient
+from fastapi import FastAPI
+from axon.models import SearchResult
+
+
+def _mock_result(url="https://ex.com", title="T", content="C", score=0.8):
+    return SearchResult(title=title, url=url, content=content, score=score)
+
+
+async def _async(val):
+    return val
+
+
+@pytest.fixture
+def client(monkeypatch):
+    from axon.services.cache import InMemoryCache
+    cache = InMemoryCache()
+
+    async def async_cache():
+        return cache
+
+    monkeypatch.setattr("axon.routers.search.get_cache", async_cache)
+
+    results_5 = [_mock_result(f"https://ex{i}.com", f"T{i}", f"C{i}") for i in range(5)]
+
+    async def fake_brave(*a, **kw):
+        return results_5
+
+    async def fake_ddg(*a, **kw):
+        return []
+
+    async def fake_rerank(q, c):
+        return [(url, text, 0.8) for url, text in c]
+
+    monkeypatch.setattr("axon.routers.search.search_brave", fake_brave)
+    monkeypatch.setattr("axon.routers.search.search_ddg", fake_ddg)
+    monkeypatch.setattr("axon.routers.search.rerank_results", fake_rerank)
+
+    from axon.routers.search import router
+    app = FastAPI()
+    app.include_router(router)
+    return TestClient(app)
+
+
+def test_search_basic_returns_200(client):
+    resp = client.post("/search", json={"query": "test"})
+    assert resp.status_code == 200
+    data = resp.json()
+    assert "results" in data
+    assert "response_time" in data
+    assert data["query"] == "test"
+
+
+def test_search_deduplicates_urls():
+    from axon.routers.search import deduplicate_results
+    r1 = _mock_result("https://x.com")
+    r2 = _mock_result("https://x.com")
+    r3 = _mock_result("https://y.com")
+    deduped = deduplicate_results([r1, r2, r3])
+    assert len(deduped) == 2
+    urls = [r.url for r in deduped]
+    assert urls.count("https://x.com") == 1
+
+
+def test_search_503_if_all_sources_fail(monkeypatch):
+    from axon.services.cache import InMemoryCache
+    cache = InMemoryCache()
+
+    async def async_cache():
+        return cache
+
+    monkeypatch.setattr("axon.routers.search.get_cache", async_cache)
+
+    async def empty(*a, **kw):
+        return []
+
+    monkeypatch.setattr("axon.routers.search.search_brave", empty)
+    monkeypatch.setattr("axon.routers.search.search_ddg", empty)
+    monkeypatch.setattr("axon.routers.search.search_serper", empty)
+
+    from axon.routers.search import router
+    app = FastAPI()
+    app.include_router(router)
+    c = TestClient(app, raise_server_exceptions=False)
+    resp = c.post("/search", json={"query": "test"})
+    assert resp.status_code == 503
+
+
+def test_search_whitelist_filter(monkeypatch):
+    from axon.services.cache import InMemoryCache
+    cache = InMemoryCache()
+
+    async def async_cache():
+        return cache
+
+    monkeypatch.setattr("axon.routers.search.get_cache", async_cache)
+
+    results = [
+        _mock_result("https://allowed.com/page"),
+        _mock_result("https://blocked.com/page"),
+    ]
+
+    async def fake_brave(*a, **kw):
+        return results
+
+    async def fake_ddg(*a, **kw):
+        return []
+
+    async def fake_rerank(q, c):
+        return [(u, t, 0.8) for u, t in c]
+
+    monkeypatch.setattr("axon.routers.search.search_brave", fake_brave)
+    monkeypatch.setattr("axon.routers.search.search_ddg", fake_ddg)
+    monkeypatch.setattr("axon.routers.search.rerank_results", fake_rerank)
+
+    from axon.routers.search import router
+    app = FastAPI()
+    app.include_router(router)
+    c = TestClient(app)
+    resp = c.post("/search", json={"query": "test", "domains_whitelist": ["allowed.com"]})
+    assert resp.status_code == 200
+    urls = [r["url"] for r in resp.json()["results"]]
+    assert all("allowed.com" in u for u in urls)
+    assert len(urls) == 1

tests/test_sources.py

@@ -0,0 +1,214 @@
+import pytest
+from unittest.mock import AsyncMock, MagicMock, patch
+from axon.models import SearchResult
+
+
+# ── Brave ──────────────────────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_brave_returns_empty_without_api_key(monkeypatch):
+    monkeypatch.setattr("axon.services.sources.brave.settings.brave_api_key", None)
+    from axon.services.sources.brave import search_brave
+    results = await search_brave("test")
+    assert results == []
+
+
+@pytest.mark.asyncio
+async def test_brave_parses_web_results(monkeypatch):
+    import axon.services.sources.brave as brave_mod
+
+    fake_data = {
+        "web": {
+            "results": [
+                {
+                    "title": "Hello",
+                    "url": "https://example.com",
+                    "description": "A description",
+                    "page_age": "2024-01-15",
+                    "meta_url": {"favicon": "https://example.com/fav.ico"},
+                }
+            ]
+        }
+    }
+
+    class FakeResponse:
+        def raise_for_status(self): pass
+        def json(self): return fake_data
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def get(self, *args, **kwargs): return FakeResponse()
+
+    monkeypatch.setattr(brave_mod, "settings", type("S", (), {"brave_api_key": "fake-key"})())
+    monkeypatch.setattr("axon.services.sources.brave.httpx.AsyncClient", FakeClient)
+
+    results = await brave_mod.search_brave("hello")
+    assert len(results) == 1
+    assert results[0].url == "https://example.com"
+    assert results[0].title == "Hello"
+    assert results[0].content == "A description"
+    assert results[0].published_date == "2024-01-15"
+    assert results[0].favicon == "https://example.com/fav.ico"
+
+
+@pytest.mark.asyncio
+async def test_brave_returns_empty_on_http_error(monkeypatch):
+    import httpx
+    monkeypatch.setattr("axon.services.sources.brave.settings.brave_api_key", "fake-key")
+
+    class FailClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def get(self, *args, **kwargs):
+            raise httpx.ConnectError("network error")
+
+    monkeypatch.setattr("axon.services.sources.brave.httpx.AsyncClient", FailClient)
+
+    from axon.services.sources.brave import search_brave
+    results = await search_brave("test")
+    assert results == []
+
+
+# ── DDG ────────────────────────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_ddg_parses_text_results(monkeypatch):
+    fake_items = [
+        {"title": "DDG Result", "href": "https://ddg.com/page", "body": "Some content"},
+    ]
+
+    class FakeDDGS:
+        def __enter__(self): return self
+        def __exit__(self, *_): pass
+        def text(self, *args, **kwargs): return iter(fake_items)
+        def news(self, *args, **kwargs): return iter([])
+
+    monkeypatch.setattr("axon.services.sources.ddg.DDGS", FakeDDGS)
+
+    from axon.services.sources.ddg import search_ddg
+    results = await search_ddg("test")
+    assert len(results) == 1
+    assert results[0].url == "https://ddg.com/page"
+    assert results[0].content == "Some content"
+
+
+@pytest.mark.asyncio
+async def test_ddg_news_uses_url_key(monkeypatch):
+    fake_news = [
+        {"title": "News", "url": "https://news.com/story", "body": "News body", "date": "2024-01-01"},
+    ]
+
+    class FakeDDGS:
+        def __enter__(self): return self
+        def __exit__(self, *_): pass
+        def text(self, *args, **kwargs): return iter([])
+        def news(self, *args, **kwargs): return iter(fake_news)
+
+    monkeypatch.setattr("axon.services.sources.ddg.DDGS", FakeDDGS)
+
+    from axon.services.sources.ddg import search_ddg
+    results = await search_ddg("news query", topic="news")
+    assert len(results) == 1
+    assert results[0].url == "https://news.com/story"
+    assert results[0].content == "News body"
+    assert results[0].published_date == "2024-01-01"
+
+
+@pytest.mark.asyncio
+async def test_ddg_returns_empty_on_error(monkeypatch):
+    class FailDDGS:
+        def __enter__(self): return self
+        def __exit__(self, *_): pass
+        def text(self, *args, **kwargs): raise RuntimeError("DDG down")
+
+    monkeypatch.setattr("axon.services.sources.ddg.DDGS", FailDDGS)
+
+    from axon.services.sources.ddg import search_ddg
+    results = await search_ddg("test")
+    assert results == []
+
+
+@pytest.mark.asyncio
+async def test_ddg_passes_timelimit(monkeypatch):
+    """Verify that time_range parameter is correctly mapped to timelimit for DDGS."""
+    captured_timelimit = {}
+
+    class FakeDDGS:
+        def __enter__(self): return self
+        def __exit__(self, *_): pass
+        def text(self, *args, **kwargs):
+            captured_timelimit["value"] = kwargs.get("timelimit")
+            return iter([])
+        def news(self, *args, **kwargs): return iter([])
+
+    monkeypatch.setattr("axon.services.sources.ddg.DDGS", FakeDDGS)
+
+    from axon.services.sources.ddg import search_ddg
+    await search_ddg("test", time_range="week")
+    assert captured_timelimit.get("value") == "w"
+
+
+# ── Serper ─────────────────────────────────────────────────────────────────
+
+@pytest.mark.asyncio
+async def test_serper_returns_empty_without_api_key(monkeypatch):
+    monkeypatch.setattr("axon.services.sources.serper.settings.serper_api_key", None)
+    from axon.services.sources.serper import search_serper
+    results = await search_serper("test")
+    assert results == []
+
+
+@pytest.mark.asyncio
+async def test_serper_parses_organic_results(monkeypatch):
+    monkeypatch.setattr("axon.services.sources.serper.settings.serper_api_key", "fake-key")
+
+    fake_data = {
+        "organic": [
+            {"title": "Serper Result", "link": "https://s.com", "snippet": "Snippet text", "date": "Jan 1, 2024"},
+        ]
+    }
+
+    class FakeResponse:
+        def raise_for_status(self): pass
+        def json(self): return fake_data
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def post(self, *args, **kwargs): return FakeResponse()
+
+    monkeypatch.setattr("axon.services.sources.serper.httpx.AsyncClient", FakeClient)
+
+    from axon.services.sources.serper import search_serper
+    results = await search_serper("test")
+    assert len(results) == 1
+    assert results[0].url == "https://s.com"
+    assert results[0].content == "Snippet text"
+
+
+@pytest.mark.asyncio
+async def test_serper_uses_news_endpoint(monkeypatch):
+    """Verify that topic='news' uses the /news endpoint."""
+    monkeypatch.setattr("axon.services.sources.serper.settings.serper_api_key", "fake-key")
+
+    captured_url = {}
+    fake_data = {"news": [{"title": "News", "link": "https://n.com", "snippet": "News snippet"}]}
+
+    class FakeResponse:
+        def raise_for_status(self): pass
+        def json(self): return fake_data
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def post(self, url, *args, **kwargs):
+            captured_url["value"] = url
+            return FakeResponse()
+
+    monkeypatch.setattr("axon.services.sources.serper.httpx.AsyncClient", FakeClient)
+
+    from axon.services.sources.serper import search_serper
+    results = await search_serper("test", topic="news")
+    assert "/news" in captured_url.get("value", "")
+    assert len(results) == 1

tests/test_synthesizer.py

@@ -0,0 +1,80 @@
+import pytest
+from unittest.mock import AsyncMock
+from axon.models import SearchResult
+
+
+def make_results(n=3):
+    return [
+        SearchResult(title=f"T{i}", url=f"https://ex{i}.com", content=f"Content {i}", score=0.9)
+        for i in range(n)
+    ]
+
+
+@pytest.mark.asyncio
+async def test_synthesize_answer_returns_string(monkeypatch):
+    fake_response = {
+        "choices": [{"message": {"content": "The answer is 42."}}]
+    }
+
+    class FakeResponse:
+        def raise_for_status(self): pass
+        def json(self): return fake_response
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def post(self, *args, **kwargs): return FakeResponse()
+
+    monkeypatch.setattr("axon.services.synthesizer.httpx.AsyncClient", FakeClient)
+
+    from axon.services.synthesizer import synthesize_answer
+    answer = await synthesize_answer("What is the answer?", make_results())
+    assert answer == "The answer is 42."
+
+
+@pytest.mark.asyncio
+async def test_synthesize_answer_empty_results():
+    from axon.services.synthesizer import synthesize_answer
+    answer = await synthesize_answer("query", [])
+    assert "No search results" in answer
+
+
+@pytest.mark.asyncio
+async def test_generate_sub_queries_parses_lines(monkeypatch):
+    fake_response = {
+        "choices": [{"message": {"content": "Query one\nQuery two\nQuery three"}}]
+    }
+
+    class FakeResponse:
+        def raise_for_status(self): pass
+        def json(self): return fake_response
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def post(self, *args, **kwargs): return FakeResponse()
+
+    monkeypatch.setattr("axon.services.synthesizer.httpx.AsyncClient", FakeClient)
+
+    from axon.services.synthesizer import generate_sub_queries
+    queries = await generate_sub_queries("main topic", count=3)
+    assert len(queries) == 3
+    assert queries[0] == "Query one"
+
+
+@pytest.mark.asyncio
+async def test_generate_sub_queries_fallback_on_error(monkeypatch):
+    import httpx
+
+    class FakeClient:
+        async def __aenter__(self): return self
+        async def __aexit__(self, *_): pass
+        async def post(self, *args, **kwargs):
+            raise httpx.ConnectError("LLM down")
+
+    monkeypatch.setattr("axon.services.synthesizer.httpx.AsyncClient", FakeClient)
+
+    from axon.services.synthesizer import generate_sub_queries
+    queries = await generate_sub_queries("main topic", count=3)
+    assert len(queries) == 3
+    assert "main topic" in queries[0]