Import documentation: CLAUDE.md
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
This commit is contained in:
arsam
@@ -0,0 +1,816 @@
|
||||
MAIN PROMPT: "Build me a private, self-hosted AI search API called "Axon" — a better alternative to Tavily. It should be a FastAPI backend that I can run on my Linux server.
|
||||
|
||||
## Stack
|
||||
- Python 3.11+
|
||||
- FastAPI
|
||||
- trafilatura (content extraction)
|
||||
- httpx (async HTTP)
|
||||
- sentence-transformers with BAAI/bge-reranker-v2-m3 (reranking)
|
||||
- redis (caching)
|
||||
- pydantic v2 (schemas)
|
||||
- uvicorn (server)
|
||||
|
||||
## Search Sources
|
||||
Implement all three and merge results, deduplicating by URL:
|
||||
1. Brave Search API (primary) — via BRAVE_API_KEY env var
|
||||
2. DuckDuckGo — using the duckduckgo-search Python package, no key needed
|
||||
3. Serper.dev (fallback/secondary) — via SERPER_API_KEY env var, only called if Brave returns fewer than 5 results
|
||||
|
||||
## Project Structure
|
||||
axon/
|
||||
├── main.py
|
||||
├── config.py
|
||||
├── models.py
|
||||
├── routers/
|
||||
│ ├── search.py
|
||||
│ ├── extract.py
|
||||
│ └── research.py
|
||||
├── services/
|
||||
│ ├── sources/
|
||||
│ │ ├── brave.py
|
||||
│ │ ├── ddg.py
|
||||
│ │ └── serper.py
|
||||
│ ├── scraper.py
|
||||
│ ├── reranker.py
|
||||
│ ├── synthesizer.py
|
||||
│ └── cache.py
|
||||
├── middleware/
|
||||
│ └── auth.py
|
||||
├── .env.example
|
||||
├── requirements.txt
|
||||
└── README.md
|
||||
|
||||
## Endpoints
|
||||
|
||||
### POST /search
|
||||
Main search endpoint.
|
||||
Request body:
|
||||
{
|
||||
"query": "string",
|
||||
"max_results": 10, // default 10, max 20
|
||||
"search_depth": "basic" | "advanced", // basic = no scraping, advanced = full scrape + rerank
|
||||
"include_answer": false, // if true, synthesize an answer via LLM
|
||||
"include_raw_content": false,
|
||||
"domains_whitelist": [], // only return results from these domains
|
||||
"domains_blacklist": [], // never return results from these domains
|
||||
"topic": "general" | "news",
|
||||
"time_range": "day" | "week" | "month" | "year" | null
|
||||
}
|
||||
|
||||
Response:
|
||||
{
|
||||
"query": "string",
|
||||
"answer": "string | null",
|
||||
"results": [
|
||||
{
|
||||
"title": "string",
|
||||
"url": "string",
|
||||
"content": "string", // snippet or extracted text
|
||||
"score": 0.95, // relevance score 0-1
|
||||
"published_date": "string | null",
|
||||
"favicon": "string | null"
|
||||
}
|
||||
],
|
||||
"response_time": 1.23,
|
||||
"credits_used": 1
|
||||
}
|
||||
|
||||
### POST /extract
|
||||
Extract clean content from a URL or list of URLs.
|
||||
Request body:
|
||||
{
|
||||
"urls": ["https://example.com"],
|
||||
"include_images": false
|
||||
}
|
||||
|
||||
Response:
|
||||
{
|
||||
"results": [
|
||||
{
|
||||
"url": "string",
|
||||
"title": "string",
|
||||
"content": "string",
|
||||
"success": true,
|
||||
"error": null
|
||||
}
|
||||
]
|
||||
}
|
||||
|
||||
### POST /research
|
||||
Deep research endpoint. Given a topic, auto-generate 3-5 sub-queries, run /search on each, deduplicate, rerank everything together, synthesize a final report.
|
||||
Request body:
|
||||
{
|
||||
"query": "string",
|
||||
"depth": "basic" | "deep", // basic = 3 sub-queries, deep = 5 sub-queries
|
||||
"output_format": "markdown" | "json"
|
||||
}
|
||||
|
||||
Response:
|
||||
{
|
||||
"query": "string",
|
||||
"report": "string", // markdown or JSON report
|
||||
"sources": [...], // all sources used
|
||||
"sub_queries": [...], // the generated sub-queries
|
||||
"response_time": 12.4
|
||||
}
|
||||
|
||||
### GET /health
|
||||
Returns server status, redis connection status, reranker model load status.
|
||||
|
||||
## Core Logic Details
|
||||
|
||||
### Search Flow (advanced depth)
|
||||
1. Query all enabled sources concurrently with asyncio.gather
|
||||
2. Merge and deduplicate results by URL
|
||||
3. For each result URL, scrape full content using trafilatura (async, with 5s timeout per URL, max 10 concurrent scrapes using asyncio.Semaphore)
|
||||
4. Chunk content to max 512 tokens per chunk
|
||||
5. Run bge-reranker-v2-m3 to score all chunks against the original query
|
||||
6. Take the top-scoring chunk per URL as that result's content
|
||||
7. Sort results by score descending
|
||||
8. If include_answer is true, send top 5 results as context to LLM via LITELLM_BASE_URL + LITELLM_API_KEY and get a synthesized answer
|
||||
|
||||
### Search Flow (basic depth)
|
||||
1. Same as above steps 1-2 (multi-source fetch + dedup)
|
||||
2. Skip scraping — use the snippet returned by the search APIs directly
|
||||
3. Still run reranker on snippets
|
||||
4. Return results
|
||||
|
||||
### Caching
|
||||
- Use Redis to cache search results for 10 minutes (key = hash of query + params)
|
||||
- Cache extracted content per URL for 1 hour
|
||||
- If REDIS_URL is not set, fall back to an in-memory dict cache
|
||||
|
||||
### Auth
|
||||
- API key auth via X-API-Key header
|
||||
- Keys stored in API_KEYS env var as a comma-separated list
|
||||
- If API_KEYS is not set, skip auth entirely (dev mode)
|
||||
|
||||
### LLM Synthesis
|
||||
- Call LLM via LITELLM_BASE_URL + LITELLM_API_KEY (standard OpenAI-compatible format)
|
||||
- Model set via SYNTHESIS_MODEL env var, default "claude-sonnet-4-6"
|
||||
- System prompt: "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."
|
||||
|
||||
## Config (.env.example)
|
||||
BRAVE_API_KEY=
|
||||
SERPER_API_KEY=
|
||||
REDIS_URL=redis://localhost:6379
|
||||
API_KEYS=your-secret-key-here
|
||||
LITELLM_BASE_URL=http://localhost:4000
|
||||
LITELLM_API_KEY=
|
||||
SYNTHESIS_MODEL=claude-sonnet-4-6
|
||||
RERANKER_MODEL=BAAI/bge-reranker-v2-m3
|
||||
PORT=8000
|
||||
MAX_CONCURRENT_SCRAPES=10
|
||||
SCRAPE_TIMEOUT=5
|
||||
|
||||
## Error Handling
|
||||
- Every endpoint must return proper HTTP status codes
|
||||
- Scraping failures for individual URLs should be silently skipped, not crash the request
|
||||
- Source API failures (Brave down, etc.) should be caught and logged, fall through to remaining sources
|
||||
- Return 503 only if ALL sources fail
|
||||
|
||||
## Requirements
|
||||
- All I/O must be fully async
|
||||
- The reranker model should be loaded once at startup into memory, not per-request
|
||||
- Log all requests with query, source count, scrape count, rerank score range, and response time
|
||||
- Add CORS middleware allowing all origins
|
||||
- Write a clean README with setup instructions and example curl commands for all endpoints"
|
||||
|
||||
# Axon Search API — Complete Project Reference
|
||||
|
||||
## What This Is
|
||||
|
||||
**Axon** is a self-hosted, private AI search API — built as a better alternative to Tavily. It is a FastAPI backend meant to run on a Linux server. It aggregates results from multiple search engines, scrapes full page content, reranks with a local ML model, and optionally synthesizes an AI-written answer using any OpenAI-compatible LLM endpoint.
|
||||
|
||||
**The goal:** `POST /search` with a query → get back ranked, cleaned, optionally AI-answered results. No rate limits you don't control. No external service owning your queries.
|
||||
|
||||
---
|
||||
|
||||
## How It Is Being Built
|
||||
|
||||
The project is being implemented via **Subagent-Driven Development** (one subagent per task, reviewed after each). The implementation plan lives at:
|
||||
|
||||
```
|
||||
docs/superpowers/plans/2026-06-18-axon-search-api.md
|
||||
```
|
||||
|
||||
Progress ledger (survives context compression):
|
||||
```
|
||||
.git/sdd/progress.md
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Overall Architecture
|
||||
|
||||
```
|
||||
Request
|
||||
│
|
||||
▼
|
||||
APIKeyMiddleware ← checks X-API-Key header
|
||||
│
|
||||
├── POST /search ← routers/search.py
|
||||
│ │
|
||||
│ ├── get_cache() check Redis / in-memory
|
||||
│ ├── search_brave() ──┐
|
||||
│ ├── search_ddg() ──┼── asyncio.gather (concurrent)
|
||||
│ └── search_serper() ─┘ (serper only if brave < 5 results)
|
||||
│ │
|
||||
│ ├── deduplicate by URL
|
||||
│ ├── filter whitelist/blacklist
|
||||
│ ├── scrape_urls() [if advanced depth]
|
||||
│ ├── rerank_results() ← CrossEncoder scores all chunks
|
||||
│ ├── synthesize_answer() [if include_answer=True]
|
||||
│ └── store in cache, return
|
||||
│
|
||||
├── POST /extract ← routers/extract.py [NOT BUILT YET]
|
||||
│ └── per-URL: httpx fetch → trafilatura extract → cache 1h
|
||||
│
|
||||
├── POST /research ← routers/research.py [NOT BUILT YET]
|
||||
│ ├── generate_sub_queries() via LLM
|
||||
│ ├── run /search for each sub-query
|
||||
│ ├── deduplicate all results
|
||||
│ └── synthesize_report()
|
||||
│
|
||||
└── GET /health ← main.py [NOT BUILT YET]
|
||||
└── returns redis status + reranker load status
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Current Build Status
|
||||
|
||||
### ✅ DONE — 10 of 14 Tasks
|
||||
|
||||
All of the following have been implemented, reviewed by a subagent, and fixed where needed.
|
||||
|
||||
---
|
||||
|
||||
#### Task 1 — Project Scaffold ✅
|
||||
**Commits:** `556b771..176e389`
|
||||
|
||||
Files created:
|
||||
- `requirements.txt` — all pip dependencies
|
||||
- `.env.example` — all configurable env vars with defaults
|
||||
- `.gitignore`
|
||||
- `axon/__init__.py`, `axon/middleware/__init__.py`, `axon/routers/__init__.py`, `axon/services/__init__.py`, `axon/services/sources/__init__.py`
|
||||
- `tests/__init__.py`
|
||||
- `axon/config.py` — `Settings` class and `settings` singleton
|
||||
|
||||
`axon/config.py` — the full thing, verbatim:
|
||||
```python
|
||||
from typing import Optional
|
||||
from pydantic_settings import BaseSettings
|
||||
|
||||
class Settings(BaseSettings):
|
||||
brave_api_key: Optional[str] = None
|
||||
serper_api_key: Optional[str] = None
|
||||
redis_url: Optional[str] = "redis://localhost:6379"
|
||||
api_keys: Optional[str] = None
|
||||
litellm_base_url: str = "http://localhost:4000"
|
||||
litellm_api_key: Optional[str] = None
|
||||
synthesis_model: str = "claude-sonnet-4-6"
|
||||
reranker_model: str = "BAAI/bge-reranker-v2-m3"
|
||||
port: int = 8000
|
||||
max_concurrent_scrapes: int = 10
|
||||
scrape_timeout: int = 5
|
||||
|
||||
@property
|
||||
def api_key_list(self) -> list[str]:
|
||||
if not self.api_keys:
|
||||
return []
|
||||
return [k.strip() for k in self.api_keys.split(",") if k.strip()]
|
||||
|
||||
model_config = {"env_file": ".env", "extra": "ignore"}
|
||||
|
||||
settings = Settings()
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Task 2 — Pydantic Models ✅
|
||||
**Commits:** `176e389..b8e6a51`
|
||||
**File:** `axon/models.py`
|
||||
|
||||
All 8 schemas. Key ones:
|
||||
|
||||
```python
|
||||
class SearchRequest(BaseModel):
|
||||
query: str
|
||||
max_results: int = Field(default=10, ge=1, le=20)
|
||||
search_depth: Literal["basic", "advanced"] = "basic"
|
||||
include_answer: bool = False
|
||||
include_raw_content: bool = False
|
||||
domains_whitelist: list[str] = []
|
||||
domains_blacklist: list[str] = []
|
||||
topic: Literal["general", "news"] = "general"
|
||||
time_range: Optional[Literal["day", "week", "month", "year"]] = None
|
||||
|
||||
class SearchResult(BaseModel):
|
||||
title: str
|
||||
url: str
|
||||
content: str
|
||||
score: float = 0.5
|
||||
published_date: Optional[str] = None
|
||||
favicon: Optional[str] = None
|
||||
|
||||
class SearchResponse(BaseModel):
|
||||
query: str
|
||||
answer: Optional[str] = None
|
||||
results: list[SearchResult]
|
||||
response_time: float
|
||||
credits_used: int = 1
|
||||
|
||||
class ExtractRequest(BaseModel):
|
||||
urls: list[str]
|
||||
include_images: bool = False
|
||||
|
||||
class ExtractResult(BaseModel):
|
||||
url: str
|
||||
title: Optional[str] = None
|
||||
content: str = ""
|
||||
success: bool
|
||||
error: Optional[str] = None
|
||||
|
||||
class ExtractResponse(BaseModel):
|
||||
results: list[ExtractResult]
|
||||
|
||||
class ResearchRequest(BaseModel):
|
||||
query: str
|
||||
depth: Literal["basic", "deep"] = "basic"
|
||||
output_format: Literal["markdown", "json"] = "markdown"
|
||||
|
||||
class ResearchResponse(BaseModel):
|
||||
query: str
|
||||
report: str
|
||||
sources: list[SearchResult]
|
||||
sub_queries: list[str]
|
||||
response_time: float
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Task 3 — Cache Service ✅
|
||||
**Commits:** `b8e6a51..200085c`
|
||||
**File:** `axon/services/cache.py`
|
||||
|
||||
Two backends with identical async interface:
|
||||
- `InMemoryCache` — dict with TTL timestamps. Used when Redis is unavailable or `REDIS_URL` unset.
|
||||
- `RedisCache` — wraps `redis.asyncio` client. Uses `setex` for TTL.
|
||||
|
||||
Key functions:
|
||||
```python
|
||||
async def get_cache() -> InMemoryCache | RedisCache:
|
||||
# Singleton. Tries Redis on first call; falls back to in-memory on failure.
|
||||
|
||||
def make_cache_key(prefix: str, **kwargs) -> str:
|
||||
# SHA-256 of JSON-serialized kwargs. Returns "prefix:<hex>"
|
||||
# Example: make_cache_key("search", query="hello") → "search:abc123..."
|
||||
```
|
||||
|
||||
TTLs used by callers (not enforced here):
|
||||
- Search results: 600 seconds (10 minutes)
|
||||
- Extracted content: 3600 seconds (1 hour)
|
||||
|
||||
---
|
||||
|
||||
#### Task 4 — Brave Search Source ✅
|
||||
**Commits:** `200085c..5da0eb9`
|
||||
**File:** `axon/services/sources/brave.py`
|
||||
|
||||
```python
|
||||
async def search_brave(
|
||||
query: str,
|
||||
count: int = 10,
|
||||
topic: str = "general",
|
||||
time_range: str | None = None,
|
||||
) -> list[SearchResult]:
|
||||
```
|
||||
|
||||
- Returns `[]` immediately if `settings.brave_api_key` is falsy
|
||||
- Returns `[]` on any HTTP/network error (never raises)
|
||||
- `topic="news"` → hits `/res/v1/news/search`; else `/res/v1/web/search`
|
||||
- Freshness param map: `day→pd`, `week→pw`, `month→pm`, `year→py`
|
||||
- Header: `X-Subscription-Token`
|
||||
- Parses `description` field (falls back to `extra_snippets[0]`) for content
|
||||
- Provides `favicon` from `meta_url.favicon`
|
||||
|
||||
---
|
||||
|
||||
#### Task 5 — DDG + Serper Sources ✅
|
||||
**Commits:** `5da0eb9..5a13863`
|
||||
**Files:** `axon/services/sources/ddg.py`, `axon/services/sources/serper.py`
|
||||
|
||||
**DDG:**
|
||||
```python
|
||||
async def search_ddg(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
|
||||
```
|
||||
- Uses `duckduckgo-search` package via `asyncio.to_thread()` (blocking call)
|
||||
- Text results: url = `item["href"]`, content = `item["body"]`
|
||||
- News results: url = `item["url"]`, content = `item["body"]`
|
||||
- Timelimit map: `day→d`, `week→w`, `month→m`, `year→y`
|
||||
- No API key needed
|
||||
|
||||
**Serper:**
|
||||
```python
|
||||
async def search_serper(query, count=10, topic="general", time_range=None) -> list[SearchResult]:
|
||||
```
|
||||
- Returns `[]` if `settings.serper_api_key` is falsy
|
||||
- `topic="news"` → `https://google.serper.dev/news`; else `/search`
|
||||
- TBS map: `day→qdr:d`, `week→qdr:w`, `month→qdr:m`, `year→qdr:y`
|
||||
- Header: `X-API-KEY`
|
||||
- Parses `organic` or `news` key in response
|
||||
- `count` capped at 10 for Serper (their limit per call)
|
||||
|
||||
---
|
||||
|
||||
#### Task 6 — Scraper Service ✅ (+ SSRF Fix)
|
||||
**Commits:** `5a13863..aaa7ffa`, then SSRF fix at `42216c4`
|
||||
**File:** `axon/services/scraper.py`
|
||||
|
||||
```python
|
||||
async def scrape_url(url: str, timeout: int = 5) -> str | None:
|
||||
# Validates URL first (SSRF check), then fetches, then extracts with trafilatura
|
||||
# Returns None on any failure
|
||||
|
||||
async def scrape_urls(
|
||||
urls: list[str],
|
||||
max_concurrent: int = 10,
|
||||
timeout: int = 5,
|
||||
) -> dict[str, str | None]:
|
||||
# Concurrent batch scrape with asyncio.Semaphore
|
||||
# Individual failures become None values in the dict
|
||||
```
|
||||
|
||||
**SSRF Protection** (added mid-build after automated security review flagged it):
|
||||
- `_is_safe_url(url)` runs before every fetch
|
||||
- Checks scheme is `http` or `https` only
|
||||
- Resolves hostname via `socket.getaddrinfo`, rejects any result that is:
|
||||
- Loopback (127.x.x.x, ::1)
|
||||
- Private (10.x, 172.16-31.x, 192.168.x)
|
||||
- Link-local (169.254.x.x — blocks AWS metadata at 169.254.169.254)
|
||||
- Multicast, reserved, or unspecified
|
||||
- `follow_redirects=False` (redirects to internal IPs are a common SSRF bypass)
|
||||
- trafilatura called via `asyncio.to_thread()` (CPU-bound extraction)
|
||||
|
||||
---
|
||||
|
||||
#### Task 7 — Reranker Service ✅
|
||||
**Commits:** `aaa7ffa..acc7a3c`
|
||||
**File:** `axon/services/reranker.py`
|
||||
|
||||
```python
|
||||
_model = None # loaded at startup, stays in memory
|
||||
|
||||
def load_reranker() -> None:
|
||||
# Loads BAAI/bge-reranker-v2-m3 CrossEncoder into _model
|
||||
# ~1GB download on first run, cached by HuggingFace after that
|
||||
|
||||
def get_reranker() -> "CrossEncoder | None":
|
||||
return _model
|
||||
|
||||
def chunk_text(text: str, max_tokens: int = 512) -> list[str]:
|
||||
# Splits into chunks of ~384 words (512 tokens × 0.75 words/token)
|
||||
# Empty string → [""]
|
||||
|
||||
async def rerank_results(
|
||||
query: str,
|
||||
contents: list[tuple[str, str]], # (url, text) pairs
|
||||
) -> list[tuple[str, str, float]]:
|
||||
# For each URL: chunks text, scores all chunks vs query
|
||||
# Picks best chunk per URL (argmax)
|
||||
# Sigmoid-normalizes raw CrossEncoder logits to [0, 1]
|
||||
# Returns sorted list of (url, best_chunk, score) — descending
|
||||
# model.predict() runs via asyncio.to_thread() (blocking/CPU)
|
||||
```
|
||||
|
||||
Score normalization: `sigmoid(x) = 1 / (1 + exp(-x))` — maps CrossEncoder logits to 0–1 range.
|
||||
|
||||
---
|
||||
|
||||
#### Task 8 — Synthesizer Service ✅
|
||||
**Commits:** `acc7a3c..6d9a057`
|
||||
**File:** `axon/services/synthesizer.py`
|
||||
|
||||
All LLM calls go to `settings.litellm_base_url + "/chat/completions"` (standard OpenAI format). `Authorization: Bearer <key>` header is only set if `settings.litellm_api_key` is non-empty.
|
||||
|
||||
System prompt used for all synthesis:
|
||||
> "You are a precise research assistant. Answer the user's query based only on the provided search results. Cite sources using [1], [2] notation. Be concise and factual."
|
||||
|
||||
```python
|
||||
async def synthesize_answer(query: str, results: list[SearchResult]) -> str:
|
||||
# Uses top 5 results as context
|
||||
# Returns "No search results available..." if results is empty
|
||||
# Returns "Answer synthesis failed: <error>" on LLM error
|
||||
|
||||
async def generate_sub_queries(query: str, count: int = 3) -> list[str]:
|
||||
# Asks LLM to output exactly `count` queries, one per line
|
||||
# On LLM error: returns [query, f"{query} overview", f"{query} recent developments"][:count]
|
||||
|
||||
async def synthesize_report(query, results, output_format="markdown") -> str:
|
||||
# Uses top 20 results as context
|
||||
# markdown format → instructs "## Summary, ## Key Findings, ## Conclusion"
|
||||
# json format → instructs keys: summary, key_findings, conclusions
|
||||
# Returns "Report synthesis failed: <error>" on LLM error
|
||||
# 60s timeout (longer than answer synthesis at 30s)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Task 9 — Auth Middleware ✅
|
||||
**Commits:** `6d9a057..bc49aa1`
|
||||
**File:** `axon/middleware/auth.py`
|
||||
|
||||
```python
|
||||
_EXEMPT = {"/health", "/docs", "/openapi.json", "/redoc"}
|
||||
|
||||
class APIKeyMiddleware(BaseHTTPMiddleware):
|
||||
# If settings.api_key_list is empty → pass everything through (dev mode)
|
||||
# If path in _EXEMPT → pass through regardless
|
||||
# Otherwise → require X-API-Key header matching one of the keys
|
||||
# Invalid/missing → 401 {"detail": "Invalid or missing API key"}
|
||||
```
|
||||
|
||||
`api_key_list` is a `@property` on `Settings` that computes from `settings.api_keys` (a raw comma-separated string). When monkeypatching in tests, patch `settings.api_keys` (the underlying field), not `api_key_list` (the property).
|
||||
|
||||
---
|
||||
|
||||
#### Task 10 — Search Router ✅ (commit done, review pending)
|
||||
**Commit:** `4193bba`
|
||||
**File:** `axon/routers/search.py`
|
||||
|
||||
Exports `deduplicate_results()` — used by the research router (Task 12).
|
||||
|
||||
Full search flow, in order:
|
||||
1. Build cache key from all request params → check cache; return immediately if hit
|
||||
2. `search_brave` + `search_ddg` via `asyncio.gather` (concurrent)
|
||||
3. If `len(brave_results) < 5`: also call `search_serper`
|
||||
4. Merge all: `brave + ddg + serper`
|
||||
5. If all empty: raise `HTTPException(503)`
|
||||
6. `deduplicate_results()` → keep first URL occurrence
|
||||
7. `_filter()` → apply `domains_whitelist` / `domains_blacklist`
|
||||
8. Slice to `max_results`
|
||||
9. If `search_depth == "advanced"`: `scrape_urls()` → replace result.content with scraped text where available
|
||||
10. `rerank_results(query, [(url, content), ...])` → updates scores
|
||||
11. Sort by score descending
|
||||
12. If not `include_raw_content`: truncate content to 500 chars
|
||||
13. If `include_answer`: `synthesize_answer()`
|
||||
14. Log: `query, source_count, scrape_count, [min_score, max_score], response_time`
|
||||
15. Store response in cache (TTL=600), return
|
||||
|
||||
---
|
||||
|
||||
### ❌ NOT DONE — Tasks 11–14
|
||||
|
||||
These four files do not exist yet:
|
||||
|
||||
---
|
||||
|
||||
#### Task 11 — Extract Router ❌
|
||||
**File to create:** `axon/routers/extract.py`
|
||||
**Endpoint:** `POST /extract`
|
||||
|
||||
What it needs to do:
|
||||
- Accept `ExtractRequest(urls: list[str], include_images: bool)`
|
||||
- For each URL: check cache (key=`"extract:<sha256 of url>"`, TTL=3600), fetch if miss
|
||||
- Fetch via httpx, extract text with `trafilatura.extract()` via `asyncio.to_thread()`
|
||||
- Also try `trafilatura.extract_metadata()` to get title
|
||||
- Individual URL failures: `ExtractResult(success=False, error="...")`
|
||||
- Return `ExtractResponse(results=[...])`
|
||||
- All URLs processed concurrently via `asyncio.gather`
|
||||
- Cache successful extractions for 1 hour
|
||||
|
||||
---
|
||||
|
||||
#### Task 12 — Research Router ❌
|
||||
**File to create:** `axon/routers/research.py`
|
||||
**Endpoint:** `POST /research`
|
||||
|
||||
What it needs to do:
|
||||
- Accept `ResearchRequest(query, depth, output_format)`
|
||||
- `depth="basic"` → 3 sub-queries; `depth="deep"` → 5 sub-queries
|
||||
- Call `generate_sub_queries(query, count)` from synthesizer
|
||||
- Run `search()` from `routers/search.py` for each sub-query (with `search_depth="advanced"`, `include_raw_content=True`)
|
||||
- Sub-query searches run concurrently via `asyncio.gather(..., return_exceptions=True)`
|
||||
- Failed sub-query searches are skipped (logged, not crashed)
|
||||
- Merge all results → `deduplicate_results()`
|
||||
- Call `synthesize_report(query, all_results, output_format)`
|
||||
- Return `ResearchResponse(query, report, sources, sub_queries, response_time)`
|
||||
|
||||
**Import note:** `from axon.routers.search import search, deduplicate_results` — calling the Python function directly, not making an HTTP request.
|
||||
|
||||
---
|
||||
|
||||
#### Task 13 — Main App + Health ❌
|
||||
**File to create:** `axon/main.py`
|
||||
|
||||
What it needs to do:
|
||||
```python
|
||||
from contextlib import asynccontextmanager
|
||||
from fastapi import FastAPI
|
||||
from fastapi.middleware.cors import CORSMiddleware
|
||||
|
||||
@asynccontextmanager
|
||||
async def lifespan(app):
|
||||
load_reranker() # loads BAAI/bge-reranker-v2-m3 once at startup
|
||||
yield
|
||||
|
||||
app = FastAPI(title="Axon Search API", version="1.0.0", lifespan=lifespan)
|
||||
|
||||
# CORS: allow_origins=["*"], allow_methods=["*"], allow_headers=["*"]
|
||||
# Auth: add APIKeyMiddleware
|
||||
# Routers: include search.router, extract.router, research.router
|
||||
|
||||
@app.get("/health")
|
||||
async def health():
|
||||
cache = await get_cache()
|
||||
return {
|
||||
"status": "ok",
|
||||
"redis": "connected" | "in-memory fallback",
|
||||
"reranker": "loaded" | "not loaded",
|
||||
}
|
||||
```
|
||||
|
||||
To run the server:
|
||||
```bash
|
||||
uvicorn axon.main:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
#### Task 14 — README ❌
|
||||
**File to create:** `README.md`
|
||||
|
||||
Must include: setup steps, env var table, curl examples for all 4 endpoints (search, extract, research, health), source fallback logic explanation, performance notes.
|
||||
|
||||
---
|
||||
|
||||
## File Tree — Current State
|
||||
|
||||
```
|
||||
/srv/Axon/
|
||||
├── .env.example
|
||||
├── .gitignore
|
||||
├── CLAUDE.md ← this file
|
||||
├── requirements.txt
|
||||
├── docs/
|
||||
│ └── superpowers/plans/
|
||||
│ └── 2026-06-18-axon-search-api.md
|
||||
├── axon/
|
||||
│ ├── __init__.py
|
||||
│ ├── config.py ✅
|
||||
│ ├── models.py ✅
|
||||
│ ├── main.py ❌ MISSING
|
||||
│ ├── middleware/
|
||||
│ │ ├── __init__.py
|
||||
│ │ └── auth.py ✅
|
||||
│ ├── routers/
|
||||
│ │ ├── __init__.py
|
||||
│ │ ├── search.py ✅
|
||||
│ │ ├── extract.py ❌ MISSING
|
||||
│ │ └── research.py ❌ MISSING
|
||||
│ └── services/
|
||||
│ ├── __init__.py
|
||||
│ ├── cache.py ✅
|
||||
│ ├── reranker.py ✅
|
||||
│ ├── scraper.py ✅ (includes SSRF protection)
|
||||
│ ├── synthesizer.py ✅
|
||||
│ └── sources/
|
||||
│ ├── __init__.py
|
||||
│ ├── brave.py ✅
|
||||
│ ├── ddg.py ✅
|
||||
│ └── serper.py ✅
|
||||
└── tests/
|
||||
├── __init__.py
|
||||
├── test_config.py ✅ 3 tests
|
||||
├── test_models.py ✅ 6 tests
|
||||
├── test_cache.py ✅ 7 tests
|
||||
├── test_sources.py ✅ 10 tests (brave + ddg + serper)
|
||||
├── test_scraper.py ✅ 4 tests
|
||||
├── test_reranker.py ✅ 7 tests
|
||||
├── test_synthesizer.py ✅ 4 tests
|
||||
├── test_auth.py ✅ 5 tests
|
||||
├── test_search_router.py ✅ 4 tests
|
||||
├── test_extract_router.py ❌ MISSING
|
||||
└── test_research_router.py ❌ MISSING (covered in test_search_router.py per plan)
|
||||
```
|
||||
|
||||
**Total tests passing right now: 50**
|
||||
|
||||
---
|
||||
|
||||
## Environment & Running Tests
|
||||
|
||||
```bash
|
||||
# The venv is at /srv/Axon/venv (Python 3.14)
|
||||
source /srv/Axon/venv/bin/activate
|
||||
|
||||
# Run all tests
|
||||
python -m pytest tests/ -v
|
||||
|
||||
# Run a specific file
|
||||
python -m pytest tests/test_search_router.py -v
|
||||
|
||||
# pytest-asyncio version: 1.4.0 (strict mode)
|
||||
# ALL async tests must be decorated with @pytest.mark.asyncio
|
||||
```
|
||||
|
||||
Known warnings (not errors):
|
||||
- Starlette currently emits a `TestClient` deprecation warning about its `httpx` integration.
|
||||
|
||||
---
|
||||
|
||||
## Environment Variables (from .env.example)
|
||||
|
||||
| Variable | Required | Default | Purpose |
|
||||
|---|---|---|---|
|
||||
| `BRAVE_API_KEY` | No | — | Enables Brave search source |
|
||||
| `SERPER_API_KEY` | No | — | Enables Serper fallback source |
|
||||
| `REDIS_URL` | No | `redis://localhost:6379` | Cache backend; omit for in-memory |
|
||||
| `API_KEYS` | No | — | Comma-separated auth keys; unset = no auth |
|
||||
| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible LLM endpoint |
|
||||
| `LITELLM_API_KEY` | No | — | Bearer token for LLM endpoint |
|
||||
| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model used for answer/report synthesis |
|
||||
| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | HuggingFace CrossEncoder model |
|
||||
| `PORT` | No | `8000` | uvicorn listen port |
|
||||
| `MAX_CONCURRENT_SCRAPES` | No | `10` | asyncio.Semaphore limit for scraping |
|
||||
| `SCRAPE_TIMEOUT` | No | `5` | Per-URL timeout in seconds |
|
||||
|
||||
---
|
||||
|
||||
## Coding Rules (enforced across all tasks)
|
||||
|
||||
1. **Python 3.11+ types only** — `str | None` not `Optional[str]`, `list[T]` not `List[T]`, `dict[K,V]` not `Dict[K,V]`. No imports from `typing` for these.
|
||||
2. **Pydantic v2** — `model_dump()` and `model_dump_json()` only. Never `.dict()` or `.json()`.
|
||||
3. **All I/O is async** — `httpx.AsyncClient`, `redis.asyncio`, `asyncio.to_thread()` for blocking calls.
|
||||
4. **Never raise from service functions** — sources, scraper, synthesizer all catch all exceptions and return empty/None/error strings.
|
||||
5. **Tests use `@pytest.mark.asyncio`** — no anyio markers.
|
||||
6. **No `Optional` imports** — this kept appearing in subagent output and was fixed each time.
|
||||
|
||||
---
|
||||
|
||||
## Gotchas & Lessons Learned During Build
|
||||
|
||||
- **`@pytest.mark.anyio` vs `asyncio`** — subagents kept using `anyio`. Every test file was checked for this. Always `@pytest.mark.asyncio`.
|
||||
- **`Optional[str]` vs `str | None`** — the plan's own code examples had `Optional` in them (for illustration). Subagents copied it. Had to fix in Tasks 3, 6.
|
||||
- **`importlib.reload` in tests** — one subagent used it to reset module state for monkeypatching. Fragile and banned. Replaced with direct `monkeypatch.setattr()`.
|
||||
- **SSRF** — automated security review flagged `scraper.py` after Task 6's commit. Fixed in Task 7's commit window by adding `_is_safe_url()` with full IP blocklist.
|
||||
- **Review of auth tests** — reviewer incorrectly flagged monkeypatching `settings.api_keys` as wrong (said it should patch `api_key_list`). But `api_key_list` is a `@property` — you can't patch it directly. Patching the underlying field is correct. Confirmed all 5 tests pass.
|
||||
- **Venv path** — the project uses `/srv/Axon/venv`. pytest-asyncio there is version 1.4.0. Must `source venv/bin/activate` before running any Python commands.
|
||||
|
||||
---
|
||||
|
||||
## Current Final State
|
||||
|
||||
Tasks 11-14 are now complete:
|
||||
1. `axon/routers/extract.py` implements `POST /extract` with per-URL caching, trafilatura extraction, metadata title extraction, concurrent processing, and SSRF protection via the scraper URL safety check.
|
||||
2. `axon/routers/research.py` implements `POST /research` by generating sub-queries, running advanced searches concurrently, skipping failed sub-searches, deduplicating sources, and synthesizing a final report.
|
||||
3. `axon/main.py` creates the runnable FastAPI app with CORS, API key middleware, all routers, lifespan reranker loading, and `GET /health`.
|
||||
4. `README.md` documents setup, configuration, endpoint curl examples, source fallback behavior, caching, and performance notes.
|
||||
|
||||
Additional cleanup:
|
||||
- `axon/config.py` and `axon/models.py` now use `str | None` style types instead of `Optional[...]`.
|
||||
- `axon/services/cache.py` now uses timezone-aware UTC datetimes, removing the Python 3.14 `datetime.utcnow()` warning.
|
||||
- `axon/services/sources/ddg.py` now prefers the current `ddgs` package and falls back to `duckduckgo_search`; `ddgs>=9.14.0` is in `requirements.txt`.
|
||||
- `axon/services/reranker.py` serializes model prediction and caps reranking to 4 chunks per document. This prevents `/research` from generating unbounded cross-encoder work on huge pages.
|
||||
- `axon/routers/research.py` runs sub-searches sequentially with 5 results per sub-query. This avoids OOM on the current server while preserving deep multi-query research behavior.
|
||||
- `axon/middleware/rate_limit.py` adds per-key/IP rate limiting.
|
||||
- `axon/middleware/request_context.py` adds `X-Request-ID` and records lightweight metrics.
|
||||
- `axon/routers/crawl.py` and `axon/routers/map.py` add Tavily-style bounded same-site crawl/map endpoints.
|
||||
- `axon/services/scraper.py` now has reusable safe fetch, extraction, URL normalization, and same-site link discovery helpers with retries.
|
||||
- `axon/services/index.py` adds a private SQLite FTS index; `POST /ingest`, `POST /index/search`, and normal `/search` can use private indexed content.
|
||||
- `axon/services/jobs.py` and `axon/routers/jobs.py` add in-process background jobs for research/crawl/map.
|
||||
- `axon/routers/tavily.py` adds `/tavily/search`, `/tavily/extract`, `/tavily/crawl`, and `/tavily/map` compatibility routes.
|
||||
- `axon/routers/validate.py` and citation helpers add basic citation coverage validation.
|
||||
- Browser extraction fallback is optional via `ENABLE_BROWSER_FALLBACK=true` if Playwright is installed.
|
||||
|
||||
Systemd deployment completed:
|
||||
- Unit file: `/etc/systemd/system/axon.service`
|
||||
- Enabled: yes
|
||||
- Runtime env file: `/srv/Axon/.env` (ignored by git, mode `600`)
|
||||
- LiteLLM model configured: `AR/openai/gpt-oss-120b:free`
|
||||
- Service command: `/srv/Axon/venv/bin/uvicorn axon.main:app --host 0.0.0.0 --port 8000`
|
||||
|
||||
Live endpoint verification completed through systemd:
|
||||
- `GET /health` -> 200, Redis connected, reranker loaded
|
||||
- `GET /openapi.json` -> 200, paths include `/extract`, `/health`, `/metrics`, `/crawl`, `/map`, `/research`, `/search`
|
||||
- `POST /extract` with `https://example.com` -> 200
|
||||
- `POST /search` basic with answer synthesis -> 200 using LiteLLM
|
||||
- `POST /search` advanced -> 200, completed in about 28 seconds
|
||||
- `POST /research` basic markdown -> 200, 15 sources, completed in about 163 seconds
|
||||
|
||||
Operational note:
|
||||
- The BGE reranker is memory-heavy. Before reranker chunk capping, `/research` OOM-killed the service at roughly 4.2 GB. After capping and sequential research searches, the tested `/research` run peaked around 2 GB and completed successfully.
|
||||
|
||||
The git branch is `master`. The project is runnable with:
|
||||
|
||||
```bash
|
||||
source /srv/Axon/venv/bin/activate
|
||||
uvicorn axon.main:app --host 0.0.0.0 --port 8000
|
||||
```
|
||||
|
||||
To run a full test suite health check right now:
|
||||
```bash
|
||||
source /srv/Axon/venv/bin/activate && python -m pytest tests/ -v
|
||||
```
|
||||
Expected: **61 passed**.
|
||||
Reference in New Issue
Block a user