diff --git a/README.md b/README.md new file mode 100644 index 0000000..0a9f7b5 --- /dev/null +++ b/README.md @@ -0,0 +1,330 @@ +# Axon Search API + +Axon is a private, self-hosted AI search API for Linux servers. It aggregates search results from Brave Search, DuckDuckGo, and Serper.dev, optionally scrapes full page content, reranks results with a local BGE cross-encoder, and can synthesize answers or research reports through any OpenAI-compatible LLM endpoint. + +## Features + +- `POST /search` for multi-source search with optional scraping, reranking, and answer synthesis +- `POST /extract` for clean content extraction from one or more URLs +- `POST /research` for multi-query research reports +- `POST /crawl` for same-site traversal with extraction +- `POST /map` for same-site URL discovery +- `GET /health` for cache and reranker status +- `GET /metrics` for lightweight request counters and latency +- `POST /ingest` and `POST /index/search` for private local document retrieval +- `/research/jobs`, `/crawl/jobs`, `/map/jobs`, and `GET /jobs/{id}` for async long-running work +- `/tavily/*` compatibility routes for easier migration +- `POST /validate/citations` for basic citation coverage checks +- API key auth via `X-API-Key` +- Per-key/IP rate limiting +- Redis cache with in-memory fallback +- CORS enabled for all origins +- SSRF protection for URL scraping and extraction + +## Setup + +Axon requires Python 3.11 or newer. + +```bash +python -m venv venv +source venv/bin/activate +pip install -r requirements.txt +cp .env.example .env +``` + +Edit `.env` with any API keys you want to enable. At least DuckDuckGo works without a key, but Brave and Serper improve coverage and reliability. + +Start the server: + +```bash +uvicorn axon.main:app --host 0.0.0.0 --port 8000 +``` + +The reranker model is loaded at startup. The first run may download the configured HuggingFace model and can take longer than later starts. + +## Configuration + +| Variable | Required | Default | Purpose | +|---|---:|---|---| +| `BRAVE_API_KEY` | No | empty | Enables Brave Search results | +| `SERPER_API_KEY` | No | empty | Enables Serper.dev fallback results | +| `REDIS_URL` | No | `redis://localhost:6379` | Redis cache URL; falls back to memory if unavailable | +| `API_KEYS` | No | empty | Comma-separated accepted API keys; empty disables auth | +| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible chat completions base URL | +| `LITELLM_API_KEY` | No | empty | Bearer token for synthesis requests | +| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model sent to the LLM endpoint | +| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | Sentence Transformers cross-encoder model | +| `PORT` | No | `8000` | Preferred server port | +| `MAX_CONCURRENT_SCRAPES` | No | `10` | Concurrent scrape limit for advanced search | +| `SCRAPE_TIMEOUT` | No | `5` | Per-URL scrape timeout in seconds | +| `EXTRACT_TIMEOUT` | No | `10` | Per-URL extraction timeout in seconds | +| `RATE_LIMIT_PER_MINUTE` | No | `60` | Per API key or client IP limit; `0` disables | +| `CRAWL_MAX_PAGES` | No | `50` | Server-side cap for crawl/map pages | +| `CRAWL_MAX_DEPTH` | No | `3` | Server-side cap for crawl/map depth | +| `INDEX_DB_PATH` | No | `axon_index.sqlite3` | SQLite FTS database path for private indexed docs | +| `ENABLE_BROWSER_FALLBACK` | No | `false` | Enables optional Playwright rendering fallback when installed | + +## Authentication + +If `API_KEYS` is unset or empty, Axon runs in development mode and does not require authentication. + +If `API_KEYS` is set, send one configured key with each protected request: + +```bash +curl -H "X-API-Key: your-secret-key" http://localhost:8000/health +``` + +`/health`, `/docs`, `/openapi.json`, and `/redoc` are exempt. + +## Endpoints + +### Health + +```bash +curl http://localhost:8000/health +``` + +Example response: + +```json +{ + "status": "ok", + "redis": "in-memory fallback", + "reranker": "loaded" +} +``` + +### Metrics + +```bash +curl http://localhost:8000/metrics +``` + +Example response: + +```json +{ + "requests_total": 42, + "requests_by_path": { + "/search": 10, + "/extract": 3 + }, + "errors_total": 0, + "average_latency_ms": 1234.5 +} +``` + +### Search + +Basic search uses search API snippets and reranks them: + +```bash +curl -X POST http://localhost:8000/search \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "query": "latest PostgreSQL indexing improvements", + "max_results": 10, + "search_depth": "basic", + "include_answer": false + }' +``` + +Advanced search scrapes result pages before reranking: + +```bash +curl -X POST http://localhost:8000/search \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "query": "private AI search APIs", + "max_results": 10, + "search_depth": "advanced", + "include_answer": true, + "include_raw_content": false, + "include_private": true, + "domains_blacklist": ["reddit.com"], + "topic": "general", + "time_range": "month" + }' +``` + +### Extract + +```bash +curl -X POST http://localhost:8000/extract \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "urls": [ + "https://example.com/article" + ], + "include_images": false + }' +``` + +Each URL is processed independently. Failed URLs return `success: false` with an error while the rest of the request continues. + +### Research + +```bash +curl -X POST http://localhost:8000/research \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "query": "state of open source vector databases", + "depth": "deep", + "search_depth": "basic", + "max_results_per_query": 5, + "output_format": "markdown" + }' +``` + +`depth: "basic"` generates 3 sub-queries. `depth: "deep"` generates 5 sub-queries. Sources are deduplicated by URL, and the final report is synthesized from the combined results. Use `search_depth: "basic"` for faster research, or `advanced` when scraped page content is worth the extra latency. + +For long-running work, create a background job: + +```bash +curl -X POST http://localhost:8000/research/jobs \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{"query": "OpenAI", "search_depth": "basic"}' + +curl http://localhost:8000/jobs/ \ + -H "X-API-Key: your-secret-key" +``` + +### Crawl + +```bash +curl -X POST http://localhost:8000/crawl \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "url": "https://docs.example.com", + "max_depth": 1, + "max_pages": 20, + "include_raw_content": false, + "instructions": "python sdk" + }' +``` + +`/crawl` stays on the same hostname as the start URL, extracts each matching page, and returns discovered links. It is intentionally bounded by `CRAWL_MAX_PAGES` and `CRAWL_MAX_DEPTH`. + +### Map + +```bash +curl -X POST http://localhost:8000/map \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "url": "https://docs.example.com", + "max_depth": 2, + "max_pages": 50, + "instructions": "api reference" + }' +``` + +`/map` returns same-site URLs without extracting full content. + +### Private Index + +Ingest private documents: + +```bash +curl -X POST http://localhost:8000/ingest \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "documents": [ + { + "id": "runbook-1", + "title": "Production Runbook", + "content": "Internal deployment and rollback notes.", + "url": "private://runbook-1" + } + ] + }' +``` + +Search only the private index: + +```bash +curl -X POST http://localhost:8000/index/search \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{"query": "rollback notes"}' +``` + +Normal `/search` includes private index hits by default. Set `"include_private": false` to use only web sources. + +### Tavily-Compatible Routes + +Axon exposes compatibility routes under `/tavily`: + +```bash +curl -X POST http://localhost:8000/tavily/search \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{"query": "OpenAI", "max_results": 5}' +``` + +Available compatibility routes: +- `/tavily/search` +- `/tavily/extract` +- `/tavily/crawl` +- `/tavily/map` + +### Citation Validation + +```bash +curl -X POST http://localhost:8000/validate/citations \ + -H "Content-Type: application/json" \ + -H "X-API-Key: your-secret-key" \ + -d '{ + "report": "OpenAI was founded in 2015 [1].", + "sources": [ + {"title": "Source", "url": "https://example.com", "content": "OpenAI was founded in 2015."} + ] + }' +``` + +## Search Source Logic + +Axon fetches Brave and DuckDuckGo concurrently. Serper.dev is called only when Brave returns fewer than 5 results. All results are merged and deduplicated by URL before filtering, scraping, reranking, and response truncation. + +If all enabled sources return no results, `/search` returns HTTP 503. Individual source failures are logged and treated as empty result sets. + +DuckDuckGo uses the current `ddgs` package when installed and falls back to `duckduckgo-search` for compatibility. + +## Caching + +- Search responses are cached for 10 minutes. +- Successful extracted URL content is cached for 1 hour. +- Redis is used when `REDIS_URL` is configured and reachable. +- If Redis is unavailable, Axon logs a warning and uses in-memory cache for the current process. + +## Performance Notes + +Advanced search is slower than basic search because it fetches and extracts each result page. Tune `MAX_CONCURRENT_SCRAPES` and `SCRAPE_TIMEOUT` for your server and network limits. + +The reranker model is loaded once during FastAPI startup and reused for requests. Use a smaller cross-encoder model if startup time, memory use, or CPU latency is too high for your deployment. + +Reranking is serialized through a process-local lock and capped to a small number of chunks per document. This keeps the self-hosted service stable on modest servers, at the cost of some recall on very long pages. + +Optional browser rendering requires installing Playwright and browser assets separately, then setting `ENABLE_BROWSER_FALLBACK=true`. + +## Development + +Run tests: + +```bash +source venv/bin/activate +python -m pytest tests/ -v +``` + +Run a focused test file: + +```bash +python -m pytest tests/test_search_router.py -v +```