arsam/Axon
1
0
Fork 0
Code Issues Pull Requests Packages Projects Releases Wiki Activity

Import documentation: README.md

Browse Source 
Import the existing local project snapshot. This commit organizes existing files; it does not represent newly developed functionality.
This commit is contained in:
arsam
2026-09-18 23:06:56 +03:30
parent f8453481d5
commit 872b407f1e
1 changed files with 330 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
README.md
+330
View File
@@ -0,0 +1,330 @@
# Axon Search API
Axon is a private, self-hosted AI search API for Linux servers. It aggregates search results from Brave Search, DuckDuckGo, and Serper.dev, optionally scrapes full page content, reranks results with a local BGE cross-encoder, and can synthesize answers or research reports through any OpenAI-compatible LLM endpoint.
## Features
- `POST /search` for multi-source search with optional scraping, reranking, and answer synthesis
- `POST /extract` for clean content extraction from one or more URLs
- `POST /research` for multi-query research reports
- `POST /crawl` for same-site traversal with extraction
- `POST /map` for same-site URL discovery
- `GET /health` for cache and reranker status
- `GET /metrics` for lightweight request counters and latency
- `POST /ingest` and `POST /index/search` for private local document retrieval
- `/research/jobs`, `/crawl/jobs`, `/map/jobs`, and `GET /jobs/{id}` for async long-running work
- `/tavily/*` compatibility routes for easier migration
- `POST /validate/citations` for basic citation coverage checks
- API key auth via `X-API-Key`
- Per-key/IP rate limiting
- Redis cache with in-memory fallback
- CORS enabled for all origins
- SSRF protection for URL scraping and extraction
## Setup
Axon requires Python 3.11 or newer.
```bash
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```
Edit `.env` with any API keys you want to enable. At least DuckDuckGo works without a key, but Brave and Serper improve coverage and reliability.
Start the server:
```bash
uvicorn axon.main:app --host 0.0.0.0 --port 8000
```
The reranker model is loaded at startup. The first run may download the configured HuggingFace model and can take longer than later starts.
## Configuration
| Variable | Required | Default | Purpose |
|---|---:|---|---|
| `BRAVE_API_KEY` | No | empty | Enables Brave Search results |
| `SERPER_API_KEY` | No | empty | Enables Serper.dev fallback results |
| `REDIS_URL` | No | `redis://localhost:6379` | Redis cache URL; falls back to memory if unavailable |
| `API_KEYS` | No | empty | Comma-separated accepted API keys; empty disables auth |
| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible chat completions base URL |
| `LITELLM_API_KEY` | No | empty | Bearer token for synthesis requests |
| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model sent to the LLM endpoint |
| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | Sentence Transformers cross-encoder model |
| `PORT` | No | `8000` | Preferred server port |
| `MAX_CONCURRENT_SCRAPES` | No | `10` | Concurrent scrape limit for advanced search |
| `SCRAPE_TIMEOUT` | No | `5` | Per-URL scrape timeout in seconds |
| `EXTRACT_TIMEOUT` | No | `10` | Per-URL extraction timeout in seconds |
| `RATE_LIMIT_PER_MINUTE` | No | `60` | Per API key or client IP limit; `0` disables |
| `CRAWL_MAX_PAGES` | No | `50` | Server-side cap for crawl/map pages |
| `CRAWL_MAX_DEPTH` | No | `3` | Server-side cap for crawl/map depth |
| `INDEX_DB_PATH` | No | `axon_index.sqlite3` | SQLite FTS database path for private indexed docs |
| `ENABLE_BROWSER_FALLBACK` | No | `false` | Enables optional Playwright rendering fallback when installed |
## Authentication
If `API_KEYS` is unset or empty, Axon runs in development mode and does not require authentication.
If `API_KEYS` is set, send one configured key with each protected request:
```bash
curl -H "X-API-Key: your-secret-key" http://localhost:8000/health
```
`/health`, `/docs`, `/openapi.json`, and `/redoc` are exempt.
## Endpoints
### Health
```bash
curl http://localhost:8000/health
```
Example response:
```json
{
"status": "ok",
"redis": "in-memory fallback",
"reranker": "loaded"
}
```
### Metrics
```bash
curl http://localhost:8000/metrics
```
Example response:
```json
{
"requests_total": 42,
"requests_by_path": {
"/search": 10,
"/extract": 3
},
"errors_total": 0,
"average_latency_ms": 1234.5
}
```
### Search
Basic search uses search API snippets and reranks them:
```bash
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"query": "latest PostgreSQL indexing improvements",
"max_results": 10,
"search_depth": "basic",
"include_answer": false
}'
```
Advanced search scrapes result pages before reranking:
```bash
curl -X POST http://localhost:8000/search \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"query": "private AI search APIs",
"max_results": 10,
"search_depth": "advanced",
"include_answer": true,
"include_raw_content": false,
"include_private": true,
"domains_blacklist": ["reddit.com"],
"topic": "general",
"time_range": "month"
}'
```
### Extract
```bash
curl -X POST http://localhost:8000/extract \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"urls": [
"https://example.com/article"
],
"include_images": false
}'
```
Each URL is processed independently. Failed URLs return `success: false` with an error while the rest of the request continues.
### Research
```bash
curl -X POST http://localhost:8000/research \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"query": "state of open source vector databases",
"depth": "deep",
"search_depth": "basic",
"max_results_per_query": 5,
"output_format": "markdown"
}'
```
`depth: "basic"` generates 3 sub-queries. `depth: "deep"` generates 5 sub-queries. Sources are deduplicated by URL, and the final report is synthesized from the combined results. Use `search_depth: "basic"` for faster research, or `advanced` when scraped page content is worth the extra latency.
For long-running work, create a background job:
```bash
curl -X POST http://localhost:8000/research/jobs \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{"query": "OpenAI", "search_depth": "basic"}'
curl http://localhost:8000/jobs/<job_id> \
-H "X-API-Key: your-secret-key"
```
### Crawl
```bash
curl -X POST http://localhost:8000/crawl \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"url": "https://docs.example.com",
"max_depth": 1,
"max_pages": 20,
"include_raw_content": false,
"instructions": "python sdk"
}'
```
`/crawl` stays on the same hostname as the start URL, extracts each matching page, and returns discovered links. It is intentionally bounded by `CRAWL_MAX_PAGES` and `CRAWL_MAX_DEPTH`.
### Map
```bash
curl -X POST http://localhost:8000/map \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"url": "https://docs.example.com",
"max_depth": 2,
"max_pages": 50,
"instructions": "api reference"
}'
```
`/map` returns same-site URLs without extracting full content.
### Private Index
Ingest private documents:
```bash
curl -X POST http://localhost:8000/ingest \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"documents": [
{
"id": "runbook-1",
"title": "Production Runbook",
"content": "Internal deployment and rollback notes.",
"url": "private://runbook-1"
}
]
}'
```
Search only the private index:
```bash
curl -X POST http://localhost:8000/index/search \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{"query": "rollback notes"}'
```
Normal `/search` includes private index hits by default. Set `"include_private": false` to use only web sources.
### Tavily-Compatible Routes
Axon exposes compatibility routes under `/tavily`:
```bash
curl -X POST http://localhost:8000/tavily/search \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{"query": "OpenAI", "max_results": 5}'
```
Available compatibility routes:
- `/tavily/search`
- `/tavily/extract`
- `/tavily/crawl`
- `/tavily/map`
### Citation Validation
```bash
curl -X POST http://localhost:8000/validate/citations \
-H "Content-Type: application/json" \
-H "X-API-Key: your-secret-key" \
-d '{
"report": "OpenAI was founded in 2015 [1].",
"sources": [
{"title": "Source", "url": "https://example.com", "content": "OpenAI was founded in 2015."}
]
}'
```
## Search Source Logic
Axon fetches Brave and DuckDuckGo concurrently. Serper.dev is called only when Brave returns fewer than 5 results. All results are merged and deduplicated by URL before filtering, scraping, reranking, and response truncation.
If all enabled sources return no results, `/search` returns HTTP 503. Individual source failures are logged and treated as empty result sets.
DuckDuckGo uses the current `ddgs` package when installed and falls back to `duckduckgo-search` for compatibility.
## Caching
- Search responses are cached for 10 minutes.
- Successful extracted URL content is cached for 1 hour.
- Redis is used when `REDIS_URL` is configured and reachable.
- If Redis is unavailable, Axon logs a warning and uses in-memory cache for the current process.
## Performance Notes
Advanced search is slower than basic search because it fetches and extracts each result page. Tune `MAX_CONCURRENT_SCRAPES` and `SCRAPE_TIMEOUT` for your server and network limits.
The reranker model is loaded once during FastAPI startup and reused for requests. Use a smaller cross-encoder model if startup time, memory use, or CPU latency is too high for your deployment.
Reranking is serialized through a process-local lock and capped to a small number of chunks per document. This keeps the self-hosted service stable on modest servers, at the cost of some recall on very long pages.
Optional browser rendering requires installing Playwright and browser assets separately, then setting `ENABLE_BROWSER_FALLBACK=true`.
## Development
Run tests:
```bash
source venv/bin/activate
python -m pytest tests/ -v
```
Run a focused test file:
```bash
python -m pytest tests/test_search_router.py -v
```