Axon/README.md

# Axon Search API

Axon is a private, self-hosted AI search API for Linux servers. It aggregates search results from Brave Search, DuckDuckGo, and Serper.dev, optionally scrapes full page content, reranks results with a local BGE cross-encoder, and can synthesize answers or research reports through any OpenAI-compatible LLM endpoint.

## Features

- `POST /search` for multi-source search with optional scraping, reranking, and answer synthesis
- `POST /extract` for clean content extraction from one or more URLs
- `POST /research` for multi-query research reports
- `POST /crawl` for same-site traversal with extraction
- `POST /map` for same-site URL discovery
- `GET /health` for cache and reranker status
- `GET /metrics` for lightweight request counters and latency
- `POST /ingest` and `POST /index/search` for private local document retrieval
- `/research/jobs`, `/crawl/jobs`, `/map/jobs`, and `GET /jobs/{id}` for async long-running work
- `/tavily/*` compatibility routes for easier migration
- `POST /validate/citations` for basic citation coverage checks
- API key auth via `X-API-Key`
- Per-key/IP rate limiting
- Redis cache with in-memory fallback
- CORS enabled for all origins
- SSRF protection for URL scraping and extraction

## Setup

Axon requires Python 3.11 or newer.

```bash
python -m venv venv
source venv/bin/activate
pip install -r requirements.txt
cp .env.example .env
```

Edit `.env` with any API keys you want to enable. At least DuckDuckGo works without a key, but Brave and Serper improve coverage and reliability.

Start the server:

```bash
uvicorn axon.main:app --host 0.0.0.0 --port 8000
```

The reranker model is loaded at startup. The first run may download the configured HuggingFace model and can take longer than later starts.

## Configuration

| Variable | Required | Default | Purpose |
|---|---:|---|---|
| `BRAVE_API_KEY` | No | empty | Enables Brave Search results |
| `SERPER_API_KEY` | No | empty | Enables Serper.dev fallback results |
| `REDIS_URL` | No | `redis://localhost:6379` | Redis cache URL; falls back to memory if unavailable |
| `API_KEYS` | No | empty | Comma-separated accepted API keys; empty disables auth |
| `LITELLM_BASE_URL` | No | `http://localhost:4000` | OpenAI-compatible chat completions base URL |
| `LITELLM_API_KEY` | No | empty | Bearer token for synthesis requests |
| `SYNTHESIS_MODEL` | No | `claude-sonnet-4-6` | Model sent to the LLM endpoint |
| `RERANKER_MODEL` | No | `BAAI/bge-reranker-v2-m3` | Sentence Transformers cross-encoder model |
| `PORT` | No | `8000` | Preferred server port |
| `MAX_CONCURRENT_SCRAPES` | No | `10` | Concurrent scrape limit for advanced search |
| `SCRAPE_TIMEOUT` | No | `5` | Per-URL scrape timeout in seconds |
| `EXTRACT_TIMEOUT` | No | `10` | Per-URL extraction timeout in seconds |
| `RATE_LIMIT_PER_MINUTE` | No | `60` | Per API key or client IP limit; `0` disables |
| `CRAWL_MAX_PAGES` | No | `50` | Server-side cap for crawl/map pages |
| `CRAWL_MAX_DEPTH` | No | `3` | Server-side cap for crawl/map depth |
| `INDEX_DB_PATH` | No | `axon_index.sqlite3` | SQLite FTS database path for private indexed docs |
| `ENABLE_BROWSER_FALLBACK` | No | `false` | Enables optional Playwright rendering fallback when installed |

## Authentication

If `API_KEYS` is unset or empty, Axon runs in development mode and does not require authentication.

If `API_KEYS` is set, send one configured key with each protected request:

```bash
curl -H "X-API-Key: your-secret-key" http://localhost:8000/health
```

`/health`, `/docs`, `/openapi.json`, and `/redoc` are exempt.

## Endpoints

### Health

```bash
curl http://localhost:8000/health
```

Example response:

```json
{
  "status": "ok",
  "redis": "in-memory fallback",
  "reranker": "loaded"
}
```

### Metrics

```bash
curl http://localhost:8000/metrics
```

Example response:

```json
{
  "requests_total": 42,
  "requests_by_path": {
    "/search": 10,
    "/extract": 3
  },
  "errors_total": 0,
  "average_latency_ms": 1234.5
}
```

### Search

Basic search uses search API snippets and reranks them:

```bash
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "query": "latest PostgreSQL indexing improvements",
    "max_results": 10,
    "search_depth": "basic",
    "include_answer": false
  }'
```

Advanced search scrapes result pages before reranking:

```bash
curl -X POST http://localhost:8000/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "query": "private AI search APIs",
    "max_results": 10,
    "search_depth": "advanced",
    "include_answer": true,
    "include_raw_content": false,
    "include_private": true,
    "domains_blacklist": ["reddit.com"],
    "topic": "general",
    "time_range": "month"
  }'
```

### Extract

```bash
curl -X POST http://localhost:8000/extract \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "urls": [
      "https://example.com/article"
    ],
    "include_images": false
  }'
```

Each URL is processed independently. Failed URLs return `success: false` with an error while the rest of the request continues.

### Research

```bash
curl -X POST http://localhost:8000/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "query": "state of open source vector databases",
    "depth": "deep",
    "search_depth": "basic",
    "max_results_per_query": 5,
    "output_format": "markdown"
  }'
```

`depth: "basic"` generates 3 sub-queries. `depth: "deep"` generates 5 sub-queries. Sources are deduplicated by URL, and the final report is synthesized from the combined results. Use `search_depth: "basic"` for faster research, or `advanced` when scraped page content is worth the extra latency.

For long-running work, create a background job:

```bash
curl -X POST http://localhost:8000/research/jobs \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{"query": "OpenAI", "search_depth": "basic"}'

curl http://localhost:8000/jobs/<job_id> \
  -H "X-API-Key: your-secret-key"
```

### Crawl

```bash
curl -X POST http://localhost:8000/crawl \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 1,
    "max_pages": 20,
    "include_raw_content": false,
    "instructions": "python sdk"
  }'
```

`/crawl` stays on the same hostname as the start URL, extracts each matching page, and returns discovered links. It is intentionally bounded by `CRAWL_MAX_PAGES` and `CRAWL_MAX_DEPTH`.

### Map

```bash
curl -X POST http://localhost:8000/map \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "url": "https://docs.example.com",
    "max_depth": 2,
    "max_pages": 50,
    "instructions": "api reference"
  }'
```

`/map` returns same-site URLs without extracting full content.

### Private Index

Ingest private documents:

```bash
curl -X POST http://localhost:8000/ingest \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "documents": [
      {
        "id": "runbook-1",
        "title": "Production Runbook",
        "content": "Internal deployment and rollback notes.",
        "url": "private://runbook-1"
      }
    ]
  }'
```

Search only the private index:

```bash
curl -X POST http://localhost:8000/index/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{"query": "rollback notes"}'
```

Normal `/search` includes private index hits by default. Set `"include_private": false` to use only web sources.

### Tavily-Compatible Routes

Axon exposes compatibility routes under `/tavily`:

```bash
curl -X POST http://localhost:8000/tavily/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{"query": "OpenAI", "max_results": 5}'
```

Available compatibility routes:
- `/tavily/search`
- `/tavily/extract`
- `/tavily/crawl`
- `/tavily/map`

### Citation Validation

```bash
curl -X POST http://localhost:8000/validate/citations \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-secret-key" \
  -d '{
    "report": "OpenAI was founded in 2015 [1].",
    "sources": [
      {"title": "Source", "url": "https://example.com", "content": "OpenAI was founded in 2015."}
    ]
  }'
```

## Search Source Logic

Axon fetches Brave and DuckDuckGo concurrently. Serper.dev is called only when Brave returns fewer than 5 results. All results are merged and deduplicated by URL before filtering, scraping, reranking, and response truncation.

If all enabled sources return no results, `/search` returns HTTP 503. Individual source failures are logged and treated as empty result sets.

DuckDuckGo uses the current `ddgs` package when installed and falls back to `duckduckgo-search` for compatibility.

## Caching

- Search responses are cached for 10 minutes.
- Successful extracted URL content is cached for 1 hour.
- Redis is used when `REDIS_URL` is configured and reachable.
- If Redis is unavailable, Axon logs a warning and uses in-memory cache for the current process.

## Performance Notes

Advanced search is slower than basic search because it fetches and extracts each result page. Tune `MAX_CONCURRENT_SCRAPES` and `SCRAPE_TIMEOUT` for your server and network limits.

The reranker model is loaded once during FastAPI startup and reused for requests. Use a smaller cross-encoder model if startup time, memory use, or CPU latency is too high for your deployment.

Reranking is serialized through a process-local lock and capped to a small number of chunks per document. This keeps the self-hosted service stable on modest servers, at the cost of some recall on very long pages.

Optional browser rendering requires installing Playwright and browser assets separately, then setting `ENABLE_BROWSER_FALLBACK=true`.

## Development

Run tests:

```bash
source venv/bin/activate
python -m pytest tests/ -v
```

Run a focused test file:

```bash
python -m pytest tests/test_search_router.py -v
```