teknav-bak/app/deep-research-report.md

# TekRAG-Persian Research Report

## Executive Summary

**TekRAG-Persian** is a strong, defensible article topic because it sits at the intersection of four active research threads that already have solid methodological foundations but have not yet clearly converged on a Persian **technical** QA benchmark: retrieval-augmented generation, hallucination measurement, data-centric AI, and Persian-language evaluation. The internal publication plan you already have also points in exactly this direction, explicitly framing the strongest manuscript as a data-centric Persian technical RAG benchmark that compares raw, cleaned, and metadata-enriched corpora. fileciteturn0file0

The most credible novelty claim is **not** “Persian QA does not exist.” Persian benchmarks do exist: ParsiNLU covers multiple NLU tasks; PQuAD is a large Persian reading-comprehension resource; PerCQA covers community QA; PCoQA covers conversational QA; PARSE introduces Persian reasoning QA; and later work expands Persian LLM evaluation into cultural and benchmark-oriented settings. The safer and stronger claim is narrower: **there is still no widely established, openly described benchmark for Persian technical, attributed, hallucination-sensitive RAG over a curated technology-news/analysis corpus**. That is the gap TekRAG-Persian should fill. citeturn6search3turn9search0turn9search1turn9search5turn17search9turn18search5

The empirical centerpiece should be a **data-centric ablation**. Instead of making the paper about inventing a new model, make it about showing what changes when the same Persian technical corpus is indexed in three or four variants: **raw**, **cleaned**, **metadata-enriched**, and optionally **provenance-constrained**. That aligns with the original RAG formulation, with retrieved-evidence citation work, with hallucination-measurement literature, and with data-centric AI’s core claim that better data engineering often produces larger gains than model-only tinkering. citeturn6search0turn8search1turn7search0turn7search2turn0academia14turn0academia15

Public search snippets from the live Teknav site already suggest the right corpus character for such a benchmark: long-form Persian articles in categories such as AI, data science, cybersecurity, hardware, software, and startup/strategy coverage, plus heavy use of mixed Persian-English technical terms such as “MCP,” “RAG,” and “AI-Native IDEs.” That combination is exactly the kind of corpus where normalization, code-switch-aware retrieval, heading preservation, and citation-grounded generation should matter. citeturn3search2turn3search3turn3search5turn3search6

For a publishable article, the safest design is a **benchmark-and-evaluation paper** with three contributions: a curated Persian technical QA dataset; a reproducible RAG evaluation protocol comparing sparse, dense, hybrid, and metadata-aware retrieval; and a hallucination/error-analysis framework tailored to Persian technical content, including ZWNJ issues, script normalization problems, transliteration drift, and mixed-script terminology. The evaluation should combine retrieval metrics, answer correctness, citation quality, atomic factual support, abstention behavior, calibration, and human judgment. citeturn19search1turn6search1turn13search2turn13search7turn7search0turn8search1turn7search1turn25view0turn27search10turn28search1

If you keep the first manuscript focused on **data quality effects** rather than full model invention, the paper becomes much easier to execute rigorously. A very defensible thesis statement is: **for Persian technical QA, hallucination is often as much a corpus-quality problem as a model-quality problem**. citeturn0academia14turn0search10turn6search0turn8search8

## Manuscript Blueprint

The outline below is optimized for a **methods/results-heavy 10,000-word article**, not for a trend essay. It assumes that you want a paper with serious experimental content, enough literature grounding for peer review, and enough narrative structure to convert later into a journal manuscript. That emphasis matches the internal Teknav plan far better than a general AI commentary piece. fileciteturn0file0

### Ten-thousand-word outline

| Manuscript section | Target words | What this section must accomplish | Suggested citations per paragraph |
|---|---:|---|---:|
| Abstract | 200 | State the problem, gap, corpus, benchmark design, systems compared, and main empirical question | 0–1 |
| Introduction | 1,200 | Motivate RAG, explain why hallucination matters in technical QA, establish Persian-specific gap, state contributions and research questions | 1–2 |
| Related Work | 1,500 | Cover RAG, dense/sparse retrieval, hallucination/factuality, attributed QA, data-centric AI, Persian NLP benchmarks, and benchmark gap | 2–4 |
| Corpus and Dataset Construction | 1,600 | Describe Teknav corpus scope, known vs unknown corpus stats, schema, cleaning, normalization, chunking, question authoring, and release policy | 1–3 |
| Methods | 1,850 | Present corpus variants, retrieval baselines, chunking strategies, prompting, RAG architectures, generators, and fine-tuning strategy | 1–2 |
| Evaluation | 1,450 | Define tasks, splits, metrics, annotation rules, human evaluation, statistical tests, confidence/calibration, and reproducibility | 1–3 |
| Results and Discussion | 1,200 | Report primary comparisons, ablations, practical trade-offs, interpretation, failure cases, and benchmark lessons | 1–2 |
| Ethics, Limitations, and Future Work | 600 | Address copyright, model drift, benchmark leakage, linguistic bias, domain limits, and roadmap extensions | 1–2 |
| Conclusion | 400 | Re-state contribution, empirical lesson, and value for future Persian RAG work | 1 |
| **Total** | **10,000** |  |  |

### Copy-ready Markdown skeleton

```markdown
# TekRAG-Persian: A Data-Centric Benchmark for Reducing Hallucination in Persian Technical Question Answering

## Abstract

## Introduction
### Why technical Persian QA is a distinct problem
### Why hallucination is a data and retrieval problem
### Why current Persian benchmarks are insufficient
### Research questions
### Contributions

## Related Work
### Retrieval-augmented generation
### Sparse, dense, and hybrid retrieval
### Hallucination, factuality, and attribution
### Data-centric AI
### Persian language resources and benchmarks
### Gap summary

## Corpus and Dataset Construction
### Teknav corpus scope and source profile
### Dataset schema
### Publicly known corpus characteristics
### Unspecified corpus statistics to report
### Cleaning and normalization
### Metadata enrichment
### Chunking
### Question authoring and evidence mapping
### Dataset versions

## Methods
### Retrieval baselines
### Metadata-aware retrieval
### Chunking variants
### Generator models
### Prompting strategies
### Provenance-aware RAG
### Fine-tuning versus retrieval-only
### Implementation details

## Evaluation
### Tasks
### Splits
### Question types and difficulty levels
### Automatic metrics
### Human evaluation
### Citation and grounding evaluation
### Calibration and abstention
### Statistical testing
### Reproducibility checklist

## Results
### Main benchmark results
### Retrieval ablations
### Chunking ablations
### Model comparison
### Cost-latency-quality trade-offs

## Error Analysis and Discussion
### Hallucination taxonomy
### Persian-specific failure modes
### Why metadata matters
### Where retrieval still fails
### Implications for low-resource technical RAG

## Ethics, Limitations, and Future Work
### Licensing and data release constraints
### Benchmark leakage and model contamination
### Annotation subjectivity
### Domain transfer limitations
### Future benchmark extensions

## Conclusion

## Appendices
### Annotation guidelines
### Sample QA pairs
### Prompt templates
### Hyperparameter grids
### Evaluation scripts
### Additional tables and figures
```

### Recommended abstract

This abstract is written to be **publishable in tone** while avoiding fabricated numerical results. It is framed around established strands in RAG, citation-grounded QA, factuality evaluation, and Persian benchmark design. citeturn6search0turn8search1turn7search0turn17search9

> Retrieval-augmented generation has become a standard approach for grounding large language models in external knowledge, yet its reliability in low-resource and mixed-script settings remains underexplored. This paper introduces **TekRAG-Persian**, a benchmark for Persian technical question answering designed to measure how corpus quality affects hallucination, grounding, and answer quality. The benchmark is built from a curated Persian technology corpus and evaluates question answering over multiple corpus variants, including raw, cleaned, and metadata-enriched representations. We compare sparse, dense, hybrid, and metadata-aware retrieval methods, together with retrieval-only and provenance-aware generation settings, across a diverse set of Persian technical questions covering definitions, comparisons, procedures, troubleshooting, and multi-hop synthesis. The evaluation protocol combines retrieval metrics with answer correctness, citation quality, atomic factual support, abstention behavior, and human judgments. The study is designed to test a data-centric hypothesis: that improvements in normalization, document structure preservation, metadata enrichment, and source attribution can substantially reduce unsupported claims in Persian technical QA, even without introducing a new base model. TekRAG-Persian aims to provide a reproducible benchmark, an error taxonomy tailored to Persian technical content, and practical guidance for building more trustworthy RAG systems in low-resource languages.

### Suggested keywords

**Persian NLP; Retrieval-Augmented Generation; Hallucination; Technical Question Answering; Data-Centric AI; Low-Resource Languages; Attribution; Grounded Generation; Information Retrieval; Citation Accuracy; Persian Text Normalization; Benchmarking**

## Literature Review and Research Gap

The paper should anchor itself in four literature blocks. The first is the RAG and retrieval line: the original RAG paper formalized retrieve-then-generate knowledge access; DPR established dense passage retrieval as a strong neural baseline; BM25 remains the indispensable sparse baseline; BM25F provides a principled way to weight structured document fields; ColBERTv2 represents a stronger late-interaction retriever; multilingual-E5 and BGE-M3 are especially relevant because they make multilingual dense retrieval practical without training a Persian-only encoder from scratch. citeturn6search0turn6search1turn19search1turn19search11turn13search12turn13search7turn13search2

The second block is factuality and attribution. Hallucination surveys now distinguish intrinsic versus extrinsic errors, and faithful versus factually correct generation. For benchmark design, the most useful evaluation papers are **FActScore** for atomic factual support, **SAFE** for search-augmented fact verification, **ALCE** for automatic answer-and-citation evaluation, **Attributed QA** for reproducible attribution-focused QA, and **RAGAS** for reference-free RAG diagnostics. These give you both terminology and concrete metric ideas. citeturn29search0turn29search1turn7search0turn7search2turn8search1turn8search8turn7search1

The third block is data-centric AI. The key framing move in your article should be that hallucination is not only a model-generation problem; it is also a **data representation, document engineering, and retrieval design** problem. Data-centric AI surveys and principle papers make that framing academically legitimate. citeturn0academia14turn0academia15turn0search10

The fourth block is Persian-specific work. Persian now has more resources than many reviewers may assume: ParsiNLU, ParsBERT, PQuAD, PerCQA, PCoQA, PersianLLaMA, FarsInstruct, PARSE, and newer Persian LLM evaluation work all matter. But these resources mostly target broad NLU, Wikipedia-style reading comprehension, conversational QA, community QA, instruction tuning, reasoning, or cultural evaluation. That leaves a gap for a **technical-domain, evidence-attributed, hallucination-aware, RAG-oriented benchmark** over Persian editorial content. Frame the novelty there, and do **not** claim absolute firstness unless you complete a much broader audit. citeturn6search2turn6search3turn9search0turn9search1turn9search5turn18search1turn18search0turn17search9turn18search5

### Primary-source reading list

| Cluster | Source | Why it matters |
|---|---|---|
| Foundational RAG | urlRetrieval-Augmented Generationturn6search0 | Original retrieve-then-generate formulation |
| Dense retrieval | urlDense Passage Retrievalturn6search5 | Canonical neural retriever baseline |
| Sparse retrieval | urlOkapi at TREC-3turn19search1 | Historical BM25 foundation |
| Structured sparse retrieval | urlSimple BM25 extension to multiple weighted fieldsturn19search11 | Metadata-aware BM25F rationale |
| Late interaction retrieval | urlColBERTv2turn13search12 | Stronger token-level retriever |
| Multilingual dense retrieval | urlMultilingual E5 technical reportturn13search7 | Strong multilingual dense baseline |
| Unified multilingual retrieval | urlBGE-M3turn13search2 | Dense, sparse, and multi-vector options in one encoder |
| Hallucination taxonomy | urlSurvey of Hallucination in NLGturn29search0 | Background terms and taxonomy |
| LLM hallucination taxonomy | urlSurvey on Hallucination in LLMsturn29search1 | LLM-specific causes and taxonomy |
| Atomic factuality | urlFActScoreturn7search3 | Atomic fact support metric |
| Search-augmented factuality | urlSAFE and LongFactturn7search2 | Automated long-form factuality evaluation |
| Citation evaluation | urlALCEturn8search1 | Automatic citation-quality benchmark |
| Attributed QA | urlAttributed Question Answeringturn8search8 | Reproducible citation-grounded QA framing |
| RAG diagnostics | urlRAGASturn7search10 | Reference-free RAG metrics |
| Data-centric framing | urlData-centric AI surveyturn0academia14 | Data-quality-first framing |
| Data-centric principles | urlPrinciples of Data-Centric AIturn0academia15 | Conceptual framing for corpus engineering |
| Persian encoder | urlParsBERT paperturn6search2 | Monolingual Persian representation baseline |
| Persian benchmark suite | urlParsiNLUturn6search15 | Broad Persian benchmark context |
| Persian reading comprehension | urlPQuADturn9search4 | Large Persian QA benchmark |
| Persian community QA | urlPerCQAturn9search8 | Answer ranking and community QA context |
| Persian conversational QA | urlPCoQAturn9search5 | Multi-turn Persian QA context |
| Persian reasoning QA | urlPARSEturn17search9 | New Persian reasoning benchmark |
| Persian LLM evaluation | urlAdvancing Persian LLM Evaluationturn18search5 | Broader Persian evaluation context |
| Persian instruction data | urlFarsInstructturn18search0 | Instruction-tuning option for ablations |
| Persian LLM baseline | urlPersianLLaMAturn18search1 | Monolingual generation baseline |
| Persian preprocessing toolkit | urlHazm docsturn9search2 | Practical normalization and tokenization |
| ZWNJ and segmentation | urlJoint Persian Word Segmentation and ZWNJ Recognitionturn27search10 | Persian-specific segmentation challenge |
| Long-context caution | urlLost in the Middleturn11search2 | Why chunking and context packing matter |
| Late chunking | urlLate Chunkingturn23search1 | Context-preserving chunk embeddings |

The dating, scope, and relevance of the sources above come directly from the linked primary papers, official docs, or official model pages. citeturn6search0turn6search5turn19search1turn19search11turn13search12turn13search7turn13search2turn29search0turn29search1turn7search3turn7search2turn8search1turn8search8turn7search10turn0academia14turn0academia15turn6search2turn6search15turn9search4turn9search8turn9search5turn17search9turn18search5turn18search0turn18search1turn9search2turn27search10turn11search2turn23search1

## Dataset and Data-Centric Pipeline

### What is known now about the Teknav corpus

From the public web, you can already justify the corpus as a **Persian technical editorial knowledge base**. Search snippets show article-style pages under `/article/...`, recent publication dates, and topical coverage including AI, data science, cybersecurity, hardware, software, and startup/strategy themes. Public snippets also show a mixed Persian-English technical register, such as “MCP,” “AI-Native IDEs,” and other untranslated jargon embedded inside Persian prose. That is exactly the profile you want for a technical RAG benchmark. citeturn3search2turn3search3turn3search5turn3search6

What is **not** available in the retrieved public materials is equally important: exact article count, exact author count, exact token count, complete per-category distribution, full metadata schema, and explicit redistribution/license terms for article text are all unspecified from the evidence gathered here. Your paper should state that clearly. Before submission, replace placeholders with an internal export and a dataset card. citeturn0search2turn3search2turn3search6turn0file0

### Reporting template for missing corpus statistics

Use the table below in the article exactly as a **dataset card scaffold**. Populate the blank values from an internal export before the paper is finalized.

| Dataset-card field | What can be stated now | What is currently unspecified and must be filled |
|---|---|---|
| Corpus source | Persian technical articles from urlTeknavturn0search2 | Internal export procedure and crawl date |
| URL pattern | Article-style URLs under `/article/<slug>` | Total number of article URLs collected |
| Public topical evidence | AI, data science, cybersecurity, hardware, software, startup/strategy | Exact topic taxonomy and final label mapping |
| Language profile | Persian with mixed English technical terms and acronyms | Proportion of mixed-script tokens, glossary size |
| Temporal coverage | At least recent 2026 articles are publicly visible | Full start and end publication dates |
| Metadata fields visible in snippets | Title, short description, date, article slug, topical cues | Whether tags, author, summaries, and canonical IDs are complete |
| Licensing status | Publicly viewable web articles | Redistribution rights for full text, chunks, and metadata |
| Corpus size | **Unknown from current public evidence** | Articles, documents, raw tokens, cleaned tokens, chunks, average article length, median chunk length |

A very workable target for a first submission is **300–600 gold questions** over **250–500 cleaned articles**, with full evidence mapping and at least **15–20% unanswerable or abstention-triggering items**. If the internal corpus is larger, index everything but keep manual QA authoring bounded and high quality. This is consistent with the internal plan’s “clean, well-documented over merely large” philosophy and with the design of recent Persian benchmarks that favor supervised quality over noisy scale. fileciteturn0file0 citeturn6search3turn9search0turn9search5turn17search9

### Cleaning and normalization pipeline

Your benchmark should compare at least three corpus variants:

| Variant | Description | Why it exists |
|---|---|---|
| Raw | Minimal extraction from HTML, weak cleanup, limited metadata | Represents the “dump it into a vector DB” baseline |
| Cleaned | Boilerplate removed, Unicode normalized, headings preserved, consistent sentence segmentation | Tests basic Persian text hygiene |
| Metadata-enriched | Cleaned text plus title, summary, author, category, tags, publication date, heading path, canonical URL, alias fields | Tests whether structure and provenance improve retrieval and grounding |
| Provenance-constrained | Metadata-enriched corpus plus generation prompt that must cite chunk IDs and abstain when evidence is insufficient | Tests whether prompt-level attribution reduces unsupported claims |

That design choice is justified by both the internal publication plan and by the broader data-centric AI literature, which argues that improving data representation and quality can be as consequential as model changes. fileciteturn0file0 citeturn0academia14turn0academia15turn6search0

The recommended cleaning stack is: **HTML extraction**, **boilerplate removal**, **Unicode normalization**, **Persian-specific orthographic normalization**, **sentence segmentation**, **heading preservation**, **metadata enrichment**, **deduplication**, and **alias generation** for mixed-script technical terms. For Persian preprocessing, the most directly useful public toolkit is **Hazm**, whose docs explicitly show normalization, sentence tokenization, word tokenization, chunking, lemmatization, and embedding helpers. Hazm’s own examples show normalization from `من کتاب های زیــــادی دارم .` to `من کتاب‌های زیادی دارم.`—exactly the kind of cleanup that matters for consistent retrieval. Persistent Persian-specific issues around segmentation and ZWNJ are also documented in dedicated sequence-labeling work on Persian word segmentation and zero-width non-joiner recognition. citeturn25view0turn27search10turn27search13

A practical implementation pipeline should look like this:

| Step | Goal | Recommended algorithms or tools | Important notes |
|---|---|---|---|
| HTML extraction | Get article body and heading structure | trafilatura, readability-lxml, BeautifulSoup | Preserve `h1–h6` hierarchy and canonical URL |
| Boilerplate removal | Remove menus, footers, related links | XPath/CSS rules + block-density filtering | Keep article summaries if useful |
| Unicode normalization | Canonicalize code points | Python `unicodedata.normalize("NFC", ...)` | Run before regex cleanup |
| Character normalization | Map Arabic variants to Persian forms | custom map + Hazm normalizer | Normalize `ي→ی`, `ك→ک`, repeated tatweel, diacritics |
| ZWNJ normalization | Restore half-space where appropriate | Hazm + rule lists + post-edits | Do **not** blindly remove all ZWNJs |
| Whitespace and punctuation cleanup | Canonical spacing | regex rules | Normalize punctuation spacing around Persian and Latin tokens |
| Sentence segmentation | Stable question-answer evidence units | Hazm SentenceTokenizer | Validate on titles and code-rich sentences |
| Mixed-script aliasing | Make `RAG`, `rag`, `تولید تقویت‌شده با بازیابی` retrievable together | glossary tables + normalized alias fields | Crucial for Persian technical corpora |
| Deduplication | Remove duplicate snippets, summaries, captions | MinHash/SimHash or embedding cosine threshold | Keep canonical source IDs |
| Metadata enrichment | Improve field-aware retrieval | title, headings, tags, date, summary, slug, category, author | Also store `heading_path` and `section_order` |

Public Teknav snippets already show why mixed-script aliasing is necessary: Persian prose is interleaved with Latin acronyms and English technical expressions, and Persian-English code-mixed corpora are a documented challenge in Persian NLP more broadly. citeturn3search3turn3search5turn28search1

### Before-and-after samples

The examples below are **illustrative synthetic samples**, but they are modeled on the exact kinds of Persian normalization issues documented in Hazm and ZWNJ-related papers and on the mixed-script patterns visible in Teknav’s public snippets. citeturn25view0turn27search10turn3search3turn3search5

| Issue | Before | After | Why it matters |
|---|---|---|---|
| Persian orthography | `مدل هاي زباني در كاربرد هاي سازماني` | `مدل‌های زبانی در کاربردهای سازمانی` | Better tokenization and matching |
| Arabic/Persian character mix | `كاربرد RAG در سيستم هاي فارسي` | `کاربرد RAG در سیستم‌های فارسی` | Prevents false lexical mismatches |
| ZWNJ inconsistency | `می شود` / `میشود` | `می‌شود` | Stable segmentation and search |
| Mixed-script acronym drift | `رگ`, `Rag`, `RAG`, `تولید تقویت شده با بازیابی` | canonical alias group for all forms | Avoid retriever fragmentation |
| HTML/boilerplate residue | `مطالب مرتبط | اشتراک گذاری | ...` | removed | Reduces noise in chunks |
| Heading loss | section text without heading context | chunk prefixed with `title > heading path` | Improves metadata-aware retrieval |

A second example should show why **two text fields** are worth keeping: `original_text` for faithful display and citation, and `normalized_index_text` for retrieval. That lets you normalize aggressively for search without losing the exact user-facing surface form. This is especially important for Persian technical content, where code-switching and stylistic variation are common. citeturn28search1turn25view0

### Chunking and metadata enrichment

Do **not** treat chunking as a boring preprocessing afterthought. Long-context studies show that models often underuse information buried in the middle of context windows, and recent chunking papers show that chunking method materially changes retrieval and answer quality. For a corpus like Teknav’s, section structure is an asset: headings often encode the local topic better than the body alone. citeturn11search2turn23search1turn23search2turn23search16

The best chunking comparison set for this paper is:

| Chunking strategy | Recommended unit | Default size | When it should help | Main risk |
|---|---|---:|---|---|
| Fixed sliding window | tokens/characters | 250, 400, 600 tokens | Cheap baseline | Splits concepts and headings |
| Sentence-recursive | sentence groups | 4–8 sentences | Cleaner local coherence | Still ignores section hierarchy |
| Heading-aware | section/subsection blocks | 150–450 tokens | Best for technical editorials | Uneven chunk sizes |
| Semantic breakpoint | embedding similarity boundaries | variable | Topic-shift sensitive | More compute, unstable splits |
| Late chunking | long-context embedding then post-pooling chunk map | 300–600 tokens | Better contextual embeddings | Harder to implement, model-dependent |

For **metadata enrichment**, store at least: `doc_id`, `chunk_id`, `title`, `section_heading`, `heading_path`, `category`, `tags`, `date`, `summary`, `canonical_url`, `slug`, `language_script_profile`, `alias_list`, and `position_in_document`. If you have person or product names in the article metadata, store them too, but do not make named-entity extraction a blocking requirement for the first paper. The highest-leverage fields are usually title, heading path, summary, tags, and date. citeturn19search11turn30search0turn11search2

The pipeline below is a good figure candidate for the paper.

```mermaid
flowchart LR
    A[HTML export] --> B[Boilerplate removal]
    B --> C[Unicode + Persian normalization]
    C --> D[Sentence segmentation]
    D --> E[Heading-aware chunking]
    E --> F[Alias generation and metadata enrichment]
    F --> G1[BM25/BM25F index]
    F --> G2[Dense vector index]
    G1 --> H[Hybrid fusion / reranking]
    G2 --> H
    H --> I[Generator with citation constraints]
    I --> J[Automatic metrics]
    I --> K[Human evaluation]
```

## Benchmark Design and Modeling Choices

### Tasks and splits

The benchmark should evaluate **more than one task**, even if the title emphasizes question answering. The cleanest task bundle is:

| Task | Output | Why it belongs |
|---|---|---|
| Passage retrieval | top-*k* chunk list | Measures evidence access directly |
| Evidence selection | gold-support hit and citation alignment | Separates retrieval from generation |
| Attributed short-form QA | concise Persian answer with chunk citations | Core benchmark task |
| Multi-hop QA | answer synthesized from multiple chunks/articles | Tests structure and aggregation |
| Unanswerable QA / abstention | refusal or insufficient-evidence response | Directly measures hallucination control |
| Optional temporal QA | answer constrained by publication date / recency | Useful because Teknav is a time-evolving technical corpus |

This task bundle follows the logic of RAG, ALCE, Attributed QA, and Persian QA resources that include answerability and multi-turn/contextual structure. It also prevents the paper from collapsing quality into a single surface-form metric. citeturn6search0turn8search1turn8search8turn9search0turn9search5

For splits, I recommend **article-level supervision splits** rather than random question-level splits. Index the full corpus during evaluation, but ensure that all labeled question-evidence pairs from a single source article live in only one of train, validation, or test. That prevents supervision leakage when you fine-tune a retriever, while keeping the deployment setting realistic. A second, harder evaluation split should be a **temporal holdout** built from the newest slice of the corpus, because the public site clearly publishes recent technical analysis on an ongoing basis. citeturn3search2turn3search5turn3search6

A very workable target split for **600 questions** is:

| Split | Count | Notes |
|---|---:|---|
| Train | 360 | Use for retriever fine-tuning or prompt development |
| Validation | 90 | Hyperparameter tuning, prompt selection |
| Test | 120 | Main public test set |
| Hidden test | 30 | Optional internal holdout for final model selection |
| **Total** | **600** | Include 90–120 unanswerable/adversarial items across the full set |

Question-type distribution should reflect real technical information seeking rather than generic trivia:

| Question type | Target share |
|---|---:|
| Definition / concept explanation | 20% |
| Comparison / trade-off | 15% |
| Procedure / how-it-works | 15% |
| Cause / mechanism | 15% |
| Troubleshooting / failure mode | 10% |
| Multi-hop synthesis | 15% |
| Temporal / version-sensitive | 5% |
| Unanswerable / adversarial | 5% |

Difficulty levels should be explicit and annotation-friendly:

| Difficulty | Definition |
|---|---|
| Easy | Single chunk, near-lexical overlap, direct answer |
| Moderate | Single article, paraphrased answer, mild code-mixing |
| Hard | Multi-chunk synthesis, needs heading awareness or terminology mapping |
| Adversarial | Unanswerable, distractor-rich, temporal mismatch, or cross-article temptation |

### Annotation schema and inter-annotator agreement

Every QA record should include at least the following fields:

| Field | Type | Notes |
|---|---|---|
| `question_id` | string | Stable ID |
| `question_fa` | text | Final Persian user-facing query |
| `question_en_gloss` | optional text | Helpful for internal audits |
| `category` | label | AI, data, cyber, hardware, software, startup/strategy, etc. |
| `question_type` | label | From the table above |
| `difficulty` | ordinal label | Easy / Moderate / Hard / Adversarial |
| `answerability` | binary | answerable / unanswerable |
| `reasoning_hops` | integer | 1, 2, or 3+ |
| `gold_answer_fa` | text | Concise reference answer |
| `supporting_doc_ids` | list | Documents required |
| `supporting_chunk_ids` | list | Minimal sufficient chunks |
| `supporting_spans` | optional list | Sentence/span offsets for attribution |
| `alias_terms` | optional list | Mixed-script term mappings |
| `notes_for_adjudication` | text | Edge cases and rationale |

Use **double annotation** on all validation and test questions, plus a stratified subset of training questions. For agreement, use **Cohen’s κ** for binary and single-label decisions such as answerability or question type; **weighted κ** for ordinal difficulty; and **Krippendorff’s α** when you have multiple annotators, missing labels, or multi-label coding such as hallucination categories or multi-span evidence overlap. Report the raw agreement matrix, not only a single aggregate score. citeturn20search0turn20search6turn20search1

A simple adjudication policy works well: if annotators disagree on answerability or minimal evidence, a third annotator resolves the item and writes one sentence of rationale. Save those rationales; they become gold material for the error-analysis appendix.

### Retrieval methods to compare

The retrieval comparison should stay disciplined. Your primary benchmark table should compare **BM25**, **dense retrieval**, **hybrid retrieval**, and **metadata-aware retrieval**, with reranking as an optional second-stage ablation. That covers the space users actually care about, and every method class has a clear literature ancestry. citeturn19search1turn6search1turn13search7turn13search2turn19search11turn30search0

| Method family | Suggested instantiation | Hyperparameters to try | Why include it |
|---|---|---|---|
| Sparse | BM25 | `k1 ∈ {0.9, 1.2, 1.5, 1.8, 2.0}`, `b ∈ {0.3, 0.5, 0.75, 0.9}` | Strong lexical baseline, often good on acronyms and surface terms |
| Field-aware sparse | BM25F | title weight `4–5`, heading `2–3`, summary `2`, tags `2`, body `1`, date filter optional | Direct test of metadata value |
| Dense | multilingual-E5-large | similarity `{cosine, dot}`, top-`k ∈ {5,10,20}` | Strong multilingual general dense retriever |
| Dense | BGE-M3 | dense mode plus optional sparse/multi-vector modes | Excellent multilingual retrieval and flexible retrieval modes |
| Late-interaction | ColBERTv2-style or multi-vector BGE-M3 setup | top-50 retrieve, rerank to top-10 | Better passage discrimination |
| Hybrid | BM25 + dense via RRF | RRF constant `60`, or weighted fusion `α ∈ {0.3,0.5,0.7}` | Usually strongest all-around first-stage setup |
| Metadata-aware dense | prepend structured header to chunk text | try title+heading prefix on/off | Direct test of structure-aware embeddings |
| Query-expansion dense | HyDE on hard queries only | one hypothetical doc per query | Useful for low-overlap paraphrase questions |

A good **minimal** benchmark comparison is:

1. BM25  
2. multilingual-E5-large  
3. BGE-M3  
4. BM25 + E5 via RRF  
5. BM25F + BGE-M3  
6. BM25F + BGE-M3 + optional reranker

That is enough to support the article’s central claim without turning the paper into a retriever zoo.

For infrastructure, a reproducible stack such as **Pyserini/Lucene for sparse retrieval** and **Faiss/HNSW for dense retrieval** is ideal because both have explicit reproducibility literature and are standard in information retrieval research. citeturn31search0turn26search0

### Chunking strategies to compare

Chunking deserves its own ablation table because long-context work and chunking studies both show that chunk boundary decisions alter what the retriever can actually surface. For technical editorials, **heading-aware chunks** are likely to beat pure fixed windows, especially on “compare,” “how,” and “why” questions where the heading already encodes topic intent. citeturn11search2turn23search1turn23search2turn12search1

| Strategy | Parameters to try | Best use case | Expected outcome |
|---|---|---|---|
| Fixed-size | 250 / 400 / 600 tokens; 10–15% overlap | cheap baseline | good recall, weaker precision |
| Recursive sentence | 4–8 sentence groups, max 450 tokens | plain prose | better coherence than fixed windows |
| Heading-aware | split on `h2/h3`, sub-split above 450 tokens | technical editorials | strongest interpretability and citation quality |
| Semantic breakpoint | sentence embeddings, threshold-based boundary detection | topic-shift articles | better topical purity, more compute |
| Late chunking | embed long section then post-pool subspans | long structured sections | potentially better dense retrieval, harder ops |

Context assembly also matters. Because “lost in the middle” effects remain real, do not blindly send 20 mediocre chunks to the generator. A better strategy is **top-3 to top-8 high-value chunks**, optionally diversified across heading paths, with the most relevant chunk first and support chunks afterward. citeturn11search2

### Generators and RAG architectures to evaluate

Model choice should be **secondary** to the data-centric thesis, but still broad enough to matter. The best design is to evaluate a small but sensible shortlist across **commercial APIs**, **strong multilingual open models**, and at least one **Persian-focused open model**. Freeze the exact model versions and dates because API models drift. citeturn14search0turn14search5turn14search6turn14search3turn22search0turn15search6turn15search9turn18search1

Current, well-documented candidates as of the benchmark freeze date are:

| Class | Candidate | Why include it | Recommended role |
|---|---|---|---|
| Commercial | urlGPT-4.1 model docsturn14search0 | Strong instruction following, long context | high-end answer-generation baseline |
| Commercial | urlClaude models overviewturn14search5 / urlClaude Sonnet 4.6turn14search1 | strong document reasoning and long context | premium grounded-generation baseline |
| Commercial | urlGemini 2.5 Pro docsturn14search6 | strong large-context document reasoning | premium reasoning baseline |
| Commercial | urlCommand R+ docsturn14search3 | explicitly positioned for complex RAG workflows | retrieval-oriented commercial baseline |
| Open-weight | urlQwen2.5-14B-Instruct model cardturn22search0 | strong multilingual generalist, affordable | best open mid-size baseline |
| Open-weight | urlQwen2.5 Technical Reportturn22search1 | supports larger family choices up to 72B | high-quality open reference family |
| Open-weight | urlQwen2.5 1M model cardturn22search3 | long-context variant | late-stage long-context ablation |
| Open-weight | urlAya Expanse docsturn15search6 | multilingual, tuned for 23 languages | strong multilingual comparison |
| Open-weight | urlAya Expanse technical reportturn15search14 | open multilingual family evaluation | justify language-general baseline |
| Open-weight | urlLlama 3.1 model cardturn15search9 | common open baseline with multilingual use | ecosystem baseline |
| Persian-focused | urlPersianLLaMAturn18search1 | monolingual Persian control model | language-focused baseline |
| Persian-focused encoder | urlParsBERT model cardturn21search6 | strong Persian encoder for retrieval/reranking | retriever and NLI helper |

The architecture shortlist should stay simple:

| Architecture | Description | Include? | Why |
|---|---|---|---|
| Retrieval-only baseline | retrieve chunks, report them or extract minimal answer | Yes | isolates retrieval quality |
| Vanilla RAG | retrieve top-*k*, generate cited answer | Yes | standard benchmark baseline |
| Provenance-aware RAG | same as above, but prompt requires chunk IDs and abstention when evidence is insufficient | Yes | directly tests hallucination reduction |
| Rerank-RAG | retrieve many, rerank, then generate | Yes | practical production variant |
| HyDE-assisted RAG | use hypothetical-doc query expansion only on hard questions | Optional | useful on paraphrase-heavy queries |
| Self-RAG / CRAG style | adaptive retrieval or corrective retrieval | Optional | interesting research ablation, but not necessary for first paper |

Use **retrieval-only** and **retriever + prompt** baselines as the main story. Treat **LoRA fine-tuning** or reflective architectures as secondary extensions; otherwise the paper drifts away from its core data-centric question. That recommendation lines up with the benchmark’s stated aim and with the literature: keep the main causal story interpretable. citeturn10search0turn10search1turn10search2turn0academia14

A practical provenance-aware answer prompt should require four things: a concise answer in Persian, sentence-level chunk citations, an explicit “insufficient evidence” branch, and a ban on uncited claims. A quote-first or evidence-first prompt can also help: ask the model to list supporting bullet evidence from retrieved chunks before writing the synthesis. That usually makes human evaluation easier.

## Evaluation, Hypotheses, and Error Analysis

### Metrics

Do not rely on one metric family. The most convincing benchmark papers separate **retrieval**, **generation**, **grounding**, **hallucination**, **abstention**, **calibration**, and **human judgment**. Existing work already gives you mature building blocks: FActScore for atomic fact support, SAFE for search-based factual verification, ALCE for citation quality, Attributed QA for reproducible attribution evaluation, and RAGAS for fast reference-free diagnostics. citeturn7search3turn7search2turn8search1turn8search8turn7search10

Use the following metric matrix:

| Dimension | Metrics | Recommended use |
|---|---|---|
| Retrieval quality | Recall@1/3/5/10, Precision@k, MRR, nDCG@10, gold-support hit rate | first-stage benchmarking |
| Evidence breadth | supporting-section coverage, multi-document coverage | useful on multi-hop questions |
| Short-answer correctness | Exact Match, token F1 | only for extractive or near-extractive items |
| Generative similarity | ROUGE-L, chrF, BLEU | optional, secondary diagnostics only |
| Grounding | sentence support rate, citation precision, citation recall, minimal-evidence sufficiency | core benchmark dimension |
| Hallucination | atomic factual support precision, unsupported-claim count, unsupported-claim rate | primary trust metric |
| Abstention | refusal precision, refusal recall, abstention F1, answerability accuracy | essential because unanswerable items are included |
| Calibration | ECE, Brier score, confidence-vs-correctness correlation | useful if model outputs confidence |
| Human evaluation | correctness, completeness, grounding, fluency, usefulness | final credibility layer |

My strongest recommendation is to make **three metrics primary** in the paper’s headline tables:

1. **Gold-support hit rate / Recall@k**  
2. **Atomic support precision**  
3. **Citation precision / recall**

Then keep EM/F1/ROUGE as subordinate tables. That design ensures your paper remains about **hallucination reduction and grounding**, not mere wording overlap.

A practical automatic metric bundle for TekRAG-Persian would be:

| Metric | Definition |
|---|---|
| **AFSP** | Atomic Factual Support Precision = supported atomic facts / total atomic facts in answer |
| **UCR** | Unsupported Claim Rate = unsupported atomic facts / total atomic facts |
| **CP** | Citation Precision = cited chunks that are actually needed or sufficient / total cited chunks |
| **CR** | Citation Recall = answer units with at least one sufficient citation / total answer units |
| **MSR** | Minimal Support Recall = gold support chunks recovered within top-*k* |
| **AR-F1** | Abstention Refusal F1 on unanswerable questions |
| **ECE** | calibration error on answer confidence or support confidence |

These metrics are easy to explain in the text and map naturally to FActScore-, ALCE-, and attribution-style evaluation. citeturn7search3turn8search1turn8search8

### Experimental protocol and reproducibility checklist

A publishable benchmark paper should freeze the following:

| Item | Must report? |
|---|---|
| Full crawl/export date | Yes |
| Corpus variant definitions and scripts | Yes |
| Article inclusion/exclusion rules | Yes |
| Exact train/val/test article IDs | Yes |
| All prompts and system prompts | Yes |
| Retriever models and checkpoints | Yes |
| Chunk sizes, overlap, heading rules | Yes |
| Top-*k* retrieval and rerank settings | Yes |
| Generator model versions / API snapshot date | Yes |
| Random seeds | Yes |
| Hardware, runtime, latency measurement method | Yes |
| Human-annotation guidelines | Yes |
| Evaluation scripts and metric formulas | Yes |
| Confidence intervals and significance tests | Yes |
| Cost reporting | Strongly recommended |
| Hidden test governance | Strongly recommended |

For open-source implementation, a reproducible stack such as Pyserini plus Faiss is ideal. For commercial APIs, record the exact model name, provider doc page, date of run, and any temperature/system settings. Do not compare API results collected months apart without saying so. citeturn31search0turn26search0turn14search0turn14search5turn14search6turn14search3

Human evaluation should review at least a stratified sample across **topic**, **difficulty**, **question type**, and **system family**. A good minimum is **150–200 evaluated answers**, double-annotated, with adjudication only after independent scoring. Use a 1–5 anchored rubric on correctness, completeness, grounding, and fluency, and separately count unsupported claims.

### Hypotheses and statistical tests

Your hypotheses should be explicit and modest:

| Hypothesis | Expected direction |
|---|---|
| H1 | Cleaned corpus > raw corpus on retrieval and grounding |
| H2 | Metadata-enriched corpus > cleaned corpus on citation precision and multi-hop QA |
| H3 | Hybrid retrieval > sparse-only and dense-only overall |
| H4 | BM25 remains disproportionately strong on acronym-heavy and code-mixed questions |
| H5 | Heading-aware chunking > fixed-size chunking on technical explanatory questions |
| H6 | Provenance-aware prompting reduces unsupported claims but may increase abstention and shorten answers |
| H7 | Stronger generators help, but data-quality gains remain visible even when generator is held fixed |

Those hypotheses are directly consistent with the RAG, attribution, long-context, and data-centric literatures, and with the observable mixed-script nature of the target corpus. citeturn6search0turn19search11turn30search0turn11search2turn0academia14turn3search3turn3search5

For statistical testing, the cleanest protocol is:

| Outcome type | Recommended test |
|---|---|
| Binary per-question outcomes | McNemar’s test |
| Per-question scalar metrics | paired bootstrap CI + Wilcoxon signed-rank |
| Rank-based retrieval metrics | paired randomization or stratified bootstrap |
| Human ordinal ratings | Wilcoxon or cumulative-link mixed model |
| Multi-comparison correction | Holm-Bonferroni |

Also report **effect sizes** and **95% confidence intervals**, not only *p*-values. That matters especially when differences are operationally real but numerically moderate.

### Hallucination taxonomy

Use a taxonomy that combines general hallucination theory with Persian technical specifics. The surveys are useful for the intrinsic/extrinsic and faithfulness/factuality distinctions, but your benchmark needs finer operational categories. citeturn29search0turn29search1

| Category | Description | Persian-style example | Likely cause | Best fix |
|---|---|---|---|---|
| Unsupported elaboration | Adds details not in retrieved text | Evidence says `RAG دقت را بهبود می‌دهد`; answer says `RAG خطا را تقریباً صفر می‌کند` | overconfident generation | provenance-aware prompt |
| Citation mismatch | Claim may be correct, citation does not support it | answer cites a chunk about `MCP` to support a claim about `بردارهای تعبیه` | poor evidence selection | better reranking, citation audit |
| Terminology drift | Incorrect translation or concept conflation | `بازیابی` used where answer really means `رتبه‌بندی` | bilingual jargon mismatch | alias glossary |
| Script normalization failure | Retrieval misses variant spellings | `می شود` and `می‌شود` not matched | inconsistent indexing | Persian normalization |
| Arabic/Persian codepoint drift | `ك` and `ک` treated differently | relevant chunk not retrieved | Unicode inconsistency | canonical character mapping |
| Mixed-script retrieval miss | English acronym present in Persian article but query uses Persian gloss | query: `تولید تقویت‌شده با بازیابی`, article uses only `RAG` | alias failure | bilingual term expansion |
| Temporal hallucination | Gives outdated or future-invented claim | answer uses older product/version facts unsupported by article date | date ignored | metadata/date filter |
| Cross-article causal leap | Combines two true facts into unsupported cause/effect | article A on clean data + article B on model collapse → answer invents causal chain not actually stated | multi-hop overreach | support-at-claim-level evaluation |
| Boundary truncation | Chunk split drops critical qualifier | `ممکن است` or `در برخی موارد` is omitted | bad chunking | heading-aware or sentence-aware chunking |
| Over-refusal | Says evidence is insufficient when answer is actually available | conservative prompt or weak retrieval | cautious system | prompt tuning |
| Under-refusal | Guesses when the corpus does not answer | classic hallucination | missing abstention logic | explicit refusal policy |

You should explicitly connect these categories to the corpus itself. Public Teknav snippets already show a mixed Persian-English technical register with terms like “MCP” and “AI-Native IDEs,” while Persian NLP work documents segmentation, ZWNJ, and code-mixed text as real processing challenges. That gives you a strong empirical rationale for a **Persian-specific hallucination/error taxonomy** rather than just recycling English categories. citeturn3search3turn3search5turn25view0turn27search10turn28search1

## Writing Assets, Ethics, and Appendices

### Visualization plan

The article will read much better if you make the **data-centric story visible**. Readers should be able to see what changed between raw and cleaned corpora, what metadata fields were added, how retrieval pipelines differ, and where hallucination types cluster. Visualization is also the easiest way to stop the manuscript from feeling like an overgrown benchmark spreadsheet.

The most useful visuals are:

| Visual | Form | Message |
|---|---|---|
| Corpus-pipeline figure | Mermaid flowchart | benchmark is about engineered data, not just models |
| Corpus composition chart | bar chart | show category and date coverage |
| Variant comparison table | table | raw vs cleaned vs metadata-enriched |
| Retrieval comparison chart | grouped bar | Recall@k, MRR, citation metrics by retriever |
| Chunking comparison chart | grouped bar | heading-aware vs fixed vs semantic vs late |
| Hallucination taxonomy chart | stacked bar | unsupported elaboration, citation mismatch, etc. by system |
| Cost-quality Pareto | scatter plot | latency/cost vs grounding quality |
| Answerability confusion matrix | heatmap | over-refusal vs under-refusal patterns |
| Error examples panel | table | concrete Persian examples and fixes |

### Recommended figures and tables

You can lift the captions below almost verbatim into the paper draft.

| Item | Recommended caption |
|---|---|
| Figure | **Benchmark construction pipeline for TekRAG-Persian, from article export and Persian-specific normalization to chunk indexing, retrieval, grounded generation, and evaluation.** |
| Figure | **Examples of Persian-specific normalization and alias generation, including ZWNJ correction, Arabic/Persian character mapping, and mixed-script terminology resolution.** |
| Figure | **Distribution of TekRAG-Persian questions by topic, question type, and difficulty.** |
| Figure | **Comparison of retrieval effectiveness across corpus variants and retrieval families.** |
| Figure | **Impact of chunking strategy on support retrieval and answer grounding.** |
| Figure | **Hallucination taxonomy for Persian technical QA, with category frequencies by model and retrieval setup.** |
| Figure | **Latency, token cost, and grounding-quality trade-offs across evaluated systems.** |
| Table | **Publicly observable and internally required corpus statistics for TekRAG-Persian.** |
| Table | **Question annotation schema and inter-annotator agreement metrics.** |
| Table | **Retrieval model comparison and hyperparameter grid.** |
| Table | **Generator model shortlist, access mode, and experimental role.** |
| Table | **Automatic and human evaluation metrics for hallucination-sensitive Persian technical QA.** |
| Table | **Representative failure cases with Persian examples, likely causes, and mitigation strategies.** |

### Section-level writing advice

| Section | Writing advice | Common mistake to avoid |
|---|---|---|
| Introduction | Open with technical QA reliability, not with generic “AI is changing the world” rhetoric | making the problem sound like a blog post |
| Related Work | Group by function: retrieval, attribution, data-centric, Persian resources | listing papers without a synthesis paragraph |
| Dataset | Be concrete: fields, statistics, cleaning rules, and licensing status | hiding missing corpus details |
| Methods | Separate corpus design, retrieval design, and generation design | mixing benchmark contribution with too many model tricks |
| Evaluation | Explain why each metric exists and what failure it captures | using ROUGE/BLEU as the main story |
| Results | Keep tables compact and tell one message per paragraph | flooding the section with all ablations at once |
| Discussion | Explain *why* metadata and cleaning help or fail | repeating result numbers without interpretation |
| Limitations | Be candid about corpus scope, licensing, and model contamination | defensive writing |
| Conclusion | Close on the data-centric lesson | overselling “first” or “SOTA” claims |

A practical citation-density rule is: **1–2 citations per paragraph in Introduction and Methods, 2–4 in Related Work, 1–3 in Dataset/Evaluation, and 1–2 in Discussion/Limitations**. A benchmark paper looks under-supported when entire paragraphs in Related Work or Dataset Construction carry no sources at all.

### Ethical considerations

The ethics section should focus on **copyright, benchmark leakage, model drift, annotator burden, and mixed-script linguistic fairness**. The biggest immediate issue is licensing: public Teknav content is visible on the web, but the retrieved public materials do **not** establish redistribution rights for full-text release as a benchmark. If rights are not explicit, release metadata, question annotations, evidence IDs, and scripts rather than dumping full text. citeturn0search2turn3search2turn0file0

A second issue is **model contamination**. Some commercial and open models may already have seen public technical web content during pretraining. This does not invalidate the benchmark, but it does mean you must distinguish **retrieval grounding** from **parametric memory**. Include no-RAG baselines, require citations, and report abstention behavior. A third issue is linguistic fairness: Persian technical writing mixes native Persian, transliterated forms, and English loanwords, so evaluation should not punish systems merely for choosing acceptable term variants when the meaning is supported. citeturn8search1turn8search8turn28search1turn28search11

### Limitations and future work

Be explicit that the first paper is likely limited to **technical editorial Persian**, not all Persian domains. Results may not transfer unchanged to legal, medical, or purely colloquial corpora. The benchmark will also likely underrepresent multimodal evidence such as figures, charts, code blocks, and tables unless you explicitly model them. Chunking studies in other domains already show that table-heavy or diagram-heavy documents remain difficult for text-only RAG. citeturn12search1turn23search16

Future work should extend in five directions: multilingual Persian-English technical retrieval; multimodal QA over tables and figures; conversational follow-up questions; adaptive retrieval methods such as Self-RAG or CRAG; and a public leaderboard with hidden test governance. On the Persian side, future benchmark alignment with PARSE, PeKA/PK-BETS-style evaluation, and culturally grounded Persian LLM assessment would also make sense. citeturn10search0turn10search1turn17search9turn18search5

### Timeline

The timeline below is realistic for a first serious benchmark paper.

```mermaid
gantt
    title TekRAG-Persian Benchmark Build Plan
    dateFormat  YYYY-MM-DD
    section Scoping
    Literature review and benchmark spec       :a1, 2026-05-10, 7d
    section Corpus
    Export, cleaning, normalization            :a2, after a1, 10d
    Metadata enrichment and chunking           :a3, after a2, 6d
    section Annotation
    Question drafting and evidence mapping     :a4, after a2, 12d
    Double annotation and adjudication         :a5, after a4, 8d
    section Systems
    Retrieve/index baselines                   :a6, after a3, 6d
    Generator baselines and prompts            :a7, after a6, 6d
    section Evaluation
    Automatic metrics and scripts              :a8, after a7, 6d
    Human evaluation                           :a9, after a8, 6d
    Statistical analysis and figures           :a10, after a9, 5d
    section Writing
    Draft methods and dataset sections         :a11, after a8, 6d
    Draft results and discussion               :a12, after a10, 6d
    Final revision and submission package      :a13, after a12, 5d
```

### Appendix package

The appendix should not be filler. It is where the paper becomes reproducible.

| Appendix item | What to include |
|---|---|
| Annotation guidelines | answerability rules, minimal-support definition, citation sufficiency rules |
| Sample QA pairs | at least 10 fully worked examples across types and difficulties |
| Prompt templates | vanilla RAG, provenance-aware RAG, abstention prompt |
| Hyperparameter grids | BM25, BM25F, dense index, reranker, top-*k*, chunk sizes |
| Evaluation scripts | retrieval scoring, atomic fact scoring, citation scoring |
| Corpus cleaning scripts | normalization maps, alias dictionary logic, deduplication rules |
| Additional tables | per-category and per-difficulty results |
| Error examples | annotated Persian failures with fixes |
| Dataset card | licenses, release format, versioning, intended use, limitations |

Two small code snippets are worth shipping in the appendix: one for normalization and one for citation scoring. The snippet below is intentionally minimal and readable.

A Persian normalization helper can look like this, using the documented normalization behavior in Hazm as the baseline. citeturn25view0turn27search10

```python
import re
import unicodedata
from hazm import Normalizer, SentenceTokenizer

ARABIC_TO_PERSIAN = str.maketrans({
    "ي": "ی",
    "ك": "ک",
})

normalizer = Normalizer()
sent_tok = SentenceTokenizer()

def normalize_for_index(text: str) -> str:
    text = unicodedata.normalize("NFC", text)
    text = text.translate(ARABIC_TO_PERSIAN)
    text = re.sub(r"\s+", " ", text).strip()
    text = normalizer.normalize(text)
    return text

def split_sentences(text: str) -> list[str]:
    return sent_tok.tokenize(normalize_for_index(text))
```

A lightweight citation-quality evaluator can be shipped like this:

```python
from collections import defaultdict

def citation_precision(answer_units, cited_chunk_ids, sufficient_support_map):
    """
    answer_units: list of atomic answer units or sentences
    cited_chunk_ids: dict[unit_id] -> list[chunk_id]
    sufficient_support_map: dict[unit_id] -> set[gold_chunk_id]
    """
    supported_citations = 0
    total_citations = 0

    for unit_id in answer_units:
        predicted = set(cited_chunk_ids.get(unit_id, []))
        gold = set(sufficient_support_map.get(unit_id, set()))
        total_citations += len(predicted)
        supported_citations += len(predicted & gold)

    return 0.0 if total_citations == 0 else supported_citations / total_citations


def citation_recall(answer_units, cited_chunk_ids, sufficient_support_map):
    covered = 0
    for unit_id in answer_units:
        predicted = set(cited_chunk_ids.get(unit_id, []))
        gold = set(sufficient_support_map.get(unit_id, set()))
        if predicted & gold:
            covered += 1
    return 0.0 if len(answer_units) == 0 else covered / len(answer_units)
```

Finally, include **three fully written sample QA records** in the appendix. A template like this is article-ready:

| Field | Example |
|---|---|
| Question | `چرا داده‌های مصنوعیِ بدون کنترل می‌توانند کیفیت مدل را کاهش دهند؟` |
| Type | Cause / mechanism |
| Difficulty | Hard |
| Answerability | Answerable |
| Gold answer | `زیرا آموزش بازگشتی روی داده‌های مصنوعی می‌تواند توزیع داده را باریک‌تر کند، رخدادهای دُم‌بلند را حذف کند، و خطاهای قبلی را تقویت کند.` |
| Required support | `doc_014#h2_2`, `doc_014#h3_1` |
| Common wrong answer | `چون داده‌های مصنوعی همیشه نویزی و غیرقابل استفاده هستند.` |
| Hallucination label | Unsupported universalization |

That kind of appendix material is exactly what turns the article from “interesting idea” into “benchmark someone can actually reproduce.”