teknav-bak/app/TekRAG-Persian-Article.md

# TekRAG-Persian: A Data-Centric Benchmark for Reducing Hallucination in Persian Technical Question Answering

## Abstract

Retrieval-augmented generation (RAG) is widely used to improve the factual grounding of large language model applications, but many deployed RAG systems still produce unsupported claims, weak citations, and incomplete answers. A frequent cause is not the language model itself, but the quality of the corpus being retrieved. This problem is amplified in Persian technical content, where articles often contain mixed Persian-English terminology, right-to-left formatting, inconsistent character forms, publication metadata, and section structures that are lost during naïve indexing. This article proposes **TekRAG-Persian**, a data-centric benchmark and evaluation framework for measuring how corpus preparation affects hallucination in Persian technical question answering. The benchmark compares three corpus variants: raw article text, cleaned section-aware text, and metadata-enriched text with source provenance. It evaluates retrieval accuracy, answer correctness, citation faithfulness, hallucination rate, unsupported claim count, and operational cost. The proposed framework is designed for technology domains such as artificial intelligence, data governance, cybersecurity, software engineering, and digital infrastructure. Rather than introducing a new model, the work argues that reliable Persian RAG requires measurable improvements in data cleaning, chunk design, semantic metadata, and provenance-aware generation. The article provides a complete research protocol that can be used to build an empirical study suitable for submission to an applied artificial intelligence, information retrieval, or low-resource NLP venue.

**Keywords:** Persian NLP; retrieval-augmented generation; hallucination; data quality; technical question answering; low-resource languages; information retrieval; provenance; RAG evaluation

## 1. Introduction

Large language models have changed how users interact with technical knowledge. Instead of searching through documentation, articles, knowledge bases, or manuals directly, users increasingly ask a natural-language question and expect a concise answer. In technical domains, this creates a serious reliability problem. A fluent answer is not necessarily a correct answer. If the system cannot identify the relevant source, preserve context, and cite evidence, it may produce a confident but unsupported explanation.

Retrieval-augmented generation is one of the most practical responses to this problem. A RAG system retrieves external passages from a corpus and passes them to a language model as context. The model then generates an answer grounded in retrieved evidence. In principle, this architecture reduces hallucination because the model no longer relies only on internal parameters. In practice, RAG quality depends heavily on the quality of the corpus, chunking strategy, metadata, retrieval model, and prompt design.

For Persian technical question answering, this dependency is especially important. Persian technology articles often mix Persian text with English acronyms and technical entities such as RAG, LLM, API, GPU, CVE, MCP, vector database, and model collapse. They may include Persian and Arabic variants of characters, inconsistent half-spaces, right-to-left formatting artifacts, HTML markup, embedded diagrams, metadata fields, and publication dates. If this material is inserted into a vector database without careful preparation, retrieval may fail even when the original article is accurate and well written.

This article proposes **TekRAG-Persian**, a benchmark framework for studying the data side of hallucination in Persian technical RAG. The central claim is simple: hallucination reduction is not only a model problem. It is also a corpus quality problem. A better prepared corpus can improve retrieval, preserve provenance, reduce unsupported claims, and make generated answers easier to verify.

The research question is:

**How do corpus cleaning, semantic metadata enrichment, and provenance-aware chunking affect retrieval accuracy and hallucination rates in Persian technical question answering using RAG?**

The proposed study compares three corpus variants:

1. **Raw corpus:** minimally processed article text.
2. **Cleaned corpus:** normalized Persian text with section-aware chunking.
3. **Metadata-enriched corpus:** cleaned chunks enriched with title, category, tags, author, publication date, summary, canonical URL, heading path, and provenance identifier.

The expected contribution is not a new language model. The contribution is a reproducible benchmark design and evaluation protocol for measuring whether data-centric preparation improves Persian technical RAG.

## 2. Background and Motivation

### 2.1 Retrieval-Augmented Generation

RAG systems combine retrieval and generation. A retriever selects relevant passages from an external corpus. A generator then uses those passages to produce an answer. This architecture is attractive because it allows knowledge to be updated without retraining the model and because answers can be connected to sources.

However, a RAG system can fail at multiple points. The parser may extract noisy text. The chunker may split a section in the middle of an argument. The embedding model may fail to match Persian paraphrases with English technical terms. The retriever may return a related but insufficient passage. The generator may ignore the retrieved text and rely on prior knowledge. The citation layer may cite a source that does not actually support the claim.

Because of this chain of dependencies, RAG evaluation must measure both retrieval and generation. A final answer can be wrong because the retriever failed, because the generator failed, or because the corpus did not preserve enough semantic context.

### 2.2 Hallucination in Technical QA

Hallucination is often described as a model inventing facts. In technical question answering, hallucination is more specific. It can include:

- unsupported technical claims;
- incorrect comparison between technologies;
- invented standards, versions, or vulnerabilities;
- wrong causal explanations;
- citations that do not support the answer;
- confident extrapolation beyond the retrieved evidence;
- mixing facts from unrelated documents.

Technical hallucinations are dangerous because they can lead to wrong engineering decisions. A hallucinated answer about cybersecurity, software architecture, or AI deployment may be expensive or unsafe. Therefore, evaluation should not only ask whether an answer sounds plausible. It should ask whether each important claim is supported by retrieved evidence.

### 2.3 Persian as a Technical RAG Challenge

Persian introduces challenges that are often hidden in English-centric RAG evaluation. The script is right-to-left, but technical terms are frequently left-to-right. Persian text may contain Arabic forms of letters such as ي and ك instead of ی and ک. Half-space usage may vary. Some phrases are translated into Persian while others remain in English. A user might ask about "پایگاه داده برداری" while the article uses "Vector Database"; another user might ask about "توهم" while the article uses "Hallucination".

These patterns make metadata and normalization important. Tags, headings, summaries, and bilingual technical terms can help bridge vocabulary gaps between user questions and source text. A raw vector index may miss these connections.

### 2.4 Data-Centric AI for RAG

Data-centric AI shifts attention from model architecture to dataset quality. In RAG, this means improving the source corpus before retrieval. Examples include:

- removing boilerplate;
- preserving headings;
- normalizing language-specific characters;
- deduplicating repeated text;
- adding semantic metadata;
- tracking provenance;
- chunking by logical sections;
- filtering stale or low-quality content.

For Persian technical content, data-centric preparation may be the difference between a system that retrieves the correct article and one that retrieves a loosely related paragraph.

## 3. Benchmark Scope

TekRAG-Persian is designed for Persian technical articles. The recommended corpus includes domains aligned with Teknav's editorial strengths:

- artificial intelligence;
- data governance;
- retrieval-augmented generation;
- cybersecurity;
- software architecture;
- hardware and infrastructure;
- technology strategy.

The corpus should include article-level metadata and structured body content. Each article should have:

- stable article ID;
- slug;
- title;
- subtitle;
- summary;
- category;
- tags;
- author;
- publication date;
- canonical URL;
- body text;
- headings;
- optional source notes;
- optional fact-check date.

The benchmark should exclude drafts, unpublished scheduled articles, private user data, credentials, analytics logs, and comments. The goal is to evaluate public editorial knowledge, not user behavior.

## 4. Dataset Construction

### 4.1 Article Parsing

The first step is to export article content into a structured intermediate format. HTML should be parsed, but not blindly flattened. Headings should be preserved because they carry semantic context. For example, a section titled "Data Provenance and Digital Identity" changes how a paragraph should be interpreted. If the paragraph is indexed without the heading, retrieval may lose context.

The structured representation should look like this:

```json
{
  "article_id": "art34",
  "slug": "ai-data-centric-2026",
  "title": "Persian article title",
  "summary": "Persian article summary",
  "category": "ai",
  "tags": ["Data-Centric AI", "Model Collapse", "Data Governance"],
  "author": "Author name",
  "published_at": "2026-05-05",
  "canonical_url": "https://www.example.com/article/ai-data-centric-2026",
  "sections": [
    {
      "heading": "Section heading",
      "text": "Section body"
    }
  ]
}
```

### 4.2 Question Set

The benchmark should contain 200 to 500 Persian questions in the first version. Each question should be mapped to one or more supporting chunks. Questions should not be generated and accepted automatically without review. Even if AI is used to draft candidate questions, human review is required.

Question categories:

- definition questions;
- comparison questions;
- causal questions;
- procedure questions;
- risk and mitigation questions;
- trend interpretation questions;
- multi-hop questions.

Example:

```json
{
  "question_id": "q001",
  "question": "چرا داده‌های مصنوعی بدون نظارت می‌توانند باعث فروپاشی مدل شوند؟",
  "topic": "ai",
  "difficulty": "medium",
  "expected_answer": "A reviewed Persian answer explaining entropy loss, recursive training, distribution drift, and loss of rare events.",
  "supporting_chunks": ["art34-sec1-chunk1", "art08-sec2-chunk1"],
  "answer_type": "explanation"
}
```

### 4.3 Gold Evidence

Each question should identify gold supporting evidence. This allows retrieval metrics to be calculated. If a question requires two chunks, a system that retrieves only one should receive partial retrieval credit but may still produce an incomplete answer.

Gold evidence is also necessary for hallucination evaluation. A generated answer should be judged against the evidence, not against general plausibility.

## 5. Corpus Variants

### 5.1 Raw Corpus

The raw corpus is the baseline. It contains minimally processed article text. It may include weak chunk boundaries, missing metadata, and text extracted without preserving full heading context. This version represents the fastest production approach and is useful because many real systems are built this way.

### 5.2 Cleaned Corpus

The cleaned corpus applies Persian-aware preprocessing:

- normalize Arabic/Persian variants of letters;
- normalize whitespace and half-space where appropriate;
- remove HTML tags and boilerplate;
- remove repeated navigation or UI text;
- preserve headings;
- split by sections;
- remove empty or tiny chunks;
- keep English technical terms intact.

This version tests whether basic text hygiene improves retrieval.

### 5.3 Metadata-Enriched Corpus

The metadata-enriched corpus adds structured context to each chunk. A chunk should include:

- article title;
- article summary;
- section heading;
- category;
- tags;
- author;
- date;
- canonical URL;
- chunk ID;
- provenance label.

The chunk text sent to retrieval can include metadata in a controlled format:

```text
Title: ...
Category: ...
Tags: ...
Published: ...
Section: ...
Source: ...

Content:
...
```

This format makes the chunk more understandable to both embedding and generation stages.

### 5.4 Provenance-Aware Generation

The optional fourth condition uses the metadata-enriched corpus with stricter prompting. The model is instructed to answer only from retrieved context and cite sources. If evidence is insufficient, it must say so. This tests whether provenance-aware generation reduces unsupported claims.

## 6. Experimental Method

### 6.1 Pipeline

For each corpus variant:

1. Parse articles.
2. Create chunks.
3. Generate embeddings.
4. Build vector index.
5. Retrieve top-k chunks for each question.
6. Generate an answer using a fixed prompt.
7. Evaluate retrieval and generation.

Top-k values should include 3, 5, and 8. This reveals whether improvements come from better ranking or from giving the model more context.

### 6.2 Prompt Template

The generation prompt should be stable across conditions:

```text
You are answering a Persian technical question using only the provided context.
If the context does not support an answer, say that the available evidence is insufficient.
Do not add claims that are not supported by the context.
Cite the provided source IDs for important claims.

Question:
{question}

Context:
{retrieved_chunks}

Answer in Persian:
```

Temperature should be low to reduce random variation. All model settings should be reported.

### 6.3 Baselines

Minimum baselines:

1. Raw RAG.
2. Cleaned RAG.
3. Metadata-enriched RAG.
4. Metadata-enriched RAG with provenance-aware prompting.

Optional baselines:

- no-RAG answer;
- BM25 retrieval;
- hybrid retrieval;
- reranked retrieval.

The no-RAG baseline is useful because it measures how often the model answers from prior knowledge rather than evidence.

## 7. Evaluation

### 7.1 Retrieval Metrics

Retrieval should be evaluated independently from generation:

- Recall@k;
- Precision@k;
- Mean Reciprocal Rank;
- supporting chunk hit rate;
- category match rate.

Supporting chunk hit rate is the most interpretable metric. It asks whether at least one required evidence chunk appears in the retrieved context.

### 7.2 Answer Metrics

Generated answers should be evaluated with:

- correctness;
- completeness;
- faithfulness;
- citation accuracy;
- unsupported claim count;
- hallucination rate.

An answer should be considered hallucinated if it contains a substantial claim that is not supported by retrieved context or gold evidence.

### 7.3 Human Evaluation

Human evaluators should score answers from 1 to 5:

1. Incorrect or hallucinated.
2. Partially relevant but mostly unsupported.
3. Mostly correct but incomplete or weakly cited.
4. Correct, grounded, and mostly complete.
5. Correct, complete, concise, and well cited.

At least two evaluators should review a subset of answers. If possible, report agreement.

### 7.4 Operational Metrics

Operational metrics:

- latency;
- input token count;
- output token count;
- total cost;
- refusal rate;
- average answer length.

These metrics matter because the best research result may not be the best production system.

## 8. Results Template

This article does not invent experimental numbers. The following tables define the required result structure.

### 8.1 Retrieval Results

| Corpus Variant | Recall@3 | Recall@5 | Recall@8 | MRR | Supporting Hit Rate |
|---|---:|---:|---:|---:|---:|
| Raw | TODO | TODO | TODO | TODO | TODO |
| Cleaned | TODO | TODO | TODO | TODO | TODO |
| Metadata-enriched | TODO | TODO | TODO | TODO | TODO |

### 8.2 Generation Results

| Condition | Correctness | Completeness | Citation Accuracy | Hallucination Rate | Unsupported Claims |
|---|---:|---:|---:|---:|---:|
| Raw RAG | TODO | TODO | TODO | TODO | TODO |
| Cleaned RAG | TODO | TODO | TODO | TODO | TODO |
| Metadata-enriched RAG | TODO | TODO | TODO | TODO | TODO |
| Provenance-aware RAG | TODO | TODO | TODO | TODO | TODO |

### 8.3 Operational Results

| Condition | Latency | Input Tokens | Output Tokens | Refusal Rate |
|---|---:|---:|---:|---:|
| Raw RAG | TODO | TODO | TODO | TODO |
| Cleaned RAG | TODO | TODO | TODO | TODO |
| Metadata-enriched RAG | TODO | TODO | TODO | TODO |
| Provenance-aware RAG | TODO | TODO | TODO | TODO |

## 9. Discussion

The expected outcome is that cleaning improves retrieval and metadata enrichment improves grounding. Cleaning should help because normalized text and section-aware chunking reduce noise. Metadata should help because it adds semantic anchors such as title, category, tags, and source identity. Provenance-aware prompting should reduce unsupported claims, although it may increase refusal rate.

If metadata enrichment improves citation accuracy more than answer correctness, that would still be valuable. It would suggest that metadata helps the generator explain where claims came from even when retrieval quality is similar. If cleaning improves Recall@k but not hallucination rate, the bottleneck may be generation rather than retrieval. If provenance-aware prompting reduces hallucination but makes answers shorter or more cautious, that tradeoff should be reported.

A key issue is mixed terminology. Persian technical articles often contain both translated and English terms. A user may ask about "توهم مدل" while the source uses "Hallucination". A user may ask about "پایگاه داده برداری" while the source uses "Vector Database". Metadata and tags can bridge these vocabulary differences.

The benchmark can also reveal which topics are hardest. Cybersecurity questions may require precise version or vulnerability details. AI questions may require conceptual explanation. Software architecture questions may require comparison and tradeoff reasoning. Multi-hop questions may remain difficult because they require retrieving evidence from multiple sections or articles.

## 10. Limitations

TekRAG-Persian is domain-specific. It focuses on Persian technical content and may not generalize to medicine, law, literature, or conversational QA. The first version may have a limited number of questions. Human evaluation introduces subjectivity. Results may depend on the embedding model, generator model, prompt design, and chunk size. If the dataset cannot be fully released, reproducibility may be limited.

These limitations should be reported directly. A narrow, honest benchmark is more useful than an exaggerated claim.

## 11. Ethics and Data Availability

The dataset should use only content that the authors have the right to process. It should exclude private user data, unpublished drafts, credentials, analytics logs, and personal information beyond public author metadata. If public article metadata is used, that should be stated.

Possible data release options:

1. release full corpus and QA pairs;
2. release only metadata, questions, expected answers, and chunk IDs;
3. release scripts and evaluation protocol so others can reproduce the benchmark on their own corpora.

The selected option should be explained in the final manuscript. If AI tools are used for editing or assistance, the selected journal's AI disclosure policy should be followed.

## 12. Conclusion

This article proposes TekRAG-Persian, a data-centric benchmark for evaluating hallucination in Persian technical RAG systems. The framework compares raw, cleaned, and metadata-enriched corpora to measure how data preparation affects retrieval accuracy, citation quality, and answer faithfulness. The main argument is that reliable Persian RAG requires more than a strong language model. It requires clean text, section-aware chunks, semantic metadata, and provenance.

The proposed benchmark gives researchers and practitioners a concrete way to measure those factors. If implemented and evaluated, it can support more trustworthy Persian technical assistants, better publisher knowledge bases, and more reproducible low-resource RAG research.

## Declarations to Complete Before Submission

### Funding

TODO: State funding source or write that the research received no external funding.

### Conflicts of Interest

TODO: Declare conflicts or state that there are no competing interests.

### Data Availability

TODO: State exactly what data will be released.

### Code Availability

TODO: Add repository link or release plan.

### Ethics Statement

TODO: Confirm that no private user data or credentials were used.

### AI-Assisted Writing Disclosure

TODO: Disclose any AI-assisted writing or editing if required by the target journal.

## References to Add Before Submission

The final manuscript should cite peer-reviewed work on:

- retrieval-augmented generation;
- hallucination in language models;
- data-centric AI;
- low-resource NLP;
- Persian NLP;
- question-answering benchmarks;
- information retrieval evaluation.

Do not submit without a real reference list formatted for the target journal.