Retrieval quality comparison

01 / QUESTION

Use fixed questions and documents; compare retrieval hits, citations and answer correctness with and without retrieval.

02 / HYPOTHESIS

Retrieval may improve grounded answers when the corpus contains the answer; it may not help missing-evidence questions.

03 / VARIABLES

Independent: Retrieval on/off; tune retrieval parameters in separate comparisons.
Controlled: Questions, corpus snapshot, model version, prompt, sampling and rubric.
Dependent: Document hits, citation support, correctness and elapsed time.

04 / DATASET

NOT DEFINED YET

05 / BASELINE

The same model without retrieval, using identical questions and output format.

06 / METRICS

Planned: Recall@k with relevance labels, supported citations, manual scores and per-question time.

07 / SUCCESS CRITERIA

Compare paired outcomes and uncertainty. Define sample size and thresholds before running; require consistent improvement with supported citations.

08 / RISKS

Leakage, label bias, generation randomness and small samples. Analyze missing-evidence questions separately.

09 / REPRODUCE

Code, configuration and data will be recorded when the experiment runs.

METHOD

Retain the A4 ten-document/fourteen-question BM25 baseline; A5 adds 22 closed configuration-claim cases covering citations, revisions, scope, fields and coverage.

RESULT

A4: 11 of 12 answerable questions hit at 1/3; one unanswerable question still retrieves candidates. A5: citation existence accepts 13/22, including 10 contract violations; all 22 full-contract statuses match frozen labels.

LIMITATIONS

No generation model or online requests. Canonical field matching is not general entailment. The complete RAG/no-RAG answer comparison remains pending; registry and requested scope are trusted inputs.

KEEP EXPLORING