Backtrace Confidence Signals for Hallucination Reduction

Practical context

A proposed study of whether visible evidence helps reviewers detect unsupported answers.

Prototype project recall or document assistance without placing every available record in a prompt.

Implementation and research boundary

Retrieval score components and source metadata can make context selection inspectable. They are inputs to review, not calibrated probabilities that an answer is correct.

An evaluation should define unsupported claims, compare against a baseline and disclose the corpus, scoring rubric and disagreements. No hallucination-reduction percentage or validated confidence calibration is available in this note.

Research question and unit of analysis

The proposed question is whether visible evidence and ranking factors help a reviewer identify unsupported statements. Define the unit as an individual factual claim, not an entire answer or a retrieval hit. A record can be relevant yet wrong, and an answer can contain both supported and unsupported claims. Keep relevance scores, source authority labels, confidence judgments and observed factual correctness as separate variables.

A proposed controlled comparison

Use a versioned question set with answerable, unanswerable and contradictory-source cases. Compare the same answering workflow with and without the evidence display; keep the model, prompts and corpus fixed where possible. Have reviewers label claim support using a written rubric and record disagreements for adjudication. If judging without the evidence display is the control condition, prevent reviewers from seeing the treatment view first. These are proposed design choices, not a report of a conducted trial.

Metrics, calibration and reporting

Report unsupported-claim rate with its denominator, abstention rate and reviewer error, rather than turning a bounded ranking score into a probability. A proposed empirical rate is unsupported factual claims divided by adjudicated factual claims; exclude or separately count nonfactual statements according to the declared rubric. If confidence probabilities are introduced later, test calibration on held-out cases. Publish annotation guidance, corpus coverage, uncertainty and failure examples. Avoid selecting only answers with available citations, which would hide failures to retrieve any useful evidence.

Evaluation before adoption

Use a documented corpus with relevant, irrelevant, conflicting and missing evidence. Review selected context and bounded behavior alongside answer quality; a successful selection test alone is not a hallucination-reduction result.