nikGo

Engineering, AI, & Cognition

Retrieval-Augmented Generation as a Capability Multiplier for Research Tasks

Controlled evidence: same-model gains, and small-model-with-RAG beating large-model-without-RAG

Executive summary

Two controlled experiments tested whether Retrieval-Augmented Generation (RAG) improves research-grade outputs:

Across both experiments, using RAG produced statistically significant improvements in 4 out of 5 evaluation dimensions (factual correctness, completeness, hallucination risk, and academic response). Coherence was the only dimension without statistically significant change.

These results align with recent research showing RAG improves factual reliability and mitigates hallucinations by grounding generation in retrieved evidence. (arXiv) They also align with findings that retrieval can fail to improve (or can even harm) aspects like response quality or integration unless carefully configured—suggesting coherence is a system-design problem, not an automatic win. (ACL Anthology)

NLP analysis of outputs shows RAG responses are significantly longer and more comprehensive, but harder to read (more academic and denser). This is a trade-off that product teams should plan for via structured prompting and post-processing.

Abstract

I evaluate Retrieval-Augmented Generation (RAG) as a system-level intervention for research-style synthesis tasks. In Experiment 1, I compare Gemini 3 Flash with RAG to Gemini 3 Flash without RAG. In Experiment 2, I compare Gemini 3 Flash with RAG against Gemini 3 Pro without RAG. Outputs are assessed via blinded pairwise evaluation by an LLM judge (ChatGPT 5.2 Thinking) across five dimensions: factual correctness, completeness, coherence, hallucination risk (lower risk scores higher), and academic response. Using Wilcoxon signed-rank tests on paired differences, I find statistically significant improvements from RAG in 4/5 dimensions in both experiments; coherence does not show statistically significant differences. Readability analyses indicate RAG outputs are significantly longer and more academically styled (worse readability scores). The results support the conclusion that RAG can act as a capability multiplier: improving research performance on the same model and enabling smaller models to outperform larger ones when retrieval is absent.

RAG is increasingly understood as more than “adding sources.” It is a strategy for externalizing knowledge access while letting the model focus on synthesis, reasoning, and explanation. This framing is consistent with a growing body of work showing that retrieval-grounding improves factual accuracy and reduces hallucinations across domains. (arXiv)

At the same time, literature also highlights that retrieval is not uniformly beneficial: under some conditions, retrieval can fail to help or can introduce mismatch between retrieved context and what the model needs, degrading downstream quality. (ACL Anthology) Approaches like evidence refinement or better selection/condensation aim to reduce such mismatch and improve reliability—particularly when multiple documents are retrieved. (ACL Anthology)

Finally, best-practice studies emphasize that RAG performance depends heavily on configuration choices (prompting, chunking, retrieval strategy, and relevance/precision trade-offs). (ACL Anthology)

2. Experimental design

2.1 Research questions

All experiments used three research prompts:

  1. Selective attention and conscious awareness
  1. Reconstructive vs reproductive memory + consequences
  1. Sleep, consolidation, and forgetting

2.2 RAG corpus

2.3 Generation conditions

Experiment 1: Flash+RAG vs Flash (no retrieval) Experiment 2: Flash+RAG vs Pro (no retrieval)

Generation settings:

2.4 Evaluation protocol (blinded, pairwise, randomized ordering)

Judge: ChatGPT 5.2 Thinking (LLM-as-judge)

Each evaluation compares two answers using an integer signed scale (−10…+10) on:

Ordering is randomized so the judge sees the two answers as A/B in random positions. Internally, Answer X is always the RAG output, but X may appear as A or B on any trial.

2.5 Statistics

3. Results

3.1 Main finding: RAG improves 4/5 dimensions in both experiments

Across both experiments, RAG produces statistically significant improvements in:

Coherence does not show statistically significant change.

This pattern is consistent with prior work: RAG tends to improve factuality and reduce hallucinations by grounding the model in retrieved evidence, while higher-level discourse qualities (like coherence) often require additional system design (prompting, evidence selection, refinement). (arXiv)

3.2 Experiment 1: Gemini 3 Flash (RAG) vs Gemini 3 Flash (no RAG)

Significant improvements were observed in 4/5 dimensions (excluding coherence).

Overall results for Experiment 1 Figure 1: Overall dimension deltas (Flash+RAG vs Flash without RAG), with significance markers

3.3 Experiment 2: Gemini 3 Flash (RAG) vs Gemini 3 Pro (no RAG)

Despite Pro offering substantially larger capacity, Flash+RAG still shows statistically significant improvements in 4/5 dimensions.

Overall results for Experiment 2 Figure 2: Overall dimension deltas (Flash+RAG vs Pro without RAG), with significance markers

3.4 Coherence: why it likely didn’t move

The coherence null result is not surprising if you treat RAG as “evidence injection.” Retrieval increases breadth and factual anchoring, but it can also:

Research explicitly documents that retrieval can help or hurt depending on conditions, and that mismatches between retrieved content and required knowledge can degrade output quality. (ACL Anthology) Best-practice work further emphasizes that prompt formulation and retrieval configuration remain crucial even inside RAG systems. (ACL Anthology)

Practical implication: RAG is not a coherence solution. It’s a grounding/completeness solution. Coherence needs structure (templates, synthesis constraints, or a post-pass).

4. NLP analysis: longer, denser, more academic (and harder to read)

Your NLP analysis shows RAG outputs are:

This matches what you’d expect from retrieval-grounded synthesis: the system has more evidence available, so it produces more detailed, qualified answers. In applied domains, retrieval-grounded systems are often used explicitly to increase reliability and coverage (e.g., medical/public health and fact-checking settings). (JMIR)

Experiment 1 Length and Readability comparisions Figure 3: Readability + length comparison panels (Flash+RAG vs Flash without RAG)

Experiment 2 Lengtha and Readability comparisons Figure 4: Readability + length comparison panels (Flash+RAG vs Pro without RAG)

5. Implications for AI industry practice

5.1 Retrieval can substitute for scale in research workflows

Experiment 2 supports a blunt operational conclusion: for research-style tasks, a smaller model with high-quality retrieval can outperform a larger model without retrieval (on factuality, completeness, and hallucination-related measures).

This complements external evidence that RAG setups can offer performance advantages and favorable trade-offs between smaller and larger models depending on retrieval quality and evaluation criteria. (MDPI)

5.2 Treat RAG as a system, not a feature toggle

Recent work emphasizes that RAG performance depends heavily on:

Given the coherence and readability trade-offs, the practical stack for research outputs is:

  1. Retrieve (favor relevance + coverage)
  2. Synthesize (claims → evidence → caveats)
  3. Compress/Rewrite for audience (preserve meaning, improve clarity)
  4. Optional: evidence consistency checks / refinement when using many sources (ACL Anthology)

6. Limitations

  1. Only three prompts (though each evaluated multiple times).
  2. Domain concentrated in cognitive science/psychology.
  3. LLM-as-judge evaluation: strong for consistency, but should be paired with human rating replication in future work.
  4. Retrieval budget is a meaningful part of the intervention; results are about system behavior (model + retrieval), not model weights alone.
  5. Coherence and readability are not guaranteed improvements under retrieval without additional design work. (ACL Anthology)

7. Conclusion

Across two experiments, RAG yields statistically significant gains in factual correctness, completeness, hallucination risk, and academic response, both when holding the base model constant and when comparing a smaller RAG system against a larger non-RAG model. Coherence does not significantly change, reinforcing a key engineering lesson: retrieval improves evidence and coverage; structure improves coherence.

These findings match and extend recent literature showing that retrieval grounding improves factual reliability and reduces hallucinations, while system configuration and refinement determine whether those gains translate into consistently well-structured answers. (arXiv)

References