Scientists Secretly Using AI for Peer Review Despite Guidelines
Finn's Take· TL;DR- Over 50% of researchers use AI for peer review despite journal policies prohibiting it, with 59% using it to write reports.
- AI-generated reviews fail to provide meaningful feedback, miss key points, and contain factual errors compared to human standards.
- Researchers employ detection methods and evasion tactics in escalating cat-and-mouse game over AI use in academic publishing.
The Hidden AI Revolution in Scientific Publishing
Academic research faces a quiet revolution as more than 50% of researchers now use artificial intelligence tools while peer reviewing manuscripts , according to a comprehensive survey of 1,600 academics. This widespread adoption occurs despite many journals explicitly discouraging or prohibiting AI assistance in the peer review process, creating a disconnect between official policies and actual practice.
The findings reveal researchers are turning to AI for diverse tasks within peer review. Among those using AI, 59% employ it to help write their peer-review reports, while 29% use it to summarize manuscripts, identify gaps, or check references . Additionally, 28% use AI to flag potential signs of misconduct, such as plagiarism and image duplication .
Quality Concerns and Performance Gaps
Despite the widespread adoption, evidence suggests AI-generated reviews may fall short of human standards. Recent experiments demonstrate significant limitations in AI's ability to provide meaningful scientific feedback. One researcher found that GPT-5 could mimic the structure of peer-review reports and use polished language, but failed to produce constructive feedback and made factual errors, with even advanced prompts failing to improve performance .
The problem extends beyond individual experiments. At a major AI conference, 21% of manuscript reviews were found to be generated by artificial intelligence , raising serious questions about review quality. Researchers reported receiving AI-generated reviews that missed the point of papers, contained incorrect numerical results, and used odd expressions .
Detection and Gaming the System
The rise of AI in peer review has sparked an arms race between detection and evasion. Studies have identified buzzword adjectives that serve as hallmarks of AI-written text in peer-review reports , helping editors spot automated reviews. However, some researchers are fighting back with sophisticated countermeasures.
Scientists have been sneaking secret messages into their papers using white text or small fonts visible only to machines, attempting to trick AI tools into providing positive peer-review reports . This cat-and-mouse game highlights the growing tension between human oversight and machine assistance in academic publishing.
The Future of Scientific Review
The widespread adoption of AI in peer review reflects deeper challenges in academic publishing, including reviewer fatigue and the increasing volume of submissions. While large language models can check statistics, catch plagiarism, and verify citations, potentially freeing human attention for more critical tasks , concerns remain about their limitations.
As one expert noted, policies must adapt to reflect this "new reality" of researchers' increasing reliance on AI tools. The challenge ahead involves balancing the efficiency gains AI can provide with maintaining the rigorous standards that scientific peer review demands. The academic community must navigate this transition carefully, ensuring that technological assistance enhances rather than undermines the quality of scientific evaluation.