Microsoft Research conducted a large randomised experiment during the 2026 International Conference on Machine Learning (ICML) to test review policies. The conference received 24,661 submitted papers, which were assessed by 17,886 reviewers.
Reviewers faced a choice between two review policies: one that banned Large Language Models (LLMs) or one that permitted LLMs for understanding, checking related work, and polishing writing. Under the stricter policy, the acceptance rate stood at 27 per cent.
Conversely, the permissive policy yielded a 26.5 per cent acceptance rate. The average review score reached 3.31 under the stricter policy.
The average review score was 3.32 under the more permissive policy. Reviews under the permissive policy showed an average length increase of 5.5 to 7 per cent.
A significant finding revealed that 22.5 per cent of reviewers admitted using an LLM despite explicit instructions forbidding it. Miro Dudík at Microsoft Research stated that the self-reported non-compliance of 23 per cent is definitely higher than what he was anticipating.
Kayvan Kousha at the University of Wolverhampton, UK, commented that banning AI in peer review is very difficult to enforce. Analysis showed that 52.2 per cent of reviews under the conservative policy were classified as fully human-written.
In contrast, 37.0 per cent of reviews under the permissive policy were classified as fully human-written.
