Permanent record
RIR-3060 · Logged 14 October 2026
Do AI quality scores agree with South African experts on South African research?
Language-model scores of research quality correlate only weakly with expert judgement at the level of single articles, and the strength of that link varies by field. Evidence from African research systems is almost absent. The idea compares AI quality scores for South African articles with blinded expert ratings and tests whether prompting for national relevance changes the agreement.
In plain words
AI tools are starting to estimate how good a research paper is. This study would check whether those scores match what South African experts think of local research, and whether asking the AI about local value changes anything.
Primary research question
How closely do large-language-model quality scores for South African research articles agree with blinded ratings by South African disciplinary experts, and does prompting for national relevance change that agreement?
Knowledge gap
What remains worth asking
Thelwall (2026) found that ChatGPT quality scores align only modestly with research quality for single articles, with large differences between fields. A study from Mauritius found near-zero agreement for national value. No comparable test exists for South Africa.
Potential contribution
Why it may matter
Universities and funders considering AI-assisted evaluation need to know whether it reflects local expert judgement before it influences careers or funding.
Academic placement
Academic fields and topic tags
Study frame: A sample of South African journal articles in two or three fields, scored by language models and by blinded South African experts. · Method signals: Expert rating study, Agreement analysis, Bibliometrics
Possible study pathways
One question, different levels
Research master’s
Compare AI and expert scores for a sample of articles in one field.
Doctoral
Compare several fields and test prompts for national relevance against expert ratings.
- originality
- Moderate
- methodology
- Moderate
- Data access
- Moderate
- ethics
- Accessible
Quality and audit
85/100 quality
Safeguard-based audit was not recorded under the prior standard.
- Fills a named geographic gap
- Clear evidence path
- Policy relevance
This older record can be checked against the current standard on request. It changes only if every academic gate and independent safeguard passes.
Sign-in required · uses one AI creditProvenance
Research Idea curation
- DOI and bibliographic metadata independently resolved through Crossref
- The source reports field differences in AI–quality agreement
- The South African comparison is an inferred direction
The public contributor code contains no name or account email.
APA 7 source
Thelwall, M. (2026). In which fields do ChatGPT scores align more closely with research quality than do citation rates? Journal of Data and Information Science. https://doi.org/10.1515/jdis-2026-0058
Results by field and limitations
This is an AI-inferred direction. The source authors did not state it as a recommendation.
Open source ↗