Is AI actually replacing traditional methodologies in Scientometrics? (and why we still have a long way to go)
Hi everyone,
I’ve recently completed a data curation and comparative study analyzing the adoption of Artificial Intelligence, Machine Learning, and LLMs in scientometric research methodologies.
I analyzed a corpus of 706 papers from two different venues:
- Quantitative Science Studies (QSS - Volumes 1 to 6): representing the international, flagship journal level.
- Brazilian Meeting on Bibliometrics and Scientometrics (EBBC - 2020, 2022, 2024): representing the Brazilian national state-of-the-art.
We filtered the papers to identify those that did not just study AI as a topic, but actively incorporated AI/ML/LLMs as the core methodological motor (replacing or enriching traditional statistics, manual curation, or classical text mining).
Here are some of the numbers we found:
- Overall AI Adoption Rate:
- QSS (International): 20.46% (80 out of 391 papers) used AI in their methodology.
- EBBC (Brazil): 3.49% (11 out of 315 papers) used AI.
- The Temporal Spike:
- Both venues saw a significant jump in 2024/2025. In EBBC, adoption went from a flat ~2% to over 5% in 2024. In QSS, it peaked at 24.59% in 2025. This is clearly the "ChatGPT effect" acting as a technological shortcut for researchers.
- The Methodological Focus:
- EBBC: Almost entirely concentrated in Natural Language Processing / Text Curation (90.9%)—mostly using LLMs out-of-the-box for text classification.
- QSS: Highly diversified, with Graph Neural Networks / Network Embeddings (41.2%) and predictive modeling leading the way.
The Core Observation (My CS / Industry Perspective)
Coming from a Computer Science background and having worked in the tech industry, these results left me with a mixed feeling:
- The Brazilian scene (EBBC) is still very raw. Most applications are extremely basic using OpenAI APIs or web interfaces to perform simple tasks that could often be done more robustly with deterministic scripts, or using prompt engineering in a very naive way without evaluating bias, token efficiency, or reproducibility.
- The international scene (QSS) is much more mature, but it still underperforms. Even at the highest level of academic publishing, I see researchers treating LLMs as magical black boxes. There is a noticeable lack of software engineering best practices. We see sub-optimal API pipelines, lack of structured outputs (like JSON schemas), no system evaluation frameworks, and a general lack of computational efficiency. It feels like many academics are lagging behind the industry standards of how to properly deploy, prompt, and scale AI-driven workflows.
It seems that while AI is indeed starting to replace or augment traditional bibliometric methodologies, the usability and efficiency of these tools among academics is still a massive bottleneck.
I’d love to hear your thoughts and experiences:
- How are you integrating AI/LLMs into your research pipelines?
- Have you noticed this gap between academic AI methodology and industry/CS standards?
- How can we, as a community, bridge this gap and help researchers from non-CS backgrounds use these advanced tools more efficiently?
Looking forward to the discussion!