📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B model, trained from scratch on a large dataset, underperformed on Italian school benchmarks despite impressive technical results. This reveals the importance of scale in developing country-specific language models.
Italy’s Minerva-3B language model, trained entirely from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored only 4.9% on the INVALSI Italian school-exam benchmark, despite significant institutional investment and open data sharing. This performance challenges assumptions about the effectiveness of large-scale native-language training for complex language tasks.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s National Research Council and the CINECA supercomputing consortium, trained models ranging from 350 million to 7 billion parameters using 128 GPUs on the Leonardo supercomputer. The project aimed to create a truly open, native-language LLM with publicly available weights, data, and code, contrasting with other European efforts like Portugal’s AMÁLIA.
While Minerva’s technical achievements include outperforming comparable multilingual models on Italian benchmarks, its performance on the INVALSI school exams was strikingly low, with a score near chance. Researchers noted that despite the large dataset and native focus, the model’s ability to handle complex academic content remains limited, suggesting that scale and data alone may not suffice for deep language understanding.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Building A large language model with Ai: A Practical Guide to Structuring LLM Systems from Scratch Using Reverse-Engineering Techniques
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
HPE NVIDIA Tesla V100 32GB HBM2 PCIe 3.0 x16 Passive GPU Computational Accelerator for AI Machine Learning HPC Deep Learning 699-2G500-0216-400 (Renewed)
NVIDIA Volta GV100 Architecture — 5,120 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
From Weights to Wisdom: The Complete Guide to Running and Adapting Opensource AI Models
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
Supercomputing for Artificial Intelligence: Foundations, Architectures, and Scaling Deep Learning Workloads
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications for National AI Investment Strategies
The results from Minerva demonstrate that even substantial investment in native-language data and training from scratch may not produce models capable of mastering complex, country-specific knowledge and tasks. This finding questions the assumption that scaling data and parameters alone will yield effective national language models, prompting a reevaluation of European sovereign-LLM strategies and funding allocations.
It underscores the need for a nuanced approach that considers not only dataset size and model scale but also the quality, diversity, and task-specific training. Policymakers and researchers must recognize that achieving meaningful country-knowledge depth may require different or additional investments beyond simply scaling up model size.
European Sovereign-LLMs and the Scale Debate
The European sovereign-LLM movement has been characterized by contrasting approaches: Portugal’s AMÁLIA, which layered Portuguese specialization onto a multilingual foundation, and Italy’s Minerva, which trained from scratch on a large native dataset. While Italy’s approach resulted in impressive technical benchmarks, its poor performance on academic content reveals a structural challenge in scaling language models to meet complex national needs.
Prior to Minerva, efforts like AMÁLIA focused on continuation pre-training with smaller, more targeted datasets, but the results have not yet demonstrated the depth of country-specific knowledge. The recent Minerva evaluation adds a critical data point: large native datasets and extensive parameter counts do not automatically translate into academic or complex language task proficiency.
“While our models outperform multilingual benchmarks, their limited performance on academic tests highlights the need for targeted, possibly task-specific training strategies.”
— Research team behind Minerva
Unresolved Questions About Scale and Effectiveness
It remains unclear whether further scaling, different training methodologies, or additional data curation could improve Minerva’s performance on complex, academic tasks. The current results suggest the need for more nuanced research into the relationship between dataset composition, model size, and task-specific proficiency.
Next Steps in European Sovereign-LLM Development
The Minerva team plans to continue iterating on their models, including ongoing experiments with continual training and dataset refinement. Policymakers and researchers are likely to reassess funding priorities and strategies, emphasizing the importance of targeted, high-quality data and specialized training for national language models. Further evaluations on different benchmarks and real-world tasks are expected to clarify how to best achieve effective country-specific AI capabilities.
Key Questions
Why did Minerva perform poorly on the Italian school benchmarks?
Despite large-scale native data and significant investment, the model’s limited performance suggests that sheer size and dataset scale may not be sufficient for mastering complex academic content. More targeted or task-specific training might be necessary.
Does this mean European efforts to develop sovereign LLMs are doomed?
Not necessarily. The results highlight challenges but also provide valuable insights into the scale and investment needed. Ongoing research and iteration are expected to refine these approaches.
What does this imply for future national AI projects?
It suggests that investments should balance scale with data quality, diversity, and task-specific training to develop models capable of deep country-specific knowledge and understanding.
Are there alternative approaches to improve performance?
Yes, strategies such as targeted fine-tuning, multi-task training, and incorporating expert-curated datasets could enhance models’ ability to handle complex, academic language tasks.
Source: ThorstenMeyerAI.com
