📊 Full opportunity report: Minerva. The opposite path. on ThorstenMeyerAI.com — validation score, market gap, and execution plan.
TL;DR
Italy’s Minerva-3B, a European sovereign LLM trained from scratch on extensive Italian data, achieved low performance on Italian academic benchmarks, highlighting the importance of scale in language models. This challenges assumptions about investment sufficiency.
Italy’s Minerva-3B, a large-scale sovereign language model trained from scratch on 2.5 trillion tokens with approximately 50% Italian content, scored just 4.9% on the INVALSI Italian school-exam benchmark, highlighting significant challenges in achieving language-specific knowledge depth despite substantial investment.
The Minerva project, led by Sapienza University of Rome and supported by Italy’s national research and supercomputing infrastructure, trained models ranging from 350 million to 7 billion parameters. Despite the extensive training data and dedicated infrastructure, Minerva-3B’s performance on the Italian academic benchmark was near chance, with a score of only 4.9%. This result contrasts sharply with the model’s impressive technical metrics and performance on other Italian benchmarks, indicating that scale alone may not suffice for complex language understanding tasks.
Researchers have concluded that while dataset composition and size are important, the overall number of parameters and training scale are critical for handling complex language tasks. The findings suggest that even substantial native-language investment may not produce the expected depth of country-specific knowledge at current model sizes, prompting a re-evaluation of strategic investment levels in European sovereign-LM projects.
Minerva.
The opposite
path.
Italy spent years building a European sovereign LLM from scratch. Then Minerva-3B scored 4.9% on the INVALSI Italian school exam.
Where AMÁLIA layered Portuguese specialization onto a multilingual foundation, Minerva trained from scratch on 2.5 trillion tokens with approximately 50% Italian content. Where AMÁLIA’s weights are not yet public, Minerva published weights, training data, and code as truly-open from day one. By every institutional measure, the Italian approach worked. But the empirical results contain a finding the press coverage has been quiet about — and it has implications that extend well beyond Italy.
Same problem. Opposite path.
European sovereign-LLM development has two primary architectural approaches. Italy chose from scratch with substantial native-language foundation. Portugal chose continuation pre-training of a multilingual model. The structural comparison surfaces what each commitment actually requires operationally.
The comparison is not “Italy did it better than Portugal.” Both projects respond to the same structural problem with different architectural strategies under different institutional and economic constraints. Italy’s national-AI investment is structurally larger by an order of magnitude — and Minerva is the visible artifact of that scale.
Engineering a Small AI Language Model: Training, Evaluation, and Deployment Without Myth
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
4.9% on INVALSI. The bitter lesson surfaces.
In June 2024, researchers evaluated Minerva-3B on the Italian school-exam benchmark. The result was unambiguous. This is not a critique of Minerva — it is a critique of the public discourse around what Minerva’s empirical results actually demonstrate.
HPE NVIDIA Tesla V100 32GB HBM2 PCIe 3.0 x16 Passive GPU Computational Accelerator for AI Machine Learning HPC Deep Learning 699-2G500-0216-400 (Renewed)
NVIDIA Volta GV100 Architecture — 5,120 CUDA Cores, 640 1st-Gen Tensor Cores delivering 14 TFLOPS FP32 and 112…
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
350M to 7B. Four parameter scales, one architecture.
The Minerva model family covers four parameter tiers, each with specific training corpora. Each scale level reveals what the from-scratch path actually requires at different operating points.
Italian + English
100B English
~50% English
+ 200B code
LLM Systems Engineering: Training and Building Large Language Models – Engineering AI Models Through Fine-Tuning, Continued Pretraining, and From-Scratch Development (AI Engineering)
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three answers. Same question.
Minerva, AMÁLIA, and OpenEuroLLM represent the three operational answers to the European sovereign-LLM question. Each makes different architectural and institutional bets. The strategic discourse benefits from treating all three as data points in the same empirical experiment.
AI research supercomputing infrastructure
As an affiliate, we earn on qualifying purchases.
As an affiliate, we earn on qualifying purchases.
Three standards the movement should adopt.
The structural critique generalizes beyond Minerva. The European sovereign-LLM movement benefits from internalizing these lessons across every subsequent national project. Italy modeled the openness standard; the movement should adopt it as norm.
Minerva is one valid answer to the European sovereign-LLM question. AMÁLIA is another. OpenEuroLLM is potentially a third. The strategic discourse benefits from treating all three as data points in the same empirical experiment rather than as competing national-prestige projects. More analysis like this is needed. Not less.
Implications of Scale and Investment in Sovereign LLMs
The low performance of Minerva-3B on academic tests raises questions about the adequacy of current investment levels in European sovereign language models. It suggests that to develop models with genuine country-specific expertise, European projects may need to significantly increase their scale and resource commitments. This challenges the narrative that smaller, locally trained models can suffice for national language understanding and emphasizes the importance of large-scale data and parameters for complex language comprehension.
These findings impact policy and strategic decisions across Europe, as they highlight potential limitations of current approaches and the need for a more realistic understanding of the scale required to produce truly effective language models tailored to national languages and knowledge.
Italy’s Strategy and the Scale of Language Model Development
Italy’s Minerva project represents a deliberate choice to train a language model from scratch on a vast corpus of Italian data, utilizing national infrastructure and funding. This approach contrasts with other European efforts like Portugal’s AMÁLIA, which opted for continuation training on multilingual foundations. Despite the significant investment—2.5 trillion tokens, 128 GPUs on the Leonardo supercomputer, and open weights—the empirical results reveal limitations in the depth of language understanding achievable at current scales.
Previous efforts have often assumed that native-language data and increased parameters would naturally lead to better performance. However, Minerva’s results suggest that scale alone may not be sufficient, and that the structural and resource commitments need to be substantially larger to realize the desired country-specific expertise.
“Even with extensive training on Italian data, the model’s performance on academic benchmarks remains near chance, indicating the need for larger models or different strategies.”
— Research team member, Orlando et al., CLiC-it 2024
Unresolved Questions About Model Scaling and Effectiveness
It remains unclear whether increasing the number of parameters, training data, or both will significantly improve Minerva’s performance on complex language tasks. The ongoing research aims to determine the scale necessary to produce meaningful country-specific knowledge, but definitive thresholds have not yet been established.
Additionally, the generalizability of these findings to other European languages and models is still under investigation, and the optimal balance between data, parameters, and architecture remains an open question.
Future Research and Scaling Efforts for European Sovereign LLMs
The Minerva team is continuing to iterate on training methodologies, including ongoing projects with larger models and refined datasets. Researchers aim to test whether further scaling can bridge the knowledge gap observed in academic benchmarks. Policy discussions are also expected to consider whether current investments are sufficient for developing truly effective country-specific language models.
Further empirical results and model releases are anticipated over the next 12-18 months, which will clarify whether increased scale can produce the desired depth of country-specific expertise.
Key Questions
Why did Minerva perform poorly on Italian academic tests despite extensive training?
Research indicates that dataset size and number of parameters are critical for complex language understanding. Despite large-scale training, Minerva’s current scale may be insufficient to develop deep country-specific knowledge, especially for academic content.
Does this mean European sovereign LLMs are ineffective?
Not necessarily. The results highlight that current models at existing scales face limitations. Increasing investment and scaling may be needed to achieve meaningful performance on complex tasks.
What are the implications for European AI policy?
Policymakers may need to reconsider funding levels and infrastructure support, emphasizing larger-scale models and more extensive data collection to develop effective national language models.
Is the low performance specific to Italian or applicable to other languages?
While the findings are specific to Italian in this context, similar challenges could arise for other languages, especially those with less extensive training data or smaller model sizes. Further research is needed.
Source: ThorstenMeyerAI.com
