Requirements for Large Language Models

Requirements for Large Language Models

If you are to build an environment for training and operation of Norwegian, commercially available language models, you must have access to the following resources:

  • Competence and experience. 

NTNU has the largest production of AI experts at bachelor's, master's and PhD level in Norway, and many are also trained at, for example, the University of Oslo. There is, however, little doubt that the educational capacity for artificial intelligence in Norway is too small. Although around 200 NTNU students annually write master's theses on artificial intelligence, relatively few of these works directly with language models. In Norway, it is currently only the University of Oslo, the National Library and NTNU that work with the really large language models.

  • Training data 

There are many indications that copyright-protected data will be necessary to train Norwegian models that are suitable for general use. 
The National Library currently manages several textual data sets that support language model training, and they sit on a lot of expertise in collecting, building and washing data sets. The universities largely use the National Library for access to Norwegian data.

  • Calculation resources (Compute) 

Apart from NTNU's Idun, the universities have limited own resources for language model training.
Sigma2 has significant supercomputers, but these too are not powerful enough for large language models. 
LUMI in Finland is an opportunity for research work with language models but cannot be used for operation and maintenance of models for regular use.

  • People 

Language models often have to go through an adjustment (alignment) in order to function in a more humane and responsible way. This is done by having a large group of people quality assess and rank responses from the model and then generalize these assessments to characterize all generated text from the model (reinforcement learning). This is costly and requires access to a representative sample of people.

Four people in a meeting room.
Associate Professor Nhien Nguyen leads NorwAI’s work package for building an Innovation Eco System. Here she discusses with NorLLM core team members, from left Jon Espen Ingvaldsen, Terje Brasethvik and Jon Atle Gulla.