Journalist or robot? Testing reporters on ingress summarizations

Many were not able to recognize robot generated summaries correctly when a team from the research center NorwAI tested the participants at the NxtMedia Conference whether they could distinguish between handwritten and robot generated summaries in "Klæbuposten".

Using Kahoot, the participants could choose which text summaries they thought was human-made and which was written via a trained Open AI GPT-3 model. It is not always easy to know which of the three examples is which.

The test at the NxtMedia Conference showed that most participants at the conference were able to distinguish between the summaries they were presented to.

- Even though the majority of participants managed to spot the summaries written by a journalist, many were fooled. The current state of large language models is impressive and mind-blowing. In the near future, we will see even more sophisticated AI models with improved capabilities for the Norwegian language, says Jon Espen Ingvaldsen, who is an adjunct associate professor at NTNU’s NorwAI Center.

With the help of Kahoot, it was easy for the participants to vote on the options:

- We used three different articles from “Klæbuposten”. We used a commercial and publicly available language model from OpenAI. Even though the model is not specialized for Nordic languages it handles these languages quite well. We ran an experiment in our lab where we fine-tuned the model for news summaries, says Jon Espen Ingvaldsen.

With 4-5000 Wikipedia articles, both full text and ledes, the model learned to create compressed excerpts that were shown to the public.

NorwAI, which has language models as one of its main areas of work, will take these experiences and results from OpenAI and other commercially available tools to benchmark them against the research center's own developed language models.

In the center NTNU collaborates with the Universities of Oslo and Stavanger on the development of its own mega-large Norwegian language model in close collaboration with partners such as Schibsted, NRK, DnB and Sparebank1 SMN. NorwAI also collaborates with similar innovation initiatives and research centers in the other Nordic countries.