Developing a reliable ChatGPT for European languages

Reliable, open, and sustainable—those are the qualities aimed for in a ChatGPT for European languages.

However, there are still big problems that need to be solved before ChatGPT can be considered beneficial to society. Marco Kuhlmann, professor at Linköping University in Sweden, develops methods to understand and computer-generate human speech. He is part of a project coordinated by Linköping University (LiU).

"Everyone was astounded when ChatGPT and later GPT-4 were released, as it was really fascinating how good they were, having been trained on such large data sets. They were suddenly able to solve tasks they had not been specifically trained for," Marco Kuhlmann says.

However, you must be able to trust the models, they must provide information that is true and not make things up. ChatGPT is very good at producing grammatically correct and convincing text, but the model has not been trained to generate texts that are factually correct.

Ensuring correctness and credibility is, therefore, among the most important issues for the researchers to solve in the three-year research project aptly named TrustLLM. LLM stands for Large Language Model, the term used for the type of language model that includes ChatGPT.

To better explain the problem, they can be compared with search engines. If you use a search engine, you still need to be cautious about the results it returns. But as it provides multiple alternatives, you can go through the list of search results to see if anything looks odd, maybe an unreliable source, for instance. You can make a selection, but when you use ChatGPT and the other models, you get only one reply and no opportunity to make your own decision as to which of the sources you trust.

The project is to benefit citizens and industry alike and is carried out in collaboration with organizations from Germany, Iceland, Denmark, Norway, and the Netherlands. One major problem with language models such as ChatGPT and others is that they are owned by large companies, often American, which creates a problem from a societal as well as a scientific perspective.

"With language models becoming increasingly commercialized, we have noted that companies publish fewer and fewer details on how they train their models, what data they use to train them, and what methods they use. This is where we can come in, with this project," says Marco Kuhlmann.

Training language models requires a lot of energy. They are also expensive to operate. The project will develop the model to be as efficient as possible, to make it a quick and sustainable tool using a minimum of energy.

"I think efficiency optimization and reducing energy requirement during training and during further development of a model, post-training, is very important, in terms of environmental impact as well as from a financial perspective. European companies don't have the same resources as the large American enterprises in this field," says Marco Kuhlmann.

He believes that the researchers may now be able to recreate methods that the companies are possibly already using and, above all, that the methods developed in the project should be published and then critically reviewed by the scientific community.

Marco Kuhlmann thinks that one of the reasons why the EU is investing in the TrustLLM project is so as not to fall too far behind. The project could result in a key technology in which Europe currently does not have as much competence as the US, especially not in the public sector. This competence can be found mainly in large international enterprises that are not always very transparent. Marco Kuhlmann hopes that TrustLLM will contribute to the EU reaching or even surpassing their level.

The technology in the project is to be made open and accessible so as to benefit many countries and companies in many ways. He points out that this is better also from a financial perspective. To develop large language models to respond to real needs, the project is looking at a number of concrete applications in medicine, media, vehicles and mobility, and public agencies.

"We have partners both in academia and industry, and they can pull in the same direction here. This openness is really something that concerns us all. I think that the fact that we are in this project with industry is also a reason why the EU wants to invest in this," says Marco Kuhlmann.

The researchers will train the language model on the largest amount of text so far in European artificial intelligence, AI. One important focus for the project is having an ethical approach by ensuring that personal data and copyright are protected in the amount of text used.

"We're working on solving problems that the entire research community is working on and is interested in progressing within, which makes this project exciting to me. I mainly contribute a research perspective. A lot of basic research needs to be done, while there are also strong financial incentives, so it's really a win-win situation," says Marco Kuhlmann.

Although the technologies to be developed by the project are supposed to be language-independent, the researchers will, at the final stages of the project, have developed a model for the Germanic languages Swedish, Danish, Icelandic, German, and Dutch.

The project is created in a kind of open ecosystem so that other researchers will then be able to take over, reuse and further develop language models for other European languages. Another objective is for the TrustLLM project and its surrounding ecosystem to contribute to making human-machine interaction more contextualized, which will have a large number of applications.

The aim is that the models developed will be the most powerful and reliable language models in Europe and that the project will be a major breakthrough in AI. In this way, the researchers hope to contribute to securing European independence in important AI technologies and create a new framework for European collaboration on large language models.

They want to lay the foundations for a European center for large language models and large-scale AI to maximize scientific, social, and financial effects.

Fredrik Heintz, professor of computer science at LiU, believes that developing its own large language models is essential for Europe, "We have to lead through innovation, not only through regulation. TrustLLM is an important step towards large-scale investments in Europe to make sure we take an active part and push our values in the form of new language models."

Provided by Linköping University