News

Deutsch

What is still blue and where does the color purple begin? We don't agree on how to define some colors ourselves and can get into discussions about them. "Colors come in so many shades, and we humans perceive them subjectively. What is orange for some is more of a reddish tone for others," says Marie Bexte.

As a research assistant in computational linguistics at CATALPA, she is investigating methods for the automatic evaluation of image-based tasks for her doctoral thesis. Her supervisor is Prof. Dr. Torsten Zesch (Research Chair of Computational Linguistics). Junior Professor Andrea Horbach is also supporting her in her work.

Supporting lecturers

"We are researching how to automatically assess students' answers to learning tasks. This should take the pressure off lecturers so that they have more time for teaching or personal interaction with students," explains Marie Bexte. The students' answers are evaluated by an AI. However, this should not only be based on the text. "We are working on enabling the AI to also recognize images. The question can then also be an image, for example diagrams that students are asked to describe." There are various AI models that should make this possible. Marie Bexte examined nine different freely accessible AI systems for her research, including LLaVA, which works in a similar way to the well-known chatbot "ChatGPT".

Recognizing text and images in context

Recognizing text and images in context is still difficult for artificial intelligence. Bexte is therefore evaluating the depth of understanding of the AI models. How well can they recognize images? Colors play a major role in this. "That's why we wanted to know how well artificial intelligence can recognize colors when humans sometimes disagree." To do this, they created an evaluation dataset and examined how accurately the AI can name colors in the AI models. For part of the data set, they also analyzed the color values described by humans. The researchers noticed that people tend to have different opinions when describing colors. Some described the cat as gray, others as brown. "The AI already partially recognizes colors, it is at least going in a good direction, but currently not at the level required for teaching. It would be important for one colored bar to be longer than the other. But the AI's understanding doesn't go that far yet," says Bexte.

The research assistant explains her research with an example: "Essentially, I provide the AI model with an image and a text. In the case of the colors, this would be a black dog in a photo and the text would say 'a black dog is running across a meadow'. We then manipulate this sentence by saying it is a white dog and see whether the model recognizes this manipulation." The researchers only make minimal changes to the text and see how sensitively the AI reacts to them. "At the moment, we're at the point where it would be good if the AI recognized that the image and text don't match. That's not the case at the moment. In the best case, the AI can recognize the manipulation in the text and correct it automatically. In the example with the dog, it would be able to determine the color of its fur itself." Marie Bexte is optimistic. Artificial intelligence will continue to learn over the next few years and will hopefully soon be able to recognize colors more reliably. This would further relieve the burden on lecturers, including those on courses that involve more than just correcting text. It is also important for accessibility that people with a visual impairment can better rely on the help of an AI.

Learning could become more independent

The automatic correction of text tasks is already working well and is being used in practice. For example, in the "TOEFL test". This is a language test that checks knowledge of the English language. The answers are evaluated in parallel by a model and a human. Bexte would like to continue researching whether the AI can recognize visual elements well. "It would be great because you could enable students to learn even more independently. They get quick feedback from the AI and can then continue practising," says Marie Bexte. She is still writing her doctoral thesis, but will finish it soon. "Before that, I still have the opportunity to do research abroad. There I can work with a dataset on image descriptions. That fits in perfectly with my research."