A team of researchers from MIT and Columbia University presented around 200 participants with a set of statements generated by OpenAI’s GPT-3 and asked them to determine whether they made sense logically. A statement might be something like “Video games cause people to be aggressive in the real world. A gamer stabbed another after being beaten in the online game Counter-Strike.”

Participants were divided into three groups. The first group’s statements came with no explanation at all. The second group’s statements each came with an explanation noting why it was or wasn’t logical. And the third group’s statements each came with a question that prompted readers to check the logic themselves.

The researchers found that the group presented with questions scored higher than the other two groups in noticing when the AI’s logic didn’t add up.

The question method also made people feel more in charge of decisions made with AI, and researchers say it can reduce the risk of overdependence on AI-generated information, according to a new peer-reviewed paper presented at the CHI Conference on Human Factors in Computing Systems in Hamburg, Germany. 

When people were given a ready-made answer, they were more likely to follow the logic of the AI system, but when the AI posed a question, “people said that the AI system made them question their reactions more and help them think harder,” says MIT’s Valdemar Danry, one of the researchers behind the study.

“A big win for us was actually seeing that people felt that they were the ones who arrived at the answers and that they were in charge of what was happening. And that they had the agency and capabilities of doing that,” he says. 

The researchers hope their method could help develop people’s critical thinking skills as they use AI chatbots in school or when searching for information online. 

They wanted to show that you can train a model that doesn’t just provide answers but helps engage their own critical thinking, says Pat Pataranutaporn, another MIT researcher who worked on the paper.