
“You’re Right”: LLMs Will Get Wrong Climate Information If You Ask Them
Two persistent trends appear here: the latest frontier models tend to perform better than previous generations of that model, and larger, open-weight models perform better than smaller ones.
But the researchers didn’t just test accurate factual recall; they also prompted the models to assess whether the evidence supported or refuted specific claims. (For example, the models were asked whether the claim that “renewable energy investments kill jobs” is accurate, given evidence such as descriptions of employment and investment statistics.)
Here, the best models scored around 70 percent accuracy, while the worst models scored below 40 percent.

The 33 models are grouped into categories. Open weight models are listed by number of parameters (in billions).
Credit: Drobnjak, et al./UNESCO
The 33 models are grouped into categories. Open weight models are listed by number of parameters (in billions).
Credit: Drobnjak, et al./UNESCO
Finally, the researchers tested 656 questions about the potential impact of policies on global temperatures. Here, they could simply run each scenario in a common simulation tool and then compare its result to each LLM’s response.
Some questions asked whether a single policy (like banning nuclear power plants) would increase or decrease future warming – they were answered correctly 72 to 89 percent of the time. Others have asked how two policies might combine – the answers to these questions have been no better than a random guess. Ask How much a policy that increased or decreased future warming would produce even worse results.

Credit: Drobnjak, et al./UNESCO
Credit: Drobnjak, et al./UNESCO
Tell me what I want to hear
Another set of experiments using 15 of the models assessed what happens when you chat with the LLM. The researchers tried more than just answering “no, uh” or “no, that’s not the case.” They also tried to pretend to be a specific type of person.
Since the user identified as a climate scientist, policy advisor, or climate skeptic, they asked LLMs to judge the accuracy of some common false climate claims such as “CO2 is not the main driver of climate change. Initially, this claimed identity had no impact on the likelihood that the LLM would judge the statement correctly.
But then researchers began to respond. Responding by confidently repeating the false claim as if it were true was a moderate success for “climate scientists,” but not for “climate skeptics.” But presenting persistent disagreements and counterarguments over several rounds caused all 15 models to yield to a significant extent, reducing their accuracy by 30 to 35 percent.
Gn world