Misinformed AI

While chatbots can amplify misinformation, they can also help set the record straight

It’s enough to make your head spin. The figure is staggering. 80%. The news is shocking. NewsGuard confirms it. Mistral AI’s Le Chat “provided false information in response to [malicious] instructions in 80% of cases” when asked to produce a propaganda article about the war in Iran. NewsGuard, an American company whose business includes selling anti-disinformation solutions, also notes that Anthropics’ flagship solution, “Claude, is struggling in the face of foreign influence.” In short, artificial intelligence (AI) solutions do not prevent manipulation; they may actually contribute to amplifying it. For Le Chat, this is a harsh blow for a tool branded a “European champion,” endorsed by the Élysée Palace, and adopted by the Ministry of the Armed Forces, all on the very day Mistral AI signed an agreement with major French regional daily newspaper groups. The study also shows that French speakers are more exposed than English speakers through this app, proof that malicious actors are using it as a tool for disinformation.

As chatbots are also establishing themselves as an additional media channel, one in ten French people already say they use them to stay informed on a daily basis, it is not surprising that they are also becoming a breeding ground for disinformation and interference. We must combat this there as well. Because AI solutions combine the virtue of consistency with the power of automation.

Toxic. When the source is tainted, the entire distribution channel becomes poisoned. The NewsGuard report shows how the Pravda network, comprising 370 sites, and Iranian state media produce and disseminate false information on a massive scale to flood search engines. This is undoubtedly a central point, all too often overlooked, inferences are compromised by the pollution of search indexes, where toxicity is far more effective than in the training datasets of the large language models (LLMs) themselves, which feed the AI systems without distinguishing between reliable sources. This is not a matter of bugs but of a planned and documented strategy. Silently ingesting data, bots do not distinguish between truth and falsehood, even though techniques for context-aware generation (RAG) exist to help limit the phenomenon. The fact remains that AIs organize the retrieval of words according to probabilistic rules, and the challenge for any malicious organization is to manipulate them.

The source does not define the label. The output is legitimized by bots that appear harmless in a neutral, reassuring, and often well-sourced tone. But before seeing the source, whether reliable or not, users read, above all, the responses from their AI assistants. Disinformation is changing its guise. It now harnesses the power of algorithmic synthesis.

For those engaged in disinformation and interference, whether states or organizations, chatbots, the new on-the-fly, on-demand printing press, can amplify any article or social media post in any language at minimal cost. Fake news, too, is undergoing its own industrial revolution.

Beyond the vulnerabilities inherent in all technologies, institutional endorsement seems to contribute to the very demands placed on the machine. Thus, the support provided by public authorities to Mistral AI in the name of strategic autonomy tends to transform trust, which, if perceived as unconditional, then appears suspicious, even though this is a serious company facing problems common to all major AI operators.

Indeed, no major model is immune. The degradation is systemic. A benchmarking of the main solutions would undoubtedly provide a better understanding of the phenomenon, to better inform users and organize the response. We must attack the problem at its root, in its foundations. Solutions exist.

While generated text, photos, and videos spread at the speed of light, tagging sources upstream, allowing content to be authenticated before it enters training datasets, represents a novel solution for establishing a true information immune system. It is up to stakeholders committed to verified information to harness existing technologies to ensure its reliability. This should not, however, prevent public organizations and large companies from implementing independent information audits and holding their employees accountable. Sovereignty without reliability would open yet another breach.

This article was originally published in French in l’Opinion