‘Hallucination’ and ‘confabulation’ are the terms used to describe those outputs of text-based generative AI that aren't true or that it fabricates*. Because these hallucinations are often embedded in outputs that also contain factually correct information, they can be difficult to identify.

One specific way this happens is when the AI claims to be using information from a study that doesn't exist, or from an expert when the expert exists but didn’t write the sentence, article or book that the generative AI claims they wrote. This is a significant threat to research integrity because we usually rely on such sources to check if something is true or not. So, if the generative AI tool makes up a source, it can make something untrue seem legitimate. It's often very hard to identify where a citation or reference is real or not, and can take a lot of work to identify whether a supposed source even exists.

It is important therefore to check any facts or claims made in generative AI output, especially if you're using them for your studies or sharing them with others. Just because something is generated by a powerful piece of software doesn't mean it's accurate.

A very blurry photo of a woman in a fur-trimmed coat.

* Hicks et al. (2024) disagree with calling errors and mistakes generated by generative AI "hallucinations" because this anthropomorphising allows the creators of generative AI tools to avoid taking responsibility for faulty outputs from the systems they have created. They also argue that when experienced by humans, hallucinations are a deviance from a 'normal' state, whereas generative AI functions in the same way regardless of whether or not the information it outputs is correct or accurate, it simply follows its same process. Additionally, they argue that calling errors hallucinations "feeds in to overblown hype about their abilities (p.37) and may lead to people misunderstanding what is happening when it gets things right - the machine does not 'know' whether the information is correct or not.

Source: Hicks, M.T., Humphries, J. and Slater, J. (2024) ‘ChatGPT is bullshit’, Ethics and Information Technology, 26(2), p. 38. Available at: https://doi.org/10.1007/s10676-024-09775-5 (Accessed: 19 June 2024).

Image: Ehimetalor Akhere Unuabona on Unsplash