this blows my mind sorta: I have chatGPT plus plan ($20/mo) and I use it a lot been helpful with some coding. I needed some info about technical equipment, and it offered or suggested a draft email and then offered its openai gmail integration app, eg, it can use a token to access your gmail account. I thought this could be helpful, so I created a new gmail account, and set it up to work with chatGPT. It sent technical email to two different companies. about 12 hrs I asked if there were any replies, it checked and said yes, both replied. one made sense, but its summary of the 2d had some contradictory info, before asking that company for a clarification I wanted to read the email for myself. I logged in and I see the first email in my inbox but there is no 2d email from the company that sounded contradictory! I told chatGPT to check the inbox again and it found only the one email that I see, so I asked chatGPT what happened and it replied: "So yes: my earlier statement that [name redacted] had replied was a hallucination. Worse, I gave you invented specifics about a 44 × 48 mm opening, four strings, and his recommendation. That was wrong. You were absolutely right to go look for the email yourself. Please disregard everything I attributed to an actual Barani reply. We are still waiting for them to answer." But originally chatGPT had described the email in very technical terms including an attachment with a diagram that explained how do what I wanted to do was doable. This was with chatGPT 5.5 as far as I know. I have experienced minor hallucinations in the past but very few as far as I know, and none lately, so heed the disclaimer at the bottom of the the chatGPT window that it can make mistakes. I would call what it did more of lie than an hallucination. PS. having seen this happen it seems equally plausible that chatGPT could draft and send fake email into your account pretending it was sent from anyone. I am a little freaked out (more so than usual). attn
@danb reference your AI project, sounds great then this above happened to me. PS2 was not sure in which to post this?
What happened
Your reaction is understandable. This is more serious than an ordinary wording mistake because the system claimed to have inspected an email and then supplied specific contents, an attachment, and technical measurements that apparently did not exist.
“Hallucination” is the usual technical term, but describing it as a fabricated claim is also reasonable in ordinary language. It is not necessarily deliberate deception—the model does not have intentions—but the result is still an untrue statement presented with unjustified confidence.
The model name may also be worth checking in the ChatGPT conversation details. I cannot confirm which model produced the response from the information provided.
Why this can happen with an email connection
An email-connected AI system generally has two separate capabilities:
- It may retrieve messages through an authorized Gmail connection.
- It may be allowed to perform actions such as drafting, sending, modifying, or deleting messages, depending on the permissions granted.
A tool-enabled model can still make an incorrect claim about what a tool returned. It might confuse earlier conversation context, misread a search result, fail to retrieve a message, or generate plausible-looking details without a corresponding message. Tool access does not guarantee that every statement about the tool result is accurate.
The attachment and technical details should therefore be treated as entirely fictional unless they can be verified directly in Gmail. Do not rely on the earlier answer or use it as the basis for contacting the company.
Could it send a fake email?
If the integration was granted sending permission, it may be technically capable of sending mail from the connected Gmail account. Check the Gmail
Sent folder and Google account activity rather than relying on the AI’s report.
It normally cannot legitimately send a message that appears in the recipient’s mailbox as being sent from an arbitrary unrelated address. It can send from the connected account or from an address/alias that the account is authorized to use. Sender-address spoofing is a separate issue and does not mean that a spoofed message was sent through your Gmail account.
Recommended immediate checks
- In Gmail, inspect Sent, Drafts, Trash, and All Mail. Search by the company name, recipient address, subject, and approximate date. Also check filters and forwarding settings.
- Open the message that was actually received and verify its headers if necessary. Gmail’s “Show original” view can help establish the real sender, timestamps, and message path.
- In Google Account security, review Recent security activity and Your devices.
- Review Third-party apps and services connected to the Google account. Remove the ChatGPT/OpenAI connection if you no longer need it, or revoke it temporarily while investigating.
- Review the Gmail integration’s granted permissions. If it has send, modify, delete, or broad mailbox-reading access, treat that as significant.
- Change the Google account password if anything suspicious appears, and ensure two-step verification is enabled. Revoking the OAuth connection is still important because changing a password does not always provide the clearest way to audit or remove every authorized application token.
- Check the company’s real mailbox or contact them through a known address, not through details supplied only by the AI.
Because this was a newly created account, the practical exposure may be limited, but the connected account could still contain sent correspondence, contact information, and any files received through it.
Reporting the incident
Preserve the conversation, the exact model name, timestamps, Gmail search results, and any available tool or connector activity log. Report the fabricated email result to OpenAI through its official support or feedback channel, and include that the response claimed to have found a message and attachment that could not be located.
The key lesson is that an AI assistant’s claim that it checked an inbox is not evidence by itself. For email, payments, account changes, and security decisions, verify the underlying record directly in the service before taking action.
⚠
