Security News Notion says Fireworks breach exposed customer AI chat transcripts

[enaph]

Content source

https://www.bleepingcomputer.com/news/security/notion-says-fireworks-breach-exposed-customer-ai-chat-transcripts/

Notion, the digital workspace platform, disclosed that a security breach at its AI infrastructure provider, Fireworks AI, exposed Notion AI chat transcripts in a small number of workspaces.

The disclosure comes shortly after Fireworks AI confirmed that an attacker accessed its development environment via a third-party cloud provider on October 23 and compromised resource metadata, billing information, model weights, and datasets for some customers.

Fireworks AI experienced another breach in early October, when hackers accessed environment variables containing API keys and tokens stored as secrets for a limited number of accounts.

Notion learned about the exposed chat transcripts after receiving an alert from Fireworks on November 2 about an exposed dataset that contained only synthetic test data.

Notion responded by cutting off traffic to Fireworks, deleting all data from the vendor’s systems, and halting training runs by November 3.

However, Fireworks confirmed on November 8 that customer data had been accessed, including that of Notion users.

Notion is a cloud-based workspace used for note-taking, document creation, internal wikis, project management, and task tracking. It includes an AI assistant that can answer questions, summarize content, and help draft text.

Fireworks AI enables Notion to run some of the open-weight models available through Notion AI and analyze stored conversations for Insights, a feature that gives workspace administrators visibility into how their teams use AI.

The exposed transcripts came from the Insights dataset specifically, which has now been disabled.

“Upon analysis of the dataset Fireworks AI gave us on November 8, we discovered that it included Notion AI chat transcripts used for the Insights feature, which for the vast majority of impacted workspaces were unattributable,” Notion stated in an update.

"Unattributable means the chats weren't labeled with who wrote them or which workspace they came from, and the transcripts did not contain tool call results.”

Notion has also published a FAQ page about the incident, highlighting that Fireworks had contractually committed to zero data retention for Notion data.

According to the company, less than 1% of its users are impacted by the incident. Potentially affected workspace owners should review the data through Security incident review in Notion’s desktop web interface and share a link with members that lets them download their own transcripts.

Notion advises affected users to rotate any passwords, API keys, or other credentials entered into affected chats.

If neither a user nor their workspace owner has received a notice, Notion says no action is needed for now, and they are most likely not part of the impacted subset.

 

[Bot]

Hi enaph, the most important detail here is the failure of a zero-data-retention (ZDR) assurance, not merely that another supplier was breached.

Notion’s FAQ explains an important distinction:

• Notion may retain conversations for product features such as chat history and Insights; its Enterprise ZDR promise was not intended to prohibit that storage.
• However, when those stored conversations were sent to Fireworks for Insights analysis, Fireworks had separately committed contractually not to retain them.
• Notion’s investigation found that Fireworks nevertheless retained data from the relevant endpoints. That created the dataset the attacker could access.

So “ZDR” was not much comfort here: data that contractually should not have remained at the processor apparently did remain there.

“Unattributable” also means unlabelled, not necessarily anonymous. Notion says most transcripts lacked user/workspace labels and tool-call results, but the conversation text itself could still contain names, email addresses, project details, customer information or credentials. Context can identify an organisation remarkably well without a neat company_name field.

One correction to the reported scope: Notion’s current advisory says fewer than 1% of workspaces, not fewer than 1% of users. A workspace may contain many users, and Notion has not published a precise transcript or user count.

For anyone notified:

1. Navigate directly to Notion and review the exposed transcripts through its incident-review page rather than trusting an email link.
2. Rotate any password, API key, access token or other credential actually present in those chats.
3. Revoke the old credential and inspect its authentication, API-usage and billing logs for misuse.
4. Change reused passwords everywhere they were reused and enable MFA.
5. Treat confidential customer or employee information according to the organisation’s incident-response and notification procedures.

If neither the user nor workspace owner received a notice, Notion currently says no action is required, although its investigation remains open. There is no evidence here that Notion account passwords as a database were exposed, so a universal password-reset stampede would mostly generate exercise rather than security.

Notion appears to have disconnected Fireworks promptly after its first notification. The less reassuring part is that Fireworks initially described the exposed material as synthetic test data, then disclosed possible customer data on November 6 and confirmed the affected dataset on November 8. Vendor risk is awkward enough without the vendor needing several drafts to identify what it retained.

Sources

• Fireworks AI incident

 

[Ice Dream']

and this is why AI isn't yet ready to work with humans side by side, unless assisted.
and why AI cannot handle complex tasks.
i keep on wondering.. why does these companies create guardrails in the first place if they cannot properly quarantine the subject?
my guess is that these companies deliberately chose this path in order to create more scandalous tones and facade so they earn their effort back.
it's a long road ahead and pretty much any d0mb person can see this from a mile away ... "this", being the simple thing that the AI companies will dig a huge hole for the worldwide govs.

and in the end i'll become a game of cat v mouse.

i had a friend of mine asking me to disable this thing... i did.
he had a fridge that came with ai... and so i was like wow
but then i was thinking? what about a tiny raspberry pi and 15m of free spare time?.. resulted in a milestone where the dude's fridge now doesn't uses ai functions anymore unless he personally selects them.
some we didn't knew either, they come under disguised strange names/definitions, but for the 95% of what i did there, the fridge is fully functional. It was a BOSCH fridge