Outputs
When an AI tool discloses data in its answers
Sometimes the leak is the tool's own answer. Treat it as a breach when the answer showed personal or confidential data to someone who should not see it.
Last reviewed against legislation.gov.uk, National Cyber Security Centre and OWASP GenAI Security Project. Sources
What it looks like
Disclosure through answers
OWASP describes LLM applications giving away personal details, health and money records, trade secrets, credentials and proprietary model details through what they output [5]. It also notes that users can expose sensitive data themselves by typing it into prompts, from where it may resurface later in answers [5].
Its example scenarios include one user being shown another user's personal data because of weak sanitisation, and careless handling of data in training leading to later disclosure [5].
A real case
ChatGPT, March 2023
OpenAI took ChatGPT offline after some users saw other people's chat titles in their history and some Plus subscribers saw other users' email addresses [9]. It blamed a bug in the redis-py library, and said roughly 1.2% of Plus subscribers who used the service during a nine-hour stretch were affected [9]. For those subscribers the visible details covered a name, an email and payment address, the card's expiry date and its last four digits, but never the full number [9].
Response
Containing it
-
1
Take the affected feature or integration offline, as OpenAI did with its service [9]. -
2
Save the prompts and outputs that show the problem. -
3
Work out whose data appeared, to whom, and for how long. -
4
Ask the supplier for the root cause in writing. -
5
Decide on reporting and telling people.
Fixes
Reduce the chance of a repeat
Check these as well:
- Scrub or mask sensitive data before it enters training or retrieval [5].
- Give the tool the least access it needs and limit its links to outside data sources [5].
- Do not rely on rules in the system prompt alone, because OWASP says prompt injection can get round them [5].
- Publish clear rules on how long entered data is kept and when it is deleted [5].
Treat the tool as an untrusted reader
The NCSC notes that crafted inputs can trick an AI model into revealing confidential information [3]. Give it only the data you would be content to see leak.
Keep reading
Read next
Examples
Four documented AI-related data exposures, ChatGPT, Samsung, Microsoft and DeepSeek, with the cause of each and the lesson for UK organisations.
Read the guide
What to do
A practical order of work for UK organisations after data leaks through an AI tool: awareness, containment, scoping, reporting, supplier and plan.
Read the guide
Timeline
A dated, sourced timeline of public AI-related data exposures and the UK guidance that followed, from the ChatGPT bug to the DeepSeek database.
Read the guideHow CyPro can help
Need a hand working through this?
Finding out whether an incident reaches you is one job. Checking your own systems, tightening access and briefing the board are several more. CyPro's responders and consultants can take any of them on.
Incident response
If you may already be compromised, our responders contain it, preserve the evidence, find out what was touched and get you running again safely.
More on cypro.co.uk
24/7 security monitoring
Our SOC analysts watch your systems day and night for the signs of an attack in progress: unexpected processes, new outbound connections, logins that do not fit.
More on cypro.co.uk
Fractional CISO for the board
A senior security lead who advises the board on what to fix first, what to accept and what to tell customers, and owns the decision record.
More on cypro.co.uk
Penetration testing
A focused test of your systems and the controls around them, so you know they hold before an attacker finds out for you.
More on cypro.co.uk
Hardening and segmentation
Our consultants put the controls in place that limit the damage when something is missed: firewall rules, network isolation and tighter admin access.
More on cypro.co.uk
Vulnerability scanning
Find every internet-facing system you run, including the forgotten ones, and see which are still on a vulnerable build.
See the scanning service
Talk to a specialist
Worried this affects you?
When data has leaked through an AI tool, its logs or its storage, the early decisions shape what you must report and to whom. An incident response team helps you contain the exposure, establish what was involved and prepare your notifications.