INC-2023-0001SEV2 · HighData leakConfirmed

ChatGPT bug exposes chat titles and payment details of other users

OpenAI took ChatGPT offline after a caching bug let some users see titles from other users’ chat histories and, for some subscribers, payment-related details.

Occurred
Disclosed
Published
Organization
OpenAI
System
ChatGPT web application
Impact
Chat titles shown to other users; payment details of 1.2% of active ChatGPT Plus subscribers potentially visible during a nine-hour window
Root cause
Bug in the open-source redis-py client, triggered by a server change

Timeline

  • 2023-03-20, 1 a.m. Pacific. According to OpenAI, a server change causes a spike in cancelled Redis requests.
  • 2023-03-20, 1 a.m. to 10 a.m. Pacific. The window in which payment-related details could be shown to the wrong subscriber.
  • 2023-03-20. OpenAI takes ChatGPT offline.
  • 2023-03-24. OpenAI publishes its account of the bug.

Trigger or entry point

OpenAI states that it “inadvertently introduced a change” to its server that sharply increased the number of cancelled Redis requests.

Failure mechanism

OpenAI used the redis-py library with Asyncio to cache user information. If a request was cancelled after it was sent but before its response was read, the connection was left in a corrupted state. The next, unrelated request on that connection could then receive the leftover data. In most cases this produced an error. In some cases the data matched the expected type and was returned as if it were valid, even though it belonged to another user.

Impact

Some users could see titles from another active user’s chat history, and possibly the first message of a newly created conversation. OpenAI also found that the bug may have exposed the name, email address, payment address, card type, last four card digits and card expiry date of 1.2% of ChatGPT Plus subscribers who were active in the nine-hour window. OpenAI states that full card numbers were not exposed, and that it believes the number of users whose data was actually seen by someone else is extremely low.

Response

OpenAI took the service offline, patched the bug, sent a fix to the Redis maintainers and notified affected users.

Mitigations

According to the secondary report, OpenAI added redundant checks so that data returned from the cache matches the requesting user.

Engineering takeaway

Failure pattern
A shared connection pool returned one user’s cached data to another after a server change raised the rate of cancelled requests.
Preventive control
Check at the point of use that cached data belongs to the requesting user, and roll server changes out in stages with error-rate monitoring.
Transferable lesson
Per-user isolation that depends on a library behaving correctly needs a second check that does not.

Editorial summary by Incidents.si, drawn from the sourced record above. It is not a statement by the organizations involved.

Sources

  1. March 20 ChatGPT outage: Here’s what happenedprimary · 2023-03-24
  2. OpenAI: A Redis bug caused a recent ChatGPT data exposure incidentsecondary · 2023-03-26

Corrections

No corrections have been made to this entry.

Report a correction