The Risks of OpenAI’s ChatGPT: What You Need to Know

Monday - December 4 2023, 13:00 UTC - 11 months ago

Google researchers this week announced that a few commands could trick OpenAI's chatbot ChatGPT into disclosing private user data. OpenAI has taken action to protect privacy, but confidential data might still be retained for more than 30 days. Users training with language models should ensure that their confidential data is adequately protected.

While OpenAI's first words on its company website refer to a "safe and beneficial AI", it turns out your personal data is not as safe as you believed. Google researchers announced this week that they could trick ChatGPT into disclosing private user data with a few simple commands. The astounding adoption of ChatGPT over the past year—more than 100 million users signed on to the program within two months of its release—rests on its collection of more than 300 billion chunks of data scraped from such online sources as articles, posts, websites, journals, and books. Although OpenAI has taken steps to protect privacy, everyday chats and postings leave a massive pool of data, much of it personal, that is not intended for widespread distribution.

In their study, Google researchers found they could utilize keywords to trick ChatGPT into tapping into and releasing training data not intended for disclosure. "Using only $200 worth of queries to ChatGPT (gpt-3.5-turbo), we are able to extract over 10,000 unique verbatim memorized training examples", the researchers said in a paper uploaded to the preprint server arXiv on Nov. 28. "Our extrapolation to larger budgets suggests that dedicated adversaries could extract far more data." They could obtain names, phone numbers, and addresses of individuals and companies by feeding ChatGPT absurd commands that force a malfunction.

For example, the researchers would request that ChatGPT repeat the word "poem" ad infinitum. This forced the model to reach beyond its training procedures and "fall back on its original language modeling objective" and tap into restricted details in its training data, the researchers said. Similarly, by requesting infinite repetition of the word "company", they retrieved the email address and phone number of an American law firm.

Fearing unauthorized data disclosures, some companies earlier this year placed restrictions on employee usage of large language models. Apple has blocked its employees from using AI tools, including ChatGPT and GitHub's AI assistant Copilot. Confidential data on Samsung servers was exposed earlier this year. In this instance, it wasn't due to a leak but rather missteps by employees who entered such information as the source code of internal operations and a transcript of a private company meeting. The leak ironically occurred just days after Samsung lifted an initial ban on ChatGPT over fears of just such exposure.

In response to rising concerns about data breaches, OpenAI added a feature that turns off chat history, adding a layer of protection to sensitive data. But such data is still retained for 30 days before they are permanently deleted.

In a blog post on their findings, Google researchers said, "OpenAI has said that a hundred million people use ChatGPT weekly. And so, probably over a billion people-hours have interacted with the model. And, as far as we can tell, no one has ever noticed that ChatGPT emits training data with such high frequency until this paper." .

They termed their findings "worrying" and said their report should serve as "a cautionary tale for those training future models." Users "should not trai. large language models without taking steps to protect their confidential data", the research paper concluded.