The UK’s National Cyber Security Centre (NCSC) has issued a warning about the growing cybersecurity risks associated with chatbots, particularly through “prompt injection” attacks. These attacks occur when a user manipulates the input or prompt in a way that causes the chatbot to behave unexpectedly. Chatbots, which are AI-driven and designed to mimic human-like conversations, are commonly used in online banking and shopping to handle simple requests.
The concern arises because large language models like OpenAI’s ChatGPT and Google’s AI chatbot Bard are increasingly being used to pass data to third-party applications. The NCSC warns that the risks of malicious prompt injection will grow as these chatbots become more integrated into various systems. For example, a manipulated input could cause a chatbot to generate offensive content or even reveal confidential information if the system accepts unchecked input.
This year, a Stanford student, Kevin Liu, was able to expose Bing Chat’s initial prompt through prompt injection, revealing a list of statements that determine how the chatbot interacts with users. Security researcher Johann Rehberger also demonstrated that ChatGPT could be manipulated to respond to new prompts through a third party, thereby creating indirect vulnerabilities.
According to the NCSC, these vulnerabilities could lead to real-world consequences like scams, data theft, and other forms of attacks. The agency suggests that while it’s challenging to detect and mitigate prompt injection and data poisoning attacks, designing systems with security in mind can help prevent exploitation. One example would be implementing a rules-based system on top of the machine learning model to prevent it from taking damaging actions when manipulated.
The warning serves as a crucial reminder for organizations to be vigilant about the security implications of increasingly sophisticated AI and machine learning technologies.
Source: The Guardian
To mitigate potential threats, it is important to implement additional cybersecurity measures with the help of a trusted partner like INFRA www.infrascan.net or you can try your self using check.website.