A UK cyber agency said no chatbot defence is yet foolproof

Start here

An organisation building a chatbot on a large language model faces a question an ordinary checklist does not cover: can the model be tricked by the data it is asked to process. On 30 August 2023 the UK's National Cyber Security Centre published two blog posts on exactly that, aimed at 'cyber security professionals' but useful to anyone deciding how much to trust a chatbot with real tasks. The reader's task is understanding this is an acknowledged, unsolved risk, not a rare edge case.

What the documents say

The first post, Exercise caution when building off LLMs, states that 'an LLM inherently cannot distinguish between an instruction and data provided to help complete the instruction,' illustrated with a hypothetical bank chatbot: an attacker hides a prompt injection inside a transaction reference, so an innocent question makes the model act on the hidden instruction instead. The second, Thinking about the security of AI systems, adds data poisoning, where training data scraped from the internet is deliberately tampered with to corrupt a model's behaviour or bias. Both converge on one point: 'at present, there are no failsafe security measures' removing the prompt injection risk entirely. Neither is a study with a sample or methodology; both are practitioner guidance describing a fast-moving field as it stood in August 2023.

Check this

NCSC's recommended check is architectural, not detective: design the system so you are 'happy with the worst case scenario of whatever the LLM-powered application is permitted to do.' A reader evaluating a chatbot-based tool can ask a version of the same question: what is the worst thing this assistant could be manipulated into doing, given everything it can read and every action it can take. If that answer is alarming, the fix is reducing what the assistant is permitted to do, not trusting a more convincing chatbot to resist manipulation.

What holds and what fails

The guidance holds as a design principle across vendors and models, since it does not depend on any one company's current defences; it treats vulnerability to injection as close to inherent. It fails to give a reader a way to detect, in the moment, whether a response was manipulated by injected content; NCSC is explicit no reliable indicator exists yet. Applying 'standard cyber security hygiene,' as the second post recommends, such as supply chain checks and access controls, protects the surrounding system even when the model cannot be made immune.

• Ask what a chatbot-based tool you rely on is permitted to do without human confirmation.
• Treat any content a chatbot is asked to summarize or process as potentially adversarial input.
• Apply ordinary account and access-control hygiene around AI tools, not just the model itself.

An agency whose job is describing threats plainly, not selling a product, tells readers the honest state of this problem: contained by design choices, not yet solved by any single defence.