AI coding assistant hallucinations are plausible-looking errors in generated code, such as invented packages, made-up functions, or unsafe security advice...
AI Coding Assistant Hallucinations: The Hidden Security Risks and How to Stop Them
AI coding assistant hallucinations are plausible-looking errors in generated code, such as invented packages, made-up functions, or unsafe security advice. They matter because developers can mistake fluent output for verified facts, then carry those mistakes into software, build systems, and customer accounts. A few practical checks can catch many of these risks before release.
Key Takeaways- AI assistants can invent package names, APIs, and security guidance that look real.
- A 2024 research study found that roughly one in five package suggestions in its test results were hallucinated; that rate is not a universal measure of every tool or project.
- Slopsquatting targets package names that AI models repeatedly invent, rather than names users mistype.
- Use dependency checks, locked versions, restricted build permissions, and human review for important security decisions.
- Measure the problem in your own workflow. Fluency is not proof.
I’m Ermenildo Castro, a Filipino engineer in Seattle. I’ve worked in software teams at Google, Microsoft, Amazon, and Zulily, where speed mattered—but code still had to survive review, testing, and production. One lesson stayed with me: a clean-looking change can hide a bad assumption. AI tools make that lesson more urgent, because they can produce polished suggestions faster than a person can check each one.
This is not an argument to ban coding assistants. Used with care, they can help explain unfamiliar code, draft routine tests, or suggest a starting point. The trouble begins when a team treats a suggestion as a source of truth. Think of generated code as a new contribution from an unverified source: useful, perhaps, but never trusted just because it sounds sure of itself. That small change in attitude makes security checks easier to build into everyday work.
What Is an AI Hallucination in Coding?
An AI coding hallucination is output that looks like a factual answer but contains something the model has made up or gotten wrong. It might be a package that does not exist, a function that is absent from the library, or an explanation that attributes a feature to the wrong tool. In security work, it can also be advice that sounds current but relies on a weak or outdated pattern.
Language models produce text by predicting likely next pieces of text from patterns learned during training. They do not automatically check every claim against a live package registry, current vendor documentation, or your repository. When they lack a useful example, they can fill the gap with a plausible guess. The wording may stay smooth and certain. That is the catch: a hesitant answer is easier to question than a confident one.
Code makes the mistake harder to spot. A fake package name can follow the naming style of a real project. An invented method can fit the style of an SDK. A code sample might even compile if it does not reach the missing dependency path during a basic test. Syntax checks can tell you that code is well formed; they cannot prove that a package is safe, a permission is needed, or an authentication method fits your threat model.
Researchers examined package suggestions from 16 code-generating models in a 2024 study. In their test results, roughly one in five suggested dependencies referred to packages that did not exist, and the researchers counted more than 205,000 distinct fabricated names. The result is a warning, not a universal score: model, prompt, programming language, and test setup all affect the outcome. The study is useful because it shows that package invention is not just a far-fetched edge case. Read the research paper, We Have a Package for You!, for the study’s method and findings.
There is a useful distinction here. A regular software bug is code that behaves incorrectly; a hallucination is an unreliable claim or suggestion that may lead to a bug, a security flaw, or wasted time. A model can also give an answer that was once correct but is now stale. The practical response is similar in all three cases: check important claims against sources you trust, then test the code in the context where it will run.
In a code review, I have seen the familiar trap: a generated example used an API name that matched the surrounding library’s style, so it looked believable at first glance. The documentation settled the question in minutes. That is why reviewers should check the behavior and the source, not just whether the diff looks tidy. A fluent answer is still only a suggestion.
Core Benefits of Catching AI Coding Hallucinations
The point of reviewing AI output is not to slow every task to a crawl. It is to keep useful speed while reducing avoidable risk. A few targeted checks protect more than code quality; they also make a team’s decisions easier to explain and repeat.
- Reduce supply-chain exposure. A new dependency can bring its own maintainers, release process, install scripts, and transitive packages. Checking the exact name and publisher before installation helps prevent a made-up suggestion from becoming trusted software. Frankly, a package with a convincing name is not a security review.
- Keep security choices tied to real requirements. A model may draft access rules, authentication flows, or cryptography examples without knowing your threat model. A human reviewer can ask the basic question: who needs this permission, and what happens if it is abused? That question catches excess access and weak defaults.
- Make code review more focused. Reviewers can spend time on the risky parts—new dependencies, identity checks, data handling, and permission changes—instead of treating every generated line as equally dangerous. A short checklist gives the team a shared method rather than relying on one person’s memory.
- Improve reliability and maintenance. Nonexistent APIs waste debugging time. Poorly chosen real libraries create upgrade and support work. Checking package health, documentation, release history, and project fit at the start can spare a team from carrying a questionable choice through years of maintenance.
- Protect the people who rely on the software. Weak authentication or a broad cloud permission can affect customers and coworkers, not just a test result. Good engineering includes care for the dignity and safety of the people on the other side of a service. That is part of doing the work responsibly.
- Create useful evidence. When teams record why a dependency was allowed or why an authentication method was chosen, future reviewers have something solid to inspect. This matters during incident response, audits, and staff changes. It also helps engineers distinguish a deliberate exception from a forgotten shortcut.
I have found that a small amount of structure beats a grand rule nobody follows. In one routine review, checking a dependency against its official registry and project documentation took less time than debating whether its name looked right. The useful habit was not suspicion of every line; it was knowing which claims needed proof. Apply checks where the impact is high, and let ordinary, low-risk work keep moving.
There is a second benefit: teams learn how to ask better questions of the tool. Instead of accepting “add a package to do this,” ask for a solution using approved dependencies, then verify the answer. Instead of asking for “secure login,” describe account needs, deployment limits, and recovery rules. Better prompts can reduce some bad suggestions, but they cannot certify the result. Here’s the kicker: good prompts are a layer of control, not a replacement for one.
Step-by-Step: Build Guardrails Into Your Development Pipeline
Good controls should catch mistakes before an install reaches a trusted build environment. Put the checks in the path developers already use, and make failures clear enough to fix without a long investigation.
- Sort work by risk. Mark changes that touch authentication, authorization, encryption, secrets, customer data, cloud permissions, or new dependencies. A generated comment or test stub does not need the same review as a change to account recovery. Make the rule simple: the higher the impact, the stronger the verification.
- Set approved dependency sources. Decide which registries and package sources are allowed for each project. Require developers to name a package and pin an exact version before it enters a lockfile. For sensitive services, use an allowlist or a review step for new dependencies. Check that the package exists in the expected registry and that its name, maintainer, and repository make sense. Do not rely on a model’s description of the package.
- Verify dependency details automatically. Run tooling that checks manifests and lockfiles, flags unknown or unapproved packages, and reports version changes. Scan for known vulnerabilities, but do not mistake a clean scan for proof that a package is trustworthy. Review provenance and release history when the project’s risk calls for it. A scanner has limits; a fake name can pass some checks if an attacker has registered it.
- Limit what a build can do. Treat installs and build steps as code execution risks. Use isolated or disposable build environments, restrict network access where practical, and give each job only the permissions it needs. Keep secrets out of untrusted pull-request builds. If a package install runs a script, that script should not inherit broad access to production credentials. The NIST Secure Software Development Framework offers a useful set of practices for building security into software work, including protecting the development environment.
- Review security-sensitive code against trusted guidance. Check authentication, cryptography, and access control against current standards and official documentation. For example, passkeys can provide phishing-resistant authentication when correctly implemented, but account recovery and device loss still need careful design. SMS codes may be a fallback in some systems, but they are not phishing-resistant and can be exposed to risks such as number takeover. Consult current NIST digital identity guidance rather than treating generated examples as a security standard.
- Ground answers in approved material. If your coding assistant supports retrieval, connect it to maintained internal documentation or trusted vendor references. Ask it to identify which source supports a recommendation. Retrieval-augmented generation can reduce unsupported claims, but it does not guarantee truth: the source may be old, incomplete, or irrelevant. Review the cited material yourself when the decision affects a critical path. The OWASP Top 10 for LLM Applications is a useful reference for risks such as misinformation and overreliance on model output.
- Keep a human decision point for critical changes. A reviewer should confirm the need for each new dependency, inspect the security behavior, and check that tests cover the expected failure cases. This is not a demand that a person manually reread every generated line. It is a way to keep accountability with the people who understand the service and its users. NIST’s Generative AI Profile can help teams think through risks and controls across the AI system lifecycle.
- Measure, learn, and adjust. Track how many new dependencies are rejected, how often package names fail registry checks, and how many AI-assisted changes need security correction. Test your assistant with prompts designed to expose weak answers, including requests for packages that are not in your approved catalog. Record the tool version and test conditions so results can be compared fairly. Do not collect private prompts or employee data without a clear reason and suitable safeguards.
I have used this kind of staged review because it makes the right action obvious: a registry check happens automatically, a risky permission change gets a reviewer, and a low-impact suggestion does not trigger a full security exercise. Teams can adapt the steps to their stack. The important point is to put checks in automation where possible, rather than expecting tired reviewers to catch every error by eye.
Comparison: Slopsquatting vs. Typosquatting
Slopsquatting is a package-supply-chain attack that targets names a language model may invent. An attacker can register one of those names in a public package registry and wait for a developer to install it. The name may look credible because it came from a coding assistant, not because the developer mistyped a familiar package. This turns a model’s made-up suggestion into a possible route for malicious code.
Typosquatting is the older, related tactic. The attacker registers a package name that resembles a popular package, hoping a person will make a typing error. Both tactics exploit trust in package names. The difference is where the name comes from: a user’s typo in one case, a model’s plausible fabrication in the other.
| Feature | Slopsquatting | Typosquatting |
|---|
| Target | A package name generated by an AI model | A package name close to a known package |
| Likely error | Trusting a plausible AI suggestion | Mistyping or misremembering a name |
| Attacker’s opening | Register a hallucinated name before a user installs it | Register a look-alike name and wait for an install |
| Useful defense | Verify the exact name in an approved registry and catalog | Check spelling, publisher, and project documentation |
| What overlaps | Both can introduce untrusted code through dependencies | Both can introduce untrusted code through dependencies |
The research paper linked above counted many fabricated names across its test output, but that does not mean every invented name was registered or used in an attack. Nor does it show that every model gives the same results. A useful security conclusion does not need inflated numbers: the attack route is plausible, package installation can run untrusted code, and low-cost checks can reduce exposure.
Repeatability is part of the concern. If a model repeatedly suggests the same false name under similar prompts, an attacker could watch for recurring suggestions and register the name. The user may then see a package that exists in the registry and assume that existence proves legitimacy. It does not. Check that the package is the intended one, comes from the expected project, and has a reason to be in your application. The model’s suggestion is a lead to verify, not an endorsement.
Common Mistakes to Avoid
Mistake 1: Trusting code because it compiles. Compilation only shows that the compiler accepted the code under the current conditions. It does not show that the dependency is legitimate, the permission is justified, or the authentication flow resists phishing. Tests help, but they prove only the behavior they exercise. For high-impact code, compare implementation choices with maintained documentation and security review criteria.
Mistake 2: Treating the package registry as a safety label. A package existing on npm or PyPI is not the same as the package being trustworthy. A malicious actor can publish a package; a legitimate package can be abandoned or compromised. Check the exact name, source, maintainer history, release pattern, and project need. A registry lookup answers “does this name exist?” It does not answer “should this code run here?”
Mistake 3: Relying on a blocklist alone. Known-bad package lists can catch threats that have already been identified. They cannot list every package an attacker might register in the future. Use layered checks: approved sources, review of new dependencies, lockfile control, build isolation, and monitoring. Frankly, no single scanner can tell you the whole story.
Mistake 4: Assuming retrieval makes an answer correct. Retrieval can point a model toward approved documentation, which is better than relying only on old training examples. But a retrieved page can be stale, the wrong version, or unrelated to the code in question. Check the version and context. A citation is a trail to inspect, not a stamp of approval.
Mistake 5: Accepting broad cloud permissions to make the sample work. Generated cloud examples may grant more access than the task requires. Review IAM actions, resource scope, trust relationships, and secret handling. Ask what the service truly needs and remove everything else. Least privilege is not busywork; it limits the damage when a key leaks or a process is abused.
Mistake 6: Assuming common authentication patterns are safe. A generated login flow may use familiar but weak patterns, or handle recovery poorly. SMS codes can still be used in some situations, but they do not provide phishing resistance. Passkeys can improve resistance to phishing, yet teams still need to plan recovery, device changes, and support procedures. Check guidance and test the full account lifecycle, not just the happy path.
Mistake 7: Making human review the only barrier. A reviewer can miss a fake dependency in a large change, especially under time pressure. Put mechanical checks in CI, reserve expert review for decisions that need judgment, and make failures visible. This approach respects the dignity of engineering work: people should use their attention for choices that require understanding, not repetitive checks a tool can perform consistently.
The truth is, “be careful” is not an operational control. It leaves each engineer to guess what deserves attention, and it fails exactly when the review queue is full. Write down the rules, automate the repeatable ones, and give reviewers a short list of risks to inspect. That is less glamorous than promising perfect AI output. It works better.
Frequently Asked Questions
What is an AI hallucination in coding?
It is a generated answer that presents an incorrect or fabricated claim as if it were reliable. Examples include a package that does not exist, a function that is not part of a library, or a security recommendation that is stale or unsafe. The code may still look polished or compile. Check the claim against a trusted source before relying on it, especially when it affects authentication, dependencies, permissions, or sensitive data.
What does slopsquatting mean?
Slopsquatting is an attack in which someone registers a package name that an AI model has hallucinated, hoping a developer will install it. It resembles typosquatting, but the target name comes from model output rather than a user’s spelling mistake. Check package names in an approved registry and against the project’s official documentation. Do not assume that a package is safe just because it exists.
How can developers prevent AI-generated package risks?
Use approved registries or dependency allowlists, verify package names and maintainers, review new dependencies, and commit lockfiles with exact versions. Run builds in restricted environments, keep secrets away from untrusted jobs, and use automated checks to flag unknown packages. For security-sensitive changes, have a qualified reviewer confirm that the dependency is needed and that its behavior matches the application’s requirements.
Do AI coding assistants always produce insecure code?
No. They can produce useful code, tests, and explanations, but their output is not automatically verified. Risk depends on the task, model, prompt, project context, and review process. Treat generated code like code from any untrusted contributor: inspect important decisions, run tests and security checks, and verify claims against maintained references. Keep a person responsible for final decisions on high-impact software.
Final Thought
AI coding assistants can save time. That does not make them authorities. They produce suggestions from patterns, while secure software depends on the facts of a particular project: its users, permissions, data, dependencies, and operating conditions. A confident answer can be a useful starting point, but confidence is not evidence. Verify what could hurt someone if it were wrong.
I still like tools that remove tedious work. I do not like pretending that speed makes review optional. A small automated check can prevent a questionable package from reaching a build, while a thoughtful engineer can catch a weak recovery flow that no simple scanner understands. The right balance is plain: automate what can be checked by rule, and ask people to judge what needs context.
The work also carries a duty to the people who depend on the software. Protect their accounts, limit access, and avoid exposing their data because a generated sample looked neat. That is stewardship in practical form. Build the checks once, keep them current, and let the assistant help without handing it the keys.