AI Coding Assistant Hallucinations: How to Stop Fake Packages and Unsafe Code
AI coding hallucinations create real security risks. They occur when an assistant invents a package, misstates an API, or drafts code that looks sound but rests on false facts, and those errors can reach a build pipeline before anyone questions the polished answer. Treat each suggestion as untrusted input.
AI assistants can save time on routine coding, but speed does not remove the need for review. I have seen a tidy code sample earn quick approval because it compiled and followed familiar patterns; that is not the same as proving it was safe or even based on a real library. A model can sound certain because it produces fluent text, not because it checked a registry or tested a security claim.
The practical question is not whether an assistant will make mistakes. It will. The useful question is whether your workflow catches those mistakes before they become dependencies, permissions, or production behavior. Frankly, that is a pipeline design problem, not a test of whether a developer can stay alert all day.
What Are AI Hallucinations in Coding?
An AI hallucination is an output that appears factual but contains invented or incorrect information. In coding, that can be a package that does not exist, an API that belongs to no real version of a library, or a security recommendation based on an outdated pattern. The assistant may present the result in clean, idiomatic code. That polish can hide the error.
Most language models generate text by predicting likely next tokens from patterns in training data and the current prompt. They do not automatically query a live package registry, read the latest documentation, or confirm that a cited function exists. When context is missing, a model may complete the pattern with a plausible guess. It does not have to intend to mislead for the result to mislead a developer.
Code makes the problem easy to miss. A made-up package can follow real naming conventions, while an invented function can resemble the other methods in an SDK. A compiler catches syntax errors and some type errors, but it cannot tell whether a package is trustworthy, whether a permission is excessive, or whether an authentication design resists phishing. Passing a build is a narrow check.
I have reviewed code where the first pass looked convincing: familiar imports, sensible variable names, and comments that explained the approach. A closer check found that one dependency name needed registry verification and that the proposed API did not match the library version in the project. The lesson was not to ban assistance. It was to inspect the assumptions the code brought along.
Retrieval tools can reduce some errors by giving a model approved documentation to consult, but they do not make every answer true. A model can misread a page, mix versions, or produce a claim that the page never made. Treat citations, documentation snippets, and generated tests as leads to verify. The OWASP GenAI Security Project’s LLM guidance addresses the need to check model outputs rather than rely on their confident presentation.
Why AI Coding Hallucinations Matter
The benefit is real. An assistant can draft routine code, explain an unfamiliar function, suggest tests, or turn a clear requirement into a first version. Used well, it gives an engineer a starting point and leaves more time for design and review. The benefit comes from faster drafting, not from handing over responsibility.
- Faster first drafts: Generated code can reduce time spent on repetitive setup, especially when a developer already knows what a correct result should do.
- More test ideas: A model can suggest edge cases that a busy engineer may have missed, then a person can decide which tests matter.
- Help with unfamiliar code: Explanations and small examples can make a large codebase easier to inspect, provided the API details are checked.
- A review prompt: Asking an assistant to list risks can help reviewers consider input validation, error handling, and permissions. It cannot replace a security review.
The supply-chain risk is more than a theoretical concern. A study of 16 code-generating models reported that roughly one in five package recommendations in its tests referred to packages that did not exist; the researchers reported more than 205,000 unique fabricated package names across their test set. Those figures describe the tested prompts and models, not every assistant or every real-world request. Still, they show why a name that looks plausible deserves a check. Read the research paper on package hallucinations for its methods and limits.
An attacker can register a fabricated name on npm, PyPI, or another registry before a developer does. If a developer installs that name, the package may run attacker-controlled code with the build agent’s permissions. This tactic is called slopsquatting: it resembles typosquatting, but the name comes from an AI-generated suggestion rather than a human typing mistake. Researchers have also observed recurring fabricated names in model outputs, though repeatability varies by model, prompt, and time.
Security advice can be just as risky. A model might draft an authentication flow that relies on SMS codes without explaining their limits, use a deprecated cipher, or create a cloud role with broad permissions that the feature does not need. For important accounts, teams should consider phishing-resistant authentication such as passkeys, while also reviewing account recovery and fallback methods. The CISA guidance on phishing-resistant MFA and NIST digital identity guidance offer stronger reference points than an unverified code sample.
There is a human cost to weak controls. A package can expose customer data, interrupt a service, or put an on-call engineer into a long night of incident response. Responsible engineering means treating tools and skills as things to steward well: protect the people who rely on the software, and do not confuse fast output with finished work.
Step-by-Step: Add Guardrails to an AI Coding Workflow
Do not depend on reviewers noticing every error by eye. Add checks where suggestions enter the repository, then reserve careful human review for the decisions that need judgment. I have found that a small set of repeatable checks works better than a vague instruction to be careful.
- Set rules for what the assistant may do. Decide whether it can suggest dependencies, change authentication code, edit infrastructure, or only draft code. Keep high-impact changes behind review. A clear boundary helps developers know when a suggestion needs extra scrutiny instead of treating all generated code as equally low risk.
- Verify every new package before installing it. Search the official registry and confirm the exact name, package owner, repository, release history, and documentation. A registry entry alone does not prove the package is safe; an attacker may have registered the name. For unfamiliar dependencies, ask why the project needs them and whether an existing approved library already covers the task.
- Pin and review dependency changes. Use lockfiles, review version changes, and apply your organization’s rules for approved packages. Check transitive dependencies too, since a trusted top-level package can bring in code your team has not examined. Tools such as GitHub dependency review can flag changes in a pull request before they merge.
- Check imports and APIs against trusted references. Confirm that every import exists in the registry and that the called method is documented for the version your project uses. Search the official docs, not just a model-generated explanation. When the assistant gives a version number, release note, or function signature, treat it as a claim to verify.
- Run automated security checks. Use dependency scanning, static analysis, secret detection, and tests that match your stack. Review generated IAM policies for wildcard actions, broad resources, and permissions unrelated to the feature. Automated tools cannot prove code is secure, but they can catch common mistakes before human review.
- Give extra review to sensitive paths. Authentication, authorization, encryption, payment handling, and infrastructure changes deserve review by someone who understands the threat. Check the design, not only whether the code runs. For authentication, inspect enrollment, recovery, rate limits, session handling, and phishing resistance—not just the happy-path login.
- Track the results. Record how often new dependencies are suggested, rejected, or found to be invalid; note which tools and prompts produced them. Review trends after model or workflow changes. This turns a vague concern into a measurable control, and it helps a team decide whether its guardrails need adjustment.
For systems that use an assistant with retrieval, restrict its references to documentation your team trusts and keep that documentation current. Retrieval-augmented generation can ground an answer in selected material, but the output still needs verification. A benchmark or internal test suite can help measure package suggestions, API accuracy, and compliance with your coding rules before the assistant is widely adopted.
Comparison Table: Slopsquatting vs. Typosquatting
Slopsquatting and typosquatting can both lead a developer to install a malicious package. The difference is how the name enters the workflow. With typosquatting, a person mistypes a real package name or selects a lookalike. With slopsquatting, an assistant may invent a convincing name, and an attacker can register it before the developer checks.
| Risk | How the name appears | Typical attacker move | Useful first control | Important limit |
|---|
| Slopsquatting | A model suggests a fabricated package name | Register a name that models tend to invent | Verify package identity and provenance before install | Registry presence does not prove safety |
| Typosquatting | A person mistypes or misreads a package name | Register a name close to a popular package | Use exact-name checks and approved dependency lists | Careful spelling cannot detect every malicious lookalike |
| Dependency confusion | A build tool resolves a package from an unintended registry | Publish a public package that competes with a private name | Configure private scopes and registry priority | Correct names can still resolve to the wrong place |
The overlap matters. All three risks involve trusting a package name too quickly, and all can affect a build agent that has access to credentials or internal systems. A package that exists may still be malicious, abandoned, compromised, or unsuitable for the project. Check ownership, source code, release history, and the permissions available during installation.
Slopsquatting adds a new way for a name to be suggested, but it does not replace old supply-chain risks. Typosquatting still matters, and registry configuration still matters. Use layered controls: exact names, private package scopes, dependency review, lockfile checks, and limited build permissions. The GitHub dependency review documentation describes one way to inspect dependency changes during code review. No single scanner can certify that every package is harmless.
The practical difference is who or what creates the risky name. That distinction helps incident teams investigate how a dependency entered a project, but the immediate response is similar: stop the install, inspect the package and its history, and check whether any build already executed it. Here’s the kicker: an unfamiliar name deserves scrutiny even when a respected tool proposed it.
Common Mistakes to Avoid
Treating fluent code as verified code. A clear explanation can make an incorrect answer feel safe, but tone is not evidence. Ask what documentation, version, and tests support the claim. I have seen reviewers accept a neatly explained suggestion faster than a rough one, even though both needed the same checks. Review behavior, not confidence.
Checking only whether a package exists. Registration proves that a name is taken, not that the package is legitimate or fit for your project. Look at its maintainer, repository, release history, documentation, and relationship to the feature. A recently created package with little history may be valid, but it should not receive automatic trust just because an assistant recommended it.
Assuming a passing build proves safety. Compilation can show that names and types fit the compiler’s rules. It cannot show that a service is protected from account takeover or that a cloud policy follows least privilege. Run security tests, inspect the design, and review the runtime behavior. Frankly, a green check is useful evidence, but it is not a security verdict.
Using broad permissions to make generated code work. If a suggested cloud policy fails, granting administrator access is not a sensible shortcut. Identify the exact action and resource the service needs, then keep permissions narrow. Apply the same care to CI secrets: limit their scope, duration, and availability to untrusted builds.
Trusting a second AI answer as independent confirmation. Two assistants can repeat the same error because their training data and patterns overlap. A model-generated citation can also point to a real page that does not support the claim. Verify against official documentation, standards, package registries, and tests that run in your own environment.
Relying on human attention alone. A reviewer can catch subtle mistakes, but code generation increases the volume of suggestions. People get tired, and repetitive checks are easy to skip. Put registry verification, lockfile review, scanning, and policy checks into automation, then use human judgment for context and risk. Most guides say review everything; few explain how to make that possible at scale.
Treating a benchmark as a permanent guarantee. A model’s behavior can change with its version, prompt, or connected tools. A one-time test does not describe future output. Retest after upgrades, keep examples of past failures, and track whether checks catch them. Measurement should guide improvements, not become a badge that excuses weak review.
Frequently Asked Questions
Can AI-generated code be trusted?
Not without checking it. AI-generated code can be useful as a draft, and it can also contain invented APIs, unsafe defaults, or incorrect explanations. Review it as you would code from an unfamiliar contributor: verify dependencies and documentation, run tests and security checks, and examine sensitive design choices. The amount of review should match the impact of the change. A comment-formatting helper needs less scrutiny than an authentication flow.
What is slopsquatting?
Slopsquatting is a supply-chain attack in which an attacker registers a package name that an AI coding assistant may hallucinate. A developer may copy the suggested name into an install command without first checking whether the package is real and trusted. It resembles typosquatting, but the name is generated rather than mistyped. Verify package names and ownership before installation, and do not run unfamiliar packages with broad build permissions.
How can a team prevent AI package hallucinations?
Require registry and ownership checks for new dependencies, use approved-package lists where practical, and review lockfile changes. Scan dependencies and run builds with limited permissions so a compromised package cannot easily reach secrets or production systems. Train developers to confirm APIs against official documentation. Track invalid suggestions over time so the team can see whether its tools and controls are improving.
Are passkeys safer than SMS codes?
Passkeys are designed to resist phishing because authentication is tied to the legitimate site or service, while SMS codes can be exposed to phishing, phone-number takeover, or message interception. That does not make every passkey setup risk-free: enrollment, device access, recovery, and fallback methods still matter. For high-impact accounts, follow current security guidance and review the whole account-recovery path, not only the sign-in screen.
Final Thought
AI assistance can make capable developers faster, and that is a reasonable benefit to keep. But speed is only useful when a team can tell a real package from a fabricated one, a current security practice from an obsolete one, and a narrowly scoped permission from a dangerous shortcut. The tool does not carry the consequences of a production incident. The people who build and operate the system do.
I have learned to trust a workflow more than a promise. Put checks into the pipeline, limit what build tools can access, review sensitive changes with care, and keep a record of the failures your controls catch. Those habits protect customers and give engineers a fair chance to do good work without asking them to spot every error by sheer attention. Use AI as an assistant, not as an authority. That is a modest rule, but it holds up when the output looks convincing.