AI coding hallucinations are plausible-looking code suggestions that contain invented facts, packages, or unsafe security choices. They can slip into a build...
AI Coding Hallucinations: How to Stop Plausible Code from Becoming a Security Risk
AI coding hallucinations are plausible-looking code suggestions that contain invented facts, packages, or unsafe security choices. They can slip into a build because they read cleanly, compile, and arrive with confidence. Treat every generated dependency and security decision as untrusted until a person or automated check verifies it.
AI tools can speed up routine work, and that speed is useful. It also makes weak suggestions easier to accept before anyone has checked the details. I have seen how a tidy code sample can draw less scrutiny than a messy one, even when the tidy sample deserves more. That is the real problem: confidence is a writing style, not evidence. A safe workflow makes verification part of the work, not a chore saved for later.
What Are AI Coding Hallucinations?
An AI coding hallucination is an output that sounds factual but contains something made up or incorrect. A model generates text by predicting likely next tokens from patterns in its training data and the context it receives. It does not automatically check each claim against a live package registry, current security guidance, or your company’s codebase. A polished answer can still be wrong.
That distinction matters. A normal bug may come from a logic mistake in code the author intended to write. A hallucination can begin earlier, with an invented library name, a made-up function, or a claim that a weak security pattern is safe. Sometimes the code does not compile. Sometimes it compiles perfectly and fails only when someone relies on it.
I have reviewed code where a suggestion looked familiar because its names and formatting matched the surrounding SDK. The import was the weak link. A reviewer scanning for logic errors could easily miss a dependency that had never been checked against the registry. Here’s the kicker: the better the generated code looks, the more important it is to check where its claims came from.
Package hallucinations have drawn particular attention. A study covering 16 code-generating models reported that roughly one in five suggested package names in its tests did not exist, and counted more than 205,000 distinct fabricated names across its test set. Those results describe that study’s prompts and models, not a universal rate for every tool or developer. Still, the finding shows how large the pool of plausible names can become. Read the research on package hallucinations in code-generating models for its methods and limits.
Researchers have described a related attack as slopsquatting. An attacker registers a package name that models repeatedly invent, then waits for a developer to install it. Unlike ordinary typosquatting, the developer may not have mistyped anything; the model supplied the wrong name. If an installation script runs with access to build secrets or source code, the risk can spread beyond one workstation.
Why AI Coding Hallucinations Matter
AI assistance can save time. A model can draft a test, explain an unfamiliar function, or produce a first pass at repetitive code. That can leave developers more room for careful review. But a fast draft is valuable only when the team can verify it.
The risk reaches further than broken builds. A fabricated or compromised package can expose credentials, change build output, or add code that runs during installation. A made-up function can also hide a mistaken assumption about how a real library handles input. Dependencies are not harmless lines in a manifest; they are code that your organization agrees to run.
Security advice needs even closer attention. Ask a model for authentication code, and it may give you a familiar pattern that misses current threats or your application’s needs. For accounts that need strong protection, phishing-resistant methods such as passkeys are worth evaluating. Passkeys are not magic, and implementation and recovery still matter, but they are designed to resist common phishing attacks. The CISA guidance on phishing-resistant MFA explains why organizations should consider stronger methods for high-value accounts.
Cryptography creates another trap. A model can name an outdated option with a convincing explanation, or combine sound pieces in an unsafe way. Cloud permissions can go wrong too: generated IAM policies may grant broad access because the model optimizes for making the example work, not for limiting access to what the service needs. Least privilege is still a human decision.
In team settings, I have found that review works best when people ask what each dependency and permission is for, rather than simply whether the patch passes tests. That question takes a little longer. It also respects the people whose data and systems the code will touch. Good engineering is a form of stewardship: the work must serve users without handing them risks they never agreed to take.
The point is not to ban coding assistants. It is to put their speed inside boundaries. Use them for drafts and explanations, then make verification a normal step before code reaches a production path. Most guides praise speed. The useful question is whether the team can check the result just as quickly.
Step-by-Step: Build Guardrails into Your Pipeline
A workable process does not require a special AI security department. It requires clear checks at the places where a suggestion can become executable code. Start with the same basic rule for every model: output is untrusted until verified.
- Limit what the assistant can change. Give the tool a focused task and a small set of approved libraries. Ask it to explain why a new dependency is needed and to prefer the existing stack when practical. A model should not quietly turn a small feature into a new supply-chain relationship.
- Verify every package against its real registry. Check the exact name on npm, PyPI, or the registry your organization uses. Confirm that the package matches the intended project, has a credible history, and is maintained by the expected owner. Registry presence alone is not proof of safety; an attacker can publish a malicious package under a plausible name.
- Review package changes and lockfiles. Inspect dependency manifests and lockfile diffs. Confirm versions, transitive dependencies, and integrity data. A lockfile helps make builds repeatable, but it does not certify that a package is benign. Use dependency review or software composition analysis to flag new, vulnerable, or disallowed components.
- Run builds with limited privileges. Keep secrets out of untrusted pull-request builds where possible. Restrict network access and permissions for build jobs that do not need them, and use isolated environments for unfamiliar code. If a package behaves badly, a restricted runner gives it less room to cause harm.
- Test behavior, not just syntax. Run unit and integration tests, static analysis, and security checks that fit the code. For authentication, verify failure cases, session handling, rate limits, and account recovery. For cloud policies, compare each permission with the operation the service actually performs. Passing tests is evidence, not a security certificate.
- Check critical advice against primary sources. For identity and cryptography, consult current standards and trusted security guidance rather than asking a model to certify its own answer. The NIST Digital Identity Guidelines and CISA’s phishing-resistant MFA guidance are useful reference points. When the generated code handles secrets or identity, require review by someone who understands the threat model.
- Measure what your tools get wrong. Track how often suggestions introduce unapproved packages, fail tests, or trigger security rules. Keep the measure narrow enough to act on; a raw count of AI-generated lines says little about risk. OWASP’s guidance on misinformation in large language models recommends checking claims against trusted sources and maintaining review for consequential decisions.
I have seen teams get better results when these checks live in the normal pull-request path, rather than in a separate policy document nobody opens. Automate the easy checks. Save human attention for intent, context, and consequences. That split is practical, not flashy, and it catches mistakes before they become routine.
AI Suggestions vs. Developer-Led Coding
The closest comparison is not between an AI tool and a perfect human. It is between AI-assisted work and developer-led work, both of which can fail. A person can choose an unsafe library or overlook a bad permission. A model can make those errors faster and repeat them in convincing language. The controls should reflect those differences without pretending either method is automatically safe.
| Area | AI-assisted coding | Developer-led coding |
|---|
| Draft speed | Often fast for boilerplate and familiar patterns | Depends on experience and task familiarity |
| Source of suggestions | Learned patterns, prompt context, and tool integrations | Direct experience, documentation, and team knowledge |
| Dependency risk | May suggest a fabricated or unfamiliar package | May select an unsuitable or poorly maintained package |
| Repeatability | The same false suggestion can recur across prompts | Similar mistakes can recur through habits or copied code |
| Security review | Needs checks for invented facts and unsafe confidence | Needs checks for bias, stale knowledge, and missed threats |
| Best safeguard | Registry checks, policy automation, tests, and human review | Peer review, tests, dependency controls, and current guidance |
The package study gives one measurable warning, not a scorecard for every coding assistant. Its finding of roughly one in five fabricated package recommendations applied to its tested models and prompts. Do not turn that number into a claim that one in five AI-written projects contains a fake dependency. Different prompts, versions, tools, and workflows can produce different results.
AI tools have a particular weakness: an invented detail can arrive wrapped in fluent reasoning. Human authors have a different weakness: confidence, habit, and pressure can make a familiar shortcut seem safe. A senior engineer can also miss a package name while scanning a large review. That is why a reliable workflow uses more than one control.
When I compare these methods, I do not ask which one is always safer. I ask what failure each one makes more likely and how quickly the team will notice it. For AI-generated dependencies, registry checks and dependency review matter. For authentication, tests and a knowledgeable security review matter more than a tidy explanation. Use the right control for the risk in front of you.
Common Mistakes to Avoid
Mistake one: trusting code because it compiles. Compilation checks whether code follows rules the compiler understands. It does not confirm that a package is trustworthy, that an API is used safely, or that the application’s threat model was understood. I have seen clean builds create a false sense of closure. A green check means one test passed; it does not answer every security question.
Mistake two: checking only whether a package exists. A real package can still be malicious, abandoned, compromised, or simply wrong for the job. Verify its owner, release history, source, and dependencies. For important software, check available provenance information and follow your organization’s approval process. Frankly, a name in a registry is the start of a review, not the end.
Mistake three: assuming a lockfile makes a dependency safe. Lockfiles help pin versions and make installations repeatable. That repeatability can also make an unsafe choice persist. Review the change that introduced the package and remove it if its purpose or origin cannot be established.
Mistake four: treating retrieval-augmented generation as a cure. Retrieval can ground an answer in documentation or code that you provide. That is useful, but a model can still misread a page, cite the wrong version, or invent a detail that the source never said. Check the cited passage yourself, especially when the result affects identity, encryption, or access control.
Mistake five: asking the model to approve its own security work. A second prompt may catch a simple error, but it is not an independent audit. Use tests, static checks, policy rules, and people who can question the design. The NIST Generative AI Profile is a useful framework for thinking about risks and controls across an AI system; it does not replace review of a specific code change.
Mistake six: giving build jobs too much access. A malicious package can do more damage when it can read secrets, reach sensitive services, or publish releases. Use limited permissions and separate trusted release jobs from less-trusted testing. Here’s the plain truth: controls matter most when something slips through.
I once watched a team focus on whether a generated patch was elegant while leaving a new dependency’s purpose unexplained. The fix was not a lecture about AI. It was a review rule: every new dependency needs an owner, a reason, and a verification trail. Simple rules beat vague warnings because reviewers can apply them under deadline pressure.
Frequently Asked Questions
What is an AI coding hallucination?
It is a plausible-looking code suggestion or technical claim that is fabricated or incorrect. Examples include an invented package, a function that does not exist, or security advice that no longer fits current guidance. It may fail immediately, but it can also compile and hide the error until the code is used.
What is slopsquatting?
Slopsquatting is a supply-chain attack in which an attacker registers a package name that a language model has repeatedly hallucinated. A developer may then install that package after trusting the suggestion. It is related to typosquatting, but the mistaken name can originate with the model rather than a human typo.
How can developers check an AI-suggested dependency?
Confirm the exact package name on the correct registry, then inspect its owner, release history, source, and dependencies. Check whether it is approved for your project and review the proposed version and lockfile changes. Do not treat registry presence or a successful build as proof that a package is safe.
Can retrieval-augmented generation stop coding hallucinations?
No. Retrieval-augmented generation can give a model approved documentation or code to use as context, which may reduce unsupported answers. It cannot guarantee that the model interpreted the source correctly or that the source is current. Check important claims against the cited documentation and test the resulting code.
Final Thought
AI coding tools are useful assistants, not authorities. Their speed can help a team make room for careful work, but only if the team keeps responsibility for what it ships. A generated answer does not own the consequences; the people and organizations deploying that code do.
That responsibility can be made concrete. Verify dependencies before installation, restrict build permissions, test security behavior, and check critical guidance against trusted sources. Reviewers should be able to ask why a package is there and what each permission allows without being treated as obstacles. That is part of respecting the work and the people who rely on it.
I would use a coding assistant for a first draft. I would not hand it the final say on a package, an authentication flow, or a cloud policy. Keep the useful speed. Keep the human judgment. When a tool gives you code that looks certain, ask the simple question that protects the whole system: what evidence says this is true?