AI coding hallucinations are confident-looking mistakes produced by AI tools, such as invented package names, incorrect function calls, or unsafe security...
AI Hallucinations in Coding: How to Stop Plausible Errors Becoming Security Risks
AI coding hallucinations are confident-looking mistakes produced by AI tools, such as invented package names, incorrect function calls, or unsafe security advice. They matter because developers may run generated code before checking it, giving a plausible error access to source code, build systems, or customer data.
Key Takeaways- Treat code and dependencies suggested by AI as untrusted until checked.
- Validate package names against official registries and review lockfile changes.
- Use automated security checks and human review for authentication, permissions, and cryptography.
- Track mistakes so you can see whether your safeguards work.
I’m Ermenildo V Castro, and I’ve spent years watching teams balance speed with the hard work of making software dependable. A neat answer is not necessarily a correct one. That gap matters more when a coding assistant can draft a dependency install command or an authentication flow in seconds, and the result looks polished enough to pass a hurried review.
This is not an argument for banning coding assistants. They can help explain unfamiliar code, draft tests, and speed up routine work. It is an argument for setting boundaries: the tool can propose, while engineers and automated controls verify. When an answer can affect a real system, confidence is not evidence.
What Are AI Hallucinations in Coding?
An AI hallucination is an output that appears credible but contains fabricated or inaccurate information. In coding, that might be a package that does not exist, a function with invented arguments, or a security recommendation that no longer meets current guidance. The model is not checking truth in the way a compiler, package registry, or security engineer does.
Language models generate text by predicting likely next tokens from patterns learned during training. They can produce useful code because programming languages, documentation, and examples contain recurring structures. But a model may also fill a knowledge gap with a plausible guess. It can write that guess with the same smooth tone it uses for a sound answer. That makes fluency a poor test of correctness.
Code adds its own traps. A made-up package name can follow the naming style of a real library. An invented method can look like the other methods in an SDK. The code may even compile if the questionable line is not reached, or if a different dependency happens to provide a similar symbol. A green build is helpful; it is not proof that the code is safe.
I have seen this in review: a tidy code change, a familiar-looking import, and a description that made the choice seem obvious. The useful question was not whether the answer sounded right. It was whether the package existed, had a maintained release, and came from the intended owner. That small check changed the review from a style discussion into a supply-chain check.
Researchers studying code-generating models have documented package hallucinations. One study tested 16 models and reported that roughly one in five recommended dependencies in its test set referred to nonexistent packages, with more than 205,000 unique fabricated package names observed. Those figures describe the study’s tests, not a universal rate for every tool or project. Still, the result shows why package suggestions deserve verification. See the study, We Have a Package for You!.
The same problem reaches beyond libraries. A model can suggest an outdated authentication design, a weak cryptographic choice, or cloud permissions broader than the task needs. Those are not merely factual slips. They can become security defects when copied into production code without review.
Why AI Hallucinations Matter to Software Security
A hallucinated package can become a supply-chain opening. If a developer asks for a library and accepts an invented name, an attacker may register that name on a public package registry first. The developer then installs software controlled by someone else. This attack has been called slopsquatting: instead of exploiting a developer’s typo, an attacker tries to claim a name that an AI model repeatedly invents.
The risk is not limited to a single bad suggestion. If the same model tends to repeat a fabricated name, an attacker may have repeated chances to catch an install. Build systems often hold valuable permissions, such as access to source code, package credentials, or deployment environments. A malicious install can therefore do more than break a local test. Its impact depends on what the installation process can reach.
Security advice can fail in quieter ways. A generated authentication example might rely on SMS codes without explaining the risks or considering stronger options. Current digital identity guidance treats SMS-based codes as restricted, while phishing-resistant methods such as passkeys can offer better protection for many high-value accounts. That does not mean passkeys fit every system or solve every authentication problem. It means generated advice should be compared with current guidance, such as NIST’s Digital Identity Guidelines and the FIDO Alliance’s passkey overview.
I once reviewed an authentication change where the explanation sounded more certain than the security model deserved. The important work was tracing what happened when a user lost a device, reused a phone number, or encountered a phishing page. The example code had not answered those questions. A reviewer had to ask them.
Cloud permissions create another route to trouble. An assistant may return a policy granting broad access because it is easier to express than a narrow permission set. Cryptography has a similar failure mode: familiar names and old examples can survive in generated answers long after security advice changes. In each case, the immediate code may look reasonable, while the consequences show up only under attack or during an incident.
There is a human cost, too. A compromised build can expose customer information, disrupt services, and create long hours for the people responsible for fixing it. Secure engineering is partly a matter of stewardship: protecting the systems and information entrusted to a team is ordinary professional responsibility, not extra polish.
Step-by-Step: Build Guardrails for AI-Generated Code
A policy that says “review AI code carefully” is too vague to protect a real pipeline. Reviews happen under deadlines, and fluent output can slip past tired eyes. Use checks that run every time, then reserve human attention for questions that need judgment.
- Set the boundary before using the tool. Decide where generated code may be used and where it needs extra review. A test stub or a small formatting helper carries a different risk from authentication, payment handling, cryptography, or deployment permissions. I have found it useful to label sensitive areas clearly in team guidance. That keeps the rule understandable: higher impact means stronger verification.
- Check dependencies against official registries. Before installing a suggested package, confirm the exact name in the registry your project uses, such as npm or PyPI. Check its publisher, release history, documentation, and whether the project is maintained. Do not assume that a name is legitimate because an AI assistant suggested it or because it resembles a known library. For widely used packages, check whether the project has a trusted source repository and whether releases follow a consistent pattern.
- Review dependency changes, not just application code. Use lockfiles to record the exact package versions being installed, and examine lockfile changes in code review. Set dependency allowlists or approval rules where they make sense. Automated dependency review can flag newly introduced packages and known vulnerabilities; GitHub documents its approach in About dependency review. A clean vulnerability report does not prove a package is trustworthy, but it can catch known problems early.
- Make installs and builds less powerful. Run dependency installation and tests with only the permissions they need. Keep secrets out of ordinary build jobs when possible, use short-lived credentials, and separate pull-request checks from release jobs. If a dependency is malicious, limited access can reduce the damage. This is basic least privilege, applied to the place where generated code often first runs.
- Test security behavior, not just whether code runs. Ask what happens when a user has no permission, submits malformed input, loses an authentication device, or tries to access another person’s data. Add tests for those cases. For security-sensitive changes, compare the implementation with trusted standards and your organization’s approved patterns. A test that proves a function returns a value may not prove that the function protects the right person.
- Ground answers in sources you control. Retrieval-augmented generation can give a model relevant internal documentation or approved examples to draw on. That can reduce unsupported guesses, but it does not guarantee correctness. Documentation can be old, retrieval can select the wrong page, and a model can still misread a passage. Treat grounding as a better starting point, not as a substitute for verification.
- Measure what your controls catch. Track issues such as nonexistent dependencies, unapproved packages, vulnerable versions, and security fixes required after AI-assisted changes. Review the numbers over time and adjust checks when a pattern repeats. Published studies can help teams design tests, but your own tools, prompts, and codebase may behave differently. The practical question is whether your safeguards catch mistakes before release.
- Keep a human accountable for high-impact decisions. Require an engineer to approve changes that touch authentication, authorization, cryptography, sensitive data, or production infrastructure. The reviewer should understand the behavior, not simply confirm that an AI-generated explanation sounds reasonable. If nobody on the team can explain a security-sensitive change, pause the merge and get the right expertise.
The OWASP GenAI Security Project includes misinformation among its risks for large language model applications. Its guidance supports checking outputs against trusted sources and keeping suitable human oversight. Read the OWASP misinformation guidance. The point is not to add a ceremonial approval step. It is to make verification part of the work, where it can happen consistently.
Comparison: AI Package Hallucination vs. Typosquatting
Slopsquatting and typosquatting both exploit package names, but they start in different places. Typosquatting relies on a person entering a name that is close to a legitimate package but spelled incorrectly. Slopsquatting relies on an AI assistant generating a plausible name that may not yet exist. When an attacker registers that suggested name, a developer who trusts the output can install the attacker’s package without making a typo.
| Risk | AI package hallucination and slopsquatting | Traditional typosquatting |
|---|
| How the package name appears | An AI tool suggests a fabricated but plausible name | A user mistypes or misremembers a real name |
| Attacker’s opportunity | Register a name models suggest repeatedly | Register common misspellings or look-alike names |
| What the developer sees | A polished recommendation that looks intentional | A package name that may look slightly wrong |
| Detection | Verify names against the official registry; review package ownership and changes | Check spelling, publisher, package metadata, and dependency records |
| Potential reach | May recur if a model repeats the same suggestion | Depends on how often users make the targeted typo |
The model study described earlier reported more than 205,000 unique fabricated names across its tests and found that some names recurred across prompts. That is evidence about the tested models, not a count of malicious packages or successful attacks in the wild. The distinction matters. Research output can indicate an opportunity for attackers, but it does not tell us how many developers have installed malicious packages as a result.
Frankly, a name that looks odd is easier to question than a name delivered in a confident code block. That makes process more reliable than intuition. Checking package existence and publisher identity is useful for both attack types, and a lockfile helps preserve the exact dependency version that passed review. Neither control replaces the other.
Common Mistakes to Avoid
Mistake 1: Treating a confident explanation as proof. AI-generated answers can sound complete because they contain the same polished language as correct ones. I have seen reviewers spend more time discussing the explanation than checking the behavior. Start with evidence: official package records, current standards, tests, and the code itself. Confidence is a writing style, not a security property.
Mistake 2: Assuming a successful build means the package is safe. A compiler checks whether code follows rules it understands. It does not reliably establish that a dependency has a trustworthy maintainer or harmless behavior. A malicious package can install cleanly and pass basic tests. Verify identity and provenance separately from build success.
Mistake 3: Reviewing only the visible source file. The code change may include new dependencies, altered lockfiles, workflow changes, or broader permissions. Here’s the kicker: a two-line source change can trigger a much larger install or build step. Review the full diff, including configuration and generated files, and ask what runs during installation and deployment.
Mistake 4: Using allowlists without maintaining them. An allowlist can reduce surprise, but a stale list may block needed updates or encourage shortcuts around the control. Define who can approve additions, what evidence they should check, and how exceptions expire. A guardrail nobody can use sensibly will eventually be bypassed.
Mistake 5: Treating retrieval as a truth machine. Supplying approved documentation can improve an answer, but it does not force the model to follow every rule or quote the right page. The documents might be incomplete or outdated. Keep source dates visible, review high-risk outputs, and test whether the system cites material that actually supports its recommendations.
Mistake 6: Asking an AI tool to grade its own security work. A second prompt can catch simple mistakes, but the same model may repeat the first error or miss an issue for the same reason. Use independent tests, static analysis, dependency checks, and people who understand the relevant security area. A second opinion is useful only if it brings a different way of checking.
Mistake 7: Making human review responsible for everything. “Be careful” does not scale when a team receives more code than reviewers can examine closely. Automate registry checks, policy checks, lockfile review, and tests. Then use human review for design choices, threat assumptions, and cases where the automated tools cannot judge intent. Most guides praise human review; fewer admit that attention is limited.
Mistake 8: Skipping ownership and incident planning. Teams should know who can remove a dependency, revoke build credentials, and respond if a package is compromised. Keep a record of why sensitive dependencies were approved. When something fails, a clear trail of decisions helps people act quickly and fairly instead of guessing under pressure.
Frequently Asked Questions
What is an AI coding hallucination?
It is code-related information generated by an AI system that looks believable but is false or unsupported. Examples include nonexistent package names, invented API methods, incorrect security explanations, or permissions that exceed the task. Verify the details against trusted sources rather than relying on how polished the output looks.
What is slopsquatting?
Slopsquatting is a supply-chain attack in which someone registers a package name that an AI coding tool may hallucinate. A developer who installs the suggested name could then run software controlled by the attacker. The term describes a possible attack method; it does not mean every fabricated package name has been registered or exploited.
How can a team check whether an AI-suggested package is real?
Search for the exact name in the official package registry, then check the publisher, release history, source repository, documentation, and project health. Review the dependency diff and lockfile before merging. If the package is unfamiliar or the evidence is thin, do not install it just because the generated code needs it.
Can retrieval-augmented generation prevent coding hallucinations?
No. Retrieval can provide relevant documentation and reduce unsupported answers, but it cannot guarantee that a model selects or interprets the right information. Documents can be outdated, and generated code can still contain mistakes. Use retrieval alongside tests, dependency checks, security review, and clear limits on what the tool may change.
Should developers stop using AI coding assistants?
Not necessarily. These tools can help with routine code, tests, and explanations when teams verify the result. The sensible choice is to match safeguards to impact: lightweight checks for low-risk suggestions, stronger review for dependencies and security-sensitive code. Keep the engineer accountable for what reaches production.
Final Thought
AI coding tools can save time. They can also produce a bad answer faster than a team can read it. That is why the answer is not simply to tell developers to be more alert. A safer approach places checks where the work happens: at dependency installation, during review, in tests, and before a build receives sensitive permissions.
In my experience, dependable engineering comes from small habits repeated under ordinary pressure. Confirm the package name. Read the permission change. Ask what the code does when a user is not supposed to have access. Those steps may feel slower in the moment, but they protect the people who rely on the service and the coworkers who maintain it. Good tools should support that care, not replace it.
The useful promise of AI assistance is faster drafting, not automatic trust. Let the model suggest. Let automated checks catch what they can. Let a qualified person own the decisions that affect security and customer data. I’m Ermenildo V Castro, and I’d rather review one more dependency today than explain an avoidable compromise tomorrow.