AI Code Hallucinations: How to Stop Fake Packages and Unsafe Security Advice
AI code hallucinations are plausible-looking errors. They occur when a coding model invents a package, API, explanation, or security rule, then presents it with the tidy confidence of something checked—even when it may be false, unsafe, or simply nonexistent. That confidence is the risk.
These errors matter because AI assistants can move from a question to a working-looking code change in seconds. A made-up package can look real enough to install, while a flawed authentication example can pass a quick review because it compiles. In my experience reviewing software, the most dangerous mistakes are often not the loud ones. They are the small, reasonable-looking choices that nobody stops to check.
This is not an argument for banning coding assistants. Used with care, they can help explain unfamiliar code, draft routine tests, and speed up repetitive work. The practical goal is narrower: keep the useful speed while adding checks that catch false claims before they reach a build or a customer. Software is work entrusted to us, and good engineering includes care for the people who rely on it.
What Are AI Code Hallucinations?
An AI code hallucination is an output that looks like a reliable fact or working code but is invented, incorrect, or unsupported. A model does not look up truth in the way a person checks a manual. It predicts likely next pieces of text from patterns it learned, and those patterns can produce an answer that sounds certain even when the model lacks the facts needed to support it.
That difference matters. A model can give you a real function name with the wrong arguments, a package that resembles a popular library, or a security explanation borrowed from an older version of a standard. It may produce code that is valid in form but wrong in effect. A compiler can catch a missing bracket; it cannot tell you that the package name is a trap or that a permission is broader than the task needs.
I have seen reviews stall over a tiny question: “Is this library actually the one we meant to use?” The generated import looked ordinary, and the surrounding code looked clean. Yet a quick check of the registry and the project’s dependency list showed that the team had no reason to add it. That pause saved more trouble than another round of polishing the code would have.
The risk grows when a model fills a gap in its knowledge with something plausible. Library names follow naming conventions. Function signatures borrow familiar patterns. Security advice may sound like a real standard because it uses the right vocabulary. The output can read smoothly without being grounded in current documentation.
Researchers have measured this problem in package suggestions. In a study of 16 code-generating models, the researchers reported that roughly one in five recommended package references pointed to packages that did not exist, and their test set contained more than 205,000 unique fabricated package names. Those results describe the study’s prompts and models, not a universal rate for every coding tool. Still, the scale is a useful warning. The paper, “We Have a Package for You!”, examines package hallucinations directly.
A fabricated package name can become a supply-chain attack when someone registers it on a public package registry and waits for a developer to install it. Security researchers call this pattern slopsquatting, a twist on typosquatting: the attacker does not need the developer to mistype a real package name if a model repeatedly invents a convincing one. An installation may run scripts or add code with access to build secrets, so the risk does not stop at a bad suggestion.
Why AI Hallucination Security Matters
AI assistance can save time. It can also make a wrong answer easier to accept. The benefits are real, but they depend on checking the output instead of treating speed as proof.
- It reduces routine drafting work. A model can create a first pass for a test, a small data conversion, or a repeated code pattern. I have used this kind of draft as a starting point, then spent my attention on edge cases and behavior rather than typing every line by hand. The draft is useful; the review still belongs to the engineer.
- It can help people understand unfamiliar code. Asking for a plain-language explanation may surface a useful path through an old service or a new framework. But explanations can leave out important limits. Check claims against the source code and current documentation, especially when the answer names an API or security feature.
- It makes review discipline more important. A suggestion that arrives in seconds can introduce dependencies, permissions, and authentication logic faster than a team can inspect them manually. Automated checks make review repeatable instead of relying on someone noticing every detail under deadline pressure.
- It can reveal assumptions worth testing. When a model proposes an approach, a developer can ask which inputs it expects, which failures it handles, and what evidence supports a library choice. Those questions are useful even when the original suggestion is wrong. Frankly, a confident explanation is often a cue to verify, not a reason to skip verification.
- It can support safer work when its role is limited. Use an assistant to draft or explain; use tests, trusted documentation, and security tools to check. The distinction is simple, but teams blur it when a generated answer looks finished.
The cost of a hallucination depends on where it lands. A bad variable name is annoying. A false claim about session expiry, cryptographic settings, or cloud access can expose real users or data. Authentication, authorization, secrets, payment handling, and deployment settings deserve a higher bar than a small formatting helper.
Security advice needs special care because it ages. For example, a model may produce an SMS one-time-code flow as if it were the strongest option for every account. SMS can serve some use cases, but phishing-resistant methods are a better fit for high-value accounts where the threat calls for them. CISA recommends phishing-resistant MFA for stronger protection, and the FIDO Alliance’s passkey guidance explains how passkeys use public-key cryptography. Passkeys reduce the risk of password reuse and phishing, though account recovery and device access still need thought.
The same problem can appear in encryption advice, cloud IAM policies, input validation, and logging. A model might suggest an old cipher suite, grant a service broad permissions, or log a token that should remain secret. Those snippets can look familiar enough to pass a casual glance. In a production system, the people affected by that shortcut are not abstractions. Their privacy and access deserve careful handling.
How to Add Guardrails to an AI-Assisted Workflow
A reliable workflow puts checks around the model, not just trust in the person reading its answer. Here is a practical sequence that works for dependency suggestions and security-sensitive code.
- Set the boundaries before asking. Tell the assistant which language, framework version, approved libraries, and runtime constraints apply. Give it a narrow task. “Use our approved authentication library and explain any new dependency” is safer than “build secure login.” Clear limits do not guarantee a correct result, but they reduce room for invented choices.
- Verify every dependency before installation. Search the official registry for the exact name. Check its publisher or maintainer, source repository, release history, documentation, and whether the project already uses it. Look for names that differ by a character from a known package. If you cannot establish why the dependency is needed and who maintains it, do not install it just because the assistant recommended it.
- Use controls in the repository and build pipeline. Keep dependency versions pinned in lockfiles, review lockfile changes, and use dependency review or software-composition analysis tools to flag new or vulnerable packages. An allowlist can help for sensitive applications, though it should have a clear process for approved additions. GitHub’s dependency review documentation describes checks teams can use to review changes to dependencies.
- Inspect install behavior and limit build privileges. Some packages run installation scripts. Use a controlled environment for new or untrusted dependencies, and avoid giving ordinary build steps access to production credentials. Short-lived tokens, limited permissions, and isolated runners reduce the damage if a dependency behaves badly. A lockfile helps make installs repeatable; it does not prove that a package is safe.
- Check security code against current, trusted sources. Do not ask a model to invent cryptography or set policy from scratch. For authentication, authorization, cloud permissions, and encryption, compare its proposal with vendor documentation, recognized standards, and your organization’s approved patterns. The OWASP guidance on LLM misinformation calls attention to unsupported or incorrect model output and the need for verification.
- Test behavior, not just syntax. Run unit and integration tests, static analysis, dependency scanning, and secret scanning. Add tests for denied access, expired sessions, invalid input, and error paths. A green build tells you that certain checks passed; it does not tell you that the design is secure or that every important case was tested.
- Require human review for consequential changes. A qualified reviewer should examine code that touches identity, access control, cryptography, data handling, and deployment policy. Ask the reviewer to verify the reasoning and sources, not just whether the code looks tidy. In review sessions I have found that one direct question—“What gives this service permission to do that?”—can uncover a broad access rule hiding in a neat snippet.
Retrieval-augmented generation, or RAG, can help by grounding an assistant’s answers in documentation chosen by the team. It can reduce reliance on old or general training patterns, but it is not a truth machine. The documents can be stale, incomplete, or irrelevant, and the model can still misread them. Keep the source set current and make citations easy to inspect.
Measurement matters too. Track how often AI suggestions add new dependencies, how often reviewers reject generated security code, and what kinds of defects tests find. Do not turn these numbers into a contest to reduce all review comments; that would reward teams for missing problems. Use the measures to find weak points in your process and improve them over time.
AI-Assisted Coding vs. Manual Coding
The useful comparison is not “AI good” versus “AI bad.” It is the set of checks each approach needs. Manual code can contain mistakes, outdated patterns, or malicious dependencies too. An assistant changes the speed and volume of suggestions, which can put pressure on review habits.
| Concern | AI-assisted coding | Manual coding |
|---|
| First draft speed | Often fast for routine patterns and examples | Depends on the developer’s familiarity and typing time |
| Package risk | May invent plausible names or recommend unneeded dependencies | Can still include typos, outdated packages, or poor choices |
| Explanation quality | Fluent, but may state unsupported claims with confidence | Depends on the author’s knowledge and willingness to check |
| Review need | Requires checking generated logic, sources, and dependencies | Requires normal code review and security testing |
| Best use | Drafting, explanation, test scaffolding, and repetitive tasks | Design decisions, ownership, and domain-specific judgment |
| Main safeguard | Automated verification plus human review | Tests, review, and trusted documentation |
Manual work is not a magic shield. Developers can copy a questionable answer from a forum, reuse an old pattern, or grant excessive access because a deadline is close. The difference is that AI can produce many polished suggestions in a short time, making it easier for teams to accept more changes than they can thoughtfully inspect.
In practice, I treat an assistant like a quick junior contributor who needs supervision: useful for a first draft, not responsible for the final decision. That is not an insult to the tool. It is a sensible way to assign responsibility. The person merging the code still owns the consequences, whether the first draft came from a model, a teammate, or an old snippet.
Common Mistakes to Avoid
The biggest error is confusing fluent output with verified output. That mistake is easy to make because the answer often has clean formatting, plausible names, and a tidy explanation. Here are habits that make the problem worse.
- Installing a package before checking its identity. A package name in generated code is only a suggestion. Confirm it in the official registry, inspect its history and source, and ask whether the project needs it at all. For unfamiliar packages, check for suspiciously thin documentation or a recent first release that does not match the assistant’s claim.
- Treating “it compiles” as “it is safe.” Compilation confirms that code meets certain language and dependency rules. It does not confirm correct authorization, safe cryptography, or appropriate data handling. I have watched perfectly valid code grant much more access than its feature required. A clean build can still contain a serious design flaw.
- Letting the assistant design security from a blank page. Broad prompts invite broad guesses. Ask it to explain an approved pattern or draft code within clear constraints, then compare the result with trusted guidance. If you cannot explain the security behavior without the model’s text, the change is not ready to merge.
- Assuming more context means more truth. Long prompts and large document collections may improve relevance, but they do not guarantee correctness. Check which sources the system used, whether those sources are current, and whether the quoted section supports the claim. A citation that merely looks official is not enough.
- Giving builds broad secrets and permissions. A package or script running in a build has the permissions the environment grants it. Use narrow credentials, short-lived access, and isolation. Do not hand every test job a production key for convenience.
- Skipping a human review because a tool passed. Static analysis, dependency scanners, and tests catch different classes of faults. None understands every business rule or threat. Automation should reduce repetitive work and catch common errors, not become a reason to stop thinking.
Most advice says “keep a human in the loop,” then leaves the hard part unexplained. The truth is that review does not scale if it depends on attention alone. Put the checks where the work happens: pull-request rules, package policies, automated tests, and clear ownership for high-risk code. Then reserve human judgment for decisions that tools cannot settle.
Frequently Asked Questions
What is an AI coding hallucination?
It is generated code or technical advice that appears credible but is false, invented, outdated, or unsupported. Examples include nonexistent package names, made-up API methods, incorrect documentation claims, and unsafe security patterns. The code may still look polished or even compile. Verify the source and behavior before relying on it.
Can AI-generated code be safe to use?
Yes, when it is treated as a draft and checked like other code. Verify dependencies, run relevant tests and security tools, compare sensitive logic with current trusted guidance, and require review for high-impact changes. No single check proves safety. The right level of review depends on what the code can access and whom it can affect.
What is slopsquatting?
Slopsquatting is a supply-chain attack in which someone registers a package name that a language model is likely to invent. A developer who accepts that suggestion may install the attacker’s package instead of a real dependency. Checking the exact package name, its history, and its publisher before installation helps prevent this.
Do passkeys make authentication secure by themselves?
No. Passkeys can provide phishing-resistant authentication and avoid password reuse, but a secure account also needs careful recovery, session handling, access controls, and protection for devices. Choose authentication methods based on the risks and users involved. Do not accept a generated login flow without checking it against current guidance and your system’s needs.
Final Thought
AI coding assistants are useful when they reduce the dull work without taking responsibility away from the people who build and run software. Their answers can be fluent, fast, and wrong all at once. That is not a reason to panic, and it is certainly not a reason to stop reviewing code. It is a reason to make verification ordinary.
I have learned that the best engineering habits are often unglamorous: check the package, read the diff, test the failure path, and ask what a permission actually allows. Those steps can feel slow when a generated answer arrives in seconds. But a small pause before installation is cheaper than cleaning up a compromised build, and a careful review of authentication code is cheaper than explaining an avoidable account exposure.
Keep the assistant in its proper role. Let it draft, explain, and suggest. Let tests, trusted references, automated controls, and accountable engineers decide what ships. Build those checks into the pipeline, and the speed can remain without asking users to bear the cost of guesses. That is good stewardship of both technical skill and the trust people place in the software we make.