AI coding assistants can invent convincing falsehoods. Their suggestions may look authoritative while naming nonexistent packages, misusing APIs, or proposing...
AI Hallucinations in Coding: The Hidden Risks Behind Confident Code
AI coding assistants can invent convincing falsehoods. Their suggestions may look authoritative while naming nonexistent packages, misusing APIs, or proposing weak security patterns, and a fast-moving build pipeline can turn those errors into dependencies, exposed services, or costly fixes. Treat every generated answer as a draft, not evidence.
Key Takeaways- A plausible answer from an AI assistant is not proof that it is correct.
- Fabricated package names can create supply-chain risk when someone registers them.
- Check dependencies, authentication, cryptography, and cloud permissions before merging.
- Measure mistakes and use human review for changes that affect security or users.
I’m Ermenildo Castro, and after years working on software at Google, Microsoft, Amazon, and Zulily, I’ve learned that polished code can still hide a very ordinary failure: nobody checked the assumption underneath it. Coding assistants make that failure easier to miss because they answer quickly and rarely look uncertain. Speed is useful; unearned trust is not.
This matters even when a team uses a well-known model and a tidy development process. Models can produce sound explanations alongside incorrect details, and the output may pass a quick visual review. The right response is not to ban these tools. It is to build checks around them, especially where code touches dependencies, identity, encryption, or production access. A careful review protects the people who rely on the software, even if they never see its source code.
What Are AI Hallucinations in Coding?
An AI hallucination is an output that sounds like a factual answer but contains invented or incorrect information. In code, that might be a package that does not exist, a library method with the wrong arguments, or a security recommendation that no longer fits current guidance. The model is not necessarily trying to deceive anyone. It is producing likely text, not certifying facts.
Most language models learn patterns from large collections of text and code. Given a prompt, they predict a sequence of likely tokens based on the prompt and their learned patterns. If the training examples do not cover a detail, or the model cannot reliably distinguish it from a similar one, it may fill the gap with a plausible guess. That guess can have the same calm tone as a correct answer. Frankly, tone tells you very little.
Code makes this tricky. Package names tend to follow familiar naming styles, and invented function names can look right beside real ones. A snippet might even compile if a project has a similarly named helper or if the mistake sits in a path that tests never exercise. A successful build is useful evidence, but it does not prove that a dependency is safe or that an authentication flow is sound.
I’ve reviewed generated code where the structure looked clean enough to invite a quick approval. Then one small detail—a method that was not in the SDK documentation, or an unpinned package pulled from a public registry—changed the review completely. The lesson was not that every generated line is suspect. It was that fluent output can hide uncertainty, so reviewers need to check the claims that matter.
A well-studied example is package hallucination. Researchers testing 16 code-generating models reported that roughly one in five package recommendations in their study pointed to packages that did not exist, with more than 205,000 unique fabricated package names across their test set. Those figures describe the models and test conditions in that research; they are not a universal rate for every assistant or prompt. See the researchers’ analysis of package hallucinations for the study details.
When a fabricated name is registered by an attacker, a careless install can pull in code the team never meant to trust. Some package managers run installation scripts, and dependencies may execute later during a build or application run. The exact exposure depends on the package manager and configuration, but the basic risk is simple: a dependency decision can grant code access to a machine, pipeline, or application.
Why AI Hallucination Controls Matter
Good guardrails let a team use coding assistance without pretending it is an authority. The payoff is fewer preventable mistakes, more useful reviews, and a clearer record of why a package or security choice was accepted. Here’s the kicker: controls are not just a brake on speed. They make fast work safer to repeat.
- They reduce supply-chain exposure. Checking package names against trusted registries, reviewing changes to lockfiles, and limiting approved dependencies can catch a fabricated suggestion before it enters a build. That is especially useful when developers work across several languages and package managers.
- They make code review more focused. A reviewer should not have to guess whether a new import exists. Automated checks can verify package identity and version, leaving people to judge whether the dependency belongs in the design and whether the code uses it safely.
- They protect security-sensitive paths. Authentication, cryptography, authorization, secrets handling, and cloud permissions deserve more scrutiny than a formatting helper. Generated code in these areas can look reasonable while quietly weakening a control.
- They improve incident response. A clear record of the prompt, model version, dependency changes, test results, and approval decisions helps a team work out what happened. It also helps teams notice repeated failure patterns instead of treating each one as a surprise.
- They preserve time for useful work. Catching a bad dependency before it spreads is usually cheaper than removing it from several services, rebuilding images, and reviewing possible exposure. The boring checks often save the exciting recovery plan.
I once watched a review stall over a generated authentication example that mixed a familiar flow with a security choice the team had not approved. The code looked tidy, and the explanation sounded confident. We had to stop and check the current guidance before deciding what to ship. That pause was not wasted time; it kept a convenience sample from becoming a production rule.
Security advice ages. SMS one-time codes, for example, still have legitimate uses, but they are not a phishing-resistant factor and can face risks such as account recovery abuse, SIM swapping, or interception. For important accounts, teams should assess phishing-resistant options such as passkeys and hardware security keys, along with recovery and device policies. CISA’s guidance on phishing-resistant multifactor authentication explains why stronger MFA matters. Passkeys also need sound implementation and recovery plans; no single feature fixes a careless identity system.
Similar caution applies to cryptography and cloud access. A model might suggest an old cipher, misuse a secure library, or write an IAM policy with broad permissions that were never requested. A correct-looking syntax check cannot tell you whether the permission is necessary. The principle of least privilege remains a human design decision, not a sentence to accept because it reads well.
This is also a matter of responsibility. Software choices affect customers, coworkers, and people who have little say in how a system is built. Treating their safety as part of the job is not ceremony; it is ordinary care in professional work.
Step-by-Step: Build Guardrails Into Your Pipeline
A useful process starts before a pull request reaches a reviewer. It does not require a grand AI policy or a new department. It requires a few specific checks, applied consistently, with stricter review where a mistake could harm users or expose systems.
- Set boundaries for AI use. Write down which tasks assistants may support and which require approved patterns or a specialist review. For example, a model can draft a test or explain a familiar API, while changes to login, authorization, key management, or production IAM need an owner who understands the system. Keep secrets and private customer data out of prompts unless your approved tool and policy explicitly allow them. I’ve found that a short, clear rule beats a long policy nobody can recall.
- Verify every new dependency. Before installation, confirm the exact package name, registry, publisher or maintainer details, repository, and intended purpose. Check that the package exists in the expected registry and that its documentation matches the proposed use. Similar names are not proof of identity. Do not let a generated command choose a package simply because it is the first result in a chat window.
- Control versions and lockfiles. Require reviewed lockfile changes and use the package manager’s normal integrity checks. Pinning versions and recording integrity information make builds more repeatable, though neither proves a package is harmless. Review transitive dependencies too: one top-level library can bring along a large tree of code. GitHub’s documentation on dependency review describes one way to inspect dependency changes in pull requests.
- Check generated imports automatically. Add a lightweight CI check that resolves new imports against approved registries and internal package sources. Flag unresolved names, unexpected registries, typos that resemble trusted packages, and packages outside an allowlist when one is appropriate. Do not rely on a registry lookup alone: a package can exist and still be malicious, abandoned, or unnecessary.
- Review security logic against current sources. For authentication, cryptography, and cloud policy, compare the generated proposal with current vendor documentation, standards, and organizational rules. Ask what threat the control addresses, what assumptions it makes, and what happens during recovery or failure. If the answer is merely that the code compiles, the review is not done. NIST’s Secure Software Development Framework offers practices that teams can use to organize secure development and review.
- Test behavior, not just syntax. Use unit and integration tests, static analysis, dependency scanning, and targeted security tests. For an authentication change, test the intended success path, a failed login, account recovery, session expiry, and relevant abuse cases. For cloud policy, inspect the effective permissions rather than only the policy text. Tools find certain classes of errors; people still need to decide whether the system is safe for its purpose.
- Add a stronger gate for critical paths. Require a named reviewer for changes involving identity, secrets, encryption, network access, or production permissions. Keep the review focused: what changed, what evidence supports it, and what rollback plan exists? A second pair of eyes is not a cure-all, but it helps catch assumptions that an author has stopped noticing.
- Measure what your own tools get wrong. Track unresolved package suggestions, rejected dependencies, security-review findings, and changes revised after verification. If you benchmark a model, record the model and version, prompt, language, and evaluation method so the result can be repeated. Avoid treating a single percentage as a permanent truth. NIST’s Generative AI Profile gives teams a broader risk-management reference; use it to inform local checks, not as a substitute for them.
Retrieval-augmented generation can also help. A tool that consults approved documentation or an internal package catalogue may reduce guesses about APIs and dependencies. But retrieved text can be stale, incomplete, or misunderstood. Grounding an answer in documentation improves the evidence available to the model; it does not turn the model into a verifier. Confirm important claims against the actual registry, version, and code behavior.
Comparison: Slopsquatting vs. Typosquatting
Slopsquatting is a supply-chain attack that takes advantage of package names invented by AI tools. An attacker can register a name a model tends to recommend, then wait for a developer to install it. Typosquatting is older: the attacker registers a name that resembles a popular package and hopes someone mistypes or overlooks the difference. The two tactics have different triggers, but both exploit trust in package names.
| Feature | Slopsquatting | Typosquatting |
|---|
| How the name appears | A model generates a plausible but nonexistent package name. | A user mistypes a real name or misses a look-alike. |
| Attacker’s first move | Register a name that appears in generated suggestions. | Register a name similar to a well-known package. |
| Typical trigger | Developer accepts an AI-generated import or install command without checking it. | Developer or automation requests the wrong package name. |
| Why it can be missed | The invented name may fit the language and sound like a real library. | A one-character difference can be hard to notice in a long command or dependency list. |
| Useful defenses | Verify existence and identity, restrict registries, review lockfiles, and monitor new dependencies. | Check exact spelling and publisher, use trusted sources, and review dependency changes. |
The difference is not that one is clever and the other is crude. Both aim to get untrusted code into a software project. The package-hallucination study linked earlier found repeated invented names across its tested model outputs, which matters because a name repeated by a model may be more attractive to an attacker than a one-off oddity. That finding does not mean every repeated suggestion will be attacked; it does mean teams should not assume fabricated names are random noise.
Neither tactic is stopped by a build that passes. A malicious package can install cleanly, and a typo can resolve to a real package with harmful code. The practical defense is to verify the package’s identity and purpose, then limit what it can do in a build environment. Run builds with only the permissions they need, keep secrets out of untrusted build steps, and investigate unexpected package changes. Boring? Yes. Optional? Not if the pipeline can reach production.
Common Mistakes to Avoid
The first mistake is trusting confident wording. An assistant can provide a neat explanation for an API that is wrong or a package that does not exist. I have seen reviewers spend more time judging how polished an answer sounded than checking the source it cited. Reverse that order: verify the claim first, then consider the explanation.
The second mistake is assuming a successful build proves security. Compilation can show that code fits certain type and syntax rules. It cannot show that a dependency is trustworthy, a token is stored safely, or a cloud role is limited to the right actions. Tests matter, but the tests must cover the behavior and risks you care about. A green check is one signal, not a blessing.
A third mistake is turning off the assistant and calling the problem solved. That may remove one source of bad suggestions, but it does not remove typos, outdated examples, copied insecure code, or ordinary human error. Most guides emphasize the tool. In practice, the weak point is often the process around it: no package review, no owner for authentication changes, and no useful test for an access policy.
Another common error is relying on a package allowlist without maintaining it. An allowlist can reduce risk, especially in high-control environments, but it can become a stale permission slip if nobody checks who owns a package or whether it is still needed. Combine it with dependency updates, ownership review, vulnerability monitoring, and a process for removing unused libraries. The goal is not to collect approved names forever; it is to keep the dependency tree understandable.
Teams also overestimate retrieval tools and automated scanners. Retrieval can surface relevant documentation, but it may not contain a breaking change or the right version. A scanner can flag known issues, but it cannot judge every design decision or catch an unknown flaw. I use automation to narrow the search, not to end it. Keep the source, version, and review decision visible.
Finally, avoid measuring only how often people use AI. Usage says little about quality. Track concrete outcomes: nonexistent dependency suggestions, rejected packages, security changes caught in review, time to remediate, and whether the same defect returns. Be careful with employee monitoring; collect only what the team needs, protect prompt and code data, and use metrics to improve the process rather than shame an individual. Responsible measurement respects both system security and the people doing the work.
Frequently Asked Questions
What is an AI hallucination in coding?
It is a generated code suggestion or explanation that sounds plausible but contains invented or incorrect details. Examples include a nonexistent dependency, a made-up API method, or outdated advice about authentication. The model predicts likely text; unless it is connected to a suitable verification tool, it does not automatically check a registry or prove that a security pattern is current. Treat the output as a proposal and verify claims that can affect the build or users.
What is slopsquatting?
Slopsquatting is a supply-chain attack based on package names invented by generative AI tools. An attacker registers a fabricated name that developers may receive from a coding assistant, then waits for someone to install it. It resembles typosquatting, but the target name comes from a model’s plausible guess rather than a user’s typing mistake. Checking the exact package, registry, publisher, and purpose helps prevent accidental installation.
How can I tell whether an AI-suggested package is real and safe?
First, confirm that the exact name exists in the expected registry. Then check its publisher or maintainer, repository, documentation, release history, and whether it actually solves the problem. Review its direct and transitive dependencies, license, known vulnerabilities, and requested permissions where relevant. Existence is only the first check: a real package can still be compromised, poorly maintained, or unnecessary. Pin and review dependency changes before merging.
Should developers stop using AI coding assistants?
Not necessarily. Assistants can help draft tests, explain unfamiliar code, and speed up routine work, but they can also produce convincing mistakes. Use them within clear boundaries, keep sensitive information out of unapproved tools, and verify dependencies and security-sensitive changes. For high-impact paths, require review by someone who understands the system. The sensible standard is neither blind trust nor blanket fear; it is useful assistance with evidence and accountability.
Final Thought
AI-generated code is not special enough to deserve a different standard of care. It is still code written into a system that other people depend on, so the usual questions remain: Who owns this dependency? What does this permission allow? Is this authentication choice current? What evidence says the code behaves as intended? A confident paragraph from a model cannot answer those questions on its own.
I like tools that save time. I also know how quickly a small shortcut becomes a shared assumption, especially when the output looks polished and the deadline is close. Keep the assistant in the workflow, but put a person and a check between its suggestion and a production decision. Verify the package. Read the current guidance. Test the failure cases. Record why the team accepted the change.
That is not needless caution. It is good engineering and decent stewardship of the systems people trust with their work, money, and private information. The best guardrail is not a warning banner that everyone learns to ignore. It is a pipeline that makes the safe choice easier, catches the common errors early, and leaves room for human judgment where the consequences are real.