AI Coding Hallucinations: How to Stop Plausible Code From Becoming a Security Risk
AI coding hallucinations can turn guesses into security bugs. A coding assistant may invent a package, recommend an unsafe login pattern, or produce cloud permissions no one asked for, all in clean-looking code that passes a quick glance. The speed is useful. The confidence is not proof.
I have worked on software teams where a small library choice could affect many services, so I know why developers want a quick answer. But autocomplete does not carry the cost of a compromised build or a weak login flow; the people who use and maintain the software do. That is why speed needs a firm check beside it.
This article explains how coding hallucinations happen, why package suggestions can create supply-chain risk, and how teams can reduce the chance that a plausible answer becomes a production problem. The goal is not to ban useful assistants. It is to keep their output inside sensible boundaries.
What Is an AI Hallucination in Coding?
An AI hallucination is an answer that appears confident and useful but contains made-up or incorrect information. In code, that can be a fictional package, an invented function, an outdated security rule, or a configuration that grants too much access. The code may still look tidy. That is the catch.
Most language models generate text by predicting likely next pieces of text from patterns learned during training. They do not automatically check whether a package exists, whether a method is in the latest version of a library, or whether a security recommendation still meets current guidance. When the model lacks a reliable pattern, it may fill the gap with something that sounds right.
Code can hide the mistake better than ordinary prose. Package names follow familiar naming rules. Function arguments often look much like those in real software libraries. And a generated example can be syntactically valid, which makes it feel more trustworthy than it deserves. Does it compile? That only answers one small question.
I learned to separate a polished answer from a verified one while reviewing code in large engineering organizations. A change can pass its tests and still use a dependency that nobody has vetted, or handle a security edge case in the wrong way. Tests confirm the behavior they cover; they do not certify every claim behind the code.
A particularly clear example is a package hallucination: a model suggests a dependency that is not published in the registry it names. Researchers testing 16 code-generating models found that roughly one in five package recommendations in their test set pointed to packages that did not exist, with more than 205,000 unique fabricated names across the study. That figure describes those tests, not every assistant or every request. Still, it is enough to make package verification a normal step. The research on package hallucinations by code-generating models explains the scale and limits of the finding.
A missing package is not automatically an attack. The risk appears when someone registers a commonly suggested name first, publishes malicious code, and a developer installs it believing the assistant found a real dependency. This is often called slopsquatting, a package-name attack that takes advantage of AI-generated guesses. It resembles typosquatting, but the trigger is a fabricated name rather than a human typo.
Why AI Coding Hallucinations Matter
They can create real exposure. A suggestion in an editor may lead to code that runs with access to developer machines, cloud credentials, build tokens, or customer data. The important question is not whether the answer sounds smart. It is what the code can do once it runs.
Here are the main risks:
- Supply-chain exposure: An invented dependency can be registered by someone else. If a developer installs it, the package may run scripts or code in a build environment. The package name alone tells you nothing about its safety.
- Weak authentication: A model may reuse familiar patterns without weighing current risks. SMS codes, for example, can be useful in some situations, but high-value accounts should consider phishing-resistant options such as passkeys based on WebAuthn and FIDO standards. A fallback method still needs careful design.
- Cryptography mistakes: A generated answer may use an old algorithm, weak parameters, or a mode that is no longer appropriate. Cryptography is a poor place to trust a convincing paragraph without checking a trusted standard.
- Excessive cloud access: A generated IAM policy may grant broad permissions because that is an easy way to make an example work. A working demo is not a good reason to give a service access to resources it never needs.
- False confidence in review: Clean formatting and a passing test can make incorrect code look finished. Reviewers still need to check the source of a dependency and the reason for each security decision.
I once reviewed a change where the code looked ordinary, the build passed, and the new library seemed to solve the task neatly. The useful review question was not whether the import looked normal; it was who published the package, what the build scripts did, and whether the project had a real reason to add it. That check changed the decision. Frankly, a few minutes of verification beat a long incident review.
The risk also grows with volume. An assistant can produce more suggestions in an hour than a reviewer can examine line by line. That does not mean every line is dangerous, but it does mean good intentions are not a control. Teams need checks that run every time, even when a developer is tired or a deadline is close.
There is a human reason to take this seriously. Software supports work, services, and personal information. Engineers are stewards of systems that other people depend on, so secure defaults are part of respecting the people on the other side of the screen. This is practical responsibility, not ceremony.
A Step-by-Step Guide to Safer AI-Assisted Coding
The safest workflow is simple to describe: limit what the assistant can change, verify what it suggests, and automate checks before code reaches production. No single control catches every mistake. A few well-placed checks make a much stronger barrier.
- Set clear rules before using the assistant. Decide which work may use generated code and which work requires extra review. Authentication, authorization, cryptography, payment handling, and infrastructure permissions deserve stronger checks than a small formatting helper. Write the rule down so it does not depend on who happens to be on call.
- Ask for sources, then verify them yourself. Request links to the official documentation and the relevant version. Treat a citation from a model as a lead, not as proof; models can invent citations too. Open the vendor or standards documentation and confirm that the suggested function, option, and behavior exist.
- Check every proposed dependency. Search the intended registry, such as npm or PyPI, and confirm the exact package name. Then check the publisher, repository, release history, license, and signs of maintenance. A package existing is only the first check. It does not mean the package is trustworthy or fit for your project.
- Use an approved dependency path. Where practical, use an allowlist, an internal package proxy, or a reviewed set of dependencies. Keep versions pinned and commit lockfiles so a later install does not quietly resolve to a different package. Review lockfile changes as code changes, not as harmless generated noise.
- Keep build privileges small. Do not hand every build job broad cloud permissions or long-lived credentials. Separate test and release roles, restrict tokens, and run untrusted build steps in a controlled environment. If a dependency behaves badly, limited access can reduce the damage.
- Review security-sensitive code against standards. For login flows, compare the design with current identity guidance and consider phishing-resistant passkeys for accounts that need stronger protection. For cryptography, use maintained libraries and documented recommendations rather than asking a model to invent a scheme. For cloud access, grant only the actions and resources the service needs.
- Add automatic checks to CI. Use dependency scanning, static analysis, secret scanning, tests, and policy checks where they fit. Verify imports against a known registry and fail builds when a dependency breaks policy. Automated checks are not magic; they are a reliable second set of eyes that does not get distracted.
- Review the change in context. Ask what the generated code does, why each dependency is needed, what data it touches, and what happens when inputs are invalid. A reviewer should be able to explain the security decision without relying on the assistant's explanation. If the code cannot be explained, it is not ready.
- Measure and improve. Track how often dependency checks find unknown packages, how often reviewers reject generated security advice, and which control catches the problem. Sample accepted suggestions too; otherwise the team only measures failures it already noticed. Use the results to update rules and training.
Retrieval-augmented generation, or RAG, can help by grounding an assistant in documentation a team controls. But it is not a cure. The material can be old, incomplete, or misread, and a model can still make a leap beyond the evidence. Keep human review and automated checks in place.
For a broader process, the NIST Secure Software Development Framework describes practices for building and maintaining safer software. The OWASP Top 10 for Large Language Model Applications also gives teams a useful map of risks associated with generative AI. Neither document replaces project-specific controls, but both offer sound places to start.
AI Coding Assistants vs. Traditional Documentation Search
The closest alternative to an AI coding assistant is not always another assistant. Many teams rely on official documentation, code search, and standard IDE completion. These approaches differ in speed and in how clearly they show the source of an answer.
| Area | AI coding assistant | Documentation search and IDE completion |
|---|
| Speed | Produces a tailored draft quickly | Requires the developer to search and assemble the answer |
| Source visibility | May not show where a claim came from | Usually points to a document, source file, or API reference |
| New code | Can draft a whole function or test | Usually helps find examples or complete known patterns |
| Hallucination risk | Can invent packages, APIs, or explanations | Search results can mislead, but the cited material is inspectable |
| Best use | Low-risk drafts, repetitive code, and exploration with review | Verifying exact behavior, versions, and security guidance |
| Required check | Confirm claims, dependencies, and behavior | Confirm the document version and apply it correctly |
Neither approach is automatically safe. Official documentation can be misunderstood, and search can surface old examples. But source visibility matters: a developer can inspect where a recommendation came from and whether it applies to the version in use. The assistant wins on speed; the careful workflow wins on evidence. Use both when it helps, and keep the evidence close to the code.
Common Mistakes to Avoid
Good controls fail when teams treat a familiar phrase as a complete security plan. Here are mistakes I have seen teams make, including teams that knew the risks but were moving too fast.
- Trusting code because it compiles. Compilation proves that the compiler accepted the code. It does not prove the API is secure, the package is maintained, or the permissions are narrow. Run the checks that match the risk.
- Assuming a package name is safe because it exists. An attacker can register a name that a model repeatedly invents. Check ownership, project history, source, and integrity—not just the registry listing.
- Installing first and investigating later. Package installation can run scripts or fetch code. Check the package before adding it to the project, and avoid running unknown install steps with sensitive credentials available.
- Using a generated answer as a security review. A model can explain its own mistake persuasively. Ask a qualified reviewer to assess important code and compare its design with trusted documentation.
- Treating passkeys as a magic switch. Passkeys can resist phishing, but account recovery, device loss, enrollment, and fallback methods still matter. A strong primary method can be weakened by an unsafe recovery path.
- Assuming RAG prevents hallucinations. Grounded answers can still be wrong, and the retrieved material may be stale. Confirm dates, versions, and exact claims.
- Giving the assistant or build job broad access. Least privilege is useful even when no model is involved. Limit tokens and cloud roles so an error has less room to cause harm.
- Measuring only dramatic failures. A package that does not exist may be caught before release, but that still reveals a weak point in the workflow. Track near misses and update controls before a more serious case arrives.
Most guides say to keep a human in the loop. That is reasonable, but it is not enough when the human receives a flood of suggestions and has no time to inspect each one. Put routine checks in the pipeline, then reserve expert attention for decisions that need judgment. That is the part many teams skip.
Frequently Asked Questions
What is a hallucination in AI-generated code?
It is code or an explanation that looks credible but contains made-up or incorrect details. Examples include a package that does not exist, a function with a fictional name, an unsafe login pattern, or an overly broad cloud policy. The code may still look clean or even pass limited tests, so verify claims against documentation and run the right checks.
What is slopsquatting?
Slopsquatting is a supply-chain attack based on package names that AI tools may invent. An attacker registers one of those names in a package registry, hoping a developer will install it from an assistant's suggestion. The defense is to verify the package name, publisher, source, and project history before installation, then use dependency controls and restricted build permissions.
Can AI coding assistants be used safely?
Yes, if teams treat their output as a draft rather than a trusted authority. Use assistants for tasks where a fast draft helps, check dependencies and sources, review security-sensitive changes, and run automated scans and tests. Keep credentials and build permissions limited. The level of review should rise with the possible harm.
Do passkeys solve AI-generated authentication mistakes?
No single login method solves every design problem. Passkeys based on WebAuthn can provide phishing-resistant authentication, making them a strong option for important accounts, but teams still need safe enrollment, recovery, and fallback plans. Check current guidance, test the whole account flow, and avoid copying an authentication snippet without understanding it. The FIDO Alliance overview of passkeys explains how they work.
Final Thought
AI coding assistants can save time, especially when a developer needs a first draft or a quick explanation. But speed should not turn a suggestion into an unchecked dependency or an unreviewed security decision. The simple rule is to verify the parts that can cause real harm, then automate whatever can be checked reliably.
In my experience, the strongest teams do not rely on one brilliant reviewer catching every mistake. They make safe choices easier through approved dependencies, small permissions, clear review rules, and CI checks that run on every change. They still use judgment, because tools cannot understand every effect on a customer or a service. But they do not make a tired person carry the whole burden.
Software is work done for other people. That makes careful design a matter of stewardship as well as engineering. Keep the useful speed, insist on evidence, and do not let confident autocomplete make the final call.
Sources