How to Test AI-Generated Code for Security Vulnerabilities

The assumption that AI-generated code is inherently safer than human-written code does not hold up under scrutiny. Research conducted across multiple coding assistant tools and multiple languages has found that AI-generated code exhibits elevated rates of particular vulnerability classes, especially when the prompt does not explicitly specify security requirements. The assistant is optimising for correctness and functionality; security is not the primary objective unless the prompt makes it one.

The practical implication is that teams adopting AI-assisted development need a testing approach that accounts for this. Existing security testing processes designed for human-authored code are a starting point, but they need to be extended in specific ways to address the failure modes that are characteristic of AI-generated code. This is not about distrusting the tool; it is about understanding what it does and does not do by default.

SAST tools catch the easy cases but miss the harder ones

Static application security testing tools are a sensible first line of defence for AI-generated code, and they catch meaningful issues. Injection vulnerabilities, hardcoded credentials, use of deprecated cryptographic functions, and some memory safety issues are within the detection capability of mature SAST tools. Running SAST on AI-generated code before it enters the main branch is a reasonable baseline control and is easier to enforce than manual review at scale.

The limitation of SAST for AI-generated code is the same as its limitation for human-written code: it catches patterns but misses logic. If the AI has generated code that correctly implements an authentication function from a syntactic perspective but makes a logical error in the condition that determines whether a session is valid, SAST is unlikely to catch it. Logic vulnerabilities, race conditions, and insecure design patterns that are implemented correctly at the code level but incorrectly at the design level require different testing approaches.

Manual security review of AI-generated code needs to be targeted

Full manual review of every line of AI-generated code is not practical at scale, and it is not necessary. What is practical is targeted manual review focused on the vulnerability classes that AI-generated code is most likely to exhibit and the functions that carry the highest security consequence if they fail. This means security reviewers need to know where AI-generated code is in the codebase, which is an argument for committing to a policy of flagging AI-generated contributions in pull requests or commit messages.

The functions that warrant mandatory human security review regardless of their origin include authentication and session management, authorisation and access control decisions, cryptographic operations, handling of externally supplied data, and any code that makes system calls or interacts with privileged resources. AI-generated code in these categories should receive the same level of scrutiny as code submitted by a junior developer who does not yet have an established track record in secure coding.

Prompt engineering affects security outcomes and should be part of the testing picture

The security of AI-generated code is partly a function of how the developer prompts the tool. A prompt that asks for a function to validate user input and explicitly specifies that the function must handle SQL injection, XSS, and path traversal will generally produce more secure output than one that simply asks for an input validation function. This is worth knowing because it means developer training on prompt construction has a measurable effect on code security outcomes.

It also means that security review of AI-assisted development processes should include assessment of the prompting practices that developers are using. If review of production code reveals that a particular class of vulnerability is recurrently present in AI-generated contributions, the fix may not be only a testing control but also a change to how developers are formulating their prompts. Security champions within development teams can play a useful role here by developing prompt templates for common security-sensitive functions that incorporate explicit security requirements.

Dynamic testing and penetration testing remain essential

Security testing of AI-generated code needs to extend beyond static analysis to include dynamic testing and, for significant codebases, penetration testing. Dynamic analysis catches runtime vulnerabilities that static analysis misses, including some injection vulnerabilities, race conditions, and authentication flaws that only manifest when the application is running. Penetration testing provides an adversarial perspective that neither SAST nor DAST fully replicates.

The scope and frequency of penetration testing should account for the proportion of the codebase that is AI-generated, particularly for applications in security-sensitive categories such as financial services, healthcare, or critical infrastructure. Where AI-generated code forms a significant part of a new application or a major feature, a targeted security review of that component before deployment is a proportionate control rather than an excess one.

We conduct security reviews of codebases that include significant AI-generated components, covering static analysis, manual review of security-sensitive functions, and penetration testing where appropriate. Contact us at info@cyberlinx.com.au to discuss what a review engagement would involve for your application.