Vulnerability-as-a-Service

I will always talk (and maybe over-talk) about how important it is to make sure one’s build systems are in place, regardless of their language choice.

As we’re yolo’ing/vibing more often with our AI assistants, we’re seeing more and more code deployed using old dependencies. This isn’t always obvious because the thing you’re building probably works fine, but under the hood it’s a complete festering mess of vulnerabilities waiting to be taken advantage of.

So why does this happen?

Well we just have to look at how these models are trained. Remember, all training is point-in-time, and this training costs a lot of money, so it’s not always feasible to have the latest and greatest info.

Another part is how LLMs work (I’m not going to explain this, you can go find out yourself) but it’s all based on probability, and generally LLMs will take the laziest path - me too, so I don’t blame them.

So it’s inevitable: if I know how to write what I’m gonna write, then I’ll write it, and just assume all the libraries and code I’m using is just fine.

What can we do?

There’s many things you can do, and really it depends what the state of your project is. Many languages that are being yolo’d/vibe’d will have some sort of package managers, and there’ll be built in tools to allow you to see just how out of date your dependencies are.

The real problem happens if you already have your thing and then you’re dealing with major upgrades of dependencies that introduce breaking changes.

Welcome to the jungle my friend.

What should we do?

Well from the get go we should make sure we’re giving our assistants as much info about the specific requirements not to use old dependencies. Generally I tend to follow this approach:

1. I ensure that the AI never edits the dependencies files (package.json, pyproject.toml, etc…)
2. I especially make sure they never touch the lock files (package-lock.json, uv.lock, etc…)
3. Pretty much all of your AI assistants’ coding tools can call CLIs so make them do so; get them to use the package managers (such as npm, uv, etc…) to decide which dependencies to install, read notices, and manage updates

I have no idea what any of this means

Ok, now you understand the title of this post. I’m not here to critise your work, who knows, I’m probably wrong, but I always try to be less wrong than the team/person who deployed a system and got hacked.

It can be as simple as telling your AI assistant:

Please make sure you’re using the latest stable dependencies, make sure you check that using the CLIs for those dependencies and not editing the package files yourself

Accidents are no mistakes right, and it’s very straightforward to spell it out in simple language that you’re wanting to build something that’s secure and meeting your requirements

What about these build systems you mentioned?

Yeah, this is important, regardless of language, you defiantly want to make sure you’re versioning your code (using something like git, svn is probably still available), a lot of the hosted solution provide you with dependency bots, scanners, etc… to tell you when your stuff’s out of date and causing potential security issues.

Bonus points for then implementing CICD using these systems; I’ll leave my opinions on that for another day, maybe.