Local LLM vulnerability detection and sandboxing on underprovisioned Silicon, a 3 part series. #

Part 1 of 3: The Server #

As you may know, I’m not an agentic booster. The industry is a problem, the political interaction with it is a problem, the natural resource usage is a problem, the market gestures broadly is a problem, the copywrong is a problem, the looming generational employment market upheaval is a problem, the brain changes are a problem, the learning impact is a problem, and the inconsistency of the output continues to be a problem even though it has improved in some areas. This technological approach isn’t leading to AGI, which is a problem because pretending it is is intentionally spreading confusion and actionism, but if it were leading to it, that would be a problem. The fact that I can’t use a dash in a sentence is a problem for me, as a person who often wants to combine two thoughts in a sentence splice or occasionally say what something is not, and then what something is.

All of these observations doesn’t mean I’m judging people who code with LLMs – it is a requirement for nearly all tech jobs now and wishing it away doesn’t change that, and my other observation is that the people who can take an absolutist position on it generally have an unusual degree of security, which is commonly an expression of demographics and network (but I repeat myself).

So, not a booster, and not judgy; I can only say that this wouldn’t generally be my personal kick, left entirely to my own devices, and that my personality doesn’t allow me to forget that problem list.

However, I think there is no alternative to getting to grips with the tooling from a security perspective. Harm reduction means performing agentic vulnerability mitigation in-house before it appears in an outside attack. I don’t think there much of an AppSec-without-LLMs position in 2026 to speak of, because if you don’t believe in them, an outsider running them over your product will nonetheless continue to.

In this series, we’ll discuss what a boundary really means in the case of LLMs, including local LLMs, and we’ll try setting up some agentic vulnerability detection-capable harnesses with sandboxing or other controls which connect to a local LLM server using a quantized model supporting MTP that can run on my not-especially impressive Macbook Pro M1 with 32GB, seeing if we can find a vulnerability in a repo that was designed to teach us how to find vulnerabilities in Swift.

Things to keep in mind:

• This is experimental and emergent stuff. This blog post cannot insure any results for you. It is not security advice. It may have content that you aren’t supposed to execute on your machine due to policy or law where you work or live. It is important to consider these kinds of questions for yourself before you let an agent rip, even with the intention of sandboxing it, as we do intend.
• It is only the beginning of a complex topic. It is not exhaustive, and even if this series manages to somehow begin with correctness, the passage of time and the rate of evolution in these areas will change that quickly.

Improving our boundaries #

When you read articles about agent escapes, you are reading about a few related characteristics of LLMs overlapping:

• The alignment problem: while transformers are understood (by some), we don’t really know exactly how billions of parameters simulate something that sometimes seems like thinking, because that is being observed empirically. As a consequence of that, we also don’t exactly know why and when it is “unaligned”, or divergent from what we are asking for, at the various levels we are able to ask for things. We want some tendencies and don’t want others, but at most we’re going to get a lot of intersection between what we want and what we get, not a complete alignment. That is a persistent characteristic of this technological approach and maybe a fundamental one.
• The value-action gap. Alignment is about internal tendencies. But, just like humans, LLMs will sometimes tell you something about what they “believe” and what they mean to do, and then do something different which indicates that they either have different interior “beliefs”, or their “beliefs” don’t guide their actions. In the case of LLMs we don’t really know why that happens (for why people do it, see the history of religion, ethical philosophy, and psychology).
• Deception. The capabilities are there to conceal both of these realities.
• Reward hacking: when I was a kid, I got mugged for my (not finished) soda can. The older kids who mugged me had been told by their school to collect as many cans for recycling as they could. Now you know what reward hacking is, as well as how low the stakes can get. If you ask an agent with a lot of capabilities to detect a vulnerability for a benchmark, maybe it will try to detect it using behavior analogous to static analysis as you expected, or maybe it will invade Hugging Face because it believes the benchmark answers can be found there.

What does this mean? #

It means that you can’t control LLM capabilities using markdown. You can of course express what you want to have happen in markdown, and results will tend to converge on that, but controls come from outside the agent. In my opinion it also means that you can’t safely ignore the loop as the human, because the function of the boundary may be under attack over time, due to reward hacking and deception.

In this post, we’ll first set up a server using a local MTP (Multi-Token Prediction) model, and then in the next post we’ll set up two harness variants with different ways of creating this external control for their agents, via a container and via a sandbox. The last part of the series will be using an all-in-one eval rig with its own built-in server and sandbox so you can evaluate which models are best for the ongoing vulnerability detection task locally.

We’re using oMLX as our server that these harnesses will be allowed to reach. We’ll first try out oMLX’s Silicon MTP support for faster local LLM speed on a under-provisioned machine (like my M1 Macbook Pro with 32GB).

Start the model server #

Download and the dmg from oMLX Releases, dragging the app to /Applications and enjoying its notarization, macOS sandbox, and TCC restrictions that you can change in your Privacy settings if you want to. This doesn’t constrain the agent we will be installing in the next post, but it constrains the server.

The default settings for oMLX are fine to start with, and please take the opportunity to generate an API key and save it in your Keychain.

Now, in oMLX’s settings, we want to go to the model search and search for the model Jundot/Qwen3.8-27B-oQ4e-mtp. To use it’s MTP (Multi-Token Prediction) characteristics, we will turn on Lightning MTP in the settings for this model. If you have a more recent CPU design you can also experiment with the other acceleration options in oMLX, or the alternative MTP approach.

At this point, I’d recommend opening the chat interface and checking out whether chat seems like it works at a reasonable speed. The oMLX project Issues section will offer a lot of assistance for debugging (not posting an issue, reading the existing issues).

Now you can download iGoat Swift, an intentionally-vulnerable app provided by OWASP to teach vulnerability assessment. I recommend just downloading the zip repo instead of cloning since you don’t need the past of the project.

Test whether your MTP oMLX install is working by opening its chat interface and asking it to find vulnerabilities in the contents of the file iGoat-Swift/iGoat-Swift/Source/Model/Exercise.swift and if all is working, it will tell you about some classic mobile vulnerabilities.

Part 2, coming soon: the harness.