The first time I ever experienced a security incident was probably 2004. This is, I mean, outside of the 'my friends on irc showed me what bind was and why the "install everything" option of redhat was a bad idea at the same time' story from when I was a kid. Greymatter is what there was before wordpress was a thing. It was another php doodad people used to make blogposts. At the time, there was a popular thing called a 'tagboard' - basically a live, public chat similar to a guestbook - but think of baby irc in a tiny two inch wide, three inch tall panel under the navbar on the left of a site. This tagboard was ancillary to greymatter - it was written in perl and hacked into the theme by hand. Super simple, just wrote to a text file, and showed the last few lines of text when viewed. I know what you're thinking but none of that was a reality in 2004. Security wasn't a thing people really thought about. If the thing worked, it went online. There was no such thing as 'dev' or 'qa'. Up was up.
Well what you would expect happened - someone figured out how to RCE the thing. It wasn't hard. Zero input sanitization. When I found it, it was being used as an irc server for a botnet c2. Backed up some simple files, full format. Nuke it from orbit, it's the only way. Since then that post exploitation action-item-list has been very thorougly turned over, ad nauseum. Every security shop on the planet will have some kind of plan or documented methodology for how to deal with an attacker who has a shell on a box. It's been the tip of the spear for something like 25 years now. There are songs about popping shells.
The thing about LLMs you have to remember is that they just squirt text. That's it. The LLM itself doesn't even live where the text is actually being squirted either. It's some script or binary or 'harness' calling an api, retrieving the text, then deciding what to do with it.
Not unlike, say, sshd - or some other kinda malicious payload on a box. It's just something squirting text into a system.
And considering this is one of the oldest standing things that has existed in security - an attacker who is simply squirting text into a box - peoples methodology for handling that scenario should tell you about their acumen. People who have been building systems for a while learned these lessons a long time ago. Problem is: those people seem to be most of the folks who are looking for work right now. That should also telegraph stuff. To me it says 'find high ground'.
The next thing to consider is that "there's only one door". Meaning the LLM has to 'get inside' wherever it is to be squirting text in the first place. If it loses connectivity, game over. The harness has to be able to talk to the model. That's just an api call over the internet, or maybe somewhere on the LAN if you're wealthy enough, but there's that plumbing to keep in mind.
From the defenders perspective, this is a very, very, old, well-worn path. I mean, assuming the defenders have .. done this before. If this concept is somehow new, then.. well. Find high ground.