Vibes & Benchmarks Ep 09: Did an AI escape its sandbox, or was that just bad engineering?

Transcript of the final episode. Speaker attribution is best-effort from a label-less recording.

Cold open

Josh (0:01) I don't think that this should be evidence for "these AIs are so powerful they can escape if we try really hard." It's more, we were running this and we had a janky harness around it and it happened to get out of it. Not, we tried super hard to isolate it and it was able to defeat us.

Ali Rohde (0:21) Hey Josh.

Josh (0:21) Hello.

Ali Rohde (0:22) You ready?

Josh (0:22) I am.

TOPIC 1 — OpenAI's model slips its sandbox

Ali Rohde (0:23) All right, let's get into it. An OpenAI model repeatedly found ways to slip its sandbox. It actually opened up a public GitHub PR when it was just supposed to post it to Slack. Is this the future, and models just getting too good, or is this bad engineering?

Josh (0:43) It's mostly just bad engineering, which is also the future. So, a little bit of both. I don't think that this should be evidence for "oh, these AIs are so powerful they can escape if we try really hard." It's more, we were running this and we had a janky harness around it and it happened to get out of it. Not, we tried super hard to isolate it and it was able to defeat us.

Ali Rohde (1:07) It's not that models are now so powerful that they can just regularly slip out of sandboxes.

Josh (1:12) No.

Ali Rohde (1:13) But they will, because other companies will have bad engineering around them.

Josh (1:18) Yep.

Ali Rohde (1:18) So, what should those companies do?

Josh (1:21) Be better. [laughter] Be best. I mean, they should probably just actually take this stuff a little bit more seriously. There are known escapes even for Docker, for example. So you might want to think about how to harden the types of infrastructure that you're using for running your coding agents.

Ali Rohde (1:41) They won't. So do we expect to see a lot more of this?

Josh (1:44) Yeah, probably. Although other people don't have the same incentive to report it. OpenAI gets to report it as "oh my god, our AI is so powerful," whereas everyone else, when they report it, it's "we're idiots and we can't run a proper sandbox."

Ali Rohde (1:58) Is this just yet another thread in ongoing cybersecurity threats?

Josh (2:04) This is already a class of problem that matters. Even if you were trying to play around with a computer virus for research purposes to understand what it's doing, or if you're trying to isolate some untrusted code for a coding tool that you offer other people — you need to have sandboxes that people can't escape.

Ali Rohde (2:25) Right.

Josh (2:25) So that's already a really important thing that people put a lot of effort into.

Ali Rohde (2:30) Sure, but now are the risks magnified?

Josh (2:33) I'd say that it's easier for agents to break out than for random attackers in the past. So it basically matters that you do a good job here.

TOPIC 2 — Fireworks' $1.5B raise and the open-weight market

Ali Rohde (2:44) Fireworks just announced a $1.5 billion Series D at a $70.5 billion valuation. It says it now serves more than 40 trillion tokens per day. It allows people and companies to use open-source, open-weight models and then helps them customize those based on their own data. I think historically you've been a little bit skeptical of this market. What's your take today?

Josh (3:11) I think that this will be a larger market in the future. My skepticism was, are people in the past doing a lot of this? Not a huge amount. If you think about how many trillion — like, 40 trillion tokens over what time period?

Ali Rohde (3:25) Per day.

Josh (3:27) Per day. Okay. Yeah, so that's a decent number, and I bet that a lot of that has happened in the recent quarter or so as we've gotten much better open models. So I would expect that to continue as we receive even better open models.

Ali Rohde (3:41) Yeah, it feels like just the start of even more amazing open models, and just the start of companies moving into their cost optimization phase versus just trying new things phase.

Josh (3:54) Yeah, maybe we should invest.

TOPIC 3 — Does fine-tuning on proprietary data matter?

Ali Rohde (3:56) How important is the customization of companies' models on their own data?

Josh (4:05) I'm not sure customizing it on your own data makes a big difference for, say, coding agents. Maybe it helps a little bit, but it probably helps a lot if you're trying to do something like a customer service bot, or something that you don't want to sound like Claude. So I think it depends on the use case. Probably many of the people who are using it are the ones where it does matter, where they want it to sound different, or they want it to do something that's out of domain for the typical coding agents or typical models.

Ali Rohde (4:34) I see. That's interesting. For coding it's just not going to be necessary, because that's also what these models are optimized for.

Josh (4:41) Yeah, I just don't think it'll have a huge impact. If you want to get it to adapt to your house style or your giant enterprise code base, you can get a little bit from that, but I suspect a lot more of the value is coming from these other applications where there's a bigger delta.

Ali Rohde (4:56) So it helps Claude not sound like Claude. What else does customizing and fine-tuning allow these companies to do?

Josh (5:05) You might be able to get better capabilities for something like legal or medical, where there's just not nearly enough data for Claude on the web or in their training sets to get what you want. Like if you're trying to write a diagnosis for "does this patient have cancer" — Claude just hasn't seen that much of it, so it might help a lot for you to be able to train on thousands or millions of examples of that.

Ali Rohde (5:27) Could that provide a moat for these vertical software companies?

Josh (5:32) Maybe. It helps to the extent that other companies they're competing with don't have access to the same data, so it's vertical-specific. Like in health care, probably other people don't have access to your medical records, right? Epic's data is not public. But something like legal filings — most of the court cases and everything end up being public. So it kind of depends on the industry.

Ali Rohde (5:55) So maybe software isn't dead.

Josh (5:58) Well, I don't know if software is dead or not, but the models here are the things that matter, even if it's not writing software.

Ali Rohde (6:06) Right, but having models that are customized on proprietary data sets might help SaaS companies survive in the future.

Josh (6:16) Yeah, if they happen to be the shape of "we have all of this custom data and it matters for us to train a custom model on it."

Ali Rohde (6:22) Yeah. I wonder if that will empower incumbents who have kind of been left out of the AI revolution, but do have those data sets.

Josh (6:32) Yeah, that's very possible. I think it's a war between incumbents being large and slow and not particularly confident, and having all this data, which is somewhat of an advantage. So where does it net out? I don't know. [laughter]

Ali Rohde (6:46) Probably still not good for the incumbents.

Josh (6:47) Yeah, exactly.

TOPIC 4 — Kimi K3 and how close open models are

Ali Rohde (6:49) In other open source news, Kimi K3 was just launched. When you saw these models released, and Kimi K3 in particular, what were your takeaways?

Josh (7:00) They all seem pretty good. I think we're about one generation from ones where I can switch over to them.

Ali Rohde (7:08) For everything?

Josh (7:09) Well, for the default work. As I've always said, Opus 4.8 is enough. I often find myself switching back to it because it's faster. I don't need Fable for everything.

Ali Rohde (7:21) When you say one generation, what does that mean?

Josh (7:24) I mean, they claim to be Opus 4.8 level right now, but they're not quite. Once they get to Opus 4.8 level, then yeah. So whatever their incremental step is, or their next model release, it's probably close enough.

Ali Rohde (7:37) When do you think that will be — like three months, or next year?

Josh (7:41) Sometimes they do point releases where they go from 3.8 to 3.9 within a few months. So I would hope that most of these have that. Other ones, they've already tuned it as much as they can and they need to wait for the next full pre-training run, and that could take a little bit longer — maybe three to six months.

TOPIC 5 — Chips: Google's model-specific play

Ali Rohde (8:01) Let's talk about chips. There was news this week that Google is planning a model-specific server chip that could run Gemini six to ten times more efficiently than its newest TPU. We've recently talked about how Google already has a bunch of different chips, so this seems to be a third in the mix. How important does this seem to Google and Gemini's future?

Josh (8:27) It seems important for the future. Looking at the details for it, it looked a little bit more prototype, small run, proof of concept — which is similar to how they did the TPUs. I think they're just smart and good at doing this now, so they'll try a smaller run and then make them larger later, because they know what they're doing and that's the right way to make chips. So it seems pretty important, because if they can really serve it at six to ten times less power, and presumably the chips will end up being a little bit cheaper to make, a little bit easier to deploy, then I think that helps them a lot on the inference cost side of things.

TOPIC 6 — Is the moat the model or the harness?

Ali Rohde (9:02) Feels like in some ways pressure is mounting on OpenAI and Anthropic.

Josh (9:08) Indeed.

Ali Rohde (9:10) Both with each other — their cold war turned hot war — and then all of these open-source models.

Josh (9:19) Yeah.

Ali Rohde (9:20) Which is interesting because it also coincides with their IPO plans.

Josh (9:25) Yeah.

Ali Rohde (9:26) So it feels like — I think they would argue that the moat is changing. It's not the models, it's the harness. It's them specializing increasingly in the whole stack. You don't agree.

Josh (9:39) No. [laughter] I don't agree. I mean, I'm sure they will argue that as the model commoditizes. But the model was the thing that takes billions of dollars to make. The harness is something that me and a few friends can throw together in a week at a hackathon.

Ali Rohde (9:53) That was literally open-sourced.

Josh (9:55) Yeah.

Ali Rohde (9:57) Then maybe the thing that's left is distribution and enterprise relationships.

Josh (10:02) That'll be a big part of it.

Ali Rohde (10:04) What else could be left? Talent, generally.

Josh (10:09) I mean, if you've got Fable, do you really need talent? [laughter]

Ali Rohde (10:13) Uh, yes.

Josh (10:15) I'm kidding. Yeah, distribution, talent — those things will matter. But also, this doesn't mean that it's the end for AI. It just means that the winners will be the ones who can do inference most profitably, right? That's why I keep saying Google seems like, still, even despite all the stumbles, it's not in a bad spot.

TOPIC 7 — Apple as the sleeping giant of local inference

Josh (10:39) Or even Apple. I saw another good case that someone made on Substack, that Apple is the sleeping giant of AI because they have all this really good hardware for local inference. So as you get good enough models that could run on your unified memory locally, then maybe most of the inference is happening there. Maybe you're willing to spend an extra few thousand dollars for your new MacBook so that it can do most of your inference. That puts them in a good spot. So not maybe this year or next year, but thinking a little bit further into the future.

Ali Rohde (11:06) Yeah, that's interesting. And they did just have their hardware lead replace Tim Cook, so they're obviously thinking a lot about hardware. You mentioned it's important for these companies to get better and better at inference. Could OpenAI and Anthropic do very well there as well?

Josh (11:22) They could. I think that's why they're doing their custom chips.

Ali Rohde (11:26) Yes.

Josh (11:27) And so it's a race there, and it's not an easy one to win.

TOPIC 8 — AMD's Helios and Nvidia's high-water mark

Ali Rohde (11:32) AMD is also working on shipping a new chip. They're going to ship Helios, its first rack-scale AI system to rival Nvidia, later this year. I think historically you've been very bearish on AMD. Is your impression of them improving?

Josh (11:51) I haven't looked recently, but I think it was mostly from their leadership being so slow to this game. Everybody has seen this coming for so long at this point, right? Now their hardware is actually pretty good, and pretty good in terms of cost performance, but you need someone who's really taking advantage of that. Maybe they're getting a little more serious about it. Maybe they could be decent competition for Nvidia in the future. That sounds great.

Ali Rohde (12:17) Is it too late, do you think?

Josh (12:19) No, I don't think it's too late. The vast majority of these things haven't been built yet — the data centers, and the chips that will run inference on. Most of the commitments are many years out still, right? So what chips are going to go in those data centers? We'll see.

Ali Rohde (12:37) Does any of this present an actual threat to Nvidia, or are they kind of crushing it no matter what?

Josh (12:42) I think they're in a good spot. The Substack article I read was making the case that Apple would be a little bit of a threat, by having much cheaper unified memory and a much easier way to string these Macs together — they have just a lightning cable, you can just connect them to get way more memory. They took that kind of connector cable off of the most recent cards in Nvidia, so it seems like they're kind of going away from that style. But they do have their DGX-park style stuff, so it's not like they're not thinking about it. They certainly are capable of doing this. I think it's a question of timing. They could do it too late and then stumble a little bit. But it's not as profitable either, so I can see why they're doing it the way they're doing it.

Ali Rohde (13:28) Do you think the market price that Nvidia is commanding today is going to be a high-water mark for them?

Josh (13:35) Oh, [sighs] I don't know. It's very overpriced, but also there's going to be a lot of this that happens in the future and these other things don't exist yet, right? Like Apple or AMD — in theory it could compete, but I can't go buy them today.

Ali Rohde (13:49) Yeah.

Josh (13:49) And I might not even be able to buy them next year in a meaningful way.

TOPIC 9 — OpenAI hires vibe coders, and why knowledge work is unsolved

Ali Rohde (13:55) In other news this week, OpenAI is now hiring vibe coders. They just hired this Harvard MBA who created this email writing extension called Sincerely — but "sincerely" spelled wrong — that deliberately adds errors into emails so they seem less AI. I actually thought this was brilliant and an interesting take. You and I talk about knowledge work a lot, and I think you've predicted the destruction of all knowledge work jobs. I have never agreed with that. And so I was actually not surprised to see this, because I think using AI well for knowledge work is still completely not solved.

Josh (14:36) Yeah, it's unsolved, but I don't think it's unsolvable. I think it's more that the labs have focused on coding, and then — wow, look, they made coding so good that now someone who isn't even a coder can make a thing that helps defeat AI detection for emails. Okay, great.

Ali Rohde (14:54) I suspect the rate limiting factor is not the model and the model's capabilities. It's actually people. I also think that coding emerged as this amazing use case because the primitives needed to use AI in coding already existed. Right? Like, what were the most important primitives?

Josh (15:16) Everything is just accessible. All the tests and infrastructure and deployment and monitoring and —

Ali Rohde (15:22) GitHub.

Josh (15:23) — reporting. Everything exists. Yeah.

Ali Rohde (15:26) That doesn't exist for knowledge work. And so I think even if the capabilities are there, right now it still depends on — and for a long time will still depend on — people setting up these systems. And they simply won't.

Josh (15:39) That's right. But the thing that's different about it is that only one person needs to set it up. As soon as one person goes and sets up Sincerely, and it's like, oh, now here's this product other people can use — now other people can go use this to defeat the AI detection on their spam cold sales emails. So it just spreads a lot faster. It doesn't require every person to reinvent the same thing.

Ali Rohde (16:03) I don't know, because a lot of knowledge work is not organization-wide. It's very particular, very specific. So someone can't easily build for me and my use case.

Josh (16:14) I don't know. I think we've seen counterexamples of that internally, as we're using our own kind of tools for non-software engineers on our team.

Ali Rohde (16:22) Yeah, but are software engineers building those?

Josh (16:25) No. The software engineers built the original tool, but then giving that tool — it's basically Claude Code but with a lot of extra stuff.

Ali Rohde (16:33) Like what? What extra stuff is useful?

Josh (16:35) Skills for how to make these things robust, or how to deploy them, or how to do the backups, or how to do it in a secure way, or how to share with people. All that stuff can be done just once, and one of our engineers can think about it and make a nice skill, and then hand off this big collection of skills and tools and interfaces to someone else who's non-technical. And at that point Claude can just go make it happen. "Oh great, you wanted a dashboard to look at how many new users there were based on this ad campaign — come back in five minutes." "Oh yeah, works like I wanted."

Wrap-up

Ali Rohde (17:08) All right, on that note, we did it.

Josh (17:11) Thanks, Ali.

Ali Rohde (17:12) See you next week.

Josh (17:13) See you.