Vibes & Benchmarks Ep 08: How do you learn engineering judgment when AI writes the code?

Transcript of the final episode. Speaker attribution is best-effort from a label-less recording.

Cold open

Josh (0:02) The labs are trying to make this story of, oh, we're gonna have recursive self-improvement, right? If you have the same rate of improvement in between now and then, that's not recursive self-improvement. It's not worth the valuations that people are going for if that's all you get. And that's still hard.

TOPIC 1 — Code is solved, software engineering isn't

Ali Rohde (0:18) Josh, you ready?

Josh (0:18) I am.

Ali Rohde (0:20) Someone said to me, "Code is solved, software engineering is not." Do you agree?

Josh (0:26) Yeah, that seems like a good way of putting it.

Ali Rohde (0:29) What does that even mean to you?

Josh (0:32) I think a good example of this is me learning Rust. I don't need to learn every single last bit of the syntax of Rust. That doesn't really matter. The macro syntax and blah blah blah — whatever. Claude will get all of that right anyway. And if it doesn't, the compiler will catch it and then Claude will fix it. But what does matter is the higher level ideas that are part of Rust, like the borrow checker and these sorts of ideas about how you allocate memory and when you deallocate it, and why you set up the program in a certain way, and what are the things you're trying to do with your program and how it's different, and what types of programs you should use Rust for, Python for. Those kinds of things still matter a lot. But the actual code is easy to just tell any language model these days to go write it.

Ali Rohde (1:20) I totally agree, though I think that has some downstream impacts. Before, to become a good software engineer you needed to write and learn a lot of code. Now you don't. But how do you — if you're not having to learn Rust yourself — how do you actually become a good software engineer?

TOPIC 2 — How you learn engineering judgment now

Josh (1:36) This is something that I struggled with and makes me worried for the future — the educational aspects of this. I don't know how I would have gotten here without having written a lot of code, and without the intuitive understanding that came from those years of doing that. Could I have gotten that faster than those years? For sure. But could I have skipped it entirely? I don't think so.

But now it enables me to learn Rust much more and use it productively much more quickly than I would otherwise, because otherwise I'd have to learn all these extra things and remember a bunch of extra kind of pointless knowledge. So all the pointless knowledge I get to skip and I get to focus on: okay, how is my Rust program architecturally different?

And this is the thing I've been thinking about for Rust — I'm not just going to start having Claude write in Rust and not review the PRs. I'm going to write a bunch of smaller programs first to understand the various trade-offs and kind of get a feel for it before I try and build something bigger. So I'll still have to do some learning, but it'll be a different level and a different kind of learning.

Ali Rohde (2:38) You employ interns and more junior employees. You, I think, are in a great position because you know all of these techniques, the higher level principles, because you went the regular traditional way. How do you help teach them?

TOPIC 3 — A code-review lesson: don't reinvent cron

Josh (2:46) I was doing this literally last night for some code review. I stopped at one of our interns' desks and I was like, "Okay, here's all your PRs. Let's just go through them and I'll tell you all the things that are wrong with them." Which he was very happy about, because we have good interns. And then at the end I was like, and here are the higher level principles that you can back out.

For example, one of the things was adding a scheduled task to one of our agents. And he wrote this little Python library that did the task scheduling part. I was like, no, no, no. Don't write your own version of a thing that already exists. Either use cron or anacron. "Oh, we could use fron." No, no, it has too few GitHub stars. Don't use that. It's unmaintained. Don't use that. Just use one of these old crappy things.

But he's like, "Oh, but it doesn't have the feature — you can't make it run every day and at a time." I was like, that's a feature, not a bug. The reason that these programs work this way is because they're about simplicity and robustness, and it's actually difficult because you get yourself into these twisty situations if you try and do the two at the same time. So you actually want to be picking these older, more robust, stable tools that you build on, and not writing software as much as you can. Your job as a software engineer is to write as little software as possible, and to use the things that are as canonical as possible, to stay in the happy simple path. That's where you want to be.

So I can deliver this lesson a little more easily and a little more directly, and he gets to learn this because he can ship things more quickly, so he gets to learn those things more quickly than in the past. But it does require someone still paying attention and wanting to go through the code review and talk about this and think about it and learn it.

TOPIC 4 — When you don't have a mentor

Ali Rohde (4:18) Yeah. What if you don't have a Josh?

Josh (4:20) Then it's harder. [laughter] I mean, you can use Claude for this. You can use a mentor or another software engineer. There's still value to other people that have other perspectives and experience.

Ali Rohde (4:32) You said you can use Claude for this. Right now I don't think it's really good at this taste stuff, even when it comes to software engineering. Do you see it getting better at this over time, and therefore being able to abstract away more and more of this over time?

Josh (4:46) It's a tool and you can use it for this even though it's not perfect at it today. So for example, for Rust — I know I've built up all these things that I know in Python, like oh, you should avoid this library, and you should do things in this way, and you should check for this kind of error. And so I gave a long list of that type of thing to Claude and I was like, okay, now tell me the same kind of things but for Rust. What are the things that I should or shouldn't do? What is this style of thing?

And so then it gave me a long list, and then for each one of those I can dig into it and think about it and look it up and ask, what's your source for that? Or play around with it. So it requires a little more engagement. It won't just tell me all the right answers. It'll tell me 66% of the right answers, and then I'll have to remove some of the false ones and I'll have to come up with the ones that it's missing. But it's still a tool that gets me a lot closer than just the years of grinding that would take otherwise.

TOPIC 5 — Paul Graham's five-year bet and the valuation question

Ali Rohde (5:38) 100%, and you still need the taste to vet all the candidates it surfaces. Paul Graham posted, "Imagine what it will be like five years from now if models have improved on Fable as much as Fable has improved on GPT-3." Do you expect to see that level of progress sustained, decreased, or accelerated?

Josh (6:02) I saw that quote because somebody retweeted it as, if that happens the labs are screwed. As in, the difference between Fable and GPT-3 is pretty big, but if that's all we get in the next five years, that's actually not that huge. GPT-3 was already pretty useful. You could talk to it, right? As ChatGPT it would answer any question about normal factual things. It wasn't perfect, but it was pretty good. And Fable's great, but the step isn't quite big. Five years is a long time. If that's all we get out of five years, is that enough? And that's a huge gap.

But the labs are trying to make this story of, oh, we're going to have recursive self-improvement, right? If you have the same rate of improvement in between now and then, that's not recursive self-improvement, right? I don't think that it's worth the valuations that people are going for if that's all you get. And that's still hard. It's still hard to go that far. I think we do get roughly that. I think that is roughly the rate, as opposed to an accelerated rate.

Ali Rohde (6:58) I think hitting that is possible.

Josh (7:00) I think hitting that is likely. I think we slightly exceed it. I don't think we get 2x that gap. My guess is we get a little bit higher, but not dramatically higher.

Ali Rohde (7:11) But you're saying that is actually less than what the frontier labs' forecasts kind of depend on.

Josh (7:17) Yeah. That's not recursive self-improvement, because the rate of change is staying roughly the same.

Ali Rohde (7:21) And that's basically what the whole economy is dependent on too.

Josh (7:26) Yep.

Ali Rohde (7:27) Does the economy feel a little bit precarious to you at times?

Josh (7:30) Oh yeah.

TOPIC 6 — J-space and Anthropic's interpretability work

Ali Rohde (7:32) We have a new term this past week — J-space, i.e. Anthropic's name for the internal part of their model that seems to be doing something like actual thinking. Is this doing actual thinking?

Josh (7:46) I'm not sure what actual thinking would be. You should ask Gary Marcus, I guess, where he wants to put that goalpost right now. [laughter]

But I do think that this was already part of the earlier Anthropic work, even from probably two or three years ago — their circuits work about the internal representations of the model, right? They even had a thing in the past where they could sort of force it to think about the Golden Gate Bridge and then everything it said would come back to the Golden Gate Bridge, by changing this activation vector. So I think this is just a continuation of that work.

It is interesting that it's this kind of interpretable high-level space that seems to correspond to some interesting concepts, or ways that we would think about our own thinking or about how the model thinks. I don't know that it's that surprising or that unexpected. If you were going to have a model that was able to do the kinds of things that it does, it's not super surprising that you'd have these kinds of representations somewhere in there, because otherwise how would it work? It kind of has to have this information.

One of the examples they gave was, when it's reading through a jailbreak prompt from a user, it kind of has this activation of jailbreak or danger or warning or whatever. That kind of has to be there in a sense, because if it wasn't, it couldn't know that it was a jailbreak. So it's cool that they've found it and they found a way to isolate it and technically talk about it. I think that's interesting.

What they're trying to use this for is more about consciousness and the models having internal experiences and things like that, and that part I'm a lot less sold on. But yeah, I think it seems fine for you to think about this from a technical perspective. There sort of has to be that kind of information somewhere.

TOPIC 7 — Consciousness, IPOs, and motivated reasoning

Ali Rohde (9:30) Do you think Anthropic wants people to believe that its models have consciousness?

Josh (9:38) Yes. I think they also want people to believe that their models are super powerful and too powerful for people to access, and all sorts of other things. I think it kind of validates their perspective or worldview in a way. And so it's a little bit motivated reasoning. It feels like — yes, these things could someday be conscious, but that's definitely not where we are today. So it's interesting in a theoretical sense, but this probably should be in a very niche journal and not on your public company who's about to IPO blog post.

Wrap-up

Ali Rohde (10:11) Josh, we made it.

Josh (10:13) Indeed. Thanks, Ali.

Ali Rohde (10:15) See you next week.

Josh (10:16) See you.