Are plan files the new 10k-line PRs?

Many people’s new way of working with LLMs is to start with an idea, have the LLM create a spec (review and approve that), then have the LLM generate a very detailed plan.

At that point, people have two options:

• They either spend a lot of time reviewing the plan, ask for changes, and finally approve it. Warning: this may take hours.
• Or they skim the plan, but it’s too long, so they YOLO it and say “LGTM 👍”.

And that made me realize that plans are the new 10k-line PRs.

Back before the days of LLMs, we used to say that the number of comments a PR received was inversely related to the number of lines it had.

• If the PR had 10 lines, it’d get 11 comments.
• If the PR had 10k lines, it’d get one comment: “LGTM 👍”.

The problem wasn’t that we were lazy.

The problem was that to understand 10k lines of code requires many, many hours of review. We’d have to transform that 2D text into a 3D mental model in our heads. We’d have to carefully consider how the new vast amounts of code fit into the existing system. We’d have to consider high-level architecture and low-level details.

Because of that, even if we spent 3 hours reviewing the PR, submitting comments, reviewing changes, and doing that 2 to 3 times until everything looked good (a process that I always found demoralizing for the reviewer and author alike), the results were still worse than if the author had split the work at the beginning into smaller slices.

So, the joke went that we would just give the 10k-line PR a thumbs up. 👍

Now, as an industry, we’re trying to figure out if we can stop reviewing code. To do that, I see some people turning to the big-plan-upfront strategy to try and specify the desired behavior ahead of time. After all, if we can do that, we can let the latest models run independently for hours!

But that suffers from the exact same problems we had when we tried to review 10k-line PRs: we have to transform the plan’s text into a 3D mental model in our heads. We have to carefully consider how the new vast amounts of code the model might generate will fit into the existing system. We have to consider high-level architecture and low-level details. And because of that, the implementation will be worse than if we split the work into smaller slices and work iteratively.

Thinner slices. More iteration.

For years, software development best practices encouraged a more iterative approach.

We were always trying to capture a small slice – a user story or maybe even just a use case – not a full-fledged specification. We would then build that and reevaluate. In the process, we’d discover some pitfalls or improvements we hadn’t foreseen at first.

Then we’d iterate.

And so, solutions would, to some degree, evolve from the kernel of the initial idea, the realities of what was already present in the codebase, and the new insights we gained with each iteration.

And I think that’s still how we can work.

My preferred flow (at least currently) is to plan the work in full-stack slices. Once I’ve split up my work into a slice, I then work on one or two use cases (sometimes in Gherkin) with the LLM. We then talk about architecture and agree on what the call stack should look like. Perhaps we do some ideation on UI or UX as well.

It’s only once we’ve agreed on all of that, that I let the LLM go. We have a plan of action, but it’s not the plan. It’s not a huge document sitting in my /plans directory. It’s encompassed by the use cases we agreed upon.

And once that’s done, we have a shippable unit of work. We can stop and evaluate what has been done and whether anything needs correction. That helps evolve the design. Then, I repeat the same cycle with the next chunk of work.

But isn’t that slow?

Not any slower than the big-plan-upfront approach. Instead of spending five hours with my eyes bleeding while I read and change the 10k-line plan, I spend that time iterating and evolving the design with the LLM.

It’s a far more engaging and enjoyable experience, and I think it creates better, more maintainable software.