Engineering in the Age of AI

June 03, 2024

Yesterday I wrote about how I use AI to self-serve a lot of legal work. Today, I’m thinking about my #1 use for AI assistance - coding.

Similar to legal work, the key is knowing what you want as an outcome and using AI to learn faster. Rarely does prompting ChatGPT to write an algorithm or an application work in a single-shot. However, rubber-ducking works quite well (now your rubber duck talks back!)

There seem to be 3 types of coding assistance that LLMs provide

1. Chat assistants - ChatGPT, Claude.ai

This is often my first step for exploration. When I need to learn about the different options I have for generating public/private keypairs for users or making sure my understanding of different clustering algorithms is correct, this is where I’ll go.

I also often go to a chat assistant for brand new code for small projects, e.g. writing single-file utilities in Swift, Bash scripts, or Github Workflows. Chat assistants are very good with small, self-contained pieces of code in languages where I’m not quite familiar with the syntax. Once the size of the context gets too large and the number of messages in the thread passes ~8, this becomes much clunkier and the LLM can start going in loops.

These fall down, however, for dependencies that are updating. ChatGPT’s suggestions for github workflow actions are often 2-3 versions out of date and sometimes have security holes, etc. It’s best to use this code as scaffolding, then to implement the details yourself.

2. Smart Code-Completion

This is github copilot’s primary interface where it integrates seamlessly into an IDE and tries to guess what to do next. Cursor and other tools built on VS Code take the same approach (and honestly haven’t offered much beyond that).

The nice things here are that the suggestions are constant and the tool has much more context about what you are working on because it’s inside your project and because it can understand where to focus because it can look at open tabs or recent files, etc.

The downside is that it’s a little harder to steer. I have to really hint what I want sometimes by writing comments or writing the first half of a line to get it to try something new (e.g. if it is stuck in a loop of suggesting comments, I can write const and get it to switch to writing code again). You can’t really tell it directly what you want.

These tools are starting to add some chat, but it’s often awkward and feels like a halfway solution where there’s a lot of work of telling the chat what specific context you need, but it’s also not quite as good at brainstorming as using ChatGPT directly.

3. Coding agents

This is the real future of coding, I just haven’t seen anyone get the UX right yet. IMO, this should be very collaborative allow switching back and forth between views of the code and conversation, and it should integrate the human into the workflow reviewing and approving changes.

I think the right solution here is to have a constant chat in your IDE always open where you can converse about the current open code, get a second opinion, get some suggestions, and if you like them just tell the AI to “do it”. The AI should have access to all your tools and be able to help with them as well, like bringing up diffs, committing to git and/or changing branches, helping to run your development environment, setting up AWS infrastructure, etc.

The closest to what I want has probably been Aider so far. This is most similar to the coding assistant I started to build last Spring (now open source). However, Aider is a little bit too quick on the draw in terms of making changes and immediately committing them, and having to start new conversations all the time.

You could also build an interesting workflow using Github’s collaboration tools, PRs, etc. but it wouldn’t be as fast as if it were live. Basically, I want to do pair programming at all times.

Altogether, it’s a very exciting time to be an engineer; I feel like we are on the cusp of order-of-magnitude productivity increases. The last year was probably 30-50% productivity increase, but the next generation will be 200-500%!