CodeBud: Bringing Back Pair Programming

Inspiration

Since GPT-3.5 dropped in 2022, the way we code has fundamentally changed. First, we copy-pasted from ChatGPT. Then AI moved into our IDEs as autocomplete. Now, we generate entire files with a single prompt.

We went from writing code to reviewing AI-generated code. Getting an MVP out fast is great, but the immense technical debt from the generated codebase and developer skills rotting has been getting worse and worse with each new model. Pair programming, that magical collaboration where a junior learns from a senior, died. Not because people didn't want it, but because AI felt "good enough."

But autocomplete isn't mentorship. Code generation isn't understanding.

We built CodeBud to fix that: an AI that doesn't write code for you, but writes code with you via natural voice conversation.

What it does

CodeBud is a voice-powered AI pair programmer embedded in VS Code, available as a simple extension.

Real-time awareness: The extension watches every keystroke, tracking your cursor position, the code around it, and any errors in real-time.

Proactive feedback: When you pause typing, CodeBud automatically reviews your recent changes and speaks up if something looks off.

Two modes:

Navigator: The AI guides and explains. You stay in control, and (as normal), can query what and why it makes design decisions. This way, you understand the codebase whilst speeding up dev time.
Driver: The AI can write code by voice command. - slowly! Rather than spitting out paragraphs of code in seconds, the AI will type out code line by line and explain the general thought behind it as it goes, ensuring you can still act as an effective navigator and maintain understanding and control over the codebase. The dashboard will record your mistakes and give feedback to help make you a better programmer, ensuring your skills do not rot away.

How we built it

The system operates as a three-tier architecture designed to bridge the gap between a local development environment and cloud-based Conversational AI.

┌─────────────────┐     ┌──────────────────┐     ┌──────────────────┐
│   VS Code Ext   │────▶│   Voice UI       │────▶│  ElevenLabs AI   │
│   (Express API) │◀────│   (React)        │◀────│  (Voice Agent)   │
└─────────────────┘     └──────────────────┘     └──────────────────┘

1. The Local Controller (VS Code Extension)

Acting as the source of truth, the extension manages the developer's local environment.

Context Provider: An internal Express API exposes real-time project metadata, active file content, and compiler diagnostics.
Execution Engine: It provides write-access to the editor, allowing the AI to perform "LSP-like" actions such as code insertion and refactoring through dedicated REST endpoints.
Deployment: Bundled via esbuild into a standalone VSIX for zero-dependency installation.

2. The Interaction Layer (React Webview)

The "Brain" of the UI that synchronizes the state between the editor and the AI.

State Sync: Continuously polls the Extension API to maintain a "Live Context" buffer.
Orchestration: Leverages the ElevenLabs Conversational AI SDK to manage the full-duplex audio stream, ensuring the AI is updated with the latest code context before every spoken response.

3. The Intelligence Layer (ElevenLabs Agent)

A cloud-hosted LLM configured for low-latency voice interaction.

Tool Use: Equipped with Client-Side Tools that map conversation intent to API calls (e.g., the AI "decides" to call /insert based on the user's request). The brain (LLM) can be changed dependent on response time and user cost sensitivity, e.g. we were using a mix of GPT 5.2 and Sonnet 4.5.
Awareness: Uses the summaries provided by the Voice UI to maintain situational awareness of the user's codebase without needing direct file-system access.

Challenges we ran into

1. Microphone access in VS Code webviews

VS Code's webview sandbox doesn't allow for microphone access. We initially built the UI as an embedded sidebar panel, only to discover voice wouldn't work.

Solution: Open the Voice UI in an external browser window instead.

2. Keeping the AI contextually aware

LLMs have no memory between turns. We needed the AI to know what file you're in, what line you're on, and what you just typed without you telling it.

Solution: A polling system sends context summaries every few seconds, and we inject structured [CONTEXT] messages into the conversation.

3. Bundling for portability

Initially, our VSIX failed to activate because node_modules wasn't included. The extension requires Express at runtime, not just build time.

Solution: Switched from tsc to esbuild, which bundles all dependencies into the output file.

Accomplishments that we're proud of

24-hour build: From concept to working demo in one hackathon
Real-time awareness: The AI genuinely knows what you're coding
Installable package: A .vsix anyone can install and not just a demo
Natural interaction: Coding by voice actually feels intuitive and you can understand what is in your codebase! -Insights: Dashboard gives you insights into your common coding pitfalls and advice on areas to focus on - so we don't lose our coding skills and end up blindly relying on LLMs to code everything.

What we learned

While VS Code extensions offer immense power, they can be more challenging to use than expected because webviews, activation events, and bundling processes are full of tradeoffs.
The integration of Voice AI necessitates a fundamental shift in the interface paradigm, moving the design focus from traditional click-based navigation to natural conversational flow.
Successful pair programming relies heavily on pacing, requiring the AI to intelligently differentiate between a developer's contemplative silence and a moment of genuine blockage.
Real-time context is the critical differentiator for specialized development tools; without it, even the most advanced AI functions as little more than a generic chatbot.

What's next for CodeBud

Native Voice Integration: Transition to using system-level audio to embed voice capabilities directly within VS Code, eliminating the need for external UI dependencies. Persistent Project Memory: Implement session-based persistence so the AI retains context across multiple days, allowing it to remember architectural decisions and previous discussions. Multi-File Awareness: Expand the AI’s scope to analyze and understand the entire project structure and inter-file dependencies, rather than being limited to the currently active file. Marketplace Release: Finalize the packaging and distribution process to publish CodeBud on the official VS Code Marketplace for seamless, one-click installation. Custom AI Personas: Introduce configurable teaching styles, allowing users to toggle between different AI personalities such as a strict code reviewer, a patient mentor, or a rapid prototyper.

CodeBud doesn't replace pair programming, it brings it back :)