This keeps coming back to me -- I've heard it quoted at conferences by people I've worked with, so I should probably write it down properly:
A good interface with a shit implementation beats a shit interface with a good implementation.
One contains the mess, making it possible to clean up later. The other spreads the mess everywhere else.
Think about what happens when the implementation is bad but the interface is good. The mess exists in one place. Users interact with something clean. When you eventually fix the implementation -- and you will, because you'll have time, because nobody is screaming at you about the interface -- you swap it out. Nobody notices. The fact that behind the curtain you had three bash scripts duct-taped together and a cron job that emails you when it fails is between you and God.
Now think about the opposite. Beautiful implementation. Elegant code. Well-tested. Twelve-factor. The works. But the interface is a mess -- inconsistent naming, leaky abstractions, seven flags you need to set correctly or it silently does the wrong thing. Every single consumer of that system has to understand its quirks. Every team that integrates with it builds workarounds. It's spread across your entire organisation, embedded in dozens of other systems, and you can never clean it up because you'd have to change everything that touches it.
The pipeline that was held together with string
At Bolt, I built a declarative pipeline config system. A data scientist would write a Python file that returned a config dict -- their pipeline definition. We used Python rather than YAML because a small number of pipelines needed to run code to generate parts of their config (reading city lists from an external source, for example). But the interface was still declarative in spirit: describe what your pipeline does, not how to do it. The data scientist's experience was: write config, open PR, merge, pipeline runs. Behind the scenes, the config generated Airflow DAGs -- hence the internal name "Airpipe" (Airflow Pipelines). Data quality checks ran via Great Expectations before and after deployment.
Underneath? My implementation was absolutely dreadful. This was not Bolt's engineering standard -- this was me shipping fast and dirty because the interface mattered more than the internals. The first version was a Python script that parsed the config, generated a bunch of bash commands, and executed them in sequence. Error handling was "if it fails, print the error and exit 1." Logging was print statements. The deployment step shelled out to the AWS CLI. There was a section I commented with # TODO: this is terrible that survived in production for four months.
But the data scientists didn't know any of that. They knew: I write a config, my model gets deployed. When things broke, I fixed them. When I rewrote the execution engine -- twice -- nothing changed for the users. Same config format. Same PR workflow. Same result.
That config was more than a technical artifact. It was the communication channel between two teams. Data scientists expressed what they wanted: preprocessing steps, instance types, rollout strategy. Data engineers decided how that became reality. A data scientist could change how models were grouped or make rollouts more gradual -- DE wouldn't even know. DE could rewrite the orchestration layer -- DS wouldn't notice, provided the contract was still honoured. The config file was the interface between two organisations' concerns. The dirt was contained on both sides of it.
The same pattern works at company scale. A north-star metric is the interface between teams that don't understand each other's work -- product, engineering, marketing, ops. Each side expresses what they need in terms of the shared number. Neither needs to understand the other's implementation.
Some parts I rewrote properly over time. Others? They survived as-is for four and a half years, outlasted my time at the company, and were still running after I left. Battle-tested, never broken, never worth the cost of rewriting when there was always something more valuable to work on. A shit implementation behind a good interface doesn't always get cleaned up. Sometimes it just becomes reliable shit, and reliable shit behind a clean interface is perfectly fine.
Compare this to a system I inherited at a previous company. Somebody had built a genuinely sophisticated deployment pipeline. Custom tooling, proper abstractions, good test coverage. Architecturally sound. But the interface was a deployment script that required editing a config file with eight fields that had to be set in a specific order -- reorder them and it silently used defaults for everything after the first unrecognised key. A dozen CLI flags, several of which interacted in undocumented ways (passing --env staging with --parallel would deploy to production; nobody knew why). Deploying a service required reading a wiki page that was perpetually six months out of date. New engineers took two weeks to learn how to deploy. Two weeks! To push code that was already written and tested.
The implementation was better than anything I would have written. But it didn't matter, because the interface was so hostile that people routed around it. They'd SSH into machines and restart services manually. They'd skip the pipeline entirely and copy files. Every workaround introduced a new failure mode. The clean implementation was pristine and untouched because nobody could figure out how to use it. The actual production deployments were cowboy nonsense happening in terminal sessions with no audit trail.
I see this pattern constantly with third-party services. You integrate with some vendor API that's atrocious -- inconsistent response formats, rate limits that aren't documented, authentication that expires unpredictably. You have two choices. You can let every part of your codebase that needs this service interact with the API directly, spreading the vendor's quirks everywhere. Or you can wrap it. Build a clean internal API that hides the horror. Your wrapper will be ugly -- full of retries, special cases, format translations, workarounds for bugs the vendor won't fix. Every other part of your system talks to your clean interface and never thinks about the vendor. Github vs Gitlab.
When the vendor fixes their API (they won't, but let's be optimistic), you update one wrapper. When you switch vendors (more likely), you rewrite one wrapper. The rest of your system doesn't know and doesn't care.
At Bolt, this was how most backend engineers interacted with AWS. An internal backend-platform library wrapped S3, SNS, SQS -- handling quirks, footguns, access checks, and deliberately restricting features that would cause problems if used carelessly. The AWS-specific dirt was contained in one library maintained by a platform team. When AWS changed something, one team updated one library. Everyone else kept shipping.
Self-service deployment at Bolt worked the same way. I've written about this before -- data scientists deploying their own models without filing tickets. The interface they saw was simple: a config file and a CLI with three commands. Run locally, run on cloud, deploy.
The implementation behind those three commands -- again, my code, not representative of Bolt's engineering at large -- was a disgrace. Docker-in-Docker nonsense. Shell scripts that checked environment variables to decide which cloud provider to talk to. The local-to-cloud parity was achieved through a combination of symlinks and lies.
But it worked. Data scientists deployed models. They didn't file tickets. They didn't wait two weeks. They didn't need to understand Kubernetes or Docker or AWS. They understood their config file and their three commands. The interface contained the dirt.
Over time, I cleaned up the implementation. Replaced the shell scripts with proper Python. Fixed the credentials management. Made the Docker setup less embarrassing. Each improvement was invisible to users. That's the luxury a good interface buys you: the freedom to be embarrassed by your implementation in private, and fix it on your own schedule.
The pull is always towards clean code, elegant solutions, something worth showing in a code review. So the instinct is to spend weeks getting the internals right and bolt on whatever interface fits in the remaining time.
The interface is the part that other humans have to live with. The implementation is the part only you have to live with.
The most useful thing is to address other people's experience first, then deal with your own.
Pieter Hintjens told a version of this story as "A Tale of Two Bridges." One engineer spent seven years designing and building a grand bridge across a gorge -- trains, highways, tollbooths. Another threw a rope across a few miles downstream. People started pulling packages across with a pulley. Someone added a footwalk. Then a market grew, then a town, then a real bridge. The grand bridge was demolished. Everyone was already crossing at the rope bridge.
The rope was a shit implementation behind a good interface (a crossing where people actually wanted to cross). It got upgraded over time from rope to wood to stone to steel. The grand bridge was a beautiful implementation behind the wrong interface (wrong location, wrong assumptions). Hintjens was talking about software protocols, but it's the same principle: get the interface right, and the implementation can evolve. Get the interface wrong, and it doesn't matter how good the engineering is. Unless you're Microsoft, seemingly.
If the dirt lives behind a good interface, you can clean it up later. If it lives in the interface itself, you're stuck with it, and so is everyone else.