We all hit bugs that feel impossible to diagnose. The hardest ones to debug are intermittent or inconsistent. How do I fix what works sometimes? If a line of code always breaks then the fix is direct. The system is linear. The action leads to a failure.
How do I debug a bug that is intermittent? How do I debug something that works sometimes?
Here are some reminders to myself next time I encounter a baffling debugging session.
(photo credit Charles Cushing)
This article is intended for startup founders, particularly for those who are first timers. There is nothing extraordinary about my VC experiences. I have done the following: raised an angel round and two institutional rounds (and failed at two rounds?) at a company I co-founded in the GovTech/Community space (SeeClickFix). I have participated in pitches as CTO of a pre-PMF (product market fit) mortgage tech company (HeavyWater). I have raised a friends-and-family round at my current Cloud Infrastructure company (https://www.cyclic.sh)
I was 16 and about to head off as the youngest member of a month-long wilderness sailing trip. The night before my departure my dad came into my room to check on my packing and inspect my gear. He got serious and gave me the following advice for how to stay alive, keep safe and make the most of my time.
It has stuck with me. It is the best, most concise advice I know for how to build a startup, be a parent or venture into the unknown.
I have written previously about how we, as technical professionals, misuse the term debt and end up sounding like idiots in a business context. This is meant as a companion, to help better communicate, to our non-professional colleagues, what we mean when we say "technical debt".
As a repeat founder I get the question from entrepreneurs and others: "how do you get your first users?"
For me, this is actually the easiest stage. I think of acquiring and growing a group of users differently based on the scale of the existing user base. There is a great book that talks about going 0 to 1 (https://en.wikipedia.org/wiki/Zero\_to\_One). What I mean here is how to go from 0 users to 1 user.
I know how to write good code.
Usually it means spending more time now to make the code better for some future state of the world. This assumes that I know what the future will look like. In software we rarely know what the future holds.
(a short story inspired by true events)
TLDR: Mere mortals implement best practices for S3 read and write performance, create a million billion s3 objects with their high performance system, try to clean up the really really expensive mess and find themselves in a dimension as vast as space and as timeless as infinity.
Does this sound familar?
Techie: We are drowning in technical debt!
Business person: Oh really! What is it costing us?
Techie: Oh, it’s not that. What I’m saying is the code doesn’t use best practices, there are alot of things hard coded and lots of spaghetti code. It makes the code really brittle.
Business person: But can we still deliver what we promised to Big Co Inc?
Techie: Yes, but we will have to do some workarounds. Working on the system is painful and what could have been a one day job will instead take a week. We really need to refactor!
Business person: Ah. That's good news. Our number one priority right now is Big Co Inc.
Conclusion: You don’t like your job because it is hard and/or annoying. I will tell your dev manager to ask you to work on the weekend.
If you have ever worked with AWS as a developer then you will know what I mean, AWS tools suck. They suck so bad there is an entire industry focused on solving for this pile of suck. For just Infrastructure as Code, for example, there are Terraform, Pulumi and Serverless Framework. AWS is famous for "listening to customers". So what gives? Why do their tools still feel clunky, complex and riddled with infuriating edge cases?
If you have used AWS you know how costs can sneak up on you. I have experience getting a heart stopping bill at a previous startup. With Cyclic I've decided I want to start adding controls and alerts from the start.
I recently found that AWS added Cloud Formation support for Budgets (AWS::Budgets::Budget). You can create a quick budget to to notify you of either actual COST limits or FORECASTED cost limits. Here is a sample Cloud Formation example.