Writing code for a computer is hard enough. You take something big and fuzzy, some large vague business outcome you want to achive. Then you break it down recursively and think about all the cases until you have clear logical statements a computer can follow.

   As I am en route to see my first total solar eclipse, I was curious how hard it would be to compute eclipses in Python. It turns out, ignoring some minor coordinate system head banging, I was able to get something half decent working in a couple of hours.

   CIA produced a fantastic book during the peak of World War called Simple Sabotage. It laid out various ways for infiltrators to ruin productivity of a company. Some of the advice is timeless, for instance the section about General interference with Organizations and Production :

   Long story short: I’m working on a super cool tool called Modal. Please check it out it lets you run things in the cloud without having to think about infrastructure. Scaling out, scheduling, containerization, using GPUs, setting up webhooks, and all kinds of other stuff.

   This is is in many respects a successor to a blog post I wrote last year about what I want from software infrastructure, but the ideas morphed in my head into something sort of wider.

   Hi It’s your friendly project management theorician. You might remember me from blog posts such as Why software projects take longer than you think, which is a blog post I wrote a long time ago positing that software projects completion time follow a log normal distribution.

   Here’s a theory I have about cloud vendors AWS, Azure, GCP : Cloud vendors will increasingly focus on the lowest layers in the stack: basically leasing capacity in their data centers through an API. Other pure software providers will build all the stuff on top of it.

   This isn’t as much of a blog post as an elaboration of a tweet I posted the other day: I think this specialization of data teams into different roles data scientist, data engineer, analytics engineer, ML engineer etc is generally a bad thing driven by the fact that tools are bad and too hard...

   I guess I should really call this a parable. The backdrop is: you have been brought in to grow a tiny data team people at a mid stage startup M annual revenue , although this story could take place at many different types of companies.

   Software infrastructure by which I include everything ending with aaS, or anything remotely similar to it is an exciting field, in particular because despite what the neo luddites may say it keeps getting better every year I love working with something that moves so quickly.

   I joined Better in early because I thought the team was crazy enough to actually change one of the largest industries in the US. For six years, I ran the tech team, hiring people, probably doing, interviews, and according to GitHub I added , lines of code and removed , .

   It’s a popular attitude among developers to rant about our tools and how broken things are. Maybe I’m an optimistic person, because my viewpoint is the complete opposite I had my first job as a software engineer in , and in the last two decades I’ve seen software engineering changing in ways...

   I spent a ton of time looking at different software providers, both as a CTO, and as a nerd advanced consumer who builds stuff in my spare time. In the last years, there has been an order of magnitude more products that cater directly to developers, through APIs, SDKs, and tooling.

   We live in a year of about , amateur epidemiologists and I have no desire to join that club . But I read something about COVID deaths that I thought was interesting and wanted to see if I could replicated it through data.

   Compensation has always been one of the most confusing parts of management to me. Getting it right is obviously extremely important. Compensation is what drives our entire economy, and you could look at the market for labor as one gigantic resource allocating machine in the same way as people look...

   Hanlon’s razor is a classic aphorism I’m sure you have heard before: Never attribute to malice that which can be adequately explained by stupidity. I’ve found that neither malice nor stupidity is the most common reason when you don’t understand why something is in a certain way.

   Let’s consider a toy model where you’re hiring for two things and that those are equally valuable. It’s not very important what those are, so let’s just call them thing A and thing B for now.

   I recently finished the excellent book Kochland. This isn’t my first interest in Koch I read The Science of Success by Charles Koch himself a couple of years ago. Charles Koch inherited a tiny company in and turned it into one of the world’s largest ones.

   Just a quick note that my team is always hiring at Better. A lot of new people have been joining the team here in NYC lately the tech team has actually grown from to in just months.

   My company has a buffet every Friday, and the lines grow to epic proportions when the food arrives. I’ve suspected for years that the classic buffet line system is a deeply flawed and inefficient method, and every time I’m stuck in the line has made me more convinced.

   No one asked for this, but I’m something like years into my career and have had my fair share of mistakes and luck so I thought I’d share some. Honestly, I feel like I’ve mostly benefitted from luck.

   This is a blog post originally featured on the Better engineering blog. If you want to link to this article or share it, please go to the original post URL Separately, I’m sorry it’s been so long with no posts on this blog.

   Anyone who built software for a while knows that estimating how long something is going to take is hard. It’s hard to come up with an unbiased estimate of how long something will take, when fundamentally the work in itself is about solving something.

   When I started building up a tech team for Better, I made a very conscious decision to pay at the high end to get people. I thought this made more sense: they cost a bit more money to hire, but output usually more than compensates for it.

   A modern tech stack typically involves at least a frontend and backend but relatively quickly also grows to include a data platform. This typically grows out of the need for ad hoc analysis and reporting but possibly evolves into a whole oil refinery of cronjobs, dashboards, bulk data copying, and...

   It started with a tweet: New years resolution: every plot I make during will contain uncertainty estimates Erik Bernhardsson bernhardsson January, Why Because I’ve been sitting in , , meetings where people endlessly debate whether the monthly number of widgets is going up...

   This is a bit of a rant but I really don’t like software that invents its own query language. There’s a trillion different ORMs out there. Another trillion databases with their own query language. Another trillion SaaS products where the only way to query is to learn some random query DSL they made...

   I get bored reading management books very easily and lately I’ve been reading about a wide range of almost arbitrary topics. One of the lenses I tend to read through is to see different management styles in different environments.

   As some of you may know, one of my side interests is approximate nearest neighbor algorithms. I’m the author of Annoy, a library with, stars on Github as of today. It offers fast approximate search for nearest neighbors with the additional benefit that you can load data super fast from disk...

   Ok, so I have to first preface this whole blog post by a few things: I really struggle with the term microservices. I can’t put my finger on exactly why. Maybe because the term is hopelessly ill defined, maybe because it’s gotten picked up by the hype train.

   I have done roughly, interviews in my life. When I started recruiting, I had so much confidence in my ability to assess people. Let me just throw a couple of algorithm questions at a candidate and then I’ll tell you if they are good or not

   I’ve been reading up on operations research lately, including queueing theory. It started out as a way to understand the very complex mortgage process I work at a mortgage startup but it’s turned into my little hammer and now I see nails everywhere.

   I started writing this blog in late , partly because I felt like it would help me improve my English and my writing skills, partly because I kept having a lot of random ideas in my head and I wanted to write them down somewhere.

   UPDATE : There are is a later blog post with newer benchmarks One of my super nerdy interests include approximate algorithms for nearest neighbors in high dimensional spaces. The problem is simple. You have say M points in some high dimensional space.

   I’m interrupting the regular programming for a quick announcement: we’re looking for data engineers at Better. You would be the first one to join and would work a lot directly with me. Some fun things you could work on these are all projects I’m working on right now :

   Turns out having a toddler isn’t super compatible with reading. I used to read books year as a teenager, but it has slowly deteriorated to maybe books, at most. And I don’t even finish all of them because life is too short

   I spent a few days during the holidays fixing up a bunch of semi dormant open source projects and I have a couple of blog posts in the pipeline about various updates. First up, I made a number of fixes to Git of Theseus which is a tool written in Python that generates statistics about Git...

   I spent six years at a company that went from people to and one contributing factor leading to my departure was that I went from a maker to a person stuck in meetings every day.

   I had an interesting idea a few weeks ago, best explained through an example. Let’s say you’re running an e commerce site I kind of do and you want to optimize the number of purchases. Let’s also say we try to learn as much as we can from users, both using A B tests but also using just basic...

   I’ve been a bit bad at posting things with a regular cadence lately, partly because I’m trying to adjust to having a toddler, partly because the hunt for clicks has caused such a high bar for me that I feel like I have to post something Pulitzer worthy.

   There are often close relationships between top level business metrics. For instance, it’s well known that retention has a super strong impact on the valuation of a subscription business. Or that the of occupied seats is super important for an airline.

   A funny thing about being a foreigner is how you realize people take broken things for granted. I’m going to go out on a limb here claiming that the US has a pretty dumb banking system.

   Just for fun, I generated these graphs of the number of letters in the word for each number. I really spent about minutes on this ok...possibly also another minutes tweaking the plots : More languages

   Here’s a dumb extremely accurate rule I’m postulating for software engineering projects: you need at least examples before you solve the right problem . This is what I’ve noticed: Don’t factor out shared code between two classes.

   I just bought Machine, Platform, Crowd: Harnessing Our Digital Future and discovered that it mentions my blog in particular the post When machine learning matters. Ok, I lied a little bit. I didn’t discover it serendipitously.

   There’s about million blog posts about the diversity memo that leaked out of Google a couple of weeks ago. I think the case for any biological difference is pretty weak, and it bothers me when people refer to an interest gap as anything else than caused by the environment.

   I just spent a few days in Italy, on the Ligurian coast. Even though we were on the west side of Italy, the Mediterranean sea was to the east, because the house was situated on a long bay.

   I’ve written before about the importance of iterating quickly but I didn’t necessarily talk about some concrete things you can do. When I’ve built up the tech team at Better, I’ve intentionally optimized for fast iteration speed above almost everything else.

   Remember when everyone had a really ugly blog with a blogroll Anyway, just think the word is funny. I follow a few hundred blogs using Feedly and Reeder and have been reading a few hundred thousand blog posts over the last years.

   How hard can it be to compute conversion rate Take the total number of users that converted and divide them with the total number of users. Done. Except... it’s a lot more complicated when you have any sort of significant time lag.