From fa13fce51ff783b4f623f6d889e3e22b27c737d6 Mon Sep 17 00:00:00 2001 From: David Eisinger Date: Mon, 3 Jul 2023 23:40:58 -0400 Subject: [PATCH] Choose boring technology --- content/notes/first-it-must-work/index.md | 6 + static/archive/mcfunley-com-5myzcq.txt | 260 ++++++++++++++++++++++ 2 files changed, 266 insertions(+) create mode 100644 static/archive/mcfunley-com-5myzcq.txt diff --git a/content/notes/first-it-must-work/index.md b/content/notes/first-it-must-work/index.md index 40876b4..2715e5c 100644 --- a/content/notes/first-it-must-work/index.md +++ b/content/notes/first-it-must-work/index.md @@ -31,6 +31,10 @@ references: url: https://techxplore.com/news/2023-06-problems.html date: 2023-07-03T21:19:28Z file: techxplore-com-8o7hmu.txt +- title: "Dan McKinley :: Choose Boring Technology" + url: https://mcfunley.com/choose-boring-technology + date: 2023-07-04T03:40:17Z + file: mcfunley-com-5myzcq.txt --- ### Thoughts on priorities in software development @@ -51,9 +55,11 @@ references: * [Imaginary Problems Are the Root of Bad Software][5] * [When to Build Millennia Sewers][6] * [We are wasting up to 20% of our time on computer problems, says study][7] +* [Choose Boring Technology][8] [3]: https://grugbrain.dev/ [4]: https://world.hey.com/dhh/even-amazon-can-t-make-sense-of-serverless-or-microservices-59625580 [5]: https://cerebralab.com/Imaginary_Problems_Are_the_Root_of_Bad_Software [6]: https://taylor.town/millennium-sewer [7]: https://techxplore.com/news/2023-06-problems.html +[8]: https://mcfunley.com/choose-boring-technology diff --git a/static/archive/mcfunley-com-5myzcq.txt b/static/archive/mcfunley-com-5myzcq.txt new file mode 100644 index 0000000..6090ee8 --- /dev/null +++ b/static/archive/mcfunley-com-5myzcq.txt @@ -0,0 +1,260 @@ + #[1]alternate + + [2] + + Dan McKinley + Math, Programming, and Minority Reports + + [3]Tweet [4]Follow @mcfunley + + [5]Choose Boring Technology + March 30th, 2015 + + Probably the single best thing to happen to me in my career was having + had [6]Kellan placed in charge of me. I stuck around long enough to see + Kellanâs technical decisionmaking start to bear fruit. I learned a + great deal from this, but I also learned a great deal as a result of + this. I would not have been free to become the engineer that wrote + [7]Data Driven Products Now! if Kellan had not been there to so + thoroughly stick the landing on technology choices. + [FRQKLCy.jpg] Being inspirational as always. + + In the year since leaving Etsy, Iâve resurrected my ability to care + about technology. And my thoughts have crystallized to the point where + I can write them down coherently. What follows is a distillation of the + Kellan gestalt, which will hopefully serve to horrify him only + slightly. + + Embrace Boredom. + + Letâs say every company gets about three innovation tokens. You can + spend these however you want, but the supply is fixed for a long while. + You might get a few more after you achieve a [8]certain level of + stability and maturity, but the general tendency is to overestimate the + contents of your wallet. Clearly this model is approximate, but I think + it helps. + + If you choose to write your website in NodeJS, you just spent one of + your innovation tokens. If you choose to use [9]MongoDB, you just spent + one of your innovation tokens. If you choose to use [10]service + discovery tech thatâs existed for a year or less, you just spent one of + your innovation tokens. If you choose to write your own database, oh + god, youâre in trouble. + + Any of those choices might be sensible if youâre a javascript + consultancy, or a database company. But youâre probably not. Youâre + probably working for a company that is at least ostensibly + [11]rethinking global commerce or [12]reinventing payments on the web + or pursuing some other suitably epic mission. In that context, devoting + any of your limited attention to innovating ssh is an excellent way to + fail. Or at best, delay success [13][1]. + + What counts as boring? Thatâs a little tricky. âBoringâ should not be + conflated with âbad.â There is technology out there that is both boring + and bad [14][2]. You should not use any of that. But there are many + choices of technology that are boring and good, or at least good + enough. MySQL is boring. Postgres is boring. PHP is boring. Python is + boring. Memcached is boring. Squid is boring. Cron is boring. + + The nice thing about boringness (so constrained) is that the + capabilities of these things are well understood. But more importantly, + their failure modes are well understood. Anyone who knows me well will + understand that itâs only with a overwhelming sense of malaise that I + now invoke the spectre of Don Rumsfeld, but I must. + [n8ElWr3.jpg] To be clear, fuck this guy. + + When choosing technology, you have both known unknowns and unknown + unknowns [15][3]. + * A known unknown is something like: we donât know what happens when + this database hits 100% CPU. + * An unknown unknown is something like: geez it didnât even occur to + us that [16]writing stats would cause GC pauses. + + Both sets are typically non-empty, even for tech thatâs existed for + decades. But for shiny new technology the magnitude of unknown unknowns + is significantly larger, and this is important. + + Optimize Globally. + + I unapologetically think a bias in favor of boring technology is a good + thing, but itâs not the only factor that needs to be considered. + Technology choices donât happen in isolation. They have a scope that + touches your entire team, organization, and the system that emerges + from the sum total of your choices. + + Adding technology to your company comes with a cost. As an abstract + statement this is obvious: if weâre already using Ruby, adding Python + to the mix doesnât feel sensible because the resulting complexity would + outweigh Pythonâs marginal utility. But somehow when weâre talking + about Python and Scala or MySQL and Redis people [17]lose their minds, + discard all constraints, and start raving about using the best tool for + the job. + + [18]Your function in a nutshell is to map business problems onto a + solution space that involves choices of software. If the choices of + software were truly without baggage, you could indeed pick a whole mess + of locally-the-best tools for your assortment of problems. + Created with Sketch. Problems Technical Solutions The way you might + choose technology in a world where choices are cheap: "pick the right + tool for the job." + + But of course, the baggage exists. We call the baggage âoperationsâ and + to a lesser extent âcognitive overhead.â You have to monitor the thing. + You have to figure out unit tests. You need to know the first thing + about it to hack on it. You need an init script. I could go on for days + here, and all of this adds up fast. + Created with Sketch. Problems Technical Solutions The way you choose + technology in the world where operations are a serious concern (i.e., + "reality"). + + The problem with âbest tool for the jobâ thinking is that it takes a + myopic view of the words âbestâ and âjob.â Your job is keeping the + company in business, god damn it. And the âbestâ tool is the one that + occupies the âleast worstâ position for as many of your problems as + possible. + + It is basically always the case that the long-term costs of keeping a + system working reliably vastly exceed any inconveniences you encounter + while building it. Mature and productive developers understand this. + + Choose New Technology, Sometimes. + + Taking this reasoning to its reductio ad absurdum would mean picking + Java, and then trying to implement a website without using anything + else at all. And that would be crazy. You need some means to add things + to your toolbox. + + An important first step is to acknowledge that this is a process, and a + conversation. New tech eventually has company-wide effects, so adding + tech is a decision that requires company-wide visibility. Your + organizational specifics may force the conversation, or [19]they may + facilitate developers adding new databases and queues without talking + to anyone. One way or another you have to set cultural expectations + that this is something we all talk about. + + One of the most worthwhile exercises I recommend here is to consider + how you would solve your immediate problem without adding anything new. + First, posing this question should detect the situation where the + âproblemâ is that someone really wants to use the technology. If that + is the case, you should immediately abort. + [rmdSx.gif] I just watched a webinar about this graph database, we + should try it out. + + It can be amazing how far a small set of technology choices can go. The + answer to this question in practice is almost never âwe canât do it,â + itâs usually just somewhere on the spectrum of âwell, we could do it, + but it would be too hardâ [20][4]. If you think you canât accomplish + your goals with what youâve got now, you are probably just not thinking + creatively enough. + + Itâs helpful to write down exactly what it is about the current stack + that makes solving the problem prohibitively expensive and difficult. + This is related to the previous exercise, but itâs subtly different. + + New technology choices might be purely additive (for example: âwe donât + have caching yet, so letâs add memcachedâ). But they might also overlap + or replace things you are already using. If thatâs the case, you should + set clear expectations about migrating old functionality to the new + system. The policy should typically be âweâre committed to migrating,â + with a proposed timeline. The intention of this step is to keep + wreckage at manageable levels, and to avoid proliferating + locally-optimal solutions. + + This process is not daunting, and itâs not much of a hassle. Itâs a + handful of questions to fill out as homework, followed by a meeting to + talk about it. I think that if a new technology (or a new service to be + created on your infrastructure) can pass through this gauntlet + unscathed, adding it is fine. + + Just Ship. + + Polyglot programming is sold with the promise that letting developers + choose their own tools with complete freedom will make them more + effective at solving problems. This is a naive definition of the + problems at best, and motivated reasoning at worst. The weight of + day-to-day operational [21]toil this creates crushes you to death. + + Mindful choice of technology gives engineering minds real freedom: the + freedom to [22]contemplate bigger questions. Technology for its own + sake is snake oil. + + Update, July 27th 2015: I wrote a talk based on this article. You can + see it [23]here. + __________________________________________________________________ + + 1. Etsy in its early years suffered from this pretty badly. We hired a + bunch of Python programmers and decided that we needed to find + something for them to do in Python, and the only thing that came to + mind was creating a pointless middle layer that [24]required years + of effort to amputate. Meanwhile, the 90th percentile search + latency was about two minutes. [25]Etsy didn't fail, but it went + several years without shipping anything at all. So it took longer + to succeed than it needed to. + 2. We often casually refer to the boring/bad intersection of doom as + “enterprise software,” but that terminology may be imprecise. + 3. In saying this Rumsfeld was either intentionally or unintentionally + alluding to [26]the Socratic Paradox. Socrates was by all accounts + a thoughtful individual in a number of ways that Rumsfeld is not. + 4. A good example of this from my experience is [27]Etsy’s activity + feeds. When we built this feature, we were working pretty hard to + consolidate most of Etsy onto PHP, MySQL, Memcached, and Gearman (a + PHP job server). It was much more complicated to implement the + feature on that stack than it might have been with something like + Redis (or [28]maybe not). But it is absolutely possible to build + activity feeds on that stack. + An amazing thing happened with that project: our attention turned + elsewhere for several years. During that time, activity feeds + scaled up 20x while nobody was watching it at all. We made no + changes whatsoever specifically targeted at activity feeds, but + everything worked out fine as usage exploded because we were using + a shared platform. This is the long-term benefit of restraint in + technology choices in a nutshell. + This isn’t an absolutist position--while activity feeds stored in + memcached was judged to be practical, implementing full text search + with faceting in raw PHP wasn't. So Etsy used Solr. + + [29]Back home + + [30]Tweet [31]Follow @mcfunley + + * [32]GitHub + * [33]LinkedIn + + [34]Feed | Copyright © 2004-2023 Dan McKinley. + +References + + 1. https://mcfunley.com/feed.xml + 2. file:/// + 3. https://twitter.com/intent/tweet + 4. https://twitter.com/mcfunley + 5. file:///choose-boring-technology + 6. http://laughingmeme.org/ + 7. file:///data-driven-products-lean-startup-2014 + 8. http://rc3.org/2015/03/24/the-pleasure-of-building-big-things/ + 9. file:///why-mongodb-never-worked-out-at-etsy + 10. https://consul.io/ + 11. https://www.etsy.com/ + 12. https://stripe.com/ + 13. file:///var/folders/q9/qlz2w5251kzdfgn0np7z2s4c0000gn/T/L21912-441TMP.html#f1 + 14. file:///var/folders/q9/qlz2w5251kzdfgn0np7z2s4c0000gn/T/L21912-441TMP.html#f2 + 15. file:///var/folders/q9/qlz2w5251kzdfgn0np7z2s4c0000gn/T/L21912-441TMP.html#f3 + 16. http://www.evanjones.ca/jvm-mmap-pause.html + 17. http://martinfowler.com/bliki/PolyglotPersistence.html + 18. https://twitter.com/coda/status/580531932393504768 + 19. https://twitter.com/mcfunley/status/578603932949164032 + 20. file:///var/folders/q9/qlz2w5251kzdfgn0np7z2s4c0000gn/T/L21912-441TMP.html#f4 + 21. https://twitter.com/handler + 22. file:///effective-web-experimentation-as-a-homo-narrans + 23. http://boringtechnology.club/ + 24. https://www.youtube.com/watch?v=eenrfm50mXw + 25. http://www.sec.gov/Archives/edgar/data/1370637/000119312515077045/d806992ds1.htm + 26. http://en.wikipedia.org/wiki/I_know_that_I_know_nothing + 27. https://speakerdeck.com/mcfunley/etsy-activity-feed-architecture + 28. https://aphyr.com/posts/283-call-me-maybe-redis + 29. file:/// + 30. https://twitter.com/intent/tweet + 31. https://twitter.com/mcfunley + 32. https://github.com/mcfunley + 33. https://www.linkedin.com/in/mcfunley + 34. https://mcfunley.com/feed.xml