mcfunley-com-5myzcq.txt (12692B)
1 [1] 2 3 Dan McKinley 4 Math, Programming, and Minority Reports 5 6 [2] Tweet [3] Follow @mcfunley 7 8 [4]Choose Boring Technology 9 March 30th, 2015 10 11 Probably the single best thing to happen to me in my career was having had [5] 12 Kellan placed in charge of me. I stuck around long enough to see Kellan’s 13 technical decisionmaking start to bear fruit. I learned a great deal from this, 14 but I also learned a great deal as a result of this. I would not have been free 15 to become the engineer that wrote [6]Data Driven Products Now! if Kellan had 16 not been there to so thoroughly stick the landing on technology choices. 17 18 [FRQKLCy] Being inspirational as always. 19 20 In the year since leaving Etsy, I’ve resurrected my ability to care about 21 technology. And my thoughts have crystallized to the point where I can write 22 them down coherently. What follows is a distillation of the Kellan gestalt, 23 which will hopefully serve to horrify him only slightly. 24 25 Embrace Boredom. 26 27 Let’s say every company gets about three innovation tokens. You can spend these 28 however you want, but the supply is fixed for a long while. You might get a few 29 more after you achieve a [7]certain level of stability and maturity, but the 30 general tendency is to overestimate the contents of your wallet. Clearly this 31 model is approximate, but I think it helps. 32 33 If you choose to write your website in NodeJS, you just spent one of your 34 innovation tokens. If you choose to use [8]MongoDB, you just spent one of your 35 innovation tokens. If you choose to use [9]service discovery tech that’s 36 existed for a year or less, you just spent one of your innovation tokens. If 37 you choose to write your own database, oh god, you’re in trouble. 38 39 Any of those choices might be sensible if you’re a javascript consultancy, or a 40 database company. But you’re probably not. You’re probably working for a 41 company that is at least ostensibly [10]rethinking global commerce or [11] 42 reinventing payments on the web or pursuing some other suitably epic mission. 43 In that context, devoting any of your limited attention to innovating ssh is an 44 excellent way to fail. Or at best, delay success [12][1]. 45 46 What counts as boring? That’s a little tricky. “Boring” should not be conflated 47 with “bad.” There is technology out there that is both boring and bad [13][2]. 48 You should not use any of that. But there are many choices of technology that 49 are boring and good, or at least good enough. MySQL is boring. Postgres is 50 boring. PHP is boring. Python is boring. Memcached is boring. Squid is boring. 51 Cron is boring. 52 53 The nice thing about boringness (so constrained) is that the capabilities of 54 these things are well understood. But more importantly, their failure modes are 55 well understood. Anyone who knows me well will understand that it’s only with a 56 overwhelming sense of malaise that I now invoke the spectre of Don Rumsfeld, 57 but I must. 58 59 [n8ElWr3] To be clear, fuck this guy. 60 61 When choosing technology, you have both known unknowns and unknown unknowns 62 [14][3]. 63 64 • A known unknown is something like: we don’t know what happens when this 65 database hits 100% CPU. 66 • An unknown unknown is something like: geez it didn’t even occur to us that 67 [15]writing stats would cause GC pauses. 68 69 Both sets are typically non-empty, even for tech that’s existed for decades. 70 But for shiny new technology the magnitude of unknown unknowns is significantly 71 larger, and this is important. 72 73 Optimize Globally. 74 75 I unapologetically think a bias in favor of boring technology is a good thing, 76 but it’s not the only factor that needs to be considered. Technology choices 77 don’t happen in isolation. They have a scope that touches your entire team, 78 organization, and the system that emerges from the sum total of your choices. 79 80 Adding technology to your company comes with a cost. As an abstract statement 81 this is obvious: if we’re already using Ruby, adding Python to the mix doesn’t 82 feel sensible because the resulting complexity would outweigh Python’s marginal 83 utility. But somehow when we’re talking about Python and Scala or MySQL and 84 Redis people [16]lose their minds, discard all constraints, and start raving 85 about using the best tool for the job. 86 87 [17]Your function in a nutshell is to map business problems onto a solution 88 space that involves choices of software. If the choices of software were truly 89 without baggage, you could indeed pick a whole mess of locally-the-best tools 90 for your assortment of problems. 91 92 Created with Sketch. Problems Technical Solutions The way you might choose 93 technology in a world where choices are cheap: "pick the right tool for the 94 job." 95 96 But of course, the baggage exists. We call the baggage “operations” and to a 97 lesser extent “cognitive overhead.” You have to monitor the thing. You have to 98 figure out unit tests. You need to know the first thing about it to hack on it. 99 You need an init script. I could go on for days here, and all of this adds up 100 fast. 101 102 Created with Sketch. Problems Technical Solutions The way you choose technology 103 in the world where operations are a serious concern (i.e., "reality"). 104 105 The problem with “best tool for the job” thinking is that it takes a myopic 106 view of the words “best” and “job.” Your job is keeping the company in 107 business, god damn it. And the “best” tool is the one that occupies the “least 108 worst” position for as many of your problems as possible. 109 110 It is basically always the case that the long-term costs of keeping a system 111 working reliably vastly exceed any inconveniences you encounter while building 112 it. Mature and productive developers understand this. 113 114 Choose New Technology, Sometimes. 115 116 Taking this reasoning to its reductio ad absurdum would mean picking Java, and 117 then trying to implement a website without using anything else at all. And that 118 would be crazy. You need some means to add things to your toolbox. 119 120 An important first step is to acknowledge that this is a process, and a 121 conversation. New tech eventually has company-wide effects, so adding tech is a 122 decision that requires company-wide visibility. Your organizational specifics 123 may force the conversation, or [18]they may facilitate developers adding new 124 databases and queues without talking to anyone. One way or another you have to 125 set cultural expectations that this is something we all talk about. 126 127 One of the most worthwhile exercises I recommend here is to consider how you 128 would solve your immediate problem without adding anything new. First, posing 129 this question should detect the situation where the “problem” is that someone 130 really wants to use the technology. If that is the case, you should immediately 131 abort. 132 133 [rmdSx] I just watched a webinar about this graph database, we should try it 134 out. 135 136 It can be amazing how far a small set of technology choices can go. The answer 137 to this question in practice is almost never “we can’t do it,” it’s usually 138 just somewhere on the spectrum of “well, we could do it, but it would be too 139 hard” [19][4]. If you think you can’t accomplish your goals with what you’ve 140 got now, you are probably just not thinking creatively enough. 141 142 It’s helpful to write down exactly what it is about the current stack that 143 makes solving the problem prohibitively expensive and difficult. This is 144 related to the previous exercise, but it’s subtly different. 145 146 New technology choices might be purely additive (for example: “we don’t have 147 caching yet, so let’s add memcached”). But they might also overlap or replace 148 things you are already using. If that’s the case, you should set clear 149 expectations about migrating old functionality to the new system. The policy 150 should typically be “we’re committed to migrating,” with a proposed timeline. 151 The intention of this step is to keep wreckage at manageable levels, and to 152 avoid proliferating locally-optimal solutions. 153 154 This process is not daunting, and it’s not much of a hassle. It’s a handful of 155 questions to fill out as homework, followed by a meeting to talk about it. I 156 think that if a new technology (or a new service to be created on your 157 infrastructure) can pass through this gauntlet unscathed, adding it is fine. 158 159 Just Ship. 160 161 Polyglot programming is sold with the promise that letting developers choose 162 their own tools with complete freedom will make them more effective at solving 163 problems. This is a naive definition of the problems at best, and motivated 164 reasoning at worst. The weight of day-to-day operational [20]toil this creates 165 crushes you to death. 166 167 Mindful choice of technology gives engineering minds real freedom: the freedom 168 to [21]contemplate bigger questions. Technology for its own sake is snake oil. 169 170 Update, July 27th 2015: I wrote a talk based on this article. You can see it 171 [22]here. 172 173 ━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━ 174 175 1. Etsy in its early years suffered from this pretty badly. We hired a bunch 176 of Python programmers and decided that we needed to find something for them 177 to do in Python, and the only thing that came to mind was creating a 178 pointless middle layer that [23]required years of effort to amputate. 179 Meanwhile, the 90th percentile search latency was about two minutes. [24] 180 Etsy didn't fail, but it went several years without shipping anything at 181 all. So it took longer to succeed than it needed to. 182 2. We often casually refer to the boring/bad intersection of doom as 183 “enterprise software,” but that terminology may be imprecise. 184 3. In saying this Rumsfeld was either intentionally or unintentionally 185 alluding to [25]the Socratic Paradox. Socrates was by all accounts a 186 thoughtful individual in a number of ways that Rumsfeld is not. 187 4. A good example of this from my experience is [26]Etsy’s activity feeds. 188 When we built this feature, we were working pretty hard to consolidate most 189 of Etsy onto PHP, MySQL, Memcached, and Gearman (a PHP job server). It was 190 much more complicated to implement the feature on that stack than it might 191 have been with something like Redis (or [27]maybe not). But it is 192 absolutely possible to build activity feeds on that stack. 193 194 An amazing thing happened with that project: our attention turned elsewhere 195 for several years. During that time, activity feeds scaled up 20x while 196 nobody was watching it at all. We made no changes whatsoever specifically 197 targeted at activity feeds, but everything worked out fine as usage 198 exploded because we were using a shared platform. This is the long-term 199 benefit of restraint in technology choices in a nutshell. 200 201 This isn’t an absolutist position--while activity feeds stored in memcached 202 was judged to be practical, implementing full text search with faceting in 203 raw PHP wasn't. So Etsy used Solr. 204 205 [28]Back home 206 [29] Tweet [30] Follow @mcfunley 207 208 • [31]GitHub 209 • [32]LinkedIn 210 211 [33]Feed | Copyright © 2004-2023 Dan McKinley. 212 213 References: 214 215 [1] https://mcfunley.com/ 216 [2] https://twitter.com/intent/tweet 217 [3] https://twitter.com/mcfunley 218 [4] https://mcfunley.com/choose-boring-technology 219 [5] http://laughingmeme.org/ 220 [6] https://mcfunley.com/data-driven-products-lean-startup-2014 221 [7] http://rc3.org/2015/03/24/the-pleasure-of-building-big-things/ 222 [8] https://mcfunley.com/why-mongodb-never-worked-out-at-etsy 223 [9] https://consul.io/ 224 [10] https://www.etsy.com/ 225 [11] https://stripe.com/ 226 [12] https://mcfunley.com/choose-boring-technology#f1 227 [13] https://mcfunley.com/choose-boring-technology#f2 228 [14] https://mcfunley.com/choose-boring-technology#f3 229 [15] http://www.evanjones.ca/jvm-mmap-pause.html 230 [16] http://martinfowler.com/bliki/PolyglotPersistence.html 231 [17] https://twitter.com/coda/status/580531932393504768 232 [18] https://twitter.com/mcfunley/status/578603932949164032 233 [19] https://mcfunley.com/choose-boring-technology#f4 234 [20] https://twitter.com/handler 235 [21] https://mcfunley.com/effective-web-experimentation-as-a-homo-narrans 236 [22] http://boringtechnology.club/ 237 [23] https://www.youtube.com/watch?v=eenrfm50mXw 238 [24] http://www.sec.gov/Archives/edgar/data/1370637/000119312515077045/d806992ds1.htm 239 [25] http://en.wikipedia.org/wiki/I_know_that_I_know_nothing 240 [26] https://speakerdeck.com/mcfunley/etsy-activity-feed-architecture 241 [27] https://aphyr.com/posts/283-call-me-maybe-redis 242 [28] https://mcfunley.com/ 243 [29] https://twitter.com/intent/tweet 244 [30] https://twitter.com/mcfunley 245 [31] https://github.com/mcfunley 246 [32] https://www.linkedin.com/in/mcfunley 247 [33] https://mcfunley.com/feed.xml