index.md (10979B)
1 --- 2 title: "Maintenance Matters: Good Tests" 3 date: 2023-11-29T09:41:18-05:00 4 draft: false 5 canonical_url: https://www.viget.com/articles/maintenance-matters-good-tests/ 6 references: 7 - title: "A year of Rails - macwright.com" 8 url: https://macwright.com/2021/02/18/a-year-of-rails.html 9 date: 2023-07-03T02:52:03Z 10 file: macwright-com-o4dndf.txt 11 --- 12 13 *This article is part of a series focusing on how developers can center 14 and streamline software maintenance. The other articles in the 15 Maintenance Matters series are: [Continuous 16 Integration](/elsewhere/maintenance-matters-continuous-integration/), 17 [Code 18 Coverage](https://www.viget.com/articles/maintenance-matters-code-coverage/), 19 [Documentation](https://www.viget.com/articles/maintenance-matters-documentation/), 20 [Default 21 Formatting](https://www.viget.com/articles/maintenance-matters-default-formatting/), [Building 22 Helpful 23 Logs](https://www.viget.com/articles/maintenance-matters-helpful-logs/), 24 [Timely 25 Upgrades](https://www.viget.com/articles/maintenance-matters-timely-upgrades/), 26 and [Code 27 Reviews](https://www.viget.com/articles/maintenance-matters-code-reviews/).* 28 29 In this latest entry to our [Maintenance 30 Matters](https://www.viget.com/articles/maintenance-matters/) series, I 31 want to talk about automated testing. Annie said it well in her intro 32 post: 33 34 > There is a lot to say about testing, but from a maintainer's 35 > perspective, let's define good tests as tests that prevent 36 > regressions. Unit tests should have clear expectations and fail when 37 > behavior changes, so a developer can either update the expectations or 38 > fix their code. Feature tests should pass when features work and break 39 > when features break. 40 41 This is a topic better suited to a book than a blog post (and indeed 42 [there are 43 many](https://bookshop.org/search?keywords=software+testing)), but I do 44 think there are a few high-level concepts that are important to 45 internalize in order to build robust, long-lasting software. 46 47 My first exposure to automated testing was with Ruby on Rails. Since 48 then, I've written production software in many different languages, but 49 nothing matches the Rails testing story. Tom MacWright said it well in 50 ["A year of 51 Rails"](https://macwright.com/2021/02/18/a-year-of-rails.html): 52 53 > Testing fully-server-rendered applications, on the other hand, is 54 > amazing. A vanilla testing setup with Rails & RSpec can give you fast, 55 > stable, concise, and actually-useful test coverage. You can actually 56 > assert for behavior and navigate through an application like a user 57 > would. These tests are solving a simpler problem - making requests and 58 > parsing responses, without the need for a full browser or headless 59 > browser, without multiple kinds of state to track. 60 61 Partly, I think Rails testing is so good because it's baked into the 62 framework: run `rails generate` to create a new model or controller and 63 the relevant test files are generated automatically. This helped 64 establish a community focus on testing, which led to a robust 65 third-party ecosystem around it. Additionally, Ruby is such a flexible 66 language that automated testing is really the only viable way to ensure 67 things are working as expected. 68 69 This post isn't about Rails testing specifically, but I wanted to be 70 clear on my perspective before we really dive in. And with that out of 71 the way, here's what we'll cover: 72 73 1. [Why Test?](#why-test) 74 2. [Types of Tests](#types-of-tests) 75 3. [Network Calls](#network-calls) 76 4. [Flaky Tests](#flaky-tests) 77 5. [Slow Tests](#slow-tests) 78 6. [App Code vs. Test Code](#app-code-vs-test-code) 79 80 ------------------------------------------------------------------------ 81 82 ### Why Test? 83 84 The single most important reason to make automated testing part of your 85 development process is that it **gives you confidence to make changes**. 86 This gets more and more important over time. With a reliable test suite 87 in place, you can refactor code, change functionality, and make upgrades 88 with reasonable certainty that you haven't broken anything. Without good 89 tests ... good luck. 90 91 Secondarily, testing: 92 93 - helps during the development process (testable code is correlated 94 with well-factored code, and it's a good way to review your work 95 before you ship it off); 96 - provides a guide to code reviewers; and 97 - serves as a kind of documentation (though not a particularly concise 98 one, and not as a replacement for proper written docs). 99 100 ### Types of Tests 101 102 I write two main kinds of tests, which I call **unit tests** and 103 **integration tests**, though my definitions differ slightly from the 104 original meanings. 105 106 - **Unit tests** call application code directly -- instantiate an 107 object, call a method on it, make assertions about the result. I 108 don't particularly care what the object under test does in the 109 course of doing its work -- calling off to other objects, performing 110 I/O, etc. (this is where I differ from the official definition). 111 - **Integration tests** test the entire system end-to-end, using a 112 framework like [Capybara](https://teamcapybara.github.io/capybara/) 113 or [Playwright](https://playwright.dev/). We sometimes refer to 114 these as "feature" tests in our codebases. 115 116 End-to-end, black-box integration tests are absolutely critical and can 117 cover most of your application's functionality by themselves. But it 118 often makes sense to wrap complex logic in a module, test that directly 119 (this is where [test-driven 120 development](https://en.wikipedia.org/wiki/Test-driven_development) can 121 come into play), and then write a simple integration test to ensure that 122 the module is getting called correctly. I avoid [mocking and 123 stubbing](https://en.wikipedia.org/wiki/Mock_object) if at all possible 124 -- again, "tests should pass when features work and break when features 125 break" -- and really only reach for it when it's the only option to hit 126 100% [code 127 coverage](https://www.viget.com/articles/maintenance-matters-code-coverage/). 128 In all cases, each test case should run against an empty database to 129 avoid ordering issues. 130 131 ### Network Calls 132 133 One important exception to the "avoid mocking" rule is third-party APIs: 134 your test suite should be entirely self-contained and shouldn't call out 135 to outside services. We use 136 [webmock](https://github.com/bblimke/webmock#real-requests-to-network-can-be-allowed-or-disabled) 137 in our Ruby apps to block access to the wider web entirely. Some 138 providers offer mock services that provide API-conformant responses you 139 can test against 140 (e.g., [stripe-mock](https://github.com/stripe/stripe-mock)). If that's 141 not an option, you can use something like 142 [VCR](https://github.com/vcr/vcr), which stores network responses as 143 files and returns cached values on subsequent calls. Beware, though: VCR 144 works impressively in small doses, but you can lose a lot of time 145 re-recording "cassettes" over time. 146 147 Rather than leaning on VCR, I've instead adopted the following approach: 148 149 1. Wrap the API integration into a standalone object/module 150 2. Create a second stub module with the same interface for use in tests 151 3. Create a [JSON Schema](https://json-schema.org/) that defines the 152 acceptable API responses 153 4. Use that schema to validate what comes back from your API modules 154 (both the real one and the stub) 155 156 If ever the responses coming from the real API fail to match the schema, 157 that indicates that your app and your tests have fallen out of sync, and 158 you need to update both. 159 160 ### Flaky Tests 161 162 Flaky tests (tests that fail intermittently, or only fail under certain 163 conditions) are bad. They eat up a lot of development time, especially 164 as build times increase. It's important to stay on top of them and 165 squash them as they arise. A single test that fails one time in five 166 maybe doesn't seem so bad, and it's easier to rerun the build than spend 167 time tracking it down. But five tests like that mean the build is 168 failing two-thirds of the time. 169 170 Some frameworks have libraries that will retry a failing test a set 171 number of times before giving up 172 (e.g., [rspec-retry](https://github.com/NoRedInk/rspec-retry), 173 [pytest-rerunfailures](https://pypi.org/project/pytest-rerunfailures/)). 174 These can be helpful, but they're a bandage, not a cure. 175 176 ### Slow Tests 177 178 The speed of your test suite is a much lower priority than the 179 performance of your application. All else being equal, faster is better, 180 but a slow test suite that fully exercises your application is vastly 181 preferable to a fast one that doesn't. Time spent performance-tuning 182 your tests can generally be better spent on other things. That said, it 183 *is* worth periodically looking for low-hanging speed-ups -- if 184 parallelizing your test runs cuts the build time in half, that's worth a 185 few hours' time investment. 186 187 During local development, I'll often run a subset of tests, either by 188 invoking a test file or specific test case directly, or by using a 189 wildcard pattern[^1] to run all the relevant tests. Combining that with 190 running the full suite in 191 [CI](/elsewhere/maintenance-matters-continuous-integration/) 192 provides a good balance of flow and rigor. At some point, if your test 193 suite is getting so slow that it's meaningfully impacting your team's 194 work, it's probably a sign that your app has gotten too large and needs 195 to be broken up into multiple discrete services. 196 197 ### App Code vs. Test Code 198 199 Tests are code, but they're not application code, and the way you 200 approach them should be slightly different. Some (or even a lot of) 201 repetition is OK; don't be too quick to refactor. Ideally, someone can 202 get a sense of what a test is doing by looking at a single screen of 203 code, as opposed to jumping around between early setup, shared examples, 204 complex factories with side-effects, etc. 205 206 I think of a test case sort of like a page in a book. I don't expect to 207 be able to open any random page in any random book and immediately grasp 208 the material, but assuming I'm otherwise familiar with the book's 209 content, I should be able to look at a single page and have a pretty 210 good sense of what's going on. A book that frequently required me to 211 jump to multiple other pages to understand a concept would not be a very 212 good book, and a test that spreads its setup across multiple other files 213 is not a very good test. 214 215 ------------------------------------------------------------------------ 216 217 Automated testing is a (perhaps **the**) critical component of 218 sustainable software development. It's not a replacement for human 219 testing, but with a reliable automated test suite in place, your testers 220 can focus on what's changed and not worry about regressions in other 221 parts of the system. It really doesn't add much time to the development 222 process (provided you know what you're doing), and any increase in 223 velocity you gain by forgoing testing is quickly erased by time spent 224 fixing bugs. 225 226 [^1]: For example, if I'm working on the part of the system that deals with sending email, I'll run all the tests with `mail` in the filename with `rspec spec/{models,features,lib}/**/*mail*`.