index.md (4234B)
1 --- 2 title: "First-Class Failure" 3 date: 2014-07-22T00:00:00+00:00 4 draft: false 5 canonical_url: https://www.viget.com/articles/first-class-failure/ 6 --- 7 8 As a developer, nothing makes me more nervous than third-party 9 dependencies and things that can fail in unpredictable 10 ways[^1]. More often 11 than not, these two go hand-in-hand, taking our elegant, robust 12 applications and dragging them down to the lowest common denominator of 13 the services they depend upon. A recent internal project called for 14 slurping in and then reporting against data from 15 [Harvest](http://www.getharvest.com/), our time tracking service of 16 choice and a fickle beast on its very best days. 17 18 I knew that both components (`/(im|re)porting/`) were prone to failure. 19 How to handle that failure in a graceful way, so that our users see 20 something more meaningful than a 500 page, and our developers have a 21 fighting chance at tracking and fixing the problem? Here's the approach 22 we took. 23 24 ## Step 1: Model the processes 25 26 Rather than importing the data or generating the report with procedural 27 code, create ActiveRecord models for them. In our case, the models are 28 `HarvestImport` and `Report`. When a user initiates a data import or a 29 report generation, save a new record to the database *immediately*, 30 before doing any work. 31 32 ## Step 2: Give 'em status 33 34 These models have a `status` column. We default it to "queued," since we 35 offload most of the work to a series of [Resque](http://resquework.org/) 36 tasks, but you can use "pending" or somesuch if that's more your speed. 37 They also have an `error` field for reasons that will become apparent 38 shortly. 39 40 ## Step 3: Define an interface 41 42 Into both of these models, we include the following module: 43 44 ```ruby 45 module ProcessingStatus 46 def mark_processing 47 update_attributes(status: "processing") 48 end 49 50 def mark_successful 51 update_attributes(status: "success", error: nil) 52 end 53 54 def mark_failure(error) 55 update_attributes(status: "failed", error: error.to_s) 56 end 57 58 def process(cleanup = nil) 59 mark_processing 60 yield 61 mark_successful 62 rescue => ex 63 mark_failure(ex) 64 ensure 65 cleanup.try(:call) 66 end 67 end 68 ``` 69 70 Lines 2--12 should be self-explanatory: methods for setting the object's 71 status. The `mark_failure` method takes an exception object, which it 72 stores in the model's `error` field, and `mark_successful` clears said 73 error. 74 75 Line 14 (the `process` method) is where things get interesting. Calling 76 this method immediately marks the object "processing," and then yields 77 to the provided block. If the block executes without error, the object 78 is marked "success." If any[^2] exception is thrown, the object marked "failure" and the 79 error message is logged. Either way, if a `cleanup` lambda is provided, 80 we call it (courtesy of Ruby's 81 [`ensure`](http://ruby.activeventure.com/usersguide/rg/ensure.html) 82 keyword). 83 84 ## Step 4: Wrap it up 85 86 Now we can wrap our nasty, fail-prone reporting code in a `process` call 87 for great justice. 88 89 ```ruby 90 class ReportGenerator 91 attr_accessor :report 92 93 def generate_report 94 report.process -> { File.delete(file_path) } do 95 # do some fail-prone work 96 end 97 end 98 99 # ... 100 end 101 ``` 102 103 The benefits are almost too numerous to count: 1) no 500 pages, 2) 104 meaningful feedback for users, and 3) super detailed diagnostic info for 105 developers -- better than something like 106 [Honeybadger](https://www.honeybadger.io/), which doesn't provide nearly 107 the same level of context. (`-> { File.delete(file_path) }` is just a 108 little bit of file cleanup that should happen regardless of outcome.) 109 110 *** 111 112 I've always found it an exercise in futility to try to predict all the 113 ways a system can fail when integrating with an external dependency. 114 Being able to blanket rescue any exception and store it in a way that's 115 meaningful to users *and* developers has been hugely liberating and has 116 contributed to a seriously robust platform. This technique may not be 117 applicable in every case, but when it fits, [it's 118 good](https://www.youtube.com/watch?v=HNfciDzZTNM&t=1m40s). 119 120 [^1]: Well, [almost nothing](https://github.com/github/hubot/blob/master/src/scripts/google-images.coffee#L5). 121 [^2]: [Any descendent of `StandardError`](http://stackoverflow.com/a/10048406), in any event.