index.md (8390B)
1 --- 2 title: "Pandoc: A Tool I Use and Like" 3 date: 2022-05-25T00:00:00+00:00 4 draft: false 5 canonical_url: https://www.viget.com/articles/pandoc-a-tool-i-use-and-like/ 6 --- 7 8 Today I want to talk to you about one of my favorite command-line tools, 9 [Pandoc](https://pandoc.org/). From the project website: 10 11 > If you need to convert files from one markup format into another, 12 > pandoc is your swiss-army knife. 13 14 I spend a lot of time writing, and I love [Vim](https://www.vim.org/), 15 [Markdown](https://daringfireball.net/projects/markdown/), and the 16 command line (and avoid browser-based WYSIWYG editors when I can), so 17 that's where a lot of my Pandoc use comes in, but it has a ton of 18 utility outside of that -- really, anywhere you need to move between 19 different text-based formats, Pandoc can probably help. A few examples 20 from recent memory: 21 22 ### Markdown â Craft Blog Post 23 24 This website you're reading presently uses [Craft 25 CMS](https://craftcms.com/), a flexible and powerful content management 26 system that doesn't perfectly match my writing 27 process[^1]. Rather 28 than composing directly in Craft, I prefer to write locally, pipe the 29 output through Pandoc, and put the resulting HTML into a text block in 30 the CMS. This gets me a few things I really like: 31 32 - Curly quotes in place of straight ones and en-dashes in place of 33 `--` (from the [`smart` 34 extension](https://pandoc.org/MANUAL.html#extension-smart)) 35 - [Daring 36 Fireball-style](https://daringfireball.net/2005/07/footnotes) 37 footnotes with return links 38 39 By default, Pandoc uses [Pandoc 40 Markdown](https://garrettgman.github.io/rmarkdown/authoring_pandoc_markdown.html) 41 when converting Markdown docs to other formats, an "extended and 42 slightly revised version" of the original syntax, which is how footnotes 43 and a bunch of other things work. 44 45 ### Markdown â Rich Text (Basecamp) 46 47 I also sometimes find myself writing decently long 48 [Basecamp](https://basecamp.com/) posts. Basecamp 3 has a fine WYSIWYG 49 editor (ðŠĶ Textile), but again, I'd rather be in Vim. Pasting HTML into 50 Basecamp doesn't work (just shows the code verbatim), but I've found 51 that if I convert my Markdown notes to HTML and open the HTML in a 52 browser, I can copy and paste that directly into Basecamp with good 53 results. Leveraging MacOS' `open` command, this one-liner does the 54 trick[^2]: 55 56 ```sh 57 cat [filename.md] \ 58 | pandoc -t html \ 59 > /tmp/output.html \ 60 && open /tmp/output.html \ 61 && read -n 1 \ 62 && rm /tmp/output.html 63 ``` 64 65 This will convert the contents to HTML, save that to a file, open the 66 file in a browser, wait for the user to hit enter, and the remove the 67 file. Without that `read -n 1`, it'll remove the file before the browser 68 has a chance to open it. 69 70 ### HTML â Text 71 72 We built an app for one of our clients that takes in news articles (in 73 HTML) via an API and sends them as emails to *their* clients (think big 74 brands) if certain criteria are met. Recently, we were making 75 improvements to the plain text version of the emails, and we noticed 76 that some of the articles were coming in without any linebreaks in the 77 content. When we removed the HTML (via Rails' [`strip_tags` 78 helper](https://apidock.com/rails/ActionView/Helpers/SanitizeHelper/strip_tags)), 79 the resulting content was all on one line, which wasn't very readable. 80 So imagine an article like this: 81 82 ```html 83 <h1>Headline</h1> <p>A paragraph.</p> <ul><li>List item #1</li> <li>List item #2</li></ul> 84 ``` 85 86 Our initial approach (with `strip_tags`) gives us this: 87 88 ``` 89 Headline A paragraph. List item #1 List item #2 90 ``` 91 92 Not great! But fortunately, some bright fellow had the idea to pull in 93 Pandoc, and some even brighter person packaged up some [Ruby 94 bindings](https://github.com/xwmx/pandoc-ruby) for it. Taking that same 95 content and running it through `PandocRuby.html(content).to_plain` gives 96 us: 97 98 ``` 99 Headline 100 101 A paragraph. 102 103 - List item #1 104 - List item #2 105 ``` 106 107 Much better, and though you can't tell from this basic example, Pandoc 108 does a great job with spacing and wrapping to generate really 109 nice-looking plain text from HTML. 110 111 ### HTML Element â Text 112 113 A few months ago, we were doing Pointless Weekend and needed a domain 114 for our 115 [Thrillr](https://www.viget.com/articles/plan-a-killer-party-with-thrillr/) 116 app. A few of us were looking through lists of fun top-level domains, 117 but we realized that AWS Route 53 only supports a limited set of them. 118 In order to get everyone the actual list, I needed a way to get all the 119 content out of an HTML `<select>` element, and you'll never guess what I 120 did (unless you guessed "use Pandoc"). In Firefox: 121 122 - Right click the select element, then click "Inspect" 123 - Find the `<select>` in the DOM view that pops up 124 - Right click it, then go to "Copy", then "Inner HTML" 125 - You'll now have all of the `<option>` elements on your clipboard 126 - In your terminal, run `pbpaste | pandoc -t plain` 127 128 The result is something like this: 129 130 ``` 131 .ac - $76.00 132 .academy - $12.00 133 .accountants - $94.00 134 .agency - $19.00 135 .apartments - $47.00 136 .associates - $29.00 137 .au - $15.00 138 .auction - $29.00 139 ... 140 ``` 141 142 ### Preview Mermaid/Markdown (`--standalone`) 143 144 A different client recently asked for an architecture diagram of a 145 complex system that [Andrew](https://www.viget.com/about/team/athomas/) 146 and I were working on, and we opted to use 147 [Mermaid](https://mermaid-js.github.io/mermaid/#/) (which is rad BTW) to 148 create sequence diagrams to illustrate all of the interactions. Both 149 GitHub and GitLab support Mermaid natively, which is really neat, but we 150 wanted a way to quickly iterate on our diagrams without having to push 151 changes to the remote repo. 152 153 We devised a simple build chain ([demo version available 154 here](https://github.com/dce/mermaid-js-demo)) that watches for changes 155 to a Markdown file, converts the Mermaid blocks to SVG, and then uses 156 Pandoc to take the resulting document and convert it to a styled HTML 157 page using the `--standalone` option ([here's the key 158 line](https://github.com/dce/mermaid-js-demo/blob/main/bin/build#L7=)). 159 Then we could simply make our changes and refresh the page to see our 160 progress. 161 162 ### Generate a PDF 163 164 Finally, and this is not something I need to do very often, but Pandoc 165 also includes several ways to create PDF documents. The simplest (IMO) 166 is to install `wkhtmltopdf`, then instruct Pandoc to convert its input 167 to HTML but use `.pdf` in the output filename, so something like: 168 169 ```sh 170 echo "# Hello\n\nIs it me you're looking for?" \ 171 | pandoc -t html -o hello.pdf 172 ``` 173 174 [The result is quite nice.](hello.pdf) 175 176 ------------------------------------------------------------------------ 177 178 I think that's about all I have to say about Pandoc for today. A couple 179 final thoughts: 180 181 - Pandoc is incredibly powerful -- I've really only scratched the 182 surface here. Look at the [man page](https://manpages.org/pandoc) to 183 get a sense of everything it can do. 184 - Pandoc is written in Haskell, and [the 185 source](https://github.com/jgm/pandoc/blob/master/src/Text/Pandoc/Readers/Markdown.hs) 186 is pretty fun to look through if you're a certain kind of person. 187 188 So install Pandoc with your package manager of choice and give it a 189 shot. I think you'll find it unexpectedly useful. 190 191 *[Swiss army knife icons created by smalllikeart - 192 Flaticon](https://www.flaticon.com/free-icons/swiss-army-knife "swiss army knife icons")* 193 194 [^1]: My writing process is (generally): 195 196 1. Write down an idea in my notebook 197 2. Gradually add a series of bullet points (this can sometimes take 198 awhile) 199 3. Once I feel like I have a solid outline, copy that into a 200 Markdown file 201 4. Start collecting links (in the `[1]:` footnote style) 202 5. Write a intro 203 6. Convert the bullet points to headers, edit + rearrange 204 7. Fill in all the sections, write jokes, etc. 205 8. Write a conclusion 206 9. Create a Gist, get feedback from the team 207 10. Convert Markdown to HTML, copy to clipboard 208 (`cat [file] | pandoc -t html | pbcopy`) 209 11. Create a new post in Craft, add a text section, flip to code 210 view, paste clipboard contents 211 12. Fill in the rest of the post metadata 212 13. ðĒ 213 214 [^2]: I've actually got this wired up as a Vim command in `.vimrc`: 215 216 ```vim 217 command Mdpreview ! cat % 218 \ | pandoc -t html 219 \ > /tmp/output.html 220 \ && open /tmp/output.html 221 \ && read -n 1 222 \ && rm /tmp/output.html 223 ```