index.md (14346B)
1 --- 2 title: "Let’s Write a Dang ElasticSearch Plugin" 3 date: 2021-03-15T00:00:00+00:00 4 draft: false 5 canonical_url: https://www.viget.com/articles/lets-write-a-dang-elasticsearch-plugin/ 6 --- 7 8 One of our current projects involves a complex interactive query builder 9 to search a large collection of news items. Some of the conditionals 10 fall outside of the sweet spot of Postgres (e.g. word X must appear 11 within Y words of word Z), and so we opted to pull in 12 [ElasticSearch](https://www.elastic.co/elasticsearch/) alongside it. 13 It's worked perfectly, hitting all of our condition and grouping needs 14 with one exception: we need to be able to filter for articles that 15 contain a term a minimum number of times (so "Apple" must appear in 16 the article 3 times, for example). Frustratingly, Elastic *totally* has 17 this information via its 18 [`term_vector`](https://www.elastic.co/guide/en/elasticsearch/reference/current/term-vector.html) 19 feature, but you can't use that data inside a query, as least as far as 20 I can tell. 21 22 The solution, it seems, is to write a custom plugin. I figured it out, 23 eventually, but it was a lot of trial-and-error as the documentation I 24 was able to find is largely outdated or incomplete. So I figured I'd 25 take what I learned while it's still fresh in my mind in the hopes that 26 someone else might have an easier time of it. That's what internet 27 friends are for, after all. 28 29 Quick note before we start: all the version numbers you see are current 30 and working as of February 25, 2021. Hopefully this post ages well, but 31 if you try this out and hit issues, bumping the versions of Elastic, 32 Gradle, and maybe even Java is probably a good place to start. Also, I 33 use `projectname` a lot in the code examples --- that's not a special 34 word and you should change it to something that makes sense for you. 35 36 ## 1. Set up a Java development environment 37 38 First off, you're gonna be writing some Java. That's not my usual 39 thing, so the first step was to get a working environment to compile my 40 code. To do that, we'll use [Docker](https://www.docker.com/). Here's 41 a `Dockerfile`: 42 43 ```dockerfile 44 FROM adoptopenjdk/openjdk12:jdk-12.0.2_10-ubuntu 45 46 RUN apt-get update && 47 apt-get install -y zip unzip && 48 rm -rf /var/lib/apt/lists/* 49 50 SHELL ["/bin/bash", "-c"] 51 52 RUN curl -s "https://get.sdkman.io" | bash && 53 source "/root/.sdkman/bin/sdkman-init.sh" && 54 sdk install gradle 6.8.2 55 56 WORKDIR /plugin 57 ``` 58 59 We use a base image with all the Java stuff but also a working Ubuntu 60 install so that we can do normal Linux-y things inside our container. 61 From your terminal, build the image: 62 63 `> docker build . -t projectname-java` 64 65 Then, spin up the container and start an interactive shell, mounting 66 your local working directory into `/plugin`: 67 68 `> docker run --rm -it -v ${PWD}:/plugin projectname-java bash` 69 70 ## 2. Configure Gradle 71 72 [Gradle](https://gradle.org/) is a "build automation tool for 73 multi-language software development," and what Elastic recommends for 74 plugin development. Configuring Gradle to build the plugin properly was 75 the hardest part of this whole endeavor. Throw this into `build.gradle` 76 in your project root: 77 78 ```gradle 79 buildscript { 80 repositories { 81 mavenLocal() 82 mavenCentral() 83 jcenter() 84 } 85 86 dependencies { 87 classpath "org.elasticsearch.gradle:build-tools:7.11.1" 88 } 89 } 90 91 apply plugin: 'java' 92 93 compileJava { 94 sourceCompatibility = JavaVersion.VERSION_12 95 targetCompatibility = JavaVersion.VERSION_12 96 } 97 98 apply plugin: 'elasticsearch.esplugin' 99 100 group = "com.projectname" 101 version = "0.0.1" 102 103 esplugin { 104 name 'contains-multiple' 105 description 'A script for finding documents that match a term a certain number of times' 106 classname 'com.projectname.containsmultiple.ContainsMultiplePlugin' 107 licenseFile rootProject.file('LICENSE.txt') 108 noticeFile rootProject.file('NOTICE.txt') 109 } 110 111 validateNebulaPom.enabled = false 112 ``` 113 114 You'll also need files named `LICENSE.txt` and `NOTICE.txt` --- mine 115 are empty, since the plugin is for internal use only. If you're going 116 to be releasing your plugin in some public way, maybe talk to a lawyer 117 about what to put in those files. 118 119 ## 3. Write the dang plugin 120 121 To write the actual plugin, I started with [this example 122 plugin](https://github.com/elastic/elasticsearch/blob/master/plugins/examples/script-expert-scoring/src/main/java/org/elasticsearch/example/expertscript/ExpertScriptPlugin.java) 123 which scores a document based on the frequency of a given term. My use 124 case was fortunately quite similar, though I'm using a `filter` query, 125 meaning I just want a boolean, i.e. does this document contain this term 126 the requisite number of times? As such, I implemented a 127 [`FilterScript`](https://www.javadoc.io/doc/org.elasticsearch/elasticsearch/latest/org/elasticsearch/script/FilterScript.html) 128 rather than the `ScoreScript` implemented in the example code. 129 130 This file lives in (deep breath) 131 `src/main/java/com/projectname/` `containsmultiple/ContainsMultiplePlugin.java`: 132 133 ```java 134 package com.projectname.containsmultiple; 135 136 import org.apache.lucene.index.LeafReaderContext; 137 import org.apache.lucene.index.PostingsEnum; 138 import org.apache.lucene.index.Term; 139 import org.elasticsearch.common.settings.Settings; 140 import org.elasticsearch.plugins.Plugin; 141 import org.elasticsearch.plugins.ScriptPlugin; 142 import org.elasticsearch.script.FilterScript; 143 import org.elasticsearch.script.FilterScript.LeafFactory; 144 import org.elasticsearch.script.ScriptContext; 145 import org.elasticsearch.script.ScriptEngine; 146 import org.elasticsearch.script.ScriptFactory; 147 import org.elasticsearch.search.lookup.SearchLookup; 148 149 import java.io.IOException; 150 import java.io.UncheckedIOException; 151 import java.util.Collection; 152 import java.util.Map; 153 import java.util.Set; 154 155 /** 156 * A script for finding documents that match a term a certain number of times 157 */ 158 public class ContainsMultiplePlugin extends Plugin implements ScriptPlugin { 159 160 @Override 161 public ScriptEngine getScriptEngine( 162 Settings settings, 163 Collection<ScriptContext<?>> contexts 164 ) { 165 return new ContainsMultipleEngine(); 166 } 167 168 // tag::contains_multiple 169 private static class ContainsMultipleEngine implements ScriptEngine { 170 @Override 171 public String getType() { 172 return "expert_scripts"; 173 } 174 175 @Override 176 public <T> T compile( 177 String scriptName, 178 String scriptSource, 179 ScriptContext<T> context, 180 Map<String, String> params 181 ) { 182 if (context.equals(FilterScript.CONTEXT) == false) { 183 throw new IllegalArgumentException(getType() 184 + " scripts cannot be used for context [" 185 + context.name + "]"); 186 } 187 // we use the script "source" as the script identifier 188 if ("contains_multiple".equals(scriptSource)) { 189 FilterScript.Factory factory = new ContainsMultipleFactory(); 190 return context.factoryClazz.cast(factory); 191 } 192 throw new IllegalArgumentException("Unknown script name " 193 + scriptSource); 194 } 195 196 @Override 197 public void close() { 198 // optionally close resources 199 } 200 201 @Override 202 public Set<ScriptContext<?>> getSupportedContexts() { 203 return Set.of(FilterScript.CONTEXT); 204 } 205 206 private static class ContainsMultipleFactory implements FilterScript.Factory, 207 ScriptFactory { 208 @Override 209 public boolean isResultDeterministic() { 210 return true; 211 } 212 213 @Override 214 public LeafFactory newFactory( 215 Map<String, Object> params, 216 SearchLookup lookup 217 ) { 218 return new ContainsMultipleLeafFactory(params, lookup); 219 } 220 } 221 222 private static class ContainsMultipleLeafFactory implements LeafFactory { 223 private final Map<String, Object> params; 224 private final SearchLookup lookup; 225 private final String field; 226 private final String term; 227 private final int count; 228 229 private ContainsMultipleLeafFactory( 230 Map<String, Object> params, SearchLookup lookup) { 231 if (params.containsKey("field") == false) { 232 throw new IllegalArgumentException( 233 "Missing parameter [field]"); 234 } 235 if (params.containsKey("term") == false) { 236 throw new IllegalArgumentException( 237 "Missing parameter [term]"); 238 } 239 if (params.containsKey("count") == false) { 240 throw new IllegalArgumentException( 241 "Missing parameter [count]"); 242 } 243 this.params = params; 244 this.lookup = lookup; 245 field = params.get("field").toString(); 246 term = params.get("term").toString(); 247 count = Integer.parseInt(params.get("count").toString()); 248 } 249 250 @Override 251 public FilterScript newInstance(LeafReaderContext context) 252 throws IOException { 253 PostingsEnum postings = context.reader().postings( 254 new Term(field, term)); 255 if (postings == null) { 256 /* 257 * the field and/or term don't exist in this segment, 258 * so always return 0 259 */ 260 return new FilterScript(params, lookup, context) { 261 @Override 262 public boolean execute() { 263 return false; 264 } 265 }; 266 } 267 return new FilterScript(params, lookup, context) { 268 int currentDocid = -1; 269 @Override 270 public void setDocument(int docid) { 271 /* 272 * advance has undefined behavior calling with 273 * a docid <= its current docid 274 */ 275 if (postings.docID() < docid) { 276 try { 277 postings.advance(docid); 278 } catch (IOException e) { 279 throw new UncheckedIOException(e); 280 } 281 } 282 currentDocid = docid; 283 } 284 @Override 285 public boolean execute() { 286 if (postings.docID() != currentDocid) { 287 /* 288 * advance moved past the current doc, so this 289 * doc has no occurrences of the term 290 */ 291 return false; 292 } 293 try { 294 return postings.freq() >= count; 295 } catch (IOException e) { 296 throw new UncheckedIOException(e); 297 } 298 } 299 }; 300 } 301 } 302 } 303 // end::contains_multiple 304 } 305 ``` 306 307 ## 4. Add it to ElasticSearch 308 309 With our code in place (and synced into our Docker container with a 310 mounted volume), it's time to compile it. In the Docker shell you 311 started up in step #1, build your plugin: 312 313 `> gradle build` 314 315 Assuming that works, you should now see a `build` directory with a bunch 316 of stuff in it. The file you care about is 317 `build/distributions/contains-multiple-0.0.1.zip` (though that'll 318 obviously change if you call your plugin something different or give it 319 a different version number). Grab that file and copy it to where you 320 plan to actually run ElasticSearch. For me, I placed it in a folder 321 called `.docker/elastic` in the main project repo. In that same 322 directory, create a new `Dockerfile` that'll actually run Elastic: 323 324 ```dockerfile 325 FROM docker.elastic.co/elasticsearch/elasticsearch:7.11.1 326 327 COPY .docker/elastic/contains-multiple-0.0.1.zip /plugins/contains-multiple-0.0.1.zip 328 329 RUN elasticsearch-plugin install 330 file:///plugins/contains-multiple-0.0.1.zip 331 ``` 332 333 Then, in your project root, create the following `docker-compose.yml`: 334 335 ```yaml 336 version: '3.2' 337 338 services: elasticsearch: 339 image: projectname_elasticsearch 340 build: 341 context: . 342 dockerfile: ./.docker/elastic/Dockerfile 343 ports: 344 - 9200:9200 345 environment: 346 - discovery.type=single-node 347 - script.allowed_types=inline 348 - script.allowed_contexts=filter 349 ``` 350 351 Those last couple lines are pretty important and your script won't work 352 without them. Build your image with `docker-compose build` and then 353 start Elastic with `docker-compose up`. 354 355 ## 5. Use your plugin 356 357 To actually see the plugin in action, first create an index and add some 358 documents (I'll assume you're able to do this if you've read this far 359 into this post). Then, make a query with `curl` (or your Elastic wrapper 360 of choice), substituting `full_text`, `yabba` and `index_name` with 361 whatever makes sense for you: 362 363 ``` 364 > curl -H "content-type: application/json" 365 -d ' 366 { 367 "query": { 368 "bool": { 369 "filter": { 370 "script": { 371 "script": { 372 "source": "contains_multiple", 373 "lang": "expert_scripts", 374 "params": { 375 "field": "full_text", 376 "term": "yabba", 377 "count": 3 378 } 379 } 380 } 381 } 382 } 383 } 384 }' 385 "localhost:9200/index_name/_search?pretty" 386 ``` 387 388 The result should be something like: 389 390 ```json 391 { 392 "took" : 6, 393 "timed_out" : false, 394 "_shards" : { 395 "total" : 1, 396 "successful" : 1, 397 "skipped" : 0, 398 "failed" : 0 399 }, 400 "hits" : { 401 "total" : { 402 "value" : 1, 403 "relation" : "eq" 404 }, 405 "max_score" : 0.0, 406 "hits" : [ 407 { 408 "_index" : "index_name", 409 "_type" : "_doc", 410 "_id" : "10", 411 ... 412 ``` 413 414 So that's that, an ElasticSearch plugin from start-to-finish. I'm sure 415 there are better ways to do some of this stuff, and if you're aware of 416 any, let us know in the comments or write your own dang blog.