Who should read this?
- Do you need to create an index pattern in Kibana and mark it as the default programmatically (so that your users aren’t faced with an extraneous choice)?
- Have you been tasked with provisioning canned queries in Kibana?
- Did an Amazon Elasticsearch Service version upgrade cause your default index patterns to stop defaulting? In other words, did an unofficial method of defaulting an index pattern stop functioning correctly?
- Are you using Amazon Elasticsearch Service version 6.3 or newer?
- Are you scouring the web in vain, copying-and-pasting questionable curl commands with nothing to show for it?
We’ve got you.
Before we proceed, let’s review the nature of the problem.
Kibana and Index Patterns
For Amazon Elasticsearch Service users, Kibana is an invaluable plugin for exploring their cluster’s indices and mapped documents within.
A prerequisite for any searching within Kibana is the index pattern. This entity gives shape to Kibana queries, forming the target index (or indices), against which Kibana will perform its searches. The results from queries against the target(s) are made available for viewing in the web console.
Fresh clusters don’t have any index patterns defined
Because there aren’t any index patterns out-of-the-box, the user will be prompted to create one. Our plan is to automate the creation of this index pattern.
Index patterns don’t mark themselves as default
Even after you create an index, manually or otherwise, the pattern isn’t marked as default. Kibana will prompt the user to select a default index pattern. Ideally, we’ll automate this step as well. Herein lies the secret sauce of this how-to.
What we’re going to do
- Set up a request path to an Amazon Elasticsearch Service cluster with aws-es-kibana
- Create an index pattern
- Retrieve the newly created index pattern
- Use the index pattern as a template for newly provisioned clusters
- Set the index pattern as the default programmatically
Prerequisites
For this how-to, you’ll need curl or awscurl installed. Go ahead and do this now.
aws-es-kibana provides a convenient way to access your cluster on AWS. If you don’t have it already, you’ll need nodejs installed before you can install aws-es-kibana.
Last, but not least, create an Amazon Elasticsearch Service domain if you don’t already have one on hand.
- Log into the AWS Console.
- Select Elasticsearch Service.
- Select Create a new domain.
- Follow the wizard steps and wait a few minutes for the cluster to become available.
Starting aws-es-kibana
aws-es-kibana is a super handy proxy for interacting with Amazon Elasticsearch Service. You don’t have to worry about V4 signing your requests – this is done on your behalf using your configured AWS credentials. You might also notice that in our examples we use http and not https. Not to worry – the cluster is still secure. aws-es-kibana provides SSL termination so that when we request something in http, the request will be completed on our behalf using https. Additionally, the paths of the URIs are structured specifically for Amazon Elasticsearch Service; please adapt as necessary for your specifics.
- Follow the installation instructions for aws-es-kibana here.
- Be sure your AWS credentials are configured.
- Log in to the AWS Console and retrieve the host name of your Amazon Elasticsearch Service cluster.
- Select Elasticsearch Service
- Select your Amazon Elasticsearch Service domain
e.g. dev-cluster - Take note of the host name embedded within the Endpoint field.
e.g. dev-cluster-4c5si47uztlr5p4liw3xh63tia.us-east-1.es.amazonaws.com
- aws-es-kibana HOST_NAME
- Confirm you can access your cluster by curling or pointing with your browser.
curl http://localhost:9200/
You should see something like the following:
{
"name" : "I6Fnsqt",
"cluster_name" : "XXXXXXXXXXXX:dev-cluster",
"cluster_uuid" : "fbxnTSk9Q_-Y80jUWWYkdA",
"version" : {
"number" : "6.3.1",
"build_flavor" : "oss",
"build_type" : "zip",
"build_hash" : "eb782d0",
"build_date" : "2019-04-01T14:21:38.700602Z",
"build_snapshot" : false,
"lucene_version" : "7.3.1",
"minimum_wire_compatibility_version" : "5.6.0",
"minimum_index_compatibility_version" : "5.0.0"
},
"tagline" : "You Know, for Search"
}All set? Let’s move on.
Bring your own index pattern
Or skip to the next section – we’ll provide one for you to use.
Generate the index pattern
You can create your own index pattern by hand, but we’re going to rely on Kibana to generate one for us. Use the Kibana web console to create the index pattern first. You can use the wizard when you first login, or you can do it later using the following steps:
- Log in to the Kibana console.
http://YOUR_ENDPOINT/_plugin/kibana/app/kibana - Select Management.
- Select Index Patterns.
- Select Create Index Pattern.
- Type something into the Index Pattern text box. I’m using logging-*.
- Select Next step.
- Select a Time Filter field name.
- Select > Show advanced options.
- In Custom index pattern ID, enter something that makes sense to you. I’m using logging-all. If you don’t choose an id, one will be generated for you, which you’ll have to identify on your own later.
- Finally, select Create index pattern.
Now, we’ll get a hold of the JSON we need with curl. In order to do that, we need to discover the id of the object we’ve created.
Search for the index pattern
awscurl –service es -H ‘Content-Type: application/json; charset=utf-8’ -H ‘kbn-xsrf: true’ -XGET ‘http://YOUR_ENDPOINT/_plugin/kibana/api/saved_objects/_find’
The object you’re looking for has a key named type with a value of index-pattern. After you know the id, you can fetch its content directly. In this example, the id is logging-all.
Get a specific index pattern by id
awscurl –service es -H ‘Content-Type: application/json; charset=utf-8’ -H ‘kbn-xsrf: true’ -XGET ‘http://YOUR_ENDPOINT/_plugin/kibana/api/saved_objects/index-pattern/logging-all’
Before re-using the result by sending it up to Kibana, you’ll need to remove some attributes – id, type, updated_at, and version. Thankfully, Kibana will let you know which properties it doesn’t like if/when you attempt to POST your index pattern should you forget to alter it beforehand.
Transmit Index Pattern to Amazon Elasticsearch Service
Once you have your index pattern in hand, we’re going to use the officially supported Saved Objects API to install it in Kibana.
In case you don’t already have an index pattern cooked up, here’s an example for creating an index pattern targeting all indices with names beginning with logging- –
awscurl --service es -H 'Content-Type: application/json; charset=utf-8' -H 'kbn-xsrf: true' -XPOST 'http://YOUR_ENDPOINT/_plugin/kibana/api/saved_objects/index-pattern/logging-all?overwrite=true' -d '{
"attributes": {
"title": "logging-*",
"notExpandable": true,
"fields":
"[{\"name\":\"_id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_index\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"_score\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_source\",\"type\":\"_source\",\"count\":0,\"scripted\":false,\"searchable\":false,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"_type\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":false},{\"name\":\"logEvents.id\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"logEvents.id.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"logEvents.message\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"logEvents.message.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"logEvents.timestamp\",\"type\":\"number\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"logGroup\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"logGroup.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"logStream\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"logStream.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"messageType\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"messageType.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true},{\"name\":\"owner\",\"type\":\"string\",\"count\":2,\"scripted\":false,\"searchable\":true,\"aggregatable\":false,\"readFromDocValues\":false},{\"name\":\"owner.keyword\",\"type\":\"string\",\"count\":0,\"scripted\":false,\"searchable\":true,\"aggregatable\":true,\"readFromDocValues\":true}]"
}
}
'If you inspect the path, you’ll see we’re giving the index pattern an id of logging-all.
Where’s my default index pattern?
Unfortunately, the index pattern we created hasn’t been set as the default. This won’t happen automatically, even if it’s the only one present. If we were to use the Kibana web console at this moment, we’d be prompted to select a default.
We can use the web console to manually select this index pattern as our default. But that’s not going to fly anymore. This needs to be automated.
Let’s just quickly browse through some documentation.
Or maybe Google.
Maybe Google just a little while longer.
POST to .kibana/config/5.6.4 … How old is this forum post?
POST to api/kibana/settings/defaultIndex? Well, which path is it?
Why are none of these commands working?!
tl;dr Just Make It Default, Please
Sure thing!
awscurl --service es -H 'Content-Type: application/json;
charset=utf-8' -H 'kbn-xsrf: true' -XPOST
'http://YOUR_ENDPOINT/_plugin/kibana/api/saved_objects/index-pattern/logging-all?overwrite=true' -d '{ "changes": { "defaultIndex": "logging-all" } }'
That’s it.
What is that, you ask? It’s the same request that the Kibana web console uses to mark an index pattern as default. Remember to change the logging-all value to match the id of your index pattern. Though undocumented, it has been tested on Amazon Elasticsearch Service / Kibana 6.3. If it’s good enough for the Kibana web console, it’s good enough for me.
Armed with this technique, you can provision and configure your Amazon Elasticsearch Service clusters with reliability and confidence! Thanks for reading!
If you want assistance with Amazon Elasticsearch Service or have other automation challenges, get in touch!
