Elasticsearch 8 turns security on and removes types

Elasticsearch 8.0 shipped on the tenth of February. The query API is almost unchanged, types are finally gone four years after their deprecation, and the thing that actually breaks every environment is that security is now on by default — the cluster generates a certificate authority, enables TLS and refuses unauthenticated requests before anybody has configured anything.

The symptom

$ docker compose up -d elasticsearch
$ curl http://localhost:9200
curl: (52) Empty reply from server

$ docker compose logs elasticsearch | tail -12
✅ Elasticsearch security features have been automatically
   configured!
✅ Authentication is enabled and cluster connections are
   encrypted.

  Password for the elastic user: kL9x-mQ2p8Zt...
  HTTP CA certificate SHA-256 fingerprint:
    9c1f4a7e3b2d4f81a6e011d0c8b3f204...

# it is https, it wants credentials, and the CA is
# self-signed and generated at first boot.

Every development environment, every integration test and every internal tool that connected over plain HTTP with no credentials stops working simultaneously. The password is printed once, in the container log, on first boot — which is a container log nobody keeps.

Why it happens

Unsecured Elasticsearch clusters exposed to the internet were a recurring news story for most of a decade, and the fix that works is not documentation — it is making the secure configuration the one you get by default. That is the right decision and it lands as a breaking change in every environment at once.

The fix

Development, where the honest answer is to disable it

services:
  elasticsearch:
    image: elasticsearch:8.6.2
    environment:
      discovery.type: single-node
      xpack.security.enabled: 'false'      # development only
      ES_JAVA_OPTS: '-Xms512m -Xmx512m'
    ports: ['9200:9200']

# and the comment that has to be next to it:
#   this file is development only. the production stack is
#   in deploy/elasticsearch.yml and has security enabled.
#   see ENG-3211.

Disabling it in development is defensible and is what most teams did; disabling it in production is the configuration that produces the news story. The two configurations diverging is exactly the staging-lies problem, so the compromise that worked was security enabled everywhere except developer laptops, with the difference documented in the environment diff report.

Everywhere else, where it stays on

$client = ClientBuilder::create()
    ->setHosts(['https://es-01.internal:9200'])
    ->setBasicAuthentication('app', $secrets->get('es_password'))
    ->setCABundle('/etc/ssl/certs/es-ca.crt')
    ->build();

// or, pinning the fingerprint rather than distributing a CA
->setSSLVerification(true)
->setCABundle($caPath)

// service accounts, which are better than a shared user:
// POST /_security/service/elastic/fleet-server/credential/token

Distributing the generated CA to every client is the operational work, and it is the same problem as any internal certificate authority — a file that has to reach every host and be rotated. Using an existing internal CA rather than the auto-generated one is more setup and removes a distribution problem that will otherwise recur at every renewal.

The API keys and service accounts are worth adopting rather than a shared basic-auth user: an API key can be scoped to specific indices and privileges, can be revoked individually, and appears in the audit log with its own identity.

Types, finally

# 6.0: deprecated
# 7.0: one type per index, and include_type_name to opt out
# 8.0: gone. any request naming a type is rejected.

$ grep -rn '_type|include_type_name|"type" =>' src/ | wc -l
0        # the application was already correct

# where they were still hiding:
#   a saved search template from 2017
#   a Kibana saved object
#   a monthly report script nobody had opened

The four-year deprecation runway means most application code was corrected long ago, and the survivors are in the places nobody greps — a stored template, a saved object, a script that runs monthly. Auditing those is a different exercise from auditing the codebase and is the one that gets skipped.

The client that refuses to talk to the wrong version

# the 8.x PHP client against a 7.17 cluster:
#   "The client noticed that the server is not a supported
#    distribution of Elasticsearch"

# so the order is fixed and there are no shortcuts:
#   1. cluster 7.x → 7.17          (the bridge release)
#   2. resolve every deprecation warning it reports
#   3. cluster 7.17 → 8.x
#   4. THEN the client

$ curl -s localhost:9200/_migration/deprecations | jq -r 
    '.index_settings | to_entries[] | "(.key): (.value[0].message)"'

The deprecation API is the piece that makes step two tractable — it reports, per index and per cluster setting, what will break on the next major. Running it against 7.17 and clearing everything it lists is the actual upgrade work, and skipping it means discovering the same list one item at a time during the rolling restart.

What the removal of types actually changed in the mapping

Types were a modelling mistake that took six years to unwind, and the reason they were removed is worth stating because it explains a constraint people still hit: two types in one index shared a Lucene index, so a field called name in two types was one field with one mapping.

// what people thought they were doing, in 2015
PUT /shop/product/1   { "name": "Desk lamp", "price": 4900 }
PUT /shop/customer/1  { "name": "A Yildirim", "tier": "gold" }

// what was actually happening: one index, one 'name'
// field, one analyser. changing it for products changed
// it for customers.

// 8.x: one index per entity, which was always the answer
PUT /products/_doc/1  { "name": "Desk lamp", "price": 4900 }
PUT /customers/_doc/1 { "name": "A Yildirim", "tier": "gold" }

An index per entity is more indices to manage and it is the arrangement that lets each one have its own mapping, its own shard count and its own lifecycle. On this cluster it turned one index with four types into four indices, three of which turned out to be small enough to have a single shard where the shared index had five.

The join field is the replacement for the parent-child relationship that types were sometimes used for, and it is worth knowing that it is a different mechanism with different performance characteristics rather than a rename. Documents in a join relationship must live on the same shard, which constrains routing in a way the type-based version did not.

The queries that behaved differently

The API being unchanged is a claim about syntax rather than about results, and the replay was worth doing because two of five hundred queries returned a different set.

4,102 captured production queries, replayed against both:

  identical set and order      4,098
  same set, different order        3
  different set                    1

the three:  aggregation buckets with equal doc counts,
            where the tiebreak differs. harmless, and
            worth knowing about before somebody reports
            it as a bug.

the one:    a query using a deprecated parameter that 8.x
            removed. it had been logging a warning since
            7.4 and nobody had read the log.

That ratio is the usual outcome and is the argument for doing the replay rather than trusting the release notes — the single genuine difference was our own deprecated usage, not their change. Capturing the queries is the work: a week of the slow log at a threshold of zero produces a large file and a representative sample.

foreach ($this->capturedQueries() as $query) {
    $old = $this->legacy->search($query);
    $new = $this->candidate->search($query);

    $oldIds = array_column($old['hits']['hits'], '_id');
    $newIds = array_column($new['hits']['hits'], '_id');

    $this->record($query, [
        'same_set'   => $oldIds == $newIds,
        'same_order' => $oldIds === $newIds,
        'top_1'      => ($oldIds[0] ?? null) === ($newIds[0] ?? null),
    ]);
}

Comparing the set and the order separately is what makes the report readable, because an ordering difference in an aggregation is a footnote and a set difference is an incident. The top_1 column is the one a product owner cares about, since a relevance change that moves the first result is visible to every user of the search box.

The rolling upgrade, and the reindex that may be required

# an index created by 6.x cannot be read by 8.x, even if
# 7.x could read it. the compatibility window is one major.

$ curl -s localhost:9200/_all/_settings 
    | jq -r 'to_entries[] | "(.key) (.value.settings.index.version.created_string)"'
products-v3   6.8.23      ← must be reindexed before 8.x
orders-v2     7.10.2      ← fine

# the reindex, into an index created by 7.17:
POST /_reindex
{ "source": { "index": "products-v3" },
  "dest":   { "index": "products-v4" } }

The one-major compatibility window is the constraint that catches long-lived clusters: an index created in 2018, migrated through 7.x without being rebuilt, cannot be opened by 8.x at all. The version each index was created with is queryable and is worth checking before planning, because a reindex of a large index is the longest step in the whole upgrade.

Verifying it worked

$ curl -s --cacert es-ca.crt -u app:$PW https://es-01:9200 
    | jq -r '.version.number, .version.build_flavor'
8.6.2
default

$ ./bin/replay-queries --sample=500 --against=production
500 queries, 500 identical result sets

$ curl -s --cacert es-ca.crt -u app:$PW 
    'https://es-01:9200/_cat/indices?v&h=index,docs.count,version'

# and the negative test
$ curl -s http://es-01:9200
curl: (52) Empty reply from server        # correct

Replaying real captured queries against both clusters and comparing result sets is the check that matters, because the API being unchanged is a claim rather than a guarantee — analysis chain defaults and scoring have both moved between majors before. Five hundred identical result sets is a stronger statement than a passing smoke test.

What this costs

TLS in every environment that keeps security enabled, which means a certificate authority to distribute, rotate and eventually renew. That is a genuine ongoing cost and it is the cost of not being the next unsecured cluster in a news story, which is a trade worth making explicitly rather than by default.

The development divergence is the sharper edge. A stack with security disabled locally and enabled everywhere else means the authentication path is never exercised until staging, and the first time a client library is misconfigured is the first time it matters. Running one integration test against a secured container — slow, awkward, and the only thing that covers it — is the compromise that caught two configuration errors before they reached staging.