On the fourteenth of January the search engine we had been running since 2017 stopped being Apache-licensed. Nothing broke, nothing needed patching that afternoon, and the version we were running was as legal on the fifteenth as it had been on the thirteenth. What changed was that the next upgrade became a decision somebody outside engineering had to be involved in.
The symptom
$ curl -s localhost:9200 | jq -r '.version.number'
7.9.3
$ curl -s https://artifacts.elastic.co/... | jq -r '.[] | .version'
7.10.2 # Apache 2.0
7.11.0 # SSPL / Elastic License 2.0
# and the question that arrived by email from Legal:
# "can we still use this"
# "what happens if we upgrade"
# "how long do we have"
# nobody on the team could answer any of the three.The immediate answer to all three is reassuring — an internal application is unambiguously permitted under either licence — and the reason it took three days to establish is that nobody had ever had to read a licence for infrastructure before. That is the real cost and it recurs at every future upgrade.
Why it happens
The dispute is between a company that builds an open-source product and a cloud provider selling it as a managed service without contributing back. Both positions are defensible and neither has anything to do with the people running a search cluster for a shop.
The consequence for a user is that a routine version bump now requires establishing whether your use is a competing managed service, which it almost certainly is not, and getting somebody to say so in writing. Doing that once is fine. Doing it at every minor version is what makes people move.
The fix
Enumerating the options honestly
stay on 7.10 no work today. no security patches after
the maintenance window closes, and it
closes. a deadline you did not choose.
accept the licence a legal review, once, and then upgrades
are normal again. the review has to be
repeated if the terms change.
fork a client library change, a divergence
that grows, and a smaller ecosystem.
managed service somebody else's problem, at a price, and
the data leaves your network.
replace entirely Postgres full-text, Meilisearch, Typesense.
a rewrite of the query layer.The fifth option is the one people skip and the one worth costing, because a great deal of what a cluster does in a mid-sized application is a filtered keyword search that a database can serve. Establishing that took an afternoon of reading the actual queries.
$ curl -s 'localhost:9200/_search?size=0' -H 'Content-Type: application/json'
-d '{"aggs":{"types":{"terms":{"field":"_index"}}}}' | jq -r '...'
# what the queries actually used, from a week of slowlog:
# match / multi_match 88%
# term and range filters 74%
# aggregations 9%
# fuzziness 6%
# more_like_this 0.2%
# percolate 0%
# 88% of it is a keyword search with filters.Nine per cent using aggregations is the number that decided it: faceted counts over a large filtered set are the thing a search engine genuinely does better than a relational database, and rewriting those against SQL would have been slower and more code. So replacement was off the table for the wrong reason — not because the search was sophisticated, but because the facets were.
What a fork actually costs
The server side is nearly free at the fork point, because the fork begins as the last Apache-licensed release with the trademarks removed. The friction is entirely in the client library, and it is deliberate rather than technical.
// the official client, from 7.14, checks a response header
// and refuses to talk to anything that is not Elasticsearch:
//
// "The client noticed that the server is not Elasticsearch
// and we do not support this unknown product"
// so the fork has its own client, with the same shape
// and a different namespace
composer require opensearch-project/opensearch-php
-use ElasticsearchClientBuilder;
+use OpenSearchClientBuilder;Counting the places that mention the vendor namespace is the whole estimate. On this codebase it was eleven files, of which nine were tests and two were the client factory — because the queries had always been built as plain arrays rather than through a fluent builder from the client library.
$ grep -rl 'Elasticsearch\' src/ tests/ config/ | wc -l
11
$ grep -rl 'Elasticsearch\' src/ | wc -l
2
# and the queries, which are the part that could have hurt:
$ grep -rn 'QueryBuilder|->query()->' src/ | wc -l
0 # arrays all the way downThat was luck rather than foresight, and it is the strongest argument for a thin abstraction that usually looks like over-engineering. A codebase using a fluent query builder from the vendor library would have had the query layer itself to port, which is a different order of work.
Testing a fork against the queries you actually run
A smoke test that indexes three documents and searches for one of them proves nothing about relevance, which is the thing that differs subtly and matters most.
// capture a week of real queries from the slowlog, then
// replay them against both clusters and compare
foreach ($this->capturedQueries() as $query) {
$old = $this->legacy->search($query);
$new = $this->candidate->search($query);
$oldIds = array_column($old['hits']['hits'], '_id');
$newIds = array_column($new['hits']['hits'], '_id');
$this->record($query, [
'same_set' => $oldIds == $newIds,
'same_order' => $oldIds === $newIds,
'top_1' => ($oldIds[0] ?? null) === ($newIds[0] ?? null),
'jaccard' => $this->jaccard($oldIds, $newIds),
]);
}4,102 captured queries, replayed against both:
identical result set and order 4,098
same set, different order 3
different set 1
the three: aggregation buckets with equal counts, where
the tiebreak differs. harmless, and worth knowing about.
the one: a query with a deprecated parameter that the
fork removed. it had been logging a warning since 2019.The single genuine difference was our bug rather than theirs, and it had been reported in a deprecation warning nobody read for two years. That is the usual outcome of a comparison like this and is worth the day it takes.
The migration, which is a reindex
There is no in-place upgrade path between two products, so the move is a full reindex into a new cluster with a period where both are written to.
// phase 1: dual write, read from the old one
$this->legacy->index($document);
try {
$this->candidate->index($document);
} catch (Throwable $e) {
// must NOT fail the request during the transition
$this->logger->warning('candidate.index.failed', [
'id' => $document['id'],
'error' => $e->getMessage(),
]);
}
// phase 2: read from the candidate for 1% of traffic,
// compare against the legacy result, log differences
// phase 3: read from the candidate, dual write continues
// phase 4: stop writing to the legacy clusterSwallowing failures on the candidate write is what keeps the transition safe and is also what makes it possible to end up with a silently incomplete index. The count comparison at the end of each phase is not optional, and neither is a full reindex before the cutover regardless of what the counts say.
$ for c in legacy candidate; do
> printf '%-10s %sn' "$c" "$(curl -s "$c:9200/products/_count" | jq .count)"
> done
legacy 412008
candidate 412008
# and the one that actually matters, because counts hide
# a document indexed with the wrong content:
$ ./bin/compare-checksums products
412008 compared, 0 differingThe alias that makes the cutover a single command
POST /_aliases
{
"actions": [
{ "remove": { "index": "products_v3", "alias": "products" } },
{ "add": { "index": "products_v4", "alias": "products" } }
]
}Every application should read and write through an alias rather than an index name, and adopting that before the migration is what turns a cutover into an atomic operation instead of a deploy. It is also what makes a reindex routine afterwards: build a new index, verify it, swap the alias, delete the old one a week later.
The application had been using the index name directly since 2017. Changing that was the first commit of the migration and would have been worth doing regardless of the licence.
Verifying it worked
$ curl -s localhost:9200 | jq -r '.version.number, .version.distribution'
1.2.0
opensearch
$ ./bin/replay-queries --against=production --sample=500
500 queries, 500 identical result sets
$ curl -s 'localhost:9200/_cat/indices?v'
health index docs.count store.size
green products_v4 412008 1.1gb
# and the search latency, which is the user-facing number
# p95 before: 41ms
# p95 after: 38msLatency being marginally better is coincidence rather than a property of the fork — the new cluster was on newer hardware. Reporting it as a benefit would have been dishonest, and it is the kind of number that gets quoted back at you a year later.
The replay against production traffic, run as a scheduled job for the first fortnight, is what caught the one genuine regression: a query with a script field that behaved differently and had been used by exactly one internal report.
What this costs
A dependency on a fork, which is a bet on a project that is nine months old at the time of the decision. That bet looks better with a large cloud provider behind it and it is still a bet, and the honest framing to give the business is that we have swapped one company’s commercial strategy for another’s.
The divergence is the part that accrues. At the fork point the two are the same software; a year later they have different features, and code written against one stops being portable to the other. Anything relying on a feature that exists in only one is a decision to stay, made implicitly, which is why the query layer being plain arrays mattered so much.
The decision that would have been better than any of this is the one nobody made in 2017: writing down what the search actually needs to do, in a document, so that this conversation could have started from requirements rather than from a licence. Four years of accumulated queries had to be read to reconstruct it, and that reading is the work that made every other part of the migration possible.