Merging three WordPress installs into one multisite

Three WordPress installs on one server, sharing a theme by way of a directory that had been copied twice and then edited three times. Three plugin sets, three update schedules, one person doing the patching. This is the merge into a single network, the order the content had to move in, and the things that stopped working afterwards.

The symptom

$ for d in site-a site-b site-c; do
>   echo "== $d"; wp --path=/var/www/$d core version
>   wp --path=/var/www/$d plugin list --update=available --field=name
> done
== site-a
4.0
== site-b
4.0
wordpress-seo
contact-form-7
== site-c
3.8.4
wordpress-seo
contact-form-7
wp-super-cache

A security release had gone out in September. It was applied to the first site the same day, to the second a week later, and to the third six weeks after that — by which point it was found by running the command above rather than by anyone remembering. Nothing had been exploited. The gap was not a plan, it was an oversight that had happened before and would happen again.

Why it happens

Separate installs share nothing, and that includes the discipline. Each one has its own core, its own plugin directory, its own update screen and its own nag. Keeping three in step is a task with no artefact and no deadline, so it competes with work that has both, and loses.

A network changes what the task is. One core, one plugin directory, one update — the sites become rows in a table rather than copies of an application. The cost of that is real and is dealt with at the end, but the maintenance argument on its own is what carried the decision.

The fix

The network first, empty

The largest site becomes the network’s primary site, because its content does not have to move at all. Everything else is imported into it.

// wp-config.php, before running the network install
define( 'WP_ALLOW_MULTISITE', true );

// and what the network install then tells you to add
define( 'MULTISITE', true );
define( 'SUBDOMAIN_INSTALL', false );
define( 'DOMAIN_CURRENT_SITE', 'example.com' );
define( 'PATH_CURRENT_SITE', '/' );
define( 'SITE_ID_CURRENT_SITE', 1 );
define( 'BLOG_ID_CURRENT_SITE', 1 );

Subdirectory rather than subdomain, because the certificate covers one host name and buying a wildcard for this was not worth it. That decision is difficult to reverse later, so it is worth making deliberately rather than accepting the default.

The plugin sets, before anything moves

A network has one plugin directory. Three installs that between them carry forty-one plugins, of which nineteen appear on more than one site at different versions, therefore need reconciling before the first import rather than after — a plugin activated on the network writes options for every site, and finding out afterwards that two of the three should never have had it is expensive.

for d in site-a site-b site-c; do
  wp --path=/var/www/$d plugin list 
    --status=active --fields=name,version --format=csv | sed "1d;s/^/$d,/"
done | sort -t, -k2 > plugins.csv

# the ones that differ in version between installs
awk -F, '{ if ($2 == prev && $3 != pv) print prev; prev=$2; pv=$3 }' plugins.csv

The output split into three groups, and the split is the useful part. Eleven plugins belonged on every site and became network-activated. Twenty-two were genuinely per-site and stayed that way, activated individually after the import. Eight were dead — installed for a campaign in 2012, still active, still running code on every request — and the merge was the first time in two years anyone had looked at the list closely enough to notice.

The import order that keeps attachments attached

A WXR export contains posts, pages, attachments, terms and users, and the importer processes them in file order. Attachments are posts whose parent is another post, so an attachment imported before its parent gets a parent ID that means something else entirely — or nothing — and the media library fills with images belonging to the wrong article.

The reliable order is terms, then users, then posts and pages, then attachments, each as its own export. The importer maps old IDs to new ones as it goes and keeps that map for the run, so the parent references resolve as long as the parent has already been seen.

#!/usr/bin/env bash
set -euo pipefail

SRC=/var/www/site-b
DST=/var/www/network
URL=example.com/journal

# two exports, so a parent is always imported before its attachments
wp --path=$SRC export --dir=/tmp/b/posts --post_type=post,page
wp --path=$SRC export --dir=/tmp/b/media --post_type=attachment

# --authors=create keeps authorship; the users become network members
for f in /tmp/b/posts/*.xml /tmp/b/media/*.xml; do
  wp --path=$DST --url=$URL import "$f" --authors=create --skip=image_resize
done

# the ids changed, so anything that stored one has to be rewritten
wp --path=$DST --url=$URL search-replace 
  'http://siteb.example.com' 'http://example.com/journal' --precise

--skip=image_resize matters on a large library: the importer regenerates every thumbnail as it goes and turns a four-minute import into an hour. Regenerating them afterwards, in one pass that can be restarted, is both faster and interruptible.

Users are global, and were not before

In a network the user table is shared: one account, one password, membership of any number of sites. Across three separate installs the same person frequently exists three times, sometimes with different email addresses and usually with different capabilities.

-- the same person, three rows, three password hashes
SELECT user_email, COUNT(*) c, GROUP_CONCAT(user_login) logins
  FROM (
    SELECT user_email, user_login FROM a_users
    UNION ALL SELECT user_email, user_login FROM b_users
    UNION ALL SELECT user_email, user_login FROM c_users
  ) t
 GROUP BY user_email
HAVING c > 1;

The merge rule has to be decided before the import, not during it, because the importer will make its own decision otherwise. Email address is the identity; the login name from the largest site wins; the highest capability across the three is kept only if it is editor or below, and administrator is never merged upward. That last clause is the one worth writing down — a contributor on two sites and an administrator on the third should not become an administrator of the network.

Duplicate email addresses with different logins have to be resolved by a person. There were nine. It took an afternoon of asking, and doing it before the import rather than after is the difference between nine questions and nine hundred rows to unpick.

Rewriting, and the uploads path that moves

Subdirectory networks need rewrite rules that strip the site segment from requests for core files, otherwise /journal/wp-admin/ is a 404. On nginx there is no generated file to paste, so this is written once and understood rather than copied.

map $uri $blogname {
    ~^(?<blogpath>/[^/]+/)files/(.*) $blogpath;
}

server {
    server_name example.com;
    root /var/www/network;

    # core and wp-content requests below a site path lose the path
    rewrite ^/[_0-9a-zA-Z-]+(/wp-.*) $1 last;
    rewrite ^/[_0-9a-zA-Z-]+(/.*.php)$ $1 last;

    # sites created before 3.5 serve uploads through ms-files.php
    location ~ ^(/[^/]+/)?files/(.+) {
        try_files /wp-content/blogs.dir/$blogname/files/$2 /wp-includes/ms-files.php?file=$2;
        access_log off;
        expires 30d;
    }

    location ~ .php$ {
        include      fastcgi_params;
        fastcgi_pass unix:/var/run/php5-fpm.sock;
    }
}

The files block is the part that catches people. A network created today puts uploads in wp-content/uploads/sites/2/ and serves them as static files; sites created before 3.5 serve them through ms-files.php, which is PHP handling every image request. A merged network can contain both, and the theme does not know which — so anything in a template that builds an upload URL by string concatenation rather than through wp_upload_dir() breaks for exactly one of the three sites.

Verifying it worked

The claim being made is that every URL that worked before still works, which is checkable rather than arguable. The three sitemaps and the access logs of the last ninety days give a list of every URL anybody has actually requested.

#!/usr/bin/env bash
# every published URL from all three sites, against the network
while read -r old new; do
  code=$(curl -s -o /dev/null -w '%{http_code}' -L "$new")
  [ "$code" = '200' ] || echo "$code  $old  ->  $new"
done < url-map.txt

$ wc -l url-map.txt
4812 url-map.txt

$ ./check-urls.sh | tee failures.txt | wc -l
37

$ sort -k1,1 failures.txt | uniq -c -w3
     31 404  /gallery/...   (attachment pages, not redirected)
      6 500  /events/...    (shortcode from a plugin not network-activated)

Thirty-seven failures out of 4,812, in two groups. The attachment pages were a redirect rule that had not accounted for a slug pattern; the five hundreds were a plugin activated on one site and not on the network, which is the single most common thing to get wrong in the first week and the reason to check status codes rather than just following links.

One check the crawl cannot make is whether the right thing is on the page. A 200 proves the URL resolves, not that the article kept its images, so the second pass is a count: attachments per site before and after, posts per author, terms per taxonomy. Three numbers per site, and any of them being off by more than the known deletions is worth stopping for.

$ wp --path=$DST --url=$URL post list --post_type=attachment --format=count
2841
# source install reported 2844 — three were already orphaned before the move

The three missing attachments had no file behind them on the old install either, which is worth confirming rather than assuming: a merge is exactly when a pre-existing inconsistency gets attributed to the merge, and an hour spent proving otherwise is an hour well spent.

What this costs

One database and one failure domain. Before, a plugin update that broke the third site broke the third site; now it breaks everything, and a database restore takes all three back to the same point whether or not they all needed it. That is a genuine loss and it is the price of the maintenance win — the same property that makes one update reach three sites makes one mistake reach them too.

Plugins are the second cost, and it is ongoing. A large number of them assume a single site: they write to a fixed option name, register a cron job without a site context, or build an uploads path from ABSPATH. Most work, some work per-site by accident, and a few need a fork. Every plugin now has a question attached to it that did not exist before, and the answer is rarely in the readme.

The database grows by nine tables per site, and anything that spans the network becomes a loop rather than a query.

foreach ( wp_get_sites( array( 'limit' => 0 ) ) as $site ) {
    switch_to_blog( $site['blog_id'] );

    $published = wp_count_posts( 'post' )->publish;

    // every early return between here and the restore leaks the switch,
    // and the next thing to run writes to the wrong site's tables
    restore_current_blog();
}

That loop is nine queries instead of one for three sites and three hundred for a hundred, so a dashboard widget written this way is fine now and is the thing that falls over later. The subtler hazard is the one in the comment: a switch_to_blog() without its matching restore does not error, it just quietly redirects the rest of the request at another site’s tables. Three sites is comfortably below the size where any of this hurts, but it is worth knowing where the ceiling is before adding the twentieth site rather than after.

Would I do it again? For three sites sharing a theme and a maintainer, yes, and the deciding factor was not the update count. It was that “which sites is this plugin on” and “which version is site C running” became questions with one answer each instead of three, which is what made the patching reliable rather than merely faster.