Türker Yıldırım

Testing a block theme without a browser

Deep dive10 min read

The block theme conversion removed forty PHP files and every test that had covered them, which had been four tests and was still more than zero. What replaced them was a directory of HTML files and a JSON settings file, neither of which a PHPUnit test knows how to call.

The symptom

$ ls tests/
bootstrap.php

$ ls templates/ parts/
templates/: index.html single.html archive.html 404.html
parts/: header.html footer.html

# and the failure mode this allows:
#   a typo in theme.json — silently ignored
#   a block markup error — a validation warning in an
#     editor screen nobody opens
#   a missing template — the wrong layout renders, with
#     no error anywhere

All three failures are silent. A misspelled key in theme.json produces a setting that does nothing and looks identical to one that works, which is the specific failure that a schema check exists to catch.

Why it happens

A block theme is data rather than code, and the tooling for testing data is validation rather than execution. The instinct is to look for a unit test and there is nothing to unit — the assertions available are that the files are well-formed, that they render, and that the rendering has not changed.

The fix

Validating theme.json against the published schema

- name: Validate theme.json
  run: |
    npx ajv-cli validate --strict=false 
      -s .github/schemas/theme-5.9.json 
      -d theme.json

    for f in styles/*.json; do
      npx ajv-cli validate --strict=false 
        -s .github/schemas/theme-5.9.json -d "$f"
    done

Vendoring the schema rather than fetching it from the network is what makes this reproducible and pins the check to the minimum supported WordPress version — a key that only exists in 6.0 should fail on a theme claiming to support 5.9. That version pinning is the whole value beyond catching typos.

$ npx ajv-cli validate --strict=false -s ... -d theme.json
theme.json invalid
[ { instancePath: '/settings/spacing',
    keyword: 'additionalProperties',
    params: { additionalProperty: 'customPadding' },
    message: 'must NOT have additional properties' } ]

# customPadding is the v1 name. it was silently ignored
# for three weeks.

Rendering a template and asserting on the markup

final class SingleTemplateTest extends WP_UnitTestCase
{
    public function test_it_renders_the_post_content(): void
    {
        $id = self::factory()->post->create( array(
            'post_content' => '<!-- wp:paragraph --><p>Hello</p>'
                            . '<!-- /wp:paragraph -->',
        ) );

        $this->go_to( get_permalink( $id ) );

        $html = do_blocks( file_get_contents(
            get_theme_file_path( 'templates/single.html' )
        ) );

        $this->assertStringContainsString( '<p>Hello</p>', $html );
        $this->assertStringContainsString( 'wp-block-post-title', $html );
    }
}

go_to sets the global query so the dynamic blocks have a post to render, which is the piece that makes this work at all — without it every post-related block renders empty and the assertions fail for the wrong reason. It tests that the template references the right blocks rather than that the page looks correct, which is narrow and is more than the theme had.

Catching a block validation error before an editor does

public function test_every_template_parses_cleanly(): void
{
    $files = glob( get_theme_file_path( 'templates/*.html' ) );

    foreach ( $files as $file ) {
        $blocks = parse_blocks( file_get_contents( $file ) );

        $this->assertNotEmpty( $blocks, basename( $file ) );

        foreach ( $this->flatten( $blocks ) as $block ) {
            if ( null === $block['blockName'] ) {
                continue;    // a raw HTML fragment, which is legal
            }

            $this->assertTrue(
                WP_Block_Type_Registry::get_instance()
                    ->is_registered( $block['blockName'] ),
                sprintf( '%s references unregistered %s',
                    basename( $file ), $block['blockName'] )
            );
        }
    }
}

Asserting that every referenced block is registered catches the template that names a block from a plugin nobody installed on this site, which renders as nothing and produces no error. It is also what catches a typo in a block name, since wp:post-titel parses perfectly and registers as an unknown block.

A visual baseline, in a container

// playwright.config.js
export default defineConfig({
  expect: { toHaveScreenshot: { maxDiffPixels: 0 } },
  use: { baseURL: 'https://turkeryildirim.com' },
})

// and the update, which MUST run in the same image the
// pipeline uses, or every baseline differs by font
// rendering alone:
//
//   docker run --rm -v "$PWD:/w" -w /w 
//     mcr.microsoft.com/playwright:v1.28.0-focal 
//     npx playwright test --update-snapshots

A zero pixel tolerance is achievable with a pinned container and is what makes the check meaningful — a non-zero threshold hides exactly the small regressions this exists to catch. Self-hosting the theme fonts rather than loading them from a CDN removes the other source of intermittent difference.

This is the test that actually caught something during the conversion: three templates where blockGap replaced a hand-written margin and the resulting spacing was two pixels different. None of the other checks would have noticed and none of them should have.

Verifying it worked

$ vendor/bin/phpunit
Tests: 14, Assertions: 61, Time: 00:08.402

$ npx ajv-cli validate --strict=false -s ... -d theme.json
theme.json valid

$ npx playwright test
  41 passed

# and the deliberate check, run once:
#   introduce a typo in theme.json → the build fails
#   rename a block in a template   → the build fails
#   change a margin by 1px         → the build fails

Deliberately breaking each thing to confirm the corresponding check fails is worth the ten minutes at the moment the checks are written, because a validation step that silently passes everything is indistinguishable from one that works.

What this costs

A suite that tests markup, which changes often — every deliberate design change is a baseline update, and a baseline updated without being looked at is worse than no baseline. The discipline is that a screenshot diff in a pull request needs a comment saying why the change is intended, and there is no tool that enforces it.

The schema check also pins the theme to a WordPress version in a way nothing else does. Supporting 5.9 and 6.1 means validating against both schemas and accepting that a key valid in one fails in the other, which in practice means validating against the older and losing the ability to catch a genuinely invalid 6.1 key. That is the honest limitation and it is still better than the alternative.