Why test things?

byStefan JohanssononAug 5, 2026read19 mininsoftware

• testing
• dependencies

For some time now, the diagrams in the React Flow post were missing.

Not broken in a way anything complained about. The page built. The React island hydrated. The title bar rendered, the caption rendered, and between them sat a canvas with a height of exactly zero pixels. React Flow dutifully laid out every node and edge inside it. Nobody could see them.

Two repositories were involved, both with test suites that I had spent real effort on. The component library, astro-components, runs around 270 unit and integration tests. This blog runs another 120 or so, plus a Playwright suite. Every one of them was green the entire time.

This post is about what that gap says about testing. Not “tests are pointless” - they aren’t, and the same repositories have caught plenty of real regressions. The point is narrower and, I think, more useful: a test suite can only protect the things someone thought to describe. The bugs that hurt are usually in the space nobody described, and coverage numbers tell you nothing about the size of that space.

What actually happened

@sjohansson/astro-reactflow wraps React Flow for Astro sites. I wrote about it in an earlier post, which is also where the empty boxes lived. The component has to be rendered with client:only="react", because React Flow measures the DOM and has no server output. That is the documented, correct usage.

The component imports its own stylesheet at the top of the file:

// ReactFlowWrapper.tsx
import "./styles.css";

That stylesheet carries the one rule the whole thing depends on:

.reactflow-pane {
  flex: 1 1 auto;
  min-height: 0;
}

The wrapper has an explicit height. The pane inside it is the flex child that is supposed to fill that height, and React Flow’s own container is height: 100% of the pane. Take the rule away and the chain collapses: the pane has no intrinsic height, so it resolves to 0px, and React Flow renders into nothing. As the React Flow troubleshooting guide puts it, “The React Flow parent container needs a width and a height to render the graph.”

So why did the rule go missing? Astro collects the CSS for a page from the modules it imports while rendering that page on the server. A client:only component is, by definition, never imported on the server. Astro does have a recovery path for this - the client build walks CSS chunks back up to their client:only parents and attaches them to the right page - but that mapping has a long history of edge cases going back to 2021, and a fresh one was fixed as recently as Astro 7.1.4 in July 2026. In this case the stylesheet was imported inside an npm package, reached only through an MDX island, and it simply never made it into the built page.

The fix, once found, was small. The integration now injects the stylesheet at Astro’s page-ssr stage, which the Integration API docs describe as existing precisely for “injecting a CSS import into every page to be optimized and resolved by Vite”. The package also gained a public styles.css export so consumers can import it by hand. That shipped as 0.5.0 on 8 September 2026.

What the tests were saying the whole time

Here is the part worth dwelling on. The package has a test for the wrapper’s height. It passed:

it("applies the default height", () => {
  const { container } = render(<ReactFlowWrapper nodes={nodes} edges={edges} />);
  expect(container.querySelector(".reactflow-wrapper")).toHaveStyle({ height: "400px" });
});

This test is not wrong. The wrapper does get height: 400px as an inline style. The test simply describes a slightly different fact from the one that mattered. It checks an attribute on one element, in jsdom, where no layout happens, where no stylesheet is loaded, and where React Flow itself is mocked out because rendering the real thing in a unit test is painful. Every one of those simplifications is reasonable on its own. Together they mean the test could never have observed a zero-height pane.

Ask the same question of the blog’s suite: 36 tests for the diagram component, and the first thing they do is vi.mock("@xyflow/react", ...). The Playwright end-to-end suite exists, and it is good at what it does, but what it does is guard the web font. Its most elaborate test renders the same string at weight 400 and 700 with font-synthesis: none, screenshots both, and fails if the pixels match:

expect(
  Buffer.compare(shot400, shot700),
  "weight 700 rendered identically to 400 with synthesis off - faux-bold or broken @font-face range",
).not.toBe(0);

That test exists because faux-bold had shipped once. Nothing in the suite has ever looked at a diagram.

None of that is negligence. It is the ordinary shape of a test suite that grew up around the things that had already gone wrong once. The stylesheet had never gone missing before, so no test asked whether it was there.

What tests are for

It helps to be precise about what a test suite actually buys you, because “quality” is too vague to act on.

The most honest description I know comes from Michael Feathers, who coined the term characterization test in Working Effectively with Legacy Code: “A characterization test is a test that characterises the actual behavior of a piece of code.” Not what the code should do. What it does, right now, today.

That framing is the whole point. At any moment you have a now-state: the code as it is, behaving as it behaves, with users depending on that behavior whether it is intended or not. Tests pin down as much of the now-state as you can afford to describe. Then, when a change arrives - a feature, a refactor, a dependency bump - you run the suite and it tells you which parts of the now-state moved. If the only things that moved are the things you meant to move, you have evidence the change is contained.

That is a genuinely valuable property, and it is the reason both of these repositories will keep their test suites and keep adding to them. Dependency updates in particular are where it pays off: the blog’s Renovate bot lands version bumps weekly, and the suite is what lets those merge without a human re-reading every changelog.

But notice what the framing does not promise. It does not promise that the now-state is correct. If the diagram was already invisible when the test was written, a characterization test would happily assert that it stays invisible. Edsger Dijkstra said it in 1969, at the NATO software engineering conference, and it has not stopped being true: “Program testing can be used to show the presence of bugs, but never to show their absence!”

Coverage is a map of where you looked

Coverage tooling makes the gap worse by making it invisible. A coverage report shows the lines a test executed, rendered in reassuring green. It cannot show the assertions nobody wrote, the environment differences nobody reproduced, or the stylesheet that was not loaded because jsdom does not load stylesheets.

Martin Fowler’s 2012 note on test coverage is still the clearest statement of this: “Test coverage is a useful tool for finding untested parts of a codebase,” but “high numbers don’t necessarily mean much, and lead to ignorance-promoting dashboards.” He adds that he would “be suspicious of anything like 100% - it would smell of someone writing tests to make the coverage numbers happy.”

Google’s testing team said much the same in 2020: “Code coverage does not guarantee that the covered lines or branches have been tested correctly, it just guarantees that they have been executed by a test.” Their internal guideline treats 60% as acceptable and 90% as exemplary, and they explicitly warn against “obsessing on how to get from 90% code coverage to 95%”.

There is a general principle underneath. Goodhart’s law, in Marilyn Strathern’s phrasing, holds that “when a measure becomes a target, it ceases to be a good measure.” Coverage is a fine measure of where tests reach. The moment a team is rewarded for the number, the cheapest way to raise it is to write tests that execute code without asserting anything meaningful about it. The dashboard goes greener and the protection does not change.

Automating test creation: the trade

This matters more now than it did in 2012, because generating tests has become nearly free. Scaffolding tools, snapshot testing, and AI assistants can produce a large suite from an existing codebase in an arvo. Both repositories in this story used that kind of help (not a single test has been handcrafted, they’re all AI generated). It is worth being clear-eyed about what you get.

What you gain:

• Breadth, fast. A generated suite reaches corners a human would not bother with: every prop combination, every branch of a parser, every error path. That breadth is real and it does catch regressions.
• A safety net for refactoring. Even mediocre characterization tests make a large refactor or a major dependency upgrade far less frightening, because something will shout if behavior shifts.
• Documentation of the now-state. A test that asserts current behavior is at least an honest record of it, which is more than most codebases have.
• Cheap regression locks. Once a bug is fixed, generating a test that pins the fix is trivial and worth doing every time.

What you pay:

• Generated tests encode the bug. A tool that writes tests from existing code will faithfully assert whatever the code currently does. If the pane is already 0px, the generated test will enshrine 0px. Automation amplifies the now-state; it does not judge it.
• They inherit the environment’s blind spots. Tests generated for jsdom will only ever see what jsdom can see. No layout, no real CSS cascade, no browser. That is exactly the class of bug in this post.
• Volume is not the same as coverage of risk. Two hundred tests that mock out the interesting dependency are two hundred tests of the mock.
• They still need a reviewer. A generated test that asserts the wrong thing is worse than no test, because it looks like evidence. Someone has to read it and ask whether the assertion describes something a user would care about.

The last point is the one I would underline. More tests raise the amount of reading a change requires, not lower it. Reviews and sanity checks do not become optional because the suite is large; they become the thing that decides whether the suite is worth anything.

Where this bug should have died

It is useful to walk the layers and be honest about which one could realistically have caught a missing stylesheet.

Unit tests were never going to see it. jsdom does not compute layout and does not load imported CSS. The height assertion above is about as far as that layer can go, and it went there.

Module or integration tests could have, with effort. The 0.5.0 fix added tests that assert the integration calls injectScript("page-ssr", ...) with the stylesheet imports. That is a real improvement, but it is also a characterization test of the fix: it protects against someone removing the injection later, not against the next thing of this shape.

A built-output check is the cheapest layer that could have caught it directly. Build a fixture site that uses the component, then grep the generated HTML for a <link> to a stylesheet containing .reactflow-pane. It is crude and it is specific, but it exercises the real build pipeline rather than a simulation of it, and it would have failed on day one.

Visual regression testing is the layer that would have caught this without anyone predicting it. Playwright’s toHaveScreenshot writes a reference image on the first run and compares subsequent runs pixel by pixel. A 400-pixel diagram collapsing to a thin strip of title and caption is not a subtle diff. Hosted services such as Chromatic and Percy do the same thing with review workflows around the diffs.

There is a second question: which repository should own that test?

• In the source repository, a visual test on a fixture site protects every consumer at once, and it runs on every change to the package. This is where the durable fix belongs.
• In the consumer repository, a screenshot of the actual post protects this page against any upstream change, including ones the package author never anticipated. When Renovate bumps the dependency, the diff would have shown the diagram disappearing, and the update would have stayed unmerged until someone looked.

The right answer is both, and the consumer-side one is arguably more important, because it is the only one that catches problems the package author did not know to test for.

When the tests passed and reality did not

This pattern is not unique to small open-source packages. Two well-documented incidents share its shape almost exactly.

CrowdStrike, 19 July 2024. A content update to the Falcon sensor crashed an estimated 8.5 million Windows machines and grounded flights, hospitals and banks for days. CrowdStrike’s own root cause analysis is remarkably candid. The template type defined 21 input fields; the code that invoked it supplied 20. That mismatch “evaded multiple layers of build validation and testing”, in part because every test case used a wildcard in the 21st field, so the out-of-bounds read never fired. The content validator “based its assessment on the expectation that the IPC Template Type would be provided with 21 inputs” and passed the bad content. The system had validation, automated tests, and a track record of successful releases. The bug lived in the one combination nobody had described.

Knight Capital, 1 August 2012. A deployment of new trading code to eight servers reached only seven of them. The eighth still carried dormant code from a feature called Power Peg, discontinued in 2003, and the new release had repurposed the flag that used to activate it. According to the SEC order, Knight “did not retest the Power Peg code” after earlier changes and “did not have a second technician review this deployment”. In roughly 45 minutes the firm sent millions of orders and “realized a $460 million loss”. The code was tested. The deployment was not something the tests could see.

Neither of these was a failure to write tests. They were failures of imagination about what the tests needed to cover, and failures of review to notice the gap. That is the same failure as a missing stylesheet, at a scale that ends companies.

Dependencies move, and so must you

There is a bigger theme sitting behind the incident. The bug surfaced because the blog updates its dependencies constantly. That is also how it got fixed: the fix went into the package, the package was bumped, the blog picked it up.

The tempting lesson is “stop updating things that work.” It is the wrong lesson, and the historical record on it is brutal.

Equifax, 2017. The Apache Struts vulnerability was disclosed on 7 March 2017 with a patch available the same day. Equifax’s own security team emailed the alert to over 400 people on 9 March. The patch was not applied to the consumer dispute portal. Attackers got in on 13 May and stayed for 76 days. The House Oversight Committee’s report found that “Equifax’s failure to patch a known critical vulnerability left its systems at risk for 145 days” and called the breach “entirely preventable”. The FTC settlement started at $575 million for around 147 million affected people.

WannaCry, May 2017. Microsoft had published the fix, MS17-010, on 14 March. Two months later the worm hit at least 80 NHS trusts in England and cancelled an estimated 19,000 appointments. The UK National Audit Office found that “all NHS organisations infected by WannaCry had unpatched or unsupported Windows operating systems”. Notably, most infected machines were on Windows 7, a supported OS at the time. They just had not taken the update.

Log4Shell, December 2021. A single logging library, embedded almost everywhere. The US Cyber Safety Review Board concluded that Log4j is an “endemic vulnerability” and that “vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer”, precisely because so many deployments will never be updated.

MOVEit, 2023. A file transfer product with a patch published on 31 May 2023. By mid-2024 Emsisoft counted more than 2,700 organisations and 95 million individuals affected, the majority through instances patched too late or not at all.

The other side of the ledger is real too. The left-pad removal in 2016 broke thousands of builds in an afternoon, and the event-stream compromise in 2018 shipped a malicious dependency to everyone who updated. Updating blindly has its own failure modes. The answer is not “never update”; it is a lockfile, a bot that proposes updates, a test suite that vets them, and a human who reads the diff when the suite cannot.

The support window keeps shrinking

“We will stay on this version until it is convenient to move” used to be a workable strategy. The reality of platform life-cycles no longer supports it.

Platform	Cadence	Support per major
Node.js	New major every six months, even numbers get LTS	About 30 months. Node 20 reached end of life on 30 April 2026.
.NET	Annual	24 months for STS, 36 months for LTS
Angular	Annual major from v22	24 months total
Python	Annual	Five years, of which only two receive bug fixes
Vite	Roughly annual	Tracks the Node.js end-of-life schedule
Astro	Majors on 30 Aug 2023, 5 Dec 2023, 3 Dec 2024, 10 Mar 2026, 22 Jun 2026	Previous major receives fixes for a limited window

For contrast, Ubuntu LTS ships every two years with five years of standard support, and Red Hat Enterprise Linux offers a ten-year lifecycle. Those are the exceptions, and they are operating systems, not application frameworks. In the JavaScript ecosystem this blog lives in, Astro shipped two major versions in the first half of 2026 alone.

Even the slow movers are ending things. Windows 10 reached end of support on 14 October 2025, and as of August 2026 StatCounter still put it on about 30% of desktop Windows machines. Every one of those is an unsupported system, whether or not anyone has scheduled the work.

Every dependency you have will go stale. The only question is whether you move on your own schedule, in small steps that a test suite can vet, or on someone else’s, in one large step after an incident.

Unsupported is a compliance state, not just a technical one

If you operate under any kind of regulatory or assurance regime, “we are on an old version” stops being an engineering backlog item and becomes a finding.

In Australia, the ACSC’s Essential Eight Maturity Model requires at every maturity level that patches for critical vulnerabilities in internet-facing services are “applied within 48 hours of release”, non-critical ones within two weeks, and that “online services that are no longer supported by vendors are removed”. Operating systems that are no longer supported “are replaced”. There is no maturity level at which running unsupported software is acceptable.

APRA’s guidance to regulated financial entities, CPG 234, is blunter still. Technology that is “end-of-life, out-of-support or in extended support is typically less secure by design, has a dated security model and can take longer, or is unable, to be updated to address new threats”. It warns that extended support arrangements “could provide a false sense of security” and expects entities to decommission systems that “cannot be adequately updated”.

Elsewhere, PCI DSS v4.0.1 requires patches for critical vulnerabilities within one month of release. The EU Cyber Resilience Act pushes the obligation onto vendors, requiring security updates for a defined support period, with its main obligations applying from December 2027. And the OAIC’s guide to securing personal information asks organizations directly whether “the latest versions of software and applications [are] in use” as part of the reasonable steps expected under the Privacy Act.

None of these frameworks care that your tests are green. They care whether you are running something the vendor still fixes.

What I am taking from this

• Tests describe the now-state. That is enormously useful for containing change and nearly useless for discovering that the now-state was wrong.
• Coverage measures reach, not protection. Treat a high number as a map of where you have looked, and spend your attention on the blank areas.
• Generated tests need reviewers more, not less. They encode whatever the code does today, including the bugs, and they inherit every blind spot of the environment they run in.
• Put at least one test as close to the user as you can afford. For a web page that means a real browser and a screenshot. A crude visual regression test would have caught this in June instead of September.
• Consumers should test their own pages. The package author cannot anticipate every integration. A screenshot of the page you actually ship is the only test that catches what upstream did not know to check.
• Keep dependencies moving. Small, frequent, vetted updates are safer than rare large ones, and unsupported software is a regulatory exposure, not a technical preference.

The diagrams are visible again. The stylesheet export, the injected import and the integration tests are all in 0.5.0. The screenshot test is next.