Claude and Drupal QA

I had Claude compare 50 pages across two versions of a Drupal site, at desktop and phone widths, and it finished in about ten minutes. The results were impressive but not quite perfect.

It’s been more than ten years, but I vividly recall working on a dev project where we used Selenium for functional testing. It was a new kind of programming hell, one I gladly abandoned after the project.

I won’t go too deeply into the process, but I recall building tests: find an element by its ID, click it, wait for the page, check the text, wait some more. Now imagine this for scores of pages. Again, I found the tedium painful, even on a project where I was paid by the hour.

I’ve done some QA with Claude before, but today I wanted to expand my tests. Could I compare two virtually identical Drupal sites running different versions of Drupal?

The impetus here is that Drupal shipped a critical security fix, and we needed to release updates quickly.

How the QA worked

I asked Claude to compare 50 pages on our production and development versions of a test site and tell me if anything looked different.

As I often do, I ended my initial prompt by telling Claude to ask me questions first. It came back with questions about which pages to crawl, which browser, etc. I answered them and let Claude rip.

Claude crawled the links in the main navigation and chose 50 pages covering every content type we use. It loaded each page on both copies of the site at a desktop width and a phone width, using my browser, and recorded the text, headings, page height, images, links, fonts, header styling, and the stylesheets and scripts each page pulls in. Then it compared the two. Anything flagged got opened by hand before it counted.

The time it saved

Checking 50 pages on two sites at two screen widths is 200 page loads, and every pair has to be compared. Doing that by hand means a lot of tab-switching and squinting. It’s the same level of drudgery as Selenium analysis back in the day.

From the moment I asked to the moment I had a published report, about ten minutes passed (ten minutes that I spent working on something else!). That includes Claude stopping to ask me a few questions first. The page loading itself took a few minutes.

Two analog clocks: 30 minutes by hand (my estimate) against 10 minutes with Claude to QA 50 pages on two sites
10 minutes from my request to a published report, questions included. The 30-minute by-hand figure is my own estimate.

If I had to do this review myself, I’d guess half an hour. It’s hard to say exactly, but I know I’d have wanted to do something else with that time.

What we found

Nothing looked like a broken upgrade. All 50 pages loaded on both sites, used the same stylesheets and scripts, and had matching headings, fonts and header styling. No images were broken, and nothing overflowed on a phone.

40 of 50 pages matched on production and development, apart from one dev-only banner
Seven of the ten differences were news and events content that dev doesn’t have.

The differences were mundane, and it seems quite clear that this report was far more comprehensive than my manual checks would have been.

What to test for this release

After the comparison, I asked Claude what I should test for this update. It read the release notes and advisories and gave me a checklist tied to what actually changed. Here are some of the things it suggested:

  • Rich text. The HTML parser update is the likeliest source of visible changes.
  • URLs, redirects and login. The Symfony routing and HTTP updates can touch aliases, redirects, query strings, error pages and sessions. Test single sign-on login and logout, and a few redirected URLs.
  • Forms and data. Submit a form, and exercise any Views exports or JSON endpoints.

What it can’t do

Be clear-eyed about the limits. Claude took screenshots and compared pages across different environments. It didn’t check the admin panel, and it can’t tell whether a difference is legitimate, such as one caused by a content change. We still need humans (thankfully). After all, how often are dev and prod sites really identical?

But I suspect that Opus 7 or Fable 9 will be much better at these sorts of tasks. In the meantime, I’m amazed by what Claude was able to do in less than ten minutes.

Leave a comment

Your email address won’t be published. Name, email and comment are required.