Test::Builder
Problems

What's wrong with Test::Builder? (Test::More)

old-age

Test::Builder is showing its age



  • First release was March 28, 2001.
  • Last significant feature added Jun 23, 2009 (subtests).
  • Test-Builder/More2 halted Test-Builder, but ultimately stalled.
  • The code has grown organically, it shows.
Speed

Test::Builder is slow



  • Old code has some inefficient practices.
  • Perl test suite takes a very long time to run.
  • Not everything benefits from the features that make it slow.
Level

Test::Builder uses a $Level variable for error reporting

  • Obscure, very few people are aware of it.
  • The job it does is super critical.
  • It does not do the job well.
  • It is easy to get wrong.
  • It is slow.
Syncing

Test::Builder has minimal concurrency support



  • Works with ithreads... sometimes.
  • Third party module required for forking, breaks ithread support.
  • You can turn off test numbers and manually plan...
  • Subtests and concurrency just do not play nicely.
But at least you can run multiple test files concurrently!
Testing

Test::Builder is hard to test (Wait, what?!)

Test::Builder revolutionized testing in perl. It opened the door to an entire ecosystem of intercompatible testing tools.

Suprisingly, testing tools build on Test::Builder is not easy.

  • Test::Builder::Tester made it possible.
  • Test::Tester made it better.
  • Both are limited in what they support.
  • Both are fragile.
  • Both make assumption.
The Test::Stream project was concieved after David Golden (XDG) submitted this pull request. It changed the indentation of a comment intended for human consumption, but broke downstream tool tests.
Tooling

Test::Builder had a limited vision

Test::Builder assumed 'ok()' and 'diag()' were all you need

  • No hooks into ok or diag.
  • No way to customize their behavior.
  • Modifying state is non-obvious and an afterthought.
  • Monkeypatching is rampant.

Test::Builder puts EVERYTHING into a monolithic singleton

  • Can't subclass test builder... not nicely anyway.
  • Subtests hack on the singleton would be rated NC-17 for gore.
  • Can't easily intercept results.
  • Tools that replace the singleton cannot work together.
Test::Stream
Solutions

What's better with Test::Stream?

New

Test::Stream adopts modern practices and ideas



  • New architecture.
  • Proper Encapsulation.
  • Separation of concerns.
  • Code was written with a goal in mind.
  • Code was written to solve problems.
Speed

Test::Stream is faster than Test::Builder

Test::Builder numbers

        Test-Simple version: 1.001014
        1000000_oks.t  38.51s user 0.44s system 100% cpu 38.941 total
        1000000_oks.t  38.52s user 0.52s system 100% cpu 39.024 total
        1000000_oks.t  38.92s user 0.35s system 100% cpu 39.254 total

Test::Stream numbers

        Test-Stream version: 1.302024
        1000000_oks.t  22.55s user 0.20s system 100% cpu 22.743 total
        1000000_oks.t  22.71s user 0.18s system 100% cpu 22.882 total
        1000000_oks.t  22.83s user 0.23s system 100% cpu 23.055 total

Test::Builder converted to use Test::Stream under the hood

        Test-Simple version: 1.302012_003
        1000000_oks.t  34.11s user 0.43s system 100% cpu 34.525 total
        1000000_oks.t  34.12s user 0.46s system 100% cpu 34.566 total
        1000000_oks.t  34.45s user 0.40s system 100% cpu 34.841 total
Context

Test::Stream does not use $Level

  • The context is a primary interface, not obscure.
  • No need to localize things.
  • The context is like a gps that keeps everything coordinated.
Syncing

Test::Stream was designed with concurrency in mind



  • Works with threads.
  • Works with forking.
  • Works with subtests.
  • It just works.
Testing

Test::Stream has tools to test testing tools

  • Facilties to intercept results as they happen.
  • Functions to verify generated events.
  • The test testing tool can test itself even!
  • Not fragile.
  • Easy to customize.
Tooling

Test::Stream tries not to assume it knows what you need

  • Results are objects (Test::Stream::Event::*)
  • You build an event, then send it off.
  • There are hooks to modify events before they are processed.
  • You can customize the output formatter, TAP is just the default.
  • Hooks to watch events after they are processed.
  • Hooks all the things!
  • Minimal singleton.
  • No need for monkeypatching.
Berlin 2015

QA Hackathon 2015 in Berlin

During the hackathon we had a long discussion about Test-Builder. Here is a summary of the findings: