Re: Readd use of TAP subtests

Hi,

On 2022-02-25 09:43:20 -0500, Andrew Dunstan wrote:
> AIUI TAP consumers are supposed to ignore lines they don't understand.

Are they?

In http://testanything.org/tap-version-13-specification.html there's:

"Lines written to standard output matching /^(not )?ok\b/ must be interpreted
as test lines. [...]All other lines must not be considered test output."

That says that all other lines aren't "test ouput". But what does that mean?
It certainly doesn't mean they can just be ignored, because obviously
^(TAP version|#|1..|Bail out) isn't to be ignored.

And then there's:

"
Anything else

Any output that is not a version, a plan, a test line, a YAML block, a
diagnostic or a bail out is incorrect. How a harness handles the incorrect
line is undefined. Test::Harness silently ignores incorrect lines, but will
become more stringent in the future. TAP::Harness reports TAP syntax errors at
the end of a test run.
"

If I were to to implement a tap parser this wouldn't make me ignore lines.

Contrasting to that:
http://testanything.org/tap-specification.html

"
Anything else

A TAP parser is required to not consider an unknown line as an error but may
optionally choose to capture said line and hand it to the test harness,
which may have custom behavior attached. This is to allow for forward
compatability. Test::Harness silently ignores incorrect lines, but will
become more stringent in the future. TAP::Harness reports TAP syntax errors
at the end of a test run.
"

I honestly don't know what to make of that. Parsers are supposed to ignore
unknown lines, Test::Harness silently ignores, but also "TAP::Harness reports
TAP syntax errors"? This seems like a self contradictory mess.

Greetings,

Andres Freund