Re: Readd use of TAP subtests

On 2/25/22 11:41, Andres Freund wrote:
> Hi,
>
> On 2022-02-25 09:43:20 -0500, Andrew Dunstan wrote:
>> AIUI TAP consumers are supposed to ignore lines they don't understand.
> Are they?
>
> In http://testanything.org/tap-version-13-specification.html there's:
>
> "Lines written to standard output matching /^(not )?ok\b/ must be interpreted
> as test lines. [...]All other lines must not be considered test output."
>
> That says that all other lines aren't "test ouput". But what does that mean?
> It certainly doesn't mean they can just be ignored, because obviously
> ^(TAP version|#|1..|Bail out) isn't to be ignored.

I don't think we're following spec 13.

>
> And then there's:
>
>
> "
> Anything else
>
> Any output that is not a version, a plan, a test line, a YAML block, a
> diagnostic or a bail out is incorrect. How a harness handles the incorrect
> line is undefined. Test::Harness silently ignores incorrect lines, but will
> become more stringent in the future. TAP::Harness reports TAP syntax errors at
> the end of a test run.
> "
>
> If I were to to implement a tap parser this wouldn't make me ignore lines.
>
>
> Contrasting to that:
> http://testanything.org/tap-specification.html
>
> "
> Anything else
>
> A TAP parser is required to not consider an unknown line as an error but may
> optionally choose to capture said line and hand it to the test harness,
> which may have custom behavior attached. This is to allow for forward
> compatability. Test::Harness silently ignores incorrect lines, but will
> become more stringent in the future. TAP::Harness reports TAP syntax errors
> at the end of a test run.
> "
>
> I honestly don't know what to make of that. Parsers are supposed to ignore
> unknown lines, Test::Harness silently ignores, but also "TAP::Harness reports
> TAP syntax errors"? This seems like a self contradictory mess.
>

I agree it's a mess. Both of these "specs" are incredibly loose. I guess
that happens when the spec comes after the implementation.

cheers

andrew

--
Andrew Dunstan
EDB: https://www.enterprisedb.com