Re: Differential Code Coverage report for Postgres

Hi,

On Fri, 5 Sept 2025 at 22:14, Álvaro Herrera <alvherre(at)kurilemu(dot)de> wrote:
>
> Thanks for working on this!
>
> On 2025-Sep-05, Nazir Bilal Yavuz wrote:
>
> > 1- One Github Actions run takes ~50 minutes for now and since this
> > runs daily it is ~1500 minutes in total for a month. If you include
> > manual triggers and failures, it is more than 1500 minutes. Github
> > allows 2000 minutes free usage of Github Actions for a month. So, if
> > we increase the time (by generating another report), then it may
> > exceed the free usage limit. Right now, I install Postgres
> > dependencies on each task; I will work on it to reduce time.
>
> I think the goal should be to run this on the pginfra machines. I
> wasn't really thinking about doing this until pginfra upgraded to Debian
> trixie, because that would have the lcov version we need; but since you
> also seem to be cloning lcov, maybe we do that also in pginfra and thus
> we could do it right away. Such a machine can use all the CPU time it
> needs. (In fact, in a totally overkill approach, we currently run the
> report every four hours or something like that. It's easy to run once
> or twice daily and run all branches instead.)

I agree with this. I cloned lcov because of the reason you wrote
above, making it work on more machines.

>
> > 2- If we want to show both reports on the same page, then it may
> > require a bit of HTML coding. I have no experience on that but I do
> > not think it will be hard.
>
> I think it's enough to have multiple reports available and a link to
> each.
>
> I see that the report shows 7, 30, 360 days of change. I wonder how
> that works, and how can we best make use of that. Are you supposed to
> run the tests every day and then run the diff with the tests from one
> week, one month, one year ago? Or is the script running the test for
> 7, 30, 360 days ago every time and comparing those with the current one?

I think it works like that, we run the tests once for master and
REL_18_STABLE; then genhtml calculates the `current commit's day
(which is the latest commit at the master) - related commit's day` and
puts the commit in the correct date bin. This is automatically done by
genhtml. genhtml binary accepts the '--date-bins' option and it is set
to '--date-bins 1,7,30,360' in the script.

> Maybe what we want is not "x days ago" but instead compare current
> branch HEAD with each previous minor release (up to the merge-base with
> master). Does genhtml let you do that? If not, "X days ago" is
> probably good enough.

I do not think so. I think if we want to do that, we need to run tests
and generate lcov reports of each minor release and run a genhtml for
each minor release's lcov files along with the master's lcov file.

--
Regards,
Nazir Bilal Yavuz
Microsoft