Re: [PATCH] Add GitLab CI to PostgreSQL

Hi,

On 2023-07-04 23:44:45 +0000, Newhouse, Robin wrote:
> I propose the attached patch to be applied on the 'master' branch
> of PostgreSQL to add GitLab CI automation alongside Cirrus CI in the PostgreSQL repository.
>
> It is not intended to be a replacement for Cirrus CI, but simply suggestion for the
> PostgreSQL project to host centrally a Gitlab CI definition for those who prefer to use
> it while developing/testing PostgreSQL.

One way to avoid duplicated CI definition could be to use for gitlab-ci to use
cirrus-cli to run the cirrus CI tests within gitlab ci.

Realistically I think adding a separate CI definition would entail committers
needing to run that CI at least occasionally. If we make the different CI envs
more similar, that becomes less of a necessity.

> +default:
> + # Base image for builds and tests unless otherwise defined
> + image: fedora:latest
> + # Extend build jobs to have longer timeout as the default GitLab
> + # timeout (1h) is often not enough
> + timeout: 3h

IMO we shouldn't add CI that doesn't complete within well under an hour, it's
too expensive workflow wise.

> +fedora:
> + stage: build
> + variables:
> + GIT_STRATEGY: fetch
> + GIT_SUBMODULE_STRATEGY: normal
> + script:
> + # Install dependencies
> + - yum install -y yum-utils perl
> + - yum-builddep -y postgresql
> + - *build-postgres-def

My experience is that installing dependencies on each run is way too slow to
be practical. I also found that it often causes temporary failures due to
network issues etc. For cirrus-ci we create VM and docker images on a regular
schedule (three times a week right now) - if there's interest in building
fedora containers that'd be easy.

I'd be open to switching one of the cirrus-CI tasks over to fedora, fwiw.

> +# From https://github.com/postgres/postgres/blob/master/.cirrus.yml
> +.create-user: &create-user-def
> + - useradd -m postgres
> + - chown -R postgres:postgres .
> + - mkdir -p ${CCACHE_DIR}
> + - chown -R postgres:postgres ${CCACHE_DIR}
> + - echo '* - memlock 134217728' > /etc/security/limits.d/postgres.conf
> + - su postgres -c "ulimit -l -H && ulimit -l -S"
> + # Can't change container's kernel.core_pattern. Postgres user can't write
> + # to / normally. Change that.
> + - chown root:postgres /
> + - chmod g+rwx /

If we need duplicated stanzas like this, we should instead move them out into
scripts that we can use from different CI environments.

> +# Similar to https://github.com/postgres/postgres/blob/master/.cirrus.yml
> +fedora meson:
> + stage: build
> + variables:
> + GIT_STRATEGY: fetch
> + GIT_SUBMODULE_STRATEGY: normal
> + script:
> + # Meson system only exists on master branch currently

Master and 16 now...

> + - if [ ! -f meson.build ]; then exit 0; fi
> + # Install dependencies
> + - yum install -y yum-utils perl perl-IPC-Run meson ninja-build
> + - yum-builddep -y postgresql
> + # Create postgres user
> + - *create-user-def
> + # Configure
> + - su postgres -c 'meson setup --buildtype=debug --auto-features=disabled -Dtap_tests=enabled build'
> + # Build
> + - su postgres -c 'ninja -C build -j 2'
> + # Minimal test
> + - su postgres -c 'meson test $MTEST_ARGS --num-processes 2 tmp_install cube/regress pg_ctl/001_start_stop'
> + # Run all tests
> + - su postgres -c 'meson test $MTEST_ARGS --num-processes 2'
> + artifacts:
> + when: always # Must be able to see logs
> + paths:
> + - build/meson-logs/testlog.txt

FWIW, that's not enough to be able to debug problems. You really also need the
log files created by failing tests.

Greetings,

Andres Freund