Currently the `gsutil rsync` command does not support the `-z` or `-Z`
options available in `gsutil cp` to compress files locally via gzip
before uploading
(https://github.com/GoogleCloudPlatform/gsutil/issues/579). As
https://cloud.google.com/storage/docs/gsutil/commands/cp states:
When you specify the -z option, the data from your files is
compressed before it is uploaded, but your actual files are left
uncompressed on the local disk. The uploaded objects retain the
Content-Type and name of the original files, but have their
Content-Encoding metadata set to gzip to indicate that the object data
stored are compressed on the Cloud Storage servers and have their
Cache-Control metadata set to no-transform.
about.gitlab.com is currently serving uncompressed HTML files because
`Cache-Control: max-age=0` is set (see
https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/87045),
and Fastly has a custom rule to skip HTML files and therefore won't
cache them.
This patches the `rsync.py` using
https://github.com/GoogleCloudPlatform/gsutil/pull/1430 to support these
command-line options so local gzip compression can be performed.
Relates to
https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/14852
This refactors our custom image building to utilize GitLab parallel jobs
with a matrix.
This makes it easier to parse what kind of matrix we are going to build
in our docker images. Furthermore instead of splitting the image name,
we can simply pull the versions of the tools from the environment
variables.
The ultimative reason: Dogfooding
The ruby-2.7 tags will always contain the latest
patch released for that minor version.
This allows us to ship patches more easily but
retains backwards compatibility with existing
repos asking for 2.7.2.
With https://gitlab.com/gitlab-com/www-gitlab-com/-/merge_requests/81890
the last usage of pandoc / LaTeX was removed from the www-gitlab-com
repo. This happened more than 2 months ago, so removing it from the
image should be rather safe, especially considering we use merge trains.
It reduces the image size by around 500 MB.
The pgbouncer build is currently broken because of a missing build
dependency. This went unnoticed because the build script previously
didn't report any build errors.
Fix the build by installing the missing libevent-dev dependency.
The install-pgbouncer script is not currently reporting any build errors
to the caller, except if the final `rm -rf` fails. Fix the issue by
executing `set -xeuo pipefail` to properly report errors of any
unchecked commands and chaining commands with `&&`'s. While the latter
shouldn't be necessary anymore because of changed options, it's easy
enough to do for this script.
With this change, we now properly notice that building pgbouncer fails
currently because of a missing libevent dependency. This failure is
fixed in a subsequent commit.
The minimum required version of git nowadays is v2.29.0, and thus we
don't create images with versions older than that anymore. We can thus
safely drop their hashes from the custom-docker-build script.
Also node-14 will be changed from 14.15.4 to 14.16.0 as intended,
while node-12 will not be changed staying with 12.4.0.
Signed-off-by: Takuya Noguchi <takninnovationresearch@gmail.com>
