Build gardener/gardener images with prow/kaniko

[oliver-goetz]

/kind enhancement

What would you like to be added:
Now that the prow image-builder app is completed and is building ci-infra apps productively and gardener apps in a test environment, we could think about the next steps to build gardener/gardener images in prow.

We could proceed in two steps:

1. Use prow for test builds replacing concourse-ci/publish job
2. Include prow image build into release pipeline

Step 1: test builds

When investigating the I noticed that the images built by concourse-ci/publish PR test are pushed to eu.gcr.io/gardener-project/gardener which is the same location where we are pushing our release images too.
This does not look ideal from my point of view, because

• we cannot guarantee that those image come from a trusted origin
• I question, that we really need to push those images anywhere. From my perspective these are primary tests if the PR could be built at all.

I would like to propose a slightly different setup for prow

• deactivate concourse-ci/publish PR test
• create a new presubmit job for gardener/gardener which runs a plain kaniko pod without any target but with the --no-push flag
  • the pod will build the entire Dockerfile
  • it neither uses cache nor pushes the resulting images anywhere, so it can run in safely the prow-work cluster without any credentials for gcr.io
  • the goal of this job is to find out, if we are able to build the PR
• Change the destination of post-gardener-build-images job to eu.gcr.io/gardener-project/gardener
  • we will build the current state of master branch
  • exclude commits which are changing the VERSION file only, that we do not build release images

Those are the gains from my point of view:

• we do not push images from "untrusted" origins to eu.gcr.io/gardener-project/gardener anymore
• we are able to validate if the prow images are working more easily
• we already reduce the load of our concourse systems

Step 2: release builds

After we verified that our builds are working as expected we can start building our release images in prow.

• create a build prow job similar to post-gardener-build-images for the release branches which is triggered by changes to VERSIONS file
• deactivate build in concourse pipeline but keep the rest
• gardener-robot-ci-* would initiate the release build by pushing a new VERSION file to the release branches

Step 2 is still a draft. I cannot see the entire concourse release pipeline yet.

Why is this needed:
Reduce load on concourse pipelines.
Improve development experience.
Build images of a trusted origin only.

[oliver-goetz]

@timebertt @rfranzke what do you think?

[rfranzke]

Thanks @oliver-goetz, looking forward!

We sometimes need the images built from PRs for local deployment/validation, so I would definitely keep it. Also, I would keep the repository for now to avoid additional complexity (using a separate repo can always be improved later on). In short: Let's for now focus only on moving the image builds to Prow while keeping the "semantics" we have today.

As for the release jobs, I currently cannot oversee how this plays together with image tagging etc. We have not yet even moved the verification jobs for the release job (https://concourse.ci.gardener.cloud/teams/gardener/pipelines/gardener-master/jobs/master-release-job) 👀

[oliver-goetz]

Okay, introducing a presubmit job, which is building images should be easy.

The release jobs need definitely some more research. I cannot even see the release job. When I try it is loading... forever. I simply might not have permissions for that.

[oliver-goetz]

Depending on how often "sometimes" is, the PR builds pushed to the registry could be optional though, like /test pull-gardener-test-build. A build only takes 5 minutes.

[timebertt]

Thanks for outlining your proposal!

Regarding step 1:
I'm definitely open for all idea that are included here, but I would also be fine with moving to prow without changing semantics as a first step.
So no strong opinion from my side.

Regarding step 2:
I'm afraid, I don't really have an overview over the release mechanics and how things will interact.
I would go for completing step 1 first and move all image builds (master + release branches – except release jobs). Once that's done, I would propose to sit down together with our concourse experts and clarify compliance implications of moving release verification and build steps to prow.

[oliver-goetz]

/reopen
Step 1 finished with #181
An approach for step 2 needs still to be investigated

[gardener-prow]

@oliver-goetz: Reopened this issue.

Details

In response to this:

/reopen
Step 1 finished with #181
An approach for step 2 needs still to be investigated

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository.

[gardener-ci-robot]

The Gardener project currently lacks enough contributors to adequately respond to all issues and PRs.
This bot triages issues and PRs according to the following rules:

• After 90d of inactivity, lifecycle/stale is applied
• After 30d of inactivity since lifecycle/stale was applied, lifecycle/rotten is applied
• After 30d of inactivity since lifecycle/rotten was applied, the issue is closed

You can:

• Mark this issue or PR as fresh with /remove-lifecycle stale
• Mark this issue or PR as rotten with /lifecycle rotten
• Close this issue or PR with /close

/lifecycle stale

[timebertt]

/remove-lifecycle stale