[Enhancement] Docs for ImageValidatingPolicy don't explain ECR setup details

[kingdonb]

Description

I was just testing ImageValidatingPolicy, I am a new-ish Kyverno user, and my use case is related to AWS+ECR.

When I set this up, I realized that it was going to do much more than a ClusterPolicy could do - my earlier use cases were focused around making sure that only approved registries can go in the cluster. This tool will fetch the digest, and do many other things! Great... but it needs permissions. I searched the docs for anything related to ECR and IAM, and came up with one blog article that was nowhere near ImageValidatingProxy.

I don't know if comprehensive docs are in order, that would be great if there was someone from AWS who could contribute that, but for me certain basic things seemed undiscoverable without experimenting - for example, which controller needs the IRSA role assignment? I guessed admission-controller first, (nope), then background-controller, turns out it was the reports-controller.

And when I got it working, I noticed some noise in the logs about ecr credentials being rewritten constantly because there was no write access in the place where they're supposed to be cached. So, I attached a volume there, and now that error is gone. I assume it's properly caching creds now, but there was no documentation supporting any of this!

I would be happy to contribute some changes to the ImageValidatingPolicy docs if Kyverno is receptive to this, but my experience is still quite limited.

Slack discussion

No response

[JimBugwadia]

Hi @kingdonb - thanks for opening this issue, to help improve the docs.

Will be great if you can contribute the steps you had to perform to make this work with Amazon ECR.

The write error seems to be a bug.

You can either contribute this as a blog post, or as a sub-section in the docs. If its the latter, we can create platform specific sections so that we can add docs for other cloud providers as well.

[kingdonb]

Thanks for responding! I'll put together what I've learned and try to find a good spot for it in the ImageValidatingPolicy page. (In the mean time let's just be sure that I get all the information down here, in the issue report...)

mutateDigest
verifyDigest

It seemed like it was easy to trigger this issue by using a cluster with private ECRs in it, and installing an ImageValidatingPolicy with either or both of these validationConfiguration options.

This was the IAM policy I wound up using:

iamPolicy: |
  {
      "Version": "2012-10-17",
      "Statement": [
          {
              "Effect": "Allow",
              "Action": "ecr:GetAuthorizationToken",
              "Resource": "*"
          },
          {
              "Effect": "Allow",
              "Action": [
                  "ecr:BatchCheckLayerAvailability",
                  "ecr:GetDownloadUrlForLayer",
                  "ecr:BatchGetImage",
                  "ecr:DescribeImages"
              ],
              "Resource": [
                  "arn:aws:ecr:us-east-1:123456789012:repository/*",
                  "arn:aws:ecr:us-east-1:012345678901:repository/*"
              ]
          }
      ]
  }

That worked, when it was applied to the reports controller (the details of how I did that are wrapped in Crossplane and a bit too complicated to share, but suffice it to say this should be straightforward to anyone who is already using IAM and IRSA, so long as it's clear which controller needs the role attached to its service account.)

I'm deploying using the Flux Kustomize Controller & OCI artifact, this patch adds the volume mount as an emptyDir (no PV)

- patch: |-
    apiVersion: v1/apps
    kind: Deployment
    metadata:
      name: not-used
    # Patch for reports controller
    spec:
      template:
        spec:
          containers:
          - name: controller
            volumeMounts:
            - name: ecr-cache
              mountPath: /home/nonroot/.ecr
          volumes:
          - name: ecr-cache
            emptyDir: {}
  target:
    kind: Deployment
    labelSelector: "app.kubernetes.io/component=reports-controller"

It may or may not have also needed this patch (I think it is needed)

- patch: |-
    apiVersion: v1
    kind: ServiceAccount
    metadata:
      name: not-used
    automountServiceAccountToken: true
  target:
    kind: ServiceAccount
    labelSelector: "app.kubernetes.io/component=reports-controller"

I'm not sure I would consider this a bug, but it's fine if you do. I'm not sure how you document this or even attempt to fix it globally without making a very AWS-specific change - I mean, but it could be a nice addition to the Helm chart as an optional choice (if it isn't already in there!)

Will put a note in my tickler and try to come back to help close this.

[Tusharjamdade]

Hi @JimBugwadia,

I wanted to check whether this issue is still considered valid. If so, I’d be happy to put together a documentation PR covering the AWS ECR setup details for ImageValidatingPolicy, based on the findings discussed here.

Please let me know if that would be helpful and if there’s a preferred location or format for the documentation.