Kubernetes Security Hands-On

Minikube

Start Minikube

minikube delete

minikube start --cni=cilium --memory=4096

In case Minikube has errors starting try

minikube delete --all --purge

rm -rf ~/.minikube/

and then the start command from above again.

RBAC

User RBAC

kubectl create namespace rbac-example

kubectl create serviceaccount -n rbac-example myuser

kubectl create rolebinding -n rbac-example myuser-view --clusterrole=view --serviceaccount=rbac-example:myuser

alias kubectl-user='kubectl --as=system:serviceaccount:rbac-example:myuser'

kubectl-user get pod -n rbac-example

kubectl-user get pod

kubectl get pod

kubectl-user auth can-i get pods -n default

kubectl create rolebinding -n default myuser-default-view --clusterrole=view --serviceaccount=rbac-example:myuser

kubectl-user auth can-i get pods -n default

kubectl-user get pod

kubectl-user auth can-i get pods --all-namespaces

Some examples

Admin access to a specific namespace:

kind: RoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: development-admin
  namespace: development
subjects:
  - kind: User
    name: jane
    apiGroup: rbac.authorization.k8s.io
  - kind: Group
    name: dev-admin
    apiGroup: rbac.authorization.k8s.io  
roleRef:
  kind: ClusterRole
  name: admin
  apiGroup: rbac.authorization.k8s.io

Read access to the whole cluster:

kind: ClusterRoleBinding
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  name: cluster-viewer
subjects:
  - kind: User
    name: jane
    apiGroup: rbac.authorization.k8s.io
  - kind: Group
    name: cluster-view
    apiGroup: rbac.authorization.k8s.io  
roleRef:
  kind: ClusterRole
  name: view
  apiGroup: rbac.authorization.k8s.io

Workload RBAC

cat prometheus.yaml

kubectl create -f prometheus.yaml

kubectl -n kube-system get pods

kubectl -n kube-system logs prometheus-0

cat prometheus-rbac.yaml

kubectl create -f prometheus-rbac.yaml

kubectl -n kube-system delete pod prometheus-0

kubectl -n kube-system get pods

kubectl -n kube-system logs prometheus-0

kubectl delete -f prometheus.yaml

Network Policies

Ingress Policies

kubectl create ns restricted

kubectl run -n restricted --image=nginx nginx-app --port=80

kubectl -n restricted get pod -o wide

kubectl run utils \
  --restart Never \
  --image webwurst/curl-utils \
  --command sleep 3000

kubectl exec utils -- curl IPOFNGINX:80

Deny all (ingress) traffic to pods in that namespace

cat default-deny.yaml

kubectl create -n restricted -f default-deny.yaml

kubectl exec utils -- curl IPOFNGINX:80

Allow traffic from busybox to nginx

kubectl label ns default name=default

cat allow-nginx.yaml

kubectl create -n restricted -f allow-nginx.yaml

kubectl exec utils -- curl IPOFNGINX:80

kubectl -n restricted run bla \
  --restart Never \
  --image webwurst/curl-utils \
  --command sleep 3000

kubectl -n restricted exec bla -- curl IPOFNGINX:80

Allow all traffic within namespace

kubectl label ns restricted name=restricted

cat allow-within-ns.yaml

kubectl create -f allow-within-ns.yaml

kubectl -n restricted exec bla -- curl IPOFNGINX:80

Egress Policies

Egress to pods within a cluster

kubectl -n restricted exec bla -- nslookup google.de

Deny all egress in namespace

cat default-deny-egress.yaml

kubectl -n restricted create -f default-deny-egress.yaml

kubectl -n restricted exec bla -- nslookup google.de

Allow DNS lookups

kubectl label ns kube-system name=kube-system

cat allow-dns.yaml

kubectl -n restricted create -f allow-dns.yaml

kubectl -n restricted exec bla -- nslookup google.de

Egress to IPs outside the cluster

kubectl -n restricted exec bla -- ping 9.9.9.9

Allow

cat allow-external.yaml

kubectl -n restricted create -f allow-external.yaml

kubectl -n restricted exec bla -- ping 9.9.9.9

PSP

Note, as of this writing, PSPs are deprecated and will be replaced in the near future. Thus, the next steps might be useful if you already have the needs or want to learn more about pod security contexts, but not necessary.

Running minikube with PSP is not trivial, you can start it by running

minikube start \
  --extra-config=apiserver.enable-admission-plugins="NamespaceLifecycle,LimitRanger,ServiceAccount,DefaultStorageClass,DefaultTolerationSeconds,NodeRestriction,PodSecurityPolicy,MutatingAdmissionWebhook,ValidatingAdmissionWebhook,ResourceQuota"

This is will take a lot of time as minikube wants to verify it is working. It will finally result in a failed start, but minikube should actually be running. Just the Kubernetes components (besides API server) won't be up. To get them running you can apply following manifest that contain default PSPs and bindings for the main components.

kubectl create -f minikube-psp.yaml

Once you have a working minikube with PSP enabled you should check out https://kubernetes.io/docs/concepts/policy/pod-security-policy/#example and https://docs.giantswarm.io/guides/securing-with-rbac-and-psp/#running-applications-that-need-privileged-access.

Sources, Further Reading and Resources

• https://docs.giantswarm.io/getting-started/rbac-and-psp/
• https://github.com/appscodelabs/tasty-kube/tree/master/minikube/1.10/psp
• https://kubernetes.io/docs/concepts/policy/pod-security-policy/#example
• https://kubernetes.io/docs/concepts/services-networking/network-policies/