Debugging

DebuggingAccess ErrorsRollout Not WorkingRuntime Debugging

Access Errors

You may at times get an error saying that an action is forbidden. This may or may not be intentional. RBAC is complicated. Please contact Hebe with the full message.

kubectl get roles
Error from server (Forbidden): roles.rbac.authorization.k8s.io is forbidden: User "you" cannot list resource "roles" in API group "rbac.authorization.k8s.io" in the namespace "default"

 

Rollout Not Working

So something's gone wrong and your Deployment doesn't seem to be rolling out properly.

First, you can check the status of the Pods with:

kubectl get pods -l app=DEPLOYMENT_NAME
Name Ready Status Restarts Age
hello-node-5776cf4bcf-4sjrt 0/1 CrashLoopBackOff 0 312s
hello-node-7694cbfdd4-zfrms 1/1 Running 0 4d

You can see if something is going wrong from the Status column. Your newly deployed pod (distinguished by being the youngest) has run into a problem.

The Status isn't always intuitive, so for more detail check for the reasons by looking at the output of:

kubectl describe pods POD_NAME
...

Events:
Type Reason Age From Message
...
Warning BackOff 92s kubelet Back-off restarting failed container

This is often super helpful, but sometimes you need more detail about what's going on in the Pod itself. In this case, use:

kubectl logs PODNAME
Error: blah blah blah

If you realize there's a problem with the most recent rollout, you can go back to the previous one with a quick:

kubectl rollout undo deploy DEPLOYMENT_NAME
deployment.apps/hello-node rolled back

 

Runtime Debugging

To stream the Pod logs to standard output:

kubectl logs PODNAME
I'm talking to the server
Now I'm doing some processing
Maybe I should write clearer logs
Nah I'm sure it'll be fine
Okay done with processing sending it back

This'll give you the normal logs for a single Pod. For prettier and more complicated things, I recommend stern.

To exec into the running container and look around:

kubectl exec -it PODNAME -- bash
> bash-5.0#