OpenDAX v4 docs

Common Kubernetes Operations

As soon as you're connected to your VM, you can interact with the MicroK8s cluster running there by using kubectl CLI. You can find the most common Kubernetes operations below.
Please, pay attention that Kubernetes workloads (Pods, Deployments, Statefulsets, etc.) are distributed between different namespaces. So for every kubectl command related to workloads you should specify the namespace using the -n flag. To list existing namespaces on the Kubernetes cluster, run kubectl get namespaces. Also, you can use --all-namespaces instead of -n <namespace> to apply a command to all namespaces. Ex: kubectl get pods --all-namespaces

Operations with Kubernetes Pod

Common operations for Pods include get, delete, logs, describe, and exec.
  • kubectl get pods -n <namespace> - list all Pods in a given namespace
  • kubectl get pods -o wide -n <namespace> - list all Pods in a given namespace, with more details
  • kubectl delete pods <pod_name> -n <namespace>- delete a Pod identified by its name. A new Pod will be automatically created by Kubernetes if a corresponding Deployment/StatefulSet/DaemonSet exists
  • kubectl logs <pod_name> -n <namespace> - dump Pod logs to the stdout
  • kubectl logs -f <pod_name> -n <namespace> - stream Pod logs to the stdout
  • kubectl describe pod <pod_name> -n <namespace> - fetch details about Pod.
Output example:
> kubectl describe pod nginx-deployment-67d4bdd6f5-w6kd7 -n core
Name: nginx-deployment-67d4bdd6f5-w6kd7
Namespace: default
Priority: 0
Node: kube-worker-1/
Start Time: Thu, 17 Feb 2022 16:51:01 -0500
Labels: app=nginx
Annotations: <none>
Status: Running
IP: 2001:db8::1
Controlled By: ReplicaSet/nginx-deployment-67d4bdd6f5
Container ID: containerd://5403af59a2b46ee5a23fb0ae4b1e077f7ca5c5fb7af16e1ab21c00e0e616462a
Image: nginx
Image ID:[email protected]:2834dc507516af02784808c5f48b7cbe38b8ed5d0f4837f16e78d00deb7e7767
Port: 80/TCP
Host Port: 0/TCP
State: Running
Started: Thu, 17 Feb 2022 16:51:05 -0500
Ready: True
Restart Count: 0
cpu: 500m
memory: 128Mi
cpu: 500m
memory: 128Mi
Environment: <none>
/var/run/secrets/ from kube-api-access-bgsgp (ro)
Type Status
Initialized True
Ready True
ContainersReady True
PodScheduled True
Type: Projected (a volume that contains injected data from multiple sources)
TokenExpirationSeconds: 3607
ConfigMapName: kube-root-ca.crt
ConfigMapOptional: <nil>
DownwardAPI: true
QoS Class: Guaranteed
Node-Selectors: <none>
Tolerations: op=Exists for 300s op=Exists for 300s
Type Reason Age From Message
---- ------ ---- ---- -------
Normal Scheduled 34s default-scheduler Successfully assigned default/nginx-deployment-67d4bdd6f5-w6kd7 to kube-worker-1
Normal Pulling 31s kubelet Pulling image "nginx"
Normal Pulled 30s kubelet Successfully pulled image "nginx" in 1.146417389s
Normal Created 30s kubelet Created container nginx
Normal Started 30s kubelet Started container nginx
  • kubectl exec -it finex-55ffdc954d-cslmp -- /bin/bash - execute a command in an interactive shell session on a given Pod If Pod doesn't have bash installed by default (and when you got OCI runtime exec failed: exec failed: container_linux.go:349: starting container process caused "exec: \"bash\": executable file not found in $PATH": unknown running previous command) use sh:
  • kubectl exec -it finex-55ffdc954d-cslmp -- /bin/sh

Operations with Kubernetes Deployment

Basic operations to do with Deployment include get, describe, edit and scale.
  • kubectl get deployments -n <namespace> - list all Deployments in a given namespace
  • kubectl describe deployment <deployment_name> -n <namespace> - display the detailed state of one or more Deployments
  • kubectl edit deployment/<deployment_name> -n <namespace> - edit Deployment resource in a text editor. This is equivalent to first get the resource, edit it, and then apply the resource with the updated version
  • kubectl scale --replicas=3 deployment/mysql -n <namespace> - scale a Deployment named mysql to 3 Pods
Output example of kubectl describe deployment <deployment_name> -n <namespace>:
> kubectl describe deployments nginx-deployment -n core
Name: nginx-deployment
Namespace: default
CreationTimestamp: Thu, 30 Nov 2017 10:56:25 +0000
Labels: app=nginx
Selector: app=nginx
Replicas: 3 desired | 3 updated | 3 total | 3 available | 0 unavailable
StrategyType: RollingUpdate
MinReadySeconds: 0
RollingUpdateStrategy: 25% max unavailable, 25% max surge
Pod Template:
Labels: app=nginx
Image: nginx:1.16.1
Port: 80/TCP
Environment: <none>
Mounts: <none>
Volumes: <none>
Type Status Reason
---- ------ ------
Available True MinimumReplicasAvailable
Progressing True NewReplicaSetAvailable
OldReplicaSets: <none>
NewReplicaSet: nginx-deployment-1564180365 (3/3 replicas created)
Type Reason Age From Message
---- ------ ---- ---- -------
Normal ScalingReplicaSet 2m deployment-controller Scaled up replica set nginx-deployment-2035384211 to 3
Normal ScalingReplicaSet 24s deployment-controller Scaled up replica set nginx-deployment-1564180365 to 1
Normal ScalingReplicaSet 22s deployment-controller Scaled down replica set nginx-deployment-2035384211 to 2
Normal ScalingReplicaSet 22s deployment-controller Scaled up replica set nginx-deployment-1564180365 to 2
Normal ScalingReplicaSet 19s deployment-controller Scaled down replica set nginx-deployment-2035384211 to 1
Normal ScalingReplicaSet 19s deployment-controller Scaled up replica set nginx-deployment-1564180365 to 3
Normal ScalingReplicaSet 14s deployment-controller Scaled down replica set nginx-deployment-2035384211 to 0

Operations with Kubernetes Jobs

A Job creates one or more Pods and will continue to retry execution of the Pods until a specified number of them successfully terminate.
Common operations to do with Jobs include get, describe.
  • kubectl get jobs -n <namespace> - list all Jobs in a given namespace
  • kubectl describe jobs <job_name> -n <namespace> - display the detailed state of one or more Jobs
  • kubectl get jobs/<job_name> -o yaml - return Job definition in yaml format. Can be used to check Job status and history.
  • kubectl get pods --selector=job-name=<job_name> --output=jsonpath='{.items[*]}' - list all Pods related to the given job
To get logs from job related pods:
pods=$(kubectl get pods --selector=job-name=pi --output=jsonpath='{.items[*]}' -n <namespaces>)
echo $pods # will list pods which created by the given job
kubectl logs -f $pods -n <namespace> # will show logs for this pods
Output example of kubectl get jobs/<job_name> -o yaml where you can find job status and history:
apiVersion: batch/v1
kind: Job
# .metadata and .spec omitted
- lastProbeTime: "2021-02-05T13:14:33Z"
lastTransitionTime: "2021-02-05T13:14:33Z"
status: "True"
type: Suspended
startTime: "2021-02-05T13:13:48Z"
If you have problems with Helm release installation or upgrade related to the hooks/jobs check this doc for troubleshooting instructions

How to check K8s certificates status

To see certificates and their status in your Kubernetes cluster run:
kubectl get certificate --all-namespaces
To get some additional information about specific certificate run:
kubectl describe certificate <name> -n <namespaces>

Make sure the certificate issue success

  1. 1.
    Check clusterissuer resource
kubectl get clusterissuer -n core
letsencrypt-prod True 7d7h
a. If READY is True continue to the 2nd step.
b. If READY is False we need to re-install clusterissuer and restart the certificate issuing process
kubectl delete clusterissuer letsencrypt-prod -n core
kubectl delete po -l '' -n core
# re-apply clusterissuer
kubectl apply -f /home/ubuntu/resources/clusterissuer.yaml
c. Repeat 1st step.
  1. 2.
    Check certificaterequest resource
kubectl get certificaterequest --all-namespaces
loading-page odax-loading-page-n9s9c True True zerossl-prod system:serviceaccount:core:cert-manager 2d23
If the READY is not True you can run describe command to check the details:
kubectl describe certificaterequest <name> -n <namespace>
  1. 3.
    Check certificate order resource
kubectl get order --all-namespaces
loading-page odax-loading-page-n9s9c-2677649829 valid 2d23h
Same as in the previous case, to check details of order if it's not valid:
kubectl describe order <name> -n <namespace>

Validate TLS certificate date and CN/subject

Print out dates of the cert (check if certificate is not expired):
openssl s_client -connect <domain>:443 -showcerts -servername <domain> |openssl x509 -noout -dates
Print out the subject/CN of the cert:
openssl s_client -connect <domain>:443 -showcerts -servername <domain> |openssl x509 -noout -subject

How to update a Deployment’s image and pull secret using kubectl

To update the image of a Deployment, you can use the command kubectl edit deployment/<deployment_name> -n <namespace> to interactively edit a Deployment, its image is defined by .spec.template.spec.containers[0].image field.
A faster way for that would be:
kubectl set image deployment/<deployment_name> <container_name>=<image> -n <namespace>
<container_name> usually matches the Deployment name, you can get it by running:
kubectl get deployment <deployment_name> -o jsonpath="{.spec.template.spec.containers[*].name} -n <namespace>"
To update secret use:
kubectl edit secrets <secret_name> -n <namespace>
If you'd like to update a specific field inside a Secret's .data key, (be aware that instead of bar you should have base64 encoded data): kubectl get secret <secret_name> -o json -n <namespace> | jq '.data["foo"]="bar"' | kubectl apply -f - After that, you have to delete your pod so changes will apply. To do that, use kubectl delete pod <pod_name> -n <namespace>