I am having lots of issues that seem to stem from upgrading my kubernetes cluster to the latest version (1.23.5). I initially had some issues with the cluster itself and the nodes but that seems to be fixed and the cluster seems to be healthy, at least it says that when I run kops validate cluster. The issue I am facing now is that my ingress-nginx pods are not running which means my load balancer has nothing to point to and therefore I cannot reach my application even though the application pods are running without issue. I used helm to create the ingress-nginx resources and will paste the files that I am trying to use below to upgrade. I have tried multiple things around this and I think the major thing I am missing is the IngressClass stuff and I have tried to include that in multiple places but I am not seeing how to do that. My cluster only has one ingress controller and there is an Ingress instance defined in the deployment for each instance of the application. You will also see the AppVersion is 0.24.0, I have tried bumping that in multiple ways and using different images in the deployment.yaml.
rbac.yml
apiVersion: v1
kind: ServiceAccount
metadata:
name: {{ .Chart.Name }}-serviceaccount
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
name: {{ .Chart.Name }}-clusterrole
rules:
- apiGroups:
- ""
resources:
- configmaps
- endpoints
- nodes
- pods
- secrets
verbs:
- list
- watch
- apiGroups:
- ""
resources:
- nodes
verbs:
- get
- apiGroups:
- ""
resources:
- services
verbs:
- get
- list
- watch
- apiGroups:
- "extensions"
resources:
- ingresses
verbs:
- get
- list
- watch
- apiGroups:
- ""
resources:
- events
verbs:
- create
- patch
- apiGroups:
- "extensions"
resources:
- ingresses/status
verbs:
- update
---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
name: {{ .Chart.Name }}-role
rules:
- apiGroups:
- ""
resources:
- configmaps
- pods
- secrets
- namespaces
verbs:
- get
- apiGroups:
- ""
resources:
- configmaps
resourceNames:
# Defaults to "<election-id>-<ingress-class>"
# Here: "<ingress-controller-leader>-<nginx>"
# This has to be adapted if you change either parameter
# when launching the nginx-ingress-controller.
- "ingress-controller-leader-nginx"
verbs:
- get
- update
- apiGroups:
- ""
resources:
- configmaps
verbs:
- create
- apiGroups:
- ""
resources:
- endpoints
verbs:
- get
---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
name: {{ .Chart.Name }}-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: Role
name: {{ .Chart.Name }}-role
subjects:
- kind: ServiceAccount
name: {{ .Chart.Name }}-serviceaccount
namespace: {{ .Release.Namespace }}
---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
name: {{ .Chart.Name }}-clusterrole-nisa-binding
roleRef:
apiGroup: rbac.authorization.k8s.io
kind: ClusterRole
name: {{ .Chart.Name }}-clusterrole
subjects:
- kind: ServiceAccount
name: {{ .Chart.Name }}-serviceaccount
namespace: {{ .Release.Namespace }}
service.yaml
---
# Main service ingesting http traffic
kind: Service
apiVersion: v1
metadata:
name: loadbalancer-ingress
labels:
app.kubernetes.io/managed-by: Helm
annotations:
{{- if .Values.loadbalancer.cert }}
service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values.loadbalancer.cert | quote }}
service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "{{- range .Values.loadbalancer.ports -}}{{- if .ssl -}}{{ .name }},{{- end -}}{{- end -}}"
{{- end }}
service.beta.kubernetes.io/aws-load-balancer-backend-protocol: {{ .Values.loadbalancer.backend_protocol | quote }}
service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
type: LoadBalancer
selector:
pod: {{ .Chart.Name }}
ports:
{{- range .Values.loadbalancer.ports }}
- name: {{ .name }}
port: {{ .port }}
targetPort: {{ .targetPort }}
{{- end }}
---
# Dummy service to stop the controller from nagging about ingress-nginx service
kind: Service
apiVersion: v1
metadata:
name: ingress-nginx
labels:
app.kubernetes.io/managed-by: Helm
spec:
ports:
- name: http
port: 10254
targetPort: 10254
selector:
pod: {{ .Chart.Name }}
---
deployment.yaml
---
apiVersion: apps/v1
kind: DaemonSet
metadata:
name: {{ .Chart.Name }}-controller
spec:
selector:
matchLabels:
pod: {{ .Chart.Name }}
template:
metadata:
labels:
pod: {{ .Chart.Name }}
annotations:
prometheus.io/port: "10254"
prometheus.io/scrape: "true"
fluentbit.io/parser: k8s-nginx-ingress
spec:
serviceAccountName: {{ .Chart.Name }}-serviceaccount
containers:
- name: {{ .Chart.Name }}-controller
image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:{{ .Chart.AppVersion }}
args:
- /nginx-ingress-controller
- --configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-nginx-configuration
- --tcp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-tcp-services
- --udp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-udp-services
- --publish-service=$(POD_NAMESPACE)/loadbalancer-ingress
- --annotations-prefix=nginx.ingress.kubernetes.io
securityContext:
allowPrivilegeEscalation: true
capabilities:
drop:
- ALL
add:
- NET_BIND_SERVICE
# www-data -> 33
runAsUser: 33
env:
- name: POD_NAME
valueFrom:
fieldRef:
fieldPath: metadata.name
- name: POD_NAMESPACE
valueFrom:
fieldRef:
fieldPath: metadata.namespace
ports:
- name: http
containerPort: 80
- name: metrics
containerPort: 10254
livenessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
readinessProbe:
failureThreshold: 3
httpGet:
path: /healthz
port: 10254
scheme: HTTP
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 10
configmap.yaml
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-nginx-configuration
data:
use-proxy-protocol: "false"
use-forwarded-headers: "true"
server-tokens: "false"
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-tcp-services
---
kind: ConfigMap
apiVersion: v1
metadata:
name: {{ .Chart.Name }}-udp-services
Chart.yaml
name: ingress-nginx
description: Cluster - Ingress Controller
version: 1
apiVersion: v1
appVersion: "0.24.0"
values.yaml
loadbalancer:
backend_protocol: http
cert: <my-cert>
ports:
- name: http
port: 80
targetPort: 80
ssl: false
- name: https
port: 443
targetPort: 80
ssl: true
Command I am running.
helm upgrade ingress-nginx --install --namespace ingress-nginx ./
Output I currently get.
W0327 19:53:47.472827 8 client_config.go:614] Neither --kubeconfig nor --master was specified. Using the inClusterConfig. This might not work.
I0327 19:53:47.473136 8 main.go:241] "Creating API client" host="https://100.64.0.1:443"
I0327 19:53:47.487201 8 main.go:285] "Running in Kubernetes cluster" major="1" minor="23" git="v1.23.5" state="clean" commit="c285e781331a3785a7f436042c65c5641ce8a9e9" platform="linux/amd64"
I0327 19:53:47.684135 8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0327 19:53:47.689215 8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
E0327 19:53:47.692044 8 main.go:124] "Searching IngressClass" err="ingressclasses.networking.k8s.io \"nginx\" is forbidden: User \"system:serviceaccount:ingress-nginx:ingress-nginx-serviceaccount\" cannot get resource \"ingressclasses\" in API group \"networking.k8s.io\" at the cluster scope" class="nginx"
W0327 19:53:47.692070 8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0327 19:53:47.739577 8 nginx.go:254] "Starting NGINX Ingress controller"
I0327 19:53:47.755865 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-tcp-services", UID:"6115a34f-4f95-4f99-970a-b65477e45808", APIVersion:"v1", ResourceVersion:"103400810", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-tcp-services
I0327 19:53:47.756010 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-udp-services", UID:"fa04d653-a070-4934-a606-a60a7f98ad6a", APIVersion:"v1", ResourceVersion:"103400812", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-udp-services
I0327 19:53:47.756196 8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-nginx-configuration", UID:"3af77ed0-e71c-49e9-bac3-b7c3fada40df", APIVersion:"v1", ResourceVersion:"103400808", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-nginx-configuration
E0327 19:53:48.844980 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:50.385656 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:52.811461 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:57.052727 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:54:05.784219 8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
I0327 19:54:26.690574 8 main.go:187] "Received SIGTERM, shutting down"
I0327 19:54:26.690593 8 nginx.go:372] "Shutting down controller queues"
E0327 19:54:26.690778 8 store.go:178] timed out waiting for caches to sync
I0327 19:54:26.690835 8 nginx.go:296] "Starting NGINX process"
I0327 19:54:26.691321 8 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I0327 19:54:26.691353 8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0327 19:54:26.718477 8 status.go:84] "New leader elected" identity="ingress-nginx-controller-72b9j"
I0327 19:54:26.733451 8 nginx.go:388] "Stopping NGINX process"
2022/03/27 19:54:26 [notice] 28#28: signal process started
I0327 19:54:27.738884 8 nginx.go:401] "NGINX process has stopped"
I0327 19:54:27.738926 8 main.go:195] "Handled quit, awaiting Pod deletion"
I0327 19:54:37.739197 8 main.go:198] "Exiting" code=0
Happy to provide any other details that would be helpful. I really appreciate the help in advance!
EDIT:
The cluster is on AWS and was created using the following k0ps command.
kops create cluster --node-count 2 --node-size t2.medium --zones ap-southeast-2a,ap-southeast-2c --master-size t2.small --master-zones ap-southeast-2c --master-count 1 --networking=calico --authorization RBAC -o yaml --dry-run > my-cluster.yaml
