Điểm:1

Updating ingress-nginx helm chart for kubernetes 1.23.5

lá cờ be

I am having lots of issues that seem to stem from upgrading my kubernetes cluster to the latest version (1.23.5). I initially had some issues with the cluster itself and the nodes but that seems to be fixed and the cluster seems to be healthy, at least it says that when I run kops validate cluster. The issue I am facing now is that my ingress-nginx pods are not running which means my load balancer has nothing to point to and therefore I cannot reach my application even though the application pods are running without issue. I used helm to create the ingress-nginx resources and will paste the files that I am trying to use below to upgrade. I have tried multiple things around this and I think the major thing I am missing is the IngressClass stuff and I have tried to include that in multiple places but I am not seeing how to do that. My cluster only has one ingress controller and there is an Ingress instance defined in the deployment for each instance of the application. You will also see the AppVersion is 0.24.0, I have tried bumping that in multiple ways and using different images in the deployment.yaml.

rbac.yml

apiVersion: v1
kind: ServiceAccount
metadata:
  name: {{ .Chart.Name }}-serviceaccount

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRole
metadata:
  name: {{ .Chart.Name }}-clusterrole
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - endpoints
      - nodes
      - pods
      - secrets
    verbs:
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - nodes
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - services
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses
    verbs:
      - get
      - list
      - watch
  - apiGroups:
      - ""
    resources:
      - events
    verbs:
      - create
      - patch
  - apiGroups:
      - "extensions"
    resources:
      - ingresses/status
    verbs:
      - update

---
apiVersion: rbac.authorization.k8s.io/v1
kind: Role
metadata:
  name: {{ .Chart.Name }}-role
rules:
  - apiGroups:
      - ""
    resources:
      - configmaps
      - pods
      - secrets
      - namespaces
    verbs:
      - get
  - apiGroups:
      - ""
    resources:
      - configmaps
    resourceNames:
      # Defaults to "<election-id>-<ingress-class>"
      # Here: "<ingress-controller-leader>-<nginx>"
      # This has to be adapted if you change either parameter
      # when launching the nginx-ingress-controller.
      - "ingress-controller-leader-nginx"
    verbs:
      - get
      - update
  - apiGroups:
      - ""
    resources:
      - configmaps
    verbs:
      - create
  - apiGroups:
      - ""
    resources:
      - endpoints
    verbs:
      - get

---
apiVersion: rbac.authorization.k8s.io/v1
kind: RoleBinding
metadata:
  name: {{ .Chart.Name }}-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: Role
  name: {{ .Chart.Name }}-role
subjects:
  - kind: ServiceAccount
    name: {{ .Chart.Name }}-serviceaccount
    namespace: {{ .Release.Namespace }}

---
apiVersion: rbac.authorization.k8s.io/v1
kind: ClusterRoleBinding
metadata:
  name: {{ .Chart.Name }}-clusterrole-nisa-binding
roleRef:
  apiGroup: rbac.authorization.k8s.io
  kind: ClusterRole
  name: {{ .Chart.Name }}-clusterrole
subjects:
  - kind: ServiceAccount
    name: {{ .Chart.Name }}-serviceaccount
    namespace: {{ .Release.Namespace }}

service.yaml

---
# Main service ingesting http traffic
kind: Service
apiVersion: v1
metadata:
  name: loadbalancer-ingress
  labels:
    app.kubernetes.io/managed-by: Helm
  annotations:
    {{- if .Values.loadbalancer.cert }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-cert: {{ .Values.loadbalancer.cert | quote }}
    service.beta.kubernetes.io/aws-load-balancer-ssl-ports: "{{- range .Values.loadbalancer.ports -}}{{- if .ssl -}}{{ .name }},{{- end -}}{{- end -}}"
    {{- end }}
    service.beta.kubernetes.io/aws-load-balancer-backend-protocol: {{ .Values.loadbalancer.backend_protocol | quote }}
    service.beta.kubernetes.io/aws-load-balancer-connection-idle-timeout: "60"
spec:
  type: LoadBalancer
  selector: 
    pod: {{ .Chart.Name }}
  ports:
    {{- range .Values.loadbalancer.ports }}
    - name: {{ .name }}
      port: {{ .port }}
      targetPort: {{ .targetPort }}
    {{- end }}

---
# Dummy service to stop the controller from nagging about ingress-nginx service
kind: Service
apiVersion: v1
metadata:
  name: ingress-nginx
  labels:
    app.kubernetes.io/managed-by: Helm
spec:
  ports:
  - name: http
    port: 10254
    targetPort: 10254
  selector:
    pod: {{ .Chart.Name }}
---

deployment.yaml

---
apiVersion: apps/v1
kind: DaemonSet
metadata:
  name: {{ .Chart.Name }}-controller
spec:
  selector:
    matchLabels:
      pod: {{ .Chart.Name }}
  template:
    metadata:
      labels:
        pod: {{ .Chart.Name }}
      annotations:
        prometheus.io/port: "10254"
        prometheus.io/scrape: "true"
        fluentbit.io/parser: k8s-nginx-ingress
    spec:
      serviceAccountName: {{ .Chart.Name }}-serviceaccount
      containers:
        - name: {{ .Chart.Name }}-controller
          image: quay.io/kubernetes-ingress-controller/nginx-ingress-controller:{{ .Chart.AppVersion }}
          args:
            - /nginx-ingress-controller
            - --configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-nginx-configuration
            - --tcp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-tcp-services
            - --udp-services-configmap=$(POD_NAMESPACE)/{{ .Chart.Name }}-udp-services
            - --publish-service=$(POD_NAMESPACE)/loadbalancer-ingress
            - --annotations-prefix=nginx.ingress.kubernetes.io
          securityContext:
            allowPrivilegeEscalation: true
            capabilities:
              drop:
                - ALL
              add:
                - NET_BIND_SERVICE
            # www-data -> 33
            runAsUser: 33
          env:
            - name: POD_NAME
              valueFrom:
                fieldRef:
                  fieldPath: metadata.name
            - name: POD_NAMESPACE
              valueFrom:
                fieldRef:
                  fieldPath: metadata.namespace
          ports:
            - name: http
              containerPort: 80
            - name: metrics
              containerPort: 10254
          livenessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /healthz
              port: 10254
              scheme: HTTP
            periodSeconds: 10
            successThreshold: 1
            timeoutSeconds: 10

configmap.yaml

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ .Chart.Name }}-nginx-configuration
data:
  use-proxy-protocol: "false"
  use-forwarded-headers: "true"
  server-tokens: "false"

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ .Chart.Name }}-tcp-services

---
kind: ConfigMap
apiVersion: v1
metadata:
  name: {{ .Chart.Name }}-udp-services

Chart.yaml

name: ingress-nginx
description: Cluster - Ingress Controller
version: 1
apiVersion: v1

appVersion: "0.24.0"

values.yaml

loadbalancer:
  backend_protocol: http
  cert: <my-cert>
  ports:
    - name: http
      port: 80
      targetPort: 80
      ssl: false
    - name: https
      port: 443
      targetPort: 80
      ssl: true

Command I am running.

helm upgrade ingress-nginx --install --namespace ingress-nginx ./

Output I currently get.

W0327 19:53:47.472827       8 client_config.go:614] Neither --kubeconfig nor --master was specified.  Using the inClusterConfig.  This might not work.
I0327 19:53:47.473136       8 main.go:241] "Creating API client" host="https://100.64.0.1:443"
I0327 19:53:47.487201       8 main.go:285] "Running in Kubernetes cluster" major="1" minor="23" git="v1.23.5" state="clean" commit="c285e781331a3785a7f436042c65c5641ce8a9e9" platform="linux/amd64"
I0327 19:53:47.684135       8 main.go:105] "SSL fake certificate created" file="/etc/ingress-controller/ssl/default-fake-certificate.pem"
I0327 19:53:47.689215       8 main.go:115] "Enabling new Ingress features available since Kubernetes v1.18"
E0327 19:53:47.692044       8 main.go:124] "Searching IngressClass" err="ingressclasses.networking.k8s.io \"nginx\" is forbidden: User \"system:serviceaccount:ingress-nginx:ingress-nginx-serviceaccount\" cannot get resource \"ingressclasses\" in API group \"networking.k8s.io\" at the cluster scope" class="nginx"
W0327 19:53:47.692070       8 main.go:127] No IngressClass resource with name nginx found. Only annotation will be used.
I0327 19:53:47.739577       8 nginx.go:254] "Starting NGINX Ingress controller"
I0327 19:53:47.755865       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-tcp-services", UID:"6115a34f-4f95-4f99-970a-b65477e45808", APIVersion:"v1", ResourceVersion:"103400810", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-tcp-services
I0327 19:53:47.756010       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-udp-services", UID:"fa04d653-a070-4934-a606-a60a7f98ad6a", APIVersion:"v1", ResourceVersion:"103400812", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-udp-services
I0327 19:53:47.756196       8 event.go:282] Event(v1.ObjectReference{Kind:"ConfigMap", Namespace:"ingress-nginx", Name:"ingress-nginx-nginx-configuration", UID:"3af77ed0-e71c-49e9-bac3-b7c3fada40df", APIVersion:"v1", ResourceVersion:"103400808", FieldPath:""}): type: 'Normal' reason: 'CREATE' ConfigMap ingress-nginx/ingress-nginx-nginx-configuration
E0327 19:53:48.844980       8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:50.385656       8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:52.811461       8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:53:57.052727       8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
E0327 19:54:05.784219       8 reflector.go:138] k8s.io/[email protected]/tools/cache/reflector.go:167: Failed to watch *v1beta1.Ingress: failed to list *v1beta1.Ingress: the server could not find the requested resource
I0327 19:54:26.690574       8 main.go:187] "Received SIGTERM, shutting down"
I0327 19:54:26.690593       8 nginx.go:372] "Shutting down controller queues"
E0327 19:54:26.690778       8 store.go:178] timed out waiting for caches to sync
I0327 19:54:26.690835       8 nginx.go:296] "Starting NGINX process"
I0327 19:54:26.691321       8 queue.go:78] "queue has been shutdown, failed to enqueue" key="&ObjectMeta{Name:initial-sync,GenerateName:,Namespace:,SelfLink:,UID:,ResourceVersion:,Generation:0,CreationTimestamp:0001-01-01 00:00:00 +0000 UTC,DeletionTimestamp:<nil>,DeletionGracePeriodSeconds:nil,Labels:map[string]string{},Annotations:map[string]string{},OwnerReferences:[]OwnerReference{},Finalizers:[],ClusterName:,ManagedFields:[]ManagedFieldsEntry{},}"
I0327 19:54:26.691353       8 leaderelection.go:243] attempting to acquire leader lease ingress-nginx/ingress-controller-leader-nginx...
I0327 19:54:26.718477       8 status.go:84] "New leader elected" identity="ingress-nginx-controller-72b9j"
I0327 19:54:26.733451       8 nginx.go:388] "Stopping NGINX process"
2022/03/27 19:54:26 [notice] 28#28: signal process started
I0327 19:54:27.738884       8 nginx.go:401] "NGINX process has stopped"
I0327 19:54:27.738926       8 main.go:195] "Handled quit, awaiting Pod deletion"
I0327 19:54:37.739197       8 main.go:198] "Exiting" code=0

Happy to provide any other details that would be helpful. I really appreciate the help in advance!

EDIT:

The cluster is on AWS and was created using the following k0ps command.

kops create cluster --node-count 2 --node-size t2.medium --zones ap-southeast-2a,ap-southeast-2c --master-size t2.small --master-zones ap-southeast-2c --master-count 1 --networking=calico --authorization RBAC -o yaml --dry-run > my-cluster.yaml
lá cờ br
Cụm K8s của bạn được định cấu hình chính xác như thế nào - phân phối nào, đó có phải là cài đặt của nhà cung cấp dịch vụ đám mây/kim loại trần không?
kirie avatar
lá cờ be
Cụm của tôi được tạo bằng k0ps và có trên AWS.

Đăng câu trả lời

Hầu hết mọi người không hiểu rằng việc đặt nhiều câu hỏi sẽ mở ra cơ hội học hỏi và cải thiện mối quan hệ giữa các cá nhân. Ví dụ, trong các nghiên cứu của Alison, mặc dù mọi người có thể nhớ chính xác có bao nhiêu câu hỏi đã được đặt ra trong các cuộc trò chuyện của họ, nhưng họ không trực giác nhận ra mối liên hệ giữa câu hỏi và sự yêu thích. Qua bốn nghiên cứu, trong đó những người tham gia tự tham gia vào các cuộc trò chuyện hoặc đọc bản ghi lại các cuộc trò chuyện của người khác, mọi người có xu hướng không nhận ra rằng việc đặt câu hỏi sẽ ảnh hưởng—hoặc đã ảnh hưởng—mức độ thân thiện giữa những người đối thoại.