[Istio] Huh? Where did my istio-proxy go? 🧐 – A Deep Dive into Kubernetes Native Sidecars

Hello! Today, we’re going to dive deep into the “Mystery of the Missing Sidecar Container”, a situation that engineers working with Kubernetes and Istio environments might have encountered at least once.

What if you injected Istio, but the proxy container isn’t visible in the YAML file, yet `kubectl` shows it normally? It’s not a bug. It’s the Kubernetes Native Sidecar feature in action. Let’s explore in detail why this change occurred and what improvements it brings. πŸš€

image


1. The Incident: “The Container Count Seems Off” πŸ€”

When deploying a pod in an Istio environment, it typically reaches a 2/2 status.

  1. Your application container
  2. The Istio-injected proxy container (istio-proxy)

However, if you examine the YAML of pods deployed in recent versions (Istio 1.27+, K8s 1.29+), you’ll notice something peculiar.

$ kubectl get pod details-v1
NAME                            READY   STATUS    RESTARTS   AGE
details-v1-766844796b-zmr86     2/2     Running   0          5m

πŸ‘‰ The `kubectl` command clearly shows READY 2/2, meaning two containers are running correctly.

But what if you check the specification with `kubectl get pod details-v1 -o yaml`?

spec:
  containers:
  - name: details  # <--- Huh? There's only one container?
    image: docker.io/istio/examples-bookinfo-details-v1:1.20.3
    ...

The `istio-proxy` is missing from the `spec.containers` section! How can it be running when it’s not visible in the YAML definition? What’s going on?


2. The Culprit is Hiding in ‘initContainers’! πŸ•΅οΈβ€β™‚οΈ

The answer lies not in `spec.containers` but in the `spec.initContainers` section. A closer look at the provided YAML file reveals the secret.

  initContainers:
  - name: istio-init
    image: docker.io/istio/proxyv2:1.28.1
    # ... (typical init container setting up iptables) ...

  - name: istio-proxy  # <--- It was hiding here!
    image: docker.io/istio/proxyv2:1.28.1
    restartPolicy: Always  # <--- ⭐ Key Point ⭐

Traditionally, `initContainers` were meant to perform ‘initialization’ tasks and then exit (Completed) before the main application starts. However, here we find `istio-proxy`, and it’s not exiting; it’s staying alive.

The secret lies in the `restartPolicy: Always` setting.


3. What is Kubernetes Native Sidecar? πŸ’‘

This is a feature called SidecarContainers, introduced in Kubernetes v1.28 (Alpha) / v1.29 (Beta, Default).

The Old Way (Old Sidecar)

Previously, `istio-proxy` was included in the regular `containers` list. However, this method had critical drawbacks:

  • No Guaranteed Startup Order: The application might start before the proxy, leading to failures when attempting network communication.
  • Termination Order Issues (Jobs): When a batch job finished, the sidecar `istio-proxy` wouldn’t terminate, causing the pod to remain in a perpetual `Running` state.

The Current Way (Native Sidecar)

Kubernetes now recognizes containers in `initContainers` with `restartPolicy: Always` as “Sidecar Containers”.

  1. Starts First: They are guaranteed to start before the main application container.
  2. Runs Continuously: After initialization, they don’t exit and share the lifecycle of the main application.
  3. Terminates Last: When the main application terminates, Kubernetes automatically shuts down the sidecar. (Solves the Job issue!)

4. Why the Sudden Change? (Version History) πŸ“…

This change is a natural progression aligned with Istio and Kubernetes version upgrades.

  • Before Istio 1.27: The Native Sidecar feature existed but was optional (required `ENABLE_NATIVE_SIDECARS=true` setting).
  • After Istio 1.27: This feature became the default.
  • Kubernetes Requirement: The cluster version must be v1.29 or higher. (Most latest versions on GKE, EKS, etc., meet this requirement).

Your current environment, a combination of “Istio 1.27+ + K8s 1.29+”, automatically applies this latest architecture without any separate configuration. πŸŽ‰


5. Reinterpreting 2/2 πŸ”’

Now, it becomes clear how `kubectl get pod` calculates the 2/2 status.

  1. Main Container (details): Defined in `spec.containers` ➑️ +1
  2. Native Sidecar (istio-proxy): In `spec.initContainers` but with `restartPolicy: Always` ➑️ +1
  3. Regular Init (istio-init): Finishes its tasks and exits (Completed) ➑️ 0

Therefore, the total is 2. It’s a perfectly normal and healthy state, so rest assured!


6. Summary and Conclusion πŸ“

This change is more than just a structural modification; it’s a highly welcome update that fundamentally resolves the “Sidecar Lifecycle Issue” which has long troubled Kubernetes engineers.

  • Proxy not in `containers` in YAML? πŸ‘‰ Check `initContainers`.
  • See `restartPolicy: Always`? πŸ‘‰ Congratulations! You are using the latest Native Sidecar feature.
  • Running Job (batch) workloads: You no longer need workarounds like `quitquitquit` for the proxy to terminate cleanly.

To all of you using the latest tech stack, don’t be alarmed by the changed YAML structure. Embrace and leverage this powerful feature! Happy cloud sailing today! β›΅οΈβ˜οΈ



Comments

Leave a Reply

Your email address will not be published. Required fields are marked *