CLOUD SECURITY LAB

[Kyverno] Replica Configuration and Resource Allocation for High Availability (HA)

β€”

by

cslab
in

Hello! This is the third installment of the Kyverno Master Guide, which is responsible for Kubernetes security and operational stability. πŸ›‘οΈ

Last time, we briefly touched upon basic installation using Helm and optimizing values.yaml. Today, we will delve deeply into the most crucial topic that should never be overlooked in a production environment: “Kyverno High Availability (HA) Configuration and Resource Allocation Strategy.” πŸš€

Kyverno is not just a simple tool; it’s the ‘gateway’ of your cluster. Did you know that if this gateway fails, the entire cluster can be paralyzed? Invest just 10 minutes to turn your cluster into an ironclad defense!

πŸ—οΈ 1. Why is High Availability (HA) essential for Kyverno?

Kyverno operates as a Kubernetes Admission Webhook. What does this mean?

When a user executes kubectl apply, the API server asks Kyverno, “Can I deploy this?” What happens if the Kyverno Pod is dead or unresponsive at that moment?

  • With Fail Close: All resource creation is blocked for security. (Service deployment impossible! πŸ”₯)
  • With Fail Open: Resources are deployed without security checks, exposing the cluster to risks.

Therefore, deploying multiple Kyverno instances in an HA (High Availability) configuration is not an option, but a necessity.

πŸ‘₯ 2. Replica Configuration Strategy: The Rule of 3

In a production environment, a minimum of 3 replicas is recommended.

Why 3 and not 2? 🧐

  • Availability during upgrades: If one replica has an issue while another is being updated, a last resort must remain.
  • Maintaining Quorum: An odd number of replicas is advantageous for stable consensus and load distribution in distributed systems.

values.yaml configuration example:

YAML

replicaCount: 3

πŸ“ 3. Pod Distribution Policy: “Don’t put all your eggs in one basket”

What if you deployed 3 replicas, but all 3 happened to be on the same worker node? The moment that node dies, Kyverno is wiped out. To prevent this, you must use Topology Spread Constraints.

  • Goal: Distribute Kyverno Pods across different nodes, and even different Availability Zones (AZs)!
  • Effect: Service continuity is maintained even if a specific node or zone fails.

values.yaml configuration example:

YAML

topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: kyverno

⚑ 4. Resource Allocation: “Don’t starve it, don’t let it overeat”

Kyverno intercepts and inspects all API requests within the cluster. As the cluster scales, Kyverno’s CPU and memory usage also increase.

β‘  Setting Requests & Limits πŸ”‹

You need to provide Kyverno with enough space to breathe stably.

  • CPU: Policy evaluation is compute-intensive, so setting it too low will slow down API response times. (Increased Latency)
  • Memory: The more resources in the cluster, the more memory Kyverno uses for caching. Allocate generously to avoid becoming a target for the OOM (Out Of Memory) killer.

β‘‘ Practical Recommended Specifications (for small to medium-sized clusters)

YAML

admissionController:
  container:
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

Tip: For large-scale clusters, consider increasing memory to 1Gi or more.

πŸ›‘οΈ 5. Safety Mechanism: PDB (Pod Disruption Budget)

During cluster maintenance or node replacement, Kubernetes moves Pods to other nodes (Eviction). To prevent the catastrophic scenario where all Kyverno Pods are terminated simultaneously, you must configure a PDB.

  • Function: A rule that enforces, “At least N Kyverno Pods must be running at all times, no matter what!”

values.yaml configuration example:

YAML

podDisruptionBudget:
  minAvailable: 1 # Keep at least one alive!

🚦 6. Webhook Timeout Tuning

This setting determines how long the API server will wait for Kyverno’s response.

  • Too short: Normal requests might be rejected due to network latency.
  • Too long: If Kyverno has an issue, the entire cluster’s API response will slow down, leading to user frustration.

Typically, around 15 seconds is the most appropriate compromise.

πŸ’Ύ Appendix: Comprehensive values.yaml Sample for Production Environments

This is a practical configuration file that consolidates all the settings discussed above: High Availability (HA), resource allocation, distribution policy, and PDB. Save this content as my-values.yaml and use it.

YAML

# =========================================================
# Kyverno Production-Ready Configuration
# =========================================================

# 1. Replica Configuration (Ensuring High Availability)
replicaCount: 3

# 2. Pod Disruption Budget (Ensuring Availability during Maintenance)
podDisruptionBudget:
  minAvailable: 1

# 3. Pod Distribution Policy (Node Failure Preparedness)
# Physically distribute Pods to prevent them from concentrating on a specific node.
topologySpreadConstraints:
  - maxSkew: 1
    topologyKey: kubernetes.io/hostname
    whenUnsatisfiable: DoNotSchedule
    labelSelector:
      matchLabels:
        app.kubernetes.io/instance: kyverno

# 4. Resource Allocation (Performance and Stability Optimization)
# Adjust the values below according to your cluster size.
admissionController:
  container:
    resources:
      requests:
        cpu: 100m
        memory: 256Mi
      limits:
        cpu: 500m
        memory: 512Mi

# 5. Webhook Timeout and Retry Settings
# Set to 15 seconds to prevent false positives due to network latency.
webhook:
  timeoutSeconds: 15

# 6. Background Scan Controller (Monitoring already deployed resources)
backgroundController:
  resources:
    requests:
      cpu: 50m
      memory: 64Mi
    limits:
      cpu: 200m
      memory: 128Mi

# 7. Resource Cleanup Controller (Activate Cleanup Policy)
cleanupController:
  enabled: true
  resources:
    requests:
      cpu: 50m
      memory: 64Mi

# 8. Enable Monitoring
# If Prometheus is installed, metrics can be collected.
metricsService:
  enabled: true

πŸš€ How to Apply Settings

Once you have prepared the configuration file above, execute the following command in your terminal to apply it to your cluster.

Bash

# Create if namespace does not exist
kubectl create namespace kyverno --dry-run=client -o yaml | kubectl apply -f -

# Install and update via Helm
helm upgrade --install kyverno kyverno/kyverno 
  --namespace kyverno 
  --values my-values.yaml

πŸ’‘ Post-Installation Final Checklist

  • [ ] Verify that 3 Pods are running on different nodes using the kubectl get pod -n kyverno command.
  • [ ] Verify that the PDB is correctly created using the kubectl get pdb -n kyverno command.
  • [ ] Check for any TLS errors during the Webhook registration process by examining kubectl logs.

πŸ” Post-Installation Verification – “Is my Kyverno truly healthy?”

A successful Helm installation command doesn’t mean all configurations are perfectly complete. You must perform these 3 checklist items to ensure Kyverno is functioning correctly as the cluster’s gateway.

β‘  Verify CRDs (Custom Resource Definitions)

Kyverno uses several custom resources to define policies. Use the command below to verify that all Kyverno-related CRDs have been created successfully.

Bash

kubectl get crd | grep kyverno

Key CRD List:

  • clusterpolicies.kyverno.io: Policies applied cluster-wide
  • policies.kyverno.io: Policies applied to specific namespaces
  • policyreports.kyverno.io: Reports recording policy compliance
  • admissionreports.kyverno.io: Reports containing inspection results during resource creation

β‘‘ Check Admission Webhook Status πŸ”—

Kyverno communicates with the Kubernetes API server via Webhooks. If this connection is broken, policies will not be applied.

Bash

# Verify Mutating Webhook
kubectl get mutatingwebhookconfigurations | grep kyverno

# Verify Validating Webhook
kubectl get validatingwebhookconfigurations | grep kyverno

The output should show entries like kyverno-resource-mutating-webhook-cfg, and the AGE should match the installation time.

β‘’ Verify System Components and Availability 🚦

Verify that the 3 replicas configured in values.yaml are well-distributed across different nodes and are in a Running state.

Bash

# Check status of all Kyverno-related Pods
kubectl get pods -n kyverno -o wide

# Check service status (especially Metrics port 8000)
kubectl get svc -n kyverno

It is important to visually confirm whether each Pod is deployed on a different NODE (i.e., if Topology Spread is applied) by using the -o wide option.

πŸ› οΈ Troubleshooting: When the Status is NotReady

If a Pod is not in a Running state or a Webhook error occurs, check the following logs.

Bash

# Real-time log check for a specific Pod
kubectl logs -f -n kyverno -l app.kubernetes.io/instance=kyverno
  • Most common causes: Likely due to TLS certificate issuance errors, insufficient resource allocation (OOM), or blocked network communication between nodes (Security Group/Network Policy).

##

🌟 Conclusion: Stable Security Starts with Infrastructure Design!

Configuring Kyverno for high availability is more than just installing a security tool; it’s about designing the stability of the entire cluster.

  1. Maintain 3 or more replicas,
  2. Prepare for physical failures with Topology Spread, and
  3. Ensure internal stability with PDB and appropriate resource allocation.

Only by laying such a strong foundation will your cluster remain stable even when complex policies are added later. πŸ’ͺ

We hope today’s guide helps you with your enjoyable Kubernetes operations!

Feel free to leave any questions in the comments! 😊

Comments

Leave a Reply

Your email address will not be published. Required fields are marked *