How difficult has network troubleshooting been in a Kubernetes environment?
Traditional tcpdump is very difficult to trace in an environment where pod IPs change frequently. However, if you are using Cilium as your CNI, you have a powerful tool called Hubble.
Hubble is an Observability tool that intuitively visualizes network flows based on Kubernetes resources (Pod, Service, Namespace). Today, we will provide a detailed summary of the main Hubble CLI commands and usage tips that you can use immediately in practice. π
1. Real-time Network Flow Observation: hubble observe π‘
This is Hubble’s most core feature. It allows you to view packet flows in real-time within the current cluster. The biggest advantage is the ability to trace communication by service name or pod name, rather than IP addresses.
π Basic Usage
Running it without any options will query all namespace traffic at the current time.
hubble observeπ Real-time Monitoring (Follow Mode)
If you want to continuously view network traffic in real-time, similar to watching logs with tail -f, use the -f option.
hubble observe -fπ Essential Filtering Options (Best Practice)
This is how to extract only the information you want from a flood of tens of thousands of packets.
- Specify a specific pod and namespace
- View only traffic related to a specific pod (–pod) in a specific namespace (-n).
hubble observe --pod my-web-server -n default- Security Audit: View only blocked traffic (–verdict) π₯
- This is extremely useful for filtering out packets blocked by NetworkPolicy. It’s an essential command for security personnel!
hubble observe --verdict DROPPED # Query only packets blocked by the firewall
hubble observe --verdict FORWARDED # Query only normally allowed packets- Filtering by Peer (Source/Destination)
- Used to find “who is trying to connect to kube-system?”
hubble observe --to-namespace kube-system # If the destination is kube-system
hubble observe --from-pod landing-page # If the source is the landing-page pod- Protocol and Port Filtering:
hubble observe --protocol tcp --port 80
π Changing Output Format (-o)
You can change the format when writing automation scripts or when you need to view more detailed information.
hubble observe -o compact # (Default) Easy-to-read summary information
hubble observe -o json # JSON output including all metadata (for detailed analysis)2. Status and Connection Check: hubble status π₯
Checks if Hubble server (Relay) and agents are operating normally.
hubble statusInterpretation of Output Example:
- Health: Connection status with Hubble Relay
- Peer Service: Connection status with other nodes in the cluster
- Flows/s: Number of packet flows collected per second (for load checking)
3. Resource and Identifier Lookup: hubble list π
Cilium manages security policies internally using Identity (numeric ID) rather than IP. This is used to check which pod or label an ID is mapped to.
- Identity Lookup:
hubble list identities
Checks which labels are mapped to which numeric IDs.
- Endpoint Lookup:
hubble list endpoints
Checks the list and status of pods (Endpoints) currently being monitored.
4. Visualization Dashboard: hubble ui π₯οΈ
Run the web UI when you are not familiar with the CLI or want to see the overall Service Map between services.
Executing this command forwards a port to your localhost, allowing you to view service connection relationships as a graph in your web browser. (Excellent for understanding MSA structures!)
hubble uiπ‘ Troubleshooting Scenarios for Experts (Tips)
How to solve common real-world problems with Hubble.
β Scenario 1: “Service connection suddenly fails!” (Network Policy Debugging)
This is the most common case. Check if there are any DROPPED packets in the last 5 seconds.
hubble observe --verdict DROPPED --last 5sTip: If the result shows “Policy denied”, it means a NetworkPolicy is blocking the traffic.
β Scenario 2: “I’m getting an HTTP 500 error, but from which path?” (L7 Visibility)
If Cilium’s L7 visibility feature is enabled, you can see not only simple TCP connections but also HTTP Methods and Paths.
hubble observe --protocol http --port 80 -fYou can immediately see logs like GET /api/v1/users => 500 in the output.
β Scenario 3: “I want to know the specific reason for the block” (Using JSON + jq)
Instead of just viewing a list, parse and view the specific Drop Reason.
hubble observe --verdict DROPPED -o json | jq '.flow.drop_reason_desc'π Conclusion
Hubble is a powerful tool that makes the black box of Kubernetes networking transparent. Beyond just checking connections, it enables security audits, application debugging, and structural understanding. Be sure to learn these commands and apply them in your work! π
If this was helpful, please give it a like! Feel free to leave any questions in the comments.
Leave a Reply