Prometheus Best Practices: Designing Metrics and Labels Correctly 🚀

Hello everyone! Today, I’d like to talk about ‘metric name and label design,’ which is the most important and fundamental part of using Prometheus, the core of monitoring systems. To accurately and efficiently understand the state of an application, it’s crucial to design metrics correctly from the start.

Shall we explore the best approach through a common scenario of tracking HTTP request counts? 🤔

###

❌ Common Mistake: Putting All Information in the Name

Let’s assume you want to track HTTP requests processed by your application, categorized by method (GET, POST) and status code (200, 404, 500). The most intuitive approach might be something like this:

http_requests_get_200_total
http_requests_post_500_total
http_requests_get_404_total
...

While it might seem clear at first glance, this approach has a fatal flaw that prevents proper utilization of Prometheus’s powerful features: it leads to ‘Metric Explosion’ 💣.

A new metric is created for every combination of HTTP method and status code. What if you add other attributes like user ID or endpoint path? The number of metrics to manage will grow exponentially, putting a significant burden on the system. Furthermore, even simple aggregation queries like “sum of all requests with status code 500” become very complex and messy. 🤯

###

✅ Prometheus Way: Add Dimensions with Labels!

The most ideal method recommended by Prometheus is this:

http_requests_total{method="GET", code="200"}

Let’s examine why this approach is superior, one by one.

1. Meaningful Metric Names 📝

The name httprequeststotal can be divided into two parts:

• http_requests: Clearly indicates that the target being measured is ‘HTTP requests’. It’s best to keep the base name as general and comprehensive as possible.
• _total: This is a standard suffix indicating that this metric is of ‘Counter’ type. A counter represents a value that continuously accumulates and increases. 📈 By explicitly stating the metric type with a suffix, anyone can easily understand its characteristics.

2. Powerful and Flexible Labels ✨

The key here is the {method=”GET”, code=”200″} part, which is the Label. Labels serve to assign multi-dimensional attributes (Dimensions) to a metric.

• Attribute Separation: Attributes that can have various values, such as method or code, are separated from the metric name and made into labels.
• Flexible Queries: With labels, you can freely slice, filter, and aggregate data using PromQL. For example, if you want to see the request rate by status code over 5 minutes, a simple query like the one below is sufficient:

# Calculate the per-second request rate over 5 minutes, grouped by status code (code)
sum by (code) (rate(http_requests_total[5m]))

If you had used metric names like in the first approach, you would have faced a terrible situation where you’d have to sum all metrics like httprequests…200total, httprequests…500total, etc., using + . 😱

###

🤔 Other Incorrect Examples

Let’s briefly touch upon why other approaches are not good.

• http_requests{method=”GET”, code=”200″, status=”total”}
• The _total suffix already indicates that it’s a counter, so adding a label like status=”total” is unnecessary duplication. It’s best to keep metrics as concise and clear as possible.
• applicationhttprequests(method=’GET’, code=’200′)
• This notation does not conform to Prometheus’s own metric and label syntax. Parentheses () and single quotes ” are not used. ❌

###

💡 In Summary

Here are the key principles to remember when designing Prometheus metrics:

1. Generalize metric names to represent the measured entity.
2. Separate attributes (dimensions) that can have various values into labels.
3. Use standard suffixes like _total for counter metrics.

By adhering to these principles, you can build a much more flexible, scalable, and powerful monitoring system. I hope this article provides a small help on your Prometheus journey! 💪

---