Prometheus on() vs ignoring(): Two Faces of Vector Matching 🎭

Hello! Today, we’ll delve into two very important but easily confusing keywords in Prometheus Query Language (PromQL): on() and ignoring(). These two play a crucial role in determining which labels to use for matching when performing operations between vectors. A clear understanding of their differences is essential for writing accurate queries!

🎯 on(): “Match only on these labels!”

on() is a keyword that instructs Prometheus to perform operations only between time series data where the specified list of labels exactly matches. All other labels not explicitly mentioned in on() are completely ignored during the matching process.

How on() Works

• Explicit Specification: Directly and clearly specify the labels to be used for matching.
• Whitelist Approach: It’s like saying, “Only allow labels in this list; ignore the rest.”

Example Scenario 📝

Let’s assume we have two metrics like these:

• http_requests_total{method=”GET”, path=”/api/users”, instance=”app-1″}
• http_requests_in_progress{method=”GET”, path=”/api/users”, instance=”app-1″}

Here, we want to calculate the ratio of in-progress requests based on method and path. We don’t want to consider the instance label. This is where on() comes in handy.

http_requests_in_progress
  / on(method, path)
http_requests_total

This query performs a division operation between data points where the method and path label values exactly match. Since the instance label is ignored even if it differs, we get the desired result.

🙈 ignoring(): “Match on everything except these labels!”

ignoring() is the opposite concept of on(). It instructs Prometheus to perform operations between time series data where all labels except the specified list of labels match.

How ignoring() Works

• Exception Specification: Directly specify the labels to be ignored during matching.
• Blacklist Approach: It’s like saying, “Ignore labels in this list, but all other labels must match.”

Example Scenario 📝

Let’s use ignoring() in the same scenario as above. We need to ignore only the instance label, and all other labels (method, path) must match.

http_requests_in_progress
  / ignoring(instance)
http_requests_total

This query finds data where all labels except the instance label (i.e., method, path) match and performs the operation. As a result, it yields the same outcome as using on(method, path).

🤔 on() vs ignoring(), When to Use Which?

Category	on()	ignoring()
Concept	Match only specified labels (Whitelist)	Ignore specified labels and match (Blacklist)
Advantage	Query intent is clear and concise	May require less query modification even if new labels are added
Recommended Scenario	When grouping or operating based on a few specific labels	When you want to exclude only specific labels that vary by environment, such as instance, pod

Typically, it’s common to ignore unique labels that distinguish individual instances, like instance or pod, and perform operations based on service-logic-related labels (e.g., method, path, status_code). In such cases, using ignoring(instance, pod) can be more flexible and convenient.

Conversely, if you want to group by only two or three specific labels (e.g., job, handler) out of many, then on(job, handler) would be a much more intuitive and clear choice.

Conclusion ✨

on() and ignoring() are powerful tools for controlling vector matching in PromQL.

• on() specifies the labels to include. 🎯
• ignoring() specifies the labels to exclude. 🙈

By accurately understanding the differences between these two keywords and choosing the appropriate one for your situation, you can write more precise and flexible monitoring queries!

---