Three Security Signals: Connecting Log, Trace, and Metric Reveals Incidents

Log, Trace, and Metric are three types of observability signals that systems emit themselves. Logs show what events occurred, Traces show where a request traveled, and Metrics show how often and how significantly the same phenomenon occurs.

In security operations, the roles of these three are also different. A sudden surge in login failures (Metric) alone cannot reveal the attack path, and a Trace alone may not be sufficient to understand what authorization decisions a user received. Storing only detailed Logs can increase costs and the risk of personal information exposure, making it easy to miss overall trends.

Therefore, rather than collecting the three signals separately, a design that connects the same incident with a common context is crucial. This article, based on OpenTelemetry’s official data model, summarizes the differences between each signal, the sequence of security incident investigation, and safe collection methods.

Three signals (Log, Trace, Metric) connecting to a single security investigation point

The Three Signals Are Not Interchangeable

OpenTelemetry refers to output data describing the activities of systems and applications as Signals. Currently, official documentation describes Trace, Metric, Log, and Baggage as supported signals, and treats Profiles and others as separate areas of development. The “three” in this article refers to focusing on Log, Trace, and Metric, which are most widely handled together in observability systems, not that other signals do not exist.


Signal	Question Best Answered	Basic Unit	Security Example	Typical Weakness

—	—	—	—	—
Log	What event occurred?	Event with timestamp and attributes	Login failure, authorization denied, configuration change	Large volume, easily mixed with sensitive information
Trace	Where did a request travel?	Trace and Span	API → Auth → Order → DB call path	Sampling may omit some requests
Metric	How often, how significantly does it occur?	Time-series numerical value	Failure rate, 401/403 ratio, latency	Less detailed context for individual events

The relationship between the three signals can be likened to detective work. A Metric is a distant alarm, a Trace is a travel path, and a Log is a field record. However, more important than the analogy is the fact that each signal has a different data structure and retention cost.

Log: Records Describing Events

A Log records an event that occurred at a specific point in time. The OpenTelemetry Log Data Model defines fields such as Timestamp, ObservedTimestamp, Severity, Body, Resource, Attributes, and optional TraceId/SpanId.

Security Logs include events that require later verification of responsibility and impact, such as authentication success/failure, authorization denial, administrator actions, secret access, and policy changes. The OWASP Logging Cheat Sheet also emphasizes logging security events in applications, as web server access logs alone cannot reveal internal application authorization decisions.

Structured Logs like the following are easier for searching and correlation analysis than free-form strings:

{
  "event.name": "authorization.denied",
  "service.name": "order-api",
  "trace_id": "4fd0...9a2c",
  "span_id": "91ab...20ef",
  "subject.type": "user",
  "resource.type": "order",
  "action": "read",
  "reason": "tenant_mismatch",
  "outcome": "failure"
}

Actual user IDs, order numbers, and IP addresses should not be unconditionally stored in their raw form. It is necessary to consider both the identifiability required for investigation and the minimization of personal information. Tokens, passwords, session cookies, and API keys should not be recorded. Log storage access control, retention period, and integrity protection are as important as application permissions.

Trace: A Map Showing the Causal Relationship of Requests

A Trace is the entire path a single request takes through multiple services and operations. Each segment of an operation is called a Span, and a Span contains the operation name, start/end times, Attributes, Events, status, and parent Span relationship. Spans within the same Trace are grouped by a common TraceId, and each Span has a SpanId.

For example, if an external request passes through an API Gateway, authentication service, order service, and database, a Trace is good for answering the following questions:

Was the authentication check actually called?
Which service initiated the delay or error?
Did a request that should have been denied reach the downstream DB call?
Did an internal service call an unexpected external API?

However, a Trace does not replace an audit Log. A Span’s status=ERROR may indicate a technical failure, but it may not sufficiently express who was denied what business authorization. Furthermore, using Sampling can result in some requests not being saved. For security purposes, it is safer not to rely solely on Trace retention for audit events that must be preserved.

Metric: The First Number to Detect Anomalies

A Metric is a numerical value measured repeatedly over time. Representative types include Sum, which handles increments like request counts; Gauge, which represents current values; and Histogram, which shows latency distributions. Since it aggregates data without storing every individual request, it is suitable for trends and alerts in large-scale systems.

Useful examples from a security perspective include:

auth_failures_total: Number of authentication failures
authorization_denied_total: Number of authorization denials
token_validation_errors_total: Token validation errors
http.server.request.duration: Request latency distribution
outbound_requests_total: Number of calls per external destination

If you include items with continuously increasing value types, such as raw user IDs, full URLs, or session IDs, in Metric Labels or Attributes, the number of time series will explode. This is called High Cardinality. This can lead not only to cost and memory issues but also to security problems where identifiers are replicated in the monitoring system. Detailed per-user investigations should be done using Traces or access-controlled Logs, and Metrics should generally use limited dimensions such as service, status code, or allowed result classifications for safety.

The Key to Connecting the Three Signals

The three signals are not automatically connected just by being placed in the same Backend. At a minimum, the following contexts must be consistently designed:


Connection Axis	Purpose	Design Example

—	—	—
Time	Compare phenomena at the same point in time	NTP synchronization, distinction between event time and observed time
Resource	Identify which execution entity it originated from	service.name, service.version, deployment environment
Trace Context	Directly link Logs and request paths	Log’s trace_id, span_id, trace_flags
Deployment Context	Compare before and after changes	Release version, Git SHA or image Digest
Business Event Classification	Consistently aggregate security events	event.name, action, outcome, reason

OpenTelemetry allows connecting Logs and Traces within the same execution context by including TraceId and SpanId in LogRecords. Even when using non-OTLP JSON Logs, official compatibility documents recommend top-level trace_id, span_id, and trace_flags fields.

Metrics and Traces can be connected with Exemplars. An Exemplar is example data that attaches context, such as trace_id and span_id, to a specific observed value of an aggregated Metric. This allows, for instance, selecting an abnormally slow point in a dashboard’s latency Histogram and navigating to the corresponding Trace.

Let’s Investigate a Security Incident with the Three Signals

Let’s assume a scenario: In the early morning, authorization denials for a specific API suddenly increased, and some requests took longer than usual.

1. Define the scope with Metrics

First, check the increase time, target service, status code, and deployment version of authorization_denied_total. Compare it with the overall traffic increase as a ratio. Looking only at the raw count might lead to mistaking normal traffic growth for an attack.

2. Find Traces using Exemplars or a time range

If there’s an Exemplar in the latency Histogram, navigate to the representative Trace. Otherwise, search for Traces using the anomalous time range, service, and path conditions. In the Trace, check for authentication service calls, DB access after authorization decisions, abnormal external calls, and repeated retries.

3. Check Logs with the same TraceId

Query structured Logs using the trace_id and span_id of the suspicious Span. Confirm business context such as event.name=authorization.denied, reason=tenant_mismatch, outcome=failure. This should be done by examining predefined safe fields, not by searching for request bodies or raw tokens.

4. Confirm changes and impact

Compare the service.version or image Digest of the three signals to see if the incident started after a specific deployment. If it’s an actual breach, block related accounts and sessions; if it’s a policy change or code defect, roll back or fix it. The queries and timeline used for the investigation should be recorded as a separate incident record.

This sequence is not an absolute rule. You can also start from a specific alert Log and expand to Traces and Metrics. The important thing is not to draw conclusions within a single signal but to verify hypotheses with other signals.

What Does OpenTelemetry Collector Do?

The OpenTelemetry Collector is a vendor-agnostic component that receives (Receiver), processes (Processor), and exports (Exporter) telemetry data. Trace, Metric, and Log can be configured with different Pipelines, and each Pipeline connects supported Receivers, Processors, and Exporters.

service:
  pipelines:
    traces:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    metrics:
      receivers: [otlp]
      processors: [memory_limiter, batch]
      exporters: [otlp]
    logs:
      receivers: [otlp]
      processors: [memory_limiter, redaction, batch]
      exporters: [otlp]

This configuration is an abbreviated example to illustrate the structure. The inclusion and stability of Processors like redaction should be verified in the Collector distribution being used, and actual Endpoint, TLS, and authentication settings for Receivers and Exporters are also separately required. OpenTelemetry official documentation also states that the Collector must be securely configured to handle sensitive data, tampering, and DoS risks.

Simply having a Collector does not automatically make all signals secure. Receiver Endpoint authentication, transmission encryption, network access restrictions, Processor order, transmission failure queues, Backend permissions, and the Collector’s own Logs and Metrics must be designed together.

Collection Methods to Absolutely Avoid

“Erase sensitive data later”

While an approach where applications first record tokens and personal information in Logs and then erase them in the Collector can be used as a defense layer, it is not the best starting point. Raw data may remain due to transmission errors, bypassed Exporters, or in local files. The safest default is for the application not to record it in the first place.

Putting sensitive information in Baggage

Baggage is a Key-Value context passed across service boundaries. It can be transmitted to external services via HTTP Headers during automatic instrumentation and lacks built-in integrity guarantees, so do not put passwords, tokens, API keys, or personal information in it. Trace Context from external sources should also not be used as trusted authorization information.

Relying solely on Trace Sampling for security events

Sampling reduces Trace costs but can discard requests that are targets for investigation. While Tail Sampling, which prioritizes the preservation of error or high-latency Traces, can be helpful, it should not be the sole storage mechanism for authentication and authorization audit records that must be preserved by regulation.

Putting user IDs in Metric Labels

This can create as many time series as there are users. Metrics should be limited to aggregatable dimensions, and individual user lookups should be performed using other signals with appropriate access control and retention policies.

Practical Application Checklist

For each security question, determined which signal (Log, Trace, or Metric) serves as the primary evidence.
Ensured all services record consistent service.name, environment, and version.
Verified that trace_id and span_id are automatically injected into request processing Logs.
Standardized Event Name, Action, Outcome, and Reason for authentication, authorization, and administrator actions.
Tested that passwords, tokens, cookies, API keys, and unnecessary personal information are not present in any of the three signals.
Managed the Metric Attribute allowlist and cardinality limits.
Separated Trace Sampling policies from audit Logs that must be preserved.
Applied authentication, TLS, and network restrictions to Collector receiving Endpoints.
Managed access rights, retention periods, encryption, and audit history for the Collector and Backend.
Actually performed training to navigate from Metric alerts → Trace → Log.

Conclusion: Design a Single Investigation Flow, Not Three Separate Signals

Logs provide detailed event information, Traces show the causal relationships of requests, and Metrics reveal scale and trends. The goal is not to choose one of the three signals, but to create a structure that views the same incident at different resolutions.

The most practical starting point is simple: select a security Metric, configure Exemplars or common Resources to allow navigation to a Trace from an anomalous section, and then verify if authorization decision Logs can be found using that TraceId. When this connection works in actual incident drills, Log, Trace, and Metric become not three types of data, but a single security investigation system.

Official References

Category: Cloud/OpenTelemetry

Tags: OpenTelemetry, Log, Trace, Metric, Observability, Security Monitoring, DevSecOps