From Evidence to Response: LLM Security Incident Investigation

In the previous article, we found anomalous sections in Prometheus, narrowed down the request path with Tempo’s Trace, and then confirmed detailed evidence with the same TraceId in Loki. However, finding traces does not mean incident response is over. We must preserve investigation results, define the scope of impact, block dangerous paths, and safely restore services.

This article will cover a hypothetical incident from start to finish: “An input Guardrail recorded a block, but a Tool call was executed afterward.” The goal is not to re-execute the attack string. It is a defensive exercise to reconstruct the incident using already collected Logs, Traces, Metrics, and deployment information, and to create a reproducible evidence package and response record.

Incident Investigation is Bigger Than Log Search

NIST SP 800-61 Rev.3 describes incident response as not a separate, one-time task, but rather integrated into an organization’s overall cybersecurity risk management activities. Preparedness accelerates detection, response, and recovery, and lessons learned from recovery, in turn, improve protection and detection capabilities.

This cycle is particularly important in LLM services. Investigating only model calls is insufficient. User authentication, tenant boundaries, system prompt versions, RAG search results, Guardrail decisions, Tool permissions, downstream changes, and output processing must all be tied together as part of a single incident.

준비 → 탐지·분석 → 범위 확정 → 증거 보존
  ↑                              ↓
개선 ← 사후 분석 ← 복구·검증 ← 차단·근절

First, Define Roles and Decision-Making Authority

Observability tools show facts but do not make response decisions. The Incident Commander coordinates priorities and decisions, while investigators create evidence and timelines. Service owners perform functional restrictions and recovery, and IAM owners check accounts, tokens, and permissions. If personal information or regulated data is involved, legal and privacy officers determine retention periods and notification requirements.

Role	Primary Responsibility	What Not to Do Alone
—	—	—
Incident Commander	Severity, Scope, Decision Record	Confirm breach without evidence
Investigator	Timeline, Query, Evidence Hash	Directly modify original evidence
Service/Platform Owner	Function Restriction, Deployment, Recovery	Delete logs before securing evidence
IAM/Data Owner	Token Revocation, Permission/Tenant Check	Delegate final authorization judgment to LLM
Privacy/Legal Officer	Determine Retention/Notification Requirements	Retain all Telemetry indefinitely

Initial severity is determined by actual impact, rather than “was an attack string seen?”. If there’s only a blocked input, it might be a detection event. If Tool execution after blocking, access to other tenant data, secret exposure, or external system changes are confirmed, the priority should be raised.

Practice Incident: Tool Executed Even After Being Blocked

The following are assumptions for this practice:

• The input_guard Span recorded security.decision=block.
• Within the same Trace, tool_policy and tool_call Spans followed.
• It is not yet known whether the Tool’s result caused actual external changes.
• The original prompt and response were not stored in the observability store.

In this state, one should not conclude that “data was leaked due to Guardrail bypass.” First, separate three hypotheses:

1. Instrumentation error: Only Span order or attributes were incorrectly recorded.
2. Control flow defect: Execution halt was missed after a block decision.
3. Permission violation: The Tool actually performed unauthorized read/write operations.

The purpose of the investigation is not to choose the most sensational explanation, but to eliminate hypotheses one by one with evidence.

Step 1: Fix Incident Number and Investigation Time Range

First, record the incident number, investigator, baseline timezone, and initial alert. Use UTC absolute time instead of the relative time “10 minutes ago” shown on screen.

incident_id: INC-LLM-2026-0007
detected_at: 2026-09-03T01:10:00Z
window_start: 2026-09-03T00:55:00Z
window_end: 2026-09-03T01:25:00Z
initial_signal: block 이후 tool_call Span 발견

Check the blocking rate and Tool call volume before and after the incident in Prometheus.

sum(rate(llm_security_requests_total{decision="block"}[5m]))
/
sum(rate(llm_security_requests_total[5m]))

sum by (tool, result) (
  increase(llm_tool_calls_total[30m])
)

Metrics indicate the time and scale of anomalies but do not include individual TraceIds as labels. This is because using different values as labels for each request would cause the number of time series to explode.

Step 2: Verify Execution Boundaries in Trace

Find the blocked input Trace in Tempo. Attribute names are examples for this practice and should be adjusted to the actual instrumentation schema.

{ name = "input_guard" && span.security.decision = "block" }

Open the Trace and verify the following sequence:

request
 ├─ authenticate
 ├─ input_guard        decision=block
 ├─ tool_policy        ← 존재하면 제어 흐름 의심
 └─ tool_call          ← 실제 실행·결과 확인 필요

Record the Span start/end times, Status, service.name, deployment version, Tool name, and authorization decision ID. Do not copy the original prompt, Access Token, or sensitive Tool arguments into the evidence memo. If necessary, record existence, length, classification, and separate storage location instead of the values themselves.

Step 3: Verify Actual Impact with Loki and Downstream Audit Logs

Find application logs with the same TraceId. It is more appropriate to place TraceId in structured metadata or log fields rather than as a high-cardinality index label in Loki.

{service_name="llm-security-app"}
| trace_id="조사할_TraceId"

Do not confirm Tool execution success solely based on application logs. Cross-verify with the target API, database, and cloud audit logs for the same request_id, operation ID, or authorization decision ID.

Verification Point	Minimum Evidence to Retain	Questions to Answer
—	—	—
Application	TraceId, RequestId, Deployment Version	Did the code path continue after blocking?
Guardrail	Rule ID, Decision, Policy Version	Which policy decided what?
IAM/Policy Engine	Subject, Tenant, Decision ID	Who was authorized with what permissions?
Tool Gateway	Tool Name, Result Code, Target Classification	Was the call actually transmitted?
Downstream	Audit Event ID, Before/After State	Were reads/writes actually reflected?
RAG	Document ID/Version, Tenant Filter Result	Was data from other Tenants included?

LLM or Guardrail judgments are not evidence of business authorization. Final access permissions are enforced by trusted Application/API layers and downstream systems, and investigators must verify those authorization records.

Step 4: Create an Evidence Package Without Modifying Originals

Investigation results must be reviewable later to reach the same conclusions. Preserve queries, lookup time ranges, exported results, configuration/deployment versions, and hashes in a separate incident directory without modifying the original storage.

mkdir -p incident-INC-LLM-2026-0007/{queries,exports,config,notes}

cp prometheus-query.txt incident-INC-LLM-2026-0007/queries/
cp trace-summary.json incident-INC-LLM-2026-0007/exports/
cp redacted-events.jsonl incident-INC-LLM-2026-0007/exports/
cp deployment-version.txt incident-INC-LLM-2026-0007/config/

find incident-INC-LLM-2026-0007 -type f ! -name SHA256SUMS 
  -print0 | sort -z | xargs -0 sha256sum 
  > incident-INC-LLM-2026-0007/SHA256SUMS

tar --sort=name --mtime='UTC 1970-01-01' 
  --owner=0 --group=0 --numeric-owner 
  -czf INC-LLM-2026-0007-evidence.tar.gz 
  incident-INC-LLM-2026-0007

sha256sum INC-LLM-2026-0007-evidence.tar.gz

On macOS, you can use shasum -a 256 instead of GNU sha256sum. Tar options also vary by implementation, so check for support in your operating environment.

In notes/timeline.md, separate observed facts from interpretations.

[사실] 01:03:14Z input_guard가 block 기록
[사실] 01:03:14Z 같은 Trace에 tool_call Span 존재
[사실] Downstream 감사 로그에 write 이벤트 없음
[해석] 호출 준비 단계까지 진행됐으나 외부 변경 증거는 확인되지 않음
[미확인] 응답이 사용자에게 반환됐는지 Gateway 로그 확인 필요

A hash is a means to verify if a file has changed, not an automatic guarantee of the reliability of the collection process. The collector, collection time, original location, transformation/masking process, and access records should also be preserved.

Ensure Telemetry Itself Does Not Become a New Exfiltration Path

OpenTelemetry’s official security guidelines warn that Telemetry can unintentionally include PII, application data, and network patterns. Since OpenTelemetry cannot automatically determine what is sensitive, implementers must define policies for minimal collection, deletion, hashing, and redaction.

For basic evidence, prioritize the following:

• TraceId, rule ID, decision, and processing steps instead of original text
• Approved version or hash instead of system prompt content
• Document ID/version and Tenant Filter results instead of RAG original text
• Operation type, target classification, and result code instead of original Tool arguments
• Limited pseudonymous identifiers necessary for incident investigation instead of user identifiers

If original text is truly necessary, a separate encrypted storage, restricted roles, short retention periods, and access audits are required, distinct from general observability storage. Legal, contractual, and regulatory requirements must be determined by the organization’s respective officers.

Step 5: Secure Evidence and Then Limit Damage

Containment, or damage limitation, is not a single action like “turn off all servers.” Apply the narrowest safe measures first, tailored to the impact and business continuity.

Confirmed Situation	Example Priority Action	Side Effects to Check
—	—	—
Specific Tool called after block	Disable relevant Tool Route	Interruption of core business
Defect in specific policy version	Rollback to previous approved version	Re-exposure to risks blocked by new policy
Potential Token exposure	Revoke/Rotate Token, Terminate Session	Failure of automated/batch jobs
Missing Tenant Filter	Retrieval API Fail-closed	Overall availability of search function
External write confirmed	Remove write permissions, restore target state	Distinction between evidence and legitimate changes

If possible, rather than taking down the entire LLM, reduce functionality by changing Tool writes to read-only, blocking dangerous routes, or requiring human approval. However, if secret leakage or widespread tenant boundary collapse is suspected, stronger isolation is needed.

All emergency changes must record the executor, approver, time, change details, rollback conditions, and associated incident number.

Step 6: Eradicate and Verify Recovery

If the cause was a control flow defect where the function did not terminate after returning a block, simply adding a policy is not enough. Enforce fail-closed in the code and add automated tests.

decision = input_guard.check(request)
if decision.blocked:
    audit(decision)
    return blocked_response()

# Only allowed requests should reach below this line.
result = tool_gateway.execute(authorized_request)

Before recovery, test at least four things:

1. tool_policy and tool_call Spans are not generated for blocked requests.
2. Allowed requests operate normally and pass downstream authorization again.
3. RAG documents and Tool targets of other Tenants cannot be accessed.
4. Metrics, Traces, Logs, and Alerts are connected in the new version.

Initially, deploy to Canary or restricted Tenants and observe error rates, blocking rates, and Tool call volumes. “Successful patch deployment” is not the completion of recovery. Conclude when normal functionality, security boundaries, and observability features are all restored.

Step 7: Feed Post-Analysis Back into Detection and Design Improvement

A post-mortem report is not a document to blame people, but an input to reduce detection and response time for the next incident.

• What were the initial signals and actual impact?
• What evidence changed the conclusion?
• How long did detection and containment each take?
• What fields in Telemetry were insufficient or excessive?
• What will be changed in Runbook, Test, Alert, and permission boundaries?

For this incident, the following improvements are natural:

예방: block 이후 즉시 종료하는 공통 Middleware
탐지: blocked Trace에 tool_call이 존재하면 Critical Alert
제한: Tool Gateway에서 사용자·Tenant 권한을 다시 검사
검증: CI에 차단 후 Side Effect 부재 테스트 추가
운영: 사건 Query와 증거 Manifest 템플릿 표준화

Practice Completion Checklist

• Recorded incident number, responsible person, UTC time range, and initial hypotheses.
• Confirmed anomaly time and scale with Metrics.
• Verified execution path after blocking in Trace.
• Cross-verified with Application and Downstream audit logs.
• Separated observed facts, interpretations, and unconfirmed items.
• Preserved exports and queries without modifying originals.
• Generated file-specific hashes and evidence package hash.
• Excluded prompts, responses, tokens, and PII from basic evidence.
• Recorded the narrowest containment and rollback conditions.
• Performed blocking, authorization, tenant, and observability regression tests after the patch.
• Reflected lessons learned in Alert, Test, Runbook, and permission design.

Conclusion

The core of LLM security incident response is not to guess why the model generated a certain phrase. It is to connect evidence to determine which entity requested under which policy and deployment version, what Guardrail and Application decided, and what actual side effects the Tool and Downstream systems created.

Prometheus shows the timing and scale of the incident, Tempo shows the execution path of the request, and Loki and audit logs show the basis of decisions and actual impact. By adding change records, hashes, access control, and recovery verification, observability data becomes actionable evidence. The final step is always the same question: “Can we detect faster, contain more narrowly, and recover more reliably next time?”

Official Resources

• NIST SP 800-61 Rev.3: Incident Response Recommendations and Considerations for Cybersecurity Risk Management
• NIST AI 600-1: Generative Artificial Intelligence Profile
• NIST AI Risk Management Framework
• OpenTelemetry Security
• OpenTelemetry: Handling Sensitive Data
• OpenTelemetry Context Propagation
• Grafana Tempo TraceQL
• Grafana Loki LogQL
• Prometheus Querying Basics

※ Incident response, evidence preservation, and notification obligations vary depending on the organization’s industry, region, contractual, and legal requirements. The commands and preservation items in this article are for educational examples, and in actual incidents, the organization’s approved incident response procedures and the guidance of legal and privacy officers take precedence.

Category: General IT/IT Security

Tags: LLM Security, Incident Response, Digital Forensics, OpenTelemetry, Loki, Tempo, Prometheus, NIST