On June 13, 2026, a web service that was in operation suddenly became inaccessible.
At first, it seemed like a simple server reboot issue.
However, it was actually a rather deep network outage involving a Linux kernel update and the Cilium eBPF data path.
The interesting point is that this outage was not analyzed by a human, step by step.
This incident response is a case where Claude connected to a remote server, checked its status, executed commands, analyzed logs, narrowed down the cause, and even performed recovery actions.
The only thing a human did was approve risky operations along the way.
Sensitive information such as service names, domains, IPs, server names, and node names has been masked with *.
However, the kernel version necessary for root cause analysis is indicated as is.
3-Line Summary
After a server reboot, the web service was inaccessible, but SSH access and Kubernetes cluster status appeared normal.
Claude directly executed commands on the remote server, sequentially analyzing Gateway, Cilium, NodePort, tcpdump, and kernel history, ultimately determining the problem to be an eBPF data path issue with the combination of the 6.8.0-124-generic kernel and Cilium 1.18.3.
Booting with the previous kernel, 6.8.0-111-generic, immediately restored the service, and Claude even fixed the GRUB default boot kernel to prevent recurrence.
1. The Start of the Outage: Server Alive, Web Service Dead
The initial symptoms were simple.
The website was inaccessible.
curl https://***/
# curl: (28) Connection timed out after 15007 millisecondsAt first, it seemed as if the server itself was down.
bash scripts/test-connection.sh
# ssh: connect to host ***.***.***.*** port 22: Operation timed outChecking Tailscale status confirmed that the server was offline.
***.***.***.*** ***-server linux active; relay "***"; offline, last seen 2d agoUp to this point, it was simple.
The server was off, and it seemed like turning it back on would solve the problem.
However, even after restarting the server, the web service remained inaccessible.
SSH access was possible.
Server resources were also within the normal range.
load average: 0.87
CPU idle: 87%
Memory free: 12GB
Disk usage: 12%
Failed systemd services: 0All Kubernetes nodes were also in a Ready state.
kubectl get nodes
# ***-control-plane Ready 204d
# ***-worker Ready 204d
# ***-worker2 Ready 204dOutwardly, the server appeared normal.
But the web service seen by users was dead.
At this point, Claude did not simply conclude, “The server is normal.”
It began to view the outage based on the entire path through which service traffic actually passes.
2. The Architecture Was Not a Simple Web Server
This environment was not simply a structure with just Nginx running.
The overall architecture was roughly as follows:
외부 사용자
↓
도메인 ***
↓
공인 IP ***.***.***.***
↓
물리 서버 ***
↓
Docker
↓
kind 기반 Kubernetes 클러스터
↓
Cilium Gateway API
↓
내부 웹 애플리케이션In other words, there were many potential points of failure.
Possible candidates were as follows:
DNS 문제
공인 IP 문제
호스트 방화벽 문제
Docker 네트워크 문제
Kubernetes Service 문제
Cilium Gateway 문제
Envoy 문제
eBPF 데이터패스 문제
백엔드 애플리케이션 문제
Linux 커널 문제If a human had looked at it directly, they would have had to ponder quite a bit about where to start checking.
But Claude began to categorize the status by layer.
It sequentially checked whether the server itself was normal, whether Kubernetes was normal, whether the Gateway was normal, whether NodePort was normal, and how far actual packets were entering.
3. The Role of Humans in This Incident Response
A crucial point in this case is that Claude did not merely offer advice.
Claude directly connected to the remote server, executed commands, interpreted the output, and then proceeded with the next inspection command.
The role of humans was limited.
사람:
- 위험한 작업 승인
- 재부팅 같은 주요 조치 승인
- 최종 결과 확인
Claude:
- 원격 서버 상태 점검
- Kubernetes 상태 점검
- Cilium Gateway 상태 분석
- IP Pool 문제 확인 및 복구
- NodePort 구간 테스트
- tcpdump 기반 패킷 분석
- Envoy 메트릭 확인
- 커널 변경 이력 추적
- 이전 커널 부팅 조치
- GRUB 기본 부팅 커널 고정
- 장애 기록 정리In other words, this response was closer to “Claude diagnosing and recovering an outage like an AIOps agent” than “an operator solving a problem by asking Claude.”
Of course, not all operations were conducted completely unsupervised.
Operations that affect the system, such as rebooting and GRUB changes, were approved by a human.
However, the outage analysis and recovery flow itself were led by Claude.
4. First Cause Candidate: Cilium Gateway Address Not Assigned
Claude first checked the Cilium Gateway status.
kubectl -n kube-system get gateway ciliumThe result was as follows:
NAME CLASS ADDRESS PROGRAMMED AGE
cilium cilium False 204dThe status was PROGRAMMED=False.
This means the Gateway was not ready to receive external traffic.
Upon checking the detailed conditions, the reason was AddressNotAssigned.
Gateway waiting for addressThat is, the Gateway was not receiving an external IP to use.
Claude immediately checked the Cilium LoadBalancer IP Pool status.
kubectl get ciliumloadbalancerippool
# No resources foundThe IP Pool resource had disappeared.
In this environment, the Cilium Gateway needed to use a specific public IP, so without an IP Pool, the Gateway could not obtain an external address.
Claude found the existing configuration and reapplied the IP Pool.
apiVersion: "cilium.io/v2alpha1"
kind: CiliumLoadBalancerIPPool
metadata:
name: "cilium-pool"
spec:
blocks:
- cidr: "***.***.***.***/32"It checked the Gateway status again.
kubectl -n kube-system get gateway ciliumNAME CLASS ADDRESS PROGRAMMED AGE
cilium cilium ***.***.***.*** True 204dThe Gateway became normal.
Up to this point, it seemed like the problem was solved.
However, the web service was still timing out.
Claude did not stop its analysis here.
It determined, “The Gateway address issue has been resolved, but if user traffic is still failing, another problem remains.”
5. Sectional Testing: How Far Is It Normal?
Next, Claude began to break down the network path into sections.
It tested by changing the source based on NodePort 30080.
| Test Section | Result |
| Inside worker container → Self 30080 | Normal |
| Another Kubernetes node → Worker node 30080 | Normal |
| Physical server host → Worker container 30080 | Timeout |
| External user → Domain HTTPS | Timeout |
This result was very important.
Kubernetes internal communication was normal.
Inter-node communication was also normal.
The backend application was not completely dead either.
The problem was the path from outside the host to the container.
Based on these results, Claude significantly narrowed down the scope of the outage.
애플리케이션 자체 문제 가능성 낮음
Kubernetes 내부 네트워크 문제 가능성 낮음
노드 간 통신 문제 가능성 낮음
호스트 → 컨테이너 NodePort 처리 경로 문제 가능성 높음
Cilium/eBPF 데이터패스 문제 가능성 상승It also checked the firewall.
ufw status
# inactiveIP forwarding was also enabled.
sysctl net.ipv4.ip_forward
# net.ipv4.ip_forward = 1Ping from the host to the container also reached normally.
Therefore, it was difficult to consider it a simple firewall issue or a complete routing disruption.
6. Decisive Evidence: SYN Arrives, But No SYN-ACK
Claude then used tcpdump.
It captured port 30080 within the worker container’s network namespace.
nsenter -t $WORKER_PID -n tcpdump -ni eth0 'tcp port 30080'Then, it attempted to connect to port 30080 from the host.
The tcpdump result was as follows:
01:18:53 IP ***.***.***.***.51640 > ***.***.***.***.30080: Flags [S]
01:18:54 IP ***.***.***.***.51640 > ***.***.***.***.30080: Flags [S]
01:18:55 IP ***.***.***.***.51640 > ***.***.***.***.30080: Flags [S]The key here is one thing.
The SYN packet arrived at the container’s eth0.
However, there was no SYN-ACK response.
A TCP connection usually starts in the following sequence:
Client → Server : SYN
Server → Client : SYN-ACK
Client → Server : ACKThis time, only the first SYN was repeatedly observed.
There was no SYN-ACK response from the server side.
In other words, packets were not completely failing to arrive.
Packets reached near the destination.
However, a response was not being generated normally within the container’s internal network processing path.
Cilium processes packets at the kernel level using eBPF.
When using the Gateway API, traffic passes through Cilium’s eBPF path and is then forwarded to Envoy.
Claude also checked Envoy metrics.
listener-insecure downstream_cx_total = 0The counter did not increase even when a connection was attempted.
This means that packets were visible up to the container’s eth0 but did not reach Envoy.
At this point, the cause candidates were quite narrowed down.
애플리케이션 문제 아님
DNS 문제 아님
단순 방화벽 문제 아님
Kubernetes 내부 통신 문제 아님
Envoy까지 트래픽이 도달하지 않음
Cilium eBPF 데이터패스 또는 커널 네트워크 처리 경로 의심7. Cilium Restart Did Not Solve the Problem
It could have been a temporary Cilium status issue, so Claude restarted Cilium-related components.
kubectl -n kube-system rollout restart ds/cilium
kubectl -n kube-system rollout restart ds/cilium-envoyHowever, the symptoms remained the same.
The Gateway was PROGRAMMED=True.
Kubernetes internal communication was also normal.
But external incoming traffic was still timing out.
At this point, Claude posed this key question:
재부팅 전에는 정상인데,
재부팅 후 장애가 발생했다면,
재부팅으로 인해 실제로 바뀐 것은 무엇인가?This question was decisive.
Neither the configuration nor the application had changed.
What was newly applied by the reboot was the kernel.
8. The Real Cause Candidate: Kernel Update
It checked the current kernel version.
uname -r
# 6.8.0-124-genericIt checked the reboot history.
last reboot -FThe result was as follows:
Sat Jun 13 00:03 boot 6.8.0-124-generic
Tue May 19 13:04 boot 6.8.0-111-genericPreviously, it was operating with the 6.8.0-111-generic kernel.
However, after this reboot, it came up with the 6.8.0-124-generic kernel.
It also checked the kernel installation history.
grep "install linux-image" /var/log/dpkg.log2026-06-04 install linux-image-6.8.0-124-genericTo summarize, this is what happened:
6월 4일:
- linux-image-6.8.0-124-generic 자동 설치
6월 4일 이후:
- 서버는 계속 기존 커널 6.8.0-111-generic으로 운영
6월 13일:
- 서버 재부팅
- 새 커널 6.8.0-124-generic으로 부팅
- Cilium eBPF 경로에서 외부 트래픽 처리 이상 발생
- 웹서비스 타임아웃Such outages often show a time lag between cause and symptom.
The kernel was already installed on June 4th.
However, it was actually applied only after the reboot on June 13th.
From an operator’s perspective, they might feel, “Nothing changed today, so why did the outage occur?”
But the actual cause came in through an automatic update a few days ago and was activated during the reboot.
Claude confirmed this point as the outage cause candidate.
9. Recovery: Booting with the Previous Kernel
The fastest recovery method was to revert to the kernel that was functioning normally before the outage.
Claude configured the system to use the previous kernel for the next boot.
grub-reboot "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-111-generic"
rebootThis operation involved a system reboot, so it proceeded after human approval.
After booting, it checked the kernel version.
uname -r
# 6.8.0-111-genericAfter waiting for Kubernetes nodes and Cilium components to start normally, it tested again.
curl http://127.0.0.1:30080/A normal response was received.
It also checked the external domain.
curl -L https://***/The result was HTTP 200.
HTTP 200The service was recovered.
This was the moment the cause candidate was virtually confirmed.
With 6.8.0-124-generic, external traffic was not processed normally through the Cilium eBPF path, and reverting to 6.8.0-111-generic restored the service.
10. Prevention of Recurrence: Fixing GRUB Default Kernel
But it shouldn’t end here.
grub-reboot is a one-time setting.
If it boots back to 6.8.0-124-generic on the next reboot, the same outage could recur.
To prevent recurrence, Claude fixed the GRUB default boot kernel to 6.8.0-111-generic.
grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-111-generic"
update-grubIt also performed a configuration check.
grub-editenv listsaved_entry=Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-111-genericNow, even on the next reboot, it will default to booting with the 6.8.0-111-generic kernel.
However, this is a temporary measure.
Fixing to an older kernel can mean missing security patches from the latest kernel.
Therefore, the following follow-up actions are necessary in the long term:
- Cilium 업그레이드 검토
- 6.8.0-124-generic 또는 이후 커널에서 재현 테스트
- Cilium eBPF 관련 설정 검토
- 커널 자동 업데이트 정책 재검토
- 재부팅 전후 외부 헬스체크 자동화
- 운영 노드 재부팅 전 사전 검증 절차 마련11. This Outage from an AIOps Perspective
This outage is not simply a case of “Claude told me the answer when I asked.”
Claude actually checked the status on the remote server, executed commands, interpreted logs, and autonomously proceeded to the next verification point.
From an AIOps perspective, it’s closer to the following flow:
1. 장애 감지
2. 서버 접속 상태 확인
3. Kubernetes 상태 확인
4. Gateway 상태 확인
5. 누락된 IP Pool 복구
6. 사용자 증상 재확인
7. NodePort 구간별 테스트
8. tcpdump로 패킷 흐름 확인
9. Envoy 메트릭 확인
10. Cilium 재시작
11. 재부팅 전후 커널 변경 확인
12. 이전 커널로 복구
13. GRUB 기본값 고정
14. 장애 기록 작성In this process, humans did not directly design and execute all commands.
Claude made judgments and executed.
Humans only approved high-risk operations.
This demonstrates that AIOps can go beyond simple monitoring or alert automation.
12. What Claude Did Particularly Well
12.1 Quickly Narrowed Down the Scope of the Outage
Initially, there were too many possible causes.
DNS
방화벽
Docker
Kubernetes
Cilium
Gateway
Envoy
eBPF
Linux 커널
애플리케이션But Claude narrowed down the candidates through sectional testing.
Kubernetes 내부 정상
노드 간 통신 정상
호스트 → 컨테이너 NodePort 실패
SYN은 도착
SYN-ACK 없음
Envoy까지 미도달
재부팅 후 커널 변경Ultimately, the cause candidate was narrowed down to the kernel + Cilium eBPF data path combination.
12.2 Connected tcpdump Results to the Cause of the Outage
The repeated SYN packets and absence of SYN-ACK in tcpdump are crucial clues.
Claude did not simply conclude, “Packets are visible” from this result.
It interpreted it as follows:
패킷은 목적지 네트워크 인터페이스까지 도착했다.
하지만 TCP 응답이 생성되지 않았다.
따라서 단순 라우팅 단절이나 방화벽 차단보다는 수신 처리 경로가 의심된다.
Cilium eBPF 또는 Envoy 전달 경로를 확인해야 한다.Thanks to this interpretation, it avoided unnecessary DNS, TLS, and application analysis.
12.3 Tracked Changes Applied by Reboot
The core of this outage was the reboot.
It was normal before the reboot, and the outage occurred after the reboot.
Claude checked the kernel history based on this difference.
uname -r
last reboot -F
grep "install linux-image" /var/log/dpkg.logAnd it found that the kernel had changed from 6.8.0-111-generic to 6.8.0-124-generic.
This was the key clue that determined the final recovery direction.
12.4 Followed Through from Recovery to Recurrence Prevention
A common mistake in incident response is thinking, “It’s alive, so it’s over.”
But Claude did not stop after a one-time boot to the previous kernel.
It fixed the GRUB default value to prevent the same problem from recurring on the next reboot.
grub-set-default "Advanced options for Ubuntu>Ubuntu, with Linux 6.8.0-111-generic"
update-grubIt handled recovery and recurrence prevention separately.
13. Advantages of AI Directly Managing Servers
This case clearly demonstrated the advantages of AI-based remote server management.
First, incident response speed increases.
Claude saw the command results and immediately proceeded to the next verification point.
It significantly reduced the time humans spend searching and deliberating.
Second, it structures complex layers.
Even in an environment intertwined with physical servers, Docker, Kubernetes, Cilium, eBPF, and Envoy, it separated problems by layer.
Third, it interprets command results.
It didn’t just list commands; it analyzed what the results meant.
Fourth, documentation after recovery is easy.
The incident response process itself was well-organized for recording.
This article, too, is organized based on the workflow performed by Claude.
Fifth, it can be developed into repeatable Runbooks.
This procedure can serve as the basis for future incident response automation.
서버 상태 확인
Kubernetes 상태 확인
Gateway 상태 확인
NodePort 구간 테스트
tcpdump 패킷 확인
커널 변경 이력 확인
복구 커널 선택
재발 방지 설정14. Caution: Giving AI Full Authority Is Dangerous
Just because this case was successful does not mean it is safe to give AI unlimited full authority over production servers.
In particular, the following operations absolutely require an approval process:
reboot
grub-reboot
grub-set-default
update-grub
kubectl delete
kubectl rollout restart
iptables
tc
sysctlEven in this case, Claude led the outage analysis and recovery, but high-risk operations like rebooting or GRUB changes were approved by a human.
A realistic operating model is as follows:
일반 조회 명령:
- Claude가 자유롭게 수행
상태 변경 명령:
- Claude가 제안 및 실행 준비
- 사람 승인 후 실행
위험 명령:
- 사람 승인 필수
- 가능하면 롤백 방법 확인 후 실행The core of AIOps is not to completely eliminate humans.
It is for AI to perform most of the practical tasks, but to have approval gates for risky decisions.
15. Lessons Learned from This Outage
First, even if the server is alive, the service can be dead.
Even if SSH access is possible, and CPU and memory are normal, the web service accessed by users can fail.
Second, AI can trace the root cause of complex outages quite deeply.
In this case, Claude analyzed not just simple log summaries but connected Gateway, Cilium, NodePort, tcpdump, Envoy, and kernel history.
Third, after a reboot, an outage requires checking for kernel changes.
uname -r
last reboot -F
grep "install linux-image" /var/log/dpkg.logFourth, eBPF-based networks are sensitive to kernel updates.
Cilium actively uses the kernel’s eBPF features, so kernel updates can affect the network data path.
Fifth, when giving AI execution privileges, the approval boundaries must be clear.
Even if AI automatically performs queries and analysis, operations like rebooting or changing boot settings should be handled based on approval for safety.
Conclusion
This outage started as a simple problem of the website not being accessible.
But the actual cause was not the web server or application.
A server reboot applied the new kernel 6.8.0-124-generic, and in this environment, Cilium 1.18.3’s eBPF data path was determined not to be processing external traffic normally.
Claude connected to the remote server, checked its status, recovered the Gateway issue, tested the NodePort section, analyzed packets with tcpdump, traced kernel history, and then reverted to the previous kernel 6.8.0-111-generic to restore the service.
The only thing a human did was approve major risky operations.
This case demonstrates the possibility that AIOps can go beyond merely summarizing alerts to actually performing server outage diagnosis and recovery processes.
Of course, caution is needed when giving AI execution privileges on production servers.
However, with an appropriate approval system and clear privilege boundaries, AI can play a role closer to an actual operational agent rather than just an auxiliary tool.
This outage was a day that confirmed that possibility.
Leave a Reply