macos-tools/contexts/ai-context-debug.md

AI Development Context - Debug Scenario

生成时间: 2025-12-30 22:59:44 | 场景: 故障排查

系统健康状态

Docker 服务

• ✅ Docker 运行正常
• 容器: 222 运行中, 160 停止

近期重启的容器

reg-cache: Restarting (2) 7 seconds ago

Unhealthy 容器

tiny-backend: Up 32 hours (unhealthy)
lingjing-plugin-daemon: Up 2 days (unhealthy)
bookstore-app: Up 2 days (unhealthy)
plane-github-sync: Up 2 days (unhealthy)

Kubernetes 集群

• ✅ 集群连接正常

异常 Pod

NAMESPACE      NAME                              READY   STATUS             RESTARTS   AGE
tiny           ljwx-tiny-flyway-mr6ng            0/1     Completed          0          2d
trivy-system   trivy-operator-77499dcb58-dkx27   0/1     InvalidImageName   0          42m

最近的事件

infra           2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/ingress-nginx-hpa                  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
argocd          2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/argocd-server-hpa                  failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
monitoring      2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/prometheus-hpa                     failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
ljwx-stock      2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/claude-stock-backend-hpa           failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
gitea-runner    2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/gitea-runner-hpa                   failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
monitoring      2m6s        Warning   FailedGetResourceMetric   horizontalpodautoscaler/grafana-hpa                        failed to get cpu utilization: unable to get metrics for resource cpu: no metrics returned from resource metrics API
cert-manager    2m3s        Warning   Unhealthy                 pod/cert-manager-webhook-dnspod-94dd64c7d-gw8jj            Liveness probe failed: Get "https://192.168.194.113:8443/healthz": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)
cert-manager    118s        Normal    Killing                   pod/cert-manager-webhook-dnspod-94dd64c7d-gw8jj            Container webhook failed liveness probe, will be restarted
ljwx-project    93s         Normal    Pulled                    pod/pcc-project-control-center-frontend-8666b49865-jvgcw   Container image "harbor.omniverseai.net:8080/ljwx-project/pcc-frontend:0.1.0" already present on machine
ljwx-project    92s         Normal    Pulled                    pod/pcc-project-control-center-backend-6df5ff97d6-97pxz    Container image "harbor.omniverseai.net/ljwx-project/pcc-backend:0.1.0" already present on machine
cert-manager    87s         Normal    Pulled                    pod/cert-manager-webhook-dnspod-94dd64c7d-gw8jj            Container image "qqshfox/cert-manager-webhook-dnspod:latest" already present on machine
ljwx-stock      73s         Normal    ClearAssignment           service/claude-stock-backend                               current IP for "ljwx-stock/claude-stock-backend" not allowed by config, will attempt for new IP assignment: ["192.168.139.2"] is not allowed in config
ljwx-stock      73s         Normal    UpdatedLoadBalancer       service/claude-stock-backend                               Updated LoadBalancer with new IPs: [192.168.139.201] -> [192.168.139.2]
ljwx-stock      73s         Normal    UpdatedLoadBalancer       service/claude-stock-frontend                              Updated LoadBalancer with new IPs: [192.168.139.200] -> [192.168.139.2]
ljwx-stock      73s         Normal    ClearAssignment           service/claude-stock-frontend                              current IP for "ljwx-stock/claude-stock-frontend" not allowed by config, will attempt for new IP assignment: ["192.168.139.2"] is not allowed in config
ljwx-stock      62s         Warning   Unhealthy                 pod/claude-stock-backend-7c9567ff69-ptvc5                  Readiness probe failed: Get "http://192.168.194.88:8000/health": context deadline exceeded (Client.Timeout exceeded while awaiting headers)
ljwx-project    60s         Warning   BackOff                   pod/pcc-project-control-center-frontend-8666b49865-jvgcw   Back-off restarting failed container frontend in pod pcc-project-control-center-frontend-8666b49865-jvgcw_ljwx-project(7626ea9c-b9ce-4633-a3dc-6526d2458310)
ljwx-project    49s         Warning   BackOff                   pod/pcc-project-control-center-backend-6df5ff97d6-97pxz    Back-off restarting failed container pcc-project-control-center-backend in pod pcc-project-control-center-backend-6df5ff97d6-97pxz_ljwx-project(3e0e39a0-0592-4468-b497-3684fbc829aa)
stock           21s         Normal    Killing                   pod/stock-backend-67d9fb75bc-j5mkg                         Container backend failed liveness probe, will be restarted
ljwx-stock      11s         Normal    Killing                   pod/claude-stock-backend-7c9567ff69-ptvc5                  Container backend failed liveness probe, will be restarted

资源使用异常的 Pod

# 检查 CPU/内存使用（需要 metrics-server）
kubectl top pods -A --sort-by=cpu 2>/dev/null | head -10 || echo "metrics-server 未安装"

资源使用

• 磁盘: 2% 已使用（904Gi 可用）
• 内存: 47.9% 空闲

常见问题诊断

1. ImagePullBackOff / ErrImagePull

# 查看 Pod 详情
kubectl describe pod <pod-name> -n <namespace>

# 自动修复（推荐）
cd /Users/brunogao/work/codes/tools/macos/k8s
./cache-image.sh --update-deployment <namespace>/<deployment>

# 手动缓存镜像
./cache-image.sh <image-name>

2. CrashLoopBackOff

# 查看日志
kubectl logs <pod-name> -n <namespace>
kubectl logs <pod-name> -n <namespace> --previous  # 查看上一次的日志

# 查看事件
kubectl describe pod <pod-name> -n <namespace>

# 进入容器调试
kubectl exec -it <pod-name> -n <namespace> -- /bin/sh

3. 服务无法访问

# 检查 Service
kubectl get svc -n <namespace>
kubectl describe svc <service-name> -n <namespace>

# 检查 Endpoints
kubectl get endpoints <service-name> -n <namespace>

# 端口转发测试
kubectl port-forward svc/<service-name> 8080:80 -n <namespace>

4. Pending 状态

# 查看原因
kubectl describe pod <pod-name> -n <namespace>

# 检查节点资源
kubectl describe nodes

# 查看 PVC 状态
kubectl get pvc -A

5. Docker 容器问题

# 查看日志
docker logs -f <container-name>
docker logs --tail 100 <container-name>

# 进入容器
docker exec -it <container-name> /bin/bash

# 重启容器
docker restart <container-name>

# 查看资源使用
docker stats <container-name>

日志位置

服务日志

• Gitea: docker logs gitea
• Harbor: docker logs harbor-core
• 系统日志: ~/ops/logs/

查看日志命令

# Docker 服务日志
docker logs -f <service-name> --tail 100

# K8s Pod 日志
kubectl logs -f <pod-name> -n <namespace>

# K8s 所有容器日志
kubectl logs <pod-name> -n <namespace> --all-containers=true

# 查看系统日志
tail -f ~/ops/logs/*.log

性能诊断

检查资源瓶颈

# Docker 资源使用
docker stats --no-stream

# K8s 资源使用
kubectl top nodes
kubectl top pods -A

# 磁盘 I/O
iostat -x 1 5

# 网络连接
netstat -an | grep ESTABLISHED | wc -l

清理资源

# 清理 Docker
docker system prune -a          # 清理所有未使用的资源
docker volume prune             # 清理未使用的卷
docker image prune -a           # 清理未使用的镜像

# 清理 K8s
kubectl delete pods --field-selector=status.phase==Failed -A
kubectl delete pods --field-selector=status.phase==Succeeded -A

快速检查清单

• Docker 守护进程运行正常: docker ps
• K8s 集群连接正常: kubectl cluster-info
• 所有关键服务运行: docker ps | grep -E "(harbor|gitea)"
• 磁盘空间充足: df -h /
• 没有异常 Pod: kubectl get pods -A --field-selector=status.phase!=Running
• 最近无严重事件: kubectl get events -A --sort-by='.lastTimestamp' | tail -20