Files

T

brunoandClaude 842fa8dbe5 feat: 完整AI SOC分析师平台 - LLM集成与前后端优化

本次提交包含完整的AI SOC（Security Operations Center）分析师平台，支持本地大模型和Claude API集成，实现端到端的安全威胁分析。

## 🚀 主要功能

### 后端架构 (FastAPI)
- **多源数据接入**: 支持JSON/CSV格式安全告警数据上传
- **智能预处理**: 基于YAML配置的字段映射和实体提取
- **图分析引擎**: NetworkX驱动的攻击链图构建和分析
- **评分系统**: 实体和子图的多维度风险评分算法
- **LLM集成**: 支持多提供商(Claude、OpenAI、本地模型)的智能报告生成
- **API接口**: 完整的RESTful API (Swagger文档: /docs)

### 前端界面 (Vue 3 + TypeScript)
- **拖拽上传**: 支持拖拽上传和实时预览
- **管道分析**: 可视化的数据处理管道状态
- **智能报告**: 动态生成的威胁分析报告和可视化
- **响应式设计**: 适配桌面和移动端的现代UI

### LLM智能分析
- **多模型支持**: qwen3(8B/32B)、deepseek-r1(70B)、Claude 4等
- **超时优化**: 分层超时配置(120s-600s)适配不同模型规模
- **内容清理**: 自动过滤模型思考过程，输出专业分析报告
- **中文优化**: 针对中文安全分析场景的prompt工程

## 🛠️ 技术栈

**后端**: FastAPI + SQLAlchemy + NetworkX + 多LLM提供商
**前端**: Vue 3 + TypeScript + Naive UI + Tailwind CSS + Vite
**数据处理**: YAML配置 + JSONPath表达式
**部署**: Docker支持 + 环境变量配置

## 🐛 修复的关键问题

1. **LLM超时问题**: 实现前后端分层超时配置，支持大模型长时间推理
2. **报告格式化**: 修复Markdown渲染和换行符显示问题
3. **数据映射**: 统一前后端字段命名和数据结构映射
4. **时间格式**: 修复时间戳解析和中文本地化显示
5. **思考标签清理**: 自动过滤qwen模型的<think>标签

## 📊 测试数据

包含完整的多阶段APT攻击场景测试数据:
- 钓鱼邮件 → 恶意软件执行 → C2通信
- 云凭证窃取 → 资源探测 → 横向渗透
- 后门用户创建 → 权限提升 → 容器控制

## 🔧 开发工具

- **模型切换**: switch_model.py 支持快速切换本地LLM模型
- **超时测试**: test_llm_timeout.py 验证不同模型性能
- **依赖安装**: install_llm_deps.py 自动安装LLM依赖

🤖 Generated with [Claude Code](https://claude.ai/code)

Co-Authored-By: Claude <noreply@anthropic.com>

2025-09-06 06:56:09 +08:00

6.8 KiB

Raw Permalink Blame History

AI SOC 分析师

AI-driven Security Operations Center analyst platform for intelligent threat detection and analysis.

Overview

This is a complete AI SOC Analyst platform that processes security alerts, builds threat graphs, and generates intelligent analysis reports. The system consists of a Vue 3 frontend and FastAPI backend with LLM-powered report generation.

Features

Multi-format Alert Ingestion: Support for JSON and CSV alert formats from various SIEM/XDR products
Intelligent Preprocessing: YAML-driven configuration for alert standardization and entity extraction
Graph-based Scoring: Entity relationship modeling and attack chain risk assessment
LLM-enhanced Reports: AI-powered threat analysis and actionable recommendations
Interactive Pipeline: Complete workflow from data ingestion to report generation
Real-time Dashboard: Vue 3 frontend for pipeline control and report visualization

Architecture

Frontend (Vue 3 + TypeScript + Vite)

src/api.ts: HTTP client with mock/real API switching via VITE_USE_MOCK
src/components/:
- UploadPanel.vue: File upload with configurable data sources
- PipelinePanel.vue: AI analysis pipeline control and monitoring
- ReportList.vue: Threat reports display with filtering
- ReportDetail.vue: Detailed analysis results drawer

Backend (FastAPI + Python)

app/api_main.py: Main FastAPI application with all endpoints
app/storage.py: In-memory storage for development and demo
app/dsl/: Configuration-driven alert preprocessing engine
app/edges.py: Graph construction from standardized alerts
app/scoring.py: Entity and subgraph risk scoring system
app/generate.py: Report generation with LLM integration
app/llm.py: OpenAI client for AI-powered analysis

Configuration System (`/configs`)

edge_scoring.yaml: Risk scoring weights and thresholds
llm.yaml: LLM configuration and prompt templates
splunk-es-alert-preprocessing.yaml: JSON alert preprocessing
csv-alerts-preprocessing.yaml: CSV alert preprocessing

Quick Start

1. Install Dependencies

Backend:

pip install -r requirements.txt

Frontend:

npm install

2. Start the Backend

# From project root
python run_backend.py

Backend will be available at:

API: http://localhost:8000
Documentation: http://localhost:8000/docs

3. Start the Frontend

npm run dev

Frontend will be available at: http://localhost:5173

4. Test the Complete Pipeline

Upload Sample Data: Use the provided test-data/sample-alerts.json file
Select Data Source: Choose "Splunk ES Alert (JSON)"
Run Pipeline: Click "构建威胁图谱" → "风险评分计算" → "生成分析报告"
View Results: Check the generated reports in the reports section

Configuration

Environment Variables

LLM Integration (Optional):

export LLM_API_BASE="https://api.openai.com/v1"
export LLM_API_KEY="your-openai-api-key"

Frontend Mock Mode:

# .env file
VITE_USE_MOCK=false  # Use real backend API
VITE_USE_MOCK=true   # Use mock data for testing

Scoring Configuration

Edit configs/edge_scoring.yaml to customize:

Severity weights (critical: 100, high: 70, etc.)
Attack stage weights (credential_access: 90, etc.)
Risk thresholds (entity: 70, subgraph: 300)
Time decay and burst detection parameters

Data Processing Pipeline

Alert Ingestion: Upload JSON/CSV files with security alerts
Preprocessing: YAML-driven field extraction and entity standardization
Graph Construction: Build entity relationships from alert data
Risk Scoring: Calculate entity and subgraph risk scores
Report Generation: Create LLM-enhanced threat analysis reports

API Endpoints

Core APIs (`/api`)

POST /ingest - Ingest alert data files
POST /dry_run - Preview preprocessing results
POST /graph/build - Build entity relationship graph
POST /scoring/run - Calculate risk scores
POST /reports/generate - Generate analysis reports
GET /reports - List generated reports
GET /reports/{id} - Get detailed report
GET /status - System status and statistics

Sample Data

The project includes realistic test data (test-data/sample-alerts.json) covering:

Phishing attack with malware execution
Cloud credential theft and exfiltration
Backdoor user creation and privilege escalation
Kubernetes compromise and container control

This demonstrates a complete attack chain spanning endpoint, cloud, and container environments.

Development Features

Mock Mode

Enable frontend mock mode for development:

// Set in .env or environment
VITE_USE_MOCK=true

In-Memory Storage

The backend uses in-memory storage for development, making it easy to:

Test different configurations quickly
Reset state by restarting the service
Run without external database dependencies

Hot Reload

Both frontend and backend support hot reload during development:

Frontend: Vite dev server with HMR
Backend: uvicorn with --reload flag

Production Deployment

For production deployment:

Database: Replace in-memory storage with PostgreSQL (models already defined)
Configuration: Set proper environment variables for LLM API keys
Security: Configure CORS, authentication, and rate limiting
Scaling: Use multiple worker processes for FastAPI backend

Advanced Configuration

Custom Data Sources

Add new data source configurations:

Create YAML file in configs/sources/
Define input mappings, transforms, and entity extraction rules
Update frontend source options in src/types.ts

Custom Scoring Rules

Modify configs/edge_scoring.yaml:

Adjust severity and attack stage weights
Configure time decay and burst detection
Set entity-specific thresholds
Customize cooldown and upgrade policies

LLM Customization

Edit configs/llm.yaml:

Change model and API settings
Customize prompt templates
Adjust temperature and token limits
Configure retry and fallback behavior

Troubleshooting

Backend won't start:

Check Python dependencies: pip install -r requirements.txt
Verify Python version: 3.10+ required

Frontend can't connect to backend:

Ensure backend is running on port 8000
Check CORS configuration in backend
Try mock mode: VITE_USE_MOCK=true

LLM reports fail:

Verify API key is set: echo $LLM_API_KEY
Check API endpoint: echo $LLM_API_BASE
System automatically falls back to template generation

No reports generated:

Check if entities exceed thresholds in scoring config
Review preprocessing results in dry-run mode
Verify graph construction produces edges

For more help, check the API documentation at http://localhost:8000/docs when the backend is running.

License

MIT License

6.8 KiB Raw Permalink Blame History