ljwx-stock/DATA_PIPELINE_INDEX.md

Data Pipeline Documentation Index

Overview

This directory contains comprehensive documentation of the Stock Recommendation System's data pipeline architecture, file organization, and implementation details.

Documentation Files

1. DATA_PIPELINE_ARCHITECTURE.md (20 KB)

Complete technical documentation of the entire data pipeline.

Contents:

• Executive summary of the pipeline
• Detailed description of all 5 pipeline layers:
  1. Data ingestion (TuShare, yfinance, OpenBB)
  2. Feature engineering (33 technical indicators)
  3. ML model training and prediction
  4. Caching system (Redis)
  5. Output and ranking
• Comprehensive feature engineering calculations with formulas
• Full request-to-recommendation data flow
• Database configuration and data models
• Data quality checks and validation
• Performance characteristics and metrics
• Configuration variables and environment setup
• Summary table of data availability

Best For: Understanding the complete architecture, feature calculations, and how all components interact.

Key Sections:

• Section 2: 5-layer architecture overview
• Section 4: 33 features explained with formulas
• Section 5: Complete request flow
• Section 6: Data availability at each stage

---

2. DATA_PIPELINE_QUICK_REFERENCE.md (13 KB)

Quick reference guide with visual diagrams and tables.

Contents:

• ASCII pipeline diagram with data flow
• Data fetching module summary with methods and cache
• Feature engineering categories and processing pipeline
• ML model layer architecture and score interpretation
• Caching system overview with hit rates
• Full request flow with timing breakdown
• Data availability by component (5 stages)
• Configuration keys and environment variables
• Performance metrics with latency breakdown
• Data quality checks and fallback strategy
• File organization and key data files
• Troubleshooting guide
• Performance optimization tips

Best For: Quick lookup, visual reference, performance troubleshooting, and optimization tips.

Key Sections:

• Section 5: Full request flow with timings
• Section 8: Performance metrics breakdown
• Section 11: Troubleshooting guide
• Section 12: Optimization tips

---

3. DATA_PIPELINE_FILE_MAP.md (20 KB)

Detailed file-by-file breakdown with code locations and exact line numbers.

Contents:

• Complete directory structure with annotations
• 7 core pipeline modules:
  1. Data ingestion (data_fetcher.py)
  2. Feature engineering (feature_engineer.py)
  3. ML training (ml_trainer.py)
  4. Caching (cache_manager.py)
  5. LLM integration (llm_ranker.py)
  6. Backtesting (backtester.py)
  7. Alternative provider (openbb_provider.py)
• 3 API route handlers with endpoint descriptions
• Configuration and database setup
• Main application entry point
• Standalone data fetching script
• Key data types and transformations (5 stages)
• Data flow connection paths (3 main paths)
• Troubleshooting with module-by-module tracing
• Important constants and settings

Best For: Finding specific code, understanding module organization, tracing data flow through modules.

Key Sections:

• Section 1: Directory structure with file purposes
• Section 4: Exact method names and line numbers
• "Data Flow Connections Summary": 3 main request paths
• "Troubleshooting": Tracing data through code

---

4. ANALYSIS_SUMMARY.txt (19 KB)

Executive summary report of the entire pipeline analysis.

Contents:

• Executive summary (5 layers overview)
• Complete pipeline architecture breakdown
• 3 data sources with details:
  1. TuShare (Chinese A-stocks)
  2. yfinance (Global fallback)
  3. OpenBB (Optional unified provider)
• Default watchlist of 4 Chinese stocks
• 33 features explained with formulas:
  • Moving averages (11)
  • Momentum indicators (8)
  • Volatility indicators (8)
  • Trend indicators (4)
• ML model architecture (XGBoost/LightGBM)
• Detailed caching system with TTLs
• Complete request flow (9 steps)
• Configuration variables
• Performance metrics and data volumes
• Data quality validation
• Key files and absolute paths
• Conclusions and recommendations

Best For: Executive overview, presentations, understanding strengths and areas for improvement.

Key Sections:

• Section 2: Pipeline layers
• Section 7: Full request flow
• Section 9: Performance metrics
• Section 12: Conclusions and next steps

---

Quick Navigation Guide

I Want To...

Understand the big picture: → Start with ANALYSIS_SUMMARY.txt (Section 2)

See how data flows through the system: → Check DATA_PIPELINE_ARCHITECTURE.md (Section 5) → Or DATA_PIPELINE_QUICK_REFERENCE.md (Section 5)

Find specific code: → Use DATA_PIPELINE_FILE_MAP.md with line numbers → All absolute paths: /root/work/stock/backend/app/modules/

Understand feature engineering: → DATA_PIPELINE_ARCHITECTURE.md (Section 2.2 for calculations) → ANALYSIS_SUMMARY.txt (Section 4 for formulas)

Troubleshoot performance: → DATA_PIPELINE_QUICK_REFERENCE.md (Section 8) → DATA_PIPELINE_QUICK_REFERENCE.md (Section 11-12)

See what data is available at each stage: → DATA_PIPELINE_QUICK_REFERENCE.md (Section 6) → DATA_PIPELINE_ARCHITECTURE.md (Section 6.2)

Understand caching: → DATA_PIPELINE_QUICK_REFERENCE.md (Section 4) → DATA_PIPELINE_ARCHITECTURE.md (Section 4)

Configure the system: → DATA_PIPELINE_QUICK_REFERENCE.md (Section 7) → ANALYSIS_SUMMARY.txt (Section 8)

Check production readiness: → ANALYSIS_SUMMARY.txt (Section 12)

---

Key Statistics

Data Pipeline Scope:

• Files analyzed: 15 modules
• Code lines: 2,000+
• API endpoints: 6+
• Features engineered: 33 technical indicators
• Cache keys: 7 types
• Default stocks: 4 Chinese A-stocks
• Data sources: 3 (TuShare, yfinance, OpenBB)

Performance:

• Full request latency: 7-14 seconds (no cache)
• Cached response: <10ms
• Expected cache hit rate: 95%
• Per stock memory: 40-150 KB
• Final response size: 5-10 KB

Pipeline Layers:

1. Data Ingestion: TuShare/yfinance (1-3 seconds)
2. Feature Engineering: 33 indicators (0.5 seconds)
3. ML Prediction: XGBoost inference (0.3 seconds)
4. Ranking: Top 10 selection (0.2 seconds)
5. LLM Explanation: Claude API (2-3 seconds)

---

Module Cross-Reference

Core Data Processing:

data_fetcher.py
    ↓ (provides OHLCV data)
feature_engineer.py
    ↓ (generates 33 features)
ml_trainer.py
    ↓ (produces scores 0-100)
llm_ranker.py
    ↓ (ranking and explanation)
API Response (JSON)

Supporting Services:

• cache_manager.py: Redis caching throughout pipeline
• config.py: Configuration for all modules
• database.py: PostgreSQL persistence (prepared)
• backtester.py: Strategy validation

API Routes:

• recommendations.py: Main endpoint, uses full pipeline
• analysis.py: Market and signal analysis
• backtest.py: Strategy backtesting

---

Data Availability Summary

Stage	Data Type	Format	Size	Update Rate
1. Raw	OHLCV	DataFrame	40-50 KB	On request
2. Features	33 indicators	DataFrame	100-150 KB	On request
3. ML Input	Normalized features	NumPy array	<1 KB	On request
4. Scores	0-100 predictions	Float array	<1 KB	60s cache
5. Output	Recommendations	JSON	5-10 KB	60s cache

---

Feature Engineering Summary

33 Total Features:

Moving Averages (11):

• MA5, MA10, MA20, MA50, MA200
• Price_MA5-200_Ratio (4)
• MA5_MA20_Cross (1 signal)

Momentum (8):

• RSI_14, MACD, MACD_Signal, MACD_Hist
• Stoch_K, Stoch_D, ROC_12, Momentum_10

Volatility (8):

• ATR_14
• BB_Middle, BB_Upper, BB_Lower, BB_Width, BB_Position
• Volatility_20, Volatility_60

Trend (4):

• Momentum_5d, Momentum_20d
• Volume_Ratio, Avg_Return_10d

---

Configuration Essentials

Required:

TUSHARE_TOKEN     → Chinese stock API access
REDIS_URL         → Cache backend
OPENROUTER_KEY    → Claude API access

Important:

STOCK_SYMBOLS              → "600000,600031,601988,601398"
MODEL_TYPE                 → "xgboost" or "lightgbm"
ENABLE_TECHNICAL_FEATURES  → true/false
CACHE_TTL_*                → Performance tuning

---

Request Flow Overview

GET /api/recommendations
    ↓
Check cache (< 10ms)
    ↓ (miss) → Full pipeline (7-14s)
    ├─ Fetch data (1-3s)
    ├─ Engineer features (0.5s)
    ├─ ML prediction (0.3s)
    ├─ Rank & select (0.2s)
    ├─ LLM explanation (2-3s)
    └─ Cache result (60s)
    ↓
Return JSON response

---

File Locations (Absolute Paths)

Core Modules:

/root/work/stock/backend/app/modules/data_fetcher.py
/root/work/stock/backend/app/modules/feature_engineer.py
/root/work/stock/backend/app/modules/ml_trainer.py
/root/work/stock/backend/app/modules/cache_manager.py
/root/work/stock/backend/app/modules/llm_ranker.py

Configuration:

/root/work/stock/backend/app/config.py
/root/work/stock/backend/app/database.py
/root/work/stock/backend/main.py

API Routes:

/root/work/stock/backend/app/routes/recommendations.py
/root/work/stock/backend/app/routes/analysis.py

---

Documentation Revision History

Date	Version	Changes
2025-11-10	1.0	Initial analysis and documentation

---

How to Use This Documentation

1. For onboarding: Start with ANALYSIS_SUMMARY.txt
2. For debugging: Use DATA_PIPELINE_FILE_MAP.md with line numbers
3. For optimization: Check DATA_PIPELINE_QUICK_REFERENCE.md Section 12
4. For architecture understanding: Read DATA_PIPELINE_ARCHITECTURE.md
5. For quick lookup: Use DATA_PIPELINE_QUICK_REFERENCE.md

---

Questions Answered

"Where does data come from?"

→ ANALYSIS_SUMMARY.txt Section 3, or DATA_PIPELINE_QUICK_REFERENCE.md Section 1

"How are recommendations generated?"

→ DATA_PIPELINE_QUICK_REFERENCE.md Section 5, or ANALYSIS_SUMMARY.txt Section 7

"What features are used in the ML model?"

→ ANALYSIS_SUMMARY.txt Section 4, or DATA_PIPELINE_ARCHITECTURE.md Section 2.2

"How fast is the system?"

→ ANALYSIS_SUMMARY.txt Section 9, or DATA_PIPELINE_QUICK_REFERENCE.md Section 8

"What's the code structure?"

→ DATA_PIPELINE_FILE_MAP.md Section 1 with line numbers

"What configuration is needed?"

→ DATA_PIPELINE_QUICK_REFERENCE.md Section 7, or ANALYSIS_SUMMARY.txt Section 8

"How does caching work?"

→ DATA_PIPELINE_QUICK_REFERENCE.md Section 4, or ANALYSIS_SUMMARY.txt Section 6

"What's the data flow?"

→ DATA_PIPELINE_QUICK_REFERENCE.md Section 5, or ANALYSIS_SUMMARY.txt Section 7

---

Last Updated: 2025-11-10
Analysis Tool: Claude Code (AI Assistant)
Codebase Version: Latest from /root/work/stock/backend/