website/news-crawler-service

OmniVerse News Crawler Service

A standalone news crawler service for healthcare and robotics industry news, designed to integrate with the OmniVerse website's "业界动态" (Industry News) section.

Features

• RSS Feed Crawling: Automatically crawls major Chinese tech and healthcare news sources
• Content Extraction: Full article content and image extraction from source URLs
• Smart Filtering: Keyword-based filtering for healthcare, robotics, and related topics
• RESTful API: Express.js API server with comprehensive news endpoints
• SQLite Database: Local database storage with indexing and caching
• Scheduled Crawling: Configurable automated crawling with cron jobs
• Content Management: Automatic cleanup of old news and duplicate detection
• Caching: In-memory caching for improved API performance

Quick Start

Prerequisites

• Node.js >= 16.0.0
• NPM or Yarn

Installation

1. Navigate to the crawler service directory:

cd news-crawler-service

2. Install dependencies:

npm install

3. Copy environment configuration:

cp .env.example .env

4. Edit .env file with your configuration:

# Set your API key
API_KEY=your_secure_api_key_here

# Configure CORS for your frontend
CORS_ORIGIN=http://localhost:5173

# Adjust crawling interval (minutes)
CRAWL_INTERVAL_MINUTES=30

Running the Service

Development Mode

npm run dev

Production Mode

npm start

Manual Crawling

npm run crawler

The service will:

• Start on port 3001 (configurable via PORT env var)
• Automatically run initial crawl after 5 seconds
• Schedule periodic crawling based on CRAWL_INTERVAL_MINUTES
• Create SQLite database at ./data/news.db
• Log to ./logs/app.log

API Endpoints

All API endpoints require authentication via X-API-Key header or apikey query parameter.

Get News List

GET /api/news?page=1&limit=10&category=medical-robotics&query=search

Parameters:

• page: Page number (default: 1)
• limit: Items per page (default: 10, max: 50)
• category: Filter by category
• query: Search in title, summary, and tags
• sortBy: Sort field (default: published_at)
• sortOrder: ASC or DESC (default: DESC)

Get Hot News

GET /api/news/hot?limit=5

Get Statistics

GET /api/stats

Get Categories

GET /api/categories

Manual Crawl Trigger

POST /api/crawl

Health Check

GET /health

Configuration

Environment Variables

Variable	Default	Description
PORT	3001	Server port
NODE_ENV	development	Environment
DATABASE_PATH	./data/news.db	SQLite database file
CRAWL_INTERVAL_MINUTES	30	Crawling interval
MAX_NEWS_AGE_DAYS	7	Auto-cleanup threshold
ENABLE_SCHEDULER	true	Enable scheduled crawling
CACHE_TTL_SECONDS	300	Cache TTL for API responses
API_KEY	required	API authentication key
CORS_ORIGIN	http://localhost:5173	Allowed CORS origins
LOG_LEVEL	info	Logging level
LOG_FILE	./logs/app.log	Log file path

News Sources

The crawler monitors these RSS sources:

• 动脉网 (vcbeat.top) - Medical robotics focus
• 36氪 (36kr.com) - Investment and funding news
• 亿欧网 (iyiou.com) - Industry trends
• 雷锋网 (leiphone.com) - Tech innovation
• 机器之心 (jiqizhixin.com) - AI and robotics

Categories

• healthcare-policy - 康养政策
• medical-robotics - 医疗机器人
• smart-elderly-care - 智慧养老
• industry-trends - 行业趋势
• tech-innovation - 技术创新
• investment - 投资融资

CLI Usage

The crawler includes a command-line interface:

# Run crawl
node src/crawler.js crawl

# Show statistics
node src/crawler.js stats

# Clean up old news
node src/crawler.js cleanup

# Show help
node src/crawler.js help

Integration with Frontend

Update Frontend Service

Modify src/services/newsService.ts in your main website:

// Update API configuration
const NEWS_API_BASE = 'http://localhost:3001/api';
const API_KEY = 'your_api_key_here';

// Add API key to requests
const headers = {
  'X-API-Key': API_KEY,
  'Content-Type': 'application/json'
};

Example Frontend Integration

import { newsService } from '../services/newsService';

// Get news with filters
const news = await newsService.getNews({
  category: 'medical-robotics',
  page: 1,
  limit: 10
});

// Get hot news
const hotNews = await newsService.getHotNews(5);

// Get statistics
const stats = await newsService.getStats();

Development

Project Structure

news-crawler-service/
├── src/
│   ├── models/
│   │   └── database.js          # SQLite database operations
│   ├── crawlers/
│   │   └── rss-crawler.js       # RSS feed crawler
│   ├── server.js                # Express API server
│   └── crawler.js               # CLI interface
├── data/                        # SQLite database
├── logs/                        # Log files
├── package.json
├── .env.example
└── README.md

Adding New News Sources

Edit src/crawlers/rss-crawler.js and add to the sources array:

{
  name: 'Source Name',
  url: 'https://example.com/rss',
  category: 'medical-robotics',
  keywords: ['医疗', '机器人', 'AI'],
  priority: 'high',
  region: '全国'
}

Testing

npm test

Production Deployment

1. Set production environment:

export NODE_ENV=production

2. Use a process manager like PM2:

npm install -g pm2
pm2 start src/server.js --name "news-crawler"

3. Configure reverse proxy (nginx example):

location /api/news {
    proxy_pass http://localhost:3001;
    proxy_set_header Host $host;
    proxy_set_header X-Real-IP $remote_addr;
}

Troubleshooting

Common Issues

1. Database locked: Ensure only one crawler instance is running
2. RSS parsing fails: Check network connectivity and RSS URL validity
3. Memory usage: Adjust cache TTL and cleanup frequency
4. Rate limiting: Increase delays between requests in crawler

Monitoring

Check logs for crawler performance:

tail -f logs/app.log

Monitor database size:

ls -lh data/news.db

License

MIT License - see LICENSE file for details.