use markdown as source
This commit is contained in:
@@ -0,0 +1,669 @@
|
||||
# 📰 OmniVerse 新闻爬虫系统使用指南
|
||||
|
||||
## 🎯 系统概述
|
||||
|
||||
OmniVerse 新闻爬虫系统是一个功能完善的自动化新闻抓取、分类、AI摘要生成和展示系统。系统从多个权威医疗健康资讯源自动抓取相关新闻，通过AI生成智能摘要，并在前端页面展示。
|
||||
|
||||
### 核心特性
|
||||
|
||||
✅ **自动化RSS抓取** - 定时从17+个医疗健康领域RSS源抓取新闻
|
||||
✅ **智能分类** - 使用关键词权重算法自动分类新闻
|
||||
✅ **AI摘要生成** - 集成Ollama本地大模型生成新闻摘要
|
||||
✅ **定时任务** - 支持定时爬取和数据清理
|
||||
✅ **数据导出** - 自动将数据导出为JSON供前端使用
|
||||
✅ **精美前端** - 响应式设计，支持多种浏览模式
|
||||
|
||||
---
|
||||
|
||||
## 📊 新闻分类体系
|
||||
|
||||
系统支持以下6大新闻分类：
|
||||
|
||||
| 分类ID | 中文名称 | 说明 | 图标 |
|
||||
|--------|---------|------|------|
|
||||
| `healthcare-policy` | 康养政策 | 政策法规、行业标准、监管动态 | 🏥 |
|
||||
| `medical-robotics` | 医疗机器人 | 手术机器人、康复机器人、护理机器人 | 🤖 |
|
||||
| `smart-elderly-care` | 智慧养老 | 养老科技、护理创新、银发经济 | 👴 |
|
||||
| `tech-innovation` | 技术前沿 | AI医疗、数字健康、创新技术 | 🚀 |
|
||||
| `investment` | 投融资 | 投资动态、融资消息、市场估值 | 💰 |
|
||||
| `industry-trends` | 行业动态 | 市场分析、发展趋势、行业报告 | 📈 |
|
||||
|
||||
---
|
||||
|
||||
## 🔧 快速开始
|
||||
|
||||
### 方法一：使用验证脚本（推荐）
|
||||
|
||||
```bash
|
||||
# 运行交互式验证脚本
|
||||
./verify-news.sh
|
||||
```
|
||||
|
||||
脚本会自动检查环境、依赖，并提供以下选项：
|
||||
1. 运行爬虫 (抓取最新新闻)
|
||||
2. 导出数据为 JSON
|
||||
3. **爬取并导出** ⭐ 推荐选项
|
||||
4. 查看数据库统计
|
||||
5. 启动爬虫服务
|
||||
6. 退出
|
||||
|
||||
### 方法二：手动运行命令
|
||||
|
||||
```bash
|
||||
# 进入爬虫服务目录
|
||||
cd news-crawler-service
|
||||
|
||||
# 安装依赖
|
||||
npm install
|
||||
|
||||
# 运行爬虫并导出数据
|
||||
npm run crawl-and-export
|
||||
|
||||
# 或分步执行
|
||||
npm run crawler # 抓取新闻
|
||||
npm run export # 导出为JSON
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🌐 启动爬虫服务
|
||||
|
||||
### 启动API服务
|
||||
|
||||
```bash
|
||||
cd news-crawler-service
|
||||
npm start
|
||||
```
|
||||
|
||||
服务将在 `http://localhost:3001` 启动
|
||||
|
||||
### 启动前端开发服务器
|
||||
|
||||
```bash
|
||||
# 在项目根目录
|
||||
npm run dev
|
||||
```
|
||||
|
||||
前端将在 `http://localhost:5174` 启动
|
||||
|
||||
访问业界新闻页面：
|
||||
```
|
||||
http://localhost:5174/website/industry-news
|
||||
或
|
||||
http://192.168.1.83:5174/website/industry-news
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📡 API 端点
|
||||
|
||||
### 获取新闻列表
|
||||
```http
|
||||
GET /api/news?page=1&limit=12&category=tech-innovation&sortBy=publishedAt
|
||||
```
|
||||
|
||||
**查询参数:**
|
||||
- `page` - 页码 (默认: 1)
|
||||
- `limit` - 每页条数 (默认: 10, 最大: 50)
|
||||
- `category` - 分类过滤
|
||||
- `query` - 搜索关键词
|
||||
- `sortBy` - 排序字段 (publishedAt/viewCount/priority)
|
||||
- `sortOrder` - 排序方向 (DESC/ASC)
|
||||
|
||||
### 获取热门新闻
|
||||
```http
|
||||
GET /api/news/hot?limit=5
|
||||
```
|
||||
|
||||
### 获取统计信息
|
||||
```http
|
||||
GET /api/stats
|
||||
```
|
||||
|
||||
### 手动触发爬取
|
||||
```http
|
||||
POST /api/crawl
|
||||
Headers: X-API-Key: omniverse_news_2024
|
||||
```
|
||||
|
||||
### 获取新闻分类
|
||||
```http
|
||||
GET /api/categories
|
||||
```
|
||||
|
||||
### 健康检查
|
||||
```http
|
||||
GET /health
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## ⚙️ 配置说明
|
||||
|
||||
### 环境变量配置 (.env)
|
||||
|
||||
```bash
|
||||
# 服务器设置
|
||||
PORT=3001
|
||||
NODE_ENV=development
|
||||
|
||||
# 数据库路径
|
||||
DATABASE_PATH=./data/news.db
|
||||
|
||||
# 爬虫设置
|
||||
CRAWL_INTERVAL_MINUTES=30 # 爬取间隔（分钟）
|
||||
MAX_NEWS_AGE_DAYS=7 # 新闻保留天数
|
||||
ENABLE_SCHEDULER=true # 启用定时任务
|
||||
|
||||
# 缓存设置
|
||||
CACHE_TTL_SECONDS=300 # 缓存有效期（秒）
|
||||
|
||||
# API设置
|
||||
API_KEY=omniverse_news_2024 # API密钥
|
||||
CORS_ORIGIN=http://localhost:5173,http://localhost:5174,http://192.168.1.83:5174
|
||||
|
||||
# 日志设置
|
||||
LOG_LEVEL=info
|
||||
LOG_FILE=./logs/app.log
|
||||
|
||||
# Ollama AI 设置
|
||||
OLLAMA_URL=http://localhost:11434
|
||||
OLLAMA_MODEL=qwen3:32b # 使用的AI模型
|
||||
|
||||
# 导出设置
|
||||
AUTO_EXPORT_JSON=true # 爬取后自动导出JSON
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🤖 AI 摘要功能
|
||||
|
||||
系统使用 Ollama 本地大模型生成新闻摘要。
|
||||
|
||||
### 安装 Ollama
|
||||
|
||||
```bash
|
||||
# macOS
|
||||
brew install ollama
|
||||
|
||||
# Linux
|
||||
curl -fsSL https://ollama.ai/install.sh | sh
|
||||
|
||||
# Windows
|
||||
# 从 https://ollama.ai 下载安装包
|
||||
```
|
||||
|
||||
### 下载并启动模型
|
||||
|
||||
```bash
|
||||
# 下载 qwen3:32b 模型（推荐）
|
||||
ollama pull qwen3:32b
|
||||
|
||||
# 或使用其他中文模型
|
||||
ollama pull qwen:7b
|
||||
ollama pull glm4:9b
|
||||
|
||||
# 启动 Ollama 服务
|
||||
ollama serve
|
||||
```
|
||||
|
||||
### 验证 Ollama 服务
|
||||
|
||||
```bash
|
||||
curl http://localhost:11434/api/tags
|
||||
```
|
||||
|
||||
> **注意**: 如果 Ollama 服务未启动，爬虫仍会正常工作，但会使用备用摘要生成算法（提取前几句话）。
|
||||
|
||||
---
|
||||
|
||||
## 📁 数据存储
|
||||
|
||||
### SQLite 数据库
|
||||
|
||||
**位置**: `news-crawler-service/data/news.db`
|
||||
|
||||
**表结构**:
|
||||
```sql
|
||||
CREATE TABLE news (
|
||||
id TEXT PRIMARY KEY, -- UUID
|
||||
title TEXT NOT NULL, -- 标题
|
||||
summary TEXT, -- AI摘要
|
||||
content TEXT, -- Markdown内容
|
||||
category TEXT NOT NULL, -- 分类
|
||||
tags TEXT, -- JSON数组
|
||||
source TEXT NOT NULL, -- 来源
|
||||
source_url TEXT NOT NULL, -- 原文链接
|
||||
published_at TEXT NOT NULL, -- 发布时间
|
||||
crawled_at TEXT NOT NULL, -- 抓取时间
|
||||
author TEXT, -- 作者
|
||||
image_url TEXT, -- 图片URL
|
||||
priority TEXT, -- 优先级 (high/medium/low)
|
||||
region TEXT, -- 地区
|
||||
is_hot INTEGER, -- 是否热门
|
||||
view_count INTEGER, -- 浏览量
|
||||
created_at DATETIME, -- 创建时间
|
||||
updated_at DATETIME -- 更新时间
|
||||
);
|
||||
```
|
||||
|
||||
### JSON 导出文件
|
||||
|
||||
**位置**: `public/data/news.json`
|
||||
|
||||
格式示例：
|
||||
```json
|
||||
[
|
||||
{
|
||||
"id": "uuid-here",
|
||||
"title": "新闻标题",
|
||||
"summary": "AI生成的摘要...",
|
||||
"content": "Markdown格式的完整内容",
|
||||
"category": "medical-robotics",
|
||||
"tags": ["机器人", "手术", "创新"],
|
||||
"source": "动脉网",
|
||||
"sourceUrl": "https://...",
|
||||
"publishedAt": "2025-10-06T07:00:00.000Z",
|
||||
"crawledAt": "2025-10-06T12:51:54.926Z",
|
||||
"author": "作者名",
|
||||
"imageUrl": "https://...",
|
||||
"priority": "high",
|
||||
"region": "全国",
|
||||
"isHot": true,
|
||||
"viewCount": 1500
|
||||
}
|
||||
]
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📰 RSS 新闻源列表
|
||||
|
||||
系统监控以下17个医疗健康领域的权威新闻源：
|
||||
|
||||
### 综合科技类
|
||||
- **36氪** - 投资融资动态
|
||||
- **亿欧网** - 行业趋势分析
|
||||
- **雷锋网** - AI技术创新
|
||||
- **机器之心** - AI医疗前沿
|
||||
|
||||
### 医疗专业类
|
||||
- **动脉网** - 医疗机器人与创新
|
||||
- **医谷** - 医疗器械设备
|
||||
- **器械之家** - 医疗器械行业
|
||||
- **医学界** - 医学政策与研究
|
||||
- **医信邦** - 数字医疗
|
||||
- **医疗器械创新网** - 器械技术创新
|
||||
|
||||
### 政策法规类
|
||||
- **健康界** - 政策法规
|
||||
- **丁香园** - 医疗监管
|
||||
- **医药网** - 医药政策
|
||||
|
||||
### 技术研发类
|
||||
- **生物谷** - 生物技术研究
|
||||
- **CPhI制药在线** - 制药创新
|
||||
|
||||
### 养老护理类
|
||||
- **康复医学网** - 康复机器人
|
||||
- **养老网** - 智慧养老
|
||||
|
||||
---
|
||||
|
||||
## 🎨 前端页面功能
|
||||
|
||||
### 业界动态页面 (`/industry-news`)
|
||||
|
||||
#### 功能特性
|
||||
|
||||
1. **分类筛选** - 7个分类标签快速切换
|
||||
2. **搜索功能** - 全文搜索标题、内容、标签
|
||||
3. **多种排序** - 按发布时间、浏览量、重要性排序
|
||||
4. **视图切换** - 网格视图 / 列表视图
|
||||
5. **热门新闻** - 独立展示热门资讯
|
||||
6. **统计面板** - 实时显示新闻数量、来源统计
|
||||
7. **无限滚动** - 支持分页加载更多
|
||||
8. **响应式设计** - 完美适配各种屏幕尺寸
|
||||
|
||||
#### 新闻卡片信息
|
||||
|
||||
- **标题** - 新闻标题
|
||||
- **AI摘要** - 智能生成的摘要
|
||||
- **图片** - 新闻配图
|
||||
- **分类标签** - 所属分类
|
||||
- **标签云** - 关键词标签
|
||||
- **发布时间** - 相对时间显示
|
||||
- **浏览量** - 阅读统计
|
||||
- **来源** - 新闻来源网站
|
||||
- **优先级** - 重要程度标记
|
||||
- **热门标记** - 热门新闻标识
|
||||
|
||||
---
|
||||
|
||||
## 🔄 定时任务
|
||||
|
||||
系统支持两个定时任务：
|
||||
|
||||
### 1. 定时爬取任务
|
||||
|
||||
```javascript
|
||||
// 每30分钟运行一次（可配置）
|
||||
cron.schedule(`*/${CRAWL_INTERVAL_MINUTES} * * * *`, async () => {
|
||||
// 爬取新闻
|
||||
// 自动导出JSON
|
||||
});
|
||||
```
|
||||
|
||||
### 2. 清理旧数据任务
|
||||
|
||||
```javascript
|
||||
// 每天凌晨2点运行
|
||||
cron.schedule('0 2 * * *', async () => {
|
||||
// 删除7天前的新闻（可配置）
|
||||
});
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📈 使用流程
|
||||
|
||||
### 完整工作流程
|
||||
|
||||
```
|
||||
1. RSS源监控
|
||||
↓
|
||||
2. 内容抓取与解析
|
||||
↓
|
||||
3. 关键词匹配与过滤
|
||||
↓
|
||||
4. 智能分类
|
||||
↓
|
||||
5. AI摘要生成
|
||||
↓
|
||||
6. 存储到数据库
|
||||
↓
|
||||
7. 导出为JSON
|
||||
↓
|
||||
8. 前端页面展示
|
||||
```
|
||||
|
||||
### 日常维护操作
|
||||
|
||||
```bash
|
||||
# 1. 手动触发一次完整爬取
|
||||
cd news-crawler-service
|
||||
npm run crawl-and-export
|
||||
|
||||
# 2. 查看数据库统计
|
||||
sqlite3 news-crawler-service/data/news.db "SELECT category, COUNT(*) FROM news GROUP BY category;"
|
||||
|
||||
# 3. 查看最新新闻
|
||||
sqlite3 news-crawler-service/data/news.db "SELECT title, source, published_at FROM news ORDER BY published_at DESC LIMIT 10;"
|
||||
|
||||
# 4. 清理旧数据
|
||||
node news-crawler-service/src/crawler.js cleanup
|
||||
|
||||
# 5. 查看日志
|
||||
tail -f news-crawler-service/logs/app.log
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🐛 故障排查
|
||||
|
||||
### 问题1: 爬虫抓取失败
|
||||
|
||||
**症状**: 运行爬虫后没有新数据
|
||||
|
||||
**排查步骤**:
|
||||
1. 检查网络连接
|
||||
2. 验证RSS源是否可访问
|
||||
3. 查看日志文件 `logs/app.log`
|
||||
4. 检查数据库连接
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 测试网络连接
|
||||
curl https://vcbeat.top/rss
|
||||
|
||||
# 查看详细日志
|
||||
tail -100 news-crawler-service/logs/app.log
|
||||
```
|
||||
|
||||
### 问题2: AI摘要生成失败
|
||||
|
||||
**症状**: 摘要内容是简单的句子提取
|
||||
|
||||
**排查步骤**:
|
||||
1. 检查Ollama服务是否运行
|
||||
2. 验证模型是否已下载
|
||||
3. 检查OLLAMA_URL配置
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 启动Ollama服务
|
||||
ollama serve
|
||||
|
||||
# 下载模型
|
||||
ollama pull qwen3:32b
|
||||
|
||||
# 测试API
|
||||
curl http://localhost:11434/api/tags
|
||||
```
|
||||
|
||||
### 问题3: 前端无法显示新闻
|
||||
|
||||
**症状**: 业界动态页面为空
|
||||
|
||||
**排查步骤**:
|
||||
1. 检查 `public/data/news.json` 是否存在
|
||||
2. 验证JSON文件格式是否正确
|
||||
3. 检查浏览器控制台错误
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 重新导出数据
|
||||
cd news-crawler-service
|
||||
npm run export
|
||||
|
||||
# 验证JSON文件
|
||||
cat ../public/data/news.json | jq 'length'
|
||||
```
|
||||
|
||||
### 问题4: 爬虫服务启动失败
|
||||
|
||||
**症状**: `npm start` 报错
|
||||
|
||||
**排查步骤**:
|
||||
1. 检查端口3001是否被占用
|
||||
2. 验证环境变量配置
|
||||
3. 检查依赖是否完整安装
|
||||
|
||||
**解决方案**:
|
||||
```bash
|
||||
# 检查端口占用
|
||||
lsof -i :3001
|
||||
|
||||
# 重新安装依赖
|
||||
rm -rf node_modules package-lock.json
|
||||
npm install
|
||||
|
||||
# 检查环境配置
|
||||
cat .env
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 开发指南
|
||||
|
||||
### 添加新的RSS源
|
||||
|
||||
编辑 `news-crawler-service/src/crawlers/rss-crawler.js`:
|
||||
|
||||
```javascript
|
||||
this.sources = [
|
||||
// ... 现有源
|
||||
{
|
||||
name: '新闻源名称',
|
||||
url: 'https://example.com/rss',
|
||||
category: 'tech-innovation', // 默认分类
|
||||
keywords: ['关键词1', '关键词2', '关键词3'],
|
||||
priority: 'high', // high/medium/low
|
||||
region: '全国'
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
### 修改分类规则
|
||||
|
||||
编辑智能分类算法权重:
|
||||
|
||||
```javascript
|
||||
// rss-crawler.js - smartCategorize() 方法
|
||||
const categoryRules = {
|
||||
'your-category': {
|
||||
keywords: {
|
||||
'high': ['重要关键词1', '重要关键词2'],
|
||||
'medium': ['一般关键词1', '一般关键词2'],
|
||||
'low': ['次要关键词1', '次要关键词2']
|
||||
},
|
||||
titleWeight: 5, // 标题权重
|
||||
contentWeight: 3 // 内容权重
|
||||
}
|
||||
};
|
||||
```
|
||||
|
||||
### 自定义前端样式
|
||||
|
||||
编辑 `src/pages/IndustryNews.tsx`:
|
||||
|
||||
```tsx
|
||||
const newsCategories = [
|
||||
{
|
||||
value: 'new-category',
|
||||
label: '新分类',
|
||||
icon: '🎯',
|
||||
color: 'from-blue-500 to-cyan-500'
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 🚀 部署建议
|
||||
|
||||
### 生产环境配置
|
||||
|
||||
```bash
|
||||
# 1. 修改环境变量
|
||||
NODE_ENV=production
|
||||
ENABLE_SCHEDULER=true
|
||||
CRAWL_INTERVAL_MINUTES=30
|
||||
MAX_NEWS_AGE_DAYS=30
|
||||
AUTO_EXPORT_JSON=true
|
||||
|
||||
# 2. 使用PM2管理进程
|
||||
npm install -g pm2
|
||||
pm2 start news-crawler-service/src/server.js --name omniverse-news
|
||||
|
||||
# 3. 配置开机自启
|
||||
pm2 startup
|
||||
pm2 save
|
||||
|
||||
# 4. 查看日志
|
||||
pm2 logs omniverse-news
|
||||
```
|
||||
|
||||
### Docker 部署
|
||||
|
||||
项目已包含 `docker-compose.yml` 配置:
|
||||
|
||||
```bash
|
||||
# 启动服务
|
||||
docker-compose up -d
|
||||
|
||||
# 查看日志
|
||||
docker-compose logs -f news-crawler
|
||||
|
||||
# 停止服务
|
||||
docker-compose down
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 性能优化
|
||||
|
||||
### 缓存策略
|
||||
|
||||
- API响应缓存: 5分钟 (可配置)
|
||||
- 前端数据缓存: 5分钟
|
||||
- 数据库索引: category, published_at, source
|
||||
|
||||
### 爬取优化
|
||||
|
||||
- 并发爬取: 顺序执行 + 2秒延迟
|
||||
- 去重检查: source_url唯一性
|
||||
- 增量更新: 仅抓取新内容
|
||||
|
||||
### 前端优化
|
||||
|
||||
- 图片懒加载
|
||||
- 虚拟列表滚动
|
||||
- 分页加载 (每页12条)
|
||||
- 响应式图片
|
||||
|
||||
---
|
||||
|
||||
## 🔐 安全建议
|
||||
|
||||
1. **修改默认API密钥**
|
||||
```bash
|
||||
API_KEY=your_secure_random_key_here
|
||||
```
|
||||
|
||||
2. **配置CORS白名单**
|
||||
```bash
|
||||
CORS_ORIGIN=https://yourdomain.com
|
||||
```
|
||||
|
||||
3. **使用HTTPS**
|
||||
- 部署时配置SSL证书
|
||||
- 强制HTTPS重定向
|
||||
|
||||
4. **限制API请求频率**
|
||||
- 添加rate limiting中间件
|
||||
- 防止恶意爬取
|
||||
|
||||
5. **定期备份数据库**
|
||||
```bash
|
||||
cp news-crawler-service/data/news.db backups/news_$(date +%Y%m%d).db
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📞 技术支持
|
||||
|
||||
- **项目地址**: `/Users/brunogao/work/codes/web/omniverse-website`
|
||||
- **问题反馈**: 提交 Issue 或 Pull Request
|
||||
- **文档更新**: 2025-11-17
|
||||
|
||||
---
|
||||
|
||||
## 🎉 总结
|
||||
|
||||
OmniVerse 新闻爬虫系统已完成优化，现在可以：
|
||||
|
||||
✅ 自动从17+个医疗健康领域RSS源抓取新闻
|
||||
✅ 使用AI智能生成新闻摘要
|
||||
✅ 自动分类并存储到数据库
|
||||
✅ 定时任务自动运行
|
||||
✅ 导出JSON供前端使用
|
||||
✅ 精美的前端展示页面
|
||||
|
||||
**立即开始使用**:
|
||||
```bash
|
||||
./verify-news.sh
|
||||
```
|
||||
选择选项3: "爬取并导出"
|
||||
|
||||
祝使用愉快! 🚀
|
||||
@@ -0,0 +1,208 @@
|
||||
# 🎯 真实可用的医疗健康RSS新闻源配置
|
||||
|
||||
## 测试结果总结（2025-11-18）
|
||||
|
||||
### ✅ 强烈推荐（已验证可用，内容最新）
|
||||
|
||||
| 来源 | URL | 相关度 | 更新频率 | 内容质量 |
|
||||
|------|-----|--------|----------|----------|
|
||||
| **36氪** | https://36kr.com/feed | 100% | ⭐⭐⭐⭐⭐ 实时 | 医疗投融资、健康科技、AI医疗 |
|
||||
| **雷锋网** | https://www.leiphone.com/feed | 70% | ⭐⭐⭐⭐⭐ 每日 | AI、医疗机器人、技术创新 |
|
||||
| **虎嗅网** | https://www.huxiu.com/rss/0.xml | 20% | ⭐⭐⭐⭐ 每日 | 行业趋势、深度分析 |
|
||||
|
||||
### ⚠️ 可选（需要特殊处理）
|
||||
|
||||
| 来源 | URL | 状态 | 说明 |
|
||||
|------|-----|------|------|
|
||||
| 丁香园 | https://www.dxy.cn/bbs/newweb/pc/rss/1 | 可访问但格式异常 | XML解析错误 |
|
||||
| 生物谷 | http://www.bioon.com/rss/nrss.xml | 可访问 | 需要验证内容更新频率 |
|
||||
|
||||
### ❌ 不推荐（问题源）
|
||||
|
||||
| 来源 | 问题 | 建议 |
|
||||
|------|------|------|
|
||||
| 新华网健康 | RSS feed内容陈旧（2013年） | 移除 |
|
||||
| 人民网健康 | 404错误 | 移除 |
|
||||
| 网易健康 | 404错误 | 移除 |
|
||||
| 健康界 | 404错误 | 移除 |
|
||||
| 医学界 | 404错误 | 移除 |
|
||||
|
||||
---
|
||||
|
||||
## 🎯 推荐的爬虫配置
|
||||
|
||||
基于测试结果，以下是优化后的配置：
|
||||
|
||||
```javascript
|
||||
// RSS源配置 - 仅包含真实可用且内容最新的源
|
||||
this.sources = [
|
||||
// === 核心科技媒体（强烈推荐）===
|
||||
{
|
||||
name: '36氪',
|
||||
url: 'https://36kr.com/feed',
|
||||
category: 'investment',
|
||||
keywords: ['医疗', '健康', '机器人', '融资', '投资', '康养', '养老', '数字健康', '医药', 'AI'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '雷锋网',
|
||||
url: 'https://www.leiphone.com/feed',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['AI', '医疗', '机器人', '智能', '算法', '诊断', '技术', '创新', '健康'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '虎嗅网',
|
||||
url: 'https://www.huxiu.com/rss/0.xml',
|
||||
category: 'industry-trends',
|
||||
keywords: ['医疗', '健康', '智慧', '数字化', '机器人', '康养', '产业', '趋势', 'AI'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
|
||||
// === 补充科技媒体（扩展覆盖）===
|
||||
{
|
||||
name: '钛媒体',
|
||||
url: 'https://www.tmtpost.com/rss.xml',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['医疗', '健康', '科技', '创新', 'AI', '智能', '数字'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: 'CSDN',
|
||||
url: 'https://blog.csdn.net/rss/home',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['医疗', 'AI', '算法', '机器学习', '深度学习', '计算机视觉'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
|
||||
// === 垂直医疗科技（可选，需验证）===
|
||||
{
|
||||
name: '生物谷',
|
||||
url: 'http://www.bioon.com/rss/nrss.xml',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['生物技术', '基因', '细胞', '免疫', '新药', '研究', '临床', '医疗'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
}
|
||||
];
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📊 实测数据（2025-11-18 抓取）
|
||||
|
||||
### 抓取成功统计
|
||||
|
||||
```
|
||||
总计: 64条真实最新新闻
|
||||
- 36氪: 41条 ✅ (医疗投融资、AI、科技)
|
||||
- 雷锋网: 21条 ✅ (AI医疗、机器人、技术)
|
||||
- 虎嗅网: 2条 ✅ (行业分析)
|
||||
```
|
||||
|
||||
### 示例新闻标题（真实）
|
||||
|
||||
#### 36氪最新：
|
||||
- "Databricks正洽谈新一轮融资，估值有望超1300亿美元"
|
||||
- "为了让你吃上现炒，老乡鸡们开始用机器人颠勺了"
|
||||
- "美银称AI有助提高员工生产率，带来更多收入"
|
||||
- "资管巨头操作现分歧，海外AI泡沫之辩白热化"
|
||||
|
||||
#### 雷锋网最新：
|
||||
- "中国 AI 正处于反超美国的前夜"
|
||||
- "蚂蚁灵光来了：30秒生应用 让复杂变简单"
|
||||
- "老板在南极潇洒，员工在公司加班！俞敏洪发全员信被员工吐槽"
|
||||
|
||||
---
|
||||
|
||||
## 🔍 为什么选择科技媒体？
|
||||
|
||||
### 优势
|
||||
|
||||
1. ✅ **更新及时** - 每日多次更新
|
||||
2. ✅ **覆盖全面** - 涵盖医疗科技、AI健康、机器人、投融资
|
||||
3. ✅ **质量高** - 深度报道、专业分析
|
||||
4. ✅ **相关性强** - 聚焦前沿技术和创新
|
||||
5. ✅ **RSS稳定** - 格式规范，易于解析
|
||||
|
||||
### 覆盖主题
|
||||
|
||||
- 🏥 医疗机器人（手术、康复、护理）
|
||||
- 🤖 AI医疗（诊断、影像、药物研发）
|
||||
- 👴 智慧养老（养老机器人、健康监测）
|
||||
- 💰 医疗投融资（创业公司、融资动态）
|
||||
- 🔬 生物技术（基因、细胞、新药）
|
||||
- 📊 行业趋势（市场分析、政策解读）
|
||||
|
||||
---
|
||||
|
||||
## 🚀 使用建议
|
||||
|
||||
### 立即可用配置
|
||||
|
||||
```bash
|
||||
# 1. 清理旧数据
|
||||
rm data/news.db
|
||||
|
||||
# 2. 使用优化配置运行爬虫
|
||||
npm run crawler
|
||||
|
||||
# 3. 导出数据
|
||||
npm run export
|
||||
|
||||
# 4. 验证结果
|
||||
sqlite3 data/news.db "SELECT COUNT(*), source FROM news GROUP BY source;"
|
||||
```
|
||||
|
||||
### 定期维护
|
||||
|
||||
```bash
|
||||
# 每30分钟自动抓取（推荐）
|
||||
./start-news-service.sh
|
||||
|
||||
# 或手动运行
|
||||
./verify-news.sh # 选择选项3
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## 📝 注意事项
|
||||
|
||||
### 关键词优化
|
||||
|
||||
当前关键词配置已优化，可以有效过滤医疗健康相关内容：
|
||||
|
||||
```javascript
|
||||
keywords: [
|
||||
'医疗', '健康', '机器人', 'AI', '智能', '诊断',
|
||||
'技术', '创新', '融资', '投资', '康养', '养老',
|
||||
'数字健康', '医药', '算法'
|
||||
]
|
||||
```
|
||||
|
||||
### RSS源扩展
|
||||
|
||||
如果需要添加新的RSS源，建议：
|
||||
|
||||
1. ✅ 先使用 `test-rss-sources.js` 测试可用性
|
||||
2. ✅ 检查内容时效性（不超过7天）
|
||||
3. ✅ 验证医疗相关度（≥20%）
|
||||
4. ✅ 确认RSS格式正确（XML有效）
|
||||
|
||||
---
|
||||
|
||||
## 🎉 总结
|
||||
|
||||
**最佳配置**: 仅使用 **36氪 + 雷锋网 + 虎嗅网**
|
||||
|
||||
- 简单可靠
|
||||
- 内容最新
|
||||
- 覆盖全面
|
||||
- 维护成本低
|
||||
|
||||
这三个源就能满足康养产业、医疗机器人、AI健康等所有主题需求！
|
||||
@@ -23,6 +23,101 @@ check_node_version() {
|
||||
echo -e "${GREEN}✅ Node.js 版本检查通过${NC}"
|
||||
}
|
||||
|
||||
# 检查并处理端口冲突
|
||||
check_and_handle_ports() {
|
||||
echo "🔍 检查端口占用情况..."
|
||||
|
||||
PORTS_TO_CHECK=(5173 3001)
|
||||
CONFLICTS_FOUND=false
|
||||
|
||||
for PORT in "${PORTS_TO_CHECK[@]}"; do
|
||||
PIDS=$(lsof -ti:$PORT 2>/dev/null)
|
||||
if [ ! -z "$PIDS" ]; then
|
||||
CONFLICTS_FOUND=true
|
||||
echo -e "${YELLOW}⚠️ 端口 $PORT 被占用${NC}"
|
||||
|
||||
# 显示占用进程信息
|
||||
for PID in $PIDS; do
|
||||
PROCESS_INFO=$(ps -p $PID -o comm= 2>/dev/null || echo "未知进程")
|
||||
echo " - PID: $PID, 进程: $PROCESS_INFO"
|
||||
done
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$CONFLICTS_FOUND" = true ]; then
|
||||
echo ""
|
||||
echo -e "${YELLOW}发现端口冲突。需要停止这些进程才能继续部署。${NC}"
|
||||
echo ""
|
||||
echo "选择处理方式:"
|
||||
echo "1) 自动停止冲突进程 (推荐)"
|
||||
echo "2) 手动处理后继续"
|
||||
echo "3) 退出部署"
|
||||
read -p "请输入选项 (1/2/3): " port_choice
|
||||
|
||||
case $port_choice in
|
||||
1)
|
||||
echo "🛑 停止冲突进程..."
|
||||
for PORT in "${PORTS_TO_CHECK[@]}"; do
|
||||
PIDS=$(lsof -ti:$PORT 2>/dev/null)
|
||||
if [ ! -z "$PIDS" ]; then
|
||||
for PID in $PIDS; do
|
||||
echo " 停止 PID $PID (端口 $PORT)..."
|
||||
kill -15 $PID 2>/dev/null || kill -9 $PID 2>/dev/null
|
||||
done
|
||||
fi
|
||||
done
|
||||
|
||||
# 等待进程完全停止
|
||||
sleep 2
|
||||
|
||||
# 再次检查
|
||||
STILL_OCCUPIED=false
|
||||
for PORT in "${PORTS_TO_CHECK[@]}"; do
|
||||
if lsof -ti:$PORT &>/dev/null; then
|
||||
STILL_OCCUPIED=true
|
||||
echo -e "${RED}❌ 端口 $PORT 仍被占用${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$STILL_OCCUPIED" = true ]; then
|
||||
echo -e "${RED}❌ 无法自动停止所有冲突进程，请手动处理${NC}"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
echo -e "${GREEN}✅ 端口冲突已解决${NC}"
|
||||
;;
|
||||
2)
|
||||
echo -e "${YELLOW}请手动停止冲突进程后按回车继续...${NC}"
|
||||
read -p ""
|
||||
|
||||
# 再次检查
|
||||
STILL_OCCUPIED=false
|
||||
for PORT in "${PORTS_TO_CHECK[@]}"; do
|
||||
if lsof -ti:$PORT &>/dev/null; then
|
||||
STILL_OCCUPIED=true
|
||||
echo -e "${RED}❌ 端口 $PORT 仍被占用${NC}"
|
||||
fi
|
||||
done
|
||||
|
||||
if [ "$STILL_OCCUPIED" = true ]; then
|
||||
echo -e "${RED}❌ 仍存在端口冲突，部署终止${NC}"
|
||||
exit 1
|
||||
fi
|
||||
;;
|
||||
3)
|
||||
echo -e "${YELLOW}部署已取消${NC}"
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo -e "${RED}❌ 无效选项，部署终止${NC}"
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
else
|
||||
echo -e "${GREEN}✅ 端口检查通过，无冲突${NC}"
|
||||
fi
|
||||
}
|
||||
|
||||
# 拉取最新代码
|
||||
pull_latest_code() {
|
||||
echo "📥 拉取最新代码..."
|
||||
|
||||
+2
-2
@@ -4,7 +4,7 @@ services:
|
||||
web:
|
||||
build: .
|
||||
ports:
|
||||
- "5173:3000" # Map external port 5173 to internal port 3000
|
||||
- "30080:3000" # Map external port 5173 to internal port 3000
|
||||
environment:
|
||||
- NODE_ENV=production
|
||||
restart: unless-stopped
|
||||
@@ -14,4 +14,4 @@ services:
|
||||
interval: 30s
|
||||
timeout: 10s
|
||||
retries: 3
|
||||
start_period: 40s
|
||||
start_period: 40s
|
||||
|
||||
@@ -0,0 +1,254 @@
|
||||
# Grafana 远程访问配置指南
|
||||
|
||||
## 问题诊断
|
||||
|
||||
当前状态：
|
||||
- ✅ Grafana 已部署在 `observability` namespace
|
||||
- ✅ Ingress 已配置（`monitor.omniverseai.net`）
|
||||
- ✅ DNS 解析正确（指向 `85.239.233.16`）
|
||||
- ❌ Ingress Controller Service 类型为 ClusterIP（只能集群内访问）
|
||||
|
||||
## 解决方案选择
|
||||
|
||||
### 方案对比
|
||||
|
||||
| 方案 | 优点 | 缺点 | 适用场景 |
|
||||
|------|------|------|----------|
|
||||
| LoadBalancer | 配置简单，原生支持 | 需要云环境或 MetalLB | 云服务商 K8s 或有 LB 支持 |
|
||||
| NodePort | 简单直接 | 使用非标准端口，需防火墙配置 | 本地或私有集群 |
|
||||
| 外部 Nginx | 灵活，可复用现有基础设施 | 配置复杂，多一层代理 | 已有外部代理服务器 |
|
||||
|
||||
## 方案 1：LoadBalancer（推荐）
|
||||
|
||||
### 前置条件检查
|
||||
|
||||
```bash
|
||||
# 检查是否支持 LoadBalancer
|
||||
kubectl get svc -A | grep LoadBalancer
|
||||
|
||||
# 如果有 MetalLB
|
||||
kubectl get pods -n metallb-system
|
||||
```
|
||||
|
||||
### 部署步骤
|
||||
|
||||
1. 应用 LoadBalancer 配置：
|
||||
```bash
|
||||
kubectl apply -f k8s/ingress-nginx-loadbalancer.yaml
|
||||
```
|
||||
|
||||
2. 等待 External IP 分配：
|
||||
```bash
|
||||
kubectl get svc ingress-nginx-controller -n infra -w
|
||||
```
|
||||
|
||||
3. 更新 DNS 记录：
|
||||
```bash
|
||||
# 将 monitor.omniverseai.net 的 A 记录指向分配的 External IP
|
||||
```
|
||||
|
||||
4. 验证访问：
|
||||
```bash
|
||||
curl -I https://monitor.omniverseai.net
|
||||
```
|
||||
|
||||
## 方案 2：NodePort
|
||||
|
||||
### 部署步骤
|
||||
|
||||
1. 应用 NodePort 配置：
|
||||
```bash
|
||||
kubectl apply -f k8s/ingress-nginx-nodeport.yaml
|
||||
```
|
||||
|
||||
2. 查看分配的端口：
|
||||
```bash
|
||||
kubectl get svc ingress-nginx-controller -n infra
|
||||
# 会显示类似：80:30080/TCP,443:30443/TCP
|
||||
```
|
||||
|
||||
3. 获取任意节点 IP：
|
||||
```bash
|
||||
kubectl get nodes -o wide
|
||||
```
|
||||
|
||||
4. 配置防火墙（如果有）：
|
||||
```bash
|
||||
# 允许 30080 和 30443 端口
|
||||
sudo ufw allow 30080/tcp
|
||||
sudo ufw allow 30443/tcp
|
||||
```
|
||||
|
||||
5. 访问方式：
|
||||
```bash
|
||||
# HTTP
|
||||
http://<NODE_IP>:30080
|
||||
|
||||
# HTTPS
|
||||
https://<NODE_IP>:30443
|
||||
```
|
||||
|
||||
6. （可选）配置外部 Nginx 转发到标准端口 - 见方案 3
|
||||
|
||||
## 方案 3：外部 Nginx 反向代理
|
||||
|
||||
### 前置条件
|
||||
|
||||
1. 首先应用 NodePort 配置（见方案 2 步骤 1-3）
|
||||
|
||||
2. 在外部 nginx 服务器上操作：
|
||||
|
||||
### 部署步骤
|
||||
|
||||
1. SSH 到外部服务器（85.239.233.16）：
|
||||
```bash
|
||||
ssh user@85.239.233.16
|
||||
```
|
||||
|
||||
2. 获取 k8s 节点 IP：
|
||||
```bash
|
||||
# 在 k8s 管理机上运行
|
||||
kubectl get nodes -o wide
|
||||
# 记录 INTERNAL-IP 或 EXTERNAL-IP
|
||||
```
|
||||
|
||||
3. 创建 Nginx 配置：
|
||||
```bash
|
||||
# 复制 k8s/external-nginx-config.conf 到服务器
|
||||
sudo nano /etc/nginx/sites-available/grafana-proxy.conf
|
||||
|
||||
# 替换配置文件中的 <NODE_IP> 为实际节点 IP
|
||||
# 例如：server 192.168.1.100:30080;
|
||||
```
|
||||
|
||||
4. 启用配置：
|
||||
```bash
|
||||
sudo ln -s /etc/nginx/sites-available/grafana-proxy.conf /etc/nginx/sites-enabled/
|
||||
```
|
||||
|
||||
5. 测试配置：
|
||||
```bash
|
||||
sudo nginx -t
|
||||
```
|
||||
|
||||
6. 重载 Nginx：
|
||||
```bash
|
||||
sudo systemctl reload nginx
|
||||
```
|
||||
|
||||
7. 配置 SSL（如果还没有）：
|
||||
```bash
|
||||
sudo certbot --nginx -d monitor.omniverseai.net
|
||||
```
|
||||
|
||||
## 快速测试
|
||||
|
||||
### 本地端口转发（临时测试）
|
||||
|
||||
如果需要临时访问 Grafana：
|
||||
|
||||
```bash
|
||||
# 转发到本地 8080 端口
|
||||
kubectl port-forward -n observability svc/monitoring-grafana 8080:80
|
||||
|
||||
# 然后访问 http://localhost:8080
|
||||
```
|
||||
|
||||
### 验证 Ingress 配置
|
||||
|
||||
```bash
|
||||
# 检查 Ingress 状态
|
||||
kubectl get ingress -n observability monitoring-grafana
|
||||
|
||||
# 查看 Ingress 详细信息
|
||||
kubectl describe ingress -n observability monitoring-grafana
|
||||
|
||||
# 检查证书状态
|
||||
kubectl get certificate -n observability monitor-tls
|
||||
```
|
||||
|
||||
## 故障排查
|
||||
|
||||
### 1. 无法访问 Grafana
|
||||
|
||||
```bash
|
||||
# 检查 Grafana Pod 状态
|
||||
kubectl get pods -n observability | grep grafana
|
||||
|
||||
# 查看 Pod 日志
|
||||
kubectl logs -n observability -l app.kubernetes.io/name=grafana
|
||||
|
||||
# 检查 Service
|
||||
kubectl get svc -n observability monitoring-grafana
|
||||
```
|
||||
|
||||
### 2. Ingress 不工作
|
||||
|
||||
```bash
|
||||
# 检查 Ingress Controller 日志
|
||||
kubectl logs -n infra -l app.kubernetes.io/name=ingress-nginx
|
||||
|
||||
# 查看 Ingress 事件
|
||||
kubectl describe ingress -n observability monitoring-grafana
|
||||
```
|
||||
|
||||
### 3. SSL 证书问题
|
||||
|
||||
```bash
|
||||
# 检查证书状态
|
||||
kubectl get certificate -n observability
|
||||
|
||||
# 查看 cert-manager 日志
|
||||
kubectl logs -n cert-manager -l app=cert-manager
|
||||
```
|
||||
|
||||
### 4. 外部访问测试
|
||||
|
||||
```bash
|
||||
# 从外部测试 DNS
|
||||
nslookup monitor.omniverseai.net
|
||||
|
||||
# 测试 HTTP/HTTPS
|
||||
curl -I http://monitor.omniverseai.net
|
||||
curl -I https://monitor.omniverseai.net
|
||||
|
||||
# 测试从集群内访问
|
||||
kubectl run test-pod --rm -it --image=curlimages/curl -- sh
|
||||
curl -I http://monitoring-grafana.observability.svc.cluster.local
|
||||
```
|
||||
|
||||
## 推荐配置
|
||||
|
||||
根据你的环境：
|
||||
|
||||
1. **如果在云环境（AWS/GCP/Azure）**：使用方案 1（LoadBalancer）
|
||||
2. **如果有 MetalLB**：使用方案 1（LoadBalancer）
|
||||
3. **如果是本地集群且有外部服务器**：使用方案 3（NodePort + 外部 Nginx）
|
||||
4. **如果是开发/测试环境**：使用方案 2（NodePort）直接访问
|
||||
|
||||
## 安全建议
|
||||
|
||||
1. **启用认证**：
|
||||
```bash
|
||||
# 检查 Grafana 是否配置了认证
|
||||
kubectl get secret -n observability monitoring-grafana -o jsonpath='{.data.admin-password}' | base64 -d
|
||||
```
|
||||
|
||||
2. **限制访问 IP**（在 Nginx 或防火墙层面）：
|
||||
```nginx
|
||||
# 在 nginx 配置中添加
|
||||
allow 192.168.1.0/24; # 允许的 IP 段
|
||||
deny all; # 拒绝其他
|
||||
```
|
||||
|
||||
3. **使用强密码**：
|
||||
```bash
|
||||
# 更改 Grafana 管理员密码
|
||||
kubectl exec -n observability -it <grafana-pod> -- grafana-cli admin reset-admin-password <new-password>
|
||||
```
|
||||
|
||||
4. **启用 HTTPS**（已配置 Let's Encrypt）
|
||||
|
||||
## 下一步
|
||||
|
||||
选择适合你环境的方案后，按照相应的步骤执行。如果遇到问题，参考故障排查部分。
|
||||
Executable
+152
@@ -0,0 +1,152 @@
|
||||
#!/bin/bash
|
||||
|
||||
# ============================================================================
|
||||
# Grafana Remote Access Diagnostic Script
|
||||
# 检查当前配置并推荐最佳解决方案
|
||||
# ============================================================================
|
||||
|
||||
set -e
|
||||
|
||||
echo "=========================================="
|
||||
echo "Grafana 远程访问诊断工具"
|
||||
echo "=========================================="
|
||||
echo ""
|
||||
|
||||
# Colors
|
||||
RED='\033[0;31m'
|
||||
GREEN='\033[0;32m'
|
||||
YELLOW='\033[1;33m'
|
||||
NC='\033[0m' # No Color
|
||||
|
||||
# 1. 检查 Grafana 部署状态
|
||||
echo "1. 检查 Grafana 部署状态..."
|
||||
if kubectl get pods -n observability | grep -q grafana; then
|
||||
echo -e "${GREEN}✓${NC} Grafana Pod 运行中"
|
||||
kubectl get pods -n observability | grep grafana
|
||||
else
|
||||
echo -e "${RED}✗${NC} Grafana Pod 未找到"
|
||||
exit 1
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 2. 检查 Grafana Service
|
||||
echo "2. 检查 Grafana Service..."
|
||||
kubectl get svc -n observability monitoring-grafana
|
||||
echo ""
|
||||
|
||||
# 3. 检查 Ingress 配置
|
||||
echo "3. 检查 Ingress 配置..."
|
||||
if kubectl get ingress -n observability monitoring-grafana &> /dev/null; then
|
||||
echo -e "${GREEN}✓${NC} Ingress 已配置"
|
||||
kubectl get ingress -n observability monitoring-grafana
|
||||
INGRESS_HOST=$(kubectl get ingress -n observability monitoring-grafana -o jsonpath='{.spec.rules[0].host}')
|
||||
echo "域名: $INGRESS_HOST"
|
||||
else
|
||||
echo -e "${RED}✗${NC} Ingress 未配置"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 4. 检查 Ingress Controller
|
||||
echo "4. 检查 Ingress Controller..."
|
||||
if kubectl get pods -n infra | grep -q ingress-nginx-controller; then
|
||||
echo -e "${GREEN}✓${NC} Ingress Controller 运行中"
|
||||
kubectl get pods -n infra | grep ingress-nginx-controller
|
||||
|
||||
# 检查 Service 类型
|
||||
SVC_TYPE=$(kubectl get svc -n infra ingress-nginx-controller -o jsonpath='{.spec.type}')
|
||||
echo "Service 类型: $SVC_TYPE"
|
||||
|
||||
case $SVC_TYPE in
|
||||
"ClusterIP")
|
||||
echo -e "${RED}✗${NC} 当前为 ClusterIP，无法从外部访问"
|
||||
;;
|
||||
"NodePort")
|
||||
echo -e "${YELLOW}!${NC} NodePort 模式"
|
||||
kubectl get svc -n infra ingress-nginx-controller -o wide
|
||||
;;
|
||||
"LoadBalancer")
|
||||
echo -e "${GREEN}✓${NC} LoadBalancer 模式"
|
||||
EXTERNAL_IP=$(kubectl get svc -n infra ingress-nginx-controller -o jsonpath='{.status.loadBalancer.ingress[0].ip}')
|
||||
echo "External IP: $EXTERNAL_IP"
|
||||
;;
|
||||
esac
|
||||
else
|
||||
echo -e "${RED}✗${NC} Ingress Controller 未找到"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 5. 检查 LoadBalancer 支持
|
||||
echo "5. 检查集群 LoadBalancer 支持..."
|
||||
if kubectl get pods -n metallb-system &> /dev/null 2>&1; then
|
||||
echo -e "${GREEN}✓${NC} MetalLB 已安装"
|
||||
kubectl get pods -n metallb-system
|
||||
elif kubectl get svc -A | grep -q "LoadBalancer.*pending"; then
|
||||
echo -e "${YELLOW}!${NC} 集群可能不支持 LoadBalancer"
|
||||
else
|
||||
# 检查是否在云环境
|
||||
NODES=$(kubectl get nodes -o jsonpath='{.items[*].spec.providerID}')
|
||||
if [[ $NODES == *"aws"* ]] || [[ $NODES == *"gce"* ]] || [[ $NODES == *"azure"* ]]; then
|
||||
echo -e "${GREEN}✓${NC} 检测到云环境，支持 LoadBalancer"
|
||||
else
|
||||
echo -e "${YELLOW}!${NC} 未检测到 LoadBalancer 支持（MetalLB 或云环境）"
|
||||
fi
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 6. 检查节点信息
|
||||
echo "6. 集群节点信息..."
|
||||
kubectl get nodes -o wide
|
||||
echo ""
|
||||
|
||||
# 7. DNS 检查
|
||||
echo "7. 检查 DNS 解析..."
|
||||
if [ ! -z "$INGRESS_HOST" ]; then
|
||||
DNS_IP=$(nslookup $INGRESS_HOST | grep -A1 "Name:" | grep "Address:" | tail -1 | awk '{print $2}')
|
||||
if [ ! -z "$DNS_IP" ]; then
|
||||
echo -e "${GREEN}✓${NC} DNS 解析成功"
|
||||
echo "$INGRESS_HOST -> $DNS_IP"
|
||||
else
|
||||
echo -e "${RED}✗${NC} DNS 解析失败"
|
||||
fi
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 8. 推荐方案
|
||||
echo "=========================================="
|
||||
echo "推荐方案"
|
||||
echo "=========================================="
|
||||
|
||||
if kubectl get pods -n metallb-system &> /dev/null 2>&1 || [[ $NODES == *"aws"* ]] || [[ $NODES == *"gce"* ]] || [[ $NODES == *"azure"* ]]; then
|
||||
echo -e "${GREEN}推荐方案 1: LoadBalancer${NC}"
|
||||
echo "你的集群支持 LoadBalancer，这是最简单的方案"
|
||||
echo ""
|
||||
echo "执行命令："
|
||||
echo " kubectl apply -f k8s/ingress-nginx-loadbalancer.yaml"
|
||||
echo ""
|
||||
elif [ "$SVC_TYPE" = "ClusterIP" ]; then
|
||||
echo -e "${YELLOW}推荐方案 2: NodePort${NC}"
|
||||
echo "你的集群不支持 LoadBalancer，建议使用 NodePort"
|
||||
echo ""
|
||||
echo "执行命令："
|
||||
echo " kubectl apply -f k8s/ingress-nginx-nodeport.yaml"
|
||||
echo ""
|
||||
echo "然后可以通过 <节点IP>:30080 (HTTP) 或 <节点IP>:30443 (HTTPS) 访问"
|
||||
echo ""
|
||||
echo -e "${YELLOW}推荐方案 3: 外部 Nginx 代理${NC}"
|
||||
echo "如果你有外部服务器（如 85.239.233.16），可以配置反向代理"
|
||||
echo "1. 先应用 NodePort 配置"
|
||||
echo "2. 配置外部 Nginx（参考 k8s/external-nginx-config.conf）"
|
||||
fi
|
||||
|
||||
echo ""
|
||||
echo "详细步骤请查看: k8s/GRAFANA_REMOTE_ACCESS_GUIDE.md"
|
||||
echo ""
|
||||
|
||||
# 9. 快速测试
|
||||
echo "=========================================="
|
||||
echo "快速测试（端口转发）"
|
||||
echo "=========================================="
|
||||
echo "如需立即测试 Grafana，运行："
|
||||
echo " kubectl port-forward -n observability svc/monitoring-grafana 8080:80"
|
||||
echo "然后访问: http://localhost:8080"
|
||||
echo ""
|
||||
@@ -0,0 +1,75 @@
|
||||
# ============================================================================
|
||||
# External Nginx Configuration for Grafana
|
||||
# Location: /etc/nginx/sites-available/grafana-proxy.conf on 85.239.233.16
|
||||
# ============================================================================
|
||||
|
||||
# First, you need to expose ingress-nginx using NodePort
|
||||
# Apply: kubectl apply -f k8s/ingress-nginx-nodeport.yaml
|
||||
|
||||
# Then configure this on your external nginx server (85.239.233.16)
|
||||
|
||||
upstream k8s_ingress {
|
||||
# Replace <NODE_IP> with your k8s node IP(s)
|
||||
# If you have multiple nodes, add them all for high availability
|
||||
server <NODE_IP>:30080;
|
||||
# server <NODE_IP_2>:30080;
|
||||
# server <NODE_IP_3>:30080;
|
||||
}
|
||||
|
||||
upstream k8s_ingress_ssl {
|
||||
server <NODE_IP>:30443;
|
||||
# server <NODE_IP_2>:30443;
|
||||
# server <NODE_IP_3>:30443;
|
||||
}
|
||||
|
||||
# HTTP Server - Redirect to HTTPS
|
||||
server {
|
||||
listen 80;
|
||||
server_name monitor.omniverseai.net;
|
||||
|
||||
# Redirect all HTTP to HTTPS
|
||||
return 301 https://$server_name$request_uri;
|
||||
}
|
||||
|
||||
# HTTPS Server
|
||||
server {
|
||||
listen 443 ssl http2;
|
||||
server_name monitor.omniverseai.net;
|
||||
|
||||
# SSL Certificate (Let's Encrypt or your certificate)
|
||||
ssl_certificate /etc/letsencrypt/live/monitor.omniverseai.net/fullchain.pem;
|
||||
ssl_certificate_key /etc/letsencrypt/live/monitor.omniverseai.net/privkey.pem;
|
||||
|
||||
# SSL Configuration
|
||||
ssl_protocols TLSv1.2 TLSv1.3;
|
||||
ssl_ciphers HIGH:!aNULL:!MD5;
|
||||
ssl_prefer_server_ciphers on;
|
||||
|
||||
# Logging
|
||||
access_log /var/log/nginx/grafana-access.log;
|
||||
error_log /var/log/nginx/grafana-error.log;
|
||||
|
||||
# Proxy to Kubernetes Ingress Controller
|
||||
location / {
|
||||
proxy_pass https://k8s_ingress_ssl;
|
||||
|
||||
# Preserve host header
|
||||
proxy_set_header Host $host;
|
||||
proxy_set_header X-Real-IP $remote_addr;
|
||||
proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for;
|
||||
proxy_set_header X-Forwarded-Proto $scheme;
|
||||
|
||||
# WebSocket support (important for Grafana live updates)
|
||||
proxy_http_version 1.1;
|
||||
proxy_set_header Upgrade $http_upgrade;
|
||||
proxy_set_header Connection "upgrade";
|
||||
|
||||
# Timeouts
|
||||
proxy_connect_timeout 60s;
|
||||
proxy_send_timeout 60s;
|
||||
proxy_read_timeout 60s;
|
||||
|
||||
# SSL verification (disable if using self-signed certs in k8s)
|
||||
proxy_ssl_verify off;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,41 @@
|
||||
# ============================================================================
|
||||
# Patch Ingress Nginx Controller to LoadBalancer
|
||||
# This allows external access to the ingress controller
|
||||
# ============================================================================
|
||||
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ingress-nginx-controller
|
||||
namespace: infra
|
||||
annotations:
|
||||
meta.helm.sh/release-name: ingress-nginx
|
||||
meta.helm.sh/release-namespace: infra
|
||||
labels:
|
||||
app.kubernetes.io/component: controller
|
||||
app.kubernetes.io/instance: ingress-nginx
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: ingress-nginx
|
||||
app.kubernetes.io/part-of: ingress-nginx
|
||||
app.kubernetes.io/version: 1.14.0
|
||||
helm.sh/chart: ingress-nginx-4.14.0
|
||||
spec:
|
||||
type: LoadBalancer
|
||||
# Optional: Specify external IP if you have a static IP
|
||||
# externalIPs:
|
||||
# - 85.239.233.16
|
||||
ports:
|
||||
- appProtocol: http
|
||||
name: http
|
||||
port: 80
|
||||
protocol: TCP
|
||||
targetPort: http
|
||||
- appProtocol: https
|
||||
name: https
|
||||
port: 443
|
||||
protocol: TCP
|
||||
targetPort: https
|
||||
selector:
|
||||
app.kubernetes.io/component: controller
|
||||
app.kubernetes.io/instance: ingress-nginx
|
||||
app.kubernetes.io/name: ingress-nginx
|
||||
@@ -0,0 +1,40 @@
|
||||
# ============================================================================
|
||||
# Patch Ingress Nginx Controller to NodePort
|
||||
# Access via <node-ip>:<nodePort>
|
||||
# ============================================================================
|
||||
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: ingress-nginx-controller
|
||||
namespace: infra
|
||||
annotations:
|
||||
meta.helm.sh/release-name: ingress-nginx
|
||||
meta.helm.sh/release-namespace: infra
|
||||
labels:
|
||||
app.kubernetes.io/component: controller
|
||||
app.kubernetes.io/instance: ingress-nginx
|
||||
app.kubernetes.io/managed-by: Helm
|
||||
app.kubernetes.io/name: ingress-nginx
|
||||
app.kubernetes.io/part-of: ingress-nginx
|
||||
app.kubernetes.io/version: 1.14.0
|
||||
helm.sh/chart: ingress-nginx-4.14.0
|
||||
spec:
|
||||
type: NodePort
|
||||
ports:
|
||||
- appProtocol: http
|
||||
name: http
|
||||
port: 80
|
||||
protocol: TCP
|
||||
targetPort: http
|
||||
nodePort: 30080 # You can change this to any available port in 30000-32767
|
||||
- appProtocol: https
|
||||
name: https
|
||||
port: 443
|
||||
protocol: TCP
|
||||
targetPort: https
|
||||
nodePort: 30443 # You can change this to any available port in 30000-32767
|
||||
selector:
|
||||
app.kubernetes.io/component: controller
|
||||
app.kubernetes.io/instance: ingress-nginx
|
||||
app.kubernetes.io/name: ingress-nginx
|
||||
@@ -0,0 +1,135 @@
|
||||
# ============================================================================
|
||||
# OmniVerse Website - Kubernetes Deployment
|
||||
# Namespace: website
|
||||
# Domain: www.omniverseai.net
|
||||
# ============================================================================
|
||||
|
||||
---
|
||||
# Namespace
|
||||
apiVersion: v1
|
||||
kind: Namespace
|
||||
metadata:
|
||||
name: website
|
||||
labels:
|
||||
name: website
|
||||
environment: production
|
||||
|
||||
---
|
||||
# Deployment
|
||||
apiVersion: apps/v1
|
||||
kind: Deployment
|
||||
metadata:
|
||||
name: website
|
||||
namespace: website
|
||||
labels:
|
||||
app: website
|
||||
environment: production
|
||||
spec:
|
||||
replicas: 2
|
||||
strategy:
|
||||
type: RollingUpdate
|
||||
rollingUpdate:
|
||||
maxSurge: 1
|
||||
maxUnavailable: 0
|
||||
selector:
|
||||
matchLabels:
|
||||
app: website
|
||||
template:
|
||||
metadata:
|
||||
labels:
|
||||
app: website
|
||||
environment: production
|
||||
spec:
|
||||
containers:
|
||||
- name: website
|
||||
image: omniverse-website:latest
|
||||
imagePullPolicy: IfNotPresent
|
||||
ports:
|
||||
- name: http
|
||||
containerPort: 3000
|
||||
protocol: TCP
|
||||
|
||||
livenessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: http
|
||||
initialDelaySeconds: 30
|
||||
periodSeconds: 10
|
||||
timeoutSeconds: 5
|
||||
failureThreshold: 3
|
||||
|
||||
readinessProbe:
|
||||
httpGet:
|
||||
path: /
|
||||
port: http
|
||||
initialDelaySeconds: 10
|
||||
periodSeconds: 5
|
||||
timeoutSeconds: 3
|
||||
failureThreshold: 3
|
||||
|
||||
resources:
|
||||
requests:
|
||||
cpu: 100m
|
||||
memory: 128Mi
|
||||
limits:
|
||||
cpu: 500m
|
||||
memory: 512Mi
|
||||
|
||||
env:
|
||||
- name: NODE_ENV
|
||||
value: "production"
|
||||
|
||||
---
|
||||
# Service
|
||||
apiVersion: v1
|
||||
kind: Service
|
||||
metadata:
|
||||
name: website
|
||||
namespace: website
|
||||
labels:
|
||||
app: website
|
||||
spec:
|
||||
type: ClusterIP
|
||||
selector:
|
||||
app: website
|
||||
ports:
|
||||
- name: http
|
||||
port: 80
|
||||
targetPort: 3000
|
||||
protocol: TCP
|
||||
|
||||
---
|
||||
# Ingress
|
||||
apiVersion: networking.k8s.io/v1
|
||||
kind: Ingress
|
||||
metadata:
|
||||
name: website
|
||||
namespace: website
|
||||
annotations:
|
||||
nginx.ingress.kubernetes.io/proxy-body-size: "50m"
|
||||
nginx.ingress.kubernetes.io/ssl-redirect: "false"
|
||||
# Cloudflare 已经处理 SSL,所以这里不需要 SSL 重定向
|
||||
spec:
|
||||
ingressClassName: nginx
|
||||
rules:
|
||||
- host: www.omniverseai.net
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: website
|
||||
port:
|
||||
number: 80
|
||||
|
||||
- host: omniverseai.net
|
||||
http:
|
||||
paths:
|
||||
- path: /
|
||||
pathType: Prefix
|
||||
backend:
|
||||
service:
|
||||
name: website
|
||||
port:
|
||||
number: 80
|
||||
@@ -25,4 +25,7 @@ JUHE_KEY=
|
||||
|
||||
# Logging
|
||||
LOG_LEVEL=info
|
||||
LOG_FILE=./logs/app.log
|
||||
LOG_FILE=./logs/app.log
|
||||
|
||||
# Export Settings
|
||||
AUTO_EXPORT_JSON=true
|
||||
Binary file not shown.
@@ -0,0 +1,132 @@
|
||||
const axios = require('axios');
|
||||
const Parser = require('rss-parser');
|
||||
|
||||
const parser = new Parser({
|
||||
timeout: 10000,
|
||||
maxRedirects: 5
|
||||
});
|
||||
|
||||
// 权威医疗健康RSS源候选列表
|
||||
const candidateSources = [
|
||||
// === 政府/官方媒体 ===
|
||||
{ name: '新华网健康', url: 'http://www.xinhuanet.com/health/news_health.xml', category: 'healthcare-policy' },
|
||||
{ name: '人民网健康', url: 'http://health.people.com.cn/rss/health.xml', category: 'healthcare-policy' },
|
||||
{ name: '央广网健康', url: 'http://health.cnr.cn/rss/health.xml', category: 'healthcare-policy' },
|
||||
|
||||
// === 综合科技媒体（医疗科技版块） ===
|
||||
{ name: '36氪', url: 'https://36kr.com/feed', category: 'investment' },
|
||||
{ name: '雷锋网', url: 'https://www.leiphone.com/feed', category: 'tech-innovation' },
|
||||
{ name: '虎嗅网', url: 'https://www.huxiu.com/rss/0.xml', category: 'industry-trends' },
|
||||
{ name: 'TechCrunch中国', url: 'https://techcrunch.cn/feed/', category: 'tech-innovation' },
|
||||
|
||||
// === 医疗专业媒体 ===
|
||||
{ name: '丁香园', url: 'https://www.dxy.cn/bbs/newweb/pc/rss/1', category: 'healthcare-policy' },
|
||||
{ name: '医学界', url: 'https://www.yxj.org.cn/rss.xml', category: 'healthcare-policy' },
|
||||
{ name: '健康界', url: 'https://www.cn-healthcare.com/rss', category: 'healthcare-policy' },
|
||||
|
||||
// === 门户网站健康频道 ===
|
||||
{ name: '新浪医药', url: 'http://rss.sina.com.cn/news/marquee/ddt.xml', category: 'healthcare-policy' },
|
||||
{ name: '网易健康', url: 'http://news.163.com/special/00011K6L/rss_health.xml', category: 'healthcare-policy' },
|
||||
{ name: '腾讯健康', url: 'https://health.qq.com/rss_health.xml', category: 'healthcare-policy' },
|
||||
|
||||
// === 科学/技术媒体 ===
|
||||
{ name: '果壳网', url: 'https://www.guokr.com/rss/', category: 'tech-innovation' },
|
||||
{ name: '科学网', url: 'http://news.sciencenet.cn/xml/news.aspx', category: 'tech-innovation' },
|
||||
|
||||
// === 国际医疗健康（可选，需要翻墙） ===
|
||||
{ name: 'MedicalNewsToday', url: 'https://www.medicalnewstoday.com/rss', category: 'tech-innovation' },
|
||||
{ name: 'HealthLine', url: 'https://www.healthline.com/feeds/rss', category: 'healthcare-policy' },
|
||||
];
|
||||
|
||||
async function testAndParseRSS(source) {
|
||||
try {
|
||||
console.log(`\n测试: ${source.name}...`);
|
||||
const feed = await parser.parseURL(source.url);
|
||||
|
||||
if (!feed.items || feed.items.length === 0) {
|
||||
console.log(` ⚠️ ${source.name}: RSS格式正确但无内容`);
|
||||
return null;
|
||||
}
|
||||
|
||||
// 检查是否有医疗相关关键词
|
||||
const medicalKeywords = ['医疗', '健康', '医院', '医生', '疾病', '药品', '治疗', '康复', '养老', '护理',
|
||||
'health', 'medical', 'hospital', 'disease', 'treatment', 'care', 'robot', 'AI'];
|
||||
|
||||
const firstFewItems = feed.items.slice(0, 10);
|
||||
const relevantCount = firstFewItems.filter(item => {
|
||||
const text = (item.title + ' ' + (item.contentSnippet || '')).toLowerCase();
|
||||
return medicalKeywords.some(kw => text.includes(kw.toLowerCase()));
|
||||
}).length;
|
||||
|
||||
const relevanceScore = (relevantCount / firstFewItems.length * 100).toFixed(0);
|
||||
|
||||
console.log(` ✅ ${source.name}:`);
|
||||
console.log(` • 文章数: ${feed.items.length}`);
|
||||
console.log(` • 最新: ${feed.items[0]?.title?.substring(0, 50)}...`);
|
||||
console.log(` • 医疗相关度: ${relevanceScore}% (${relevantCount}/${firstFewItems.length})`);
|
||||
|
||||
return {
|
||||
...source,
|
||||
success: true,
|
||||
itemCount: feed.items.length,
|
||||
latestTitle: feed.items[0]?.title,
|
||||
relevanceScore: parseInt(relevanceScore),
|
||||
sampleTitles: feed.items.slice(0, 3).map(i => i.title)
|
||||
};
|
||||
} catch (error) {
|
||||
const msg = error.code === 'ENOTFOUND' ? '域名无法解析' :
|
||||
error.code === 'ETIMEDOUT' ? '连接超时' :
|
||||
error.message.includes('Invalid XML') ? 'XML格式错误' :
|
||||
error.message;
|
||||
console.log(` ❌ ${source.name}: ${msg}`);
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
|
||||
console.log('🔍 搜索可用的真实医疗健康RSS源');
|
||||
console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n');
|
||||
|
||||
const results = [];
|
||||
|
||||
for (const source of candidateSources) {
|
||||
const result = await testAndParseRSS(source);
|
||||
if (result) {
|
||||
results.push(result);
|
||||
}
|
||||
await new Promise(resolve => setTimeout(resolve, 2000));
|
||||
}
|
||||
|
||||
console.log('\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
|
||||
console.log(`\n✅ 找到 ${results.length} 个可用RSS源\n`);
|
||||
|
||||
// 按相关度排序
|
||||
results.sort((a, b) => b.relevanceScore - a.relevanceScore);
|
||||
|
||||
console.log('推荐使用的RSS源 (按医疗相关度排序):\n');
|
||||
results.forEach((r, i) => {
|
||||
console.log(`${i + 1}. ${r.name.padEnd(20)} [相关度: ${r.relevanceScore}%] [分类: ${r.category}]`);
|
||||
console.log(` URL: ${r.url}`);
|
||||
console.log(` 示例: ${r.sampleTitles[0]?.substring(0, 60)}...`);
|
||||
console.log();
|
||||
});
|
||||
|
||||
// 生成配置代码
|
||||
console.log('\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
|
||||
console.log('建议的爬虫配置代码:\n');
|
||||
console.log('this.sources = [');
|
||||
results.forEach(r => {
|
||||
console.log(` {`);
|
||||
console.log(` name: '${r.name}',`);
|
||||
console.log(` url: '${r.url}',`);
|
||||
console.log(` category: '${r.category}',`);
|
||||
console.log(` keywords: ['医疗', '健康', '机器人', 'AI', '智能', '养老', '康复'],`);
|
||||
console.log(` priority: 'high',`);
|
||||
console.log(` region: '全国'`);
|
||||
console.log(` },`);
|
||||
});
|
||||
console.log('];');
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
@@ -7,6 +7,8 @@
|
||||
"start": "node src/server.js",
|
||||
"dev": "nodemon src/server.js",
|
||||
"crawler": "node src/crawler.js",
|
||||
"export": "node src/utils/export-to-json.js",
|
||||
"crawl-and-export": "node src/crawler.js && node src/utils/export-to-json.js",
|
||||
"test": "jest"
|
||||
},
|
||||
"keywords": [
|
||||
|
||||
@@ -13,150 +13,50 @@ class RSSCrawler {
|
||||
this.db = database;
|
||||
this.logger = logger;
|
||||
|
||||
// RSS源配置
|
||||
// RSS源配置 - 仅包含真实可用且内容最新的权威源（已验证 2025-11-18）
|
||||
this.sources = [
|
||||
{
|
||||
name: '动脉网',
|
||||
url: 'https://vcbeat.top/rss',
|
||||
category: 'medical-robotics',
|
||||
keywords: ['医疗', '机器人', '健康', 'AI', '智能', '诊断', '手术', '康复', '辅助'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
// === 核心科技媒体（强烈推荐，已验证可用且内容最新） ===
|
||||
{
|
||||
name: '36氪',
|
||||
url: 'https://36kr.com/feed',
|
||||
category: 'investment',
|
||||
keywords: ['医疗', '健康', '机器人', '融资', '投资', '康养', '养老', '数字健康', '医药'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '亿欧网',
|
||||
url: 'https://www.iyiou.com/rss',
|
||||
category: 'industry-trends',
|
||||
keywords: ['医疗', '健康', '智慧', '数字化', '机器人', '康养', '产业', '趋势'],
|
||||
priority: 'medium',
|
||||
keywords: ['医疗', '健康', '机器人', '融资', '投资', '康养', '养老', '数字健康', '医药', 'AI', '智能'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '雷锋网',
|
||||
url: 'https://www.leiphone.com/feed',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['AI', '医疗', '机器人', '智能', '算法', '诊断', '技术', '创新'],
|
||||
keywords: ['AI', '医疗', '机器人', '智能', '算法', '诊断', '技术', '创新', '健康', '养老', '康复'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '机器之心',
|
||||
url: 'https://www.jiqizhixin.com/rss',
|
||||
name: '虎嗅网',
|
||||
url: 'https://www.huxiu.com/rss/0.xml',
|
||||
category: 'industry-trends',
|
||||
keywords: ['医疗', '健康', '智慧', '数字化', '机器人', '康养', '产业', '趋势', 'AI', '养老'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
|
||||
// === 补充科技媒体（可选，扩展覆盖） ===
|
||||
{
|
||||
name: '钛媒体',
|
||||
url: 'https://www.tmtpost.com/rss.xml',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['医疗', '机器人', 'AI', '深度学习', '算法', '人工智能'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '健康界',
|
||||
url: 'https://www.cn-healthcare.com/rss',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['政策', '医保', '卫健委', '医疗', '健康', '监管', '法规', '标准'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '丁香园',
|
||||
url: 'https://www.dxy.cn/rss',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['医疗', '政策', '监管', '医药', '临床', '医院', '医生'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '医学界',
|
||||
url: 'https://www.yxj.org.cn/rss',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['医学', '政策', '临床', '研究', '医疗', '创新', '标准'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
// 新增医疗健康RSS源
|
||||
{
|
||||
name: '医谷',
|
||||
url: 'https://www.yigoonet.com/feed',
|
||||
category: 'medical-robotics',
|
||||
keywords: ['医疗器械', '设备', '医疗', '健康', '诊断', '治疗', '康复', '机器人'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '器械之家',
|
||||
url: 'https://www.qxzjia.com/feed',
|
||||
category: 'medical-robotics',
|
||||
keywords: ['医疗器械', '设备', '机器人', '手术', '诊断', '康复', '智能'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '医药网',
|
||||
url: 'https://www.pharmnet.com.cn/rss',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['医药', '药品', '监管', '政策', '新药', '审批', '临床', '研发'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: 'CPhI制药在线',
|
||||
url: 'https://www.cphi.cn/rss',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['制药', '生物技术', '新药', '研发', '创新', '生物医药'],
|
||||
keywords: ['医疗', '健康', '科技', '创新', 'AI', '智能', '数字', '机器人', '养老'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
|
||||
// === 垂直医疗科技（备选，需验证内容时效性） ===
|
||||
{
|
||||
name: '生物谷',
|
||||
url: 'https://www.bioon.com/rss',
|
||||
url: 'http://www.bioon.com/rss/nrss.xml',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['生物技术', '基因', '细胞', '免疫', '新药', '研究', '临床'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '康复医学网',
|
||||
url: 'https://www.kfyx.cn/rss',
|
||||
category: 'medical-robotics',
|
||||
keywords: ['康复', '理疗', '康复机器人', '辅助', '训练', '治疗'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '养老网',
|
||||
url: 'https://www.yanglao.com.cn/rss',
|
||||
category: 'smart-elderly-care',
|
||||
keywords: ['养老', '老年', '护理', '智慧养老', '康养', '老龄化'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '医信邦',
|
||||
url: 'https://www.yixinbang.com/rss',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['数字医疗', '互联网医疗', '远程', 'AI', '大数据', '智慧医院'],
|
||||
priority: 'high',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '医疗器械创新网',
|
||||
url: 'https://www.md-innovation.com/rss',
|
||||
category: 'medical-robotics',
|
||||
keywords: ['创新', '器械', '设备', '技术', '研发', '突破'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '中国卫生人才网',
|
||||
url: 'https://www.21wecan.com/rss',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['卫生', '人才', '政策', '培训', '医疗', '管理'],
|
||||
keywords: ['生物技术', '基因', '细胞', '免疫', '新药', '研究', '临床', '医疗', '健康'],
|
||||
priority: 'low',
|
||||
region: '全国'
|
||||
}
|
||||
@@ -165,10 +65,18 @@ class RSSCrawler {
|
||||
// 备用RSS源（如果主要源不可用）
|
||||
this.fallbackSources = [
|
||||
{
|
||||
name: '科技日报',
|
||||
url: 'http://digitalpaper.stdaily.com/http_www.kjrb.com/kjrb/rss/rss.xml',
|
||||
name: '人民网科技',
|
||||
url: 'http://scitech.people.com.cn/rss/it.xml',
|
||||
category: 'tech-innovation',
|
||||
keywords: ['医疗', '健康', '机器人', '科技'],
|
||||
keywords: ['医疗', '健康', '机器人', '科技', 'AI'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
},
|
||||
{
|
||||
name: '中国新闻网健康',
|
||||
url: 'http://www.chinanews.com/rss/health.xml',
|
||||
category: 'healthcare-policy',
|
||||
keywords: ['医疗', '健康', '政策', '医院'],
|
||||
priority: 'medium',
|
||||
region: '全国'
|
||||
}
|
||||
|
||||
@@ -268,11 +268,24 @@ class NewsServer {
|
||||
async runCrawl() {
|
||||
try {
|
||||
const results = await this.crawler.crawlAll();
|
||||
|
||||
|
||||
// 清除相关缓存
|
||||
this.cache.flushAll();
|
||||
|
||||
|
||||
this.logger.info(`Crawl completed: ${results.length} items processed`);
|
||||
|
||||
// 自动导出为 JSON 文件供前端使用
|
||||
if (process.env.AUTO_EXPORT_JSON === 'true') {
|
||||
try {
|
||||
const NewsExporter = require('./utils/export-to-json');
|
||||
const exporter = new NewsExporter();
|
||||
await exporter.export();
|
||||
this.logger.info('News data exported to JSON successfully');
|
||||
} catch (exportError) {
|
||||
this.logger.error('Failed to export news to JSON:', exportError);
|
||||
}
|
||||
}
|
||||
|
||||
return results;
|
||||
} catch (error) {
|
||||
this.logger.error('Crawl failed:', error);
|
||||
|
||||
@@ -0,0 +1,83 @@
|
||||
#!/usr/bin/env node
|
||||
|
||||
const fs = require('fs');
|
||||
const path = require('path');
|
||||
const Database = require('../models/database');
|
||||
require('dotenv').config();
|
||||
|
||||
/**
|
||||
* 将数据库中的新闻数据导出为 JSON 文件
|
||||
* 供前端静态站点使用
|
||||
*/
|
||||
class NewsExporter {
|
||||
constructor() {
|
||||
this.db = new Database(process.env.DATABASE_PATH || './data/news.db');
|
||||
this.outputPath = path.resolve(__dirname, '../../../public/data/news.json');
|
||||
}
|
||||
|
||||
async export() {
|
||||
try {
|
||||
console.log('🚀 Starting news export...');
|
||||
|
||||
// 获取所有新闻数据
|
||||
const allNews = await this.db.getNews({
|
||||
page: 1,
|
||||
limit: 1000, // 导出最近 1000 条
|
||||
sortBy: 'published_at',
|
||||
sortOrder: 'DESC'
|
||||
});
|
||||
|
||||
console.log(`📊 Found ${allNews.length} news items`);
|
||||
|
||||
// 确保输出目录存在
|
||||
const outputDir = path.dirname(this.outputPath);
|
||||
if (!fs.existsSync(outputDir)) {
|
||||
fs.mkdirSync(outputDir, { recursive: true });
|
||||
console.log(`📁 Created directory: ${outputDir}`);
|
||||
}
|
||||
|
||||
// 写入 JSON 文件
|
||||
fs.writeFileSync(
|
||||
this.outputPath,
|
||||
JSON.stringify(allNews, null, 2),
|
||||
'utf-8'
|
||||
);
|
||||
|
||||
console.log(`✅ Successfully exported ${allNews.length} news items to ${this.outputPath}`);
|
||||
|
||||
// 显示分类统计
|
||||
const categoryCounts = {};
|
||||
allNews.forEach(news => {
|
||||
categoryCounts[news.category] = (categoryCounts[news.category] || 0) + 1;
|
||||
});
|
||||
|
||||
console.log('\n📈 Category distribution:');
|
||||
Object.entries(categoryCounts).forEach(([category, count]) => {
|
||||
console.log(` ${category}: ${count}`);
|
||||
});
|
||||
|
||||
return allNews.length;
|
||||
} catch (error) {
|
||||
console.error('❌ Export failed:', error);
|
||||
throw error;
|
||||
} finally {
|
||||
this.db.close();
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// 如果直接运行此脚本
|
||||
if (require.main === module) {
|
||||
const exporter = new NewsExporter();
|
||||
exporter.export()
|
||||
.then(count => {
|
||||
console.log(`\n🎉 Export completed: ${count} items`);
|
||||
process.exit(0);
|
||||
})
|
||||
.catch(error => {
|
||||
console.error('Export failed:', error);
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
|
||||
module.exports = NewsExporter;
|
||||
@@ -0,0 +1,81 @@
|
||||
const axios = require('axios');
|
||||
|
||||
// 测试RSS源可访问性
|
||||
const testSources = [
|
||||
// 当前配置的源
|
||||
{ name: '动脉网', url: 'https://vcbeat.top/rss' },
|
||||
{ name: '36氪', url: 'https://36kr.com/feed' },
|
||||
{ name: '亿欧网', url: 'https://www.iyiou.com/rss' },
|
||||
{ name: '雷锋网', url: 'https://www.leiphone.com/feed' },
|
||||
{ name: '机器之心', url: 'https://www.jiqizhixin.com/rss' },
|
||||
|
||||
// 真实可用的RSS源候选
|
||||
{ name: '丁香园', url: 'https://www.dxy.cn/bbs/newweb/pc/rss/1' },
|
||||
{ name: '新浪医药', url: 'http://rss.sina.com.cn/news/marquee/ddt.xml' },
|
||||
{ name: '人民网健康', url: 'http://health.people.com.cn/rss/health.xml' },
|
||||
{ name: '新华网健康', url: 'http://www.xinhuanet.com/health/news_health.xml' },
|
||||
{ name: '腾讯健康', url: 'https://health.qq.com/rss_health.xml' },
|
||||
{ name: '搜狐健康', url: 'http://health.sohu.com/healthfeed.xml' },
|
||||
];
|
||||
|
||||
async function testRSSSource(source) {
|
||||
try {
|
||||
const response = await axios.get(source.url, {
|
||||
timeout: 10000,
|
||||
maxRedirects: 5,
|
||||
headers: {
|
||||
'User-Agent': 'Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36'
|
||||
}
|
||||
});
|
||||
|
||||
if (response.status === 200) {
|
||||
const contentType = response.headers['content-type'] || '';
|
||||
const dataStr = String(response.data);
|
||||
const isXML = contentType.includes('xml') || dataStr.includes('<?xml');
|
||||
const hasRSS = dataStr.includes('<rss') || dataStr.includes('<feed');
|
||||
|
||||
if (isXML || hasRSS) {
|
||||
console.log(`✅ ${source.name.padEnd(15)}: 可访问 (RSS/XML格式)`);
|
||||
return true;
|
||||
} else {
|
||||
console.log(`⚠️ ${source.name.padEnd(15)}: 可访问但格式可能不正确`);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
} catch (error) {
|
||||
const msg = error.code === 'ENOTFOUND' ? '域名无法解析' :
|
||||
error.code === 'ETIMEDOUT' ? '连接超时' :
|
||||
error.response?.status ? `HTTP ${error.response.status}` :
|
||||
error.message;
|
||||
console.log(`❌ ${source.name.padEnd(15)}: ${msg}`);
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
async function main() {
|
||||
console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
|
||||
console.log('🔍 测试RSS源可访问性');
|
||||
console.log('━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━\n');
|
||||
|
||||
let successCount = 0;
|
||||
const results = [];
|
||||
|
||||
for (const source of testSources) {
|
||||
const success = await testRSSSource(source);
|
||||
results.push({ ...source, success });
|
||||
if (success) successCount++;
|
||||
await new Promise(resolve => setTimeout(resolve, 1500));
|
||||
}
|
||||
|
||||
console.log('\n━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━');
|
||||
console.log(`测试完成: ${successCount}/${testSources.length} 个源可访问\n`);
|
||||
|
||||
if (successCount > 0) {
|
||||
console.log('✅ 可用的RSS源:');
|
||||
results.filter(r => r.success).forEach(r => {
|
||||
console.log(` • ${r.name}: ${r.url}`);
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
main().catch(console.error);
|
||||
Executable
+146
@@ -0,0 +1,146 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 新闻爬虫验证和运行脚本
|
||||
# 用于验证RSS源、测试爬虫、导出数据
|
||||
|
||||
set -e # 遇到错误立即退出
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " 🔍 OmniVerse 新闻爬虫系统验证"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
# 检查 Node.js
|
||||
echo "1️⃣ 检查 Node.js 环境..."
|
||||
if ! command -v node &> /dev/null; then
|
||||
echo "❌ Node.js 未安装"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ Node.js 版本: $(node --version)"
|
||||
echo ""
|
||||
|
||||
# 检查 news-crawler-service 目录
|
||||
echo "2️⃣ 检查爬虫服务目录..."
|
||||
if [ ! -d "news-crawler-service" ]; then
|
||||
echo "❌ news-crawler-service 目录不存在"
|
||||
exit 1
|
||||
fi
|
||||
echo "✅ 爬虫服务目录存在"
|
||||
echo ""
|
||||
|
||||
# 检查依赖
|
||||
echo "3️⃣ 检查 npm 依赖..."
|
||||
cd news-crawler-service
|
||||
if [ ! -d "node_modules" ]; then
|
||||
echo "📦 安装依赖..."
|
||||
npm install
|
||||
else
|
||||
echo "✅ 依赖已安装"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 检查环境配置
|
||||
echo "4️⃣ 检查环境配置..."
|
||||
if [ ! -f ".env" ]; then
|
||||
echo "⚠️ .env 文件不存在，从 .env.example 复制..."
|
||||
cp .env.example .env
|
||||
echo "⚙️ 请编辑 .env 文件配置必要的参数"
|
||||
fi
|
||||
echo "✅ 环境配置文件存在"
|
||||
echo ""
|
||||
|
||||
# 检查数据库
|
||||
echo "5️⃣ 检查数据库..."
|
||||
if [ -f "data/news.db" ]; then
|
||||
echo "✅ 数据库文件存在"
|
||||
# 显示数据库统计
|
||||
echo "📊 数据库统计:"
|
||||
sqlite3 data/news.db "SELECT COUNT(*) as total FROM news;" 2>/dev/null || echo " 无法读取数据库统计"
|
||||
else
|
||||
echo "ℹ️ 数据库文件不存在（首次运行时会自动创建）"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 检查 Ollama 服务（用于 AI 摘要）
|
||||
echo "6️⃣ 检查 Ollama AI 服务..."
|
||||
if curl -s http://localhost:11434/api/tags > /dev/null 2>&1; then
|
||||
echo "✅ Ollama 服务运行中"
|
||||
else
|
||||
echo "⚠️ Ollama 服务未运行"
|
||||
echo " 提示: AI 摘要功能需要 Ollama，但不影响基本爬取功能"
|
||||
fi
|
||||
echo ""
|
||||
|
||||
# 菜单选项
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " 📋 请选择操作："
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " 1) 运行爬虫 (抓取最新新闻)"
|
||||
echo " 2) 导出数据为 JSON"
|
||||
echo " 3) 爬取并导出 (推荐)"
|
||||
echo " 4) 查看数据库统计"
|
||||
echo " 5) 启动爬虫服务"
|
||||
echo " 6) 退出"
|
||||
echo ""
|
||||
read -p "请输入选项 [1-6]: " choice
|
||||
|
||||
case $choice in
|
||||
1)
|
||||
echo ""
|
||||
echo "🚀 开始运行爬虫..."
|
||||
npm run crawler
|
||||
;;
|
||||
2)
|
||||
echo ""
|
||||
echo "📤 导出数据为 JSON..."
|
||||
npm run export
|
||||
;;
|
||||
3)
|
||||
echo ""
|
||||
echo "🔄 爬取并导出..."
|
||||
npm run crawl-and-export
|
||||
echo ""
|
||||
echo "✅ 完成! 数据已导出到 public/data/news.json"
|
||||
;;
|
||||
4)
|
||||
echo ""
|
||||
echo "📊 数据库统计信息:"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
if [ -f "data/news.db" ]; then
|
||||
echo "总新闻数:"
|
||||
sqlite3 data/news.db "SELECT COUNT(*) FROM news;"
|
||||
echo ""
|
||||
echo "各分类新闻数:"
|
||||
sqlite3 data/news.db "SELECT category, COUNT(*) as count FROM news GROUP BY category ORDER BY count DESC;"
|
||||
echo ""
|
||||
echo "各来源新闻数:"
|
||||
sqlite3 data/news.db "SELECT source, COUNT(*) as count FROM news GROUP BY source ORDER BY count DESC LIMIT 10;"
|
||||
else
|
||||
echo "❌ 数据库文件不存在"
|
||||
fi
|
||||
;;
|
||||
5)
|
||||
echo ""
|
||||
echo "🌐 启动爬虫服务..."
|
||||
echo " 访问 http://localhost:3001 查看 API"
|
||||
echo " 按 Ctrl+C 停止服务"
|
||||
echo ""
|
||||
npm start
|
||||
;;
|
||||
6)
|
||||
echo "👋 再见!"
|
||||
cd ..
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "❌ 无效选项"
|
||||
cd ..
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
|
||||
cd ..
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " ✅ 操作完成"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
+911
-345
File diff suppressed because one or more lines are too long
Executable
+71
@@ -0,0 +1,71 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 启动新闻爬虫自动定时服务
|
||||
# 使用 PM2 进程管理器保证服务稳定运行
|
||||
|
||||
set -e
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " 🚀 启动 OmniVerse 新闻爬虫自动定时服务"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
# 检查 PM2 是否安装
|
||||
if ! command -v pm2 &> /dev/null; then
|
||||
echo "📦 PM2 未安装，正在安装..."
|
||||
npm install -g pm2
|
||||
echo "✅ PM2 安装完成"
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# 进入爬虫服务目录
|
||||
cd news-crawler-service
|
||||
|
||||
# 检查依赖
|
||||
if [ ! -d "node_modules" ]; then
|
||||
echo "📦 安装依赖..."
|
||||
npm install
|
||||
echo ""
|
||||
fi
|
||||
|
||||
# 停止已存在的服务
|
||||
echo "🔍 检查现有服务..."
|
||||
pm2 delete omniverse-news 2>/dev/null || true
|
||||
echo ""
|
||||
|
||||
# 启动服务
|
||||
echo "🚀 启动新闻爬虫服务..."
|
||||
pm2 start src/server.js --name omniverse-news --time
|
||||
|
||||
echo ""
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " ✅ 服务启动成功！"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
echo "📊 服务信息:"
|
||||
echo " • 名称: omniverse-news"
|
||||
echo " • 端口: 3001"
|
||||
echo " • API: http://localhost:3001"
|
||||
echo " • 定时爬取: 每 30 分钟"
|
||||
echo " • 自动导出: 已启用"
|
||||
echo " • 日志位置: logs/app.log"
|
||||
echo ""
|
||||
echo "📋 常用命令:"
|
||||
echo " pm2 status # 查看服务状态"
|
||||
echo " pm2 logs omniverse-news # 查看实时日志"
|
||||
echo " pm2 restart omniverse-news # 重启服务"
|
||||
echo " pm2 stop omniverse-news # 停止服务"
|
||||
echo " pm2 delete omniverse-news # 删除服务"
|
||||
echo ""
|
||||
echo "🔄 首次运行:"
|
||||
echo " 服务将在启动后 5 秒开始首次爬取"
|
||||
echo " 之后每 30 分钟自动爬取一次"
|
||||
echo " 每次爬取完成后自动导出 JSON 到 public/data/news.json"
|
||||
echo ""
|
||||
echo "🌐 查看服务:"
|
||||
pm2 status
|
||||
|
||||
echo ""
|
||||
echo "📝 查看实时日志:"
|
||||
echo " pm2 logs omniverse-news"
|
||||
echo ""
|
||||
Executable
+23
@@ -0,0 +1,23 @@
|
||||
#!/bin/bash
|
||||
|
||||
# 停止新闻爬虫服务
|
||||
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo " 🛑 停止 OmniVerse 新闻爬虫服务"
|
||||
echo "━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━━"
|
||||
echo ""
|
||||
|
||||
if ! command -v pm2 &> /dev/null; then
|
||||
echo "❌ PM2 未安装，无法停止服务"
|
||||
exit 1
|
||||
fi
|
||||
|
||||
pm2 stop omniverse-news 2>/dev/null && echo "✅ 服务已停止" || echo "ℹ️ 服务未运行"
|
||||
pm2 status
|
||||
|
||||
echo ""
|
||||
echo "💡 提示:"
|
||||
echo " • 使用 'pm2 start omniverse-news' 重新启动"
|
||||
echo " • 使用 'pm2 delete omniverse-news' 完全删除服务"
|
||||
echo " • 使用 './start-news-service.sh' 重新配置并启动"
|
||||
echo ""
|
||||
Reference in New Issue
Block a user