Signal Pipeline

The signal pipeline ingests news and social media content from six sources, classifies and deduplicates articles, links them to relevant markets, and broadcasts signals to connected clients via WebSocket.

Architecture

The pipeline lives in src/modules/monitor/ingestion/ and runs on the signal-ingestion BullMQ queue, managed by the SignalIngestionScheduler.

┌─────────────┐
│   GDELT     │──┐
│   RSS       │  │
│   NewsData  │  │     ┌──────────────┐     ┌──────────────┐
│   CryptoNews│──┼──▶  │  Classifier  │──▶  │  Dedup       │
│   X/Twitter │  │     │  Service      │     │  Service      │
│   Telegram  │──┘     └──────────────┘     └──────┬───────┘
                                                    │
                                                    ▼
                                           ┌──────────────┐
                                           │ Market Linker │
                                           └──────┬───────┘
                                                    │
                                                    ▼
                                           ┌──────────────┐
                                           │  Geo Service  │
                                           └──────┬───────┘
                                                    │
                                                    ▼
                                           ┌──────────────┐
                                           │  WebSocket   │
                                           │  Broadcast   │
                                           │ monitor:signals│
                                           └──────────────┘

News Connectors

Each connector fetches content from its source and returns normalized article objects:

Pipeline Stages

1. Classification

The SignalClassifierService categorizes each article into one of the signal categories: politics, military, financials, crypto, tech, energy, or macro. It also assigns a severity level (critical, high, or low) based on the content's potential market impact.

2. Deduplication

The SignalDedupService identifies and filters duplicate articles across sources. The same breaking news story may appear in RSS, GDELT, and NewsData within minutes -- deduplication ensures only one signal reaches the database and WebSocket subscribers.

3. Market Linking

The SignalMarketLinkerService analyzes article content and matches it to relevant markets in the database. A news article about a presidential candidate, for example, would be linked to election prediction markets. Linked signals include marketId and relatedMarketIds fields.

4. Geo-Tagging

The SignalGeoService extracts geographic information from articles and assigns coordinates (lat, lng), country name, ISO country code, and country flag emoji. This data powers the 3D globe visualization on the monitor page.

5. Broadcast

Processed signals are saved as MonitorSignal records and broadcast to all clients subscribed to the monitor:signals WebSocket channel.

Signal Data Model

Each signal stored in the MonitorSignal table includes:

Separate Pipelines

The news signal pipeline is distinct from the signal-detector queue, which handles market-data-based signal detection (whale moves, smart money patterns). The news pipeline focuses exclusively on external content sources, while the detector queue analyzes on-chain and exchange trading patterns.