Market Ingestion

The ingestion pipeline continuously fetches market data from two prediction market exchanges, normalizes it into a unified schema, and stores it in PostgreSQL. The pipeline runs on BullMQ repeatable jobs managed by the MarketSyncScheduler.

Exchange Connectors

Polymarket

Polymarket data comes through two APIs:

• Gamma API -- Market listings, metadata, categories, and event groupings
• CLOB API -- Real-time prices, order book depth, and trade history. Also provides historical price candles (fidelity=60 for hourly, fidelity=1440 for daily)

The connector fetches the top 30 events by volume and all markets within those events.

Kalshi

Kalshi data comes through the Trade API v2:

• Market listings, prices, and event metadata
• Daily price candles (period_interval=1440) for historical data
• Order book snapshots

The connector also fetches the top 30 events by volume, but total event count is hard-capped.

Sync Schedules

The market-sync BullMQ queue runs three types of repeatable jobs:

Price-Sync Memory Optimizations

The price-sync job processes ~4,500 markets every 60 seconds. Two critical optimizations prevent the Node process from growing past 5 GB:

1. Delta-only snapshots -- MarketPriceSnapshot rows are only inserted when |price_change| > 0.0001, avoiding thousands of redundant writes per cycle
2. Deferred price-change queries -- The expensive updatePriceChanges() query (computing 5m/1h/6h/24h changes across all markets) runs only during full-sync (every 5 minutes), not on every price-sync cycle

Normalization

The NormalizerService converts exchange-specific data into the unified Market model:

Polymarket Raw Data ──┐
                      ├──▶ NormalizerService ──▶ Prisma Market Record
Kalshi Raw Data ──────┘

Key normalizations:

• Slug generation -- Consistent URL-safe slugs from market titles
• Category mapping -- Exchange-specific categories mapped to Vezta categories
• Price format -- Both exchanges normalize to yesPrice and noPrice as Decimal(10, 6)
• Volume -- Standardized to USD with Decimal(18, 2) precision
• Timestamps -- startedAt uses the exchange's creation time (Polymarket createdAt, Kalshi open_time); createdAt is when Vezta ingested the market

Price History Backfill

A separate price-history-backfill job (every 60 seconds) populates historical price data for charts:

The backfill only inserts data points older than the earliest existing snapshot to avoid duplicates. Markets are marked priceHistoryBackfilled = true once backfill completes.

Chart Data

The getChartData endpoint returns OHLC data aggregated via SQL date_bin():