Bulk Export of Prediction Market Data: A Practical Guide

A bulk export of prediction market data means pulling full trade, order book and market history from venues like Polymarket, Kalshi and Limitless into files or a warehouse in one controlled job, instead of paging through live endpoints by hand. The hard part is not the download. It is getting identical markets lined up across venues, keeping the pull repeatable, and not losing rows when pagination or retention windows shift under you. This guide covers how to plan the export, which formats fit which job, how to normalize and validate the result, and how to keep it reproducible.

Assymetrixdata.assymetrix.comUnify Prediction Market DataAssymetrix brings Polymarket, Kalshi, and Limitless data together through one API, with historical coverage spanning nearly one billion rows of trading activity.Explore the Data API

TL;DR

• Decide the grain first (markets, trades, quotes, order book snapshots, wallet activity). The grain drives file size, format and cost more than anything else.

• Use columnar files (Parquet) for analysis and CSV only for hand-off to spreadsheets. Partition by venue and date.

• Venue APIs paginate and apply their own retention rules, so check the official docs for each venue and write resumable, idempotent jobs.

• Normalize to canonical market IDs before you merge venues, or cross-venue joins will silently drop or duplicate rows.

• Version every export with a manifest (query window, schema version, row counts, checksums) so a backtest can be rerun exactly.

Table of Contents

• What “bulk export” means for prediction markets

• Step 1: Choose the grain of the export

• Step 2: Pick a format that matches the job

• Step 3: Build the export job so it can be rerun

• Step 4: Normalize before you merge venues

• Step 5: Validate the export

• Step 6: Version the export

• DIY exports versus a unified source

• Our perspective

• Using the export: three common workflows

• Where our Data API fits

• FAQ

What “bulk export” means for prediction markets

Most results for this phrase describe generic spreadsheet exports. Prediction markets are different in three ways that shape the work.

The data is event-driven and large. A single active market can produce thousands of trades a day. Across three venues and several years, you are dealing with hundreds of millions of rows, not thousands. Our own historical layer spans roughly 1.5 terabytes and nearly one billion rows of trading activity, which is the scale where ad hoc scraping stops being practical.

Each venue models a market differently. Kalshi lists contracts under events and series with its own tickers. Polymarket markets are tied to on-chain conditions and token identifiers, and Limitless has its own market structure. The same real-world question can appear under three different identifiers with three different price conventions.

Retention and endpoints change. Venues split live and archived data, adjust response fields and revise limits. An export script that worked last quarter can quietly return a shorter universe this quarter. Read each venue’s official documentation, such as the Kalshi API docs and the Polymarket developer docs, before you hardcode anything about windows, fields or limits.

Step 1: Choose the grain of the export

Write down what one row represents before you write any code. These are the common grains, from smallest to largest.

If you only need to test whether market prices were calibrated against outcomes, candles plus resolution data are enough. If you want to study slippage or build a fill model, you need trades and quotes. Choosing trades when candles would do multiplies storage and processing time by orders of magnitude.

For how price data connects to forecasting quality, see our guide to prediction market accuracy, and for what trade-level data reveals, prediction market microstructure.

Step 2: Pick a format that matches the job

Parquet for analysis

Parquet stores columns separately and compresses well, so scanning one column such as price across hundreds of millions of rows is fast. It also preserves types, which matters when you have timestamps, integer contract counts and decimal prices. Partition directories by venue and by date (for example venue=kalshi/date=2026-03-14/) so queries touch only the days they need.

CSV for hand-off

CSV is the right choice when the recipient lives in a spreadsheet or a legacy tool. Keep it to a bounded slice: one venue, one category, one date range. Spreadsheets have row limits, and a full trade history will not fit. Use ISO 8601 timestamps in UTC, a dot as the decimal separator and a header row with stable column names.

JSON Lines for raw capture

If you want to keep the original venue payloads untouched, store them as newline-delimited JSON, compressed. This gives you a raw layer you can reprocess if your normalization logic changes, without hitting the venue again.

Object storage as the staging layer

Most teams land exports in object storage and query from there. We cover that pattern in detail in S3 prediction market data: sources, schemas and pipelines.

Step 3: Build the export job so it can be rerun

A bulk job that cannot resume is a job that will eventually waste a day of work. These habits prevent most failures.

1. Page with a cursor, not an offset, when the venue offers one. Offsets drift as new rows arrive during the pull. Cursors anchor you to a position. Check each venue’s docs for the pagination style it uses.

2. Pull in time windows. Split the history into day or week chunks and write each chunk to its own file. A failed chunk is retried alone.

3. Make writes idempotent. Name each output file from its venue, grain and window so rerunning overwrites the same file rather than appending duplicates.

4. Respect rate limits and back off. Honor the limits in each venue’s docs, use exponential backoff on errors and cap concurrency per venue.

5. Record the live and archive boundary. Where a venue serves recent and older data through different endpoints, find the boundary programmatically on every run instead of hardcoding a date. Missing history often returns a clean empty response, not an error, so row-count checks matter.

6. Log counts per chunk. Store expected versus received rows. A sudden drop for one day is a signal, not noise.

A minimal Python skeleton

import pathlib, json, time

def export_window(fetch_page, venue, start, end, out_dir):
    path = pathlib.Path(out_dir) / f"venue={venue}" / f"{start}_{end}.jsonl"
    if path.exists():
        return path  # idempotent: skip finished chunks
    path.parent.mkdir(parents=True, exist_ok=True)
    tmp = path.with_suffix(".tmp")
    cursor, rows = None, 0
    with open(tmp, "w") as f:
        while True:
            page = fetch_page(start, end, cursor)
            for r in page["items"]:
                f.write(json.dumps(r) + "
")
                rows += 1
            cursor = page.get("next_cursor")
            if not cursor:
                break
            time.sleep(0.2)
    tmp.rename(path)
    return path

import pathlib, json, time

def export_window(fetch_page, venue, start, end, out_dir):
    path = pathlib.Path(out_dir) / f"venue={venue}" / f"{start}_{end}.jsonl"
    if path.exists():
        return path  # idempotent: skip finished chunks
    path.parent.mkdir(parents=True, exist_ok=True)
    tmp = path.with_suffix(".tmp")
    cursor, rows = None, 0
    with open(tmp, "w") as f:
        while True:
            page = fetch_page(start, end, cursor)
            for r in page["items"]:
                f.write(json.dumps(r) + "
")
                rows += 1
            cursor = page.get("next_cursor")
            if not cursor:
                break
            time.sleep(0.2)
    tmp.rename(path)
    return path

The temporary file and rename step means a crash never leaves a half-written chunk that looks complete. For a fuller walkthrough with a client library, see our Python prediction market data guide.

Step 4: Normalize before you merge venues

Raw exports from three venues are not one dataset. Merging them without normalization is the most common source of wrong results.

Canonical market IDs

Assign each real-world question a canonical ID and map every venue’s identifier to it. Without this, a cross-venue price gap analysis compares unrelated contracts, or counts the same market twice. Matching is partly mechanical (resolution date, underlying event, outcome wording) and partly judgment, so store the mapping as data with a reviewed flag rather than burying it in code. Our guide to canonical IDs for cross-venue data walks through the approach.

Price and size conventions

Standardize everything to one convention: price as a probability between 0 and 1, size in contracts, timestamps in UTC with millisecond precision where available. Field names and units differ between live and archived responses on some venues, so map them explicitly and test with a few known trades.

Outcome and resolution fields

Export resolution status and the final outcome as their own table, joined by canonical ID. Calibration and skill studies need to know which markets resolved, which were voided and when. Keep unresolved markets in a separate partition so they are never mistaken for losses.

Wallet identity

For on-chain venues, addresses are public, which makes wallet-level work possible. Keep addresses as lowercase strings and store them in a dedicated column. If you plan Smart Money or skill scoring work, the Polymarket on-chain analysis guide covers what to extract.

Step 5: Validate the export

Run these checks before anyone builds on the files.

• Row counts per day match the sum of what the pages returned, and no day inside the window is empty without a reason (such as a market holiday for a sports category).

• Uniqueness: a trade ID, or a venue plus timestamp plus sequence key, appears once.

• Range checks: prices sit between 0 and 1, sizes are positive, timestamps fall inside the window.

• Referential integrity: every trade maps to a known market, and every market in scope has a canonical ID.

• Spot checks: pick five random trades and compare them against the venue’s own interface or a second source.

Step 6: Version the export

A backtest result is only meaningful if you can rebuild its inputs. Write a manifest file next to every export containing the query window, the grains included, the schema version, the code commit, row counts per partition and file checksums. When a venue changes a field, you bump the schema version and keep the old export intact.

We describe this workflow, using three datasets as the example, in one reproducible workflow for prediction market datasets. For research that depends on a deep history, our note on reproducible backtests with one billion rows covers the access side.

DIY exports versus a unified source

Building your own exporter is a sensible choice for one venue and one grain. It gets expensive as the scope grows. The costs are rarely in the first download. They show up in maintenance: three schemas, three pagination styles, three retention policies, a mapping table to keep reviewed and a backfill that has to be rerun whenever a venue revises its API.

Our perspective

Teams usually underestimate the export problem because the first download works. The trouble starts when the second venue arrives and the same event carries two identifiers, two price conventions and two retention rules. At that point, most of the engineering time goes into plumbing rather than research.

We built Assymetrix as a venue-neutral layer so the join, the normalization and the history are handled once. Our Data API is designed to give developers, quants, bots and AI agents one integration across Polymarket, Kalshi and Limitless, backed by roughly 1.5 TB of historical data and nearly one billion rows of trading activity. Our view is that an export should be a view of a consistent dataset, not a reconstruction effort.

If you do build your own pipeline, keep the raw layer, keep the ID mapping as data and version everything. Those three habits matter more than any tool choice.

Dean

Using the export: three common workflows

Calibration and accuracy studies

Export daily or hourly candles and resolutions for a category, bucket the prices into probability bands and compare each band’s average price with its resolution rate. The data is small enough for a laptop and answers whether a category is well calibrated.

Execution and microstructure research

Export trades and quotes for a handful of liquid markets over a few months. Compute spreads, depth and the price impact of trades by size. This needs the large grains, so partition by date and process one partition at a time.

Smart Money and skill scoring

Export wallet-level trades plus resolutions, compute realized outcomes per address and rank by risk-adjusted results with a minimum number of resolved markets. This is where canonical IDs and clean resolution data matter most, because a mislabeled outcome flips a wallet’s score.

Where our Data API fits

If you want the multi-venue history without building the exporter, our Data API provides unified access to real-time and historical data across Polymarket, Kalshi and Limitless, with bulk export features for the grains described above.

For the feed itself, including streaming and historical access, read the real-time and historical API guide. Teams feeding agents can start with our guide to feeding data into an AI trading agent.

FAQ

What is the best format for a large prediction market export?

Parquet, partitioned by venue and date, is the best default for analysis because it compresses well and keeps column types. Use CSV only for small slices going to spreadsheet users. Keep a compressed raw JSON Lines layer if you may need to reprocess.

Can I export Polymarket and Kalshi data into one table?

Yes, but only after normalizing identifiers, price units and timestamps. Map each venue’s market identifier to a canonical ID, convert prices to a 0 to 1 probability and store everything in UTC. Skipping the ID mapping is the most common cause of duplicated or missing rows.

How far back can I export history?

It depends on the venue and on how it splits recent and archived data, so check each venue’s official documentation and test the boundary on every run. Our own historical layer spans roughly 1.5 TB and nearly one billion rows across Polymarket, Kalshi and Limitless.

How do I make an export reproducible?

Write a manifest with the query window, grains, schema version, code commit, row counts and checksums, and never overwrite an old export when the schema changes. A backtest can then be rerun against exactly the same inputs.