3 Prediction Market Datasets, One Reproducible Workflow

Start with the named unified releases, not raw API polling: Polymarket-v1 on Hugging Face, the OpenMarket unified export, and Kalshi-derived unified datasets like the Winnie the Pooh collection give you a reproducible starting point. The reliable workflow pairs a tagged, deduped release (or a canonical-data API plus a snapshot manifest) with checksums and a recorded revision hash, never an untracked live pull as your sole source of truth. Expect Parquet files hosted on Hugging Face or S3, and check for a CITATION.md before you build anything on top of them.

TL;DR:

• The Polymarket v1 archive covers November 21, 2022, through April 28, 2026, with 1.2 billion trades across 1.3 million markets and settlement verified aggressor direction.

• OpenMarket unified releases remove close to 21% of raw records; use snapshots only to reconstruct order book states or audit recorder uptime.

• Pin the exact Hugging Face revision or GitHub tag, and preserve file checksums, a snapshot manifest, and citation details with your analysis.

• Trade prints show executed prices, not available order book depth, so they cannot alone support reliable liquidity or slippage assumptions in a backtest.

Assymetrixdata.assymetrix.comUnify Prediction Market DataAssymetrix brings Polymarket, Kalshi, and Limitless data together through one API for research and trading tools.Explore the Data API

Table of Contents

• A catalog of the major public prediction market dataset releases

• Release types and how to read a version tag before you trust it

• Pulling a specific dataset version from Hugging Face, S3, and GitHub

• Deduplication, canonical IDs, and schema normalization across venues

• Keeping a reproducible manifest alongside every analysis

• Verifying provenance claims before you cite a dataset

• Assymetrix Data API as a complement to published releases

• Author checklist for choosing and versioning a dataset

• Get normalized, versioned prediction market data through our API

• FAQ

• Sources

A catalog of the major public prediction market dataset releases

Three releases anchor most reproducible work in this space right now, and each serves a different research need.

Polymarket-v1 is a frozen trade archive covering Polymarket’s entire v1 contract lifecycle, from November 21, 2022 through April 28, 2026. It holds 1.2 billion trade records across 1.3 million markets, with $61 billion in nominal volume, and the paper points readers to a Hugging Face dataset release for direct access. What sets it apart from most trade archives is ground-truth aggressor direction, derived from on-chain settlement-layer OrderFilled events rather than inferred through heuristics. That distinction matters for anyone computing order flow imbalance or VPIN, since heuristic-classified direction and settlement-verified direction produce measurably different microstructure statistics.

OpenMarket takes a different shape. The project also carries archival notes marking certain releases as frozen, with a recommended source tag (v0.5.2) for anyone citing it in published work.

Kalshi-derived unified datasets, such as the Winnie the Pooh collection on Hugging Face, follow a third pattern: public S3 bulk exports converted into a single normalized Parquet corpus. That particular release documents 14,118,806 markets spanning January 1, 2023 through February 10, 2025, structured as a time-series belief history rather than a raw trade tape.

Beyond these three, community tooling repositories are worth tracking for new releases and adaptable pipelines:

• GitHub dataset repositories tagged with venue names (Polymarket, Kalshi) often precede a formal Hugging Face publication by months.

• Hugging Face’s dataset search surfaces smaller, venue-specific exports that complement the three primary releases above.

• Collector and backtest frameworks such as prediction-market-analysis provide ingestion code you can point at new releases as they appear.

Checking these sources before you start a project saves you from rebuilding a pipeline that already exists in a citable, versioned form.

Release types and how to read a version tag before you trust it

Public prediction market data arrives in three distinct shapes, and conflating them is the most common mistake in this space.

1. Full snapshot exports are point-in-time recorder archives: every API poll or websocket capture written to disk as-is, with no deduplication. They exist because raw capture is cheap and preserves every observed state, which matters if you need to reconstruct exact order-book conditions at a specific timestamp.

2. Unified or deduped releases collapse overlapping snapshots into a single clean timeline. OpenMarket’s unified export is the clearest public example: it exists specifically because the full export contains heavy redundancy that distorts naive row counts and biases backtests toward overrepresented periods.

3. Feature or ML-ready exports go a step further, pre-computing derived columns (returns, implied probability deltas, rolling volume) on top of a unified base, meant to be loaded directly into a training pipeline.

Release tags usually encode which type you’re looking at. A convention like v0.4.3-unified tells you the suffix identifies the processing stage, not just a version bump. Before using any tagged release, open its accompanying merge_quality_report.json or dedupe report if one is published: it should state input row count, output row count, and the percentage of duplicates removed, which lets you verify the release matches its own documentation rather than taking the tag on faith.

OpenMarket’s dedupe process removed close to 21% of records when building its unified release, collapsing 916 million raw rows into 727 million unified rows. That ratio is a useful benchmark: if you build your own deduplication pipeline from raw snapshots and land far outside that range, something in your overlap detection likely needs review.

The rule of thumb that holds across most research tasks: default to the unified release unless your specific question requires full-snapshot recovery, such as reconstructing a disputed resolution event or auditing recorder uptime; for guidance on how to test these strategies, see our retail trader’s guide to backtesting trading strategies. Backtesting and machine learning pipelines built on full, undeduplicated exports will overweight any period with denser polling, which skews everything downstream without an obvious symptom in the output.

Pulling a specific dataset version from Hugging Face, S3, and GitHub

Reproducibility depends on fetching the exact bytes a paper or report cites, not whatever happens to be the latest commit.

On Hugging Face, every dataset has a revision hash tied to each commit to its repository. Pin that hash explicitly when you load data, rather than defaulting to the main branch, and record it alongside your results. The dataset card itself usually states the recommended citation and the revision that corresponds to a published paper’s reported statistics, which is worth checking against your own load before running anything.

For S3-hosted bulk exports, such as the Kalshi-derived pipeline behind the Winnie the Pooh dataset, look for a snapshot manifest listing each file’s path, size, and checksum. Download against that manifest rather than listing the bucket yourself, since bucket contents can grow between your first and second visit. Incremental exports typically append new partition folders by date, so a manifest comparison also tells you whether you’re missing a partition before you start processing.

GitHub releases attach versioned artifacts to tags, which is different from cloning the default branch. OpenMarket’s release notes, for instance, point to a specific source tag for its archived, frozen state. A few practical defaults:

• Download the tagged release artifact when you need an exact, citable version; clone the repository only when you need to inspect pipeline code or contribute a fix.

• Verify any published checksum against your downloaded file before loading it into an analysis environment.

• Record the tag name itself in your project’s methods section or README, not just “the OpenMarket dataset.”

Pro Tip: Keep a local versions.json file in every research repo that maps each dataset you depend on to its exact revision hash or release tag, so a six-month-old project remains reproducible without you needing to remember which commit you used.

Venue APIs still have a role, but it’s a narrower one than most new projects assume. Use a live API (Polymarket’s or Kalshi’s) for incremental sync, keeping an existing dataset current between formal releases. For any reproducible historical analysis meant to be checked by someone else, a published bulk release beats a fresh API pull every time, since the release is fixed and citable while a live API response reflects whatever state the venue happens to be in at query time.

Deduplication, canonical IDs, and schema normalization across venues

Cross-venue research runs into a structural problem before any modeling question: the same market activity often gets captured in multiple overlapping recorder snapshots, and each venue names things differently.

Overlap between consecutive snapshot captures is common and sizable. Practitioner notes on the Polymarket-v1 release describe raw snapshots routinely producing around 20% overlapping rows between consecutive captures, and OpenMarket’s own unified build confirms a comparable figure in practice, with its dedupe process removing close to 21% of input rows. Any pipeline that ingests raw snapshots without a dedupe step will inherit that overlap directly into downstream statistics.

A workable canonical ID strategy rests on a few concrete practices:

• Join on a stable asset_id or market_id rather than a composite key built from title strings, which drift in formatting across venues and even across snapshots of the same venue.

• Freeze a metadata snapshot (market title, resolution criteria, open and close timestamps) alongside each data release, since venue-side metadata can be edited after a market opens.

• Keep a separate provenance column tracking which snapshot or API pull each row originated from, so a later audit can trace any suspicious row back to its source.

Schema normalization across venues requires agreeing on a minimum trade-tape structure before merging anything: timestamp (ideally in a single timezone, typically UTC), market identifier, side, price, size, and a settlement or resolution flag where applicable. Time alignment deserves particular care, since venues timestamp events at different points in their matching pipeline, and a naive join on wall-clock time can misorder trades that occurred within the same second.

A roughly 21% duplicate rate between snapshot captures appears in both the OpenMarket unified build and independent practitioner notes on raw Polymarket exports, which makes it a reasonable default assumption when sizing your own dedupe pipeline.

Before trusting a merged dataset for backtesting, run a short verification pass: compare row counts before and after deduplication against any published merge quality report, recompute aggregate statistics like daily market counts or total volume and check them against the source documentation, and spot-check a provenance checksum on a sample of files to confirm nothing was corrupted in transit. These checks take minutes and catch the overwhelming majority of silent data corruption issues before they reach a model.

Keeping a reproducible manifest alongside every analysis

A dataset version is only useful if the next person working with it, including a future version of yourself, can reconstruct exactly what you used.

1. Store a snapshot_manifest.json alongside your analysis code, listing every file path, size, and checksum for the data you loaded.

2. Save the merge_quality_report.json or equivalent dedupe report from whichever unified release you’re building on, not just a link to where you found it.

3. Record the exact release tag or Hugging Face revision hash in a CITATION.md file in your project repository, following the citation format the dataset’s own documentation recommends.

4. Package a small sample Parquet slice (a few thousand rows) with your published code, so a reviewer can run a quick confirmatory check without downloading the full dataset.

Lightweight tooling supports this without much overhead: an object-storage manifest file that lists checksums per partition, an incremental snapshot index that tracks which dates have been pulled, and a packaged sample-parquet file sized for a quick smoke test rather than a full reproduction. None of this requires dedicated infrastructure, just discipline about what gets committed alongside the analysis code itself.

Pro Tip: Treat your dataset manifest as part of your code review, not an afterthought: a pull request that changes which dataset revision an analysis depends on should be as visible as a change to the model itself.

When you publish results, surface the version information directly in your methods section rather than a footnote: name the release, its tag or revision hash, and the dedupe statistics it carries. A paper that states “we used the OpenMarket v0.4.3-unified release (727M rows, Hugging Face revision abc123)” gives a reviewer everything needed to reproduce the input data, while “we used public Polymarket data” gives them nothing to act on.

Verifying provenance claims before you cite a dataset

Citing a dataset correctly is a smaller task than verifying what it actually claims to contain, and the second step is the one researchers skip most often.

Start with the dataset’s own CITATION.md or equivalent documentation and embed both the dataset revision and a file checksum directly in your methods section, not just a dataset name and access date. A few concrete validation steps go a long way:

• When a release claims something specific, like Polymarket-v1’s ground-truth aggressor direction, look for the join report or settlement-layer verification file the publishers reference rather than accepting the claim from the abstract alone.

• Re-run a small subset of the publisher’s own reported statistics (row counts, date ranges, market counts) against your downloaded copy before building anything on top of it.

• Publish your own validation script and a small sample slice as supplementary material, so a reviewer can reproduce your verification step, not just your final result.

One caution worth carrying through any analysis: venue-provided price histories and trade prints represent realized trades, not order-book depth. A price chart built from trade prints tells you what happened at the time of execution, not what size could have been filled at that price, which matters if your research touches liquidity or slippage assumptions.

Assymetrix Data API as a complement to published releases

Published dataset releases solve the reproducibility problem for a fixed point in time, but most research and trading workflows also need current data that stays aligned with the same canonical structure. The Assymetrix Data API aggregates real-time and historical data across Polymarket, Kalshi, and Limitless into a single schema, built on an archive of approximately 1.5 terabytes of historical data spanning nearly one billion rows of trading activity.

Three use cases come up repeatedly for teams working from a named public release. The first is incremental sync: once you’ve loaded a tagged release like OpenMarket’s unified export, a canonical API lets you extend that timeline forward without re-polling raw venue endpoints yourself or reintroducing the overlap problem that unified releases were built to solve. The second is canonical ID alignment: because the API normalizes identifiers across venues at ingestion, joining a Polymarket market to its Kalshi counterpart for cross-venue arbitrage research doesn’t require rebuilding a matching layer from scratch. The third is layering in analytics that published static datasets don’t include, such as Smart Money wallet tracking and trader skill scoring, which depend on a continuously updated data pipeline rather than a frozen snapshot.

For developers starting from scratch, the Python quickstart walks through authentication and a first data pull, while the developer API guide covers REST and WebSocket access patterns for both backtesting pipelines and live agents. Teams building on top of bulk historical backfill, rather than a static release, can start from the historical data access guide, which documents canonical ID structure and backfill mechanics in more depth.

Author checklist for choosing and versioning a dataset

A short checklist catches most of the mistakes that surface months after a project ships.

• Confirm the exact release tag and manifest before writing a line of analysis code, not after you notice numbers don’t match a published report.

• Run a small validation backtest on a sample slice first, checking your pipeline’s row counts and summary statistics against the dataset’s own documentation.

• Preserve the CITATION.md, checksums, and manifest files in your project repository from day one, since recreating them retroactively is far harder than keeping them from the start.

• Publish a sample Parquet slice with your analysis code so a reviewer can verify your pipeline without a full download.

Two pitfalls account for most reproducibility failures in this space: data drift between a cited release and whatever a live API now returns, and incomplete resolution events in snapshots captured before a market finished settling. The mitigation for both is the same discipline repeated throughout this guide: pin your version, record your manifest, and verify before you build.

— Dean

Get normalized, versioned prediction market data through our API

We built the Assymetrix Data API to give researchers and developers one canonical schema across Polymarket, Kalshi, and Limitless, backed by an archive of roughly 1.5 terabytes of historical data covering nearly one billion rows of trading activity. Canonical IDs mean you join markets across venues without building your own matching layer, and REST and WebSocket access cover both incremental live sync and bulk historical backfill from a single integration.

If your current workflow starts from a public release and needs a current, continuously normalized extension, our Data API developer guide walks through authentication, canonical ID structure, and backfill mechanics, and the Python quickstart gets a first data pull running in a short setup. Start there to see how canonical, versioned access fits into a reproducible pipeline you’re already building.

FAQ

What is the difference between a full snapshot and a unified dataset release?

A full snapshot export is a raw, point-in-time recorder archive that preserves every observed state without removing duplicates. A unified release, like OpenMarket’s deduped export, collapses overlapping captures into a single clean timeline and is the better default for backtesting or machine learning.

How much duplication should I expect in raw prediction market snapshots?

Overlap between consecutive raw snapshot captures commonly runs close to 20%, based on practitioner notes tied to the Polymarket-v1 release.

What does ground-truth aggressor direction mean in the Polymarket-v1 dataset?

Ground-truth aggressor direction in the Polymarket-v1 database is derived directly from on-chain settlement-layer OrderFilled events, rather than inferred through a classification heuristic. This distinction affects microstructure metrics like order flow imbalance, which can differ materially between ground-truth and heuristic-classified data.

How do I cite the exact version of a dataset I used in my research?

Record the dataset’s release tag or exact Hugging Face revision hash, along with any accompanying manifest or dedupe report, and follow the format in the dataset’s own CITATION.md file where one is published. Embedding this detail directly in your methods section, rather than a general dataset name, is what allows a reviewer to reproduce your input data.

Should I use a live venue API or a published dataset release for backtesting?

Published bulk releases are the better choice for reproducible historical backtests, since they represent a fixed, citable version that won’t change between your run and a reviewer’s. Live venue APIs are better suited to incremental sync, keeping an already-versioned dataset current between formal releases.

Sources

• Polymarket-v1 Database (arXiv:2606.04217)

• OpenMarket (GitHub)

• Winnie the Pooh - Prediction Markets Dataset (Hugging Face)