Data quality

Every venue is captured twice, from two regions, and merged by the venue's own sequence numbers. Anything still missing after the merge is written to a table you can query. Corrections are published as new revisions, and old files are never edited.

Dual-region capture

Each venue stream is captured independently in Tokyo (ap-northeast-1, primary) and Ireland (eu-west-1, secondary). From Phase 2, N. Virginia (us-east-1) is added as a third region. The two capture sessions share nothing: separate hosts, separate AWS accounts, separate network paths and separate connections to the venue. A venue disconnect, a regional network event or a bad deploy on one side leaves the other side recording.

Live and historical merges

The live stream cannot wait for Ireland on every message without adding about 105 ms of delay for a Tokyo-hosted venue. It forwards the first copy of each message to reach it, which for a Tokyo-hosted venue is almost always the primary. When the primary detects a sequence break, the stream holds that symbol for up to 750 ms waiting for the secondary to supply the missing messages, then either emits them (marked region = euw1) or emits a gap notice and resyncs.

The historical merge runs again after the UTC day closes, with no time limit, over both raw copies. It can fill messages that reached Ireland after the live holdback expired. Historical files are therefore at least as complete as the live stream was, and the gaps table lists only what the final merge could not fill.

Sequence checks

A gap you cannot see is worse than a gap you can. We check continuity with the strongest signal each venue provides:

For trade streams without a sequence, we compare trade ids where the venue assigns them contiguously, and cross-check trade counts against the venue's REST history after the day closes. Every venue other than Hyperliquid is planned, subject to a data agreement with the exchange; the rules above for those venues follow their public API documentation and will be confirmed with each venue before capture starts.

The gaps table

Every interval that survives the merge without data gets a row in gaps. Gaps are recorded and never filled: we do not interpolate, forward-fill or synthesize rows, and order books are never backfilled from other sources. Intervals where data is present but a quality bound was missed, such as a capture clock outside its error budget, are not gaps; they are published with the daily quality metrics below.

All columns are on Schema reference. Every Parquet file also states how many gap rows overlap it in its fdc.gap_count metadata, so a pipeline can check completeness without a query.

import pandas as pd
from fdc import Client

fdc = Client()
gaps = pd.DataFrame(fdc.gaps(venue="hyperliquid", symbol="BTC-PERP",
                             start="2026-06-01", end="2026-07-01"))

# drop trades that fall inside any gap before computing a statistic
trades = fdc.trades(venue="hyperliquid", symbol="BTC-PERP",
                    start="2026-06-01", end="2026-07-01").to_pandas()
for g in gaps.itertuples():
    trades = trades[~trades.ts_capture.between(g.ts_start, g.ts_end)]

curl -s https://api.financedatacloud.com/v1/gaps \
  -H "Authorization: Bearer $FDC_API_KEY" \
  -G -d venue=hyperliquid -d symbol=BTC-PERP \
  -d start=2026-06-01 -d end=2026-07-01

{"gap_id":"gap_hl_20260611_0007","venue":"hyperliquid","dataset":"book_l3",
 "symbol":null,"reason":"venue_outage",
 "ts_start":1781179412804113201,"ts_end":1781179431266048873,
 "seq_start":1020315471708160,"seq_end":1020315634237439,
 "regions_missing":["apne1","euw1"],"resync":true,"detected_by":"seq","detected_at":1781179413560220514,
 "revision":1,"note":"API nodes stopped publishing; both regions disconnected within 40 ms."}

The gaps table is available on every plan, including test keys.

Venue maintenance

Venues announce maintenance windows through their own channels. We track announcements from each venue's status page and API channels, and treat the window like any other interruption: data before and after is kept, the interval itself is recorded.

• Feed down, trading down. One gaps row per affected dataset with reason = venue_outage and the venue's notice reference in note. instruments gets a halted row for the window.
• Feed up, trading halted. No gap, because the feed is complete. The instruments status change to halted or reduce_only tells you the book was not live.
• After the window. The book restarts from a venue snapshot, and the gap row has resync = true. Venues sometimes cancel resting orders during maintenance; the first snapshot after the window shows the book they restored, and the difference is not reported as cancel events.
• Funding. If a venue skips or delays a funding settlement during maintenance, the settled row appears when the venue publishes it, with ts_funding as the venue states. A skipped settlement has no row.

Revisions

Published data is never modified in place. When we correct something, we publish a new revision of each affected file and record it in the changelog. Code that read revision 1 last month can read revision 1 again today and get the same bytes.

What causes a revision

• A normalization bug: a field mapped incorrectly for a venue or a period.
• Recovered data: messages found in the raw tier after the daily merge, for example from a delayed regional upload.
• Venue corrections: trades the venue later busts or amends, and reference data the venue republishes.

Schema changes are a separate mechanism with their own rules; see the versioning section of Schema reference.

How a revision is published

• The file at the canonical S3 path is replaced by the new revision, with fdc.revision incremented in its metadata.
• Earlier revisions stay readable as S3 object versions of the same key for 24 months, and through the revision parameter on REST and in the SDK.
• The changelog entry lists the dataset, venues, symbols, date range, new revision number, cause, and counts of rows added, removed and changed.
• Account contacts get an email on the day of publication. The gaps table is revised the same way: a gap filled by recovered data is removed in the new revision and remains readable in the earlier one.

DATA REVISION  book_l2 · hyperliquid · all symbols
Range          2026-02-03 to 2026-02-05 (UTC dates of ts_capture)
Revision       1 -> 2
Cause          Hourly snapshots omitted levels with size below 1e-8.
Rows           +18,442 added · 0 removed · 0 changed
Action         Re-download if you replay from hourly snapshots in this range.

To pin a revision in a pipeline that must be reproducible, pass it explicitly:

fdc.download(dataset="book_l2", venue="hyperliquid", symbol="BTC-PERP",
             start="2026-02-03", end="2026-02-06", revision=1, path="./pinned")

Quality metrics

We compute the following per venue, dataset and UTC day, and publish them on Status. The same numbers drive our internal alerts.

If a metric misses its target, the affected day is annotated on Status and in the file metadata. To report something that looks wrong in the data, write to daniel@financedatacloud.com with the venue, symbol and a seq or time range; confirmed issues are fixed through the revision process above.