Backfill¶
Fill a data store with historical records over a date range, safely restartable and gap-aware. Generalizes the Dukascopy download pattern to any source — API, files, ticks or bars.
Trigger: /backfill
How It Works¶
- Identify source, entity, date range, granularity, and destination.
- Detect what's present — query the destination for existing coverage and the gaps inside the range, not just the ends. Only fetch what is missing.
- Chunk by day or week so the job is resumable and memory-bounded.
- Fetch with retries and backoff, respecting rate limits. Log and skip a persistently failing chunk rather than aborting the whole run.
- Write idempotently — upsert /
INSERT OR REPLACE/ partition-overwrite keyed on (entity, timestamp). - Verify per chunk and report honest coverage: fetched, skipped, failed, and any gaps remaining.
Gap-Aware, Idempotent¶
Bad: re-download the whole range every run
Good: fetch only missing days, upsert
Honest Coverage Reporting¶
Report exactly what's still missing
Notes¶
- Weekends and market holidays are legitimately empty for forex — use a trading calendar so real holes stand out.
- Idempotency is the point: key every write on (entity, timestamp) so re-runs never duplicate.
- Store raw responses when feasible — re-transforming locally beats re-hitting the API.
- After backfilling the trading bot's history, the natural next step is MC Analysis.
- Source and destination credentials are read from the environment — never hardcoded in a script, printed in logs, or echoed while connecting.