PII Audit¶
Audit a repository and its data files for exposed personally identifiable information — real names, SSNs/SINs, emails, phone numbers, addresses, DOB, or payment card numbers in tracked data files, code, logs, or fixtures.
Trigger: /pii-audit
How It Works¶
- Redaction rule first — every finding is reported as file, line (or column + row count), and PII type, plus a masked sample (
***-**-1234,j***@***.com). The real value never appears in output. - PII types enumerated — SSN, Canadian SIN, email, phone, address, DOB, credit card (Luhn-valid), and full names paired with any of those in the same record.
- Column/field names checked first — schema definitions, CSV/Parquet headers,
CREATE TABLE, and Pydantic/dataclass fields for PII-shaped names, cheapest and highest-signal pass. - Tracked data files sampled — CSV/Parquet/Excel/SQLite/DuckDB files tracked by git get a row sample checked against the PII patterns; reported as match counts, never matched values.
- Code and notebooks scanned for PII interpolated into log/print statements, and fixtures/seed scripts with real-looking data instead of synthetic (
faker) data. - History scan — data files deleted in a later commit are still in every clone; huge repos get a bounded scan, stated honestly.
- Missing controls flagged — no masking/hashing step before a dashboard or export, no
.gitignoreentry for a raw-data directory. - Severity-ranked report — EXPOSED (pushed: a disclosure conversation, not just a code fix), exposed-local-only (scrub via Git Rescue), hardcoded fixtures, and hygiene gaps — ending in a one-line verdict: CLEAN / HYGIENE GAPS / EXPOSED.
- Stop after reporting — masking, hashing, deletion, and history rewrites happen only when the user asks.
Redacted Reporting¶
Bad: the audit becomes the leak
A real SSN and name are now in the conversation, the terminal scrollback, and any transcript that captures them.Good: type, location, and masked sample, never the real value
Verdict Honesty¶
Bad: green because no obviously-named file was checked
The scan only grepped filenames for "customer" or "pii"; a differently-named export with the same columns was never opened.Good: scope and gaps stated
Verdict: EXPOSED (1) + HYGIENE GAPS (2)
- data/customers.csv column "ssn" → real SIN-shaped values, pushed → untrack + /git-rescue, then a disclosure conversation.
- etl/transform.py:41 logs the full "email" column via logging.info() → strip from the log statement.
- benefits_finder/raw/ has no .gitignore entry → add it before the next real-data drop.
Notes¶
- A person's PII isn't a secret you rotate. Once a SIN or DOB tied to a real person is pushed, deleting it later doesn't undo the exposure — findings say so plainly.
- Never print a full matched value to confirm a hit — masked samples and counts are enough.
- Read-only by design: no edits, no masking, no history rewrites. History surgery is Git Rescue's job; data-quality issues are Data Profile's job.
- Column-name matches can be false positives — check what the column actually holds before calling it a finding.
- Covers PII in data; credentials and API tokens are Secrets Audit's scope — run both when onboarding a new data source.
- Natural habit points: before sharing a dataset externally, before Deploy Streamlit or Deploy Web App ships a dashboard over real data, and after onboarding any new data source.