Background

On January 1, 2020, the California Consumer Privacy Act gave every California consumer the right to ask a company what personal information it holds on them — the categories, the specific pieces, where it came from, and who it was disclosed to — and the right to have it deleted. The company gets 45 days to answer. Not 45 days to start; 45 days to deliver.

That is a reasonable ask of a startup with one Postgres database. Capital One has roughly 100 million customers and decades of systems: core banking, card processing, servicing, collections, marketing, a data lake, a warehouse, and a long tail of everything else. Answering "what do you know about me?" means answering it everywhere, per person, on a legal clock, without ever handing the wrong person someone else's financial data.

I built the data fulfillment pipeline that did it — the machinery that took a consumer request, found that human being across the estate, assembled what was actually in scope, delivered it, and proved it had done so.

Challenge

The naive version of this is a script that queries a database. The reasons that doesn't survive contact with a bank:

• A bank isn't exempt — its data is. This is the detail almost everyone gets backwards. The Gramm-Leach-Bliley carve-out in CCPA is a data-level exemption, not an entity-level one. A bank is never "CCPA-exempt." Information collected while providing a financial product is generally out of scope; information about website visitors with no relationship, marketing and prospecting data, and employees and job applicants is squarely in. Worse, the same field flips status by context — an IP address captured during a loan application is likely exempt, while the identical IP from a marketing page is not. Scope is a property of how a record was collected, not of the column it lives in. You cannot resolve that with a schema allowlist; it has to be carried as provenance.
• You can't delete — or disclose — what you can't find. The obligation covers every system, and "we couldn't locate it" isn't a defense; a disproportionate-effort claim requires a detailed factual explanation, not an assertion. Discovery is the hard part, and it's the part that fans out across dozens of stores that share no conventions.
• Identity verification is a two-sided error function. Disclosing specific pieces of personal information requires a high degree of certainty. Under-verify and you hand a stranger a customer's financial history — that's a breach, and breaches are exactly where a bank's GLBA shield stops protecting it. Over-verify and you've built unlawful friction. And verification does not toll the clock: the 45 days run from receipt, so every hour spent proving who someone is comes out of the budget for answering them.
• There is no single join key. "Match the customer" presumes a customer already exists as one object. In reality a person is scattered across systems that each minted their own identifier, plus records with no clean key at all. Entity resolution isn't a feature of the pipeline — it's a prerequisite for the pipeline to start.
• Deletion is a distributed-systems problem with legal semantics. It fans out to service providers and third parties, who must fan out again. Backups are deferred, not forgiven — the obligation reattaches the moment an archive is restored. A DELETE that a restore silently resurrects is a violation with a latency fuse.
• The proof is the product. Regulators reconstruct violations arithmetically, per consumer, from your own numbers — and penalties land per consumer, on the order of $10K each. At 100M customers, an architectural gap isn't one mistake; it's N of them, where N is everyone affected. "We complied" is only true if it's provable for every individual request.

Solution

The pipeline treats a privacy request as a distributed query with a legal deadline: resolve the human, fan out across the estate, consolidate what's in scope, deliver it, and leave an evidence trail behind every step.

One query language for two engines

Customer data didn't live in one place or speak one dialect — the relevant stores split across PostgreSQL and Snowflake. Rather than maintain two divergent sets of hand-written extraction queries (and two sets of bugs), I co-developed Query Builder, a Python module that dynamically generated queries for both engines from a single definition. Adding a system, or changing what a request had to collect, became a change to data rather than a change to code — which is what let the collection layer keep up with an estate that never stopped moving.

Orchestrated fan-out on a deadline

Discovery is the slow, parallel, failure-prone stage: many systems, each with its own latency and its own ways of failing. Orchestrating it with Apache Airflow made the deadline a scheduling problem instead of a hope — per-system retries, explicit dependencies, and an execution history that doubles as audit evidence. Extraction ran in Docker containers on ECS, which kept each system's connector isolated with its own scoped credentials, so a job touching one store couldn't become a blast radius across the others.

Consolidation in S3, delivery on rails

Per-system extracts staged into S3, where they were joined into a single per-consumer record and filtered down to what was genuinely in scope. Request state and the audit log lived in RDS/PostgreSQL — the transactional record of what was asked, what was verified, what was queried, and what was returned. A Java Spring Boot service layer fronted the pieces that needed to be callable by the rest of the bank.

Closing the loop with Kafka

A fulfilled request isn't done when the data is assembled — the consumer has to be told, and downstream systems have to act. I built the component that ETL'd customer data into AWS and notified the team behind the customer-facing React application over Kafka so customer outreach could go out, and wrote the Kafka consumer that processed consolidated privacy requests staged in S3 and drove customer notification.

Kafka is the right shape for this specifically because notification and deletion propagation fan out to systems you don't control. A synchronous call chain across that many consumers can't be made reliable; a durable, replayable log can. Replay is also the answer to the backup problem — when archived data comes back, the events that must be re-applied to it are still there.

Making it provable

Compliance work is only worth what you can demonstrate, and the same is true of the code that does it. I raised unit test coverage past 90%, which did more for Jenkins CI/CD reliability than any single feature: a pipeline whose job is to be trusted with 100 million people's financial data, on a legal clock, cannot be a pipeline anyone is afraid to deploy.

Results

The pipeline turned an open-ended legal obligation into a repeatable, orchestrated, auditable operation covering roughly 100 million customers — requests resolved to a real human, fanned out across the estate, consolidated to what was actually in scope, delivered, and evidenced end to end, inside the statutory window.

Why this matters for a business:

• Privacy compliance is a data-engineering problem wearing a legal costume. The statute is short; the difficulty is entirely discovery, identity resolution, and propagation across systems that were never designed to answer questions about a single person. Companies that treat it as a policy exercise end up doing it by hand — at industry-cited costs north of a thousand dollars per request, which is a headcount plan, not an architecture.
• The audit trail isn't instrumentation — it's the deliverable. Penalties are computed per consumer, from your own records. That inverts the usual priority: the log isn't there to debug the pipeline, the pipeline exists to produce the log.
• Scale removes the excuses, not the deadline. The 45-day clock is identical for a company with one database and a bank with a hundred million customers. There is no volume defense, which means the only viable answer is automation that fans out as wide as the estate does.
• Structural gaps multiply. Regulators haven't punished companies for losing a single request; they've punished them when the mechanism was incomplete. At this scale one missing system is not one violation — it's one per affected person.

Conclusion

CCPA fulfillment at a bank is the intersection of the two things that make data work hard: nobody knows where all the data is, and the law doesn't care. Getting it right meant building for discovery across a heterogeneous estate, resolving fragmented records into actual people, propagating durably to systems outside our control, and proving every step — on a clock that started the moment a consumer hit submit.

If you're facing a version of this — a compliance obligation that's really a distributed data problem, or a pipeline that has to be right rather than merely fast — that's the kind of work I do. Book a call and I'll walk you through how I'd approach it.