Streaming Data Engineering

Real-Time Payment Transaction Processing with Databricks & PySpark: A Streaming Case Study

How Spark Structured Streaming, Databricks Auto Loader, Delta Lake, and the Medallion Architecture turn raw payment events on AWS S3 into real-time fraud signals and trusted analytics.

By EdgeWave SoftwareOctober 16, 202611 min read

Payments never sleep. Every second, transaction events arrive from gateways, terminals, and mobile apps — and the business that can analyse them in real time can stop fraud, protect revenue, and understand customer behaviour as it happens. This case study shows how a continuous streaming pipeline built with PySpark, Databricks Auto Loader, and Delta Lake turns raw JSON payment events on AWS S3 into trusted, analytics-ready data.

Why data engineering matters in payments

Payment data is high-volume, high-velocity, and high-stakes. A batch report that runs once a day cannot flag a suspicious transaction burst happening right now, and a pipeline that drops or duplicates events quietly corrupts every financial number downstream.

Data engineering provides the streaming backbone: continuous ingestion, enforced schemas, exactly-once-style deduplication, and windowed aggregations that make real-time fraud detection and operational reporting possible.

High-velocity events

Thousands of JSON payment events per minute arrive from multiple channels and must be processed continuously, not overnight.

Fraud window is seconds

Anomalous high-value transactions must be flagged while they can still be stopped — batch latency is not acceptable.

Late and out-of-order data

Events arrive late or out of sequence due to retries and network delays, corrupting naive counts and totals.

Untrusted raw data

Malformed JSON, missing fields, and duplicate deliveries from the source systems cannot reach financial reports.

Step 1 — Continuous ingestion with Auto Loader

The pipeline begins with Databricks Auto Loader (cloudFiles), which incrementally and efficiently discovers new raw JSON payment transaction files as they land in AWS S3 — no file listing scans, no re-processing of old data.

Schema enforcement — every event is validated against a declared transaction schema (transaction ID, customer, merchant, amount, currency, timestamp); malformed records are routed to a quarantine location instead of crashing the stream.
Schema evolution — when the payment provider adds fields, Auto Loader handles the change gracefully rather than breaking the pipeline.
Checkpointing — Structured Streaming checkpoints guarantee the stream restarts exactly where it stopped after any failure or deployment.

Step 2 — Deduplication, event-time windows, and watermarks

Raw streams are messy. The PySpark Structured Streaming layer cleans them with three essential techniques:

Deduplication — dropDuplicates on the transaction ID within the watermark window removes redelivered events, so every payment is counted once.
Event-time windowing — aggregations are computed on the time the transaction actually happened, not when it arrived, keeping analytics correct under delays.
Watermarking — a watermark defines how long the stream waits for late-arriving data before finalising a window, bounding state size while still accepting realistically late events.

Together these turn an unreliable event firehose into a deterministic, trustworthy stream of payment facts.

Step 3 — Real-time anomaly detection aggregations

On the cleaned stream, the pipeline computes rolling behavioural metrics that power fraud and risk monitoring:

Rolling transaction counts by customer and by merchant over sliding time windows — a sudden spike in velocity is a classic fraud signal.
Aggregate transaction values per customer and merchant per window, compared against normal behaviour.
High-value transaction flagging — events exceeding dynamic thresholds are marked anomalous and pushed to the Gold layer for immediate alerting and investigation.

Because these aggregates update continuously, risk teams see suspicious patterns in seconds rather than discovering them in tomorrow's report.

Step 4 — Medallion Architecture on Delta Lake

All data is structured using the Medallion Architecture inside Delta Lake, so every consumer reads from the layer appropriate to their need:

Layer	Contents	Consumers
Bronze	Raw JSON events exactly as ingested from S3, with schema enforcement and full history	Engineering, replay & audit
Silver	Cleaned, deduplicated, validated transactions with enforced types	Data science, downstream pipelines
Gold	Windowed aggregates by customer and merchant, anomaly flags, reporting datasets	Fraud dashboards, BI, alerts

Delta Lake's ACID transactions, time travel, and optimised file layout keep read performance fast for downstream analytics and SQL reporting — even while the stream is writing continuously.

Business outcomes

Fraud signals in seconds

Rolling counts and value aggregates flag anomalous high-value transactions while they can still be acted on.

Numbers finance can trust

Schema enforcement, deduplication, and watermarking mean every transaction is counted exactly once, in the right time window.

One platform, many consumers

The Medallion layers serve engineering replay, data science, fraud monitoring, and executive reporting from the same governed data.

Built to scale

Auto Loader and Structured Streaming absorb growing payment volumes and new event sources without re-engineering.

Key takeaways — the use of data engineering in payments

Streaming is a business capability. Real-time ingestion and aggregation are what make fraud prevention and live operational insight possible.
Correctness before speed. Deduplication, event-time windows, and watermarks are what make streaming numbers believable.
Medallion layers create trust. Bronze, Silver, and Gold separate raw evidence from decision-ready data.
Delta Lake is the foundation. ACID guarantees and optimised storage let one lake serve streams and dashboards at once.

Build your streaming platform

Turn payment events into real-time intelligence.

EdgeWave Software designs and delivers streaming data platforms — Auto Loader ingestion, Structured Streaming, Delta Lake, and Medallion Architecture — on Databricks, AWS, and Azure.

Talk to our team