Spring Boot Batch
01 — The brief
A scheduled data-processing platform was outgrowing its window. Every night six jobs had to validate, transform and hand off roughly ten million records to two downstream services — notification and reporting. The runtime crept; the on-call window quietly stretched; the team agreed: re-architect, do not bandage.
The non-negotiable constraints were simple. The nightly window could not be extended. The historical correctness of every record had to be preserved. And the new system had to be testable to the point where a coverage report could be opened in code review and treated as evidence, not aspiration.
02 — Approach
The shape of Spring Batch is partition, step, listener. I worked through the existing jobs and asked, for each step, whether the I/O pattern matched the work. Several reads that were chunked by row were better off chunked by foreign-key cluster; several writers were doing one round-trip too many; one transformation was doing in code what the database could do in a single set-based statement.
In parallel I authored a JUnit suite from scratch. Service tests, repository tests, and end-to-end step tests that spin a job up against an embedded database. The coverage report crossed 90% across three projects. More importantly, the tests run fast enough to live in CI, and the green test report became the artefact reviewers actually opened.
- Reading cluster-chunked, paged, deterministic order
- Processing pure functions, no hidden side-effects
- Writing set-based where possible, idempotent always
- Listeners structured logs, metrics, retry policies
- Testing JUnit 5 + Testcontainers + JobLauncherTestUtils
None of this was free. Every choice below bought speed or safety by paying for it somewhere else — the discipline was in choosing where to pay.
-
Trade-off 01
Cluster-chunked reads over row-by-row
Chunking by foreign-key cluster collapses many small reads into few large ones — far fewer round-trips per partition. The cost: reads now depend on a deterministic key order, so restart-after-failure has to be reasoned about explicitly rather than assumed.
-
Trade-off 02
Set-based SQL over in-code transformation
One transformation the database could do in a single statement was being done row-by-row in Java. Moving it into SQL was a large win in runtime — paid for by pushing logic into the database, where it is less portable and has to be tested against a real engine, not a mock.
-
Trade-off 03
Idempotent writers over fire-and-forget
Making the writer idempotent means a re-run can never produce a duplicate downstream — correctness becomes a property of the code, not a hope of the operator. The cost is real write-time work: a dedup key and a check that a naive writer would skip.
-
Trade-off 04
Real-database tests over mocks
Testcontainers spins an actual database for the end-to-end step tests, so SQL and mapping bugs surface in CI instead of production. The cost is slower tests than mocks would give — kept in check by reserving the heavy tests for the steps that touch real I/O.
The fastest batch job is the one that does not read what it does not need.
03 — Results
The total nightly runtime fell by forty percent. Test coverage held above ninety on every push. The notification and reporting services downstream stopped seeing the occasional duplicate or late record, because idempotency was now a property of the writer rather than a hope of the operator.
04 — What’s next
The work continues. The next problems are about observability — making the cost of every step legible at a glance — and about pushing parts of the pipeline into event-driven shapes where the nightly window is no longer the right unit at all.