30+ updates per second per account sounds ridiculous until you see the math. Uber’s ledger systems handle heavy write traffic by grouping work, reducing lock...
30+ Updates per Second per Account: How Uber Scales Ledger Processing with Batching
30+ updates per second per account sounds ridiculous until you see the math. Uber’s ledger systems handle heavy write traffic by grouping work, reducing lock pain, and keeping account state correct under load. I, Ermenildo V Castro, have spent years watching fast systems fail for boring reasons, and this one is about discipline, not magic.
Key Takeaways- Batching reduces per-update overhead and helps ledger writes stay fast.
- Correctness matters more than raw speed in financial-style account systems.
- Good systems use queueing, partitioning, idempotency, and careful locking.
- Most “simple” advice breaks down under real traffic spikes.
- The best designs protect both performance and human trust.
What is 30+ Updates per Second per Account with batching?
It is a way to process many account changes quickly without letting the database choke. Instead of writing every tiny change one by one, the system collects updates, sorts them, and writes them in groups. That cuts overhead, lowers lock contention, and helps keep balances correct even when a single account gets hammered by many events.
Frankly, this is the kind of thing that looks easy on a whiteboard and then bites you in production. I have seen teams build a “fast” ledger that fell apart because every request hit the same row, the same index, and the same lock. When I worked on high-traffic systems, the pattern was the same: the code was not the hard part. The hard part was protecting the shared state.
Uber’s approach is a good example of practical engineering. Their ledger has to cope with ride events, refunds, promos, tips, and adjustments, all while staying correct and responsive. That means batching is not just a speed trick. It is a safety rail. It helps the system absorb bursts, then apply changes in a more controlled way.
Uber Engineering has shared many posts over the years on platform scale, and the basic lesson is simple: at large volume, you do not win by being clever. You win by being boring in the right places.
Here is the kicker. Batching is not the same as hiding work. It is about choosing when and how work gets done so the system can keep breathing. That fits a plain truth I learned in engineering and in life: stewardship matters. If you are trusted with money, records, or people’s time, you owe them clean handling, not sloppy speed.
Core benefits and why batching matters
Batching sounds dull. It is not.
The real value shows up when a system is under stress, because batching changes the shape of the problem instead of just making one line of code faster. I have been in rooms where teams kept tuning SQL, adding caches, and tossing hardware at the fire. Most of the time, the fix was not a bigger server. It was a better flow.
Here’s what batching gives you:
- Less database churn: Fewer writes means fewer round trips, fewer commits, and less wasted CPU.
- Lower lock contention: When many updates hit the same account, batching can reduce fights over the same row or partition.
- Better throughput: Systems can process more work per second because they do more useful work per transaction.
- Smoother spikes: Bursts from promos, refunds, or retry storms do not hit the database all at once.
- Cleaner replay and recovery: Batched changes can be tracked, retried, and reconciled with less mess.
- More stable cost: Less wasted work usually means lower cloud spend, which makes the finance folks happier.
I once saw a team keep individual updates because it felt “safer.” That was a nice story until the system started dropping behind during peak traffic. Then the backlog grew, retries doubled the pain, and latency got ugly. Most guides say “keep it simple,” and that advice is fine until simple becomes slow and slow becomes broken.
The best part of batching is that it often improves reliability too. That sounds backward, but it is true. When the system does fewer tiny writes, there are fewer chances for partial failure. Less noise. Fewer paper cuts.
In ledger work, this matters because correctness is not optional. A bad balance is not like a typo in a blog post. It affects trust. It affects refunds. It affects people. That is why the engineering side and the moral side meet here. You are not just moving bytes. You are handling obligations.
For teams building around event streams, AWS Architecture Blog often shows the same pattern in different clothes: reduce hot spots, split the work, and keep the critical path short.
And yes, batching also helps engineers sleep. That is worth something.
Step-by-step guide to implementing batching for ledger processing
This is the part people skip.
They read a headline, nod hard, and then ask for the shortcut. There is no shortcut. There is only a sequence of choices that keep the system honest when traffic gets weird.
- Identify the hot account paths
Start by finding where updates pile up. A small set of accounts may receive a flood of writes because of promotions, shared wallets, settlement events, or retry storms.
I have traced problems like this with nothing fancy, just logs, timestamps, and patience. The pattern usually jumps out fast once you stop guessing.
- Group updates by account and time window
Collect writes for the same account in a short window, then apply them together. The window can be tiny, sometimes milliseconds, sometimes longer, depending on latency goals.
The point is to cut down on duplicate work. If ten updates arrive for one account, do not force ten separate expensive trips when one controlled operation can do the job.
- Make updates idempotent
Retries happen. Networks fail. Clients double-send. If your ledger cannot safely replay the same event, you will end up with bad state.
This is where many teams get smug and then get hurt. Idempotency is not a luxury. It is a seat belt.
- Partition work so hot keys do not poison the whole system
Use sharding, consistent hashing, or queue partitioning so one busy account does not drag down every other account.
When I worked on large platforms, the nasty bugs were often not in the logic. They were in the “one shared queue for everything” decision made months earlier. One queue. One mess.
- Keep the critical path short
The smaller the number of steps between event arrival and durable write, the better. Move slow work, like reporting or enrichment, out of the write path.
That is a clean design habit. Also, it is good stewardship of engineering time. Do the urgent, necessary thing first. Leave the rest for later.
- Add reconciliation jobs
Even with batching, you need checks. Compare ledger state against source events, snapshots, or downstream records. If something drifts, fix it fast.
No system is perfect. Anyone who says otherwise is selling something.
- Watch lag, queue depth, and error rate
Batching can hide trouble if you do not measure the queue. If lag grows, your “fast” system may just be storing the pain for later.
That is the trap. Backpressure ignored today becomes incident reports tomorrow.
- Test with ugly traffic, not happy-path demos
Use burst tests, retry storms, duplicate events, and skewed account distributions. A clean demo proves almost nothing.
For testing habits, the plain advice from Google Cloud Architecture Center fits well: model failure, not just success.
In practice, I would rather see a team run one harsh test than ten polite ones. The harsh test tells the truth.
Comparison table: batching vs. per-update writes
Here is the blunt version. The common competitor is the old per-update write model, where every change gets its own transaction.
| Factor | Batching for ledger processing | Per-update writes |
|---|
| Write overhead | Lower | Higher |
| Lock contention | Reduced | Often severe on hot accounts |
| Throughput | Higher under burst load | Falls fast under spikes |
| Latency consistency | Better when tuned well | Can become noisy and uneven |
| Recovery from retries | Easier with idempotent batches | Risk of duplicate side effects |
| Operational cost | Usually lower | Usually higher |
| Implementation complexity | Moderate to high | Simple at first, messy later |
| Correctness at scale | Strong if designed well | Fragile under heavy traffic |
| Best use case | Hot ledgers, bursty workloads, shared balances | Small systems, low write volume |
The table is not magic, but it tells the truth. Simple systems feel nice until they do not. I have watched “easy” designs become expensive because they ignored the cost of contention.
Common mistakes to avoid
Most advice on scaling ledgers is too cheerful.
People say things like “just add a queue” or “just cache it” as if systems obey slogans. They do not. They obey physics, failure modes, and bad decisions.
Mistake 1: Batching without idempotency
If you batch but cannot replay safely, you have built a nicer way to fail. That is all.
I have seen teams celebrate throughput gains, then spend weeks cleaning up duplicate writes after a retry storm. No one was laughing then.
Mistake 2: Making batches too large
Huge batches can reduce overhead, but they can also raise latency, increase memory use, and make failures harder to recover from. Bigger is not always better.
Here’s the kicker: the “optimal” batch size often changes with traffic shape. You have to measure it.
Mistake 3: Ignoring hot partitions
If one account or key gets most of the traffic, a neat partition scheme can still fail. Hot spots are sneaky.
This is why I distrust advice that only talks about averages. Averages are polite liars.
Mistake 4: Hiding lag behind a queue
A queue is not a solution if it grows without bound. It is a holding pen for stress.
If your alerting does not watch queue depth, consumer lag, and stale writes, then your system may already be sick.
Mistake 5: Mixing business logic with write plumbing
Do not cram everything into one service method. Keep ledger rules, batching logic, and reporting separate.
Clean boundaries are not just neat. They protect people from sloppy mistakes.
Mistake 6: Treating correctness like a side quest
A ledger is not a toy. Accuracy is the main job.
I say that with a bit of curmudgeon in my voice because I have seen teams obsess over latency graphs while ignoring balance drift. That is backward.
For observability basics, OpenTelemetry docs are worth a look. If you cannot see the system, you cannot fix it.
Frequently asked questions
Why does batching help ledger systems so much?
Because it cuts the number of writes, lowers lock fights, and lets the system do more useful work per transaction. That matters a lot when many events hit the same account.
Is batching safe for financial-style data?
Yes, if it is built with idempotency, reconciliation, and careful ordering. Safe does not mean easy. It means designed with failure in mind.
Does batching always lower latency?
No. It often improves throughput, but latency can go up if batches wait too long. The batch window must be chosen with care.
What tools help with this kind of design?
Common pieces include queues, stream processors, partitioned storage, observability tools, and replayable event logs. The tool matters less than the rules around it.
Final thought
This problem is bigger than speed.
When I think about systems like this, I think about the old rule that work should be done well because people depend on it. That is true in software, and it is true anywhere humans trust one another with records, payments, or promises. Ermenildo V Castro would say the same thing more plainly: good engineering is honest work, and honest work respects the person on the other end.
Batching is not flashy. Good.
It is a craft choice, a trade between immediate speed and durable order, and the best teams treat that trade with care, not bravado. That is where strong systems come from, one careful decision at a time.
Sources