Apache Celeborn is a remote shuffle service that stores the intermediate data Spark and Flink jobs need between processing stages. LinkedIn says its deployment...
Apache Celeborn at LinkedIn: How Remote Shuffle Cut Failures, Cost, and Runtime
Apache Celeborn is a remote shuffle service that stores the intermediate data Spark and Flink jobs need between processing stages. LinkedIn says its deployment cut daily shuffle fetch failures from about 750 to 20, reduced compute costs by 10%, and improved application runtimes by 14% compared with its previous Magnet setup. The key change was separating shuffle storage from compute workers.
LinkedIn runs more than 300,000 Spark jobs each day across thousands of production nodes, using roughly 200 petabyte-hours of compute. That workload makes shuffle performance a practical concern, not a small tuning detail. When a job moves data between stages, slow reads or failed fetches can hold up many tasks at once.
I’m Ermenildo Castro, and I’ve spent much of my career thinking about what happens when software meets production scale. I’ve seen teams focus on faster processors while a quieter bottleneck—data movement—keeps the whole service waiting. Celeborn’s results are worth studying, but they are LinkedIn’s reported results, not a promise that every company will see the same gains.
What Is Apache Celeborn?
Apache Celeborn is a remote shuffle service: a separate system that stores and serves intermediate data produced during distributed data processing. A Spark or Flink job is split into tasks. Some tasks create data, while later tasks read it and continue the work. The handoff between those stages is called a shuffle, because records must be grouped or moved across machines before the next stage can run.
Shuffle data can be large. It can also arrive in many small pieces, and reading those pieces randomly from disk can slow a job. Apache Spark’s documentation describes shuffle as an operation that redistributes data across partitions and often involves network transfers, serialization, and disk I/O. Those are several opportunities for contention. The Apache Spark scheduling documentation also explains how external shuffle services can help preserve shuffle files when executors are removed.
LinkedIn previously used Magnet, a push-based shuffle implementation contributed to Apache Spark 3.2. Magnet combines shuffle blocks for reducer partitions into larger files, which helps reduce random disk reads. But the shuffle service still ran inside the compute cluster. Its external shuffle services could compete with compute workloads for CPU, memory, disk I/O, and network capacity.

Celeborn changes where that work happens. Map tasks push intermediate data to dedicated Celeborn workers, which aggregate and store the partitions. Reducers then fetch data from those workers. Compute and shuffle storage can grow separately, and shuffle files are less tied to the life of an individual compute executor. The Apache Celeborn project documentation describes the service’s role in providing remote shuffle for distributed processing engines.
The distinction is not that Magnet does nothing useful. It improves shuffle handling within a Spark cluster. Celeborn moves the shuffle service out into its own worker fleet, with a separate control plane. In my experience, that boundary matters when one resource pool is being asked to do two jobs at once. A more capable disk cannot always fix a design that makes compute and data movement compete for the same machine.
Core Benefits and Why the Change Matters
The reported failure reduction is striking. LinkedIn says daily shuffle fetch failures dropped from roughly 750 to 20 after the move to Celeborn. A fetch failure can cause task retries, waste completed work, and keep a larger job from finishing. In a service running hundreds of thousands of jobs per day, a recurring failure rate can become a steady tax on teams and infrastructure.
Cost and speed improved too. LinkedIn reports a 10% reduction in compute costs and a 14% improvement in application runtimes compared with Magnet. Those figures should be read as results from LinkedIn’s workloads and deployment, not as a universal benchmark. Still, they show why it is useful to measure more than raw shuffle throughput. A system can move data faster and still fail to lower total job cost if it needs too many workers or creates new operational work.
The main benefits are practical:
- Less resource contention. Dedicated workers take much of the shuffle storage and serving work away from compute executors. That can make CPU and memory usage easier to plan, while reducing competition for local disk and network capacity.
- Independent scaling. Teams can add shuffle capacity without adding the same amount of compute capacity, or adjust compute without automatically expanding shuffle storage. This matters when a workload has brief, heavy data movement but modest processing needs.
- More resilience to executor changes. Since shuffle data is stored on separate workers, it is less dependent on the lifecycle of the executor that created it. That separation does not make failures impossible; it gives the system another place to manage data and recovery.
- Better support for backfills. LinkedIn is moving toward a unified Flink architecture for real-time and historical processing. Large historical corrections can create substantial shuffle traffic. Shared remote shuffle gives those workloads a dedicated service rather than forcing every processing cluster to hold all the data locally.
- More focused performance tuning. LinkedIn separately improved shuffle duration by moving storage to NVMe SSDs and reduced the shuffle worker fleet by about 35% using Zstandard compression instead of LZ4. Each change targets a different part of the cost: storage speed versus bytes stored and moved.
Here’s the kicker: infrastructure improvements are only useful if they help people get reliable results. In one kind of production incident I’ve seen repeatedly, engineers spend hours chasing a slow application, then discover that a shared disk or network path is saturated. Separating the service can make the cause easier to see and the fix less disruptive. That is good engineering stewardship: use expensive capacity carefully, and avoid making users wait because an internal resource pool was designed without clear limits.

Step-by-Step Guide to Evaluating a Remote Shuffle Service
A remote shuffle service is not a drop-in cure for every slow job. Start with evidence, then test the architecture against your own failure patterns. LinkedIn’s setup offers useful examples, but its scale and workload mix are specific to its environment.
- Measure the current shuffle problem. Record fetch failures, task retries, shuffle read and write time, executor time spent waiting, disk pressure, network use, and job duration. Break results down by workload rather than relying only on fleet-wide averages. When I review performance data, I look for a repeated pattern first: are failures concentrated on certain hosts, large partitions, or busy hours? Averages can hide the machine that keeps causing trouble.
- Separate symptoms from causes. A slow reducer may be waiting on a disk, network, overloaded shuffle service, or an upstream task that has not finished writing its data. Check logs and metrics together. Do not label every delay a shuffle bottleneck just because the shuffle stage is visible in the job interface. Confirm whether compute and shuffle service processes actually compete for CPU, memory, I/O, or network capacity.
- Choose a small, representative pilot. Include more than a short, easy job. Test large shuffles, failure recovery, backfills, and workloads with uneven partitions. Compare the current system and the candidate service using the same inputs where possible. The Apache Celeborn documentation is a useful starting point for supported concepts and deployment details, but a test in your own environment is what tells you whether the service fits your workload.
- Plan reliability before tuning speed. LinkedIn deployed a five-node Raft-based control plane across availability zones, using Apache Ratis for metadata replication, leader election, and failover. It also added network-aware replication and connected Celeborn to its host health system, so workers approaching maintenance could be deprioritized and degraded workers excluded. Apache Ratis provides the consensus tools behind this type of replicated service; its project site explains the library. Your design still needs clear failure limits, recovery tests, and ownership for on-call response.
- Tune the data path one change at a time. LinkedIn moved shuffle storage to NVMe SSDs, used 100 Gbps network interfaces, and optimized its data plane and JVM. It also reports that slot selection became six times faster. Compression is another lever: Zstandard reduced LinkedIn’s required shuffle worker fleet by about 35% compared with LZ4. Measure the CPU cost of compression against saved storage and network traffic; the best choice depends on hardware, data, and job shape.
- Set a rollout and rollback plan. Start with a small share of traffic, compare failures and completion time, and expand only when results hold across workload types. Track the cost of the whole system, including the shuffle worker fleet, control plane, network, storage, and engineering support. A pilot should include a safe fallback to the previous path. This is where teams often get sloppy: a benchmark can show that a service works, but production traffic reveals whether it works on a bad day.
- Keep measuring after launch. LinkedIn reports that the share of executor runtime spent waiting for shuffle operations fell by about 10–15% from its peak. That kind of measure can show whether a change helped the full application, not just one component. Watch for new limits as workloads change, especially as backfills grow or more pipelines move to Flink. The Apache Flink overview describes Flink’s stream and batch processing model, which makes it relevant when evaluating a shared platform for real-time and historical jobs.

Apache Celeborn vs. Magnet: What Is the Difference?
Magnet and Celeborn both address shuffle performance, but their architectures place the work in different locations. Magnet aggregates shuffle blocks into larger files and runs its merger services within the Spark compute cluster. Celeborn sends intermediate data to dedicated workers and uses a separate control plane. That difference affects resource sharing, scaling, and operations.
| Area | Apache Celeborn at LinkedIn | Magnet at LinkedIn |
|---|
| Shuffle storage | Dedicated remote workers store and serve intermediate data. | Shuffle services and storage remain within the compute cluster. |
| Relationship to compute | Storage capacity can scale separately from compute executors. | Shuffle service shares cluster resources with compute workloads. |
| Data handling | Map tasks push data to Celeborn workers; reducers fetch from those workers. | Aggregates blocks by reducer partition into larger files to reduce random reads. |
| Reliability design | Separate service and replicated control plane; LinkedIn added health-aware worker selection and network-aware replication. | Uses merger External Shuffle Services in the Spark cluster. |
| LinkedIn-reported comparison | About 20 daily shuffle fetch failures, 10% lower compute cost, and 14% faster application runtimes. | Previous implementation used as the comparison point for those reported results. |
That table needs a caution label. LinkedIn’s numbers describe its own before-and-after comparison, and the available report does not establish that every workload or cluster will get the same result. The difference in architecture is more general: Celeborn puts shuffle storage on its own workers, while Magnet improves how shuffle data is organized within the existing compute cluster.
Magnet may remain a sensible choice when a team’s shuffle load is manageable and adding a separate service would create more operational burden than value. Celeborn is more compelling when compute and shuffle repeatedly compete for resources, shuffle volume changes independently of compute demand, or large backfills put pressure on existing clusters. Frankly, the decision should follow measured constraints, not the desire to copy a large company’s stack.
Common Mistakes to Avoid
Treating a reported improvement as a guarantee is the first mistake. LinkedIn reports meaningful gains, but its fleet runs more than 300,000 Spark jobs a day and uses a particular mix of hardware, applications, and operational systems. Your workload may have smaller shuffles, different data, or another bottleneck. Use the figures to form a test plan, not to write a business case before collecting your own numbers.
Measuring only speed is another trap. A job that finishes sooner is useful, but it may consume more worker capacity or require more complicated support. Track job runtime, fetch failures, retries, compute cost, shuffle fleet cost, storage use, network traffic, and on-call impact. LinkedIn’s report includes several kinds of evidence, from application runtime and compute costs to worker fleet size and failure counts. That is more informative than a single throughput chart.
Ignoring operations can undo a sound design. A remote service adds workers, control-plane components, health checks, replication behavior, and new failure modes. LinkedIn’s engineering account says operating Celeborn reliably across thousands of nodes and hundreds of thousands of daily applications required substantial work beyond the upstream project. That is a reminder to assign ownership, define alerts, test maintenance events, and document recovery steps before broad rollout.
Changing too many variables at once makes results hard to explain. If a team changes storage type, compression, network hardware, JVM settings, and worker count in the same experiment, it may see a gain without knowing what caused it. Test changes in stages. Record the hardware and configuration, keep job inputs comparable, and repeat tests when the workload has meaningful variation.
Assuming more powerful hardware solves every problem is wishful thinking. LinkedIn’s 100 Gbps interfaces and NVMe disks can help, but the results also depended on service separation, replication choices, health-aware selection, compression, and software tuning. A faster network does not fix a worker that should have been removed from service. A larger disk does not fix a system that creates too many small reads. Diagnose first.
Finally, do not forget who bears the cost of unreliable data jobs. Failed backfills can delay corrections and reports that other teams depend on. Good engineering should respect that work and the people doing it, while using shared computing resources responsibly. The point is not to buy every new tool. It is to build systems that deliver dependable results without wasting the time and capacity entrusted to the team.
Frequently Asked Questions
What does Apache Celeborn do?
Apache Celeborn stores and serves shuffle data outside the compute workers running Spark or Flink tasks. A task writes intermediate data to Celeborn workers, and later tasks read it from there. The separation can reduce competition between processing and shuffle storage, while allowing the two pools to scale independently. It does not remove the need to monitor network, disk, and worker health.
How is Celeborn different from Magnet?
At LinkedIn, Magnet aggregates shuffle blocks into larger files but keeps its merger External Shuffle Services within the Spark compute cluster. Celeborn runs a separate shuffle service with dedicated workers and a replicated control plane. The key difference is where the shuffle work and storage live. Magnet improves shuffle handling within the cluster; Celeborn separates that service from compute.
What results did LinkedIn report after adopting Celeborn?
LinkedIn reported that daily shuffle fetch failures fell from about 750 to 20, compute costs decreased by 10%, and application runtimes improved by 14% compared with Magnet. It also reported other optimizations, including a roughly 35% smaller required shuffle worker fleet with Zstandard compression and 40% shorter overall shuffle duration after moving storage to NVMe. These are LinkedIn’s results, not guaranteed outcomes for other deployments.
Is remote shuffle worth the added operational work?
It may be worth evaluating when shuffle failures are frequent, compute and shuffle resources compete, or large backfills create sharp demand for intermediate storage. A separate service also adds a fleet and control plane that someone must operate. Measure your current failure rate, job delays, and total cost, then test a representative workload. If shuffle is not a real bottleneck, the extra service may not be worth maintaining.
LinkedIn’s Celeborn deployment shows what can happen when a team treats shuffle as a core part of its data platform rather than a side effect of compute. Its reported gains came from a combination of remote storage, reliability work, and careful tuning—not from a single configuration switch. I’ve learned that production engineering rewards this kind of discipline: make the constraint visible, protect the service during failure, and measure whether the change helps the people relying on the result. The sound choice is the one your own data supports.