Basics
RoCE vs InfiniBand: choosing a fabric
Both move data kernel-free — the difference is the network underneath and what it costs you.
5 min read
Same verbs, different wire
InfiniBand is a purpose-built fabric: lossless by design, credit-based flow control, sub-microsecond latency, and it needs InfiniBand switches and HCAs end to end. RoCE runs the same RDMA verbs over Ethernet — your existing switching works, and hosts converge storage, compute, and management on one wire.
What RoCE asks of your network
RoCEv2 needs a (near) lossless Ethernet: PFC and ECN configured correctly, or you'll see throughput collapse under congestion. Getting PFC right at scale is the main operational cost of choosing RoCE.
Rule of thumb
All-to-all GPU training or tight MPI coupling with a dedicated network: InfiniBand still wins. Everything else — storage, general clusters, mixed traffic, or anywhere you can't justify a second fabric: RoCE, provided the switching is configured for it.