Sysctl settings for RDMA pods in K8s
Comments
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Ranked by a public rubric, not by votes: what a comment adds counts, not who wrote it. How comments are ranked · Comment rules
Summary
Oguz Pastirmaci's write-up traces intermittent RDMA failures on 16-rail RoCE GPU nodes to Linux sysctls that are not set where Kubernetes places the interface. Interfaces take their settings from the pod namespace's conf/default. In his DRANET tests on OKE A100 nodes, setting arp_ignore=1 and arp_announce=2 fixed ARP mix-ups, and NCCL reached about 190 GB/s. For IPv6, accept_ra=2 was needed. The results come from one author's clusters, and some figures lack independent sources.