Broadcom NIC Driver Installation on Ubuntu 24.04.2¶

The following procedure is for installing the out-of-box Broadcom NIC driver and RoCE libraries on Ubuntu 24.04.2.

NOTE This is for a single-node install. Scale with nodeshell. Arguments to make these steps unattended have been included.

Install dependencies for using RoCE with Broadcom NICs¶

DEBIAN_FRONTEND=noninteractive apt-get install -V -y rdma-core rdmacm-utils infiniband-diags libibverbs-dev ibverbs-utils

Install the Broadcom netxtreme-peer-mem-dkms package¶

Note that the Broadcom bnxt_en and bnxt_re dkms modules are included in the netxtreme-peer-mem-dkms package do they don't have to be installed separately (in fact, if they are, dkms status will complain about the built and installed versions of those modules not matching).

DEBIAN_FRONTEND=noninteractive apt-get install -V -y /cluster/drivers/bnxt_en/drum_40687/tmp/ubuntu/drivers_linux/peer_mem/dkms/netxtreme-peer-mem-dkms_232.0.155.5_all.deb

Install the Broadcom bnxt_re_conf package¶

DEBIAN_FRONTEND=noninteractive apt-get install -V -y /cluster/drivers/bnxt_en/drum_40687/tmp/ubuntu/drivers_linux/bnxt_re/bnxt_re_conf/bnxt_re_conf_232.0.155.5-1_all.deb

Build and install libbnxt_re¶

cd /cluster/drivers/bnxt_en/drum_40687/tmp/ubuntu/drivers_linux/bnxt_rocelib/tmp/libbnxt_re-232.0.155.5
make clean
sh autogen.sh
./configure --sysconfdir=/etc
make
make install all
cd

Prevent bnxt_en from being included in initramfs¶

The out-of-box Broadcom kernel modules will be installed under /lib/modules in the root filesystem, but, by default, won't be included in the initramfs when it is updated. This has the effect of the in-box bnxt_en module being loaded on boot instead of the out-of-box version and also preventing the out-of-box versions of the bnxt_en and Broadcom peer mem modules from loading. The following will cause the bnxt_en module to be omitted from the initramfs when the initramfs is updated, so the bnxt_en module will instead be loaded from the root filesystem, enabling the Broadcom out-of-box drivers to be loaded on boot. This approach works at least for Ubuntu 24.04.2 installed to local storage where the Broadcom NIC drivers aren't necessary to mount the root filesystem.

install -m 0755 /dev/null /etc/initramfs-tools/hooks/omit
cat <<'EOF' > /etc/initramfs-tools/hooks/omit
#!/bin/sh
PREREQ=""
prereqs()
{
    echo "$PREREQ"
}
case $1 in
prereqs)
    prereqs
    exit 0
    ;;
esac
. /usr/share/initramfs-tools/hook-functions
# Begin real processing below this line
OMIT_DRIVERS="bnxt_en.ko.zst"
for i in ${OMIT_DRIVERS}; do
    find ${DESTDIR} -name ${i} -delete
done
EOF

Update library path¶

echo "/usr/local/lib" >> /etc/ld.so.conf
ldconfig

Update the initramfs¶

update-initramfs -u

Reboot¶

reboot

Check to make sure updated Broadcom NIC drivers are loaded¶

root@n928:~# cat /sys/module/bnxt_en/version
1.10.3-232.0.155.5
root@n928:~# cat /sys/module/bnxt_re/version
232.0.155.5
root@n928:~# cat /sys/module/ib_peer_mem/version
232.0.155.5