Cross Compiling a Kernel Driver for Raspberry Pi 3 on Ubuntu

Finding the exact kernel version, downloading the toolchain, and compiling a .ko for Pi from Ubuntu

Vivek Bhadra  |  Raspberry Pi Linux Kernel Cross Compilation Device Drivers Ubuntu

Getting Raspberry Pi up and running

To bring up your Raspberry Pi (ver 3 B) with NOOBS follow the instructions here.

Find the kernel version of NOOBS installation

Go to Pi and open a terminal. Then type the below:

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

In the above it shows the kernel version this NOOBS installation has is 4.19.75-v7+. Now, for cross compilation we need the exact version of the Kernel source code, otherwise we will get the below error:

pi@raspberrypi:~ $ sudo insmod GobiNet.ko
insmod: ERROR: could not insert module GobiNet.ko: Invalid module format

Linux kernels for Raspberry Pi are located here:

https://github.com/raspberrypi/linux

By default the one coming right now is version 4.19.y. This doesn’t exactly match with the one we need i.e. 4.19.75-v7+. Next you can look into the Branches here:

https://github.com/raspberrypi/linux/branches

But, I can’t see the exact match. So now I look into the Releases tab. This looks like has the one we want:

https://github.com/raspberrypi/linux/releases

In this page I can see the Kernel release which was on the 24th Sep, 2019. This exactly matches with the one in NOOBS:

pi@raspberrypi:~ $ uname -a
Linux raspberrypi 4.19.75-v7+ #1270 SMP Tue Sep 24 18:45:11 BST 2019 armv7l GNU/Linux

Download this Kernel source code in your local machine (Ubuntu in my case). The tarball is called linux-raspberrypi-kernel_1.20190925-1.tar.gz. Extract the tarball with the below command:

tar -xvf linux-raspberrypi-kernel_1.20190925-1.tar.gz

In my setup after extraction the linux source is located in the below path:

/home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1

Now, go inside the linux kernel source folder and do the below:

cd /home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1
make mrproper

This will ensure we have started with a clean kernel source. Now, come to your Pi again. What we need now is the kernel configuration which was used for the NOOBS kernel compilation. The Kernel configuration now a days kept in the kernel itself as a module. So to grab hold of the kernel .config file do the below in Pi:

pi@raspberrypi:~ $ sudo modprobe configs
pi@raspberrypi:~ $ zcat /proc/config.gz > /tmp/.config

Copy this .config file in a sd card. Bring it back to the Ubuntu PC where your downloaded Kernel source tree lives. Then copy the .config file into the linux kernel source tree as below:

vbhadra@vbhadra-8200-elit-convertable-microtower:~/Downloads/linux-raspberrypi-kernel_1.20190925-1$ sudo mount /dev/sdb1 ~/usb/
vbhadra@vbhadra-8200-elit-convertable-microtower:~/Downloads/linux-raspberrypi-kernel_1.20190925-1$ ls -la ~/usb/
total 748
-rw-r--r--  1 vbhadra vbhadra 163003 Sep 26 02:23  .config
vbhadra@vbhadra-8200-elit-convertable-microtower:~/Downloads/linux-raspberrypi-kernel_1.20190925-1$ cp ~/usb/.config  .

Download the Pi toolchain

You will need the toolchain to cross compile the Kernel and your module for Pi environment. Download the toolchain from the git hub with the below command:

git clone https://github.com/raspberrypi/tools ~/tools

Locate where your tools folder is. I usually create a rpi folder in my home folder and do everything there. In my setup, the tools are located here: /home/vbhadra/rpi/tools. Now, you need to point where your tools (cross compiler) is for compiling the kernel. To do that do the below:

export CCPREFIX=/home/vbhadra/rpi/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-gnueabi-

Compile the kernel to create the Module.symvers file. To compile the kernel do the below in your Ubuntu setup:

make ARCH=arm CROSS_COMPILE=${CCPREFIX} -j 12

Let the above compile the Kernel which will take a while, wait patiently!

Now check if you have got a file created called in your linux folder:

vbhadra@vbhadra-8200-elit-convertable-microtower:~/Downloads/linux-raspberrypi-kernel_1.20190925-1$ ls -la Module.symvers
-rw-r--r-- 1 vbhadra vbhadra 841463 Feb 12 13:31 Module.symvers

Compile your module

I am trying to cross compile GobiNet and GobiSerial drivers for SieraWireless LTE device EM7565.

You can grab hold of the driver from SieraWireless website. This is not about any specific driver and their intricacies. We will only focus on the important bits. Our goal is to cross compile any custom driver for Raspberry Pi (hw version 3 B).

Once you have any driver code compiled for another platform, you want to cross compile it. First, you should check the driver Makefile. In this case, I have the below driver Makefile for GobiNet driver:

obj-m := GobiNet.o
GobiNet-objs := GobiUSBNet.o QMIDevice.o QMI.o
KDIR := /lib/modules/$(shell uname -r)/build
PWD := $(shell pwd)
OUTPUTDIR=/lib/modules/`uname -r`/kernel/drivers/net/usb/
#KBUILD_CFLAGS += -DQOS_SIMULATE
#KBUILD_CFLAGS += -DTX_XMIT_SIERRA -DTX_URB_MONITOR
ifdef TX_URB_MONITOR
ifeq ($(TX_URB_MONITOR), 1)
KBUILD_CFLAGS += -DTX_XMIT_SIERRA -DTX_URB_MONITOR
else
TX_URB_MONITOR:=0
endif
else
TX_URB_MONITOR:=0
endif
ifeq ($(TX_URB_MONITOR), 1)
	ccflags-y:=-DTX_URB_MONITOR
	GobiNet-objs += usbnet_3_0_6.o \
	            usbnet_2_6_35.o usbnet_3_10_21.o usbnet_3_12_xx.o usbnet_4_4_xx.o
endif

RAWIP := 0
ifeq ($(RAWIP), 1)
	ccflags-y:=-DDATA_MODE_RP
endif

PI_KDIR := ~/k/linux-rpi-3.6.y
PI_CCPREFIX=~/toolchain/rpi/tools-master/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-gnueabi-

OW_KDIR := ~/openwrt/trunk/build_dir/target-mips_r2_uClibc-0.9.33.2/linux-ar71xx_generic/linux-3.8.11
OW_CCPREFIX=~/openwrt/trunk/staging_dir/toolchain-mips_r2_gcc-4.6-linaro_uClibc-0.9.33.2/bin/mips-openwrt-linux-

MARVELL_KDIR := ~/toolchain/qmi_mxwell/kernel-2.6.31
MARVELL_CCPREFIX := ~/toolchain/qmi_mxwell/toolchain/bin/arm-none-linux-gnueabi-

M64_KDIR := ~/toolchain/arm64bit/linux/linux-4.4.1-devel-16.04.1
M64_CCPREFIX := ~/toolchain/arm64bit/be/aarch64ebv8-marvell-linux-gnu-5.2.1_x86_64_20151117/bin/aarch64_be-marvell-linux-gnu-

PPC_KDIR := ~/toolchain/ppc/kernel_ppc
PPC_CCPREFIX :=~/toolchain/ppc/fsl/1.1/sysroots/i686-fslsdk-linux/usr/bin/ppc64e5500-fsl-linux/powerpc64-fsl-linux-

HIKEY9_KDIR := /farm/android/Hikey_9.0_Trunk/kernel/hikey-linaro
HIKEY9_CCPREFIX := /farm/android/Hikey_9.0_Trunk/prebuilts/gcc/linux-x86/aarch64/aarch64-linux-android-4.9/bin/aarch64-linux-android-

all:
	$(MAKE) -C $(KDIR) M=$(PWD) modules

m64:
	$(MAKE) ARCH=arm64 CROSS_COMPILE=${M64_CCPREFIX} -C $(M64_KDIR) M=$(PWD) modules

ppc:
	$(MAKE) ARCH=powerpc CROSS_COMPILE=${PPC_CCPREFIX} -C $(PPC_KDIR) M=$(PWD) modules

marvell:
	$(MAKE) ARCH=arm CROSS_COMPILE=${MARVELL_CCPREFIX} -C $(MARVELL_KDIR) M=$(PWD) modules

pi:
	$(MAKE) ARCH=arm CROSS_COMPILE=${PI_CCPREFIX} -C $(PI_KDIR) M=$(PWD) modules

ow:
	$(MAKE) ARCH=mips CROSS_COMPILE=${OW_CCPREFIX} -C $(OW_KDIR) M=$(PWD) modules

hikey9:
	$(MAKE) ARCH=arm64 CROSS_COMPILE=${HIKEY9_CCPREFIX} -C $(HIKEY9_KDIR) M=$(PWD) modules

install: all
	mkdir -p $(OUTPUTDIR)
	cp -f GobiNet.ko $(OUTPUTDIR)
	depmod

clean:
	rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions Module.* modules.order .cache.mk *.o.ur-safe

The only thing we are interested in here is the target. In this file, there is a target type called “pi.” That’s the one we want. Look deeper into this target and associated macros:

pi:
	$(MAKE) ARCH=arm CROSS_COMPILE=${PI_CCPREFIX} -C $(PI_KDIR) M=$(PWD) modules

Notice the CROSS_COMPILE is set to PI_CCPREFIX which is what you need to fix. In this example we have discussed about the tools we will use for the Pi Kernel compilation. The same tool will come handy here as well. Point the PI_CCPREFIX to your tool/cross compiler path as below:

PI_CCPREFIX=/home/vbhadra/rpi/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-gnueabi-

Other important macro is PI_KDIR. This points to the Kernel folder against which we are going to cross compile. Point this to your previously downloaded kernel source folder as below:

PI_KDIR := /home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1

I also modified the clean target as below:

clean:
	make -C $(PI_KDIR) M=$(PWD) clean

But this is not necessary. Save these changes and go back to the driver folder and issue the make clean command to start with a clean slate and then issue make pi command as below:

~/rpi/GobiNet$ make clean
 #rm -rf *.o *~ core .depend .*.cmd *.ko *.mod.c .tmp_versions Module.* modules.order .cache.mk *.o.ur-safe
 make -C /home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1 M=/home/vbhadra/rpi/GobiNet clean
 make[1]: Entering directory '/home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1'
 make[1]: Leaving directory '/home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1'
 vbhadra@vbhadra-8200-elit-convertable-microtower:~/rpi/GobiNet$ make pi
 make ARCH=arm CROSS_COMPILE=/home/vbhadra/rpi/tools/arm-bcm2708/arm-bcm2708-linux-gnueabi/bin/arm-bcm2708-linux-gnueabi-  -C /home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1 M=/home/vbhadra/rpi/GobiNet modules
 make[1]: Entering directory '/home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1'
 CC [M]  /home/vbhadra/rpi/GobiNet/GobiUSBNet.o
 In file included from ./include/net/sock.h:58:0,
 from ./include/net/inet_sock.h:26,
 from ./include/net/ip.h:31,
 from /home/vbhadra/rpi/GobiNet/GobiUSBNet.c:94:
 ./include/linux/memcontrol.h: In function 'task_in_memcg_oom':
 ./include/linux/memcontrol.h:546:2: warning: return makes integer from pointer without a cast [enabled by default]
 /home/vbhadra/rpi/GobiNet/GobiUSBNet.c: At top level:
 /home/vbhadra/rpi/GobiNet/GobiUSBNet.c:112:2: warning: #warning "Remove memcontrol.h task_in_memcg_oom warning"
 /home/vbhadra/rpi/GobiNet/GobiUSBNet.c:113:2: warning: #warning "Comment '#define MEMCG_NOT_FIX' above after fix applied."
 CC [M]  /home/vbhadra/rpi/GobiNet/QMIDevice.o
 CC [M]  /home/vbhadra/rpi/GobiNet/QMI.o
 LD [M]  /home/vbhadra/rpi/GobiNet/GobiNet.o
 Building modules, stage 2.
 MODPOST 1 modules
 CC      /home/vbhadra/rpi/GobiNet/GobiNet.mod.o
 LD [M]  /home/vbhadra/rpi/GobiNet/GobiNet.ko
 make[1]: Leaving directory '/home/vbhadra/Downloads/linux-raspberrypi-kernel_1.20190925-1'

Check if there is a new kernel module been compiled called GobiNet.ko (in your case your defined obj-m name). In this case the driver defined module name was GobiNet in the Makefile as below:

obj-m := GobiNet.o

So I will check if there is a GobiNet.ko been generated:

vbhadra@vbhadra-8200-elit-convertable-microtower:~/rpi/GobiNet$ ls -la GobiNet.ko
-rw-r--r-- 1 vbhadra vbhadra 217987 Feb 21 11:17 GobiNet.ko

Copy this .ko file in a sdcard and insert into a Pi. Copy the .ko file from the sdcard to home folder in Pi. Then do the below to insert it into the Pi Kernel:

pi@raspberrypi:~ $ sudo insmod GobiNet.ko
pi@raspberrypi:~ $ dmesg
[ 2526.062621] GobiNet: 2019-11-22/SWI_2.60
[ 2526.064306] usbcore: registered new interface driver GobiNet

As you can see now the cross compiled GobiNet driver is up and running in the Pi Kernel.

The other kernel driver for SieraWireless LTE EM7565 device is called GobiSerial. I have compiled this driver code in the same way. But when I try to insert this in the Pi kernel I get the below error:

pi@raspberrypi:~ $ sudo insmod GobiSerial.ko
insmod: ERROR: could not insert module GobiSerial.ko: Unknown symbol in module
pi@raspberrypi:~ $ dmesg
[   65.759262] GobiSerial: Unknown symbol usb_serial_generic_open (err -2)
[   65.759290] GobiSerial: Unknown symbol usb_serial_suspend (err -2)
[   65.759328] GobiSerial: Unknown symbol usb_serial_generic_resume (err -2)
[   65.759348] GobiSerial: Unknown symbol usb_serial_deregister_drivers (err -2)
[   65.759407] GobiSerial: Unknown symbol usb_serial_generic_write (err -2)
[   65.759427] GobiSerial: Unknown symbol usb_serial_register_drivers (err -2)

I resolved the problem in the next post here.