How to replace the Kernel image in NOOBS with your custom Kernel

In one of the previous posts I have described Bringing up Rasberry Pi 3 with NOOBS. But after I brought up the Raspberry Pi (ver 3 B) with NOOBS I started having some difficultly with the NOOBS kernel version. Some of the Linux drivers I was working with works only with particular version of Linux Kernel and hence it became necessary for me to figure out how to replace the NOOBS kernel image with my own built shiny cross compiled image, so that I have the flexibility of using any version of Kernel I want. I think it could of use for you as well and hence writing the steps up here so that I can refer it later and also maybe someone out there finds it helpful. The original instructions are based out here. The rest of the post is based on Raspberry Pi version 3 B which is what I am working on right now. So the instructions are valid and tested for version 3. It may vary in other Pi hardware version, but the basic steps would remain by and large same.

Bring up Pi with NOOBS

Follow this link to first bring up your Raspberry Pi with NOOBS as described here.

Cross Compile the Kernel

Required utilities

Install the below software in your linux distribution. I am using Ubuntu hence the command to install the dependency software are as below.

$ sudo apt install git bc bison flex libssl-dev make
$ sudo apt install git bc bison flex libssl-dev make libc6-dev libncurses5-dev

Get Linux Source

git clone --depth=1

Get the toolchain

git clone ~/tools

Set the tool PATH

$ echo PATH=\$PATH:~/tools/arm-bcm2708/gcc-linaro-arm-linux-gnueabihf-raspbian/bin >> ~/.bashrc
$ source ~/.bashrc

Cross compile the kernel

cd linux/

Generate the .config file

$ make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- bcm2709_defconfig


make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- zImage modules dtbs -j 12

Prepare the SD Card from Pi

Now, you will need the SD card where you have installed NOOBS. One of the mistakes I was doing is thinking installing NOOBS means copying the extracted NOOBS files into the SD card. It is stupid I know, but instructions may not be always very clear (or your head may not be working at all on a day). So make sure you actually have followed all the steps described here. What it will ensure is that the NOOBS actually creates all the partitions required in the SD card and then copy the right files into the right partitions. Once that is done the installation is complete. This is done by NOOBS right after you do the following steps as described in the above link:

  1. Copy the extracted NOOBS files into the sd card
  2. Inserted the SD card into the Rasberry Pi (I am using version 3 B).
  3. Powered up the Pi and be patient when NOOBS does the neccessary things for your to install the Rasbian.

Generally if you have followed all the steps described in the post Bringing up Rasberry Pi 3 with NOOBS then all you need to do is to eject the SD card from the Pi and insert it into you local linux machine (where you are doing the cross compilation).

Locate the SD card in Host

Once you have attached the SD card to you local PC (picture below in case of any confusion), go to the console (I am using a Ubuntu PC).

I use the above card reader, choose you own, it doesn’t matter but essentially, you would probably need a card reader. Now, in you console issue the below command to find out where your SD card has gone:

sda 8:0 0 465.8G 0 disk
├─sda1 8:1 0 512M 0 part /boot/efi
└─sda2 8:2 0 465.3G 0 part /
sdc 8:32 1 29.7G 0 disk
├─sdc1 8:33 1 2.4G 0 part
├─sdc2 8:34 1 1K 0 part
├─sdc5 8:37 1 32M 0 part /media/vbhadra/SETTINGS
├─sdc6 8:38 1 256M 0 part /media/vbhadra/boot1
└─sdc7 8:39 1 27G 0 part /media/vbhadra/root
sr0 11:0 1 1024M 0 rom

It looks like as above (just the interesting bits copy pasted) after issuing lsblk command on my Ubuntu console.

Now, sda is your hard drive. So leave that alone. The rest is sdc which is the SD card media in this case. Under sdc you will see the 5 partitions that NOOBS has created for us. The most important partitions for us are sdc6 and sdc7. SDC6 is FAT partition called boot and SDC7 is an ext4 type partitions called root.

Mount SD card partitions on Host Linux

We need to install the newly built kernel modules to the SD card partitions. Also, we need to copy paste other stuffs to the SD card partitions. So for that purpose we are up to mounting these partitions sdc6 and sdc7 into suitable folders on the host PC (Ubuntu in my case). To do that follow the below steps on the host PC console:

~/temp/linux$ mkdir mnt
~/temp/linux$ mkdir mnt/fat32
~/temp/linux$ mkdir mnt/ext4
~/temp/linux$ sudo mount /dev/sdb6 mnt/fat32
~/temp/linux$ sudo mount /dev/sdc6 mnt/fat32
~/temp/linux$ sudo mount /dev/sdc7 mnt/ext4 

~/temp/linux$ ls mnt/
ext4/  fat32/ 

~/temp/linux$ ls mnt/fat32/
bcm2708-rpi-b.dtb       bcm2710-rpi-2-b.dtb       bcm2835-rpi-a-plus.dtb   bcm2835-rpi-zero-w.dtb    config.txt     fixup_cd.dat        kernel7.img       overlays      start_db.elf
bcm2708-rpi-b-plus.dtb  bcm2710-rpi-3-b.dtb       bcm2835-rpi-b.dtb        bcm2836-rpi-2-b.dtb       COPYING.linux  fixup.dat           kernel7l.img      start4cd.elf  start.elf
bcm2708-rpi-cm.dtb      bcm2710-rpi-3-b-plus.dtb  bcm2835-rpi-b-plus.dtb   bcm2837-rpi-3-b.dtb       fixup4cd.dat   fixup_db.dat        kernel8.img       start4db.elf  start_x.elf
bcm2708-rpi-zero.dtb    bcm2710-rpi-cm3.dtb       bcm2835-rpi-b-rev2.dtb   bcm2837-rpi-3-b-plus.dtb  fixup4.dat     fixup_x.dat         kernel.img        start4.elf
bcm2708-rpi-zero-w.dtb  bcm2711-rpi-4-b.dtb       bcm2835-rpi-cm1-io1.dtb  bootcode.bin              fixup4db.dat   issue.txt           LICENCE.broadcom  start4x.elf
bcm2709-rpi-2-b.dtb     bcm2835-rpi-a.dtb         bcm2835-rpi-zero.dtb     cmdline.txt               fixup4x.dat    kernel7-backup.img  os_config.json    start_cd.elf

~/temp/linux$ ls mnt/ext4/
bin  boot  dev  etc  home  lib  lost+found  media  mnt  opt  proc  root  run  sbin  srv  sys  tmp  usr  var
~/temp/linux$ mount 
/dev/sdc6 on /media/vbhadra/boot1 type vfat (rw,nosuid,nodev,relatime,uid=1000,gid=1000,fmask=0022,dmask=0022,codepage=437,iocharset=iso8859-1,shortname=mixed,showexec,utf8,flush,errors=remount-ro,uhelper=udisks2)
/dev/sdc7 on /media/vbhadra/root type ext4 (rw,nosuid,nodev,relatime,uhelper=udisks2)

Install the newly build module into SD card

~/temp/linux$ sudo env PATH=$PATH make ARCH=arm CROSS_COMPILE=arm-linux-gnueabihf- INSTALL_MOD_PATH=mnt/ext4 modules_install

Copy the rest of the files into SD card

Copy the Kernel Image

~/temp/linux$ sudo cp mnt/fat32/$KERNEL.img mnt/fat32/$KERNEL-backup.img
~/temp/linux$ sudo cp arch/arm/boot/zImage mnt/fat32/$KERNEL.img

Copy the dtb file

~/temp/linux$ sudo cp arch/arm/boot/dts/*.dtb mnt/fat32/
~/temp/linux$ sudo cp arch/arm/boot/dts/overlays/*.dtb* mnt/fat32/overlays/
~/temp/linux$ sudo cp arch/arm/boot/dts/overlays/README mnt/fat32/overlays/

Umount the sd card partitions

~/temp/linux$ sudo umount mnt/fat32
~/temp/linux$ sudo umount mnt/ext4

Boot the Pi with the SD card

Take the SD card and insert it into the Pi back again. Then power up the Pi. The rest should be taken care by NOOBS. So wait and watch. You shouldn’t see any errors at all. In case of any error, please post it in the below comments section so that I can have a look. Once the Pi comes up open a console and check if your Pi is actually running your newly built kernel image. See the below section for that.

Verify the kernel version on Pi

pi@raspberrypi:~ $ uname -a 
Linux raspberrypi 4.19.106-v7+ #1 SMP Tue Mar 3 12:57:47 GMT 2020 armv7l GNU/Linux

As you might have already noticed the new kernel version is 4.19.106-v7+. Th earlier on my Pi was 4.19.75-v7+. Also, notice the time stamp. As it shows the time stamp in my Pi is showing today’s date and time, Tue Mar 3 12:57:47 GMT 2020, another proof that it is my cross built kernel which is running in Pi not the original NOOBS kernel which came with the NOOBS distribution.

All done, have fun!

Unknown symbol usb_serial* in GobiSerial LTE Linux Kernel driver

I was working with the SieraWireless LTE device drivers recently and I got into a strange issue with one of the drivers called GobiSerial. I have compiled this driver code in the same way as described here. But when I try to insert this in the Pi kernel I get the below error:

pi@raspberrypi:~ $ sudo insmod GobiSerial.ko
insmod: ERROR: could not insert module GobiSerial.ko: Unknown symbol in module
pi@raspberrypi:~ $ dmesg
[ 65.759262] GobiSerial: Unknown symbol usb_serial_generic_open (err -2)
[ 65.759290] GobiSerial: Unknown symbol usb_serial_suspend (err -2)
[ 65.759328] GobiSerial: Unknown symbol usb_serial_generic_resume (err -2)
[ 65.759348] GobiSerial: Unknown symbol usb_serial_deregister_drivers (err -2)
[ 65.759407] GobiSerial: Unknown symbol usb_serial_generic_write (err -2)
[ 65.759427] GobiSerial: Unknown symbol usb_serial_register_drivers (err -2)

To debug this issue first I would like to know if the symbols like usb_serial_generic_open exists in the Pi kernel or not. Apparently not, but just to verify that I would use the /proc/kallsyms tool provided by Linux Kernel. This utility prints all the symbols (exported) available in the kernel space. So I would try to grep the missing symbols from all the avialable symbols in the kernel like this:

pi@raspberrypi:~ $ cat /proc/kallsyms | grep -i "usb_serial_generic_open"
pi@raspberrypi:~ $

In the above command nothing is returned indicating my hunch the symbols are not present (exported/available) in the Pi Kernel.
Now, by the look of it the symbols should be coming from a Kernel module/driver usb_serial, this is my hunch by the look of the symbols which are missing. My next hunch is some Kernel module are not inserted unless there is a reason for it. Like if you haven’t inserted anything on the USB serial the usb_serial driver won’t be inserted into the kernel. Let’s try this. I will attach the SieraWireless LTE device into the USB port of the Pi. Before I do this I usually clean my Kernel messages buffer as below:

sudo dmesg -c

I do this so that when I insert the device I can only focus on the kernel messages coming out for this device. Now insert the device, wait for a second and then try the dmesg again, and you should be able to see something like this on the console:

[ 252.661876] usb 1-1.3: reset low-speed USB device number 4 using dwc_otg
[ 253.511884] usb 1-1.5: reset low-speed USB device number 5 using dwc_otg
[ 261.551829] usb 1-1.2: new high-speed USB device number 8 using dwc_otg
[ 261.682434] usb 1-1.2: config 1 has an invalid interface number: 8 but max is 3
[ 261.682451] usb 1-1.2: config 1 has no interface number 1
[ 261.683111] usb 1-1.2: New USB device found, idVendor=1199, idProduct=9091, bcdDevice= 0.06
[ 261.683123] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 261.683133] usb 1-1.2: Product: Sierra Wireless EM7565 Qualcomm® Snapdragon™ X16 LTE-A
[ 261.683143] usb 1-1.2: Manufacturer: Sierra Wireless, Incorporated
[ 261.683153] usb 1-1.2: SerialNumber: UF92377624041506
[ 261.686019] QMAP Disabled
[ 261.688132] GobiNet 1-1.2:1.8 eth1: register 'GobiNet' at usb-3f980000.usb-1.2, GobiNet Ethernet Device, d2:bd:e1:49:f1:08
[ 261.689996] USB Speed : USB 2.0
[ 261.805791] usbcore: registered new interface driver usbserial_generic
[ 261.805832] usbserial: USB Serial support registered for generic
[ 261.806590] usbcore: registered new interface driver cdc_wdm
[ 261.814689] usbcore: registered new interface driver qcserial
[ 261.814756] usbserial: USB Serial support registered for Qualcomm USB modem
[ 261.815707] qcserial 1-1.2:1.0: Qualcomm USB modem converter detected
[ 261.816223] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB0
[ 261.819226] qcserial 1-1.2:1.2: Qualcomm USB modem converter detected
[ 261.819635] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB1
[ 261.819970] qcserial 1-1.2:1.3: Qualcomm USB modem converter detected
[ 261.820236] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB2
[ 261.820411] usbcore: registered new interface driver qmi_wwan
[ 261.920268] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready

Looks like the default linux kernel serial drivers been just inserted. Curious huh? Now, check if our missing symbol is back in the Kernel or not as below:

pi@raspberrypi:~ $ cat /proc/kallsyms | grep -i "usb_serial_generic_open"
00000000 r __ksymtab_usb_serial_generic_open [usbserial]
00000000 r __kstrtab_usb_serial_generic_open [usbserial]
00000000 t usb_serial_generic_open [usbserial]

Hurray!! It is back. Now try inserting the GobiSerial driver module again:

pi@raspberrypi:~ $ sudo insmod GobiSerial.ko
pi@raspberrypi:~ $

Seems like the previous error didn’t appear. Lets check the dmesg again, and here we go:

[ 326.647915] usbserial: USB Serial support registered for GobiSerial
[ 326.647943] GobiSerial: 2019-11-22/SWI_2.39:GobiSerial

Issue resolved.
But soon I realized that attaching the device to the RPI hw is actually insmod-ing the qcserial and the qmi_wwan kernel modules which are Qualcomm drivers. But if you are dealing with the device EM7565, in the release notes it prohibits to have qcserial and qmi_wwan drivers in the kernel and prescribes to blacklist these in the below file:

$ sudo vim /etc/modprobe.d/blacklist-modem.conf
# Uncomment these entries in order to blacklist unwanted modem drivers
# blacklist snd-atiixp-modem
# blacklist snd-intel8x0m
# blacklist snd-via82xx-modem
blacklist qcserial
blacklist qmi_wwan

If you do that these two drivers won’t be loaded when you attach your EM7565 device to the USB port. So, once I do that and restart the system my above resolution to the problem stops working again :(.

Actually qcserial driver is loading the usbserial and once you take the qcserial out of the equation, it all stops working again.

So finally what I did was to:

sudo insmod /lib/modules/4.19.75-v7+/kernel/drivers/usb/serial/usbserial.ko
sudo insmod GobiSerial.ko

and it works fine.

This works but not yet the best way to sort this problem. To sort this problem in the cleanest possible way would be to install the GobiNet and GobiSerial and then do a demod to resolve the dendencies. Follow the below steps for this:

$ export OUTPUTDIR=/lib/modules/`uname -r`/kernel/drivers/net/usb/
$ mkdir -p $OUTPUTDIR
$ cp -f GobiNet.ko $(OUTPUTDIR)
$ cp -f GobiSerial.ko $(OUTPUTDIR)
$ depmod
$ modprobe GobiNet
$ modprobe GobiSerial

Now, restart the system, and you should have everything in place.
To be sure do the below:

pi@raspberrypi:~ $ lsmod | grep -i "Gobi*"
GobiSerial 20480 1
usbserial 40960 3 GobiSerial
GobiNet 438272 0

When we do depmod, it resolves the dependancies of the various modules in the Kernel /lib/modules/ folder. Then at the time of modprobe the dependecy drivers/modules are loaded first and then the actual module of interest.