Unknown symbol usb_serial* in GobiSerial LTE Linux Kernel driver

I was working with the SieraWireless LTE device drivers recently and I got into a strange issue with one of the drivers called GobiSerial. I have compiled this driver code in the same way as described here. But when I try to insert this in the Pi kernel I get the below error:

pi@raspberrypi:~ $ sudo insmod GobiSerial.ko
insmod: ERROR: could not insert module GobiSerial.ko: Unknown symbol in module
pi@raspberrypi:~ $ dmesg
[ 65.759262] GobiSerial: Unknown symbol usb_serial_generic_open (err -2)
[ 65.759290] GobiSerial: Unknown symbol usb_serial_suspend (err -2)
[ 65.759328] GobiSerial: Unknown symbol usb_serial_generic_resume (err -2)
[ 65.759348] GobiSerial: Unknown symbol usb_serial_deregister_drivers (err -2)
[ 65.759407] GobiSerial: Unknown symbol usb_serial_generic_write (err -2)
[ 65.759427] GobiSerial: Unknown symbol usb_serial_register_drivers (err -2)

To debug this issue first I would like to know if the symbols like usb_serial_generic_open exists in the Pi kernel or not. Apparently not, but just to verify that I would use the /proc/kallsyms tool provided by Linux Kernel. This utility prints all the symbols (exported) available in the kernel space. So I would try to grep the missing symbols from all the avialable symbols in the kernel like this:

pi@raspberrypi:~ $ cat /proc/kallsyms | grep -i "usb_serial_generic_open"
pi@raspberrypi:~ $

In the above command nothing is returned indicating my hunch the symbols are not present (exported/available) in the Pi Kernel.
Now, by the look of it the symbols should be coming from a Kernel module/driver usb_serial, this is my hunch by the look of the symbols which are missing. My next hunch is some Kernel module are not inserted unless there is a reason for it. Like if you haven’t inserted anything on the USB serial the usb_serial driver won’t be inserted into the kernel. Let’s try this. I will attach the SieraWireless LTE device into the USB port of the Pi. Before I do this I usually clean my Kernel messages buffer as below:

sudo dmesg -c

I do this so that when I insert the device I can only focus on the kernel messages coming out for this device. Now insert the device, wait for a second and then try the dmesg again, and you should be able to see something like this on the console:

[ 252.661876] usb 1-1.3: reset low-speed USB device number 4 using dwc_otg
[ 253.511884] usb 1-1.5: reset low-speed USB device number 5 using dwc_otg
[ 261.551829] usb 1-1.2: new high-speed USB device number 8 using dwc_otg
[ 261.682434] usb 1-1.2: config 1 has an invalid interface number: 8 but max is 3
[ 261.682451] usb 1-1.2: config 1 has no interface number 1
[ 261.683111] usb 1-1.2: New USB device found, idVendor=1199, idProduct=9091, bcdDevice= 0.06
[ 261.683123] usb 1-1.2: New USB device strings: Mfr=1, Product=2, SerialNumber=3
[ 261.683133] usb 1-1.2: Product: Sierra Wireless EM7565 Qualcomm® Snapdragon™ X16 LTE-A
[ 261.683143] usb 1-1.2: Manufacturer: Sierra Wireless, Incorporated
[ 261.683153] usb 1-1.2: SerialNumber: UF92377624041506
[ 261.686019] QMAP Disabled
[ 261.688132] GobiNet 1-1.2:1.8 eth1: register 'GobiNet' at usb-3f980000.usb-1.2, GobiNet Ethernet Device, d2:bd:e1:49:f1:08
[ 261.689996] USB Speed : USB 2.0
[ 261.805791] usbcore: registered new interface driver usbserial_generic
[ 261.805832] usbserial: USB Serial support registered for generic
[ 261.806590] usbcore: registered new interface driver cdc_wdm
[ 261.814689] usbcore: registered new interface driver qcserial
[ 261.814756] usbserial: USB Serial support registered for Qualcomm USB modem
[ 261.815707] qcserial 1-1.2:1.0: Qualcomm USB modem converter detected
[ 261.816223] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB0
[ 261.819226] qcserial 1-1.2:1.2: Qualcomm USB modem converter detected
[ 261.819635] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB1
[ 261.819970] qcserial 1-1.2:1.3: Qualcomm USB modem converter detected
[ 261.820236] usb 1-1.2: Qualcomm USB modem converter now attached to ttyUSB2
[ 261.820411] usbcore: registered new interface driver qmi_wwan
[ 261.920268] IPv6: ADDRCONF(NETDEV_UP): eth1: link is not ready

Looks like the default linux kernel serial drivers been just inserted. Curious huh? Now, check if our missing symbol is back in the Kernel or not as below:

pi@raspberrypi:~ $ cat /proc/kallsyms | grep -i "usb_serial_generic_open"
00000000 r __ksymtab_usb_serial_generic_open [usbserial]
00000000 r __kstrtab_usb_serial_generic_open [usbserial]
00000000 t usb_serial_generic_open [usbserial]

Hurray!! It is back. Now try inserting the GobiSerial driver module again:

pi@raspberrypi:~ $ sudo insmod GobiSerial.ko
pi@raspberrypi:~ $

Seems like the previous error didn’t appear. Lets check the dmesg again, and here we go:

[ 326.647915] usbserial: USB Serial support registered for GobiSerial
[ 326.647943] GobiSerial: 2019-11-22/SWI_2.39:GobiSerial

Issue resolved.
But soon I realized that attaching the device to the RPI hw is actually insmod-ing the qcserial and the qmi_wwan kernel modules which are Qualcomm drivers. But if you are dealing with the device EM7565, in the release notes it prohibits to have qcserial and qmi_wwan drivers in the kernel and prescribes to blacklist these in the below file:

$ sudo vim /etc/modprobe.d/blacklist-modem.conf
# Uncomment these entries in order to blacklist unwanted modem drivers
# blacklist snd-atiixp-modem
# blacklist snd-intel8x0m
# blacklist snd-via82xx-modem
blacklist qcserial
blacklist qmi_wwan

If you do that these two drivers won’t be loaded when you attach your EM7565 device to the USB port. So, once I do that and restart the system my above resolution to the problem stops working again :(.

Actually qcserial driver is loading the usbserial and once you take the qcserial out of the equation, it all stops working again.

So finally what I did was to:

sudo insmod /lib/modules/4.19.75-v7+/kernel/drivers/usb/serial/usbserial.ko
sudo insmod GobiSerial.ko

and it works fine.

This works but not yet the best way to sort this problem. To sort this problem in the cleanest possible way would be to install the GobiNet and GobiSerial and then do a demod to resolve the dendencies. Follow the below steps for this:

$ export OUTPUTDIR=/lib/modules/`uname -r`/kernel/drivers/net/usb/
$ mkdir -p $OUTPUTDIR
$ cp -f GobiNet.ko $(OUTPUTDIR)
$ cp -f GobiSerial.ko $(OUTPUTDIR)
$ depmod
$ modprobe GobiNet
$ modprobe GobiSerial

Now, restart the system, and you should have everything in place.
To be sure do the below:

pi@raspberrypi:~ $ lsmod | grep -i "Gobi*"
GobiSerial 20480 1
usbserial 40960 3 GobiSerial
GobiNet 438272 0

When we do depmod, it resolves the dependancies of the various modules in the Kernel /lib/modules/ folder. Then at the time of modprobe the dependecy drivers/modules are loaded first and then the actual module of interest.

How to integrate a custom device driver into nuttx tree

I had written a blog on how to integrate a custom driver into Linux kernel tree previously here. Integrating a custom driver in nuttx kernel is pretty similar. But writing this down would probably be a good idea for any future reference. So below are the steps to follow to integrate a custom driver in nuttx.

Read More »

How to create and run cppcheck executable from source code in linux machine (C++11 Static Analysis)

I do not have administrative rights on a linux server (openSuse Linux) I am working on right now. So installing anything is not an option or at least an easy option. But I wanted to run cppcheck on my new C++11 source code. Here is what I found to achieve that:

Read More »

How to integrate a custom device driver into kernel tree

Find out where you want to place the device driver source code. In my case I had to integrate a device driver for a device called ring oscillator. The category of the device was not clear to me and I couldn’t find a suitable place under “linux/drivers” folder except “linux/drivers/misc”. So I decided to place my driver source code under linux/drivers/misc. You may decide to place it where you think  it is appropriate. It is up to to decide.

Read More »

How to run an external program from a python script

This blog will try to demonstrate how a python script can be used to automate the running of an external program with data input file located at a particular location.

Lets say we have the below c++ program which reads from a data file called myData.txt located for this example at user home i.e. ~/. The user home is usually /home/<username> for example /home/vbhadra in my case. Use your linux user name instead while trying it yourself. To find out what is your home directory you can use echo $HOME in linux command prompt. Our objective in this blog is to demonstrate the pythoin script which will run any external program binary with data input. So we will have a less focus on the c++ program.

Read More »