How to pipe one bash command output to another within a python script

I was recently working with something which required me to build a code base and flash it into the board (hw) and then once that is done open a serial console (where the hw is connected through USB) and issue further commands on the hw. Now, that is a regular job and I wrote a python script which does all the manual works of cleaning, building etc. But flashing the binary onto the hardware is where the problem started coming. The first time I flash the board there is no issue. However, once I opened a serial console, next time when I try to flash the binary the script finds the serial console to be already occupied.

BTW the flash command also uses the serial connection to flash the binary on the hardware. So the only solution seems to be killing the serial console program manually and then try to flash it again which works. To kill the serial console all you need to do is to find out the process id of the serial console program (like minicom or screen) and then use “kill -9 command” on that process id. To find that process id I do the below bash command:

$ ps -eaf | grep -w 'SCREEN'
vbhadra   7471  7470  0 13:10 ?        00:00:00 SCREEN /dev/ttyUSB0 115200
vbhadra   7473  6368  0 13:10 pts/0    00:00:00 grep --color=auto -w SCREEN

My serial console utility is called “SCREEN”. As you can see in the above code snippet that there is an instance of the SCREEN program is currently running in the system. Notice that the SCREEN program is occupying the USB serial port /dev/ttyUSB0. To release the serial console from SCREEN program I will do the below:

fuser -k /dev/ttyUSB0

To do the same in the script I need to do a bit of filtering on the above output lines. So what I am essentially doing manually is that I am checking the name of the program SCREEN which is the 8th field in the below output line:

vbhadra   7471  7470  0 13:10 ?        00:00:00 SCREEN /dev/ttyUSB0 115200

filed[1] – vbhadra (user name)
filed[2] – 7471 (parent pid)

filed[8] – SCREEN (name of the program I am after)

If the name of the program is SCREEN then I am looking at the next field i.e. /dev/ttyUSB0 and using that in the below command:

fuser -k /dev/ttyUSB0

To get the individual fields from the above output line we can use awk command as below:

ps -eaf | grep -w 'SCREEN' | awk '{print $8 " " $9}'

If you issue the above command on a console (and you have instance SCREEN program running in your system), you should see something similar to this:

$ ps -eaf | grep -w 'SCREEN' | awk '{print $8 " " $9}'
SCREEN /dev/ttyUSB0
grep --color=auto

The above output needs further parsing. But before we go into that the problem in hand is to be able to issue the above command from a python script.

To do this from python we can leverage the sub-process module of python and the input/output PIPEs of the sub-process module. Do the below steps in your python script:

$ python
Python 2.7.15+ (default, Oct  7 2019, 17:39:04)
[GCC 7.4.0] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> import subprocess as sp
>>> out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
>>>

In the above two line what we have done is a created a new object of the subprocess module, passed “ps -eaf” as our new subprocess to be launched, and we have connected the stdout (this belongs to the out subprocess which we have just created) to the sp.PIPE and the stderr (this also belongs to the out object) to the sp.PIPE.

Now the output of the command “ps -eaf” and the possible error from the “ps -eaf” command has been captured in our out object. How do we know that it has been captured? Let’s print the out object.

>>> print out
subprocess.Popen object at 0x7fe3e4b35090
>>>

Not quite what we wanted. Because the out itself is just a pointer to the subprocess.Popen object. To see the stdout and the stderr do the below:

>>> o, r = out.communicate()
>>>

I cannot elaborate much about the communicate() here. Simply put it returns a tuple which is stddata and the stderror back to the called. In the above code we have used a tuple (o, r) to capture the tuple returned by the communicate() method. The name (o, r) has no special significance, I could have used my name and surname for that matter as well.
Now you can print o and r which represents the stdout and the stderr (output and error) from the command “ps -eaf”:

>>> print o
UID        PID  PPID  C STIME TTY          TIME CMD
root         1     0  0 Nov01 ?        00:00:10 /sbin/init splash
root         2     0  0 Nov01 ?        00:00:00 [kthreadd]
root         3     2  0 Nov01 ?        00:00:00 [rcu_gp]
root         4     2  0 Nov01 ?        00:00:00 [rcu_par_gp]
...
...
>>> print r 

>>>

Now, the next challenge is to connect the output of the “ps -eaf” to the grep utility to filter only the output lines containing SCREEN. To do that we can create another subprocess object and connect the output of the “ps -eaf” command i.e. out.stdout to the input i.e. stdin of the newly created subprocess object. Have a look at the below code snippet:

>>> output = sp.Popen(['grep', '-i', 'SCREEN'], stdin=out.stdout, stdout=sp.PIPE)
Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib/python2.7/subprocess.py", line 386, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python2.7/subprocess.py", line 812, in _get_handles
    p2cread = stdin.fileno()
ValueError: I/O operation on closed file
>>>

In the above code you can see the line, stdin=out.stdout. This means that the stdin (input) of the current object is connected to the stdout of the out object which we created previously. Also, stdout=sp.PIPE means that the stdout (output) of the current subprocess is connected to the PIPE of the subprocess command.
You might have noticed the above adventure was not quite successful as it tripped over a bunch of errors. And why is that? It is because of the below operation we did after creating the out object:

out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
o, r = out.communicate()

Once we use communicate() on the out object, the input/output files descriptors are closed and hence the error. The communicate was only for demonstration purposes. Now we will do the real stuff. Do the below again:

>>> out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
>>> output = sp.Popen(['grep', '-i', 'SCREEN'], stdin=out.stdout, stdout=sp.PIPE)
>>>

Now, the earlier error is gone. Now you can test it again as before if you like:

>>> out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
>>> output = sp.Popen(['grep', '-w', 'SCREEN'], stdin=out.stdout, stdout=sp.PIPE)
>>> o, e = output.communicate()
>>> print o
vbhadra   8151  8150  0 13:56 ?        00:00:00 SCREEN /dev/ttyUSB0 115200

>>> print e
None
>>>

As you can see, we have been able to filter only the line containing the SCREEN.
Now, ready to pipe this output to the awk command. Do the below as explained earlier:

>>> result = sp.Popen(['awk', '{print $8 " " $9}'], stdin=output.stdout, stdout=sp.PIPE)
Traceback (most recent call last):
  File "", line 1, in
  File "/usr/lib/python2.7/subprocess.py", line 386, in __init__
    errread, errwrite), to_close = self._get_handles(stdin, stdout, stderr)
  File "/usr/lib/python2.7/subprocess.py", line 812, in _get_handles
    p2cread = stdin.fileno()
ValueError: I/O operation on closed file
>>>

Same problem again. As we have already operated communicate() the I/O error above is coming. So do the below all over again:

>>> out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
>>> output = sp.Popen(['grep', '-w', 'SCREEN'], stdin=out.stdout, stdout=sp.PIPE)
>>> result = sp.Popen(['awk', '{print $8 " " $9}'], stdin=output.stdout, stdout=sp.PIPE)
>>> r, e = result.communicate()
>>> print r
SCREEN /dev/ttyUSB0

>>>
>>>

As you can we have almost stripped the whole output except the interesting parts for our purposes. Now we are ready to parse the final output result for just grabbing the serial terminal string only. Do the below:

>>> import subprocess as sp
>>> out = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
>>> output = sp.Popen(['grep', '-w', 'SCREEN'], stdin=out.stdout, stdout=sp.PIPE)
>>> result = sp.Popen(['awk', '{print $8 " " $9}'], stdin=output.stdout, stdout=sp.PIPE)
>>> o, e = result.communicate()
>>> str = o.split()
>>> if str[0] == "SCREEN":
...     print str[1]
...
/dev/ttyUSB0
>>>

Now in the above code snippet from the python shell, you can see we have sucessfully extracted the serial port which we would like to release, /dev/ttyUSB0. Now to release we will replace the print statement above as below:

>>> if str[0] == "SCREEN":
...     ret = sp.check_call(['fuser', '-k', str[1]])
...
/dev/ttyUSB0:         8837
>>>

All of these can be embedded in a single script as below:

#!/usr/bin/env python

import subprocess as sp

def grep_and_return():
    o = sp.Popen(['ps', '-eaf'], stdout=sp.PIPE, stderr=sp.PIPE)
    out = sp.Popen(['grep', '-w', 'SCREEN'], stdin=o.stdout, stdout=sp.PIPE)
    output = sp.Popen(['awk', '{print $8 " " $9}'], stdin=out.stdout, stdout=sp.PIPE)
    r, e = output.communicate()
    ret = r.split()
    if ret[0] == "SCREEN":
        print ret[1]
        #output = sp.check_call(['fuser', '-k', ret[1]])

def main():
    ret = grep_and_return()

if __name__ == "__main__":
    main()

The above code can be downloaded from github here.

For any question please post below!

Have fun scripting!

 

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out /  Change )

Google photo

You are commenting using your Google account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s