6/22/2020 Executing Shell Commands with Python

Executing Shell Commands

with Python
By Sajjad Heydari •
2 Comments (/executing-shell-commands-with-python/#disqus_thread)

Introduction
Repetitive tasks are ripe for automation. It is common for developers and system
administrators to automate routine tasks like health checks and le backups with
shell scripts. However, as those tasks become more complex, shell scripts may
become harder to maintain.

Fortunately, we can use Python instead of shell scripts for automation. Python
provides methods to run shell commands, giving us the same functionality of those
shells scripts. Learning how to run shell commands in Python opens the door for us
to automate computer tasks in a structured and scalable way.

In this article, we will look at the various ways to execute shell commands in
Python, and the ideal situation to use each method.

Using os.system to Run a Command

Python allows us to immediately execute a shell command that's stored in a string
using the os.system() function.

Let's start by creating a new Python le called echo_adelle.py and enter the
following:

import os

os.system("echo Hello from the other side!")

https://stackabuse.com/executing-shell-commands-with-python/ 1/12
6/22/2020 Executing Shell Commands with Python

The rst thing we do in our Python le is import the os module, which contains
the system function that can execute shell commands. The next line does exactly
that, runs the echo command in our shell through Python.

In your Terminal, run this le with using the following command, and you should
see the corresponding output:

$ python3 echo_adelle.py
Hello from the other side!

As the echo commands prints to our stdout , os.system() also displays the
output on our stdout stream. While not visible in the console, the os.system()
command returns the exit code of the shell command. An exit code of 0 means it
ran without any problems and any other number means an error.

Let's create a new le called cd_return_codes.py and type the following:

import os

home_dir = os.system("cd ~")

print("`cd ~` ran with exit code %d" % home_dir)
unknown_dir = os.system("cd doesnotexist")
print("`cd doesnotexis` ran with exit code %d" % unknown_dir)

In this script, we create two variables that store the result of executing commands
that change the directory to the home folder, and to a folder that does not exist.
Running this le, we will see:

$ python3 cd_return_codes.py
`cd ~` ran with exit code 0
sh: line 0: cd: doesnotexist: No such file or directory
`cd doesnotexist` ran with exit code 256

The rst command, which changes the directory to the home directory, executes
successfully. Therefore, os.system() returns its exit code, zero, which is stored in
home_dir . On the other hand, unknown_dir stores the exit code of the failed
bash command to change the directory to a folder that does not exist.

https://stackabuse.com/executing-shell-commands-with-python/ 2/12
6/22/2020 Executing Shell Commands with Python

The os.system() function executes a command, prints any output of the

command to the console, and returns the exit code of the command. If we would
like more ne grained control of a shell command's input and output in Python, we
should use the subprocess module.

Running a Command with subprocess

The subprocess module (https://docs.python.org/3/library/subprocess.html) is
Python's recommended way to executing shell commands. It gives us the exibility
to suppress the output of shell commands or chain inputs and outputs of various
commands together, while still providing a similar experience to os.system() for
basic use cases.

In a new led called list_subprocess.py , write the following code:

import subprocess

list_files = subprocess.run(["ls", "-l"])

print("The exit code was: %d" % list_files.returncode)

In the rst line, we import the subprocess module, which is part of the Python
standard library. We then use the subprocess.run() function to execute the
command. Like os.system() , the subprocess.run() command returns the exit
code of what was executed.

Unlike os.system() , note how subprocess.run() requires a list of strings as

input instead of a single string. The rst item of the list is the name of the
command. The remaining items of the list are the ags and the arguments of the
command.

Note: As a rule of thumb, you need to separate the arguments based on space, for
example ls -alh would be ["ls", "-alh"] , while ls -a -l -h , would be
["ls", "-a", -"l", "-h"] . As another example, echo hello world would be

https://stackabuse.com/executing-shell-commands-with-python/ 3/12
6/22/2020 Executing Shell Commands with Python

["echo", "hello", "world"] , whereas echo "hello world" or echo hello\

world would be ["echo", "hello world"] .

Run this le and your console's output would be similar to:

$ python3 list_subprocess.py
total 80
-rw-r--r--@ 1 stackabuse staff 216 Dec 6 10:29 cd_return_codes.py
-rw-r--r--@ 1 stackabuse staff 56 Dec 6 10:11 echo_adelle.py
-rw-r--r--@ 1 stackabuse staff 116 Dec 6 11:20 list_subprocess.py
The exit code was: 0

Now let's try to use one of the more advanced features of subprocess.run() ,
namely ignore output to stdout . In the same list_subprocess.py le, change:

list_files = subprocess.run(["ls", "-l"])

To this:

list_files = subprocess.run(["ls", "-l"], stdout=subprocess.DEVNULL)

The standard output of the command now pipes to the special /dev/null device,
which means the output would not appear on our consoles. Execute the le in your
shell to see the following output:

$ python3 list_subprocess.py
The exit code was: 0

Subscribe to our Newsletter

Get occassional tutorials, guides, and jobs in your inbox. No spam ever.
Unsubscribe at any time.

Enter your email...

Subscribe

https://stackabuse.com/executing-shell-commands-with-python/ 4/12
6/22/2020 Executing Shell Commands with Python

What if we wanted to provide input to a command? The subprocess.run()

facilitates this by its input argument. Create a new le called
cat_subprocess.py , typing the following:

import subprocess

useless_cat_call = subprocess.run(["cat"], stdout=subprocess.PIPE, text=True, input

="Hello from the other side")
print(useless_cat_call.stdout) # Hello from the other side

We use subprocess.run() with quite a few commands, let's go through them:

stdout=subprocess.PIPE tells Python to redirect the output of the

command to an object so it can be manually read later
text=True returns stdout and stderr as strings. The default return type
is bytes.
input="Hello from the other side" tells Python to add the string as
input to the cat command.

Running this le produces the following output:

Hello from the other side

We can also raise an Exception without manually checking the return value. In a
new le, false_subprocess.py , add the code below:

import subprocess

failed_command = subprocess.run(["false"], check=True)

print("The exit code was: %d" % failed_command.returncode)

In your Terminal, run this le. You will see the following error:

https://stackabuse.com/executing-shell-commands-with-python/ 5/12
6/22/2020 Executing Shell Commands with Python

$ python3 false_subprocess.py
Traceback (most recent call last):
File "false_subprocess.py", line 4, in <module>
failed_command = subprocess.run(["false"], check=True)
File "/usr/local/python/3.7.5/Frameworks/Python.framework/Versions/3.7/lib/python
3.7/subprocess.py", line 512, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command '['false']' returned non-zero exit status 1.

By using check=True , we tell Python to raise any exceptions if an error is

encountered. Since we did encounter an error, the print statement on the nal
line was not executed.

The subprocess.run() function gives us immense exibility that os.system()

doesn't when executing shell commands. This function is a simpli ed abstraction
of the subprocess.Popen class, which provides additional functionality we can
explore.

Running a Command with Popen

The subprocess.Popen class exposes more options to the developer when
interacting with the shell. However, we need to be more explicit about retrieving
results and errors.

By default, subprocess.Popen does not stop processing of a Python program if

its command has not nished executing. In a new le called list_popen.py , type
the following:

import subprocess

list_dir = subprocess.Popen(["ls", "-l"])

list_dir.wait()

This code is equivalent to that of list_subprocess.py . It runs a command using

subprocess.Popen , and waits for it to complete before executing the rest of the
Python script.

https://stackabuse.com/executing-shell-commands-with-python/ 6/12
6/22/2020 Executing Shell Commands with Python

Let's say we do not want to wait for our shell command to complete execution so
the program can work on other things. How would it know when the shell
command has nished execution?

The poll() method returns the exit code if a command has nished running, or
None if it's still executing. For example, if we wanted to check if list_dir was
complete instead of wait for it, we would have the following line of code:

list_dir.poll()

To manage input and output with subprocess.Popen , we need to use the

communicate() method.

In a new le called cat_popen.py , add the following code snippet:

import subprocess

useless_cat_call = subprocess.Popen(["cat"], stdin=subprocess.PIPE, stdout=subproce

ss.PIPE, stderr=subprocess.PIPE, text=True)
output, errors = useless_cat_call.communicate(input="Hello from the other side!")
useless_cat_call.wait()
print(output)
print(errors)

The communicate() method takes an input argument that's used to pass input
to the shell command. The communicate method also returns both the stdout
and stderr when they are set.

Having seen the core ideas behind subprocess.Popen , we have now covered
three ways to run shell commands in Python. Let's re-examine their characteristics
so we'll know which method is best suited for a project's requirements.

Which one should I use?

If you need to run one or a few simple commands and do not mind if their output
goes to the console, you can use the os.system() command. If you want to
manage the input and output of a shell command, use subprocess.run() . If you
https://stackabuse.com/executing-shell-commands-with-python/ 7/12
6/22/2020 Executing Shell Commands with Python

want to run a command and continue doing other work while it's being executed,
use subprocess.Popen .

Here is a table with some usability differences that you can also use to inform your
decision:

os.system subprocess.run subprocess.Popen

Requires parsed arguments no yes yes

Waits for the command yes yes no

Communicates with stdin and stdout no yes yes

Returns return value object object

Conclusion
Python allows you to execute shell commands, which you can use to start other
programs or better manage shell scripts that you use for automation. Depending on
our use case, we can use os.system() , subprocess.run() or
subprocess.Popen to run bash commands.

Using these techniques, what external task would you run via Python?

 python (/tag/python/), shell (/tag/shell/), bash (/tag/bash/)

ter.com/share?
20Shell%20Commands%20with%20Python&url=https://stackabuse.com/executing-
-with-python/)
w.facebook.com/sharer/sharer.php?u=https://stackabuse.com/executing-shell-
-python/)
w.linkedin.com/shareArticle?mini=true%26url=https://stackabuse.com/executing-
-with-python/%26source=https://stackabuse.com)

(/author/sajjad/)

https://stackabuse.com/executing-shell-commands-with-python/ 8/12
6/22/2020 Executing Shell Commands with Python

About Sajjad Heydari (/author/sajjad/)

Subscribe to our Newsletter

Get occassional tutorials, guides, and jobs in your inbox. No spam ever. Unsubscribe
at any time.

Enter your email...

Subscribe

 Previous Post (/creating-command-line-utilities-with-pythons-argparse/)  

Next Post  (/encoding-and-decoding-base64-strings-in-python/)

Ad

https://stackabuse.com/executing-shell-commands-with-python/ 9/12
6/22/2020 Executing Shell Commands with Python

Follow Us

  
(https://twitter.com/StackAbuse) (https://www.facebook.com/stackabuse)
(https://stackabuse.com/rss/)

Newsletter

Subscribe to our newsletter! Get occassional tutorials, guides, and reviews in your
inbox.

Enter your email...

Subscribe

No spam ever. Unsubscribe at any time.

Ad

Want a remote job?

https://stackabuse.com/executing-shell-commands-with-python/ 10/12
6/22/2020 Executing Shell Commands with Python

Quant Trading Algorithm Developer

Binance 3 hours ago (https://hireremote.io/remote-job/2099-quant-trading-algorithm-developer-at-
binance)

python (https://hireremote.io/remote-python-jobs) blockchain (https://hireremote.io/remote-

blockchain-jobs) nance (https://hireremote.io/remote- nance-jobs) cpp

(https://hireremote.io/remote-cpp-jobs)

Software Engineer
Skylight 1 day ago (https://hireremote.io/remote-job/2097-software-engineer-at-skylight)

react-js (https://hireremote.io/remote-react-js-jobs) python (https://hireremote.io/remote-

python-jobs) ruby-on-rails (https://hireremote.io/remote-ruby-on-rails-jobs) healthcare

(https://hireremote.io/remote-healthcare-jobs)

Data Analyst
The Democratic Party 2 days ago (https://hireremote.io/remote-job/2087-data-analyst-at-the-
democratic-party)

python (https://hireremote.io/remote-python-jobs) data science (https://hireremote.io/remote-

data-science-jobs)

  More jobs (https://hireremote.io)

Jobs via HireRemote.io (https://hireremote.io)

Prepping for an interview?

(https://stackabu.se/daily-coding-problem)

Improve your skills by solving one coding problem every day

Get the solutions the next morning via email

Practice on actual problems asked by top companies, like:

     

   Daily Coding Problem (https://stackabu.se/daily-coding-problem)

Ad

https://stackabuse.com/executing-shell-commands-with-python/ 11/12
6/22/2020 Executing Shell Commands with Python

Recent Posts

ES6 Template/String Literals in Node.js (/es6-template-string-literals-in-nodejs/)

Calculating Mean, Median and Mode in Python (/calculating-mean-median-and-mode-in-python/)

Getting Started with Thymeleaf in Java and Spring (/getting-started-with-thymeleaf-in-java-and-

spring/)

Tags

algorithms (/tag/algorithms/) amqp (/tag/amqp/) angular (/tag/angular/)

announcements (/tag/announcements/) apache (/tag/apache/) api (/tag/api/)

arduino (/tag/arduino/) arti cial intelligence (/tag/arti cial-intelligence/)

asynchronous (/tag/asynchronous/) aws (/tag/aws/)

Follow Us

  
(https://twitter.com/StackAbuse) (https://www.facebook.com/stackabuse)
(https://stackabuse.com/rss/)

Copyright © 2020, Stack Abuse (https://stackabuse.com). All Rights Reserved.

Disclosure (/disclosure) • Privacy Policy (/privacy-policy) • Terms of Service (/terms-of-service)

https://stackabuse.com/executing-shell-commands-with-python/ 12/12