How To Run Linux Commands In Python : Running Subprocess Module Commands

Running Linux commands in Python allows you to automate server tasks and file operations with minimal code. If you’ve ever wondered how to run Linux commands in Python, you’re in the right place—this guide covers everything from basic execution to advanced error handling.

Python makes it incredibly easy to interact with the Linux shell. Whether you need to check disk space, move files, or run complex bash scripts, Python’s built-in modules have you covered. Let’s jump right in.

Why Run Linux Commands From Python?

Automating repetitive tasks is one of the best reasons to combine Python with Linux commands. Instead of typing the same commands manually, you can write a Python script that does all the work for you.

Here are a few common use cases:

System administration tasks like monitoring CPU usage
File management operations (copy, move, delete)
Running backup scripts
Deploying applications to servers
Parsing command output for reports

Python gives you the flexability to handle errors, capture output, and chain commands together. This makes it far more powerful than writing simple shell scripts.

How To Run Linux Commands In Python

The most common way to run Linux commands in Python is using the subprocess module. It’s part of Python’s standard library, so no extra installations are needed.

Let’s start with a simple example. To run the ls command, you’d write:

import subprocess
subprocess.run(["ls", "-l"])

This prints the directory listing directly to your terminal. The run() function waits for the command to finish before continuing.

Capturing Command Output

Often you’ll want to capture the output of a command for further processing. Use the capture_output=True parameter:

import subprocess
result = subprocess.run(["ls", "-l"], capture_output=True, text=True)
print(result.stdout)

The text=True parameter returns strings instead of bytes. This makes it easier to work with the output directly.

Handling Errors Gracefully

Commands can fail. You can check the return code to see if something went wrong:

import subprocess
result = subprocess.run(["rm", "nonexistent_file"], capture_output=True, text=True)
if result.returncode != 0:
    print(f"Error: {result.stderr}")

You can also use check=True to raise an exception automatically:

import subprocess
try:
    subprocess.run(["rm", "nonexistent_file"], check=True)
except subprocess.CalledProcessError as e:
    print(f"Command failed with error: {e}")

Running Commands With Shell=True

Sometimes you need to run a command exactly as you’d type it in the terminal. Use shell=True for this:

import subprocess
subprocess.run("ls -l | grep .py", shell=True)

Warning: Using shell=True with user input is a security risk. Only use it with trusted commands.

Using The Os Module For Simple Commands

The os module also lets you run commands, but it’s less flexible than subprocess. Use os.system() for quick, one-off commands:

import os
os.system("echo Hello from Python")

The problem with os.system() is that you can’t capture the output easily. It just returns the exit code. For most tasks, subprocess is the better choice.

Running Commands As A Different User

You can run commands with elevated privileges using sudo. Be careful with this:

import subprocess
subprocess.run(["sudo", "apt", "update"])

This will prompt for a password unless you have passwordless sudo configured.

Working With Command Arguments

When your command has arguments, pass them as a list. This is safer than using a string:

import subprocess
subprocess.run(["grep", "error", "/var/log/syslog"])

If you have dynamic arguments from user input, always use the list form to avoid shell injection attacks.

Passing Variables To Commands

You can use Python variables directly in the command list:

import subprocess
filename = "report.txt"
subprocess.run(["cat", filename])

This keeps your code clean and secure.

Running Multiple Commands Sequentially

To run several commands one after another, you can chain them with && or just call subprocess.run() multiple times:

import subprocess
subprocess.run(["cd", "/tmp"], shell=True)
subprocess.run(["ls"])

For more complex workflows, consider using a bash script and calling it from Python.

Using Pipes Between Commands

Pipes let you send output from one command to another. With subprocess, you can do this manually:

import subprocess
p1 = subprocess.Popen(["ls"], stdout=subprocess.PIPE)
p2 = subprocess.Popen(["grep", ".py"], stdin=p1.stdout, stdout=subprocess.PIPE)
p1.stdout.close()
output = p2.communicate()[0]
print(output.decode())

This is more advanced but gives you full control.

Handling Long-Running Commands

Some commands take a long time. Use timeout to avoid hanging forever:

import subprocess
try:
    subprocess.run(["sleep", "10"], timeout=5)
except subprocess.TimeoutExpired:
    print("Command timed out")

This is essential for production scripts where reliability matters.

Running Commands In The Background

You can run commands without waiting for them to finish. Use Popen without calling wait():

import subprocess
subprocess.Popen(["notify-send", "Task started"])

This launches the command and continues immediately.

Real-World Examples

Let’s look at some practical examples you can use today.

Checking Disk Space

import subprocess
result = subprocess.run(["df", "-h"], capture_output=True, text=True)
print(result.stdout)

You can parse this output to alert you when disk usage is high.

Finding Large Files

import subprocess
result = subprocess.run(["find", "/", "-type", "f", "-size", "+100M"], capture_output=True, text=True)
large_files = result.stdout.splitlines()
print(f"Found {len(large_files)} large files")

Automating Backups

import subprocess
import datetime
date = datetime.datetime.now().strftime("%Y-%m-%d")
subprocess.run(["tar", "-czf", f"backup-{date}.tar.gz", "/home/user/documents"])

This creates a timestamped backup archive.

Common Pitfalls And Solutions

Even experienced developers make mistakes. Here are some to watch out for.

Forgetting To Handle Spaces In Paths

If your file path has spaces, always use the list form:

# Wrong
subprocess.run("cat /path/with spaces/file.txt", shell=True)

# Correct
subprocess.run(["cat", "/path/with spaces/file.txt"])

Ignoring The Working Directory

Commands run in the current working directory by default. Change it with the cwd parameter:

import subprocess
subprocess.run(["ls"], cwd="/tmp")

Not Using Text Mode

Without text=True, output comes as bytes. This can cause confusion:

result = subprocess.run(["echo", "hello"], capture_output=True)
print(result.stdout)  # b'hello\n'

Add text=True to get a normal string.

Advanced Techniques

Once you’re comfortable with basics, try these advanced methods.

Using The Shutil Module

For file operations, shutil is often simpler than running shell commands:

import shutil
shutil.copy("source.txt", "destination.txt")
shutil.move("old.txt", "new.txt")

This is cross-platform and avoids shell dependencies.

Running Commands With Environment Variables

Set environment variables using the env parameter:

import subprocess
my_env = {"MY_VAR": "hello"}
subprocess.run(["echo", "$MY_VAR"], env=my_env, shell=True)

Using The Pathlib Module

For file system operations, pathlib provides a modern, object-oriented approach:

from pathlib import Path
p = Path("/home/user")
for file in p.glob("*.txt"):
    print(file.name)

This is often cleaner than running ls and parsing output.

Security Best Practices

Running shell commands from Python can be dangerous. Follow these rules:

Never use shell=True with untrusted input
Always use the list form for commands
Validate and sanitize any user-supplied arguments
Use shlex.quote() if you must use strings
Run scripts with the least privileges necessary

These practices will keep your system safe.

Sanitizing User Input

import subprocess
import shlex
user_input = "some; rm -rf /"
safe_input = shlex.quote(user_input)
subprocess.run(["echo", safe_input])

This prevents command injection attacks.

Comparing Subprocess Methods

Here’s a quick comparison of the main methods:

subprocess.run() – Best for most tasks, waits for completion
subprocess.Popen() – More control, can run in background
subprocess.call() – Older, similar to run() but less features
os.system() – Simple but limited, avoid for serious work
os.popen() – Deprecated, don’t use

Stick with subprocess.run() for 90% of your needs.

Debugging Tips

When things don’t work as expected, try these steps:

Print the command you’re running to verify it’s correct
Check the return code
Examine stderr for error messages
Run the same command manually in the terminal
Use subprocess.list2cmdline() to see how Python interprets your command

These steps will help you find issues quickly.

Example Debugging Session

import subprocess
cmd = ["grep", "pattern", "file.txt"]
print(f"Running: {subprocess.list2cmdline(cmd)}")
result = subprocess.run(cmd, capture_output=True, text=True)
print(f"Return code: {result.returncode}")
print(f"Stdout: {result.stdout}")
print(f"Stderr: {result.stderr}")

This shows exactly what’s happening.

Performance Considerations

Running external commands has overhead. Each subprocess.run() call creates a new process. For many commands, consider:

Batching commands into a single shell script
Using Python libraries instead of shell commands
Using concurrent.futures for parallel execution

For most automation tasks, the overhead is negligible.

Parallel Execution Example

import subprocess
from concurrent.futures import ThreadPoolExecutor

def run_cmd(cmd):
    return subprocess.run(cmd, capture_output=True)

commands = [["ls"], ["pwd"], ["date"]]
with ThreadPoolExecutor() as executor:
    results = executor.map(run_cmd, commands)

This runs multiple commands concurrently.

Integration With Other Python Libraries

Combine shell commands with Python’s ecosystem for powerful solutions.

Using With Pandas

import subprocess
import pandas as pd

result = subprocess.run(["ps", "aux"], capture_output=True, text=True)
lines = result.stdout.splitlines()
data = [line.split(None, 10) for line in lines[1:]]
df = pd.DataFrame(data)
print(df.head())

This lets you analyze process data with Pandas.

Using With Requests

import subprocess
import requests

result = subprocess.run(["curl", "-s", "https://api.example.com"], capture_output=True, text=True)
data = result.stdout
response = requests.post("https://myapp.com/log", json={"output": data})

This sends command output to a web service.

Cross-Platform Considerations

If your script needs to run on Windows too, avoid Linux-specific commands. Use Python’s built-in modules instead:

Use os.listdir() instead of ls
Use shutil.copy() instead of cp
Use os.remove() instead of rm

If you must run Linux commands, check the platform first:

import sys
import subprocess

if sys.platform == "linux":
    subprocess.run(["ls"])
else:
    print("This script only works on Linux")

Frequently Asked Questions

What Is The Best Way To Run Linux Commands In Python?

The subprocess.run() function is the recommended method. It’s flexible, secure, and part of the standard library.

Can I Run Linux Commands In Python On Windows?

Not directly. You’d need WSL (Windows Subsystem for Linux) or Cygwin to run Linux commands on Windows.

How Do I Capture Both Stdout And Stderr?

Use capture_output=True and access result.stdout and result.stderr separately.

Is It Safe To Use Shell=True?

Only with trusted input. Never use it with user-provided data unless you sanitize it first.

How Do I Run A Command As Root From Python?

Use sudo in your command list, like ["sudo", "apt", "update"]. You’ll need appropriate permissions.

Conclusion

Now you know how to run Linux commands in Python effectively. Start with subprocess.run() for most tasks, and gradually explore advanced features like pipes and parallel execution. Remember to always prioritize security by avoiding shell=True with untrusted input. With these techniques, you can automate almost any Linux task from your Python scripts.

Practice by automating one small task today—like checking disk space or backing up a folder. You’ll quickly see how powerful this combination can be. Happy coding!