Bashing through Bash

A 12-Lesson Journey from Zero to Confident Shell User

Why Learn Bash? AI Loves It.

You don't learn Bash to memorize hundreds of commands — you learn it to unlock leverage.

Modern AI assistants are incredibly strong at generating and using Bash commands to automate tasks, inspect systems, move data, transform files, run builds, and glue tools together. But they're only as effective as the person directing them. When you understand how Bash works, even at a practical level, you gain the ability to aim AI precisely at real-world outcomes.

Bash is the universal control layer across servers, containers, CI pipelines, developer machines, and cloud systems. It's the fastest path between an idea and an automated action. With a working knowledge of Bash, you can confidently tell your AI assistant to chain commands, process outputs, fix environments, and build repeatable workflows — instead of treating the terminal like a black box.

You don't need to remember everything. AI can handle syntax. What matters is that you understand the model of how commands, pipes, files, and processes fit together.

Learn Bash, and you turn AI from a helpful tool into a high-power operator working under your direction.

Who This Is For

You use a computer every day. You've probably opened a terminal before. Maybe you've copied and pasted a few commands from Stack Overflow. But you've never really learned bash — not properly, not from the ground up.

This course fixes that. In 12 lessons of roughly 30 minutes each, you'll go from "what does ls do?" to writing real scripts that automate your work and make you genuinely faster.

No prior terminal experience required. Just curiosity and a willingness to type things and see what happens.

How to Use These Lessons

Each lesson is designed to be completed in a single sitting of about 30 minutes. Every lesson follows the same structure:

  • Concepts — what you're learning and why it matters
  • Commands & Syntax — the actual tools, with clear explanations
  • Try It Yourself — hands-on exercises to cement what you've learned
  • Key Takeaways — a quick summary before you move on

The most important thing: type everything out yourself. Don't copy and paste. The muscle memory matters more than you think.

Lesson Index

# Lesson What You'll Learn
01 Welcome to the Terminal What bash is, how to open a terminal, navigating the filesystem
02 Files and Directories Creating, copying, moving, renaming, and deleting files and folders
03 Reading and Searching Files Viewing file contents with cat, less, head, tail, and searching with grep
04 Pipes and Redirection Chaining commands together, redirecting output to files, building data pipelines
05 Permissions and Ownership Understanding rwx, chmod, chown, and why "permission denied" happens
06 Your First Bash Script Writing and running scripts, variables, user input, and making scripts executable
07 Conditionals and Logic If/else, test expressions, comparison operators, and case statements
08 Loops and Iteration For loops, while loops, loop control, and processing multiple files
09 Functions and Script Organisation Writing functions, passing arguments, return values, and structuring scripts
10 Text Processing Power Tools sed, awk, cut, sort, uniq — transforming and analysing text like a pro
11 Process Management and Job Control Running processes, background jobs, signals, and scheduling with cron
12 Real-World Scripting Error handling, debugging, best practices, and building a complete project

What You'll Need

  • A terminal application (Terminal on macOS, any terminal emulator on Linux)
  • A text editor you're comfortable with (VS Code, nano, vim — anything works)
  • About 30 minutes per lesson
  • A folder to practice in (we'll set this up in Lesson 01)

A Note on macOS vs Linux

These lessons work on both macOS and Linux. The vast majority of commands are identical. Where differences exist, they're noted. If you're on Windows, use WSL (Windows Subsystem for Linux) to follow along.

Let's get started. Open your terminal and head to Lesson 01.

Lesson 01: Welcome to the Terminal

Time: ~30 minutes

What Is Bash?

Bash stands for "Bourne Again Shell." It's a program that takes commands you type and tells your computer what to do. That's it. Every time you open a terminal window, you're starting a bash session (or a similar shell like zsh — more on that shortly).

The terminal might look intimidating with its blinking cursor and blank screen, but here's the thing: it's just a different way to do what you already do with your mouse. Instead of clicking on a folder to open it, you type a command. Instead of dragging a file to the trash, you type a command. The difference is that typing commands is faster, repeatable, and automatable in ways that clicking around never will be.

Bash vs Shell vs Terminal — Clearing Up the Confusion

These terms get used interchangeably, but they mean different things:

  • Terminal (or terminal emulator) — the application window you type into. It's just a container.
  • Shell — the program running inside the terminal that interprets your commands. Bash is one shell. Zsh is another. Fish is another.
  • Bash — a specific shell, and the one we're learning. It's the default on most Linux systems and was the default on macOS until Catalina (which switched to zsh).

If you're on macOS and using zsh, don't worry. Everything in this course works in zsh too. The differences are minor and won't matter until you're much more advanced.

Opening Your Terminal

macOS: Press Cmd + Space, type "Terminal", hit Enter. Or find it in Applications → Utilities → Terminal.

Linux: Press Ctrl + Alt + T on most distributions. Or find your terminal application in your app menu.

You should see something like this:

kevin@macbook ~ %

or

kevin@ubuntu:~$

This is your prompt. It's telling you the shell is ready and waiting for you to type something. The ~ means you're in your home directory (we'll cover that next). The $ or % is just the prompt character — it's not something you type.

Your First Commands

Type each of these and press Enter. Watch what happens.

pwd — Print Working Directory

pwd

This tells you where you are in the filesystem right now. You should see something like /Users/kevin (macOS) or /home/kevin (Linux). This is your home directory — your personal space on the computer.

ls — List

ls

This shows you what's in the current directory. You should recognise these — they're the same folders you see in Finder or your file manager: Desktop, Documents, Downloads, etc.

echo — Print Text

echo "Hello from the terminal"

This just prints whatever you give it. Simple, but you'll use it constantly — for debugging, for displaying information, for writing to files.

Your computer's filesystem is a tree. At the top is the root, written as /. Everything lives below it. Your home directory is a branch on that tree.

Here's a simplified view:

/                       ← root
├── Users/              ← (macOS) or /home/ (Linux)
│   └── kevin/          ← your home directory (~)
│       ├── Desktop/
│       ├── Documents/
│       └── Downloads/
├── usr/
│   ├── bin/            ← where many commands live
│   └── local/
├── etc/                ← system configuration files
├── tmp/                ← temporary files
└── var/                ← variable data (logs, etc.)

cd — Change Directory

This is how you move around.

cd Documents

You just moved into your Documents folder. Run pwd to confirm — it should show /Users/kevin/Documents or similar.

Now go back:

cd ..

The .. means "one level up" — the parent directory. You're back in your home directory.

Some important shortcuts:

cd ~          # go to your home directory (from anywhere)
cd            # same thing — cd with no argument goes home
cd -          # go back to the previous directory you were in
cd /          # go to the root of the filesystem
cd ../..      # go up two levels

Path Types: Absolute vs Relative

There are two ways to specify a location:

Absolute paths start from the root / and spell out the full location:

cd /Users/kevin/Documents/Projects

Relative paths start from where you are now:

cd Documents/Projects

Both get you to the same place. Use whichever makes sense. If you're already close to where you want to be, a relative path is shorter. If you want to be unambiguous, use absolute.

Getting Help

Almost every command has built-in documentation. There are two ways to access it:

man — Manual Pages

man ls

This opens the full manual for ls. It tells you every option, every flag, every behaviour. Use arrow keys to scroll, press q to quit.

Manual pages can be dense. That's OK. You don't need to read the whole thing — just scan for what you need.

--help Flag

Most commands support a shorter help summary:

ls --help

(On macOS, some built-in commands don't support --help. Use man instead.)

type and which — Finding Commands

Want to know where a command lives or what it is?

type ls        # tells you what kind of command ls is
which python   # tells you the full path to the python executable

Useful ls Options

Plain ls gives you the basics. These flags give you more:

ls -l          # long format — shows permissions, size, date
ls -a          # show hidden files (files starting with .)
ls -la         # combine both — this is the one you'll use most
ls -lh         # human-readable file sizes (KB, MB instead of bytes)
ls -lt         # sort by modification time (newest first)
ls -lS         # sort by file size (largest first)

Hidden files are files whose names start with a dot, like .bashrc or .gitconfig. They're hidden by default to reduce clutter, but they're often important configuration files. The -a flag reveals them.

Setting Up Your Practice Space

Let's create a folder for these lessons. We'll learn mkdir properly in the next lesson, but for now, just type:

cd ~
mkdir -p bash-lessons
cd bash-lessons
pwd

You should see something like /Users/kevin/bash-lessons. This is where you'll do all your practice work for the rest of the course.

Tab Completion — Your New Best Friend

This is possibly the most useful thing you'll learn today. Start typing a command or filename and press Tab:

cd Docu<Tab>

Bash will auto-complete to cd Documents/ (assuming that's the only match). If there are multiple matches, press Tab twice to see all options.

This saves you an enormous amount of typing and prevents typos. Use it constantly. If you take one habit away from this lesson, let it be this.

Command History

Bash remembers everything you type.

  • Press the Up arrow to cycle through previous commands
  • Press the Down arrow to go forward
  • Type history to see a numbered list of recent commands
  • Press Ctrl + R and start typing to search your history — this is incredibly useful
history          # see your command history
history | tail   # see just the last 10 commands

Try It Yourself

Complete these exercises in your terminal. Don't copy and paste — type them out.

  1. Open your terminal and run pwd. Note your home directory path.
  2. Run ls -la in your home directory. Find three hidden files (starting with .).
  3. Navigate to /tmp using an absolute path. Run pwd to confirm. Navigate back home with cd ~.
  4. Navigate to your Documents folder. Then use cd - to jump back. Use cd - again. Notice how it toggles between two locations.
  5. Use ls -lt in your home directory to find the most recently modified file or folder.
  6. Navigate to your bash-lessons folder. Run ls -la. It should be empty (except for . and ..).
  7. Use man ls to find out what the -R flag does. Try it out.
  8. Use Ctrl + R and search your history for "pwd". Press Enter to re-run it.

Key Takeaways

  • The terminal is just a text-based way to interact with your computer. Nothing magical, nothing scary.
  • pwd tells you where you are. ls shows what's here. cd moves you around.
  • Absolute paths start with /. Relative paths start from your current location.
  • .. means the parent directory. ~ means your home directory.
  • Use Tab for auto-completion. Use Up arrow and Ctrl + R for history. These save you enormous time.
  • Use man or --help when you're not sure what a command does.

Next up: Lesson 02 — Files and Directories

Lesson 02: Files and Directories

Time: ~30 minutes

The Building Blocks

Everything on your computer is either a file or a directory (folder). That sounds obvious, but in the terminal, understanding this deeply matters. Directories are just special files that contain references to other files. There's no magic — just a tree of names pointing to data on disk.

In this lesson, you'll learn to create, copy, move, rename, and delete files and directories — all from the command line. Once you're comfortable with these commands, you'll rarely need a graphical file manager again.

Creating Directories with mkdir

cd ~/bash-lessons
mkdir projects

That creates a directory called projects inside your current location. Verify with ls.

Creating Nested Directories

What if you want to create a directory several levels deep?

mkdir projects/web/frontend

This fails if projects/web doesn't exist yet. The -p flag fixes that — it creates all necessary parent directories:

mkdir -p projects/web/frontend
mkdir -p projects/web/backend
mkdir -p projects/scripts

The -p flag is also safe to use on directories that already exist — it won't throw an error or overwrite anything.

Creating Files

There are several ways to create files. Each has its place.

touch — Create an Empty File

touch notes.txt

If the file doesn't exist, it creates an empty one. If it already exists, it updates the file's modification timestamp without changing its contents. This makes touch safe to run repeatedly.

Create several files at once:

touch file1.txt file2.txt file3.txt

echo with Redirection — Create a File with Content

echo "This is my first file" > hello.txt

The > operator redirects the output of echo into a file. If the file exists, it gets overwritten. We'll cover redirection in depth in Lesson 04.

Text Editors — Create and Edit Files

For anything more than a line or two, you'll want a text editor:

nano notes.txt     # simple terminal editor, good for beginners
vim notes.txt      # powerful but has a learning curve
code notes.txt     # opens in VS Code (if installed)

If you're new to terminal editors, use nano. It shows keyboard shortcuts at the bottom of the screen. Ctrl + O saves, Ctrl + X exits.

Copying Files and Directories with cp

Copying a File

cp hello.txt hello-backup.txt

This creates a duplicate. The original is untouched.

Copying a File to Another Directory

cp hello.txt projects/

This puts a copy of hello.txt inside the projects directory.

Copying a File to Another Directory with a New Name

cp hello.txt projects/greeting.txt

Copying Directories

To copy a directory and everything inside it, you need the -r (recursive) flag:

cp -r projects projects-backup

Without -r, bash will refuse to copy a directory. This is a safety measure — copying a directory means copying potentially thousands of files, and bash wants you to be explicit about that.

Moving and Renaming with mv

In bash, moving and renaming are the same operation. The mv command changes where a file lives — or what it's called.

Renaming a File

mv hello.txt greetings.txt

The file hello.txt no longer exists. It's now called greetings.txt.

Moving a File to Another Directory

mv greetings.txt projects/

The file is now at projects/greetings.txt. It's no longer in the current directory.

Moving and Renaming at the Same Time

mv projects/greetings.txt projects/web/welcome.txt

Moving Directories

Unlike cp, mv doesn't need a -r flag for directories. It just works:

mv projects-backup old-projects

A Useful Safety Flag

The -i flag makes mv ask for confirmation before overwriting an existing file:

mv -i newfile.txt existingfile.txt

If existingfile.txt already exists, bash will ask you before replacing it. Without -i, it overwrites silently. Consider using -i when you're moving files into directories where name collisions might occur.

Deleting Files and Directories with rm

This is the one command you need to be careful with. There is no trash bin in the terminal. When you rm a file, it's gone. No undo, no recovery.

Deleting a File

rm file1.txt

Gone. Immediately. No confirmation.

Deleting Multiple Files

rm file2.txt file3.txt

Deleting with Confirmation

rm -i notes.txt

Bash will ask "remove notes.txt?" and wait for your y or n.

Deleting Empty Directories

rmdir projects/scripts

rmdir only works on empty directories. This is a safety feature.

Deleting Directories and Their Contents

rm -r old-projects

The -r (recursive) flag tells rm to delete the directory and everything inside it. Every file. Every subdirectory. All of it.

The Dangerous Command

You'll see this in tutorials and Stack Overflow answers:

rm -rf something/

The -f (force) flag suppresses all confirmation prompts and ignores errors. Combined with -r, it deletes everything without asking. This is useful in scripts but dangerous when typed manually.

Never run rm -rf / or rm -rf ~. The first attempts to delete your entire filesystem. The second deletes your entire home directory. Modern systems have safeguards against the first one, but the second will ruin your day.

A good habit: always double-check the path before pressing Enter on any rm -r command. Read it twice.

Wildcards and Globbing

Wildcards let you match multiple files with a pattern instead of naming each one individually.

* — Matches Anything

ls *.txt           # all files ending in .txt
ls project*        # all files starting with "project"
cp *.txt projects/ # copy all .txt files into projects/
rm *.log           # delete all .log files

? — Matches Exactly One Character

ls file?.txt       # matches file1.txt, file2.txt, but not file10.txt

[...] — Matches Any Character in the Set

ls file[123].txt   # matches file1.txt, file2.txt, file3.txt
ls file[1-5].txt   # matches file1.txt through file5.txt

Using Wildcards Safely

Before deleting with wildcards, preview what you're about to delete:

ls *.tmp           # see what matches
rm *.tmp           # then delete

Or use rm -i *.tmp to get confirmation for each file.

Checking File Properties

file — Determine File Type

file notes.txt          # "ASCII text"
file /usr/bin/bash      # "ELF 64-bit LSB executable..."
file photo.jpg          # "JPEG image data..."

The file command looks at the actual contents, not just the extension. A .txt file that contains binary data will be reported as binary.

stat — Detailed File Information

stat notes.txt

This shows you everything: size, permissions, owner, creation date, modification date, inode number. It's more detail than you usually need, but it's there when you want it.

wc — Word Count

wc notes.txt        # lines, words, characters
wc -l notes.txt     # just the line count
wc -w notes.txt     # just the word count

du — Disk Usage

du -sh projects/     # total size of a directory, human-readable
du -sh */            # size of each subdirectory

Try It Yourself

Work through these in your ~/bash-lessons directory.

  1. Create this directory structure in a single command:

    practice/
├── drafts/
├── final/
└── archive/

    (Hint: mkdir -p can take multiple arguments.)

  2. Create five files: note1.txt through note5.txt using touch.

  3. Write "Draft version" into note1.txt using echo and >.

  4. Copy note1.txt into the practice/drafts/ directory.

  5. Move note2.txt and note3.txt into practice/drafts/.

  6. Rename note4.txt to important.txt.

  7. Copy the entire practice directory to practice-backup.

  8. Delete note5.txt and important.txt.

  9. Use ls *.txt to see what .txt files remain in your current directory.

  10. Use du -sh practice/ to check the size of your practice directory.

Key Takeaways

  • mkdir -p creates directories and any missing parents. It's always safe to use.
  • touch creates empty files or updates timestamps. echo "text" > file creates files with content.
  • cp copies files. Add -r for directories.
  • mv both moves and renames. It works on files and directories without any flags.
  • rm deletes permanently. There is no undo. Use rm -i when you want confirmation. Use rm -r for directories.
  • Wildcards (*, ?, [...]) let you work with groups of files. Always preview with ls before deleting with wildcards.

Next up: Lesson 03 — Reading and Searching Files

Lesson 03: Reading and Searching Files

Time: ~30 minutes

Why This Matters

You'll spend more time reading files than writing them. Log files, configuration files, code, data exports, documentation — being able to quickly view, search, and navigate file contents from the terminal is one of the most practical skills you can develop.

This lesson covers the essential tools for looking at files and finding things inside them.

Setting Up Practice Files

Before we start, let's create some files to work with:

cd ~/bash-lessons
mkdir -p lesson03
cd lesson03

Create a sample log file:

for i in $(seq 1 100); do
  echo "$(date '+%Y-%m-%d %H:%M:%S') [INFO] Processing record $i" >> server.log
done
echo "2025-01-15 10:23:45 [ERROR] Connection refused: database timeout" >> server.log
echo "2025-01-15 10:24:01 [WARN] Retrying connection (attempt 1)" >> server.log
echo "2025-01-15 10:24:15 [ERROR] Connection refused: database timeout" >> server.log
echo "2025-01-15 10:25:00 [INFO] Connection restored" >> server.log

Don't worry about the for loop syntax yet — that's in Lesson 08. Just run it and you'll have a realistic log file to practice with.

Create a sample configuration file:

cat > config.txt << 'EOF'
# Application Configuration
app_name=MyWebApp
version=2.4.1
debug=false

# Database Settings
db_host=localhost
db_port=5432
db_name=production
db_user=admin

# API Settings
api_key=sk-abc123def456
api_timeout=30
api_retries=3

# Feature Flags
enable_caching=true
enable_logging=true
enable_notifications=false
EOF

Viewing Entire Files

cat — Concatenate and Print

cat config.txt

cat dumps the entire file to your terminal. It's the simplest tool — and the right choice for short files. The name comes from "concatenate" because it can also join files together:

cat file1.txt file2.txt     # prints both files in sequence

cat with Line Numbers

cat -n config.txt

The -n flag adds line numbers. Useful when you need to reference specific lines.

When cat Is the Wrong Tool

If a file is hundreds or thousands of lines long, cat will flood your terminal. You'll see only the last screenful of output and lose everything above it. For large files, use less.

Viewing Large Files with less

less server.log

less opens the file in a scrollable viewer. Unlike cat, it doesn't dump everything at once — it shows one screen at a time and lets you navigate.

Navigation in less

Key Action
Space or f Forward one page
b Back one page
d Forward half a page
u Back half a page
g Go to the beginning
G Go to the end
q Quit
/pattern Search forward for "pattern"
?pattern Search backward for "pattern"
n Next search match
N Previous search match

The search function inside less is incredibly useful. Open your log file and try:

/ERROR

This jumps to the first occurrence of "ERROR". Press n to find the next one.

Why less and Not more

You'll sometimes see the more command mentioned. less is the improved version of more (hence the joke: "less is more"). Use less. It does everything more does, plus backward scrolling, better searching, and more.

Viewing Parts of Files

Sometimes you don't need the whole file — just the beginning or the end.

head — View the Beginning

head server.log          # first 10 lines (default)
head -20 server.log      # first 20 lines
head -1 server.log       # just the first line

Useful for checking the structure of a file or seeing its headers.

tail — View the End

tail server.log          # last 10 lines (default)
tail -20 server.log      # last 20 lines
tail -1 server.log       # just the last line

tail -f — Follow a File in Real Time

This is one of the most used commands in development and operations:

tail -f server.log

This shows the last 10 lines and then waits. Whenever new lines are added to the file, they appear immediately. It's how you watch logs in real time while debugging a running application.

Press Ctrl + C to stop following.

Searching Inside Files with grep

grep is one of the most powerful and frequently used tools in bash. It searches for patterns in files and prints every line that matches.

Basic Usage

grep "ERROR" server.log

This prints every line in server.log that contains the text "ERROR".

Useful grep Flags

grep -i "error" server.log       # case-insensitive search
grep -n "ERROR" server.log       # show line numbers
grep -c "ERROR" server.log       # count matching lines (just the number)
grep -v "INFO" server.log        # invert match — show lines that DON'T contain "INFO"

Searching Multiple Files

grep "database" *.txt            # search all .txt files
grep -r "TODO" ~/projects/       # search recursively through a directory
grep -rl "TODO" ~/projects/      # just list filenames that contain matches

The -r flag makes grep search through directories recursively. The -l flag shows only filenames, not the matching lines themselves.

Combining Flags

Flags can be combined. This is common and encouraged:

grep -in "error" server.log      # case-insensitive with line numbers
grep -rn "TODO" ~/projects/      # recursive with line numbers
grep -cv "^#" config.txt         # count lines that aren't comments

Context Around Matches

Sometimes you need to see what's around a match, not just the matching line:

grep -A 2 "ERROR" server.log    # show 2 lines After each match
grep -B 2 "ERROR" server.log    # show 2 lines Before each match
grep -C 2 "ERROR" server.log    # show 2 lines of Context (before and after)

This is invaluable when reading logs — the error itself often only makes sense with surrounding context.

Basic Pattern Matching in grep

grep supports regular expressions — patterns that match text. You don't need to master regex right now, but a few basics go a long way.

The Dot — Match Any Character

grep "db_...." config.txt     # matches "db_" followed by any 4 characters

Start and End of Line

grep "^#" config.txt          # lines that start with #
grep "false$" config.txt      # lines that end with "false"
grep "^$" config.txt          # empty lines

The ^ means "start of line" and $ means "end of line."

Character Classes

grep "[0-9]" config.txt       # lines containing any digit
grep "^[^#]" config.txt       # lines that don't start with # (inside brackets, ^ means "not")

Extended Regular Expressions with -E

The -E flag enables extended regex, which gives you more pattern options:

grep -E "ERROR|WARN" server.log        # matches ERROR or WARN
grep -E "attempt [0-9]+" server.log    # matches "attempt" followed by one or more digits

We'll revisit regex more in Lesson 10. For now, just knowing ^, $, ., [...], and | (with -E) covers most of what you'll need day to day.

Other Useful Viewing Commands

sort — Sort Lines

sort config.txt              # sort alphabetically
sort -r config.txt           # reverse sort
sort -n numbers.txt          # numeric sort (so 10 comes after 9, not after 1)

uniq — Remove Duplicate Lines

sort config.txt | uniq       # remove duplicates (file must be sorted first)
sort config.txt | uniq -c    # show count of each unique line

uniq only removes adjacent duplicates, which is why you almost always pair it with sort.

diff — Compare Two Files

cp config.txt config-new.txt
echo "debug=true" >> config-new.txt
diff config.txt config-new.txt

diff shows you the differences between two files. It's the foundation of how version control systems like Git track changes.

Try It Yourself

  1. Use cat -n to view config.txt with line numbers. What's on line 8?

  2. Use less to open server.log. Search for "ERROR" using /ERROR. How many errors are there? (Use n to find each one, or exit and use grep -c.)

  3. Use head -5 server.log to see the first 5 log entries.

  4. Use tail -5 server.log to see the last 5 entries.

  5. Use grep to find all lines in config.txt that contain "enable". How many are there?

  6. Use grep -v "^#" config.txt | grep -v "^$" to show only the actual configuration values (no comments, no blank lines).

  7. Use grep -c "INFO" server.log to count the number of INFO messages.

  8. Use grep -C 1 "ERROR" server.log to see context around each error.

  9. Create a copy of config.txt, change one value in it using nano, then use diff to see the difference.

  10. Run grep -E "ERROR|WARN" server.log to find all problems in the log at once.

Key Takeaways

  • cat is for small files. less is for large files. Know when to use which.
  • head and tail show the beginning and end of files. tail -f follows files in real time — essential for watching logs.
  • grep is your search tool. Learn its flags: -i (case-insensitive), -n (line numbers), -r (recursive), -v (invert), -c (count), -A/-B/-C (context).
  • Basic regex patterns (^, $, ., [...], |) make your searches far more powerful.
  • sort, uniq, and diff round out your text analysis toolkit.

Next up: Lesson 04 — Pipes and Redirection

Lesson 04: Pipes and Redirection

Time: ~30 minutes

The Big Idea

In Lesson 03, you used commands one at a time — run grep, see the output, run another command. That works, but the real power of bash comes from connecting commands together.

The Unix philosophy is: make each tool do one thing well, then combine them. grep searches. sort sorts. wc counts. Alone, they're useful. Chained together, they're a data processing pipeline that can rival a script in any language — written in a single line.

This lesson teaches you how to chain commands with pipes and control where output goes with redirection. Once you understand these two concepts, the terminal stops feeling like a one-trick tool and starts feeling like a workshop.

Standard Streams

Every command in bash has three data streams:

  • stdin (standard input) — where data comes in. Usually your keyboard.
  • stdout (standard output) — where results go. Usually your terminal screen.
  • stderr (standard error) — where error messages go. Also usually your terminal screen.

These streams are numbered: stdin is 0, stdout is 1, stderr is 2. These numbers matter when you want to redirect specific streams, which we'll get to shortly.

When you run ls, the list of files goes to stdout (your screen). If you ls a directory that doesn't exist, the error message goes to stderr (also your screen). They look the same, but bash treats them differently — and you can redirect them independently.

Pipes: Connecting Commands

The pipe operator | takes the stdout of one command and sends it to the stdin of the next command.

ls -la | less

This runs ls -la, but instead of printing to the screen, it sends the output into less so you can scroll through it. The first command produces data; the second command receives it.

Building a Pipeline

You can chain as many commands as you want:

cat server.log | grep "ERROR" | wc -l

What happens here, step by step:

  1. cat server.log — outputs the entire log file
  2. grep "ERROR" — receives that output, keeps only lines containing "ERROR"
  3. wc -l — receives those filtered lines, counts them

The result is a single number: how many error lines are in the log.

More Pipeline Examples

Find the 5 largest files in a directory:

ls -lS | head -6

(Head 6 because ls -l includes a "total" line at the top.)

Show unique error types in a log:

grep "ERROR" server.log | sort | uniq

Count how many times each error message appears:

grep "ERROR" server.log | sort | uniq -c | sort -rn

This is where it gets elegant. Four commands, each doing one thing, producing a frequency-sorted list of error messages. No temporary files, no scripting language needed.

Find the 10 most recently modified files:

ls -lt | head -11

See which processes are using the most memory:

ps aux | sort -k4 -rn | head -10

Output Redirection

Pipes send output to another command. Redirection sends output to a file.

> — Write to a File (Overwrite)

echo "Hello World" > output.txt
ls -la > filelist.txt
grep "ERROR" server.log > errors.txt

If the file exists, it gets replaced. If it doesn't exist, it gets created. The output no longer appears on screen — it goes into the file instead.

>> — Append to a File

echo "First line" > log.txt
echo "Second line" >> log.txt
echo "Third line" >> log.txt
cat log.txt

The >> operator adds to the end of the file instead of overwriting. This is how you build up files incrementally, and it's essential for logging.

The Overwrite Trap

This is a common mistake:

sort data.txt > data.txt    # DANGER — this empties the file!

Bash sets up the redirection (creating/emptying data.txt) before the command runs. So sort reads from an already-empty file. The result is an empty file. Always redirect to a different filename, then rename if needed:

sort data.txt > data-sorted.txt
mv data-sorted.txt data.txt

Input Redirection

< — Read from a File

Most commands can take a filename as an argument, so you don't use < as often. But it exists:

sort < unsorted.txt
wc -l < server.log

This is functionally the same as sort unsorted.txt, but the mechanism is different: with <, bash opens the file and feeds it to the command's stdin. With sort unsorted.txt, the sort command opens the file itself.

Here Documents (<<) — Inline Multi-line Input

This lets you feed multiple lines of text to a command:

cat << EOF
Line one
Line two
Line three
EOF

The word after << (here, EOF) is a delimiter. Everything between the first EOF and the last EOF becomes the input. You can use any word as the delimiter, but EOF is conventional.

This is extremely useful in scripts for creating files or feeding input to commands:

cat << EOF > config.ini
[database]
host=localhost
port=5432
EOF

Redirecting stderr

By default, error messages go to your screen even when you redirect stdout. This is usually what you want — if something goes wrong, you want to see it. But sometimes you need to control error output separately.

Redirect Only Errors to a File

ls /nonexistent 2> errors.txt

The 2> redirects stream number 2 (stderr) to a file. Normal output still goes to the screen.

Redirect stdout and stderr Separately

command > output.txt 2> errors.txt

Output goes to one file, errors go to another. This is useful in scripts where you want to log errors separately.

Redirect Both to the Same File

command > all-output.txt 2>&1

The 2>&1 means "send stderr to the same place as stdout." The order matters here — the stdout redirect must come first.

There's also a shorthand:

command &> all-output.txt

This does the same thing and is easier to read.

Discard Output Entirely

command > /dev/null 2>&1

/dev/null is a special file that discards everything written to it. This is how you silence a command completely. You'll see this in scripts when you care about a command's exit status but not its output.

Combining Pipes and Redirection

Pipes and redirection work together. The pipe chains commands; the redirection at the end captures the final result:

grep "ERROR" server.log | sort | uniq -c | sort -rn > error-report.txt

This runs the entire pipeline and saves the final output to a file. The intermediate steps aren't saved anywhere — they flow through the pipe and are gone.

You can also redirect at different points in the pipeline, though this is less common:

grep "ERROR" server.log 2>/dev/null | sort | uniq -c > report.txt

Here, any errors from grep are silently discarded, while the successful output flows through the pipeline and into a file.

The tee Command — Output to Screen AND File

Sometimes you want to see the output and save it. tee does exactly that:

grep "ERROR" server.log | tee errors.txt

This prints the matching lines to your screen and simultaneously writes them to errors.txt. It's like a T-junction for your data stream.

Use tee -a to append instead of overwrite:

echo "New error found" | tee -a errors.txt

tee is also useful mid-pipeline for debugging — you can see what's flowing through at a specific point:

cat server.log | grep "ERROR" | tee /dev/stderr | wc -l

This shows you the matching lines (via tee to stderr) and also gives you the count (via wc -l to stdout).

Practical Pipeline Patterns

Here are some pipeline patterns you'll use repeatedly:

Find and Count

# How many Python files are in this project?
find . -name "*.py" | wc -l

# How many unique IP addresses are in the access log?
awk '{print $1}' access.log | sort -u | wc -l

Filter and Format

# Show only active config values (no comments, no blank lines), sorted
grep -v "^#" config.txt | grep -v "^$" | sort

Extract and Aggregate

# Find the most common words in a file
cat document.txt | tr ' ' '\n' | sort | uniq -c | sort -rn | head -20

This pipeline: converts spaces to newlines (one word per line), sorts them, counts unique occurrences, sorts by count descending, and shows the top 20.

Search Across Multiple Files

# Find all TODO comments in a project and list by file
grep -rn "TODO" ~/projects/ | sort -t: -k1,1

Try It Yourself

First, make sure you're in the lesson03 directory (or wherever your server.log and config.txt files are).

  1. Use a pipe to count how many lines in server.log contain "INFO": grep "INFO" server.log | wc -l

  2. Extract all the error and warning lines from server.log, sort them, and save to problems.txt.

  3. Show only the configuration keys (not comments, not blank lines) from config.txt, sorted alphabetically. Save the result to active-config.txt.

  4. Run ls -la /etc — if the output is too long, pipe it to less.

  5. Run a command that produces an error (like ls /nonexistent) and redirect only the error to oops.txt. Verify the file contains the error message.

  6. Use echo and >> to create a file with three lines, one command at a time. Verify with cat.

  7. Use tee to both display and save the output of grep "ERROR" server.log.

  8. Build a pipeline that finds all unique log levels in server.log (the words in square brackets like INFO, ERROR, WARN). Hint: grep -oE '\[.*?\]' or think about how to extract them with cut.

Key Takeaways

  • The pipe | sends the output of one command into the input of the next. This is the backbone of bash's power.
  • > writes output to a file (overwriting). >> appends. Never redirect to the same file you're reading from.
  • stderr (2>) and stdout (>) are separate streams. You can redirect them independently.
  • /dev/null is the black hole — redirect there to discard output.
  • tee splits output to both the screen and a file. Use it when you need to see and save simultaneously.
  • Pipelines let you build complex data transformations from simple commands. Think of each command as a step in an assembly line.

Next up: Lesson 05 — Permissions and Ownership

Lesson 05: Permissions and Ownership

Time: ~30 minutes

Why Permissions Exist

Every file and directory on your system has rules about who can do what with it. These rules exist for good reason: they prevent you from accidentally deleting system files, they keep other users out of your private data, and they control which programs can execute.

If you've ever seen "Permission denied" in your terminal, this lesson explains why — and how to fix it.

Reading Permissions

Run ls -la in any directory:

drwxr-xr-x  5 kevin  staff   160 Jan 15 10:30 projects
-rw-r--r--  1 kevin  staff   425 Jan 15 09:15 config.txt
-rwxr-xr-x  1 kevin  staff  1024 Jan 14 14:22 deploy.sh

That first column is the permission string. Let's break it down using -rw-r--r-- as an example:

-  rw-  r--  r--
│  │    │    │
│  │    │    └── Others (everyone else): read only
│  │    └─────── Group (staff): read only
│  └──────────── Owner (kevin): read and write
└─────────────── File type: - = file, d = directory, l = symlink

The Three Permission Types

  • r (read) — can view the file's contents, or list a directory's contents
  • w (write) — can modify the file, or add/remove files in a directory
  • x (execute) — can run the file as a program, or enter/traverse a directory

The Three User Classes

Permissions are set for three groups of people:

  • Owner (u) — the user who owns the file, usually whoever created it
  • Group (g) — a group of users that the file belongs to
  • Others (o) — everyone else on the system

Reading the ls -la Output

The other columns tell you:

-rw-r--r--  1  kevin  staff  425  Jan 15 09:15  config.txt
│           │  │      │      │    │              │
│           │  │      │      │    │              └── filename
│           │  │      │      │    └── modification date
│           │  │      │      └── file size in bytes
│           │  │      └── group owner
│           │  └── user owner
│           └── number of hard links
└── permissions

Directory Permissions

Permissions work slightly differently for directories:

  • r on a directory means you can list its contents (ls)
  • w on a directory means you can add or remove files inside it
  • x on a directory means you can cd into it and access files within it

A directory with r-- but no x is an odd case: you can see the names of files inside it, but you can't actually read or access them. You usually want r and x together for directories.

Changing Permissions with chmod

There are two ways to use chmod: symbolic mode (letters) and numeric mode (numbers). Both do the same thing.

Symbolic Mode

The format is: chmod [who][operator][permission] file

Who: u (owner), g (group), o (others), a (all three) Operator: + (add), - (remove), = (set exactly) Permission: r, w, x

chmod u+x script.sh        # give the owner execute permission
chmod g-w config.txt        # remove write permission from the group
chmod o-rwx private.txt     # remove all permissions from others
chmod a+r readme.txt        # give everyone read permission
chmod u=rwx,g=rx,o=r file   # set exact permissions for each class

Numeric (Octal) Mode

Each permission has a numeric value:

  • r = 4
  • w = 2
  • x = 1

You add these values together for each user class to get a three-digit number:

rwx = 4+2+1 = 7
rw- = 4+2+0 = 6
r-x = 4+0+1 = 5
r-- = 4+0+0 = 4
--- = 0+0+0 = 0

So a three-digit number represents owner, group, and others:

chmod 755 script.sh        # rwxr-xr-x — owner can do everything, others can read/execute
chmod 644 config.txt       # rw-r--r-- — owner can read/write, others can only read
chmod 700 private/         # rwx------ — only the owner can access
chmod 600 secrets.txt      # rw------- — only the owner can read/write

Common Permission Patterns

Numeric Symbolic Meaning Typical Use
755 rwxr-xr-x Owner full, others read/execute Scripts, programs, directories
644 rw-r--r-- Owner read/write, others read Regular files, configs
700 rwx------ Owner only, full access Private directories
600 rw------- Owner only, read/write SSH keys, sensitive files
666 rw-rw-rw- Everyone read/write Rarely a good idea
777 rwxrwxrwx Everyone everything Almost never use this

Making Scripts Executable

This is the most common chmod use case you'll encounter:

echo '#!/bin/bash' > myscript.sh
echo 'echo "It works!"' >> myscript.sh
./myscript.sh                # Permission denied
chmod +x myscript.sh         # shorthand for chmod a+x
./myscript.sh                # It works!

When you write chmod +x, it adds execute permission for everyone. This is what you'll do every time you create a new bash script.

Recursive Permission Changes

chmod -R 755 projects/      # apply 755 to the directory and everything inside it

Be careful with -R. It applies the same permissions to files and directories, which isn't always what you want. Files usually don't need execute permission. A safer approach:

find projects/ -type d -exec chmod 755 {} \;    # directories get 755
find projects/ -type f -exec chmod 644 {} \;    # files get 644

Don't worry about the find -exec syntax yet — we'll cover it later. Just know this pattern exists for when you need it.

Ownership with chown

Every file has an owner and a group. You can change these with chown:

chown kevin file.txt           # change the owner to kevin
chown kevin:staff file.txt     # change owner and group
chown :staff file.txt          # change only the group
chown -R kevin:staff projects/ # change recursively

You generally need sudo (superuser privileges) to change ownership:

sudo chown kevin:staff file.txt

What Is sudo?

sudo means "superuser do." It runs a single command with administrator privileges. The system will ask for your password.

sudo ls /root                  # view a protected directory
sudo chmod 644 /etc/somefile   # change permissions on a system file

Use sudo only when necessary. If a command works without it, don't add it. Running everything as superuser is a bad habit that can lead to accidentally modifying system files.

Understanding "Permission Denied"

When you see this error, work through this checklist:

  1. Check the permissions: ls -la file.txt — do you have the permission you need?
  2. Check the owner: Is the file owned by you? If not, do you have group or other permissions?
  3. Check the parent directory: Do you have x permission on every directory in the path?
  4. Try the fix:
    • Need to read a file? chmod u+r file.txt
    • Need to run a script? chmod u+x script.sh
    • Need to write to a directory? chmod u+w directory/
    • File owned by root? sudo may be needed.

The umask — Default Permissions

When you create a new file or directory, it gets default permissions. The umask controls what those defaults are.

umask           # shows current umask (usually 022)

The umask is subtracted from the maximum permissions:

  • Files max: 666 (rw-rw-rw-) — files don't get execute by default
  • Directories max: 777 (rwxrwxrwx)

With a umask of 022:

  • New files: 666 - 022 = 644 (rw-r--r--)
  • New directories: 777 - 022 = 755 (rwxr-xr-x)

You rarely need to change the umask, but understanding it explains why your files get the permissions they do.

Special Permissions (Brief Overview)

You'll occasionally encounter these. You don't need to memorise them now, but knowing they exist helps when you see unfamiliar permission strings.

Setuid (s in owner execute): When a setuid program runs, it executes with the permissions of the file's owner, not the person running it. passwd uses this to modify system files.

Setgid (s in group execute): Similar to setuid but for the group. On directories, new files inherit the directory's group.

Sticky bit (t in others execute): On directories, prevents users from deleting files they don't own. The /tmp directory uses this — everyone can create files there, but only the owner can delete their own.

-rwsr-xr-x   ← setuid (note the s in owner execute)
drwxrwsr-x   ← setgid on a directory
drwxrwxrwt   ← sticky bit (note the t)

Try It Yourself

cd ~/bash-lessons
mkdir -p lesson05
cd lesson05

  1. Create a file called public.txt with some content. Set its permissions so everyone can read it but only you can write to it. Verify with ls -la.

  2. Create a script called hello.sh that contains #!/bin/bash and echo "Hello!". Try to run it with ./hello.sh. Fix the permission error and run it again.

  3. Create a directory called private. Set its permissions to 700. Verify that ls -la shows rwx------.

  4. Create a file called readonly.txt. Remove your own write permission with chmod u-w readonly.txt. Try to append to it with echo "test" >> readonly.txt. What happens? Restore write permission.

  5. Check the permissions on /etc/passwd and /etc/shadow using ls -la. Notice the difference — one is world-readable, the other is not. This is by design.

  6. Run umask to see your current default. Create a new file and directory and verify their permissions match what you'd expect from the umask.

Key Takeaways

  • Every file has three sets of permissions (owner, group, others) with three types each (read, write, execute).
  • chmod changes permissions. Use symbolic mode (chmod u+x) for quick changes, numeric mode (chmod 755) when you want to set everything at once.
  • chmod +x script.sh is what you'll use most often — making scripts executable.
  • chown changes ownership. Usually requires sudo.
  • "Permission denied" means you lack a specific permission. Check with ls -la and fix with chmod.
  • Use sudo sparingly and intentionally. Don't make it a habit.
  • 755 for directories and scripts, 644 for regular files, 600 for sensitive files — these three patterns cover most situations.

Next up: Lesson 06 — Your First Bash Script

Lesson 06: Your First Bash Script

Time: ~30 minutes

From Commands to Scripts

Up to now, you've been typing commands one at a time. That's fine for quick tasks, but as soon as you find yourself running the same sequence of commands more than twice, it's time to put them in a script.

A bash script is just a text file containing commands. When you run it, bash executes each line in order — exactly as if you'd typed them yourself. That's all a script is. No compilation, no special tooling. Write it, make it executable, run it.

Anatomy of a Script

Create your first script:

cd ~/bash-lessons
mkdir -p lesson06
cd lesson06

Open a new file in your editor:

nano greet.sh

Type this:

#!/bin/bash

# A simple greeting script
echo "Hello! Today is $(date '+%A, %B %d, %Y')."
echo "You are logged in as: $USER"
echo "Your current directory is: $PWD"

Save and exit (Ctrl + O, then Ctrl + X in nano).

Now make it executable and run it:

chmod +x greet.sh
./greet.sh

Let's break down what's happening.

The Shebang Line

#!/bin/bash

The first line of every bash script. The #! (called a "shebang" or "hashbang") tells the system which interpreter to use. /bin/bash means "run this file using bash." Without this line, the system might try to interpret your script with a different shell.

Some systems have bash at /usr/bin/bash. A more portable alternative is:

#!/usr/bin/env bash

This finds bash wherever it's installed. Use either one — both work. Just pick one and be consistent.

Comments

# This is a comment

Lines starting with # (other than the shebang) are comments. Bash ignores them. Write comments to explain why you're doing something, not what you're doing — the code already shows the what.

Running the Script

The ./ prefix tells bash to look for the script in the current directory. Without it, bash searches your PATH (a list of standard directories) and won't find your script.

Alternatively, you can run a script without making it executable:

bash greet.sh

This explicitly tells bash to interpret the file. It works, but making scripts executable with chmod +x is the standard practice.

Variables

Variables store values. In bash, you create them with = and access them with $.

#!/bin/bash

name="Kevin"
project="Project 412"
count=42

echo "Name: $name"
echo "Project: $project"
echo "Count: $count"

Critical Rules for Variables

No spaces around the = sign. This is the number one mistake beginners make.

name="Kevin"       # correct
name = "Kevin"     # WRONG — bash thinks "name" is a command with arguments "=" and "Kevin"

Use quotes when the value contains spaces:

greeting="Hello World"     # correct
greeting=Hello World       # WRONG — "World" becomes a separate command

Use $ to read a variable, nothing to set it:

city="Perth"          # setting — no $
echo "$city"          # reading — use $
echo "I live in $city"

Curly Braces for Clarity

When a variable name could be ambiguous, use ${variable}:

file="report"
echo "${file}_final.txt"    # report_final.txt
echo "$file_final.txt"      # WRONG — looks for variable named "file_final"

The curly braces tell bash exactly where the variable name ends.

Variable Naming Conventions

  • Use lowercase for local variables in scripts: filename, count, output_dir
  • Use UPPERCASE for environment variables and constants: PATH, HOME, MAX_RETRIES
  • Use underscores to separate words: log_file, not logfile or logFile

Environment Variables

Some variables are already set by the system. These are called environment variables:

echo "$HOME"        # your home directory
echo "$USER"        # your username
echo "$PATH"        # directories bash searches for commands
echo "$SHELL"       # your default shell
echo "$PWD"         # current working directory
echo "$HOSTNAME"    # your computer's name

You can create your own environment variables with export:

export API_KEY="sk-abc123"

The export keyword makes the variable available to child processes — programs you launch from this shell. Without export, the variable only exists in the current shell.

Command Substitution

You can capture the output of a command and use it as a value:

current_date=$(date '+%Y-%m-%d')
file_count=$(ls | wc -l)
my_ip=$(curl -s ifconfig.me)

echo "Date: $current_date"
echo "Files in current directory: $file_count"
echo "My IP: $my_ip"

The $(command) syntax runs the command and substitutes its output. You'll see an older syntax using backticks `command` — it does the same thing but is harder to read and can't be nested. Use $().

Reading User Input

The read command waits for the user to type something:

#!/bin/bash

echo "What is your name?"
read name
echo "Hello, $name!"

read with a Prompt

read -p "Enter your name: " name
echo "Hello, $name!"

The -p flag displays a prompt on the same line.

read with a Default Value

read -p "Enter port (default 8080): " port
port=${port:-8080}
echo "Using port: $port"

The ${variable:-default} syntax means "use the variable's value, but if it's empty, use this default instead." This is one of the most useful patterns in bash scripting.

Reading Sensitive Input

read -sp "Enter password: " password
echo    # print a newline since -s suppresses it
echo "Password is ${#password} characters long."

The -s flag hides what the user types (for passwords). ${#variable} gives the length of a string.

Script Arguments

Instead of asking for input interactively, you can pass arguments when running the script:

./myscript.sh arg1 arg2 arg3

Inside the script, these are accessed with special variables:

#!/bin/bash

echo "Script name: $0"
echo "First argument: $1"
echo "Second argument: $2"
echo "All arguments: $@"
echo "Number of arguments: $#"

Save this as args.sh, make it executable, and try:

./args.sh hello world

A Practical Example

#!/bin/bash

# backup.sh — create a timestamped backup of a file
# Usage: ./backup.sh filename

if [ $# -eq 0 ]; then
    echo "Usage: $0 filename"
    echo "Creates a timestamped backup of the specified file."
    exit 1
fi

source_file="$1"
timestamp=$(date '+%Y%m%d_%H%M%S')
backup_file="${source_file}.backup_${timestamp}"

cp "$source_file" "$backup_file"
echo "Backed up: $source_file → $backup_file"

Don't worry about the if statement — that's next lesson. Just note the pattern: check that arguments were provided, use them as variables, do the work.

Quoting: Singles, Doubles, and None

Quoting in bash is a source of subtle bugs. Here are the rules:

Double Quotes — Variable Expansion

name="Kevin"
echo "Hello, $name"       # Hello, Kevin — variables are expanded
echo "Path is $HOME"      # Path is /Users/kevin
echo "Date: $(date)"      # Date: Thu Jan 15... — command substitution works

Double quotes preserve spaces and expand variables. Use double quotes around variables as a default habit:

filename="my file.txt"
cat "$filename"            # correct — treats "my file.txt" as one argument
cat $filename              # WRONG — bash sees two arguments: "my" and "file.txt"

Single Quotes — Literal Text

echo 'Hello, $name'       # Hello, $name — the literal text, no expansion
echo 'Cost is $5.00'      # Cost is $5.00 — useful when you want literal dollar signs

Single quotes prevent all expansion. What you type is exactly what you get.

No Quotes — Word Splitting

files="file1.txt file2.txt"
ls $files                  # works — bash splits into two arguments
ls "$files"                # fails — bash treats the whole string as one filename

Without quotes, bash splits on whitespace and expands wildcards. This is sometimes useful but usually a source of bugs. The safe default is to always quote your variables unless you specifically want word splitting.

Exit Codes

Every command returns a number when it finishes: 0 means success, anything else means failure.

ls /tmp
echo $?        # 0 — success

ls /nonexistent
echo $?        # 2 (or non-zero) — failure

The $? variable holds the exit code of the last command. You can set your own in scripts:

#!/bin/bash

if [ -f "$1" ]; then
    echo "File exists."
    exit 0
else
    echo "File not found."
    exit 1
fi

Exit codes matter because they let other tools and scripts check whether your script succeeded.

Try It Yourself

  1. Create a script called sysinfo.sh that prints:

    • Your username
    • Your home directory
    • The current date and time
    • The number of files in your home directory Make it executable and run it.

  2. Create a script called greet.sh that takes a name as an argument and prints "Hello, [name]! Welcome." If no name is given, print a usage message and exit with code 1.

  3. Create a script called mkproject.sh that:

    • Takes a project name as an argument
    • Creates a directory with that name
    • Creates README.md, notes.txt, and a src/ subdirectory inside it
    • Prints a confirmation message

  4. Create a script that uses read -p to ask for a city name, then prints "You chose [city]!" Use a default of "Perth" if nothing is entered.

  5. Experiment with quoting. Create a variable containing a filename with spaces. Try accessing it with and without double quotes to see the difference.

Key Takeaways

  • A bash script is a text file with a #!/bin/bash shebang, made executable with chmod +x.
  • Variables are set with name=value (no spaces!) and read with $name.
  • Always quote your variables: "$variable". This prevents word-splitting bugs.
  • $(command) captures command output. Use it to store dates, counts, or any dynamic value.
  • $1, $2, etc. are script arguments. $# is the argument count. $@ is all arguments.
  • read -p gets user input. ${var:-default} provides fallback values.
  • Exit codes (exit 0 for success, exit 1 for failure) let other tools know if your script worked.

Next up: Lesson 07 — Conditionals and Logic

Lesson 07: Conditionals and Logic

Time: ~30 minutes

Making Decisions

So far, your scripts have been straight-line sequences: do this, then this, then this. Real scripts need to make decisions. Does a file exist before trying to read it? Did a command succeed or fail? Did the user provide the right number of arguments?

This lesson teaches you how to branch — how to make your scripts do different things based on conditions.

The if Statement

The basic structure:

if [ condition ]; then
    # commands to run if condition is true
fi

A concrete example:

#!/bin/bash

if [ -f "config.txt" ]; then
    echo "Config file found."
fi

The -f test checks whether a file exists and is a regular file. The [ ... ] is actually a command (an alias for the test command), and the spaces inside the brackets are mandatory.

if/else

if [ -f "config.txt" ]; then
    echo "Loading configuration..."
    source config.txt
else
    echo "No config file found. Using defaults."
fi

if/elif/else

if [ "$1" = "start" ]; then
    echo "Starting service..."
elif [ "$1" = "stop" ]; then
    echo "Stopping service..."
elif [ "$1" = "status" ]; then
    echo "Checking status..."
else
    echo "Usage: $0 {start|stop|status}"
    exit 1
fi

Note: elif is bash's "else if." You can chain as many as you need.

Test Expressions

The [ ... ] brackets support many different tests. Here are the ones you'll use constantly.

File Tests

Test True when...
-f file File exists and is a regular file
-d dir Directory exists
-e path Path exists (file or directory)
-r file File is readable
-w file File is writable
-x file File is executable
-s file File exists and is not empty
-L file File is a symbolic link 
if [ -d "$HOME/projects" ]; then
    echo "Projects directory exists."
fi

if [ ! -f "output.log" ]; then
    echo "No log file yet."
fi

The ! negates a test — "if NOT."

String Comparisons

Test True when...
"$a" = "$b" Strings are equal
"$a" != "$b" Strings are not equal
-z "$a" String is empty (zero length)
-n "$a" String is not empty 
if [ "$USER" = "root" ]; then
    echo "Running as root — be careful!"
fi

if [ -z "$1" ]; then
    echo "No argument provided."
    exit 1
fi

Always quote your variables inside [ ... ]. Without quotes, an empty variable causes a syntax error:

name=""
[ $name = "Kevin" ]    # ERROR: [ = "Kevin" ] — bash is confused
[ "$name" = "Kevin" ]  # Works fine: [ "" = "Kevin" ] — evaluates to false

Numeric Comparisons

Numbers use different operators than strings:

Test Meaning
$a -eq $b Equal
$a -ne $b Not equal
$a -lt $b Less than
$a -le $b Less than or equal
$a -gt $b Greater than
$a -ge $b Greater than or equal 
count=$(ls | wc -l)

if [ "$count" -gt 100 ]; then
    echo "That's a lot of files: $count"
elif [ "$count" -gt 10 ]; then
    echo "Moderate number of files: $count"
else
    echo "Just a few files: $count"
fi

Why -eq instead of =? Because = does string comparison. The string "10" comes before "9" alphabetically, but numerically 10 is greater than 9. The -eq family does proper number comparison.

Double Brackets: [[ ... ]]

Bash has an improved test syntax with double brackets. It's more forgiving and supports more features:

if [[ "$name" = "Kevin" ]]; then
    echo "Hello Kevin"
fi

Advantages of [[ ... ]]

Pattern matching:

if [[ "$filename" == *.txt ]]; then
    echo "It's a text file."
fi

Regex matching:

if [[ "$email" =~ ^[a-zA-Z]+@[a-zA-Z]+\.[a-zA-Z]+$ ]]; then
    echo "Looks like a valid email."
fi

No word-splitting issues:

name=""
if [[ $name = "Kevin" ]]; then    # works even without quotes
    echo "Hi"
fi

Logical operators inside the brackets:

if [[ "$age" -ge 18 && "$age" -le 65 ]]; then
    echo "Working age."
fi

As a rule: use [[ ... ]] in bash scripts. Use [ ... ] only when you need strict POSIX compatibility (which is rare in practice).

Logical Operators

Inside [[ ... ]]

if [[ "$a" -gt 0 && "$a" -lt 100 ]]; then     # AND
    echo "Between 1 and 99"
fi

if [[ "$day" = "Saturday" || "$day" = "Sunday" ]]; then   # OR
    echo "Weekend!"
fi

if [[ ! -f "lock.file" ]]; then                # NOT
    echo "No lock file — safe to proceed."
fi

Combining [ ... ] with External Operators

With single brackets, you can't use && and || inside. Use -a and -o instead, or combine multiple bracket expressions:

# These two are equivalent:
if [ "$a" -gt 0 ] && [ "$a" -lt 100 ]; then
    echo "In range"
fi

# -a and -o work inside single brackets but are considered outdated:
if [ "$a" -gt 0 -a "$a" -lt 100 ]; then
    echo "In range"
fi

Another reason to prefer [[ ... ]].

Short-Circuit Evaluation

&& and || can be used outside of if statements for quick conditional execution:

# Run the second command only if the first succeeds
mkdir -p backups && echo "Backup directory ready."

# Run the second command only if the first fails
[ -f "config.txt" ] || echo "Warning: config not found!"

# A common pattern for validation
[ -z "$1" ] && echo "Usage: $0 filename" && exit 1

This is useful for one-liners but can get hard to read for complex logic. Use proper if statements when clarity matters.

The case Statement

When you're comparing one variable against many possible values, case is cleaner than a chain of elif:

#!/bin/bash

case "$1" in
    start)
        echo "Starting the service..."
        ;;
    stop)
        echo "Stopping the service..."
        ;;
    restart)
        echo "Restarting..."
        ;;
    status)
        echo "Service is running."
        ;;
    *)
        echo "Usage: $0 {start|stop|restart|status}"
        exit 1
        ;;
esac

Each pattern ends with ). Each block ends with ;;. The * pattern matches anything not matched above — it's the default case. The whole thing ends with esac ("case" backwards).

Pattern Matching in case

case "$filename" in
    *.txt)
        echo "Text file"
        ;;
    *.jpg|*.png|*.gif)
        echo "Image file"
        ;;
    *.sh)
        echo "Shell script"
        ;;
    *)
        echo "Unknown type"
        ;;
esac

The | lets you match multiple patterns for the same block.

Arithmetic with (( ... ))

For numeric operations and comparisons, double parentheses give you a more natural syntax:

x=10
y=3

if (( x > y )); then
    echo "$x is greater than $y"
fi

if (( x % 2 == 0 )); then
    echo "$x is even"
fi

(( count++ ))           # increment
(( total = x + y ))     # arithmetic assignment
echo "$total"           # 13

Inside (( ... )), you don't need $ before variable names (though it still works). You can use familiar operators: >, <, >=, <=, ==, !=, %, +, -, *, /.

Putting It Together: A Practical Script

#!/bin/bash

# deploy.sh — a simple deployment checker
# Usage: ./deploy.sh [environment]

environment="${1:-staging}"

echo "=== Deployment Check for: $environment ==="

# Validate environment
case "$environment" in
    staging|production|development)
        echo "Environment: $environment — valid."
        ;;
    *)
        echo "Error: Unknown environment '$environment'."
        echo "Valid options: staging, production, development"
        exit 1
        ;;
esac

# Check prerequisites
if [[ ! -f "app.conf" ]]; then
    echo "Error: app.conf not found. Cannot deploy."
    exit 1
fi

if [[ "$environment" = "production" ]]; then
    read -p "You're deploying to PRODUCTION. Are you sure? (yes/no): " confirm
    if [[ "$confirm" != "yes" ]]; then
        echo "Deployment cancelled."
        exit 0
    fi
fi

echo "All checks passed. Ready to deploy to $environment."

Try It Yourself

  1. Write a script called filecheck.sh that takes a filename as an argument and reports whether it exists, whether it's a file or directory, and whether it's readable, writable, and executable.

  2. Write a script called age.sh that asks the user for their age and responds differently based on the value (under 18, 18-65, over 65). Handle non-numeric input gracefully.

  3. Write a script called extension.sh that takes a filename as an argument and uses a case statement to print what type of file it is based on the extension (.txt, .sh, .py, .jpg, etc.).

  4. Write a script that checks if a command-line tool is installed (like git, python3, or docker) using command -v toolname and reports whether it's available or not.

  5. Modify one of your earlier scripts to validate its arguments properly — check that the right number of arguments were given and that any files referenced actually exist.

Key Takeaways

  • if [ condition ]; then ... fi is the basic branching structure. Add elif and else as needed.
  • Use [[ ... ]] in bash scripts — it's safer and more powerful than [ ... ].
  • File tests (-f, -d, -e, -r, -w, -x) are how you check the filesystem.
  • String comparison uses = and !=. Numeric comparison uses -eq, -ne, -lt, -gt, etc.
  • Always quote variables inside tests: [[ "$var" = "value" ]].
  • case statements are cleaner than long elif chains when matching one variable against many values.
  • (( ... )) gives you natural arithmetic syntax.
  • && and || let you do quick conditional execution without a full if statement.

Next up: Lesson 08 — Loops and Iteration

Lesson 08: Loops and Iteration

Time: ~30 minutes

Why Loops Matter

You have 200 image files to rename. Or 50 servers to check. Or a log file you need to process line by line. Doing any of this manually would be tedious and error-prone. Loops let you say "do this thing for each item in this list" and walk away.

Loops are where scripting starts to save you real time. A task that would take 20 minutes by hand takes 3 seconds in a loop.

The for Loop

The for loop iterates over a list of items:

for name in Alice Bob Charlie; do
    echo "Hello, $name!"
done

Output:

Hello, Alice!
Hello, Bob!
Hello, Charlie!

The variable name takes each value in turn. The code between do and done runs once per value.

Looping Over Files

This is the most common use of for loops:

for file in *.txt; do
    echo "Processing: $file"
done

The *.txt glob expands to a list of all .txt files in the current directory. Each one gets assigned to $file in turn.

Looping Over Command Output

for user in $(cat users.txt); do
    echo "Setting up account for: $user"
done

$(cat users.txt) expands to the contents of the file, split on whitespace. Each word becomes one iteration.

A word of caution: this splits on all whitespace, including spaces within lines. If your file has lines with spaces, use a while read loop instead (covered below).

Looping Over a Range of Numbers

for i in {1..10}; do
    echo "Iteration $i"
done

Brace expansion generates the sequence. You can also specify a step:

for i in {0..100..5}; do
    echo "$i"
done

This counts from 0 to 100 in steps of 5.

C-style for Loop

If you're coming from a C, Java, or JavaScript background, this syntax will feel familiar:

for (( i=1; i<=10; i++ )); do
    echo "Count: $i"
done

This is useful when you need precise control over the counter.

The while Loop

A while loop runs as long as its condition is true:

count=1
while [ $count -le 5 ]; do
    echo "Count: $count"
    (( count++ ))
done

This counts from 1 to 5. The (( count++ )) increments the counter each time. Without it, you'd have an infinite loop.

Reading Files Line by Line

This is the correct way to process a file line by line, preserving spaces and special characters:

while IFS= read -r line; do
    echo "Line: $line"
done < input.txt

Breaking this down:

  • IFS= prevents leading/trailing whitespace from being stripped
  • read -r reads one line, -r prevents backslash interpretation
  • done < input.txt feeds the file into the loop's stdin

This is safer than for line in $(cat file) because it handles spaces, tabs, and special characters correctly.

Processing a File with Line Numbers

line_number=0
while IFS= read -r line; do
    (( line_number++ ))
    echo "$line_number: $line"
done < data.txt

Reading from a Pipe

grep "ERROR" server.log | while IFS= read -r line; do
    echo "Found error: $line"
done

A subtle gotcha: when you pipe into while, the loop runs in a subshell. Variables set inside the loop won't be available after the loop ends. If you need variables to persist, use process substitution instead:

count=0
while IFS= read -r line; do
    (( count++ ))
done < <(grep "ERROR" server.log)
echo "Found $count errors"   # this works because of < <(...)

The until Loop

until is the inverse of while — it runs as long as the condition is false:

count=1
until [ $count -gt 5 ]; do
    echo "Count: $count"
    (( count++ ))
done

This produces the same output as the while example above. Use whichever reads more naturally for your situation. In practice, while is used far more often.

Waiting for Something

until shines when you're waiting for a condition to become true:

echo "Waiting for server.lock to disappear..."
until [ ! -f "server.lock" ]; do
    sleep 1
done
echo "Lock file gone. Proceeding."

Loop Control

break — Exit the Loop Early

for file in *.log; do
    if [ ! -r "$file" ]; then
        echo "Cannot read $file — stopping."
        break
    fi
    echo "Processing $file"
done

break immediately exits the nearest enclosing loop.

continue — Skip to the Next Iteration

for file in *; do
    if [ -d "$file" ]; then
        continue    # skip directories
    fi
    echo "File: $file"
done

continue skips the rest of the current iteration and moves to the next one.

Practical Loop Patterns

Batch Rename Files

for file in *.jpeg; do
    mv "$file" "${file%.jpeg}.jpg"
done

The ${file%.jpeg} syntax removes .jpeg from the end of the variable. So photo.jpeg becomes photo, and then .jpg is appended to get photo.jpg.

Process CSV Data

while IFS=',' read -r name email role; do
    echo "Name: $name, Email: $email, Role: $role"
done < employees.csv

Setting IFS=',' tells read to split on commas instead of whitespace. Each comma-separated value goes into its own variable.

Create Multiple Directories

for month in {01..12}; do
    mkdir -p "2025/$month"
done

Check Multiple Servers

for server in web1 web2 web3 db1 db2; do
    if ping -c 1 -W 2 "$server" > /dev/null 2>&1; then
        echo "$server: UP"
    else
        echo "$server: DOWN"
    fi
done

Find and Process Specific Files

find . -name "*.tmp" -mtime +7 | while IFS= read -r file; do
    echo "Deleting old temp file: $file"
    rm "$file"
done

This finds all .tmp files older than 7 days and deletes them.

Retry Logic

max_attempts=5
attempt=1

while [ $attempt -le $max_attempts ]; do
    echo "Attempt $attempt of $max_attempts..."
    
    if curl -s -o /dev/null -w "%{http_code}" https://example.com | grep -q "200"; then
        echo "Success!"
        break
    fi
    
    echo "Failed. Waiting 5 seconds..."
    sleep 5
    (( attempt++ ))
done

if [ $attempt -gt $max_attempts ]; then
    echo "All $max_attempts attempts failed."
    exit 1
fi

Nested Loops

Loops can contain other loops:

for dir in project1 project2 project3; do
    echo "=== $dir ==="
    for file in "$dir"/*.txt; do
        echo "  Found: $file"
    done
done

Use nested loops sparingly. If you're going more than two levels deep, consider whether there's a simpler approach (like find).

The select Loop — Simple Menus

select creates an interactive numbered menu:

echo "Choose an environment:"
select env in development staging production quit; do
    case "$env" in
        development|staging|production)
            echo "Deploying to $env..."
            break
            ;;
        quit)
            echo "Goodbye."
            exit 0
            ;;
        *)
            echo "Invalid option. Try again."
            ;;
    esac
done

This displays a numbered list and waits for input. It loops until break or exit is called.

Common Mistakes

Forgetting quotes around variables with spaces:

# WRONG — breaks on filenames with spaces
for file in $(ls); do ...

# RIGHT
for file in *; do ...

Modifying a list while iterating over it: Don't delete files from a directory while looping over that directory's contents with a glob. Collect the filenames first, then delete.

Infinite loops without an exit: Always make sure your while condition will eventually become false, or include a break.

Using for to read lines from a file:

# WRONG — splits on all whitespace, not just newlines
for line in $(cat file.txt); do ...

# RIGHT
while IFS= read -r line; do ... done < file.txt

Try It Yourself

  1. Write a loop that creates files day01.txt through day31.txt.

  2. Write a script that takes a directory as an argument and counts how many files vs directories are inside it (one level only, not recursive).

  3. Write a script that reads a file of names (one per line) and prints a greeting for each one.

  4. Write a retry loop that tries to create a directory and retries up to 3 times with a 2-second delay between attempts (simulate failure by trying to create a directory in a location that doesn't exist).

  5. Write a script that loops through all .sh files in a directory and reports which ones are executable and which aren't.

  6. Use a select menu to let the user choose between three options and display a different message for each.

Key Takeaways

  • for item in list iterates over a list. Use it for files (*.txt), sequences ({1..10}), and explicit lists.
  • while [ condition ] loops as long as the condition is true. Use it for counters, retries, and reading files.
  • while IFS= read -r line; do ... done < file is the correct way to read a file line by line.
  • break exits a loop. continue skips to the next iteration.
  • Always quote your variables in loops: "$file" not $file.
  • The ${variable%pattern} syntax strips text from the end of a variable — essential for renaming.
  • Don't use for line in $(cat file) — it breaks on spaces. Use while read instead.

Next up: Lesson 09 — Functions and Script Organisation

Lesson 09: Functions and Script Organisation

Time: ~30 minutes

Why Functions?

As your scripts grow, they get harder to read, harder to debug, and harder to maintain. Functions solve this by letting you name a block of code and reuse it. Instead of a 200-line script that does everything in sequence, you get small, named pieces that each do one thing.

Functions also make your scripts self-documenting. When the main body of your script reads validate_input, create_backup, deploy_files, you can understand what it does without reading every line.

Defining and Calling Functions

The basic syntax:

greet() {
    echo "Hello, World!"
}

# Call it
greet

That's it. Define the function (name, parentheses, curly braces), then call it by name. The parentheses in the definition are always empty — they're just syntax, not a parameter list.

You can also write it with the function keyword:

function greet {
    echo "Hello, World!"
}

Both forms work. The first (without function) is more portable and more common. Pick one and be consistent.

Important: Functions must be defined before they're called. Bash reads scripts top to bottom, so define your functions at the top.

Function Arguments

Functions receive arguments the same way scripts do — through positional parameters $1, $2, etc.:

greet() {
    local name="$1"
    echo "Hello, $name!"
}

greet "Kevin"
greet "Alice"

Inside the function, $1 refers to the first argument passed to the function, not to the script. $@ is all arguments, $# is the count.

log_message() {
    local level="$1"
    shift                  # remove the first argument
    local message="$@"     # everything remaining
    echo "[$(date '+%H:%M:%S')] [$level] $message"
}

log_message "INFO" "Server started successfully"
log_message "ERROR" "Connection to database failed"

The shift command removes the first positional parameter and shifts the rest down — $2 becomes $1, $3 becomes $2, and so on. It's useful when the first argument is a flag or category and the rest is variable-length data.

Local Variables

By default, variables in bash are global — they're visible everywhere in the script. Inside functions, this causes problems:

set_name() {
    name="Alice"       # this modifies the global "name"
}

name="Kevin"
set_name
echo "$name"           # prints "Alice" — the function changed it!

The local keyword restricts a variable to the function:

set_name() {
    local name="Alice"   # only exists inside this function
    echo "Inside: $name"
}

name="Kevin"
set_name
echo "Outside: $name"   # prints "Kevin" — unchanged

Rule of thumb: Always use local for variables inside functions. The only exception is when you intentionally want to modify a global variable.

Return Values

Functions can return an exit code (a number from 0 to 255):

is_even() {
    local num="$1"
    if (( num % 2 == 0 )); then
        return 0     # success = true
    else
        return 1     # failure = false
    fi
}

if is_even 4; then
    echo "4 is even"
fi

if ! is_even 7; then
    echo "7 is odd"
fi

return 0 means success (true). return 1 means failure (false). This mirrors how all Unix commands work — 0 is success, non-zero is failure.

Returning Data (Not Just Status)

Since return only handles numbers 0-255, you can't use it to return strings or large numbers. Instead, use echo and capture the output:

get_extension() {
    local filename="$1"
    echo "${filename##*.}"
}

ext=$(get_extension "report.pdf")
echo "Extension is: $ext"    # pdf

The function's echo output gets captured by $(...). This is the standard pattern for functions that produce data.

You can also use a global variable, though it's less clean:

get_extension() {
    RESULT="${1##*.}"
}

get_extension "report.pdf"
echo "Extension is: $RESULT"

Prefer the echo-and-capture pattern. It's more explicit and doesn't rely on side effects.

Parameter Validation

Good functions check their inputs:

create_backup() {
    local source="$1"
    local dest_dir="$2"
    
    if [[ -z "$source" || -z "$dest_dir" ]]; then
        echo "Error: create_backup requires source and destination" >&2
        return 1
    fi
    
    if [[ ! -f "$source" ]]; then
        echo "Error: Source file '$source' not found" >&2
        return 1
    fi
    
    if [[ ! -d "$dest_dir" ]]; then
        mkdir -p "$dest_dir"
    fi
    
    local timestamp=$(date '+%Y%m%d_%H%M%S')
    local basename=$(basename "$source")
    cp "$source" "$dest_dir/${basename}.${timestamp}.bak"
    echo "Backed up: $source → $dest_dir/${basename}.${timestamp}.bak"
}

Notice the >&2 on error messages — this sends them to stderr instead of stdout, so they don't interfere with output that might be captured.

Organising a Script

Here's a structure that works well for scripts of any size:

#!/bin/bash
#
# deploy.sh — Deploy the application to a target environment
# Usage: ./deploy.sh <environment> [version]
#

# --- Configuration ---
readonly APP_NAME="myapp"
readonly LOG_DIR="/var/log/$APP_NAME"
readonly DEFAULT_VERSION="latest"

# --- Functions ---

usage() {
    echo "Usage: $0 <environment> [version]"
    echo ""
    echo "Environments: development, staging, production"
    echo "Version defaults to '$DEFAULT_VERSION' if not specified."
    exit 1
}

log() {
    local level="$1"
    shift
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*"
}

validate_environment() {
    local env="$1"
    case "$env" in
        development|staging|production) return 0 ;;
        *) return 1 ;;
    esac
}

check_prerequisites() {
    local missing=0
    
    for cmd in git docker curl; do
        if ! command -v "$cmd" > /dev/null 2>&1; then
            log "ERROR" "Required command not found: $cmd"
            (( missing++ ))
        fi
    done
    
    return "$missing"
}

deploy() {
    local env="$1"
    local version="$2"
    
    log "INFO" "Deploying $APP_NAME version $version to $env"
    log "INFO" "Deployment complete."
}

# --- Main ---

main() {
    local environment="${1:-}"
    local version="${2:-$DEFAULT_VERSION}"
    
    if [[ -z "$environment" ]]; then
        usage
    fi
    
    if ! validate_environment "$environment"; then
        log "ERROR" "Invalid environment: $environment"
        usage
    fi
    
    if ! check_prerequisites; then
        log "ERROR" "Missing prerequisites. Aborting."
        exit 1
    fi
    
    deploy "$environment" "$version"
}

main "$@"

The Pattern Explained

  1. Header comment — script name, purpose, and usage
  2. Configuration — constants at the top, using readonly
  3. Functions — each one does one thing, defined before use
  4. main function — the entry point, contains the script's logic flow
  5. main "$@" — the only line at the top level, passing all script arguments to main

This pattern is used by professionals for a reason: it's readable, testable, and maintainable. The main function isn't strictly necessary for small scripts, but it becomes valuable as scripts grow.

Sourcing Files

As scripts get larger, you can split functions into separate files and include them:

# utils.sh
log() {
    echo "[$(date '+%H:%M:%S')] $*"
}

validate_file() {
    [[ -f "$1" ]]
}

# main.sh
#!/bin/bash
source ./utils.sh    # or: . ./utils.sh

log "Starting up"
if validate_file "config.txt"; then
    log "Config found"
fi

source (or its shorthand .) executes the contents of a file in the current shell. Functions and variables defined in the sourced file become available immediately.

This is how you build a library of reusable functions across multiple scripts.

Useful String Manipulations for Functions

You'll use these inside functions constantly:

Extracting Parts of Strings

filepath="/home/kevin/documents/report.pdf"

echo "${filepath##*/}"       # report.pdf   (basename — everything after last /)
echo "${filepath%/*}"        # /home/kevin/documents  (dirname — everything before last /)
echo "${filepath##*.}"       # pdf           (extension — everything after last .)
echo "${filepath%.pdf}"      # /home/kevin/documents/report  (remove extension)

The ## removes the longest match from the front. The % removes the shortest match from the end. %% removes the longest match from the end. # removes the shortest match from the front.

Replacing Text in Variables

text="Hello World"
echo "${text/World/Bash}"     # Hello Bash  (replace first occurrence)
echo "${text//l/L}"           # HeLLo WorLd (replace all occurrences)

Uppercasing and Lowercasing

name="kevin"
echo "${name^}"       # Kevin   (capitalise first letter)
echo "${name^^}"      # KEVIN   (all uppercase)

SHOUT="HELLO"
echo "${SHOUT,,}"     # hello   (all lowercase)

Default Values

echo "${name:-Anonymous}"     # use "Anonymous" if name is unset or empty
echo "${name:=Anonymous}"     # same, but also assigns the default to name
echo "${name:+Found}"         # prints "Found" if name IS set (otherwise nothing)

Try It Yourself

  1. Write a function file_info that takes a filename and prints its size, line count, and permissions on separate lines. Call it on several files.

  2. Write a logging library: create a file called logger.sh with functions log_info, log_warn, and log_error that print timestamped, levelled messages. Source it from another script and use it.

  3. Write a function confirm that asks the user a yes/no question and returns 0 for yes, 1 for no. Use it like: if confirm "Deploy to production?"; then ....

  4. Refactor one of your earlier scripts (from any previous lesson) to use functions. Extract repeated logic, add a usage function, and structure it with the pattern shown above.

  5. Write a function sanitise_filename that takes a string and replaces spaces with underscores, removes special characters, and converts to lowercase. Test it with various inputs.

Key Takeaways

  • Functions are defined with name() { ... } and called by name. They must be defined before they're called.
  • Functions receive arguments via $1, $2, etc. — the same as scripts.
  • Always use local for variables inside functions to avoid polluting the global scope.
  • return sets an exit status (0-255). To return data, echo it and capture with $(function_name).
  • Send error messages to stderr with >&2.
  • The standard script structure is: header, constants, functions, main function, main "$@".
  • source includes other files, letting you build reusable function libraries.
  • Bash has built-in string manipulation (${var##pattern}, ${var%pattern}, ${var/old/new}) that's faster than calling external tools.

Next up: Lesson 10 — Text Processing Power Tools

Lesson 10: Text Processing Power Tools

Time: ~30 minutes

Beyond grep

You already know grep for finding lines that match a pattern. But bash has an entire toolkit for transforming, extracting, and reshaping text. These tools date back to the 1970s, but they're still used daily by developers and sysadmins because they work, they're fast, and they compose beautifully with pipes.

This lesson covers cut, tr, sort, uniq, sed, and awk — the workhorses of text processing.

Setup

Let's create some data to practice with:

cd ~/bash-lessons
mkdir -p lesson10
cd lesson10

cat > employees.csv << 'EOF'
id,name,department,salary,start_date
1,Alice Johnson,Engineering,95000,2021-03-15
2,Bob Smith,Marketing,72000,2020-07-01
3,Charlie Brown,Engineering,88000,2022-01-10
4,Diana Prince,Sales,78000,2019-11-20
5,Eve Williams,Engineering,102000,2018-05-01
6,Frank Castle,Marketing,68000,2023-02-14
7,Grace Hopper,Engineering,115000,2017-09-01
8,Henry Ford,Sales,82000,2021-06-15
9,Iris Chang,Marketing,75000,2022-08-01
10,Jack Ryan,Sales,91000,2020-01-10
EOF

cat > access.log << 'EOF'
192.168.1.10 - - [15/Jan/2025:10:23:45] "GET /index.html" 200 1024
192.168.1.15 - - [15/Jan/2025:10:23:46] "GET /about.html" 200 2048
192.168.1.10 - - [15/Jan/2025:10:23:47] "POST /api/login" 401 128
192.168.1.22 - - [15/Jan/2025:10:23:48] "GET /index.html" 200 1024
192.168.1.10 - - [15/Jan/2025:10:23:49] "POST /api/login" 200 256
192.168.1.15 - - [15/Jan/2025:10:23:50] "GET /dashboard" 200 4096
192.168.1.33 - - [15/Jan/2025:10:23:51] "GET /index.html" 404 512
192.168.1.22 - - [15/Jan/2025:10:23:52] "GET /style.css" 200 768
192.168.1.10 - - [15/Jan/2025:10:23:53] "GET /api/data" 200 8192
192.168.1.33 - - [15/Jan/2025:10:23:54] "GET /missing.html" 404 512
EOF

cut — Extract Columns

cut pulls out specific fields from each line.

By Delimiter and Field Number

cut -d',' -f2 employees.csv           # names only
cut -d',' -f2,4 employees.csv         # names and salaries
cut -d',' -f2-4 employees.csv         # names through salaries (range)

-d',' sets the delimiter to comma. -f2 selects field 2.

By Character Position

cut -c1-10 access.log                 # first 10 characters of each line

Practical Use

Pull IP addresses from the access log:

cut -d' ' -f1 access.log

Extract just the department column (skipping the header):

tail -n +2 employees.csv | cut -d',' -f3

tail -n +2 starts from line 2, effectively skipping the header.

tr — Translate Characters

tr replaces or deletes characters. It works on individual characters, not words or patterns.

Replace Characters

echo "Hello World" | tr 'a-z' 'A-Z'      # HELLO WORLD
echo "Hello World" | tr 'A-Z' 'a-z'      # hello world
echo "hello-world" | tr '-' '_'           # hello_world

Squeeze Repeated Characters

echo "too    many    spaces" | tr -s ' '  # too many spaces

-s squeezes consecutive identical characters into one.

Delete Characters

echo "Price: $42.50" | tr -d '$'          # Price: 42.50
echo "Hello123World" | tr -d '0-9'        # HelloWorld

Converting Line Endings

A common use — converting Windows line endings to Unix:

tr -d '\r' < windows_file.txt > unix_file.txt

sort and uniq — Ordering and Deduplication

You've seen these before, but they have more depth than basic usage suggests.

sort Options

sort employees.csv                         # alphabetical (default)
sort -t',' -k4 -n employees.csv           # sort by salary (field 4, numeric)
sort -t',' -k4 -rn employees.csv          # sort by salary, highest first
sort -t',' -k3,3 -k4,4rn employees.csv   # sort by department, then salary descending

-t',' sets the field separator. -k4 sorts by field 4. -n sorts numerically. -r reverses. You can chain -k options for multi-level sorting.

uniq Requires Sorted Input

uniq only removes adjacent duplicates. Always sort first:

# Unique departments
tail -n +2 employees.csv | cut -d',' -f3 | sort | uniq

# Count employees per department
tail -n +2 employees.csv | cut -d',' -f3 | sort | uniq -c | sort -rn

uniq -c prefixes each line with a count. uniq -d shows only duplicates. uniq -u shows only unique lines.

Alternatively, sort -u combines sorting and deduplication in one step:

tail -n +2 employees.csv | cut -d',' -f3 | sort -u

sed — Stream Editor

sed processes text line by line, applying transformations. It's a deep tool — we'll cover the essentials.

Find and Replace

sed 's/Engineering/Eng/' employees.csv          # replace first occurrence per line
sed 's/Engineering/Eng/g' employees.csv         # replace ALL occurrences per line

The s/old/new/ command is sed's bread and butter. The g flag means "global" — replace all matches on the line, not just the first.

Case-Insensitive Replace

sed 's/alice/ALICE/Ig' employees.csv            # I flag for case-insensitive

(The I flag is a GNU sed extension — it works on Linux but not on default macOS sed.)

Delete Lines

sed '1d' employees.csv                          # delete line 1 (the header)
sed '/^$/d' file.txt                            # delete empty lines
sed '/^#/d' config.txt                          # delete comment lines

sed -n '3p' employees.csv                       # print only line 3
sed -n '2,5p' employees.csv                     # print lines 2-5
sed -n '/Engineering/p' employees.csv           # print lines matching a pattern

-n suppresses default output. p explicitly prints matching lines. Without -n, sed prints every line plus the matched lines again.

Multiple Operations

sed -e 's/Engineering/ENG/' -e 's/Marketing/MKT/' -e 's/Sales/SLS/' employees.csv

Or use a semicolon:

sed 's/Engineering/ENG/; s/Marketing/MKT/; s/Sales/SLS/' employees.csv

In-Place Editing

sed -i 's/old/new/g' file.txt             # Linux: edits the file directly
sed -i '' 's/old/new/g' file.txt          # macOS: requires empty string after -i

Always back up before using -i, or use -i.bak to create an automatic backup:

sed -i.bak 's/old/new/g' file.txt        # edits file.txt, saves original as file.txt.bak

Capture Groups

# Swap first and last names
echo "Johnson, Alice" | sed 's/\(.*\), \(.*\)/\2 \1/'
# Output: Alice Johnson

\(...\) captures a group. \1, \2 reference captured groups.

awk — Pattern Scanning and Processing

awk is the most powerful text processing tool in the standard toolkit. It's essentially a small programming language. We'll cover the most useful parts.

Basic Structure

awk 'pattern { action }' file

For each line in the file: if it matches the pattern, execute the action. If no pattern is given, the action runs on every line.

Fields

awk automatically splits each line into fields. By default, it splits on whitespace:

echo "Alice 95000 Engineering" | awk '{ print $1 }'          # Alice
echo "Alice 95000 Engineering" | awk '{ print $3, $1 }'      # Engineering Alice

$1 is the first field, $2 the second, etc. $0 is the entire line. $NF is the last field.

Setting the Field Separator

For CSV data:

awk -F',' '{ print $2 }' employees.csv                       # print names
awk -F',' '{ print $2, $4 }' employees.csv                   # names and salaries

Filtering Rows

awk -F',' '$3 == "Engineering"' employees.csv                 # engineering only
awk -F',' '$4 > 90000' employees.csv                          # salary > 90000
awk -F',' '$4 > 90000 { print $2, $4 }' employees.csv        # names and salaries > 90k

Built-in Variables

Variable Meaning
NR Current line number (row count)
NF Number of fields in current line
FS Field separator (same as -F)
$0 Entire current line 
awk -F',' 'NR > 1 { print NR-1, $2 }' employees.csv         # skip header, number rows
awk '{ print NF, $0 }' access.log                             # show field count per line

Calculations

# Average salary
awk -F',' 'NR > 1 { sum += $4; count++ } END { print "Average:", sum/count }' employees.csv

# Total salary by department
awk -F',' 'NR > 1 { dept[$3] += $4 } END { for (d in dept) print d, dept[d] }' employees.csv

The END block runs after all lines have been processed. Arrays in awk are associative (like dictionaries/hash maps).

Formatted Output

awk -F',' 'NR > 1 { printf "%-20s %s %10s\n", $2, $3, "$"$4 }' employees.csv

printf gives you control over column widths and alignment — same syntax as C's printf.

Combining the Tools

The real power comes from combining these tools in pipelines.

Top 3 Most Active IPs in the Access Log

cut -d' ' -f1 access.log | sort | uniq -c | sort -rn | head -3

List Engineering Employees Sorted by Salary

awk -F',' '$3 == "Engineering" { print $4, $2 }' employees.csv | sort -rn

Find All 404 Errors and the Requested URLs

awk '$9 == "404" { print $7 }' access.log | sort -u

(Field numbers differ here because the access log uses spaces as separators.)

Replace Department Names and Create a New File

sed '1d' employees.csv | awk -F',' '{ gsub(/Engineering/, "ENG", $3); print }' OFS=','

Try It Yourself

  1. Extract all unique IP addresses from access.log and count how many requests each made. Sort by count, highest first.

  2. Calculate the average salary per department from employees.csv using awk.

  3. Use sed to remove the header from employees.csv and replace all commas with tabs. Save to employees.tsv.

  4. Find employees who started in 2021 or later (compare the start_date field) using awk.

  5. From access.log, extract just the HTTP status codes (200, 401, 404) and count how many of each occurred.

  6. Create a pipeline that produces a formatted report showing each department, the number of employees, and the total salary spend, sorted by total salary.

Key Takeaways

  • cut extracts columns by delimiter (-d) and field number (-f). Quick and simple.
  • tr transforms individual characters — case conversion, character deletion, squeeze repeats.
  • sort + uniq is the standard pattern for counting and deduplication. Always sort before uniq.
  • sed does find-and-replace (s/old/new/g), line deletion, and line extraction. Use -i for in-place editing.
  • awk is a mini-language for field-based processing. It handles filtering, calculations, and formatted output in one tool.
  • These tools combine through pipes to form powerful data processing pipelines with zero setup.

Next up: Lesson 11 — Process Management and Job Control

Lesson 11: Process Management and Job Control

Time: ~30 minutes

Everything Is a Process

Every time you run a command, your system creates a process — a running instance of a program. When you type ls, a process starts, does its work, and exits. When you start a web server, that process stays running until you stop it. Your terminal itself is a process. Your shell is a process running inside it.

Understanding processes lets you see what's running, stop things that are stuck, run tasks in the background, and schedule scripts to run automatically.

Viewing Processes

ps — Process Status

ps                   # your processes in this terminal session
ps aux               # all processes on the system (detailed)

The aux flags mean: a = all users, u = user-oriented format, x = include processes without a terminal.

The output looks like:

USER       PID  %CPU %MEM    VSZ   RSS TTY      STAT START   TIME COMMAND
kevin     1234  0.0  0.1  12345  6789 pts/0    Ss   10:30   0:00 bash
kevin     5678  2.5  1.2  98765 43210 pts/0    Sl   10:31   0:15 node server.js
root         1  0.0  0.1   1234   567 ?        Ss   09:00   0:02 /sbin/init

Key columns:

  • PID — Process ID. Every process has a unique number.
  • %CPU / %MEM — resource usage
  • STAT — state (S = sleeping, R = running, Z = zombie, T = stopped)
  • COMMAND — what's running

Finding Specific Processes

ps aux | grep node              # find Node.js processes
ps aux | grep -v grep | grep node   # same, but exclude the grep itself

The second version is a common pattern — grep matches its own process too, and -v grep filters that out.

A cleaner alternative:

pgrep -la node                  # find processes by name

top and htop — Live Process Monitor

top                             # built-in, always available
htop                            # better interface (install with: sudo apt install htop)

top shows processes sorted by CPU usage, updating in real time. Press q to quit, M to sort by memory, P to sort by CPU.

htop is the same idea with a nicer interface, colour coding, and mouse support. If it's not installed, it's worth installing.

Stopping Processes

kill — Send Signals to Processes

kill 5678                       # send SIGTERM (polite "please stop") to PID 5678
kill -9 5678                    # send SIGKILL (forced stop, no cleanup)

SIGTERM (the default) asks the process to shut down gracefully — it can save state, close connections, and clean up. SIGKILL (-9) forces immediate termination with no cleanup. Always try SIGTERM first.

Common Signals

Signal Number Meaning
SIGTERM 15 Terminate gracefully (default)
SIGKILL 9 Force kill immediately
SIGHUP 1 Hang up (often used to reload config)
SIGINT 2 Interrupt (same as Ctrl+C)
SIGSTOP 19 Pause the process
SIGCONT 18 Resume a paused process

Killing by Name

pkill node                      # kill all processes named "node"
pkill -f "python server.py"     # kill processes matching the full command line
killall node                    # similar to pkill (slightly different on macOS vs Linux)

Be careful with pkill and killall — they match broadly. Make sure you're not killing something you need.

Background Jobs and Job Control

Running Commands in the Background

Normally, when you run a command, your terminal waits until it finishes. For long-running tasks, you can send them to the background:

sleep 60 &                      # the & runs it in the background

The shell immediately gives you back the prompt. The & at the end is the key.

You'll see output like:

[1] 12345

This means job number 1, process ID 12345.

Viewing Background Jobs

jobs                            # list background jobs in this shell

[1]+  Running                 sleep 60 &

Moving Jobs Between Foreground and Background

Suspend a running command: Press Ctrl + Z. This pauses the process and puts it in the background (stopped).

^Z
[1]+  Stopped                 vim notes.txt

Resume in the background:

bg                              # resume the most recent stopped job in the background
bg %1                           # resume job number 1 specifically

Bring back to the foreground:

fg                              # bring the most recent background job to the foreground
fg %1                           # bring job 1 to the foreground

A Typical Workflow

  1. Start a long task: ./build.sh
  2. Realise it's going to take a while: press Ctrl + Z
  3. Send it to the background: bg
  4. Do other work in the terminal
  5. Check on it: jobs
  6. Bring it back if needed: fg

Running Tasks After Disconnect

When you close your terminal, all its child processes receive SIGHUP and typically die. For long-running tasks on remote servers, you need a way to keep them alive.

nohup — No Hangup

nohup ./long-task.sh &

nohup prevents the process from receiving SIGHUP when the terminal closes. Output goes to nohup.out by default.

nohup ./long-task.sh > output.log 2>&1 &

This redirects both stdout and stderr to a specific log file.

screen and tmux — Terminal Multiplexers

For serious work on remote servers, screen or tmux is better than nohup. They create persistent terminal sessions that survive disconnects.

Quick tmux overview:

tmux                            # start a new session
tmux new -s mysession           # start a named session

Inside tmux:

  • Ctrl+B, D — detach (leave session running, return to normal terminal)
  • tmux ls — list sessions
  • tmux attach -t mysession — reattach

This is how professionals run long tasks on remote servers — start a tmux session, launch the task, detach, disconnect from the server, and reattach later to check on it.

Scheduling Tasks with cron

cron runs scripts on a schedule — every hour, every day, every Monday at 3am. It's the Unix way to automate recurring tasks.

Editing Your Crontab

crontab -e                      # open your cron schedule in an editor
crontab -l                      # list your current cron jobs

Cron Schedule Format

┌───────────── minute (0-59)
│ ┌───────────── hour (0-23)
│ │ ┌───────────── day of month (1-31)
│ │ │ ┌───────────── month (1-12)
│ │ │ │ ┌───────────── day of week (0-7, 0 and 7 are Sunday)
│ │ │ │ │
* * * * * command to run

Examples

# Run a backup every day at 2:30 AM
30 2 * * * /home/kevin/scripts/backup.sh

# Run a cleanup every Sunday at midnight
0 0 * * 0 /home/kevin/scripts/cleanup.sh

# Run a health check every 15 minutes
*/15 * * * * /home/kevin/scripts/healthcheck.sh

# Run a report on the 1st of every month at 9 AM
0 9 1 * * /home/kevin/scripts/monthly-report.sh

# Run every weekday at 8 AM
0 8 * * 1-5 /home/kevin/scripts/morning-tasks.sh

Cron Tips

Always use absolute paths in cron jobs. Cron doesn't load your shell profile, so it doesn't know about your PATH or any aliases.

# WRONG
backup.sh

# RIGHT
/home/kevin/scripts/backup.sh

Redirect output to a log:

0 2 * * * /home/kevin/scripts/backup.sh >> /home/kevin/logs/backup.log 2>&1

Without redirection, cron tries to email the output to you (which usually goes nowhere on modern systems). Redirect to a log file so you can check for errors.

Test your script manually first. Run it by hand before putting it in cron. Then check the log after the first scheduled run.

Monitoring Resources

df — Disk Free Space

df -h                           # human-readable disk usage for all filesystems
df -h /home                     # specific filesystem

free — Memory Usage

free -h                         # human-readable memory usage (Linux only)

On macOS, use vm_stat or top instead.

uptime — System Load

uptime                          # how long the system has been running, load averages

The load averages (three numbers) represent the average number of processes waiting for CPU time over the last 1, 5, and 15 minutes. On a single-core system, a load of 1.0 means the CPU is fully utilised. On a 4-core system, 4.0 means full utilisation.

The xargs Command

xargs takes input from a pipe and converts it into arguments for another command. It bridges the gap between commands that produce output and commands that expect arguments.

# Delete all .tmp files found by find
find . -name "*.tmp" | xargs rm

# With filenames that might contain spaces
find . -name "*.tmp" -print0 | xargs -0 rm

# Run a command for each input line
cat servers.txt | xargs -I {} ping -c 1 {}

-I {} defines a placeholder. Each line from stdin replaces {} in the command.

# Create directories from a list
echo -e "logs\ncache\ntmp" | xargs mkdir -p

# Compress each file individually
ls *.log | xargs -I {} gzip {}

Try It Yourself

  1. Run ps aux and pipe it through grep to find your shell process. Note its PID.

  2. Start a sleep 300 command, then immediately press Ctrl + Z to suspend it. Use jobs to verify it's stopped. Resume it in the background with bg. Use jobs again to confirm it's running.

  3. Start a background task with sleep 120 &. Find its PID with jobs -l or ps. Kill it with kill.

  4. Write a simple script that prints the current date and time to a log file. Set up a cron job to run it every minute. After a few minutes, check the log to confirm it's working. Then remove the cron job.

  5. Use find and xargs to find all .txt files in your bash-lessons directory and count the total number of lines across all of them.

  6. Use df -h to check your disk space. Use du -sh ~/bash-lessons to see how much space your lesson files are using.

Key Takeaways

  • Every running program is a process with a unique PID. ps aux shows them all, top/htop monitors them live.
  • kill PID sends SIGTERM (graceful). kill -9 PID forces termination. Always try graceful first.
  • & runs a command in the background. Ctrl + Z suspends. bg resumes in background. fg brings to foreground.
  • nohup keeps processes alive after terminal disconnect. tmux is the professional solution for persistent sessions.
  • cron schedules recurring tasks. Use crontab -e to edit, always use absolute paths, and redirect output to log files.
  • xargs converts piped input into command arguments — essential for connecting find output to other commands.

Next up: Lesson 12 — Real-World Scripting

Lesson 12: Real-World Scripting

Time: ~30 minutes

Putting It All Together

You've learned the individual tools. Now it's time to use them together the way professionals do — with error handling, debugging, good practices, and real-world patterns.

This final lesson covers the things that separate a quick hack from a reliable script: handling failure, making scripts debuggable, writing defensively, and building something complete.

Error Handling

The Problem with Ignoring Errors

By default, bash keeps running even when a command fails:

cd /nonexistent/directory       # fails silently
rm important_file.txt           # this still runs — in whatever directory you're actually in

This is dangerous. A failed cd means subsequent commands run in the wrong place. This has caused real-world data loss.

set -e — Exit on Error

#!/bin/bash
set -e

cd /nonexistent/directory       # script stops here
echo "This never runs"

With set -e, the script exits immediately when any command returns a non-zero exit code. This is the single most important line you can add to a script.

set -u — Error on Undefined Variables

#!/bin/bash
set -u

echo "$UNDEFINED_VARIABLE"     # script stops here with an error

Without set -u, undefined variables silently expand to empty strings. With it, you catch typos and missing configuration immediately.

set -o pipefail — Catch Pipe Failures

Normally, a pipeline's exit code is the exit code of the last command. This hides failures:

false | true                    # exit code is 0 (true succeeded)

With pipefail:

set -o pipefail
false | true                    # exit code is 1 (false failed)

The Standard Safety Header

Put this at the top of every serious script:

#!/bin/bash
set -euo pipefail

This single line catches the three most common classes of silent failures. Some people add set -x during development for debugging (covered next).

Handling Expected Failures

set -e is aggressive — sometimes commands should be allowed to fail:

# Method 1: Use || true to explicitly allow failure
grep "PATTERN" file.txt || true

# Method 2: Use an if statement
if ! grep -q "PATTERN" file.txt; then
    echo "Pattern not found, continuing anyway."
fi

# Method 3: Temporarily disable set -e
set +e
risky_command
result=$?
set -e
if [ $result -ne 0 ]; then
    echo "Command failed with code $result"
fi

Debugging

set -x — Trace Execution

#!/bin/bash
set -x

name="Kevin"
echo "Hello, $name"

Output:

+ name=Kevin
+ echo 'Hello, Kevin'
Hello, Kevin

Lines prefixed with + show exactly what bash is executing, with all variables expanded. This is invaluable for understanding why a script isn't doing what you expect.

You can turn tracing on and off within a script:

set -x          # start tracing
problematic_section
set +x          # stop tracing

Debug a Script Without Editing It

bash -x myscript.sh            # run with tracing without modifying the file

Logging for Debugging

Build a simple logging function into your scripts:

readonly LOG_FILE="/tmp/myscript.log"

log() {
    local level="$1"
    shift
    echo "[$(date '+%Y-%m-%d %H:%M:%S')] [$level] $*" | tee -a "$LOG_FILE"
}

log "INFO" "Script started"
log "DEBUG" "Processing file: $filename"
log "ERROR" "Failed to connect to database"

tee -a prints to the screen and appends to the log file simultaneously.

Defensive Scripting Patterns

Trap — Cleanup on Exit

trap lets you run code when your script exits, whether normally or due to an error:

#!/bin/bash
set -euo pipefail

TEMP_DIR=$(mktemp -d)
trap 'rm -rf "$TEMP_DIR"' EXIT

# Use TEMP_DIR freely — it gets cleaned up no matter what
cp important_file.txt "$TEMP_DIR/"
cd "$TEMP_DIR"
# ... do work ...

The trap '...' EXIT ensures the cleanup command runs when the script exits for any reason — success, failure, or Ctrl + C. This prevents temp files from accumulating.

Other signals you can trap:

trap 'echo "Interrupted!"; exit 1' INT          # Ctrl+C
trap 'echo "Terminated!"; exit 1' TERM           # kill signal
trap 'cleanup_function' EXIT                      # any exit

Checking Dependencies

require_command() {
    if ! command -v "$1" > /dev/null 2>&1; then
        echo "Error: Required command '$1' not found." >&2
        exit 1
    fi
}

require_command git
require_command docker
require_command jq

Put this near the top of scripts that depend on external tools.

Safe Temporary Files

TEMP_FILE=$(mktemp)            # creates /tmp/tmp.XXXXXXXXXX
TEMP_DIR=$(mktemp -d)          # creates a temporary directory

trap 'rm -rf "$TEMP_FILE" "$TEMP_DIR"' EXIT

Never hardcode temp file paths like /tmp/myscript.tmp — if two instances run simultaneously, they'll conflict. mktemp generates unique names.

Confirming Dangerous Operations

confirm() {
    local message="${1:-Are you sure?}"
    read -p "$message [y/N]: " response
    [[ "$response" =~ ^[Yy]$ ]]
}

if confirm "Delete all log files?"; then
    rm -f *.log
    echo "Deleted."
else
    echo "Cancelled."
fi

The [y/N] convention means the default (if you just press Enter) is No. Capital letter indicates the default.

Best Practices Checklist

These are habits that will serve you well in every script you write.

Always include the safety header:

#!/bin/bash
set -euo pipefail

Quote all variable expansions:

cp "$source" "$destination"    # always quote

Use readonly for constants:

readonly CONFIG_DIR="/etc/myapp"
readonly MAX_RETRIES=5

Use local in functions:

process_file() {
    local filename="$1"
    local output
    # ...
}

Provide usage information:

usage() {
    cat << EOF
Usage: $(basename "$0") [OPTIONS] <filename>

Options:
    -v, --verbose    Enable verbose output
    -d, --dry-run    Show what would be done without doing it
    -h, --help       Show this help message

Examples:
    $(basename "$0") data.csv
    $(basename "$0") -v --dry-run report.txt
EOF
    exit "${1:-0}"
}

Parse options properly:

VERBOSE=false
DRY_RUN=false

while [[ $# -gt 0 ]]; do
    case "$1" in
        -v|--verbose) VERBOSE=true; shift ;;
        -d|--dry-run) DRY_RUN=true; shift ;;
        -h|--help) usage 0 ;;
        -*) echo "Unknown option: $1" >&2; usage 1 ;;
        *) break ;;
    esac
done

if [[ $# -eq 0 ]]; then
    echo "Error: filename required." >&2
    usage 1
fi

filename="$1"

A Complete Project: Log Analyser

Let's build a real script that incorporates everything from this course. This script analyses a web server access log and produces a summary report.

#!/bin/bash
set -euo pipefail

#
# log-analyser.sh — Analyse web server access logs
# Usage: ./log-analyser.sh [-n TOP_N] [-o OUTPUT] <logfile>
#

# --- Configuration ---
readonly DEFAULT_TOP_N=10
readonly SCRIPT_NAME=$(basename "$0")

# --- Functions ---

usage() {
    cat << EOF
Usage: $SCRIPT_NAME [-n TOP_N] [-o OUTPUT] <logfile>

Analyse a web server access log and produce a summary report.

Options:
    -n NUM      Number of top results to show (default: $DEFAULT_TOP_N)
    -o FILE     Write report to file instead of stdout
    -h          Show this help message

Examples:
    $SCRIPT_NAME access.log
    $SCRIPT_NAME -n 5 -o report.txt access.log
EOF
    exit "${1:-0}"
}

log_info() {
    echo "[INFO] $*" >&2
}

check_file() {
    local file="$1"
    if [[ ! -f "$file" ]]; then
        echo "Error: File '$file' not found." >&2
        exit 1
    fi
    if [[ ! -r "$file" ]]; then
        echo "Error: File '$file' is not readable." >&2
        exit 1
    fi
}

generate_report() {
    local logfile="$1"
    local top_n="$2"
    local total_requests
    local unique_ips

    total_requests=$(wc -l < "$logfile")
    unique_ips=$(awk '{print $1}' "$logfile" | sort -u | wc -l)

    cat << EOF
============================================
  ACCESS LOG ANALYSIS REPORT
  Generated: $(date '+%Y-%m-%d %H:%M:%S')
  Log file:  $logfile
============================================

OVERVIEW
  Total requests:  $total_requests
  Unique IPs:      $unique_ips

TOP $top_n IP ADDRESSES BY REQUEST COUNT
$(awk '{print $1}' "$logfile" | sort | uniq -c | sort -rn | head -"$top_n" | awk '{printf "  %-18s %s requests\n", $2, $1}')

HTTP STATUS CODE BREAKDOWN
$(awk '{print $9}' "$logfile" | sort | uniq -c | sort -rn | awk '{printf "  %-6s %s responses\n", $2, $1}')

TOP $top_n REQUESTED PAGES
$(awk '{print $7}' "$logfile" | sort | uniq -c | sort -rn | head -"$top_n" | awk '{printf "  %-30s %s hits\n", $2, $1}')

ERRORS (4xx and 5xx responses)
$(awk '$9 >= 400 {printf "  %-18s %-8s %s\n", $1, $9, $7}' "$logfile" | head -20)

============================================
  End of report
============================================
EOF
}

# --- Main ---

main() {
    local top_n=$DEFAULT_TOP_N
    local output_file=""

    # Parse options
    while [[ $# -gt 0 ]]; do
        case "$1" in
            -n) top_n="$2"; shift 2 ;;
            -o) output_file="$2"; shift 2 ;;
            -h) usage 0 ;;
            -*) echo "Unknown option: $1" >&2; usage 1 ;;
            *)  break ;;
        esac
    done

    # Validate arguments
    if [[ $# -eq 0 ]]; then
        echo "Error: Log file required." >&2
        usage 1
    fi

    local logfile="$1"
    check_file "$logfile"

    log_info "Analysing $logfile (top $top_n results)..."

    # Generate report
    if [[ -n "$output_file" ]]; then
        generate_report "$logfile" "$top_n" > "$output_file"
        log_info "Report written to: $output_file"
    else
        generate_report "$logfile" "$top_n"
    fi

    log_info "Done."
}

main "$@"

Test it with the access log from Lesson 10:

chmod +x log-analyser.sh
./log-analyser.sh ../lesson10/access.log
./log-analyser.sh -n 3 -o report.txt ../lesson10/access.log

Where to Go from Here

You now have a solid foundation. Here's how to keep building:

Practice daily. Use the terminal for things you'd normally do with a GUI. The more you use it, the faster you get.

Read other people's scripts. Look at the scripts in /etc/init.d/ on a Linux system, or browse shell scripts on GitHub. Reading good code teaches you patterns you won't learn from tutorials.

Build tools for yourself. Automate your own workflows: project setup scripts, deployment helpers, backup routines, data processing pipelines. The best way to learn is to solve your own problems.

Explore deeper topics when you need them:

  • Regular expressions — more powerful pattern matching
  • jq — command-line JSON processor (essential for working with APIs)
  • curl and wget — HTTP requests from the command line
  • ssh and scp — remote server management
  • make — build automation
  • docker — containerisation (heavily uses bash)

Know when bash isn't the right tool. Bash is excellent for gluing commands together, automating system tasks, and quick data processing. For complex data structures, serious error handling, or anything over a few hundred lines, consider Python. The best developers know when to use each tool.

Try It Yourself — Final Exercises

  1. Take the log analyser script above, save it, and run it. Read through the code and make sure you understand every line. Modify it to add a new section — perhaps showing requests by hour of day.

  2. Write a project initialisation script that creates a directory structure, initialises a git repo, creates a .gitignore, and generates a README.md with the project name and current date. Accept the project name as an argument.

  3. Write a system health check script that reports: disk usage, memory usage, top 5 CPU-consuming processes, and network connectivity (ping a known host). Format the output as a clean report. Add an option to save to a file or send to stdout.

  4. Write a file organiser script that takes a directory of mixed files and sorts them into subdirectories by extension (e.g., all .pdf files into pdf/, all .jpg files into images/). Include a --dry-run flag that shows what would happen without actually moving files.

  5. Revisit the first script you wrote in Lesson 06. Refactor it using everything you've learned: add the safety header, use functions, handle errors, parse arguments, add a usage message. Compare the before and after.

Key Takeaways

  • set -euo pipefail should be in every serious script. It catches silent failures.
  • set -x traces execution — essential for debugging. bash -x script.sh does the same without editing.
  • trap '...' EXIT ensures cleanup runs no matter how the script exits.
  • mktemp creates safe temporary files. Never hardcode temp paths.
  • Quote everything. Use local in functions. Use readonly for constants.
  • Parse options with a while/case loop. Always provide a usage function.
  • Build scripts incrementally: get the basic logic working, then add error handling, then add options and polish.

Course Summary

Over 12 lessons, you've gone from opening a terminal to writing production-quality scripts. Here's what you've covered:

Lesson Topic Core Skills
01 Welcome to the Terminal Navigation, pwd, ls, cd, tab completion
02 Files and Directories mkdir, touch, cp, mv, rm, wildcards
03 Reading and Searching cat, less, head, tail, grep
04 Pipes and Redirection |, >, >>, 2>, /dev/null, tee
05 Permissions chmod, chown, rwx, sudo
06 First Scripts Shebang, variables, read, arguments, quoting
07 Conditionals if/elif/else, [[ ]], case, test expressions
08 Loops for, while, until, break, continue, while read
09 Functions local, return, source, script structure
10 Text Processing cut, tr, sort, uniq, sed, awk
11 Process Management ps, kill, jobs, bg/fg, cron, xargs
12 Real-World Scripting Error handling, debugging, best practices

The terminal is now a tool you can think in. Keep using it.

Congratulations on completing Bashing through Bash.