Bash Text Processing Cheatsheet: grep, sed, awk, cut, and sort

The Dark Arts of Bash Text Processing: Mastering grep, sed, awk, cut, and sort

As developers, we've all been there - stuck in a terminal, staring at a sea of text, wondering how to extract the one piece of information we need. The command line can be a powerful tool, but its text processing capabilities can seem like a mysterious black box. In this post, we'll demystify the dark arts of Bash text processing and provide you with a cheatsheet of essential one-liners for common tasks.

Table of Contents

• Introduction to Bash Text Processing
• Extracting Fields with cut and awk
• Filtering Lines with grep
• Replacing Text with sed
• Sorting and Uniquing with sort
• Counting Patterns with grep and wc
• Key Takeaways
• FAQ

Introduction to Bash Text Processing

Bash text processing is a set of command-line tools that allow you to manipulate and transform text data. We'll focus on five essential tools: grep, sed, awk, cut, and sort. These tools are the bread and butter of any developer's toolkit, and mastering them will make you a more efficient and effective coder.

Extracting Fields with cut and awk

When working with text data, it's often necessary to extract specific fields or columns. The cut command is perfect for this task. Let's say we have a file called users.txt containing a list of user data:

cat users.txt

John Doe,25,USA
Jane Doe,30,Canada
Bob Smith,35,UK

To extract the second field (age), we can use the following command:

cut -d, -f2 users.txt

25
30
35

Alternatively, we can use awk to achieve the same result:

awk -F, '{print $2}' users.txt

25
30
35

We recommend using awk for more complex field extraction tasks, as its syntax is more flexible and powerful.

Filtering Lines with grep

grep is one of the most popular command-line tools, and for good reason. It allows you to search for patterns in text data and filter out unwanted lines. Let's say we want to find all lines containing the word "Doe":

grep "Doe" users.txt

John Doe,25,USA
Jane Doe,30,Canada

We can also use grep to filter out lines that don't match a certain pattern. For example, to exclude lines containing the word "USA":

grep -v "USA" users.txt

Jane Doe,30,Canada
Bob Smith,35,UK

Replacing Text with sed

sed is a powerful tool for replacing text in files. Let's say we want to replace all occurrences of "Doe" with "Smith":

sed 's/Doe-Smith/g' users.txt

John Smith,25,USA
Jane Smith,30,Canada
Bob Smith,35,UK

Note the g flag at the end of the command, which stands for "global" and replaces all occurrences in each line.

Sorting and Uniquing with sort

sort is a versatile tool for sorting and uniquing text data. Let's say we want to sort our users.txt file by age:

sort -t, -k2 -n users.txt

John Doe,25,USA
Jane Doe,30,Canada
Bob Smith,35,UK

We can also use sort to remove duplicate lines:

sort -u users.txt

John Doe,25,USA
Jane Doe,30,Canada
Bob Smith,35,UK

Counting Patterns with grep and wc

Finally, let's say we want to count the number of lines containing a certain pattern. We can use grep and wc to achieve this:

grep "Doe" users.txt | wc -l

2

Key Takeaways

• Use cut for simple field extraction tasks, and awk for more complex ones.
• Use grep for filtering lines based on patterns.
• Use sed for replacing text in files.
• Use sort for sorting and uniquing text data.
• Use grep and wc for counting patterns.

FAQ

Q: What's the difference between cut and awk?

A: cut is a simpler tool for extracting fields, while awk is more powerful and flexible.

Q: Can I use grep to filter out multiple patterns?

A: Yes, you can use the -v flag to exclude multiple patterns.

Q: How do I sort a file by multiple columns?

A: You can use the -k flag to specify multiple columns, separated by commas.