Sed command in Linux

In this article, we discuss the linux sed command, a non-interactive editor which reads a set of commands and executes them in a given file.

Table of contents.

1. Introduction.
2. Syntax.
3. Options.
4. Commands.
5. Characters.
6. Regular expressions.
7. Addresses.
8. Question.
9. Summary.
10. References.

Introduction.

The sed command is a very useful stream editor for filtering and transforming text.
It performs text transformations on an input stream by making only one pass over the input(s).

Syntax.

sed [option]... {SCRIPT} [input_file]

OR

sed [option] 'command' [input_file]

An example

Run the command to obtain a text file 'input.txt' with ping information.

ping 8.8.8.8 -c 4 > input.txt

If the ping was successful, the file should look like the following;

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.
64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=17.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=16.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=118 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=118 time=15.5 ms

Using sed command we can replace all occurrences of 'bytes' with 'bits' and redirect the results into another file 'output.txt' by using any of following commands.

sed 's/bytes/bits/' input.txt > output.txt
sed 's/bytes/bits/' < input.txt > output.txt
cat input.txt | sed 's/bytes/bits/' - > output.txt

The s/bytes/bits/ finds and replaces the first with the second word.
Note that the input.txt file doesn't change.

If we want to change the file 'input.txt' in-place, that is, without using an output file.
We use the 'i' option as shown below.

sed -i 's/bytes/bits/' input.txt

now bytes will be replaced with bits in the 'input.txt' file permanently.

Looking at the command sed -i 's/bytes/bits/' input.txt, the '/' acts as a delimiter, however in some cases we may want to perform the sed command on a file with '/', that is;

Assuming we want to perform operations on text with the following structure, for example, we may want to remove the text '/home' using the sed command.

/home/user/Documents/file1/test.py
/home/user/Documents/file1/test2.py
/home/user/Documents/file1/test3.py

In this case we need to change the delimiter / to another since they contradict with the text we want to remove.

sed 's./home..' path.txt

From the above command we have used . as the delimiter and this works.
The command will remove the text '/home'.

Output

/user/Documents/file1/test.py
/user/Documents/file1/test2.py
/user/Documents/file1/test3.py

We can also replace an occurrence of a word from a particular line.

sed '3 s/bits/mbs/g' output.txt

The command will replace the word 'bits' from the third line with the word 'mbps'.

PING 8.8.8.8 (8.8.8.8) 56(84) bits of data.
64 bits from 8.8.8.8: icmp_seq=1 ttl=118 time=17.2 ms
64 mbps from 8.8.8.8: icmp_seq=2 ttl=118 time=16.6 ms
64 bits from 8.8.8.8: icmp_seq=3 ttl=118 time=13.6 ms
64 bits from 8.8.8.8: icmp_seq=4 ttl=118 time=15.5 ms

Options.

With sed command we can define parameters. By using parameters we can determine how a command is to be interpreted.

The following are commonly used options with the sed command.

Option	Use
-e	Indicate that one or more sed scripts are used.
-f	Specify that a script is pulled from a file.
-n	Results from a command are not output.
-i	Creates a temporary file that replaces the source file.

Commands.

A command is responsible for defining actions it has to perform to the source file.
Commonly used commands are;

Flags	Description
g	Globally replace all occurrences of a string in a file.
i	Ignore case sensitivity during substitutions while replacing.
p	print: display selected lines.
w	write: write lines into the text file.
x	xchange: swap pattern space and hold space with each other.
=	specifies the number of selected lines.

An example
This command will manipulate the third line and prints it ignoring the rest of the text file contents.

sed -n '3 s/bytes/kbps/p' input.txt

Output

64 kbps from 8.8.8.8: icmp_seq=2 ttl=118 time=16.6 ms

Characters.

Characters	Description
*	Matches a sequence of zero or more instances of the previous characters.
.	Matches any single character including a new line.
$	Matches at the end of the regular expression/pattern space.
^	Acts as a special character and matches the beginning of a regular expression/pattern space.

Regular expressions.

Sed command supports regular expressions and therefore can be used to match complex patterns.

An example

sed -n '/ping/p' test.txt 

The command will match any text with the pattern 'ping' and output it. In this case it returns the first pattern found.

Output

--- 8.8.8.8 ping statistics ---

We can also match all words starting with a specific character.
An example
To get all lines starting with the number 64 we write,

sed -n '/^64/ p' test.txt 

The ^ matches the start of a line.

We also have the option of targeting a specific string by using its starting and ending character.

An example

sed -n '/s$/p' test.txt

Output

64 bytes from 8.8.8.8: icmp_seq=1 ttl=118 time=17.2 ms
64 bytes from 8.8.8.8: icmp_seq=2 ttl=118 time=16.6 ms
64 bytes from 8.8.8.8: icmp_seq=3 ttl=118 time=13.6 ms
64 bytes from 8.8.8.8: icmp_seq=4 ttl=118 time=15.5 ms
4 packets transmitted, 4 received, 0% packet loss, time 3005ms

From the command we obtain all text ending with the character s by using the $ symbol.

We can also match letter cases, that is print words containing upper case letters.

An example

sed -n '/[A-Z]/p' test.txt

Output

PING 8.8.8.8 (8.8.8.8) 56(84) bytes of data.

Addresses.

Inputs are called addresses in SED and therefore the goal of command.
An address can be a line number e.g 1, 3, 7, a regular expression, or a dollar sign $ to represent the last line of the file.
An exclamation mark ! after an address or address range applies the command to every line except the lines named by the address.
A null regex '//' is replaced by the last regex used.

Deleting the first line only.

sed '1d' test.txt

Deleting all lines except the first

sed '1!d' test.txt

Print the lines from 1 to 3

sed -n '1, 3p' test.txt

Question.

Given the following csv file.

id,firstname,lastname,email,email2,profession
100,Lynea,Ashok,Lynea.Ashok@yopmail.com,Lynea.Ashok@gmail.com,developer
101,Xylina,Gaulin,Xylina.Gaulin@yopmail.com,Xylina.Gaulin@gmail.com,doctor
102,Kellen,Chaing,Kellen.Chaing@yopmail.com,Kellen.Chaing@gmail.com,firefighter
103,Jorry,Klotz,Jorry.Klotz@yopmail.com,Jorry.Klotz@gmail.com,doctor
104,Jasmina,Prouty,Jasmina.Prouty@yopmail.com,Jasmina.Prouty@gmail.com,developer
105,Jennica,Juliet,Jennica.Juliet@yopmail.com,Jennica.Juliet@gmail.com,firefighter
106,Estell,Larochelle,Estell.Larochelle@yopmail.com,Estell.Larochelle@gmail.com,developer
107,Marline,Federica,Marline.Federica@yopmail.com,Marline.Federica@gmail.com,worker
108,Kristina,Deny,Kristina.Deny@yopmail.com,Kristina.Deny@gmail.com,police officer
109,Alia,Oscar,Alia.Oscar@yopmail.com,Alia.Oscar@gmail.com,firefighter

1. Can you print all the lines of people who are developers? (hint: use two commands in succession).
2. Send all gmail addresses to a emails.txt output file.
3. Replace yopmail.com with yahoo.com.

Summary.

Sed command is used to work on text files such as log files, configuration files etc, it can make rote tasks such as editing multiple files or searching for information in them very easy.

What we have learnt here can be combined with other similar powerful linux commands such as find to to save alot of time.

References.

1. sed manual
2. Run the command man sed for the sed manual page.