Open-Source Internship opportunity by OpenGenus for programmers. Apply now.
In this article we discuss piping and redirection whereby we can alter the flow of input or output streams. This technique enables a user to create custom commands by combining multiple smaller commands to achieve specific task that cannot be accomplished by a single Linux command.
Table of contents.
- Introduction.
- Creating files.
- Redirection.
- Piping.
- Summary.
- References.
Introduction.
Streams are inputs to and outputs from a program in Linux, we can redirect these streams.
Like everything else in Linux, streams are also treated as files, in that you can read and write to files similarly you can read and write to streams.
There are 3 standard streams namely,
- standard input (stdin) - this is input from the keyboard.
- standard output (stdout) - this is output from commands, usually displayed on the terminal.
- standard error (stderr) - this is error output.
Files associated with a process will have unique numbers for identification known as file descriptors. These number are useful whenever an action is required to be performed on a file.
Streams also have their descriptors, that is, 0, 1, 2 for stdin, stdout and stderr respectively.
In this article we shall be using Linux commands discussed in a previous articles, the links are in the references section.
Creating files.
Lets start by creating three files as follows,
mkdir TestDir && cd TestDir && { echo "id,firstname,lastname,email,age,profession
100,Tabbatha,Velick,Tabbatha.Velick@yahoo.com,52,police officer
102,Arlina,Wu,Arlina.Wu@yahoo.com,52,developer" > text1.txt; echo "103,Maye,Flita,Maye.Flita@yahoo.com,46,developer
104,Max,Juan,Max.Juan@yahoo.com,31,developer
106,Netty,Pauly,Netty.Pauly@yahoo.com,58,doctor" > text2.txt; echo "109,Gui,Cohdwell,Gui.Cohdwell@yahoo.com,32,firefighter" > text3.txt; cat text1.txt; cat text2.txt; cat text3.txt;}
The command first creates a new directory TestDir, the changes from the current directory to TestDir using the cd command, then echos text into the files and finally concatenates all files to print output to the terminal.
Redirection.
Redirection involves reading files and passing their contents as input to commands and sending output from commands to files.
Redirecting output
>: This redirects output from a command to a file, one must have write permissions. If the file has other content, it is usually lost, otherwise if the file doesn't exist a new file is created.
An example
To get all lines with the text police officer and redirect output to another file, out we write,
grep "police officer" *.txt > out
>>: This also redirects output from a command to a file, write permissions are also necessary. If the file has other content, the redirected content is appended, otherwise if the file doesn't exist a new file is created.
To append lines with the word fire fighter to the out file we write,
grep "fire fighter" *.txt >> out
Now the previous contents of out are not erased,the new content is appended.
Redirecting input
<: This is useful for redirecting a file into an application, in that, file contents we act as the standard input for the application/command.
Another example
For example to replace all commas with '|', we can write.
sed 's/,/ | /g' < out
From the above command, the sed command receives input from out file.
A here-document redirects input into an interactive shell of program.
The form of the here document is as follows.
command << delimiter
document
delimiter
An example
cat << test
> testing 1
> testing 2
> test
testing 1
testing 2
Redirecting errors
To redirect errors we ca write,
cat bogusFileName 2> error.txt
Now when you concatenate error.txt the error that was to be displayed on the terminal will have been redirected there.
Notice how we use the descriptor number 2.
Redirecting into streams
We can also redirect into streams, for example to redirect to the stderr stream, we can write,
ls >&2
We can also combine streams by redirecting from one to another, e.g to combine stderr and stdout we ca write,
ls -l 2>&1n
Piping.
Piping involves passing output from one command as input to another.
An example
cat text1.txt | wc
From the above command we concatenate contents of text1.txt, the output of this command is then piped to the wc command.
We can also combine multiple pipes as follows,
cut -d ',' -f 6 text1.txt | tr a-z A-Z | tac -r -s 'x\|[^x]' | sort
From the above command we get the professions column from text1.txt file using the sed command then pipe the output to the trcommand which converts the output(now its input) to uppercase letters then pipes the output we obtain to the tac command as input which reverses the words and finally the sort command sorts the input received and the result is printed.
We can also combine the command with redirection i.e redirecting the output or error to a file.
Summary.
The difference between a pipe and a redirect is that while a pipe passes standard output as standard input to another command, a redirect sends a output to a file or reads a file as standard input to a command.
Piping and redirection is very useful, a case on point is stress testing programs, whereby we have two algorithms and need to test them on varying inputs. To do this in a simple automated manner we can create an input file with randomized inputs then pass its contents as parameters to both algorithms.
