9.9. Redirection and Pipes

Normally, standard output (stdout) from a command goes to the screen, standard input (stdin) comes from the keyboard, and error messages (stderr) go to the screen. The shell allows you to use the special redirection metacharacters to redirect the input/output to or from a file. The redirection operators (<, >, >>, >&) are followed by a filename. This file is opened by the shell before the command on the left-hand side is executed.

Pipes, represented by a vertical bar (|) symbol, allow the output of one command to be sent to the input of another command. The command on the left-hand side of the pipe is called the writer because it writes to the pipe. The command on the right-hand side of the pipe is the reader because it reads from the pipe. See Table 9.5 for a list of redirection and pipe metacharacters.

Table 9.5. Redirection Metacharacters
Metacharacter	Meaning
`command < file`	Redirects input from `file` to `command`.
`command > file`	Redirects output from `command` to `file`.
`command >& file`	Redirects output and errors to `file`.
`command >> file`	Redirects output of `command` and appends it to `file`.
`command >>& file`	Redirects and appends output and errors of `command` to `file`.
command << WORD <input> WORD	Redirects input from first `WORD` to terminating `WORD` to `command`; start of `here document`. User input goes here. It will be treated as a doubly quoted string of text. `WORD` marks the termination of input to `command`; end of `here document`.
`command \| command`	Pipes output of first `command` to input of second `command`.
`command \|& command`	Pipes output and errors of first `command` to input of second `command`.
`command >! file`	If the `noclobber` variable is set, override its effects for this command and either open or overwrite `file`.
`command >>! file`	Override `noclobber` variable; if `file` does not exist, it is created and output from `command` is appended to it.
`command >>&! file`	Override `noclobber` variable; if `file` does not exist, it is created and both output and errors are appended to it.

9.9.1 Redirecting Input

Instead of the input coming from the terminal keyboard, it can be redirected from a file. The shell will open the file on the right-hand side of the < symbol and the program on the left will read from the file. If the file does not exist, the error No such file or directory will be reported by the C shell.

Example 9.36.

The file memo is opened by the shell, and the input is redirected to the mail program. Simply, the user bob is sent a file called memo by the mail program.

9.9.2 The here document

The here document is another way to redirect input to a command. It is used in shell scripts for creating menus and processing input from other programs. Normally, programs that accept input from the keyboard are terminated with Ctrl-D (^D). The here document provides an alternate way of sending input to a program and terminating the input without typing ^D. The << symbol is followed by a user-defined word, often called a terminator. Input will be directed to the command on the left-hand side of the << symbol until the user-defined terminator is reached. The final terminator is on a line by itself, and cannot be surrounded by any spaces. Variable and command substitution are performed within the here document. (Normally, a here document is used in shell scripts to create menus and provide input to commands such as mail, bc, ex, ftp, etc.)

Example 9.37.

The cat program, without arguments, waits for keyboard input.
The user types input at the keyboard.
The user types ^D to terminate input to the cat program.
The cat program sends its output to the screen.

Example 9.38.

The cat program will receive input from the first DONE to the terminating DONE. The words are user-defined terminators.
These lines are input. When the word DONE is reached, no more input is accepted.
The final terminator marks the end of input. There cannot be any spaces on either side of this word.
The text between the first word DONE and the final word DONE is the output of the cat command (from "here" to "here") and is sent to the screen. The final DONE must be against the left margin with no space or other text to the right of it.

Example 9.39.

The shell variable name is assigned the username steve. (Normally, this example would be included in a shell script.)
The variable name is expanded within the here document.
The mail program will receive input until the terminator EOF is reached.
Command substitution is performed within the here document; that is, the command within the backquotes is executed and the output of the command is replaced within the string.
The terminator EOF is reached, and input to the mail program is stopped.

9.9.3 Redirecting Output

By default, the standard output of a command or commands goes to the terminal screen. To redirect standard output from the screen to a file, the > symbol is used. The command is on the left-hand side of the > symbol, and a filename is on the right-hand side. The shell will open the file on the right-hand side of the > symbol. If the file does not exist, the shell will create it; if it does exist, the shell will open the file and truncate it. Often files are inadvertently removed when using redirection. (A special C/TC shell variable, called noclobber, can be set to prevent redirection from clobbering an existing file. See Table 9.6 on page 439.)

Table 9.6. The `noclobber` Variable and Redirection
`noclobber` Is Not Set		If File Exists	If File Does Not Exist
	`command > file`	`file` is overwritten	`file` is created
	`command >> file`	`file` is appended to	`file` is created
`noclobber` Is Set
	`command > file`	Error message	`file` is created
	`command >> file`	`file` is appended to	Error message
Overwriting `noclobber`
	`command >! file`	If the `noclobber` variable is set, override its effects for this `command` and either open or truncate `file`, redirecting output of `command` to `file`.
	`command >>! file`	Override `noclobber` variable; if `file` does not exist, it is created and output from `command` is appended to it. (See Example 9.45.)

Example 9.40.

The contents of file1 and file2 are concatenated and the output is sent to file3. Remember that the shell opens file3 before it attempts to execute the cat command. If file3 already exists and contains data, the data will be lost. If file3 does not exist, it will be created.

9.9.4 Appending Output to an Existing File

To append output to an existing file, the >> symbol is used. If the file on the right-hand side of the >> symbol does not exist, it is created; if it does exist, the file is opened and output is appended to the end of the file.

Example 9.41.

The standard output of the date command is redirected and appended to outfile.

9.9.5 Redirecting Output and Error

The >& symbol is used to redirect both standard output and standard error to a file. Normally, a command is either successful and sends its output to stdout, or fails and sends its error messages to stderr. Some recursive programs, such as find and du, send both standard output and errors to the screen as they move through the directory tree. By using the >& symbol, both standard output and standard error can be saved in a file and examined. The C/TC shell does not provide a symbol for redirection of only standard error, but it is possible to get just the standard error by executing the command in a subshell. See Example 9.42 and Figure 9.2.

Example 9.42.

The output of the date command is sent to standard output, the screen.
The output and errors are sent to outfile.
Because there were no errors, the standard output is sent to outfile and the contents of the file are displayed.

Example 9.43.

To copy a file, the cp command requires both a source file and a destination file. The cp command makes a copy of file1 and puts the copy in file2. Because the cp command is given the correct syntax, nothing is displayed to the screen. The copy was successful.
This time the destination file is missing and the cp command fails, sending an error to stderr, the terminal.
The >& symbol is used to send both stdout and stderr to errorfile. Because the only output from the command is the error message, that is what is saved in errorfile.
The contents of errorfile are displayed, showing that it contains the error message produced by the cp command.

9.9.6 Separating Output and Errors

Standard output and standard error can be separated by enclosing the command in parentheses. When a command is enclosed in parentheses, the C/TC shell starts up a subshell, handles redirection from within the subshell, and then executes the command. By using the technique shown in Example 9.44, the standard output can be separated from the errors.

Example 9.44.

The find command will start at the current directory, searching for all files ending in .c, and will print the output to outputfile. If an error occurs, that will also go into outputfile.
The find command is enclosed within parentheses. The shell will create a subshell to handle the command. Before creating the subshell, the words outside the parentheses will be processed; that is, the badstuff file will be opened for both standard output and error. When the subshell is started, it inherits the standard input, output, and errors from its parent. The subshell then has standard input coming from the keyboard, and both standard output and standard error going to the badstuff file. Now the subshell will handle the > operator. The stdout will be assigned the file goodstuff. The output is going to goodstuff, and the errors are going to badstuff. See Figure 9.3.
Figure 9.3. Separating stdout and stderr.

[View full size image]

9.9.7 The noclobber Variable

The special shell built-in variable noclobber, when set, protects you from clobbering files with redirection. See Table 9.6.

Example 9.45.

The contents of filex are displayed on the screen.
The output of the date command is redirected to filex. The file is truncated and its original contents overwritten.
The contents of filex are displayed.
The noclobber variable is set.
Since filex already exists and noclobber is set, the shell reports that the file exists and will not allow it to be overwritten.
The output of ls is redirected to filex because the >! operator overrides the effects of noclobber.
The effects of the >! symbol were temporary. It does not turn off noclobber. It simply overrides noclobber for the command where it is implemented.
The noclobber variable is unset.