7.3. Command-Line Processing

We've seen how the shell uses read to process input lines: it deals with single quotes (`'), double quotes (""), and backslashes (\); it separates lines into words, according to delimiters in the environment variable IFS; and it assigns the words to shell variables. We can think of this process as a subset of the things the shell does when processing command lines.

We've touched upon command-line processing throughout this book; now is a good time to make the whole thing explicit. Each line that the shell reads from the standard input or a script is called a pipeline; it contains one or more commands separated by zero or more pipe characters (|). For each pipeline it reads, the shell breaks it up into commands, sets up the I/O for the pipeline, then does the following for each command (Figure 7-1):

Splits the command into tokens that are separated by the fixed set of metacharacters: SPACE, TAB, NEWLINE, ;, (, ), <, >, |, and &. Types of tokens include words, keywords, I/O redirectors, and semicolons.
Checks the first token of each command to see if it is a keyword with no quotes or backslashes. If it's an opening keyword, such as if and other control-structure openers, function, {, or (, then the command is actually a compound command. The shell sets things up internally for the compound command, reads the next command, and starts the process again. If the keyword isn't a compound command opener (e.g., is a control-structure "middle" like then, else, or do, an "end" like fi or done, or a logical operator), the shell signals a syntax error.
Checks the first word of each command against the list of aliases. If a match is found, it substitutes the alias's definition and goes back to Step 1; otherwise, it goes on to Step 4. This scheme allows recursive aliases (see Chapter 3). It also allows aliases for keywords to be defined, e.g., alias aslongas=while or alias procedure=function.
Performs brace expansion. For example, a{b,c} becomes ab ac.
Substitutes the user's home directory ($HOME) for tilde if it is at the beginning of a word. Substitutes user's home directory for ~user.^[7]
^[7] Two obscure variations on this: the shell substitutes the current directory ($PWD) for ~+ and the previous directory ($OLDPWD) for ~-. In bash 2.0 there are two more: ~N+ and ~N-. These are replaced by the corresponding element in the directory stack as given by the dirs command.
Performs parameter (variable) substitution for any expression that starts with a dollar sign ($).
Does command substitution for any expression of the form $(string).
Evaluates arithmetic expressions of the form $((string)).
Takes the parts of the line that resulted from parameter, command, and arithmetic substitution and splits them into words again. This time it uses the characters in $IFS as delimiters instead of the set of metacharacters in Step 1.
Performs pathname expansion, a.k.a. wildcard expansion, for any occurrences of *, ?, and [/] pairs.
Uses the first word as a command by looking up its source according to the rest of the list in Chapter 4, i.e., as a function command, then as a built-in, then as a file in any of the directories in $PATH.
Runs the command after setting up I/O redirection and other such things.

That's a lot of steps—and it's not even the whole story! But before we go on, an example should make this process clearer. Assume that the following command has been run:

Further assume that a file exists called .hist537 in user alice's home directory, which is /home/alice, and that there is a double-dollar-sign variable $$ whose value is 2537 (we'll see what this special variable is in the next chapter).

Although this list of steps is fairly straightforward, it is not the whole story. There are still five ways to modify the process: quoting; using command, builtin, or enable; and using the advanced command eval.

7.3.1. Quoting

You can think of quoting as a way of getting the shell to skip some of the 12 steps above. In particular:

Table 7-7 has simple examples to show how these work; they assume the statement person=hatter was run and user alice's home directory is /home/alice.

If you are wondering whether to use single or double quotes in a particular shell programming situation, it is safest to use single quotes unless you specifically need parameter, command, or arithmetic substitution.

Table 7-7. Examples of quoting rules
Expression	Value
$person	hatter
"$person"	hatter
\$person	$person
`$person'	$person
"'$person'"	'hatter'
~alice	/home/alice
"~alice"	~alice
`~alice'	~alice

7.3.2. command, builtin, and enable

Before moving on to the last part of the command-line processing cycle, we'll take a look at the command lookup order that we touched on in Chapter 4 and how it can be altered with several shell built-ins.

The default order for command lookup is functions, followed by built-ins, with scripts and executables last. There are three built-ins that you can use to override this order: command, builtin, and enable.

command removes alias and function lookup.^[9] Only built-ins and commands found in the search path are executed. This is useful if you want to create functions that have the same name as a shell built-in or a command in the search path and you need to call the original command from the function. For instance, we might want to create a function called cd that replaces the standard cd command with one that does some fancy things and then executes the built-in cd:

In this case we avoid plunging the function into a recursive loop by placing command in front of cd. This ensures that the built-in cd is called and not the function.

Table 7-8. command options
Option	Description
-p	Uses a default value for PATH
-v	Prints the command or pathname used to invoke the command
-V	A more verbose description than with -v
-	Turns off further option checking

The -p option is a default path which guarantees that the command lookup will find all of the standard UNIX utilities. In this case, command will ignore the directories in your PATH.^[10]

builtin is very similar to command but is more restrictive. It looks up only built-in commands, ignoring functions and commands found in PATH. We could have replaced command with builtin in the cd example above.

The last command enables and disables shell built-ins—it is called enable. Disabling a built-in allows a shell script or executable of the same name to be run without giving a full pathname. Consider the problem many beginning UNIX shell programmers have when they name a script test. Much to their surprise, executing test usually results in nothing, because the shell is executing the built-in test, rather than the shell script. Disabling the built-in with enable overcomes this.^[11]

Table 7-9 lists the options available with enable.^[12] Some options are for working with dynamically loadable built-ins. See Appendix C for details on these options, and how to create and load your own built-in commands.

Table 7-9. enable options
Option	Description
-a	Displays every built-in and whether it is enabled or not
-d	Deletes a built-in loaded with -f
-f `filename`	Loads a new built-in from the shared-object `filename`
-n	Disables a built-in or displays a list of disabled built-ins
-p	Displays a list of all of the built-ins
-s	Restricts the output to POSIX "special" built-ins

Of these options, -n is the most useful; it is used to disable a built-in. enable without an option enables a built-in. More than one built-in can be given as arguments to enable, so enable -n pushd popd dirs would disable the pushd, popd, and dirs built-ins.^[13]

You can find out what built-ins are currently enabled and disabled by using the command on its own, or with the -p option; enable or enable -p will list all enabled built-ins, and enable -n will list all disabled built-ins. To get a complete list with their current status, you can use enable -a.

The -s option restricts the output to POSIX `special' built-ins. These are :, ., source, break, continue, eval, exec, exit, export, readonly, return, set, shift, trap, and unset.

7.3.3. eval

We have seen that quoting lets you skip steps in command-line processing. Then there's the eval command, which lets you go through the process again. Performing command-line processing twice may seem strange, but it's actually very powerful: it lets you write scripts that create command strings on the fly and then pass them to the shell for execution. This means that you can give scripts "intelligence" to modify their own behavior as they are running.

The eval statement tells the shell to take eval's arguments and run them through the command-line processing steps all over again. To help you understand the implications of eval, we'll start with a trivial example and work our way up to a situation in which we're constructing and running commands on the fly.

eval ls passes the string ls to the shell to execute; the shell prints a list of files in the current directory. Very simple; there is nothing about the string ls that needs to be sent through the command-processing steps twice. But consider this:

Instead of producing a paginated file listing, the shell will treat | and more as arguments to ls, and ls will complain that no files of those names exist. Why? Because the pipe character "appears" in Step 6 when the shell evaluates the variable, after it has actually looked for pipe characters. The variable's expansion isn't even parsed until Step 9. As a result, the shell will treat | and more as arguments to ls, so that ls will try to find files called | and more in the current directory!

Now consider eval $listpage instead of just $listpage. When the shell gets to the last step, it will run the command eval with arguments ls, |, and more. This causes the shell to go back to Step 1 with a line that consists of these arguments. It finds | in Step 2 and splits the line into two commands, ls and more. Each command is processed in the normal (and in both cases trivial) way. The result is a paginated list of the files in your current directory.

Now you may start to see how powerful eval can be. It is an advanced feature that requires considerable programming cleverness to be used most effectively. It even has a bit of the flavor of artificial intelligence, in that it enables you to write programs that can "write" and execute other programs.^[14] You probably won't use eval for everyday shell programming, but it's worth taking the time to understand what it can do.

As a more interesting example, we'll revisit Task 4-1, the very first task in the book. In it, we constructed a simple pipeline that sorts a file and prints out the first N lines, where N defaults to 10. The resulting pipeline was:

The first argument specified the file to sort; $2 is the number of lines to print.

Now suppose we change the task just a bit so that the default is to print the entire file instead of 10 lines. This means that we don't want to use head at all in the default case. We could do this in the following way:

In other words, we decide which pipeline to run according to whether $2 is null. But here is a more compact solution:

The last expression in this line evaluates to the string | head -\$2 if $2 exists (is not null); if $2 is null, then the expression is null too. We backslash-escape dollar signs (\$) before variable names to prevent unpredictable results if the variables' values contain special characters like > or |. The backslash effectively puts off the variables' evaluation until the eval command itself runs. So the entire line is either:

if $2 is null. Once again, we can't just run this command without eval because the pipe is "uncovered" after the shell tries to break the line up into commands. eval causes the shell to run the correct pipeline when $2 is given.

Next, we'll revisit Task 7-2 from earlier in this chapter, the start script that lets you start a command in the background and save its standard output and standard error in a logfile. Recall that the one-line solution to this task had the restriction that the command could not contain output redirectors or pipes. Although the former doesn't make sense when you think about it, you certainly would want the ability to start a pipeline in this way.

The only restriction that this imposes on the user is that pipes and other such special characters be quoted (surrounded by quotes or preceded by backslashes).

Here's a way to apply eval in conjunction with various other interesting shell programming concepts.

make is known primarily as a programmer's tool, but it seems as though someone finds a new use for it every day. Without going into too much extraneous detail, make basically keeps track of multiple files in a particular project, some of which depend on others (e.g., a document depends on its word processor input file(s)). It makes sure that when you change a file, all of the other files that depend on it are processed.

For example, assume you're using the troff word processor to write a book. You have files for the book's chapters called ch1.t, ch2.t, and so on; the troff output for these files are ch1.out, ch2.out, etc. You run commands like troff chN.t > chN.out to do the processing. While you're working on the book, you tend to make changes to several files at a time.

In this situation, you can use make to keep track of which files need to be reprocessed, so that all you need to do is type make, and it will figure out what needs to be done. You don't need to remember to reprocess the files that have changed.

How does make do this? Simple: it compares the modification times of the input and output files (called sources and targets in make terminology), and if the input file is newer, then make reprocesses it.

You tell make which files to check by building a file called makefile that has constructs like this:

This essentially says, "For target to be up to date, it must be newer than all of the sources. If it's not, run the commands to bring it up to date." The commands are on one or more lines that must start with TABs: e.g., to make ch7.out:

Now suppose that we write a shell function called makecmd that reads and executes a single construct of this form. Assume that the makefile is read from standard input. The function would look like the following code.

This function reads the line with the target and sources; the variable colon is just a placeholder for the :. Then it checks each source to see if it's newer than the target, using the -nt file attribute test operator that we saw in Chapter 5. If the source is newer, it reads, prints, and executes the commands until it finds a line that doesn't start with a TAB or it reaches end-of-file. (The real make does more than this; see the exercises at the end of this chapter.) After running the commands (which are stripped of the initial TAB), it breaks out of the for loop, so that it doesn't run the commands more than once.

As a final example of eval, we'll look again at procimage, the graphics utility that we developed in the last three chapters. Recall that one of the problems with the script as it stands is that it performs the process of scaling and bordering regardless of whether you want them. If no command-line options are present, a default size, border width, and border color are used. Rather than invent some if then logic to get around this, we'll look at how you can dynamically build a pipeline of commands in the script; those commands that aren't needed simply disappear when the time comes to execute them. As an added bonus, we'll add another capability to our script: image enhancement.

Looking at the procimage script you'll notice that the NetPBM commands form a nice pipeline; the output of one operation becomes the input to the next, until we end up with the final image. If it weren't for having to use a particular conversion utility, we could reduce the script to the following pipeline (ignoring options for now):

In this code fragment, scale is set to a default of the empty string. If -s is not given on the command line, then the final line evaluates with $scale as the empty string and the pipeline will "collapse" into:

Using this principle, we can modify the previous version of the procimage script and produce a pipeline version. For each input file we need to construct and run a pipeline based upon the options given on the command line. Here is the new version:

This version has been simplified somewhat from the previous one in that it no longer needs a temporary file to hold the converted file. It is also a lot easier to read and understand. To show how easy it is to add further processing to the script, we'll now add one more NetPBM utility.

NetPBM provides a utility to enhance an image and make it sharper: pnmnlfilt. This utility is an image filter that samples the image and can enhance edges in the image (it can also smooth the image if given the appropriate values). It takes two parameters that tell it how much to enhance the image. For the purposes of our script, we'll just choose some optimal values and provide an option to switch enhancement on and off in the script.

To put the new capability in place all we have to do is add the new option (-S) to the getopts case statement, update the usage line, and add a new variable to the pipeline. Here is the new code:

We could go on forever with increasingly complex examples of eval, but we'll settle for concluding the chapter with a few exercises. The questions in Exercise 3 are really more like items on the menu of food for thought.

Here are a couple of ways to enhance procimage, the graphics utility:
1. Add an option, -q, that allows the user to turn on and off the printing of diagnostic information from the NetPBM utilities. You'll need to map -q to the -quiet option of the utilities. Also, add your own diagnostic output for those utilities that don't print anything, e.g., the format conversions.
2. Add an option that allows the user to specify the order that the NetPBM processes take place, i.e., whether enhancing the image comes before bordering, or bordering comes before resizing. Rather than using an if construct to make the choice amongst hard-coded orders, construct a string dynamically which will look similar to this:
  "eval $convert $filename $scale $border $sharpness $standardise > $outfile"
3. You'll then need eval to evaluate this string.
The function makecmd in the solution to Task 7-3 represents an oversimplification of the real make's functionality. make actually checks file dependencies recursively, meaning that a source on one line in a makefile can be a target on another line. For example, the book chapters in the example could themselves depend on some figures in separate files that were made with a graphics package.
1. Write a function called readtargets that goes through the makefile and stores all of the targets in a variable or temporary file.
2. makecmd merely checks to see if any of the sources are newer than the given target. It should really be a recursive routine that looks like this:
  function makecmd ( ) { target=$1 get sources for $target for each source src; do if $src is also a target in this makefile then makecmd $src fi if [ $src -nt $target ]; then run commands to make target return fi done }
3. Implement this.
4. Write the "driver" script that turns the makecmd function into a full make program. This should make the target given as argument, or if none is given, the first target listed in the makefile.
5. The above makecmd still doesn't do one important thing that the real make does: allow for "symbolic" targets that aren't files. These give make much of the power that makes it applicable to such an incredible variety of situations. Symbolic targets always have a modification time of 0, so that make always runs the commands to make them. Modify makecmd so that it allows for symbolic targets. (Hint: the crux of this problem is to figure out how to get a file's modification time. This is quite difficult.)
Here are some problems that really test your knowledge of eval and the shell's command-line processing rules. Solve these and you're a true bash hacker!
1. Advanced shell programmers sometimes use a little trick that includes eval: using the value of a variable as the name of another variable. In other words, you can give a shell script control over the names of variables to which it assigns values. The latest version of bash has this built in in the form of ${!varname}, where varname contains the name of another variable that will be the target of the operation. This is known as indirect expansion. How would you do this only using eval? (Hint: if $object equals "person", and $person is "alice", then you might think that you could type echo $$object and get the response alice. This doesn't actually work, but it's on the right track.)
2. You could use the above technique together with other eval tricks to implement new control structures for the shell. For example, see if you can write a script that emulates the behavior of a for loop in a conventional language like C or Pascal, i.e., a loop that iterates a fixed number of times, with a loop variable that steps from 1 to the number of iterations (or, for C fans, 0 to iterations-1). Call your script loop to avoid clashes with the keywords for and do.
3. The pushd, popd, and dirs functions that we built up in previous chapters can't handle directories with spaces in their names (because DIR_STACK uses a space as a delimiter). Use eval to overcome this limitation. (Hint: use eval to implement an array. Each array element is called array1, array2, ... arrayn, and each array element contains a directory name.)
4. (The following doesn't have that much to do with the material in this chapter per se, but it is a classic programming exercise:) Write the function alg2rpn used in the section on command blocks. Here's how to do this: Arithmetic expressions in algebraic notation have the form expr op expr, where each expr is either a number or another expression (perhaps in parentheses), and op is +, -, x, /, or % (remainder). In RPN, expressions have the form expr expr op. For example: the algebraic expression 2+3 is 2 3 + in RPN; the RPN equivalent of (2+3) x (9-5) is 2 3 + 9 5 - x. The main advantage of RPN is that it obviates the need for parentheses and operator precedence rules (e.g., x is evaluated before +). The dc program accepts standard RPN, but each expression should have "p" appended to it, which tells dc to print its result; e.g., the first example above should be given to dc as 2 3 + p.
5. You need to write a routine that converts algebraic notation to RPN. This should be (or include) a function that calls itself (a recursive function) whenever it encounters a subexpression. It is especially important that this function keep track of where it is in the input string and how much of the string it "eats up" during its processing. (Hint: make use of the pattern-matching operators discussed in Chapter 4 to ease the task of parsing input strings.) To make your life easier, don't worry about operator precedence for now; just convert to RPN from left to right: e.g., treat 3+4x5 as (3+4)x5 and 3x4+5 as (3x4)+5. This makes it possible for you to convert the input string on the fly, i.e., without having to read in the whole thing before doing any processing.
6. Enhance your solution to the previous exercise so that it supports operator precedence in the "usual" order: x, /, % (remainder) +, -. For example, treat 3+4x5 as 3+(4x5) and 3x4+5 as (3x4)+5.
7. Here is something else to really test your skills; write a graphics utility script, index, that takes a list of image files, reduces them in size and creates an "index" image. An index image is comprised of thumbnail-sized versions of the original images, placed neatly in columns and rows, and with a caption underneath (usually the name of the original file). Besides the list of files, you'll need some options, including the number of columns to create and the size of the thumbnail images. You might also like to include an option to specify the gap between each image. The new NetPBM utilities you'll need are pbmtext and pnmcat. You'll also need pnmscale and one or more of the conversion utilities, depending upon whether you decide to take in various formats (as we did for procimage) and what output format you decide on. pbmtext takes as an argument some text and converts the text into a PNM bitmap. pnmcat is a little more complex. Like cat, it concatenates things; in this case, images. You can specify as many PNM files as you like as arguments and pnmcat will put them together into one long image. By using the -lr and -tb options, you can specify whether you want the images to be placed one after the other going from left to right, or from top to bottom. The first option to pnmcat is the background color. It can be either -black for a black background, or -white for a white background. We suggest -white to match the pbmtext black text on a white background. You'll need to take each file, run the filename through pbmtext, and use pnmcat to place it underneath a scaled down version of the original image. Then you'll need to continue doing this for each file and use pnmcat to connect them together. In addition, you'll have to keep tabs on how many columns you have completed and when to start a new row. Note that you'll need to build up the rows individually and use pnmcat to connect them together. pnmcat won't do this for you automatically.