3.14. I/O Redirection

There are always three default "files" open, stdin (the keyboard), stdout (the screen), and stderr (error messages output to the screen). These, and any other open files, can be redirected. Redirection simply means capturing output from a file, command, program, script, or even code block within a script (see Example 3-3 and Example 3-4) and sending it as input to another file, command, program, or script.

Each open file gets assigned a file descriptor. [1] The file descriptors for stdin, stdout, and stderr are 0, 1, and 2, respectively. For opening additional files, there remain descriptors 3 to 9. It is sometimes useful to assign one of these additional file descriptors to stdin, stdout, or stderr as a temporary duplicate link. [2] This simplifies restoration to normal after complex redirection and reshuffling (see Example 3-118).

   1    >
   2       # Redirect stdout to a file.
   3       # Creates the file if not present, otherwise overwrites it.
   4 
   5       ls -lR > dir-tree.list
   6       # Creates a file containing a listing of the directory tree.
   7 
   8    : > filename
   9       # The > truncates file "filename" to zero length.
  10       # The : serves as a dummy placeholder, producing no output.
  11 
  12    >>
  13       # Redirect stdout to a file.
  14       # Creates the file if not present, otherwise appends to it.
  15 
  16    2>&1
  17       # Redirects stderr to stdout.
  18       # Error messages get sent to same place as standard output.
  19 
  20    i>&j
  21       # Redirects file descriptor i to j.
  22       # All output of file pointed to by i gets sent to file pointed to by j.
  23 
  24    >&j
  25       # Redirects, by default, file descriptor 1 (stdout) to j.
  26       # All stdout gets sent to file pointed to by j.
  27 
  28    0<
  29     <
  30       # Accept input from a file.
  31       # Companion command to ">", and often used in combination with it.
  32       #
  33       # grep search-word <filename
  34 
  35 
  36    [j]<>filename
  37       # Open file "filename" for reading and writing, and assign file descriptor "j" to it.
  38       # If "filename" does not exist, create it.
  39       # If file descriptor "j" is not specified, default to fd 0, stdin.
  40       #
  41       # An application of this is writing at a specified place in a file. 
  42       echo 1234567890 > File    # Write string to "File".
  43       exec 3<> File             # Open "File" and assign fd 3 to it.
  44       read -n 4 <&3             # Read only 4 characters.
  45       echo -n . >&3             # Write a decimal point there.
  46       exec 3>&-                 # Close fd 3.
  47       cat File                  # ==> 1234.67890
  48       # Random access, by golly.
  49 
  50 
  51 
  52    |
  53       # Pipe.
  54       # General purpose process and command chaining tool.
  55       # Similar to ">", but more general in effect.
  56       # Useful for chaining commands, scripts, files, and programs together.
  57       cat *.txt | sort | uniq > result-file
  58       # Sorts the output of all the .txt files and deletes duplicate lines,
  59       # finally saves results to "result-file".

Multiple instances of input and output redirection and/or pipes can be combined in a single command line.

   1 command < input-file > output-file
   2 
   3 command1 | command2 | command3 > output-file

Multiple output streams may be redirected to one file.

   1 ls -yz >> command.log 2>&1
   2 # Capture result of illegal options "yz" to "ls" in file "command.log".
   3 # Because stderr redirected to the file, any error messages will also be there.

Closing File Descriptors

n<&-

Close input file descriptor n.

0<&-, <&-

Close stdin.

n>&-

Close output file descriptor n.

1>&-, >&-

Close stdout.

Child processes inherit open file descriptors. This is why pipes work. To prevent an fd from being inherited, close it.

   1 # Redirecting only stderr to a pipe.
   2 
   3 exec 3>&1                              # Save current "value" of stdout.
   4 ls -l 2>&1 >&3 3>&- | grep bad 3>&-    # Close fd 3 for 'ls' and 'grep'.
   5 exec 3>&-                              # Now close it for the remainder of the script.
   6 
   7 # Thanks, S.C.

For a more detailed introduction to I/O redirection see Appendix D.

3.14.1. Using exec

The exec <filename command redirects stdin to a file. From that point on, all stdin comes from that file, rather than its normal source (usually keyboard input). This provides a method of reading a file line by line and possibly parsing each line of input using sed and/or awk.


Example 3-118. Redirecting stdin using exec

   1 #!/bin/bash
   2 # Redirecting stdin using 'exec'.
   3 
   4 
   5 exec 6<&0   # Link file descriptor #6 with stdin.
   6 
   7 exec < data-file   # stdin replaced by file "data-file"
   8 
   9 read a1   # Reads first line of file "data-file".
  10 read a2   # Reads second line of file "data-file."
  11 
  12 echo
  13 echo "Following lines read from file."
  14 echo "-------------------------------"
  15 echo $a1
  16 echo $a2
  17 
  18 echo; echo; echo
  19 
  20 exec 0<&6 6<&-
  21 # Now restore stdin from fd #6, where it had been saved,
  22 # and close fd #6 ( 6<&- ) to free it for other processes to use.
  23 # <&6 6<&-    also works.
  24 
  25 echo -n "Enter data  "
  26 read b1  # Now "read" functions as expected, reading from normal stdin.
  27 echo "Input read from stdin."
  28 echo "----------------------"
  29 echo "b1 = $b1"
  30 
  31 echo
  32 
  33 exit

3.14.2. Redirecting Code Blocks

Blocks of code, such as while, until, and for loops, even if/then test blocks can also incorporate redirection of stdin. Even a function may use this form of redirection (see Example 3-141). The < operator at the the end of the code block accomplishes this.


Example 3-119. Redirected while loop

   1 #!/bin/bash
   2 
   3 if [ -z "$1" ]
   4 then
   5   Filename=names.data  # Default, if no filename specified.
   6 else
   7   Filename=$1
   8 fi  
   9 # Filename=${1:-names.data}  can replace the above test.
  10 # (parameter substitution)
  11 
  12 count=0
  13 
  14 echo
  15 
  16 while [ "$name" != Smith ]  # Why is variable $name in quotes?
  17 do
  18   read name         # Reads from $Filename, rather than stdin.
  19   echo $name
  20   let "count += 1"
  21 done <"$Filename"   # Redirects stdin to file $Filename. 
  22 
  23 echo; echo "$count names read"; echo
  24 
  25 # Note that in some older shell scripting languages,
  26 # the redirected loop would run as a subshell.
  27 # Therefore, $count would return 0, the initialized value outside the loop.
  28 # Bash and ksh avoid starting a subshell whenever possible,
  29 # so that this script, for example, runs correctly.
  30 # Thanks to Heiner Steven for pointing this out.
  31 
  32 exit 0


Example 3-120. Alternate form of redirected while loop

   1 #!/bin/bash
   2 
   3 # This is an alternate form of the preceding script.
   4 
   5 # Suggested by Heiner Steven
   6 # as a workaround in those situations when a redirect loop
   7 # runs as a subshell, and therefore variables inside the loop
   8 # do not keep their values upon loop termination.
   9 
  10 
  11 if [ -z "$1" ]
  12 then
  13   Filename=names.data  # Default, if no filename specified.
  14 else
  15   Filename=$1
  16 fi  
  17 
  18 
  19 exec 3<&0                 # Save stdin to file descriptor 3.
  20 exec 0<"$Filename"        # Redirect standard input.
  21 
  22 count=0
  23 echo
  24 
  25 
  26 while [ "$name" != Smith ]
  27 do
  28   read name         # Reads from redirected stdin ($Filename).
  29   echo $name
  30   let "count += 1"
  31 done <"$Filename"   # Loop reads from file $Filename. 
  32 
  33 
  34 exec 0<&3               # Restore old stdin.
  35 exec 3<&-               # Close temporary fd 3.
  36 
  37 echo; echo "$count names read"; echo
  38 
  39 exit 0


Example 3-121. Redirected until loop

   1 #!/bin/bash
   2 # Same as previous example, but with "until" loop.
   3 
   4 if [ -z "$1" ]
   5 then
   6   Filename=names.data  # Default, if no filename specified.
   7 else
   8   Filename=$1
   9 fi  
  10 
  11 # while [ "$name" != Smith ]
  12 until [ "$name" = Smith ]     # Change  !=  to =.
  13 do
  14   read name         # Reads from $Filename, rather than stdin.
  15   echo $name
  16 done <"$Filename"   # Redirects stdin to file $Filename. 
  17 
  18 # Same results as with "while" loop in previous example.
  19 
  20 exit 0


Example 3-122. Redirected for loop

   1 #!/bin/bash
   2 
   3 if [ -z "$1" ]
   4 then
   5   Filename=names.data  # Default, if no filename specified.
   6 else
   7   Filename=$1
   8 fi  
   9 
  10 line_count=`wc $Filename | awk '{ print $1 }'`  # Number of lines in target file.
  11 # Very contrived and kludgy, nevertheless shows that
  12 # it's possible to redirect stdin within a "for" loop...
  13 # if you're clever enough.
  14 #
  15 # More concise is     line_count=$(wc < "$Filename")
  16 
  17 
  18 for name in `seq $line_count`  # Recall that "seq" prints sequence of numbers.
  19 # while [ "$name" != Smith ]   --   more complicated than a "while" loop   --
  20 do
  21   read name         # Reads from $Filename, rather than stdin.
  22   echo $name
  23   if [ "$name" = Smith ]   # Need all this extra baggage here.
  24   then
  25     break
  26   fi  
  27 done <"$Filename"   # Redirects stdin to file $Filename. 
  28 
  29 exit 0


Example 3-123. Redirected if/then test

   1 #!/bin/bash
   2 
   3 if [ -z "$1" ]
   4 then
   5   Filename=names.data  # Default, if no filename specified.
   6 else
   7   Filename=$1
   8 fi  
   9 
  10 TRUE=1
  11 
  12 if [ "$TRUE" ]   # if true    and   if :   also work.
  13 then
  14  read name
  15  echo $name
  16 fi <"$Filename"
  17 # Reads only first line of file.
  18 # An if/then test has no way of iterating unless embedded in a loop.
  19 
  20 exit 0

Note

Here documents are a special case of redirected code blocks.

3.14.3. Applications

Clever use of I/O redirection permits parsing and stitching together snippets of command output (see Example 3-79). This permits generating report and log files.


Example 3-124. Logging events

   1 #!/bin/bash
   2 # logevents.sh, by Stephane Chazelas.
   3 
   4 # Event logging to a file.
   5 # Must be run as root (for write access in /var/log).
   6 
   7 
   8 FD_DEBUG1=3
   9 FD_DEBUG2=4
  10 FD_DEBUG3=5
  11 
  12 # Uncomment a line below to activate script.
  13 # LOG_EVENTS=1
  14 # LOG_VARS=1
  15 
  16 
  17 log()  # Writes time and date to log file.
  18 {
  19 echo "$(date)  $*" >&7     # This *appends* the date to the file.
  20                            # See below.
  21 }
  22 
  23 
  24 
  25 case $LOG_LEVEL in
  26  1) exec 3>&2         4> /dev/null 5> /dev/null;;
  27  2) exec 3>&2         4>&2         5> /dev/null;;
  28  3) exec 3>&2         4>&2         5>&2;;
  29  *) exec 3> /dev/null 4> /dev/null 5> /dev/null;;
  30 esac
  31 
  32 FD_LOGVARS=6
  33 if [[ $LOG_VARS ]]
  34 then exec 6>> /var/log/vars.log
  35 else exec 6> /dev/null               # Bury output.
  36 fi
  37 
  38 FD_LOGEVENTS=7
  39 if [[ $LOG_EVENTS ]]
  40 then
  41   # then exec 7 >(exec gawk '{print strftime(), $0}' >> /var/log/event.log)
  42   # Above line will not work in Bash, version 2.04.
  43   exec 7>> /var/log/event.log        # Append to "event.log".
  44   log                                # Write time and date.
  45 else exec 7> /dev/null               # Bury output.
  46 fi
  47 
  48 echo "DEBUG3: beginning" >&${FD_DEBUG3}
  49 
  50 #command1 >&5 2>&4
  51 ls -l >&5 2>&4
  52 
  53 #command2 
  54 echo "Done"
  55 
  56 echo "sending mail" >&${FD_LOGEVENTS}   # Writes "sending mail" to fd #7.
  57 
  58 
  59 exit 0

Notes

[1]

A file descriptor is simply a number that the operating system assigns to an open file to keep track of it. Consider it a simplified version of a file pointer. It is analogous to a file handle in C.

[2]

Using file descriptor 5 might cause problems. When Bash creates a child process, as with exec, the child inherits fd 5 (see Chet Ramey's archived e-mail, SUBJECT: RE: File descriptor 5 is held open). Best leave this particular fd alone.