Advanced Bash-Scripting Guide

A complete guide to shell scripting, using Bash

Mendel Cooper

Brindlesoft

thegrendel@theriver.com

Revision History
Revision 0.4	08 July 2001	Revised by: mc
More bugfixes, much more material, more scripts - a complete revision and expansion of the book
Revision 0.3	12 February 2001	Revised by: mc
Another major update.
Revision 0.2	30 October 2000	Revised by: mc
Bugs fixed, plus much additional material and more example scripts.
Revision 0.1	14 June 2000	Revised by: mc
Initial release.

This document is both a tutorial and a reference on shell scripting with Bash. It assumes no previous knowledge of scripting or programming, but progresses rapidly toward an intermediate/advanced level of instruction. [1] The exercises and heavily-commented examples invite active reader participation. Still, it is a work in progress. The intention is to add much supplementary material in future updates to this document, as it evolves into a comprehensive book that matches or surpasses any of the shell scripting manuals in print.

The latest version of this document, as an archived "tarball" including both the SGML source and rendered HTML, may be downloaded from the author's home site.

This is a major update on version 0.3. -- more bugs swatted, plus much additional material and example scripts added. This project has now reached the equivalent of a 300-page book. See NEWS for a revision history.

Table of Contents

1. Why Shell Programming?

2. Starting Off With a Sha-Bang

2.1. Invoking the script
2.2. Shell wrapper, self-executing script

3. Tutorial / Reference

3.1. Exit and Exit Status

3.2. Special characters used in shell scripts

3.3. Introduction to Variables and Parameters

3.3.1. Parameter Substitution

3.4. Quoting

3.5. Tests

3.5.1. File test operators
3.5.2. Comparison operators (binary)
3.5.3. Nested if/then Condition Tests

3.6. Operations and Related Topics

3.6.1. Operators
3.6.2. Numerical Constants

3.7. Variables Revisited

3.7.1. Typing variables: declare or typeset
3.7.2. Indirect References to Variables
3.7.3. $RANDOM: generate random integer
3.7.4. The Double Parentheses Construct

3.8. Loops and Branches

3.8.1. Loops
3.8.2. Nested Loops
3.8.3. Loop Control Commands
3.8.4. Testing and Branching

3.9. Internal Commands and Builtins

3.9.1. Job Control Commands

3.10. External Filters, Programs and Commands

3.10.1. Basic Commands
3.10.2. Complex Commands
3.10.3. Time / Date Commands
3.10.4. Text Processing Commands
3.10.5. File and Archiving Commands
3.10.6. Communications Commands
3.10.7. Terminal Control Commands
3.10.8. Math Commands
3.10.9. Miscellaneous Commands

3.11. System and Administrative Commands

3.12. Command Substitution

3.13. Arithmetic Expansion

3.14. I/O Redirection

3.14.1. Using exec
3.14.2. Redirecting Code Blocks
3.14.3. Applications

3.15. Here Documents

3.16. Recess Time

3.17. Regular Expressions

3.17.1. A Brief Introduction to Regular Expressions
3.17.2. Using REs in Scripts

3.18. Subshells

3.19. Restricted Shells

3.20. Process Substitution

3.21. Functions

3.21.1. Complex Functions and Function Complexities
3.21.2. Local Variables and Recursion

3.22. Aliases

3.23. List Constructs

3.24. Arrays

3.25. Files

3.26. /dev and /proc

3.27. Of Zeros and Nulls

3.28. Debugging

3.29. Options

3.30. Gotchas

3.31. Scripting With Style

3.32. Miscellany

3.32.1. Interactive and non-interactive shells and scripts
3.32.2. Tests and Comparisons
3.32.3. Optimizations
3.32.4. Assorted Tips

3.33. Bash, version 2

5. End Notes

5.1. Author's Note
5.2. Tools Used to Produce This Book

Bibliography

A. Contributed Scripts

B. A Sed and Awk Micro-Primer

B.1. Sed
B.2. Awk

C. Exit Codes With Special Meanings

D. A Detailed Introduction to I/O and I/O Redirection

E. Localization

F. A Sample .bashrc File

G. Converting DOS Batch Files to Shell Scripts

H. Copyright

List of Examples
2-1. cleanup: A script to clean up the log files in /var/log
2-2. cleanup: An enhanced and generalized version of above script.
2-3. shell wrapper
2-4. A slightly more complex shell wrapper
2-5. A shell wrapper around an awk script
2-6. Perl embedded in a bash script
3-1. exit / exit status
3-2. Negating a condition using !
3-3. Code blocks and I/O redirection
3-4. Saving the results of a code block to a file
3-5. Backup of all files changed in last day
3-6. Variable assignment and substitution
3-7. Using param substitution and :
3-8. Length of a variable
3-9. Pattern matching in parameter substitution
3-10. Renaming file extensions:
3-11. Using pattern matching to parse arbitrary strings
3-12. Echoing Weird Variables
3-13. Escaped Characters
3-14. What is truth?
3-15. Equivalence of [ ] and test
3-16. Arithmetic Tests using (( ))
3-17. arithmetic and string comparisons
3-18. testing whether a string is null
3-19. zmost
3-20. Using Arithmetic Operations
3-21. Compound Condition Tests Using && and ||
3-22. Representation of numerical constants:
3-23. $IFS and whitespace
3-24. Timed Input
3-25. Once more, timed input
3-26. Am I root?
3-27. arglist: Listing arguments with $* and $@
3-28. Inconsistent $* and $@ behavior
3-29. $* and $@ when $IFS is empty
3-30. underscore variable
3-31. Variable Assignment
3-32. Variable Assignment, plain and fancy
3-33. Positional Parameters
3-34. wh, whois domain name lookup
3-35. Using shift
3-36. Using declare to type variables
3-37. Indirect References
3-38. Passing an indirect reference to awk
3-39. Generating random numbers
3-40. Rolling the die with RANDOM
3-41. Reseeding RANDOM
3-42. C-type manipulation of variables
3-43. Simple for loops
3-44. for loop with two parameters in each [list] element
3-45. Fileinfo: operating on a file list contained in a variable
3-46. Operating on files with a for loop
3-47. Missing in [list] in a for loop
3-48. Generating the [list] in a for loop with command substitution
3-49. A grep replacement for binary files
3-50. Checking all the binaries in a directory for authorship
3-51. Listing the symbolic links in a directory
3-52. A C-like for loop
3-53. Using efax in batch mode
3-54. Simple while loop
3-55. Another while loop
3-56. while loop with multiple conditions
3-57. C-like syntax in a while loop
3-58. until loop
3-59. Nested Loop
3-60. Effects of break and continue in a loop
3-61. Breaking out of multiple loop levels
3-62. Continuing at a higher loop level
3-63. Using case
3-64. Creating menus using case
3-65. Using command substitution to generate the case variable
3-66. Checking for alphabetic input
3-67. Creating menus using select
3-68. Creating menus using select in a function
3-69. printf in action
3-70. Using getopts to read the options/arguments passed to a script
3-71. Showing the effect of eval
3-72. Forcing a log-off
3-73. A version of "rot13"
3-74. Effects of exec
3-75. Using set with positional parameters
3-76. "unsetting" a variable
3-77. Using export to pass a variable to an embedded awk script
3-78. Variable assignment, using read
3-79. Using read with file redirection
3-80. Letting let do some arithmetic.
3-81. Changing the current working directory
3-82. "Including" a data file
3-83. Waiting for a process to finish before proceeding
3-84. Using ls to create a table of contents for burning a CDR disk
3-85. Badname, eliminate file names in current directory containing bad characters and whitespace.
3-86. Log file using xargs to monitor system log
3-87. copydir, copying files in current directory to another, using xargs
3-88. Using expr
3-89. Using date
3-90. Using cmp to compare two files within a script.
3-91. Generating 10-digit random numbers
3-92. Using tail to monitor the system log
3-93. Emulating "grep" in a script
3-94. Checking words in a list for validity
3-95. toupper: Transforms a file to all uppercase.
3-96. lowercase: Changes all filenames in working directory to lowercase.
3-97. du: DOS to UNIX text file conversion.
3-98. rot13: rot13, ultra-weak encryption.
3-99. Formatted file listing.
3-100. Using column to format a directory listing
3-101. nl: A self-numbering script.
3-102. Using cpio to move a directory tree
3-103. stripping comments from C program files
3-104. Exploring /usr/X11R6/bin
3-105. basename and dirname
3-106. uudecoding encoded files
3-107. Monthly Payment on a Mortgage
3-108. Base Conversion
3-109. Using seq to generate loop arguments
3-110. Capturing Keystrokes
3-111. secret password: Turning off terminal echoing
3-112. Keypress detection
3-113. pidof helps kill a process
3-114. Checking a CD image
3-115. Adding a new hard drive
3-116. Creating a filesystem in a file
3-117. killall, from /etc/rc.d/init.d
3-118. Redirecting stdin using exec
3-119. Redirected while loop
3-120. Alternate form of redirected while loop
3-121. Redirected until loop
3-122. Redirected for loop
3-123. Redirected if/then test
3-124. Logging events
3-125. dummyfile: Creates a 2-line dummy file
3-126. broadcast: Sends message to everyone logged in
3-127. Multi-line message using cat
3-128. Multi-line message, with tabs suppressed
3-129. Here document with parameter substitution
3-130. Parameter substitution turned off
3-131. upload: Uploads a file pair to "Sunsite" incoming directory
3-132. "Anonymous" Here Document
3-133. Variable scope in a subshell
3-134. List User Profiles
3-135. Running parallel processes in subshells
3-136. Running a script in restricted mode
3-137. Simple function
3-138. Function Taking Parameters
3-139. Maximum of two numbers
3-140. Converting numbers to Roman numerals
3-141. Real name from username
3-142. Local variable visibility
3-143. Recursion, using a local variable
3-144. Aliases within a script
3-145. unalias: Setting and unsetting an alias
3-146. Using an "and list" to test for command-line arguments
3-147. Another command-line arg test using an "and list"
3-148. Using "or lists" in combination with an "and list"
3-149. Simple array usage
3-150. Some special properties of arrays
3-151. Of empty arrays and empty elements
3-152. An old friend: The Bubble Sort
3-153. Complex array application: Sieve of Erastosthenes
3-154. Complex array application: Exploring a weird mathematical series
3-155. Simulating a two-dimensional array, then tilting it
3-156. Finding the process associated with a PID
3-157. On-line connect status
3-158. Hiding the cookie jar
3-159. Setting up a swapfile using /dev/zero
3-160. test23, a buggy script
3-161. test24, another buggy script
3-162. Trapping at exit
3-163. Cleaning up after Control-C
3-164. Tracing a variable
3-165. Subshell Pitfalls
3-166. String expansion
3-167. Indirect variable references - the new way
3-168. Using arrays and other miscellaneous trickery to deal four random hands from a deck of cards
A-1. manview: A script for viewing formatted man pages
A-2. mailformat: Formatting an e-mail message
A-3. rn: A simple-minded file rename utility
A-4. encryptedpw: A script for uploading to an ftp site, using a locally encrypted password
A-5. copy-cd: A script for copying a data CD
A-6. behead: A script for removing mail and news message headers
A-7. ftpget: A script for downloading files via ftp
A-8. password: A script for generating random 8-character passwords
A-9. fifo: A script for making daily backups, using named pipes
A-10. Generating prime numbers using the modulo operator
A-11. tree: A script for displaying a directory tree
A-12. string functions: C-like string functions
A-13. Object-oriented database
F-1. Sample .bashrc file
G-1. VIEWDATA.BAT: DOS Batch File
G-2. viewdata.sh: Shell Script Conversion of VIEWDATA.BAT

Notes

[1]	...all the while sneaking in little snippets of UNIX wisdom and lore.

		Next
		Why Shell Programming?

| Français | English |