Bash Shell Expansions: Brace Expansion, Parameter Expansion and more

In this article we will cover all the basic features of Bash Shell expansion.  Some of the most complex and interesting expansions are the Brace Expansion and Parameter Expansion which have many features and options which are powerful but only mastered over time by BASH programmers and linux devops folks.  Word Splitting is also quite interesting and sometime overlooked.  Filename, Arithmetic Expansion and Variable substitution are well known.  We will cover numerous topics and show examples of the command and most useful syntaxes for each syntax.  So let’s get started.

• Environment
• Command Substitution
• Process Substitution
• Variable Substitution
• Brace Expansion
• Parameter Expansion
• Positional Parameters
• Tilde Expansion
• Arithmetic Substitution
• Word Splitting
• Filename Expansion
• Conclusion

Environment

In order to test all of the bash shell expansions features we need to make sure we are running a recent version of bash.  Below is the system information for this article. The tests in this article are running on Ubuntu 19.10 as shown below.

The bash version for these tests is bash version 5, which is quite recent.  Older versions of bash are missing a bunch of features.

Command Substitution

Command substitution allows the running of one or multiple commands and capturing outputs and actions from those commands and including them in another command all in one line or less lines than running all the commands separately.  Command Substitution has two syntaxes; the more popular syntax is the backtick syntax where the command to be executed is enclosed in two backquotes, or backticks.  The other syntax which is equally powerful encloses commands in $() syntax and the output can be used from that new expansion.  Let’s look at a number of examples of Command Substitution below.

Simple command substitution using $() syntax to run the date command.

Simple command substitution using backtick syntax to run the date command.

Using the stdin operator at the beginning of command substitution syntax is a fancy way to read the text of a file into a variable and use it in a command on the shell as below.

Read a file into a variable to be used in a command using the cat command and Command Substitution.

Same as above, read a file and use it in Command Substitution using backticks and cat command.

Combine embedded Command Substitution with another Command Substitution using both $() and backticks together

Embedded Command Substitution inside another one using two $() syntax operations

Use output from a command as arguments into another command, with the backtick syntax.  We will get a list of files by running cat which contains one file per line and then pass that into the rm command which will remove each file

Same as above but with $() syntax, pass command output from cat into rm command to delete files.

Store the output from a cat command into a variable and then loop through the variable so you can more clearly see what is happening.

Same as above but use backticks syntax to run the cat command and store the output in a variable and then loop through the files int the variable.

Use the Command Substitution with stdin operator to read a file line by line into a variable and then loop through the variable afterwords

Process Substitution

Process Substitution is a documented feature of bash; it’s quite cryptic in my opinion.  In fact I have not found many good use cases to recommend for it.  One example is included here for completeness where we use Process Substitution to get the output of a command and then use it another command.  We will print the list of files in reverse order with sort command in this example after fetching files from the ls command.

Variable Substitution

Variable Substitution is what you can consider basic usage of variables and substituting the value of the variable when it is referenced.  Its fairly intuitive, a few examples are provided below.

Simple variable assignment and usage where we put a string into variable X and then print it to stdout

Check if a variable is assigned something or null, in this case it’s assigned so we print it to stdout

Check if a variable is assigned something or null, in this case it’s not set so we print “is null” instead of the value.

Brace Expansion

Brace Expansion is a super powerful feature of bash that allows you to write more compact scripts and commands.  It has many different features and options described below.  Within braces your syntax is interpreted into a more verbose syntax depending when you enter into the curly braces.  Let’s look at a number of examples for Brace Expansion.

Each version of the items in the list in braces is executed. So we go from one echo command and print 3 versions of the word below separated by spaces.

Expressions in the expansion causes execution multiple times.  To prove this we use the date and sleep command to validate that the date command is run once for each iteration of the pattern in the Brace Expansion.

Expansions using numbers with .. will cause sequential numbers to be expanded in a numerical sequence

Reverse order brace expansion with sequence of numbers

Using an optional increment value to specify the numerical increments of brace expansion

Lexicographical brace expansion will iterate through letters in the alphabet in the order of the locale

Reverse order lexicographical brace expansion

Lexicographical brace expansion with increment specified will iterate through a list of characters from the begin to the end point but skip characters according the increment value

Multiplicative brace expansion with 2 brace expansions in one command

Brace expansion to use the same root two times in a command.  This creates a foo.tgz tar file from a directory under name foo.  Its a handy syntax where you are using it inside another loop and want to assume that the base of the word is used multiple times.  This example shows it with tar, but it can also be used with mv and cp as per this example.

Parameter Expansion

Parameter expansion is also a nice compact syntax with many capabilities such as: allow scripts to set default values for unset variables or options, string substring operations, search and replace substitutions and other use cases.  Examples are below.
 
Check for null and use the parameter if not null or the default value.  In this case X is not null so it will be used

Check for null and use the parameter if not null or the default value.  In this case X is null so the default value will be used

Check if the variable is NULL and set and echo it if it is NULL.  X is assigned 2 and printed $X.  This can both set the variable and use it in the command with the ${:=} syntax.

Substring expansion will substitute from an offset point a certain number of characters in the string

Change the offset to the second character and print 7 characters of substring

Substring from beginning of string but cut off last 2 characters

Get a string length with this version of parameter expansion

Search and replace within a variable.  In this example replace the first lowercase o with uppercase O

Search and replace within a variable but with all matches replaced because of the leading slash in the search pattern.

Patterns starting with #, mean the match must start at the beginning of the string in order to be substituted

Example where searching for match at beginning of string, but failing because match is later in the string

Patterns beginning with % will only match at the end of string like in this example.

Example for end of string match which fails because the match is in the beginning of string.

Use shopt with nocasematch to do case insensitive replacement.

Turn off shopt with nocasematch to do case sensitive replacement.

Search for environment variables that match a pattern.

Get a list of matching variables and then loop through each variable and print its value

Make a string all uppercase

Positional Parameters

Positional Parameters are normally thought of as command line parameters, how to use them are shown with examples below.

Parameter $0 is the script name that is running and then $1, $2, $3 etc are command line parameters passed to a script.

Parameter $* is a single variable with all the command line arguments concatenated.

Parameter $# is a number with the quantity of positional parameters passed to a script in this case below there are 2 arguments passed.

Tilde Expansion

Tilde expansion is commonly seen with usernames and home directories, examples are shown below.

Tilde Expansion to get the HOME directory of the current user, using just tilde without the username.

Refer to a specific user’s home directory, not the current user with Tilde and the username

Arithmetic Substitution

Arithmetic Substitution allows bash to do mathematical operations in the shell or in a script.  Examples of common usages are shown below.

Simple Arithmetic Substitution with $ and double parentheses

Post increment operator will update the value by one after the current command, note there is an equivalent post decrement not shown here.

Pre increment operator will update the value by one just before the current command, note there is an equivalent pre decrement operator not shown here.

Exponent operator can raise a number to a power exponentially

Left bitwise shift; in this case shift the bits of the decimal number 8 to the left which essentially multiplies it by 2

Right bitwise shift; in this case shift the bits of the decimal number 8 to the right which essentially divides the number by 2

Bitwise AND Operator will compare the numbers bit by bit and and the result will be the bits that are all set.

Bitwise OR Operator will compare the numbers bit by bit and the results will be the bits where either of the inputs has the bit set.

Arithmetic equality operator will test for truth and return 1 or 0

Arithmetic inequality operator will test for non-equality and return 1 or 0

The Conditional Operator will test the first argument if true, replace with the second argument and if false replace with the third.  In this case 5 equals 4+1 so the first conditional is true and 9 is returned.  5 does not equal 4+2 so in the second echo 7 is returned.

You can use hexadecimal numbers in arithmetic expansions, in this case 0xa is equivalent to 10 and 10+7 = 17.

Word Splitting

Using the IFS environment variable to register a delimiter, and using the read and readarray commands we can parse strings into an array of tokens and then count the tokens and operate on them.  Examples are shown below.
 
Use IFS parameter as delimiter, read tokens into an array split by IFS which is set to a space character, and then print out the tokens one by one

There are 2 words in the text.

User readarray without IFS and specify delimiter in the readarray command.  Note this is just an example where we split a directory path based on the slash delimiter.  In this case the code has included the empty string before the first slash which would need to be adjusted in a real usage, but we are just showing how to call readarray to split a string into tokens in an array with a delimiter.

There are 6 words in the text.

Filename Expansion

When wanting to refer to a list of files or directories in the filesystem, a bash command or bash script can use Filename Expansion to generate a list of files and directories from simple commands.  Examples are shown below.

The * character expands to a wildcard and picks up all matching files with the rest of the wild card string.  Here we pick up all files ending in .txt and pass them into the du command for checking disk size.

The ? character will only match a single character not an infinite number of characters, and therefore in this example will only pickup filenames with a single character followed by .txt.

Characters in brackets expand to match any of the characters.  In this example a.txt and c.txt are picked up by the expansion

Characters in brackets can be a range of characters and we see here all files from a to c range followed by .txt suffix are picked up

Conclusion

We have covered many types of shell expansions in this article, and I hope the simple examples can serve as a cookbook for what is possible in bash to make you more productive with shell expansions.  As further references I recommend reading the full Bash Manual, and also the many good articles on NixCraft website about bash scripting including Shell Expansions.  We have other articles that may be of interest to you on LinuxHint including: 30 Bash Script Examples, Bash Lowercase Uppercase Strings, Bash Pattern Matching, and Bash Split String Examples.  Also we have a popular free 3 hour course on Bash Programming you can find on YouTube.