--------------------
Shell Metacharacters
> `prog >file` direct standard output to file, (over-writes file)
>> `prog >>file` append standard output to file
< `prog <file` take standard input from file
| `p1 | p2` connect standard output of p1 to standard input of p2
<<str here document: standard input follows, up to next str on a line by
itself, one way to represent a multiline string
* match any string of zero or more characters in filenames (a glob
expansion, not a regex)
? match any single character in filenames
[ccc] match any single character from ccc in filenames; ranges
like 0-9 or a-z are legal
; command terminator: `p1;p2` does p1, then p2
& like ; but doesn't wait for p1 to finish
`...` (grave/backtick) run command(s) in ...; output replaces `...`
(...) run command(s) in ... in a sub-shell
{...} run command(s) in ... in current shell (rarely used)
$1, $2 etc. $0...$9 replaced by positional arguments to shell file
$var value of shell variable var
${var} value of var; avoids confusion when concatenated with text
\ \c take character c literally, \newline discarded
'...' take ... string literally
"..." take ... string literally after $, `...` and \ interpreted
--
$# Provides the total number of arguments passed to the shell script or function.
$*, $@ Represents all the command-line arguments at once.
"$*" Represents all the command-line arguments as a single string.
"$@" Represents all the command-line arguments as separate, individual strings.
------------------------------
POSIX built-in shell variables (preceded by $, e.g. "$#")
# Number of arguments given to current process.
@ Command-line arguments to current process. Inside double quotes, expands
to individual arguments.
* Command-line arguments to current process. Inside double quotes, expands
to a single argument.
- (hyphen) Options given to shell on invocation.
? Exit status of previous command.
$ Process ID of shell process.
PPID Process ID of parent process.
! Process ID of last background command. Use this to save process ID numbers
for later use with the wait command.
0 (zero) The name of the shell program, in most cases, including full
invocation path.
PWD Current working directory, containing no components of type symbolic link,
no components that are dot, and no components that are dot-dot when the
shell is initialized.
ENV Used only by interactive shells upon invocation; the value of $ENV is
parameter-expanded. The result should be a full pathname for a file to
be read and executed at startup. This is an XSI requirement.
HOME Home (login) directory.
IFS Internal field separator; i.e., the list of characters that act as word
separators. Normally set to space, tab, and newline.
LANG Default name of current locale; overridden by the other LC_* variables.
LC_ALL Name of current locale; overrides LANG and the other LC_* variables.
LC_COLLATE Name of current locale for character collation (sorting) purposes.
LC_CTYPE Name of current locale for character class determination during
pattern matching.
LC_MESSAGES Name of current language for output messages.
LINENO Line number in script or function of the line that just ran.
NLSPATH The location of message catalogs for messages in the language given
by $LC_MESSAGES (XSI).
PATH Search path for commands.
PS1 Primary command prompt string. Default is "$ ".
PS2 Prompt string for line continuations. Default is "> ".
PS4 Prompt string for execution tracing with set -x. Default is "+ ".
More POSIX built-ins can be found here (but seriously, don't trust all of them):
http://pubs.opengroup.org/onlinepubs/009695399/utilities/xcu_chap02.html
------------------------------------
Finding Programs in your enviornment
which <utility> Returns the path to the utility if it exists,
heritage from C shell.
command -v <utility> Returns the path to the utility, -v flag keeps it
silent, -V is default behavior. Very portable.
whereis <utility> Looks up executable, man page, and source- not based
on the current environment.
----------------------------------------------
Evaluation and Substitution of Shell Variables
$var Value of var; nothing if var undefined. ENV variables are, by
convention, typically typed all caps.
${var} Same as $var, useful if alphanumerics follow variable name.
${var-thing} Value of var if defined, otherwise thing, $var remains unchanged.
${var:-word} If var exists and isn't null, return it's value; otherwise,
return word.
Purpose: To return a default value if the variable is undefined.
Example: ${count:-0} evaluates to 0 if count is undefined.
${var:=word} If var exists and isn't null, return it's value; otherwise set it
to word and return it's value.
Purpose: To set a variable to a default value if it is undefined.
Example: ${count:=0} sets count to 0 if it is undefined.
${var:?message} If var exists and isn't null, return it's value; otherwise,
print 'var: message', and abort the current command of script.
Omitting message produces the default message parameter null or
not set. Note, however, that interactive shells do not have to
abort. (Behavior and exit code varies across shells.)
Purpose: To catch errors that result from variables being
undefined.
Example: ${count:?"undefined!"} prints 'count: undefined!' and
exits if count is undefined.
${var:+word} If var exists and isn't null, return word; otherwise, return null.
Purpose: To test for the existence of a variable.
Example: ${count:+1} returns 1 (which could mean "true") if count
is defined.
-----------------------------------------
POSIX dirname(1) and basename(1) Examples
path dirname basename
"/usr/lib" "/usr" "lib"
"/usr/" "/" "usr"
"usr" "." "usr"
"/" "/" "/"
"." "." "."
".." "." ".."
Command Results
dirname / /
dirname // / or //
dirname /a/b/ /a
dirname //a//b// //a
dirname Unspecified
dirname a . ($? = 0)
dirname "" . ($? = 0)
dirname /a /
dirname /a/b /a
dirname a/b a
-----------------------------------
Pattern-matching Varialbe Operators
${variable#pattern} If the pattern matches the beginning of the variable’s
value, delete the shortest part that matches and return
the rest.
Example: ${path#/*/} Result: tolstoy/mem/long.file.name
${variable##pattern} If the pattern matches the beginning of the variable’s
value, delete the longest part that matches and return
the rest.
Example: ${path##/*/} Result: long.file.name
${variable%pattern} If the pattern matches the end of the variable’s value,
delete the shortest part that matches and return the
rest.
Example: ${path%.*} Result: /home/tolstoy/mem/long.file
${variable%%pattern} If the pattern matches the end of the variable’s value,
delete the longest part that matches and return the rest.
Example: ${path%%.*} Result: /home/tolstoy/mem/long
-------------------------------
Subshell and code block summary
Construct Delimiters Separate Process Recognized where
- - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
Subshell ( ) Yes Anywhere on the line
Code block { } No After newline, semicolon, or keyword
----------------
Test Expressions
for use with test(1), and [ , as builtin and utility.
(By the way, did you know that '/bin/[' exists as a binary?)
string string is not null
-b file file is a block device file.
-b file file is a block device file.
-c file file is a character device file.
-d file file is a directory.
-e file file exists.
-f file file is a regular file.
-g file file has its setgid bit set.
-h file file is a symbolic link.
-L file file is a symbolic link. (Same as –h.)
-n string string is non-null.
-p file file is a named pipe (FIFO file).
-r file file is readable.
-S file file is a socket.
-s file file is not empty.
-tn File descriptor n points to a terminal.
-u file file has its setuid bit set.
-w file file is writable.
-x file file is executable, or file is a directory that can be searched.
-z string string is null.
s1=s2 Strings s1 and s2 are the same.
s1!=s2 Strings s1 and s2 are not the same.
n1 -eq n2 Integers n1 and n2 are equal.
n1 -ne n2 Integers n1 and n2 are not equal.
n1 -lt n2 n1 is less than n2.
n1 -gt n2 n1 is greater than n2.
n1 -le n2 n1 is less than or equal to n2.
n1 -ge n2 n1 is greater than or equal to n2.
---------------------
echo Escape Sequences
\a Alert character, usually the ASCII BEL character.
\b Backspace.
\c Suppress the final newline in the output. Furthermore, any characters
left in the argument, and any following arguments, are ignored (not
printed).
\f Formfeed.
\n Newline.
\r Carriage return.
\t Horizontal tab.
\v Vertical tab.
\\ A literal backslash character.
\0ddd Character represented as a 1- to 3-digit octal value.
-----------------------
printf Escape Sequences
\a Alert character, usually the ASCII BEL character.
\b Backspace.
\c Suppress any final newline in the output.a Furthermore, any characters
left in the argument, any follow- ing arguments, and any characters left
in the format string are ignored (not printed).
\f Formfeed.
\n Newline.
\r Carriage return.
\t Horizontal tab.
\v Vertical tab.
\\ A literal backslash character.
\ddd Character represented as a 1- to 3-digit octal value. Valid only in the
format string. \0ddd Character represented as a 1- to 3-digit octal
value.
--------------------------------
POSIX BRE and ERE metacharacters
\ BRE/ERE Usually, turn off the special meaning of the following
character. Occasionally, enable a special meaning for the
following character, such as for \(...\) and \{...\}.
. BRE/ERE Match any single character except NUL. Individual programs
may also disallow match- ing newline.
* BRE/ERE Match any number (or none) of the single character that
immediately precedes it. For EREs, the preceding character
can instead be a regular expression. For example, since .
(dot) means any character, .* means “match any number of any
character.” For BREs, * is not special if it’s the first
character of a regular expression.
^ BRE/ERE Match the following regular expression at the beginning of
the line or string. BRE: spe- cial only at the beginning of
a regular expression. ERE: special everywhere.
$ BRE/ERE Match the preceding regular expression at the end of the
line or string. BRE: special only at the end of a regular
expression. ERE: special everywhere.
[...] BRE/ERE Termed a bracket expression, this matches any one of the
enclosed characters. A hyphen (-) indicates a range of
consecutive characters. (Caution: ranges are locale-
sensitive, and thus not portable.) A circumflex (^) as the
first character in the brackets reverses the sense: it
matches any one character not in the list. A hyphen or close
bracket (]) as the first character is treated as a member of
the list. All other metacharac- ters are treated as members
of the list (i.e., literally). Bracket expressions may
contain collating symbols, equivalence classes, and
character classes (described shortly).
\{n,m\} BRE Termed an interval expression, this matches a range of
occurrences of the single character that immediately
precedes it. \{n\} matches exactly n occurrences, \{n,\ }
matches at least n occurrences, and \{n,m\} matches any
number of occurrences between n and m. n and m must be
between 0 and RE_DUP_MAX (minimum value: 255), inclusive.
\( \) BRE Save the pattern enclosed between \( and \) in a special
holding space. Up to nine subpatterns can be saved on a
single pattern. The text matched by the subpat- terns can be
reused later in the same pattern, by the escape sequences
\1 to \9. For example, \(ab\).*\1 matches two occurrences of
ab, with any number of characters in between.
\n BRE Replay the nth subpattern enclosed in \( and \) into the
pattern at this point. n is a number from 1 to 9, with 1
starting on the left.
{n,m} ERE Just like the BRE \{n,m\} earlier, but without the
backslashes in front of the braces.
+ ERE Match one or more instances of the preceding regular expression.
? ERE Match zero or one instances of the preceding regular expression.
| ERE Match the regular expression specified before or after.
() ERE Apply a match to the enclosed group of regular expressions.
-------------------------------------------
Simple regular expression matching examples
scully The six letters scully, anywhere on a line
^scully The six letters scully, at the beginning of a line
scully$ The six letters scully, at the end of a line
^scully$ A line containing exactly the six letters scully, and nothing else
[Sc]ully Either the six letters Scully, or the six letters scully, anywhere
on a line
scu.ly three letters scu, any character, and the two letters ly, anywhere
on a line
scu.*ly The three letters scu, any sequence of zero or more characters, and
the two letters ly, anywhere on a line (e.g., scuyly, scully,
scuWHOlly, and so on)
-----------------------
POSIX character classes
[:alnum:] Alphanumeric characters
[:alpha:] Alphabetic characters
[:blank:] Space and tab characters
[:cntrl:] Control characters
[:digit:] Numeric characters
[:graph:] Nonspace characters
[:lower:] Lowercase characters
[:print:] Printable characters
[:punct:] Punctuation characters
[:space:] Whitespace characters
[:upper:] Uppercase characters
[:xdigit:] Uppercase characters
------------------------
printf Format Specifiers
%b The corresponding argument is treated as a string containing escape
%c ASCII character. Print the first character of the corresponding argument.
%d, %i Decimal integer.
%e Floating-point format ([-]d.precisione[+-]dd).
%E Floating-point format ([-]d.precisionE[+-]dd).
%f Floating-point format ([-]ddd.precision).
%g %e or %f conversion, whichever is shorter, with trailing zeros removed.
%G %E or %f conversion, whichever is shorter, with trailing zeros removed.
%o Unsigned octal value.
%s String.
%u Unsigned decimal value.
%x Unsigned hexadecimal number. Use a–f for 10 to 15.
%X Unsigned hexadecimal number. Use A–F for 10 to 15.
%% Literal %.
--------------------
Meaning of precision
%d, %i, %o, %u, %x, %X
The minimum number of digits to print. When the value has fewer digits,
it is padded with leading zeros. The default precision is 1.
%e, %E The minimum number of digits to print. When the value has fewer digits,
it is padded with zeros after the decimal point. The default precision
is 6. A precision of 0 inhibits printing of the decimal point.
%f The number of digits to the right of the decimal point.
%g, %G The maximum number of significant digits.
%s The maximum number of characters to print.
----------------
Flags for printf
– Left-justify the formatted value within the field.
+ Always prefix numeric values with a sign, even if the value is positive.
# Use an alternate form: %o has a preceding 0; %x and %X are prefixed with
0x and 0X, respectively; %e, %E, and %f always have a decimal point in
the result; and %g and %G do not have trailing zeros removed.
0 Pad output with zeros, not spaces. This happens only when the field width
is wider than the converted result. In the C language, this flag applies
to all output formats, even nonnumeric ones. For the printf command, it
applies only to the numeric formats.
---------------
Basic wildcards
? Any single character
* Any string of characters
[set] Any character in set
[!set] Any character not in set
---------------------------------
Using the set construct wildcards
[abc] a, b, or c
[.,;] Period, comma, or semicolon
[-_] Dash or underscore
[a-c] a, b, or c
[a-z] Any lowercase letter
[!0-9] ny nondigit
[0-9!] ny digit, or an exclamation mark
[a-zA-Z] Any lower- or uppercase letter
[a-zA-Z0-9_-] Any letter, any digit, underscore, or dash
---------
expr operators
e1|e2 If e1 is nonzero or non-null, its value is used. Otherwise, if e2 is
nonzero or non-null, its value is used. Otherwise, the final value is
zero.
e1&e2 If e1 and e2 are non-zero or non-null, the return value is that of e1.
Otherwise, the final value is zero.
e1=e2 Equal.
e1!=e2 Not equal.
e1<e2 Less than.
e1<=e2 Less than or equal to.
e1>e2 Greater than.
e1>=e2 Greater than or equal to.
These operators cause expr to print 1 if the indicated comparison is
true, 0 otherwise. If both oper- ands are integers, the comparison is
numeric; otherwise, it’s a string comparison.
e1+e2 The sum of e1 and e2.
e1-e2 The difference of e1 and e2.
e1*e2 The product of e1 and e2.
e1/e2 The integer division of e1 by e2 (truncates).
e1%e2 The remainder of the integer division of e1 by e2 (truncates).
e1:e2 Match of e1 to BRE e2; see the expr(1) manpage for details.
( expression ) The value of expression; used for grouping, as in most
programming languages.
integer A number consisting only of digits, although an optional leading
minus sign is allowed. Sadly, unary plus is not supported.
string A string value that cannot be mistaken for a number or an operator.
--------------------
arithmetic operators
++ --
Left to Right Increment and decrement, prefix and postfix
+-!~
Right to Left Unary plus and minus; logical and bitwise negation
* /%
Left to Right Multiplication, division, and remainder
+-
Left to Right Addition and subtraction
<< >>
Left to Right Bit-shift left and right
< <= > >=
Left to Right Comparisons
= = !=
Left to Right Equal and not equal
&
Left to Right Bitwise AND
^
Left to Right Bitwise Exclusive OR
|
Left to Right Bitwise OR
&&
Left to Right Logical AND (short-circuit)
||
Left to Right Logical OR (short-circuit)
?:
Right to Left Conditional expression
= += -= *= /= %= &= ^= <<= >>= |=
Right to Left Assignment operators
--------------------
Sort key field types
b Ignore leading whitespace.
d Dictionary order.
f Fold letters implicitly to a common lettercase.
g Compare as general floating-point numbers. GNU version only.
i Ignore nonprintable characters.
n Compare as (integer) numbers.
r Reverse the sort order.
---------------------
Shell options for set
(be warned, set is the interface for misfit features in all shells)
-a allexport POSIX Export all subsequently defined variables.
-b notify POSIX Print job completion messages right away, instead of waiting for next prompt. Intended for interactive use.
-B braceexpand bash Enable brace expansion. On by default. See “Brace Expansion” [14.3.4] for more information.
-C noclobber POSIX Don’t allow > redirection to existing files. The >| oper- ator overrides the setting of this option. Intended for interactive use.
-e errexit POSIX Exit the shell when a command exits with nonzero status.
-f noglob POSIX Disable wildcard expansion.
-m monitor POSIX Enable job control (on by default). Intended for inter- active use.
-n noexec POSIX Read commands and check for syntax errors, but don’t execute them. Interactive shells are allowed to ignore this option.
-u nounset POSIX Treat undefined variables as errors, not as null.
-v verbose POSIX Print commands (verbatim) before running them.
-x xtrace POSIX Print commands (after expansions) before running them.
ignoreeof POSIX Disallow Ctrl-D to exit the shell.
nolog POSIX Disable command history for function definitions.
vi POSIX Use vi-style command-line editing. Intended for interactive use.
-A ksh88,ksh93 Array assignment. set +A does not clear the array. Refer to Korn shell “Indexed Arrays”for more information.
-H trackall (ksh) Same as POSIX -f noglob, Disable wildcard expansion.
-s ksh88, ksh93 Sort the positional parameters.
Common shell options which, for portability, should be avoided:
-h hashall (bash) POSIX Locate and remember the location of commands called from function bodies when the function is defined, instead of when the function is executed (XSI).
-k histexpand bash Enable !-style history expansion. On by default.
-P physical bash Use the physical directory structure for commands that change directory.
history bash Enable command history. On by default.
posix bash Enable full POSIX compliance.
markdirs ksh88, ksh93 Append a / to directories when doing wildcard expansion.
pipefail ksh93 Make pipeline exit status be that of the last command that fails, or zero if all OK. ksh93n or newer.
viraw ksh88, ksh93 Use vi-style command-line editing. Intended for interactive use. This mode can be slightly more CPU- intensive than set -o vi.
-p privileged bash, ksh88, ksh93 Attempt to function in a more secure mode. The details differ among the shells; see your shell’s docu- mentation.
-t bash, ksh88, ksh93 Read and execute one command and then exit. This is obsolete; it is for compatibility with the Bourne shell and should not be used.
bgnice ksh88, ksh93 Automatically lower the priority of all commands run in the background (with &).
emacs bash, ksh88, ksh93 Use emacs-style command-line editing. Intended for interactive use.
gmacs ksh88, ksh93 Use GNU emacs-style command-line editing. Intended for interactive use.
---------------------------------------------------------
POSIX shell built-in commands, a real-world portable list
(If utility exists, built-in will be run unless utility is explicitly called)
: (colon) Do nothing (just do expansions of arguments).
. (dot) Read file and execute it's contents in current shell,
(to 'source' a file in bash).
alias Set up shorthand for command or command line (interactive use).
bg Put a job in background (interactive use).
break Exit from surrounding for, while, or until loop.
cd Change working directory
command Locate built-in and external commands; find a built-in command instead of an identically named function.
continue Skip to next iteration of for, while, or until loop.
eval Process arguements as a command line.
exec Replace shell with given program or change I/O for shell.
exit Exit from shell.
export Create environment variables.
false Do nothing, unsuccessfully.
fc Work with command history (interactive use).
fg Put background job in foreground (interactive use).
getopts Process command-line options, (also less portable getopt).
jobs List
kill Send signals.
newgrp Start new shell with new group ID (obsolete).
pwd Print working directory.
read Read a line from standard input.
readonly Make variables read-only (unassignable).
return Return from surrounding function.
set Set options or positional parameters.
shift Shift command-line arguements.
times Print accumulated user and system CPU times for the shell and it's children.
trap Set up signal-catching routine.
true Do nothing, successfully.
umask Set/show file permission mask.
unalias Remove alias definitions (interactive use).
unset Remove definitions of variables or functions.
wait Wait for background job(s) to finish.