The GNU Awk User's Guide

This file documents awk, a program that you can use to select particular records in a file and perform operations upon them.

This is Edition 1.0.6 of Effective AWK Programming,
for the 3.0.6 version of the GNU implementation
of AWK.

Preface		What this Info file is about; brief history and acknowledgements.
1. Introduction		What is the awk language; using this Info file.
2. Getting Started with awk		A basic introduction to using awk. How to run an awk program. Command line syntax.
3. Useful One Line Programs		Short, sample awk programs.
4. Regular Expressions		All about matching things using regular expressions.
5. Reading Input Files		How to read files and manipulate fields.
6. Printing Output		How to print using awk. Describes the

                                print and printf statements.  
                                Also describes redirection of output.

7. Expressions		Expressions are the basic building blocks of statements.
8. Patterns and Actions		Overviews of patterns and actions.
9. Control Statements in Actions		The various control statements are described in detail.
10. Built-in Variables
11. Arrays in awk		The description and use of arrays. Also includes array-oriented control statements.
12. Built-in Functions		The built-in functions are summarized here.
13. User-defined Functions		User-defined functions are described in detail.
14. Running awk		How to run gawk.
15. A Library of awk Functions		A Library of awk Functions.
16. Practical awk Programs		Many awk programs with complete explanations.
17. The Evolution of the awk Language		The evolution of the awk language.
A. gawk Summary		gawk Options and Language Summary.
B. Installing gawk		Installing gawk under various operating systems.
C. Implementation Notes		Something about the implementation of

                                gawk.

D. Glossary		An explanation of some unfamiliar terms.
GNU GENERAL PUBLIC LICENSE		Your right to copy and distribute gawk.
Index		Concept and Variable Index.

History of awk and gawk		The history of gawk and awk.
The GNU Project and This Book		Brief history of the GNU project and this Info file.
Acknowledgements		Acknowledgements.
1.1 Using This Book		Using this Info file. Includes sample input files that you can use.
1.2 Typographical Conventions		Typographical Conventions.
1.3 Data Files for the Examples		Sample data files for use in the awk programs illustrated in this Info file.
2.1 A Rose By Any Other Name		What name to use to find awk.
2.2 How to Run awk Programs		How to run gawk programs; includes command line syntax.
2.2.1 One-shot Throw-away awk Programs		Running a short throw-away awk program.
2.2.2 Running awk without Input Files		Using no input files (input from terminal instead).
2.2.3 Running Long Programs		Putting permanent awk programs in files.
2.2.4 Executable awk Programs		Making self-contained awk programs.
2.2.5 Comments in awk Programs		Adding documentation to gawk programs.
2.3 A Very Simple Example		A very simple example.
2.4 An Example with Two Rules		A less simple one-line example with two rules.
2.5 A More Complex Example		A more complex example.
2.6 awk Statements Versus Lines		Subdividing or combining statements into lines.
2.7 Other Features of awk		Other Features of awk.
2.8 When to Use awk		When to use gawk and when to use other things.
4.1 How to Use Regular Expressions		How to Use Regular Expressions.
4.2 Escape Sequences		How to write non-printing characters.
4.3 Regular Expression Operators		Regular Expression Operators.
4.4 Additional Regexp Operators Only in gawk		Operators specific to GNU software.
4.5 Case-sensitivity in Matching		How to do case-insensitive matching.
4.6 How Much Text Matches?		How much text matches.
4.7 Using Dynamic Regexps		Using Dynamic Regexps.
5.1 How Input is Split into Records		Controlling how data is split into records.
5.2 Examining Fields		An introduction to fields.
5.3 Non-constant Field Numbers		Non-constant Field Numbers.
5.4 Changing the Contents of a Field		Changing the Contents of a Field.
5.5 Specifying How Fields are Separated		The field separator and how to change it.
5.5.1 The Basics of Field Separating		How fields are split with single characters or simple strings.
5.5.2 Using Regular Expressions to Separate Fields		Using regexps as the field separator.
5.5.3 Making Each Character a Separate Field		Making each character a separate field.
5.5.4 Setting FS from the Command Line		Setting FS from the command line.
5.5.5 Field Splitting Summary		Some final points and a summary table.
5.6 Reading Fixed-width Data		Reading constant width data.
5.7 Multiple-Line Records		Reading multi-line records.
5.8 Explicit Input with getline		Reading files under explicit program control using the getline function.
5.8.1 Introduction to getline		Introduction to the getline function.
5.8.2 Using getline with No Arguments		Using getline with no arguments.
5.8.3 Using getline Into a Variable		Using getline into a variable.
5.8.4 Using getline from a File		Using getline from a file.
5.8.5 Using getline Into a Variable from a File		Using getline into a variable from a file.
5.8.6 Using getline from a Pipe		Using getline from a pipe.
5.8.7 Using getline Into a Variable from a Pipe		Using getline into a variable from a pipe.
5.8.8 Summary of getline Variants		Summary Of getline Variants.
6.1 The print Statement		The print statement.
6.2 Examples of print Statements		Simple examples of print statements.
6.3 Output Separators		The output separators and how to change them.
6.4 Controlling Numeric Output with print		Controlling Numeric Output With print.
6.5 Using printf Statements for Fancier Printing		The printf statement.
6.5.1 Introduction to the printf Statement		Syntax of the printf statement.
6.5.2 Format-Control Letters		Format-control letters.
6.5.3 Modifiers for printf Formats		Format-specification modifiers.
6.5.4 Examples Using printf		Several examples.
6.6 Redirecting Output of print and printf		How to redirect output to multiple files and pipes.
6.7 Special File Names in gawk		File name interpretation in gawk.

                                gawk allows access to inherited file
                                descriptors.

6.8 Closing Input and Output Files and Pipes		Closing Input and Output Files and Pipes.
7.1 Constant Expressions		String, numeric, and regexp constants.
7.1.1 Numeric and String Constants		Numeric and string constants.
7.1.2 Regular Expression Constants		Regular Expression constants.
7.2 Using Regular Expression Constants		When and how to use a regexp constant.
7.3 Variables		Variables give names to values for later use.
7.3.1 Using Variables in a Program		Using variables in your programs.
7.3.2 Assigning Variables on the Command Line		Setting variables on the command line and a summary of command line syntax. This is an advanced method of input.
7.4 Conversion of Strings and Numbers		The conversion of strings to numbers and vice versa.
7.5 Arithmetic Operators		Arithmetic operations (`+', `-', etc.)
7.6 String Concatenation		Concatenating strings.
7.7 Assignment Expressions		Changing the value of a variable or a field.
7.8 Increment and Decrement Operators		Incrementing the numeric value of a variable.
7.9 True and False in awk		What is "true" and what is "false".
7.10 Variable Typing and Comparison Expressions		How variables acquire types, and how this affects comparison of numbers and strings with

                                `<', etc.

7.11 Boolean Expressions

Combining comparison expressions using boolean operators `||' ("or"), `&&'

                                ("and") and `!' ("not").

7.12 Conditional Expressions		Conditional expressions select between two subexpressions under control of a third subexpression.
7.13 Function Calls		A function call is an expression.
7.14 Operator Precedence (How Operators Nest)		How various operators nest.
8.1 Pattern Elements		What goes into a pattern.
8.1.1 Kinds of Patterns		A list of all kinds of patterns.
8.1.2 Regular Expressions as Patterns		Using regexps as patterns.
8.1.3 Expressions as Patterns		Any expression can be used as a pattern.
8.1.4 Specifying Record Ranges with Patterns		Pairs of patterns specify record ranges.
8.1.5 The BEGIN and END Special Patterns		Specifying initialization and cleanup rules.
8.1.5.1 Startup and Cleanup Actions		How and why to use BEGIN/END rules.
8.1.5.2 Input/Output from BEGIN and END Rules		I/O issues in BEGIN/END rules.
8.1.6 The Empty Pattern		The empty pattern, which matches every record.
8.2 Overview of Actions		What goes into an action.
9.1 The if-else Statement		Conditionally execute some awk statements.
9.2 The while Statement		Loop until some condition is satisfied.
9.3 The do-while Statement		Do specified action while looping until some condition is satisfied.
9.4 The for Statement		Another looping statement, that provides initialization and increment clauses.
9.5 The break Statement		Immediately exit the innermost enclosing loop.
9.6 The continue Statement		Skip to the end of the innermost enclosing loop.
9.7 The next Statement		Stop processing the current input record.
9.8 The nextfile Statement		Stop processing the current file.
9.9 The exit Statement		Stop execution of awk.
10.1 Built-in Variables that Control awk		Built-in variables that you change to control

                                awk.

10.2 Built-in Variables that Convey Information		Built-in variables where awk gives you information.
10.3 Using ARGC and ARGV		Ways to use ARGC and ARGV.
11.1 Introduction to Arrays
11.2 Referring to an Array Element		How to examine one element of an array.
11.3 Assigning Array Elements		How to change an element of an array.
11.4 Basic Array Example		Basic Example of an Array
11.5 Scanning All Elements of an Array		A variation of the for statement. It loops through the indices of an array's existing elements.
11.6 The delete Statement		The delete statement removes an element from an array.
11.7 Using Numbers to Subscript Arrays		How to use numbers as subscripts in

                                awk.

11.8 Using Uninitialized Variables as Subscripts		Using Uninitialized variables as subscripts.
11.9 Multi-dimensional Arrays		Emulating multi-dimensional arrays in

                                awk.

11.10 Scanning Multi-dimensional Arrays		Scanning multi-dimensional arrays.
12.1 Calling Built-in Functions		How to call built-in functions.
12.2 Numeric Built-in Functions		Functions that work with numbers, including

                                int, sin and rand.

12.3 Built-in Functions for String Manipulation

Functions for string manipulation, such as

                                split, match, and
                                sprintf.

12.4 Built-in Functions for Input/Output		Functions for files and shell commands.
12.5 Functions for Dealing with Time Stamps		Functions for dealing with time stamps.
13.1 Function Definition Syntax		How to write definitions and what they mean.
13.2 Function Definition Examples		An example function definition and what it does.
13.3 Calling User-defined Functions		Things to watch out for.
13.4 The return Statement		Specifying the value a function returns.
14.1 Command Line Options		Command line options and their meanings.
14.2 Other Command Line Arguments		Input file names and variable assignments.
14.3 The AWKPATH Environment Variable		Searching directories for awk programs.
14.4 Obsolete Options and/or Features		Obsolete Options and/or features.
14.5 Undocumented Options and Features		Undocumented Options and Features.
14.6 Known Bugs in gawk		Known Bugs in gawk.
15.1 Simulating gawk-specific Features		What to do if you don't have gawk.
15.2 Implementing nextfile as a Function		Two implementations of a nextfile function.
15.3 Assertions		A function for assertions in awk programs.
15.4 Rounding Numbers		A function for rounding if sprintf does not do it correctly.
15.5 Translating Between Characters and Numbers		Functions for using characters as numbers and vice versa.
15.6 Merging an Array Into a String		A function to join an array into a string.
15.7 Turning Dates Into Timestamps		A function to turn a date into a timestamp.
15.8 Managing the Time of Day		A function to get formatted times.
15.9 Noting Data File Boundaries		A function for handling data file transitions.
15.10 Processing Command Line Options		A function for processing command line arguments.
15.11 Reading the User Database		Functions for getting user information.
15.12 Reading the Group Database		Functions for getting group information.
15.13 Naming Library Function Global Variables		How to best name private global variables in library functions.
16.1 Re-inventing Wheels for Fun and Profit		Clones of common utilities.
16.1.1 Cutting Out Fields and Columns		The cut utility.
16.1.2 Searching for Regular Expressions in Files		The egrep utility.
16.1.3 Printing Out User Information		The id utility.
16.1.4 Splitting a Large File Into Pieces		The split utility.
16.1.5 Duplicating Output Into Multiple Files		The tee utility.
16.1.6 Printing Non-duplicated Lines of Text		The uniq utility.
16.1.7 Counting Things		The wc utility.
16.2 A Grab Bag of awk Programs		Some interesting awk programs.
16.2.1 Finding Duplicated Words in a Document		Finding duplicated words in a document.
16.2.2 An Alarm Clock Program		An alarm clock.
16.2.3 Transliterating Characters		A program similar to the tr utility.
16.2.4 Printing Mailing Labels		Printing mailing labels.
16.2.5 Generating Word Usage Counts		A program to produce a word usage count.
16.2.6 Removing Duplicates from Unsorted Text		Eliminating duplicate entries from a history file.
16.2.7 Extracting Programs from Texinfo Source Files		Pulling out programs from Texinfo source files.
16.2.8 A Simple Stream Editor		A Simple Stream Editor.
16.2.9 An Easy Way to Use Library Functions		A wrapper for awk that includes files.
17.1 Major Changes between V7 and SVR3.1		The major changes between V7 and System V Release 3.1.
17.2 Changes between SVR3.1 and SVR4		Minor changes between System V Releases 3.1 and 4.
17.3 Changes between SVR4 and POSIX awk		New features from the POSIX standard.
17.4 Extensions in the Bell Laboratories awk		New features from the Bell Laboratories version of awk.
17.5 Extensions in gawk Not in POSIX awk		The extensions in gawk not in POSIX

                                awk.

A.1 Command Line Options Summary		Recapitulation of the command line.
A.2 Language Summary		A terse review of the language.
A.3 Variables and Fields		Variables, fields, and arrays.
A.3.1 Fields		Input field splitting.
A.3.2 Built-in Variables		awk's built-in variables.
A.3.3 Arrays		Using arrays.
A.3.4 Data Types		Values in awk are numbers or strings.
A.4 Patterns		Patterns and Actions, and their component parts.
A.4.1 Pattern Summary		Quick overview of patterns.
A.4.2 Regular Expressions		Quick overview of regular expressions.
A.5 Actions		Quick overview of actions.
A.5.1 Operators		awk operators.
A.5.2 Control Statements		The control statements.
A.5.3 I/O Statements		The I/O statements.
A.5.4 printf Summary		A summary of printf.
A.5.5 Special File Names		Special file names interpreted internally.
A.5.6 Built-in Functions		Built-in numeric and string functions.
A.5.7 Time Functions		Built-in time functions.
A.5.8 String Constants		Escape sequences in strings.
A.6 User-defined Functions		Defining and calling functions.
A.7 Historical Features		Some undocumented but supported "features".
B.1 The gawk Distribution		What is in the gawk distribution.
B.1.1 Getting the gawk Distribution		How to get the distribution.
B.1.2 Extracting the Distribution		How to extract the distribution.
B.1.3 Contents of the gawk Distribution		What is in the distribution.
B.2 Compiling and Installing gawk on Unix		Installing gawk under various versions of Unix.
B.2.1 Compiling gawk for Unix		Compiling gawk under Unix.
B.2.2 The Configuration Process		How it's all supposed to work.
B.3 How to Compile and Install gawk on VMS		Installing gawk on VMS.
B.3.1 Compiling gawk on VMS		How to compile gawk under VMS.
B.3.2 Installing gawk on VMS		How to install gawk under VMS.
B.3.3 Running gawk on VMS		How to run gawk under VMS.
B.3.4 Building and Using gawk on VMS POSIX		Alternate instructions for VMS POSIX.
B.4 MS-DOS and OS/2 Installation and Compilation		Installing and Compiling gawk on MS-DOS and OS/2
B.5 Installing gawk on the Atari ST		Installing gawk on the Atari ST.
B.5.1 Compiling gawk on the Atari ST		Compiling gawk on Atari
B.5.2 Running gawk on the Atari ST		Running gawk on Atari
B.6 Installing gawk on an Amiga		Installing gawk on an Amiga.
B.7 Reporting Problems and Bugs		Reporting Problems and Bugs.
B.8 Other Freely Available awk Implementations		Other freely available awk implementations.
C.1 Downward Compatibility and Debugging		How to disable certain gawk extensions.
C.2 Making Additions to gawk		Making Additions To gawk.
C.2.1 Adding New Features		Adding code to the main body of gawk.
C.2.2 Porting gawk to a New Operating System		Porting gawk to a new operating system.
C.3 Probable Future Extensions		New features that may be implemented one day.
C.4 Suggestions for Improvements		Suggestions for improvements by volunteers.

Preface

This Info file teaches you about the awk language and how you can use it effectively. You should already be familiar with basic system commands, such as cat and ls,(1) and basic shell facilities, such as Input/Output (I/O) redirection and pipes.

Implementations of the awk language are available for many different computing environments. This Info file, while describing the awk language in general, also describes a particular implementation of awk called gawk (which stands for "GNU Awk"). gawk runs on a broad range of Unix systems, ranging from 80386 PC-based computers, up through large scale systems, such as Crays. gawk has also been ported to MS-DOS and OS/2 PC's, Atari and Amiga micro-computers, and VMS.

History of awk and gawk		The history of gawk and awk.
The GNU Project and This Book		Brief history of the GNU project and this Info file.
Acknowledgements		Acknowledgements.

History of awk and gawk

The name awk comes from the initials of its designers: Alfred V. Aho, Peter J. Weinberger, and Brian W. Kernighan. The original version of awk was written in 1977 at AT&T Bell Laboratories. In 1985 a new version made the programming language more powerful, introducing user-defined functions, multiple input streams, and computed regular expressions. This new version became generally available with Unix System V Release 3.1. The version in System V Release 4 added some new features and also cleaned up the behavior in some of the "dark corners" of the language. The specification for awk in the POSIX Command Language and Utilities standard further clarified the language based on feedback from both the gawk designers, and the original Bell Labs awk designers.

The GNU implementation, gawk, was written in 1986 by Paul Rubin and Jay Fenlason, with advice from Richard Stallman. John Woods contributed parts of the code as well. In 1988 and 1989, David Trueman, with help from Arnold Robbins, thoroughly reworked gawk for compatibility with the newer awk. Current development focuses on bug fixes, performance improvements, standards compliance, and occasionally, new features.

The GNU Project and This Book

The Free Software Foundation (FSF) is a non-profit organization dedicated to the production and distribution of freely distributable software. It was founded by Richard M. Stallman, the author of the original Emacs editor. GNU Emacs is the most widely used version of Emacs today.

The GNU project is an on-going effort on the part of the Free Software Foundation to create a complete, freely distributable, POSIX compliant computing environment. (GNU stands for "GNU's not Unix".) The FSF uses the "GNU General Public License" (or GPL) to ensure that source code for their software is always available to the end user. A copy of the GPL is included for your reference (see section GNU GENERAL PUBLIC LICENSE). The GPL applies to the C language source code for gawk.

A shell, an editor (Emacs), highly portable optimizing C, C++, and Objective-C compilers, a symbolic debugger, and dozens of large and small utilities (such as gawk), have all been completed and are freely available. As of this writing (early 1997), the GNU operating system kernel (the HURD), has been released, but is still in an early stage of development.

Until the GNU operating system is more fully developed, you should consider using Linux, a freely distributable, Unix-like operating system for 80386, DEC Alpha, Sun SPARC and other systems. There are many books on Linux. One freely available one is Linux Installation and Getting Started, by Matt Welsh. Many Linux distributions are available, often in computer stores or bundled on CD-ROM with books about Linux. (There are three other freely available, Unix-like operating systems for 80386 and other systems, NetBSD, FreeBSD,and OpenBSD. All are based on the 4.4-Lite Berkeley Software Distribution, and they use recent versions of gawk for their versions of awk.)

This Info file itself has gone through several previous, preliminary editions. I started working on a preliminary draft of The GAWK Manual, by Diane Close, Paul Rubin, and Richard Stallman in the fall of 1988. It was around 90 pages long, and barely described the original, "old" version of awk. After substantial revision, the first version of the The GAWK Manual to be released was Edition 0.11 Beta in October of 1989. The manual then underwent more substantial revision for Edition 0.13 of December 1991. David Trueman, Pat Rankin, and Michal Jaegermann contributed sections of the manual for Edition 0.13. That edition was published by the FSF as a bound book early in 1992. Since then there have been several minor revisions, notably Edition 0.14 of November 1992 that was published by the FSF in January of 1993, and Edition 0.16 of August 1993.

Edition 1.0 of Effective AWK Programming represents a significant re-working of The GAWK Manual, with much additional material. The FSF and I agree that I am now the primary author. I also felt that it needed a more descriptive title.

Effective AWK Programming will undoubtedly continue to evolve. An electronic version comes with the gawk distribution from the FSF. If you find an error in this Info file, please report it! See section Reporting Problems and Bugs, for information on submitting problem reports electronically, or write to me in care of the FSF.

Acknowledgements

I would like to acknowledge Richard M. Stallman, for his vision of a better world, and for his courage in founding the FSF and starting the GNU project.

The initial draft of The GAWK Manual had the following acknowledgements:

Many people need to be thanked for their assistance in producing this manual. Jay Fenlason contributed many ideas and sample programs. Richard Mlynarik and Robert Chassell gave helpful comments on drafts of this manual. The paper A Supplemental Document for awk by John W. Pierce of the Chemistry Department at UC San Diego, pinpointed several issues relevant both to awk implementation and to this manual, that would otherwise have escaped us.

The following people provided many helpful comments on Edition 0.13 of The GAWK Manual: Rick Adams, Michael Brennan, Rich Burridge, Diane Close, Christopher ("Topher") Eliot, Michael Lijewski, Pat Rankin, Miriam Robbins, and Michal Jaegermann.

The following people provided many helpful comments for Edition 1.0 of Effective AWK Programming: Karl Berry, Michael Brennan, Darrel Hankerson, Michal Jaegermann, Michael Lijewski, and Miriam Robbins. Pat Rankin, Michal Jaegermann, Darrel Hankerson and Scott Deifik updated their respective sections for Edition 1.0.

Robert J. Chassell provided much valuable advice on the use of Texinfo. He also deserves special thanks for convincing me not to title this Info file How To Gawk Politely. Karl Berry helped significantly with the TeX part of Texinfo.

David Trueman deserves special credit; he has done a yeoman job of evolving gawk so that it performs well, and without bugs. Although he is no longer involved with gawk, working with him on this project was a significant pleasure.

Scott Deifik, Darrel Hankerson, Kai Uwe Rommel, Pat Rankin, and Michal Jaegermann (in no particular order) are long time members of the gawk "crack portability team." Without their hard work and help, gawk would not be nearly the fine program it is today. It has been and continues to be a pleasure working with this team of fine people.

Jeffrey Friedl provided invaluable help in tracking down a number of last minute problems with regular expressions in gawk 3.0.

David and I would like to thank Brian Kernighan of Bell Labs for invaluable assistance during the testing and debugging of gawk, and for help in clarifying numerous points about the language. We could not have done nearly as good a job on either gawk or its documentation without his help.

I would like to thank Marshall and Elaine Hartholz of Seattle, and Dr. Bert and Rita Schreiber of Detroit for large amounts of quiet vacation time in their homes, which allowed me to make significant progress on this Info file and on gawk itself. Phil Hughes of SSC contributed in a very important way by loaning me his laptop Linux system, not once, but twice, allowing me to do a lot of work while away from home.

Finally, I must thank my wonderful wife, Miriam, for her patience through the many versions of this project, for her proof-reading, and for sharing me with the computer. I would like to thank my parents for their love, and for the grace with which they raised and educated me. I also must acknowledge my gratitude to G-d, for the many opportunities He has sent my way, as well as for the gifts He has given me with which to take advantage of those opportunities. Arnold Robbins
Atlanta, Georgia
February, 1997

1. Introduction

If you are like many computer users, you would frequently like to make changes in various text files wherever certain patterns appear, or extract data from parts of certain lines while discarding the rest. To write a program to do this in a language such as C or Pascal is a time-consuming inconvenience that may take many lines of code. The job may be easier with awk.

The awk utility interprets a special-purpose programming language that makes it possible to handle simple data-reformatting jobs with just a few lines of code.

The GNU implementation of awk is called gawk; it is fully upward compatible with the System V Release 4 version of awk. gawk is also upward compatible with the POSIX specification of the awk language. This means that all properly written awk programs should work with gawk. Thus, we usually don't distinguish between gawk and other awk implementations.

Using awk you can:

• manage small, personal databases

• generate reports

• validate data

• produce indexes, and perform other document preparation tasks

• even experiment with algorithms that can be adapted later to other computer languages

1.1 Using This Book		Using this Info file. Includes sample input files that you can use.
1.2 Typographical Conventions		Typographical Conventions.
1.3 Data Files for the Examples		Sample data files for use in the awk programs illustrated in this Info file.

1.1 Using This Book

The term awk refers to a particular program, and to the language you use to tell this program what to do. When we need to be careful, we call the program "the awk utility" and the language "the awk language." The term gawk refers to a version of awk developed as part the GNU project. The purpose of this Info file is to explain both the awk language and how to run the awk utility.

The main purpose of the Info file is to explain the features of awk, as defined in the POSIX standard. It does so in the context of one particular implementation, gawk. While doing so, it will also attempt to describe important differences between gawk and other awk implementations. Finally, any gawk features that are not in the POSIX standard for awk will be noted.

The term awk program refers to a program written by you in the awk programming language.

See section Getting Started with awk, for the bare essentials you need to know to start using awk.

Some useful "one-liners" are included to give you a feel for the awk language (see section Useful One Line Programs).

Many sample awk programs have been provided for you (see section A Library of awk Functions; also see section Practical awk Programs).

The entire awk language is summarized for quick reference in gawk Summary. Look there if you just need to refresh your memory about a particular feature.

If you find terms that you aren't familiar with, try looking them up in the glossary (see section D. Glossary).

Most of the time complete awk programs are used as examples, but in some of the more advanced sections, only the part of the awk program that illustrates the concept being described is shown.

While this Info file is aimed principally at people who have not been exposed to awk, there is a lot of information here that even the awk expert should find useful. In particular, the description of POSIX awk, and the example programs in A Library of awk Functions, and Practical awk Programs, should be of interest.

Dark Corners

Who opened that window shade?!?
Count Dracula

Until the POSIX standard (and The Gawk Manual), many features of awk were either poorly documented, or not documented at all. Descriptions of such features (often called "dark corners") are noted in this Info file with "(d.c.)". They also appear in the index under the heading "dark corner."

1.2 Typographical Conventions

This Info file is written using Texinfo, the GNU documentation formatting language. A single Texinfo source file is used to produce both the printed and on-line versions of the documentation. This section briefly documents the typographical conventions used in Texinfo.

Examples you would type at the command line are preceded by the common shell primary and secondary prompts, `$' and `>'. Output from the command is preceded by the glyph "-|". This typically represents the command's standard output. Error messages, and other output on the command's standard error, are preceded by the glyph "error-->". For example:

$ echo hi on stdout -| hi on stdout $ echo hello on stderr 1>&2 error--> hello on stderr

Characters that you type at the keyboard look like this. In particular, there are special characters called "control characters." These are characters that you type by holding down both the CONTROL key and another key, at the same time. For example, a Control-d is typed by first pressing and holding the CONTROL key, next pressing the d key, and finally releasing both keys.

1.3 Data Files for the Examples

Many of the examples in this Info file take their input from two sample data files. The first, called `BBS-list', represents a list of computer bulletin board systems together with information about those systems. The second data file, called `inventory-shipped', contains information about shipments on a monthly basis. In both files, each line is considered to be one record.

In the file `BBS-list', each record contains the name of a computer bulletin board, its phone number, the board's baud rate(s), and a code for the number of hours it is operational. An `A' in the last column means the board operates 24 hours a day. A `B' in the last column means the board operates evening and weekend hours, only. A `C' means the board operates only on weekends.

aardvark 555-5553 1200/300 B alpo-net 555-3412 2400/1200/300 A barfly 555-7685 1200/300 A bites 555-1675 2400/1200/300 A camelot 555-0542 300 C core 555-2912 1200/300 C fooey 555-1234 2400/1200/300 B foot 555-6699 1200/300 B macfoo 555-6480 1200/300 A sdace 555-3430 2400/1200/300 A sabafoo 555-2127 1200/300 C

The second data file, called `inventory-shipped', represents information about shipments during the year. Each record contains the month of the year, the number of green crates shipped, the number of red boxes shipped, the number of orange bags shipped, and the number of blue packages shipped, respectively. There are 16 entries, covering the 12 months of one year and four months of the next year.

Jan 13 25 15 115 Feb 15 32 24 226 Mar 15 24 34 228 Apr 31 52 63 420 May 16 34 29 208 Jun 31 42 75 492 Jul 24 34 67 436 Aug 15 34 47 316 Sep 13 55 37 277 Oct 29 54 68 525 Nov 20 87 82 577 Dec 17 35 61 401 Jan 21 36 64 620 Feb 26 58 80 652 Mar 24 75 70 495 Apr 21 70 74 514

If you are reading this in GNU Emacs using Info, you can copy the regions of text showing these sample files into your own test files. This way you can try out the examples shown in the remainder of this document. You do this by using the command M-x write-region to copy text from the Info file into a file for use with awk (See section `Miscellaneous File Operations' in GNU Emacs Manual, for more information). Using this information, create your own `BBS-list' and `inventory-shipped' files, and practice what you learn in this Info file.

If you are using the stand-alone version of Info, see Extracting Programs from Texinfo Source Files, for an awk program that will extract these data files from `gawk.texi', the Texinfo source file for this Info file.

2. Getting Started with awk

The basic function of awk is to search files for lines (or other units of text) that contain certain patterns. When a line matches one of the patterns, awk performs specified actions on that line. awk keeps processing input lines in this way until the end of the input files are reached.

Programs in awk are different from programs in most other languages, because awk programs are data-driven; that is, you describe the data you wish to work with, and then what to do when you find it. Most other languages are procedural; you have to describe, in great detail, every step the program is to take. When working with procedural languages, it is usually much harder to clearly describe the data your program will process. For this reason, awk programs are often refreshingly easy to both write and read.

When you run awk, you specify an awk program that tells awk what to do. The program consists of a series of rules. (It may also contain function definitions, an advanced feature which we will ignore for now. See section User-defined Functions.) Each rule specifies one pattern to search for, and one action to perform when that pattern is found.

Syntactically, a rule consists of a pattern followed by an action. The action is enclosed in curly braces to separate it from the pattern. Rules are usually separated by newlines. Therefore, an awk program looks like this:

pattern { action } pattern { action } ...

2.1 A Rose By Any Other Name		What name to use to find awk.
2.2 How to Run awk Programs		How to run gawk programs; includes command line syntax.
2.3 A Very Simple Example		A very simple example.
2.4 An Example with Two Rules		A less simple one-line example with two rules.
2.5 A More Complex Example		A more complex example.
2.6 awk Statements Versus Lines		Subdividing or combining statements into lines.
2.7 Other Features of awk		Other Features of awk.
2.8 When to Use awk		When to use gawk and when to use other things.

2.1 A Rose By Any Other Name

The awk language has evolved over the years. Full details are provided in The Evolution of the awk Language. The language described in this Info file is often referred to as "new awk."

Because of this, many systems have multiple versions of awk. Some systems have an awk utility that implements the original version of the awk language, and a nawk utility for the new version. Others have an oawk for the "old awk" language, and plain awk for the new one. Still others only have one version, usually the new one.(2)

All in all, this makes it difficult for you to know which version of awk you should run when writing your programs. The best advice we can give here is to check your local documentation. Look for awk, oawk, and nawk, as well as for gawk. Chances are, you will have some version of new awk on your system, and that is what you should use when running your programs. (Of course, if you're reading this Info file, chances are good that you have gawk!)

Throughout this Info file, whenever we refer to a language feature that should be available in any complete implementation of POSIX awk, we simply use the term awk. When referring to a feature that is specific to the GNU implementation, we use the term gawk.

2.2 How to Run awk Programs

There are several ways to run an awk program. If the program is short, it is easiest to include it in the command that runs awk, like this:

awk 'program' input-file1 input-file2 ...

where program consists of a series of patterns and actions, as described earlier. (The reason for the single quotes is described below, in One-shot Throw-away awk Programs.)

When the program is long, it is usually more convenient to put it in a file and run it with a command like this:

awk -f program-file input-file1 input-file2 ...

2.2.1 One-shot Throw-away awk Programs		Running a short throw-away awk program.
2.2.2 Running awk without Input Files		Using no input files (input from terminal instead).
2.2.3 Running Long Programs		Putting permanent awk programs in files.
2.2.4 Executable awk Programs		Making self-contained awk programs.
2.2.5 Comments in awk Programs		Adding documentation to gawk programs.

2.2.1 One-shot Throw-away awk Programs

Once you are familiar with awk, you will often type in simple programs the moment you want to use them. Then you can write the program as the first argument of the awk command, like this:

awk 'program' input-file1 input-file2 ...

where program consists of a series of patterns and actions, as described earlier.

This command format instructs the shell, or command interpreter, to start awk and use the program to process records in the input file(s). There are single quotes around program so that the shell doesn't interpret any awk characters as special shell characters. They also cause the shell to treat all of program as a single argument for awk and allow program to be more than one line long.

This format is also useful for running short or medium-sized awk programs from shell scripts, because it avoids the need for a separate file for the awk program. A self-contained shell script is more reliable since there are no other files to misplace.

Useful One Line Programs, presents several short, self-contained programs.

As an interesting side point, the command

awk '/foo/' files ...

is essentially the same as

egrep foo files ...

2.2.2 Running awk without Input Files

You can also run awk without any input files. If you type the command line:

awk 'program'

then awk applies the program to the standard input, which usually means whatever you type on the terminal. This continues until you indicate end-of-file by typing Control-d. (On other operating systems, the end-of-file character may be different. For example, on OS/2 and MS-DOS, it is Control-z.)

For example, the following program prints a friendly piece of advice (from Douglas Adams' The Hitchhiker's Guide to the Galaxy), to keep you from worrying about the complexities of computer programming (`BEGIN' is a feature we haven't discussed yet).

$ awk "BEGIN { print \"Don't Panic!\" }" -| Don't Panic!

This program does not read any input. The `\' before each of the inner double quotes is necessary because of the shell's quoting rules, in particular because it mixes both single quotes and double quotes.

This next simple awk program emulates the cat utility; it copies whatever you type at the keyboard to its standard output. (Why this works is explained shortly.)

$ awk '{ print }' Now is the time for all good men -| Now is the time for all good men to come to the aid of their country. -| to come to the aid of their country. Four score and seven years ago, ... -| Four score and seven years ago, ... What, me worry? -| What, me worry? Control-d

2.2.3 Running Long Programs

Sometimes your awk programs can be very long. In this case it is more convenient to put the program into a separate file. To tell awk to use that file for its program, you type:

awk -f source-file input-file1 input-file2 ...

The `-f' instructs the awk utility to get the awk program from the file source-file. Any file name can be used for source-file. For example, you could put the program:

BEGIN { print "Don't Panic!" }

into the file `advice'. Then this command:

awk -f advice

does the same thing as this one:

awk "BEGIN { print \"Don't Panic!\" }"

which was explained earlier (see section Running awk without Input Files). Note that you don't usually need single quotes around the file name that you specify with `-f', because most file names don't contain any of the shell's special characters. Notice that in `advice', the awk program did not have single quotes around it. The quotes are only needed for programs that are provided on the awk command line.

If you want to identify your awk program files clearly as such, you can add the extension `.awk' to the file name. This doesn't affect the execution of the awk program, but it does make "housekeeping" easier.

2.2.4 Executable awk Programs

Once you have learned awk, you may want to write self-contained awk scripts, using the `#!' script mechanism. You can do this on many Unix systems(3) (and someday on the GNU system).

For example, you could update the file `advice' to look like this:

#! /bin/awk -f BEGIN { print "Don't Panic!" }

After making this file executable (with the chmod utility), you can simply type `advice' at the shell, and the system will arrange to run awk(4) as if you had typed `awk -f advice'.

$ advice -| Don't Panic!

Self-contained awk scripts are useful when you want to write a program which users can invoke without their having to know that the program is written in awk.

Caution: You should not put more than one argument on the `#!' line after the path to awk. This will not work. The operating system treats the rest of the line as a single agument, and passes it to awk. Doing this will lead to confusing behavior: most likely a usage diagnostic of some sort from awk.

Some older systems do not support the `#!' mechanism. You can get a similar effect using a regular shell script. It would look something like this:

: The colon ensures execution by the standard shell. awk 'program' "$@"

Using this technique, it is vital to enclose the program in single quotes to protect it from interpretation by the shell. If you omit the quotes, only a shell wizard can predict the results.

The "$@" causes the shell to forward all the command line arguments to the awk program, without interpretation. The first line, which starts with a colon, is used so that this shell script will work even if invoked by a user who uses the C shell. (Not all older systems obey this convention, but many do.)

2.2.5 Comments in awk Programs

A comment is some text that is included in a program for the sake of human readers; it is not really part of the program. Comments can explain what the program does, and how it works. Nearly all programming languages have provisions for comments, because programs are typically hard to understand without their extra help.

In the awk language, a comment starts with the sharp sign character, `#', and continues to the end of the line. The `#' does not have to be the first character on the line. The awk language ignores the rest of a line following a sharp sign. For example, we could have put the following into `advice':

# This program prints a nice friendly message. It helps # keep novice users from being afraid of the computer. BEGIN { print "Don't Panic!" }

You can put comment lines into keyboard-composed throw-away awk programs also, but this usually isn't very useful; the purpose of a comment is to help you or another person understand the program at a later time.

Caution: As mentioned in One-shot Throw-away awk Programs, you can enclose small to medium programs in single quotes, in order to keep your shell scripts self-contained. When doing so, don't put an apostrophe (i.e., a single quote) into a comment (or anywhere else in your program). The shell will interpret the quote as the closing quote for the entire program. As a result, usually the shell will print a message about mismatched quotes, and if awk actually runs, it will probably print strange messages about syntax errors. For example:

awk 'BEGIN { print "hello" } # let's be cute'

2.3 A Very Simple Example

The following command runs a simple awk program that searches the input file `BBS-list' for the string of characters: `foo'. (A string of characters is usually called a string. The term string is perhaps based on similar usage in English, such as "a string of pearls," or, "a string of cars in a train.")

awk '/foo/ { print $0 }' BBS-list

When lines containing `foo' are found, they are printed, because `print $0' means print the current line. (Just `print' by itself means the same thing, so we could have written that instead.)

You will notice that slashes, `/', surround the string `foo' in the awk program. The slashes indicate that `foo' is a pattern to search for. This type of pattern is called a regular expression, and is covered in more detail later (see section Regular Expressions). The pattern is allowed to match parts of words. There are single-quotes around the awk program so that the shell won't interpret any of it as special shell characters.

Here is what this program prints:

$ awk '/foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C

In an awk rule, either the pattern or the action can be omitted, but not both. If the pattern is omitted, then the action is performed for every input line. If the action is omitted, the default action is to print all lines that match the pattern.

Thus, we could leave out the action (the print statement and the curly braces) in the above example, and the result would be the same: all lines matching the pattern `foo' would be printed. By comparison, omitting the print statement but retaining the curly braces makes an empty action that does nothing; then no lines would be printed.

2.4 An Example with Two Rules

The awk utility reads the input files one line at a time. For each line, awk tries the patterns of each of the rules. If several patterns match then several actions are run, in the order in which they appear in the awk program. If no patterns match, then no actions are run.

After processing all the rules (perhaps none) that match the line, awk reads the next line (however, see section The next Statement, and also see section The nextfile Statement). This continues until the end of the file is reached.

For example, the awk program:

/12/ { print $0 } /21/ { print $0 }

contains two rules. The first rule has the string `12' as the pattern and `print $0' as the action. The second rule has the string `21' as the pattern and also has `print $0' as the action. Each rule's action is enclosed in its own pair of braces.

This awk program prints every line that contains the string `12' or the string `21'. If a line contains both strings, it is printed twice, once by each rule.

This is what happens if we run this program on our two sample data files, `BBS-list' and `inventory-shipped', as shown here:

$ awk '/12/ { print $0 } > /21/ { print $0 }' BBS-list inventory-shipped -| aardvark 555-5553 1200/300 B -| alpo-net 555-3412 2400/1200/300 A -| barfly 555-7685 1200/300 A -| bites 555-1675 2400/1200/300 A -| core 555-2912 1200/300 C -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sdace 555-3430 2400/1200/300 A -| sabafoo 555-2127 1200/300 C -| sabafoo 555-2127 1200/300 C -| Jan 21 36 64 620 -| Apr 21 70 74 514

Note how the line in `BBS-list' beginning with `sabafoo' was printed twice, once for each rule.

2.5 A More Complex Example

Here is an example to give you an idea of what typical awk programs do. This example shows how awk can be used to summarize, select, and rearrange the output of another utility. It uses features that haven't been covered yet, so don't worry if you don't understand all the details.

ls -lg | awk '$6 == "Nov" { sum += $5 } END { print sum }'

This command prints the total number of bytes in all the files in the current directory that were last modified in November (of any year). (In the C shell you would need to type a semicolon and then a backslash at the end of the first line; in a POSIX-compliant shell, such as the Bourne shell or Bash, the GNU Bourne-Again shell, you can type the example as shown.)

The `ls -lg' part of this example is a system command that gives you a listing of the files in a directory, including file size and the date the file was last modified. Its output looks like this:

-rw-r--r-- 1 arnold user 1933 Nov 7 13:05 Makefile -rw-r--r-- 1 arnold user 10809 Nov 7 13:03 gawk.h -rw-r--r-- 1 arnold user 983 Apr 13 12:14 gawk.tab.h -rw-r--r-- 1 arnold user 31869 Jun 15 12:20 gawk.y -rw-r--r-- 1 arnold user 22414 Nov 7 13:03 gawk1.c -rw-r--r-- 1 arnold user 37455 Nov 7 13:03 gawk2.c -rw-r--r-- 1 arnold user 27511 Dec 9 13:07 gawk3.c -rw-r--r-- 1 arnold user 7989 Nov 7 13:03 gawk4.c

The first field contains read-write permissions, the second field contains the number of links to the file, and the third field identifies the owner of the file. The fourth field identifies the group of the file. The fifth field contains the size of the file in bytes. The sixth, seventh and eighth fields contain the month, day, and time, respectively, that the file was last modified. Finally, the ninth field contains the name of the file.

The `$6 == "Nov"' in our awk program is an expression that tests whether the sixth field of the output from `ls -lg' matches the string `Nov'. Each time a line has the string `Nov' for its sixth field, the action `sum += $5' is performed. This adds the fifth field (the file size) to the variable sum. As a result, when awk has finished reading all the input lines, sum is the sum of the sizes of files whose lines matched the pattern. (This works because awk variables are automatically initialized to zero.)

After the last line of output from ls has been processed, the END rule is executed, and the value of sum is printed. In this example, the value of sum would be 80600.

These more advanced awk techniques are covered in later sections (see section Overview of Actions). Before you can move on to more advanced awk programming, you have to know how awk interprets your input and displays your output. By manipulating fields and using print statements, you can produce some very useful and impressive looking reports.

2.6 awk Statements Versus Lines

Most often, each line in an awk program is a separate statement or separate rule, like this:

awk '/12/ { print $0 } /21/ { print $0 }' BBS-list inventory-shipped

However, gawk will ignore newlines after any of the following:

, { ? : || && do else

A newline at any other point is considered the end of the statement. (Splitting lines after `?' and `:' is a minor gawk extension. The `?' and `:' referred to here is the three operand conditional expression described in Conditional Expressions.)

If you would like to split a single statement into two lines at a point where a newline would terminate it, you can continue it by ending the first line with a backslash character, `\'. The backslash must be the final character on the line to be recognized as a continuation character. This is allowed absolutely anywhere in the statement, even in the middle of a string or regular expression. For example:

awk '/This regular expression is too long, so continue it\ on the next line/ { print $1 }'

We have generally not used backslash continuation in the sample programs in this Info file. Since in gawk there is no limit on the length of a line, it is never strictly necessary; it just makes programs more readable. For this same reason, as well as for clarity, we have kept most statements short in the sample programs presented throughout the Info file. Backslash continuation is most useful when your awk program is in a separate source file, instead of typed in on the command line. You should also note that many awk implementations are more particular about where you may use backslash continuation. For example, they may not allow you to split a string constant using backslash continuation. Thus, for maximal portability of your awk programs, it is best not to split your lines in the middle of a regular expression or a string.

Caution: backslash continuation does not work as described above with the C shell. Continuation with backslash works for awk programs in files, and also for one-shot programs provided you are using a POSIX-compliant shell, such as the Bourne shell or Bash, the GNU Bourne-Again shell. But the C shell (csh) behaves differently! There, you must use two backslashes in a row, followed by a newline. Note also that when using the C shell, every newline in your awk program must be escaped with a backslash. To illustrate:

% awk 'BEGIN { \ ? print \\ ? "hello, world" \ ? }' -| hello, world

Here, the `%' and `?' are the C shell's primary and secondary prompts, analogous to the standard shell's `$' and `>'.

awk is a line-oriented language. Each rule's action has to begin on the same line as the pattern. To have the pattern and action on separate lines, you must use backslash continuation--there is no other way.

Note that backslash continuation and comments do not mix. As soon as awk sees the `#' that starts a comment, it ignores everything on the rest of the line. For example:

$ gawk 'BEGIN { print "dont panic" # a friendly \ > BEGIN rule > }' error--> gawk: cmd. line:2: BEGIN rule error--> gawk: cmd. line:2: ^ parse error

Here, it looks like the backslash would continue the comment onto the next line. However, the backslash-newline combination is never even noticed, since it is "hidden" inside the comment. Thus, the `BEGIN' is noted as a syntax error.

When awk statements within one rule are short, you might want to put more than one of them on a line. You do this by separating the statements with a semicolon, `;'.

This also applies to the rules themselves. Thus, the previous program could have been written:

/12/ { print $0 } ; /21/ { print $0 }

Note: the requirement that rules on the same line must be separated with a semicolon was not in the original awk language; it was added for consistency with the treatment of statements within an action.

2.7 Other Features of awk

The awk language provides a number of predefined, or built-in variables, which your programs can use to get information from awk. There are other variables your program can set to control how awk processes your data.

In addition, awk provides a number of built-in functions for doing common computational and string related operations.

As we develop our presentation of the awk language, we introduce most of the variables and many of the functions. They are defined systematically in 10. Built-in Variables, and Built-in Functions.

2.8 When to Use awk

You might wonder how awk might be useful for you. Using utility programs, advanced patterns, field separators, arithmetic statements, and other selection criteria, you can produce much more complex output. The awk language is very useful for producing reports from large amounts of raw data, such as summarizing information from the output of other utility programs like ls. (See section A More Complex Example.)

Programs written with awk are usually much smaller than they would be in other languages. This makes awk programs easy to compose and use. Often, awk programs can be quickly composed at your terminal, used once, and thrown away. Since awk programs are interpreted, you can avoid the (usually lengthy) compilation part of the typical edit-compile-test-debug cycle of software development.

Complex programs have been written in awk, including a complete retargetable assembler for eight-bit microprocessors (see section D. Glossary, for more information) and a microcode assembler for a special purpose Prolog computer. However, awk's capabilities are strained by tasks of such complexity.

If you find yourself writing awk scripts of more than, say, a few hundred lines, you might consider using a different programming language. Emacs Lisp is a good choice if you need sophisticated string or pattern matching capabilities. The shell is also good at string and pattern matching; in addition, it allows powerful use of the system utilities. More conventional languages, such as C, C++, and Lisp, offer better facilities for system programming and for managing the complexity of large programs. Programs in these languages may require more lines of source code than the equivalent awk programs, but they are easier to maintain and usually run more efficiently.

3. Useful One Line Programs

Many useful awk programs are short, just a line or two. Here is a collection of useful, short programs to get you started. Some of these programs contain constructs that haven't been covered yet. The description of the program will give you a good idea of what is going on, but please read the rest of the Info file to become an awk expert!

Most of the examples use a data file named `data'. This is just a placeholder; if you were to use these programs yourself, you would substitute your own file names for `data'.

Since you are reading this in Info, each line of the example code is enclosed in quotes, to represent text that you would type literally. The examples themselves represent shell commands that use single quotes to keep the shell from interpreting the contents of the program. When reading the examples, focus on the text between the open and close quotes.

awk '{ if (length($0) > max) max = length($0) }

END { print max }' data

This program prints the length of the longest input line.

awk 'length($0) > 80' data

This program prints every line that is longer than 80 characters. The sole rule has a relational expression as its pattern, and has no action (so the default action, printing the record, is used).

expand data | awk '{ if (x < length()) x = length() }

END { print "maximum line length is " x }'

This program prints the length of the longest line in `data'. The input is processed by the expand program to change tabs into spaces, so the widths compared are actually the right-margin columns.

awk 'NF > 0' data

This program prints every line that has at least one field. This is an easy way to delete blank lines from a file (or rather, to create a new file similar to the old file but from which the blank lines have been deleted).

awk 'BEGIN { for (i = 1; i <= 7; i++)

print int(101 * rand()) }'

This program prints seven random numbers from zero to 100, inclusive.

ls -lg files | awk '{ x += $5 } ; END { print "total bytes: " x }'

This program prints the total number of bytes used by files.

ls -lg files | awk '{ x += $5 }

END { print "total K-bytes: " (x + 1023)/1024 }'

This program prints the total number of kilobytes used by files.

awk -F: '{ print $1 }' /etc/passwd | sort

This program prints a sorted list of the login names of all users.

awk 'END { print NR }' data

This program counts lines in a file.

awk 'NR % 2 == 0' data

This program prints the even numbered lines in the data file. If you were to use the expression `NR % 2 == 1' instead, it would print the odd numbered lines.

4. Regular Expressions

A regular expression, or regexp, is a way of describing a set of strings. Because regular expressions are such a fundamental part of awk programming, their format and use deserve a separate chapter.

A regular expression enclosed in slashes (`/') is an awk pattern that matches every input record whose text belongs to that set.

The simplest regular expression is a sequence of letters, numbers, or both. Such a regexp matches any string that contains that sequence. Thus, the regexp `foo' matches any string containing `foo'. Therefore, the pattern /foo/ matches any input record containing the three characters `foo', anywhere in the record. Other kinds of regexps let you specify more complicated classes of strings.

4.1 How to Use Regular Expressions		How to Use Regular Expressions.
4.2 Escape Sequences		How to write non-printing characters.
4.3 Regular Expression Operators		Regular Expression Operators.
4.4 Additional Regexp Operators Only in gawk		Operators specific to GNU software.
4.5 Case-sensitivity in Matching		How to do case-insensitive matching.
4.6 How Much Text Matches?		How much text matches.
4.7 Using Dynamic Regexps		Using Dynamic Regexps.

4.1 How to Use Regular Expressions

A regular expression can be used as a pattern by enclosing it in slashes. Then the regular expression is tested against the entire text of each record. (Normally, it only needs to match some part of the text in order to succeed.) For example, this prints the second field of each record that contains the three characters `foo' anywhere in it:

$ awk '/foo/ { print $2 }' BBS-list -| 555-1234 -| 555-6699 -| 555-6480 -| 555-2127

Regular expressions can also be used in matching expressions. These expressions allow you to specify the string to match against; it need not be the entire current input record. The two operators, `~' and `!~', perform regular expression comparisons. Expressions using these operators can be used as patterns or in if, while, for, and do statements. (See section Control Statements in Actions.)

exp ~ /regexp/

This is true if the expression exp (taken as a string) is matched by regexp. The following example matches, or selects, all input records with the upper-case letter `J' somewhere in the first field:

$ awk '$1 ~ /J/' inventory-shipped -| Jan 13 25 15 115 -| Jun 31 42 75 492 -| Jul 24 34 67 436 -| Jan 21 36 64 620

So does this:

awk '{ if ($1 ~ /J/) print }' inventory-shipped

exp !~ /regexp/

This is true if the expression exp (taken as a character string) is not matched by regexp. The following example matches, or selects, all input records whose first field does not contain the upper-case letter `J':

$ awk '$1 !~ /J/' inventory-shipped -| Feb 15 32 24 226 -| Mar 15 24 34 228 -| Apr 31 52 63 420 -| May 16 34 29 208 ...

When a regexp is written enclosed in slashes, like /foo/, we call it a regexp constant, much like 5.27 is a numeric constant, and "foo" is a string constant.

4.2 Escape Sequences

Some characters cannot be included literally in string constants ("foo") or regexp constants (/foo/). You represent them instead with escape sequences, which are character sequences beginning with a backslash (`\').

One use of an escape sequence is to include a double-quote character in a string constant. Since a plain double-quote would end the string, you must use `\"' to represent an actual double-quote character as a part of the string. For example:

$ awk 'BEGIN { print "He said \"hi!\" to her." }' -| He said "hi!" to her.

The backslash character itself is another character that cannot be included normally; you write `\\' to put one backslash in the string or regexp. Thus, the string whose contents are the two characters `"' and `\' must be written "\"\\".

Another use of backslash is to represent unprintable characters such as tab or newline. While there is nothing to stop you from entering most unprintable characters directly in a string constant or regexp constant, they may look ugly.

Here is a table of all the escape sequences used in awk, and what they represent. Unless noted otherwise, all of these escape sequences apply to both string constants and regexp constants.

\\

A literal backslash, `\'.

\a

The "alert" character, Control-g, ASCII code 7 (BEL).

\b

Backspace, Control-h, ASCII code 8 (BS).

\f

Formfeed, Control-l, ASCII code 12 (FF).

\n

Newline, Control-j, ASCII code 10 (LF).

\r

Carriage return, Control-m, ASCII code 13 (CR).

\t

Horizontal tab, Control-i, ASCII code 9 (HT).

\v

Vertical tab, Control-k, ASCII code 11 (VT).

\nnn

The octal value nnn, where nnn are one to three digits between `0' and `7'. For example, the code for the ASCII ESC (escape) character is `\033'.

\xhh...

The hexadecimal value hh, where hh are hexadecimal digits (`0' through `9' and either `A' through `F' or `a' through `f'). Like the same construct in ANSI C, the escape sequence continues until the first non-hexadecimal digit is seen. However, using more than two hexadecimal digits produces undefined results. (The `\x' escape sequence is not allowed in POSIX awk.)

\/

A literal slash (necessary for regexp constants only). You use this when you wish to write a regexp constant that contains a slash. Since the regexp is delimited by slashes, you need to escape the slash that is part of the pattern, in order to tell awk to keep processing the rest of the regexp.

\"

A literal double-quote (necessary for string constants only). You use this when you wish to write a string constant that contains a double-quote. Since the string is delimited by double-quotes, you need to escape the quote that is part of the string, in order to tell awk to keep processing the rest of the string.

In gawk, there are additional two character sequences that begin with backslash that have special meaning in regexps. See section Additional Regexp Operators Only in gawk.

In a string constant, what happens if you place a backslash before something that is not one of the characters listed above? POSIX awk purposely leaves this case undefined. There are two choices.

• Strip the backslash out. This is what Unix awk and gawk both do. For example, "a\qc" is the same as "aqc".

• Leave the backslash alone. Some other awk implementations do this. In such implementations, "a\qc" is the same as if you had typed "a\\qc".

In a regexp, a backslash before any character that is not in the above table, and not listed in Additional Regexp Operators Only in gawk, means that the next character should be taken literally, even if it would normally be a regexp operator. E.g., /a\+b/ matches the three characters `a+b'.

For complete portability, do not use a backslash before any character not listed in the table above.

Another interesting question arises. Suppose you use an octal or hexadecimal escape to represent a regexp metacharacter (see section Regular Expression Operators). Does awk treat the character as a literal character, or as a regexp operator?

It turns out that historically, such characters were taken literally (d.c.). However, the POSIX standard indicates that they should be treated as real metacharacters, and this is what gawk does. However, in compatibility mode (see section Command Line Options), gawk treats the characters represented by octal and hexadecimal escape sequences literally when used in regexp constants. Thus, /a\52b/ is equivalent to /a\*b/.

To summarize:

1. The escape sequences in the table above are always processed first, for both string constants and regexp constants. This happens very early, as soon as awk reads your program.

2. gawk processes both regexp constants and dynamic regexps (see section Using Dynamic Regexps), for the special operators listed in Additional Regexp Operators Only in gawk.

3. A backslash before any other character means to treat that character literally.

4.3 Regular Expression Operators

You can combine regular expressions with the following characters, called regular expression operators, or metacharacters, to increase the power and versatility of regular expressions.

The escape sequences described in 4.2 Escape Sequences, are valid inside a regexp. They are introduced by a `\'. They are recognized and converted into the corresponding real characters as the very first step in processing regexps.

Here is a table of metacharacters. All characters that are not escape sequences and that are not listed in the table stand for themselves.

\

This is used to suppress the special meaning of a character when matching. For example:

\$

matches the character `$'.

^

This matches the beginning of a string. For example:

^@chapter

matches the `@chapter' at the beginning of a string, and can be used to identify chapter beginnings in Texinfo source files. The `^' is known as an anchor, since it anchors the pattern to matching only at the beginning of the string.

It is important to realize that `^' does not match the beginning of a line embedded in a string. In this example the condition is not true:

if ("line1\nLINE 2" ~ /^L/) ...

$

This is similar to `^', but it matches only at the end of a string. For example:

p$

matches a record that ends with a `p'. The `$' is also an anchor, and also does not match the end of a line embedded in a string. In this example the condition is not true:

if ("line1\nLINE 2" ~ /1$/) ...

.

The period, or dot, matches any single character, including the newline character. For example:

.P

matches any single character followed by a `P' in a string. Using concatenation we can make a regular expression like `U.A', which matches any three-character sequence that begins with `U' and ends with `A'.

In strict POSIX mode (see section Command Line Options), `.' does not match the NUL character, which is a character with all bits equal to zero. Otherwise, NUL is just another character. Other versions of awk may not be able to match the NUL character.

[...]

This is called a character list. It matches any one of the characters that are enclosed in the square brackets. For example:

[MVX]

matches any one of the characters `M', `V', or `X' in a string.

Ranges of characters are indicated by using a hyphen between the beginning and ending characters, and enclosing the whole thing in brackets. For example:

[0-9]

matches any digit. Multiple ranges are allowed. E.g., the list [A-Za-z0-9] is a common way to express the idea of "all alphanumeric characters."

To include one of the characters `\', `]', `-' or `^' in a character list, put a `\' in front of it. For example:

[d\]]

matches either `d', or `]'.

This treatment of `\' in character lists is compatible with other awk implementations, and is also mandated by POSIX. The regular expressions in awk are a superset of the POSIX specification for Extended Regular Expressions (EREs). POSIX EREs are based on the regular expressions accepted by the traditional egrep utility.

Character classes are a new feature introduced in the POSIX standard. A character class is a special notation for describing lists of characters that have a specific attribute, but where the actual characters themselves can vary from country to country and/or from character set to character set. For example, the notion of what is an alphabetic character differs in the USA and in France.

A character class is only valid in a regexp inside the brackets of a character list. Character classes consist of `[:', a keyword denoting the class, and `:]'. Here are the character classes defined by the POSIX standard.

[:alnum:]

Alphanumeric characters.

[:alpha:]

Alphabetic characters.

[:blank:]

Space and tab characters.

[:cntrl:]

Control characters.

[:digit:]

Numeric characters.

[:graph:]

Characters that are printable and are also visible. (A space is printable, but not visible, while an `a' is both.)

[:lower:]

Lower-case alphabetic characters.

[:print:]

Printable characters (characters that are not control characters.)

[:punct:]

Punctuation characters (characters that are not letter, digits, control characters, or space characters).

[:space:]

Space characters (such as space, tab, and formfeed, to name a few).

[:upper:]

Upper-case alphabetic characters.

[:xdigit:]

Characters that are hexadecimal digits.

For example, before the POSIX standard, to match alphanumeric characters, you had to write /[A-Za-z0-9]/. If your character set had other alphabetic characters in it, this would not match them. With the POSIX character classes, you can write /[[:alnum:]]/, and this will match all the alphabetic and numeric characters in your character set.

Two additional special sequences can appear in character lists. These apply to non-ASCII character sets, which can have single symbols (called collating elements) that are represented with more than one character, as well as several characters that are equivalent for collating, or sorting, purposes. (E.g., in French, a plain "e" and a grave-accented "è" are equivalent.)

Collating Symbols

A collating symbol is a multi-character collating element enclosed in `[.' and `.]'. For example, if `ch' is a collating element, then [[.ch.]] is a regexp that matches this collating element, while [ch] is a regexp that matches either `c' or `h'.

Equivalence Classes

An equivalence class is a locale-specific name for a list of characters that are equivalent. The name is enclosed in `[=' and `=]'. For example, the name `e' might be used to represent all of "e," "è," and "é." In this case, [[=e]] is a regexp that matches any of `e', `é', or `è'.

These features are very valuable in non-English speaking locales.

Caution: The library functions that gawk uses for regular expression matching currently only recognize POSIX character classes; they do not recognize collating symbols or equivalence classes.

[^ ...]

This is a complemented character list. The first character after the `[' must be a `^'. It matches any characters except those in the square brackets. For example:

[^0-9]

matches any character that is not a digit.

|

This is the alternation operator, and it is used to specify alternatives. For example:

^P|[0-9]

matches any string that matches either `^P' or `[0-9]'. This means it matches any string that starts with `P' or contains a digit.

The alternation applies to the largest possible regexps on either side. In other words, `|' has the lowest precedence of all the regular expression operators.

(...)

Parentheses are used for grouping in regular expressions as in arithmetic. They can be used to concatenate regular expressions containing the alternation operator, `|'. For example, `@(samp|code)\{[^}]+\}' matches both `@code{foo}' and `@samp{bar}'. (These are Texinfo formatting control sequences.)

*

This symbol means that the preceding regular expression is to be repeated as many times as necessary to find a match. For example:

ph*

applies the `*' symbol to the preceding `h' and looks for matches of one `p' followed by any number of `h's. This will also match just `p' if no `h's are present.

The `*' repeats the smallest possible preceding expression. (Use parentheses if you wish to repeat a larger expression.) It finds as many repetitions as possible. For example:

awk '/\(c[ad][ad]*r x\)/ { print }' sample

prints every record in `sample' containing a string of the form `(car x)', `(cdr x)', `(cadr x)', and so on. Notice the escaping of the parentheses by preceding them with backslashes.

+

This symbol is similar to `*', but the preceding expression must be matched at least once. This means that:

wh+y

would match `why' and `whhy' but not `wy', whereas `wh*y' would match all three of these strings. This is a simpler way of writing the last `*' example:

awk '/\(c[ad]+r x\)/ { print }' sample

?

This symbol is similar to `*', but the preceding expression can be matched either once or not at all. For example:

fe?d

will match `fed' and `fd', but nothing else.

{n}

{n,}

{n,m}

One or two numbers inside braces denote an interval expression. If there is one number in the braces, the preceding regexp is repeated n times. If there are two numbers separated by a comma, the preceding regexp is repeated n to m times. If there is one number followed by a comma, then the preceding regexp is repeated at least n times.

wh{3}y

matches `whhhy' but not `why' or `whhhhy'.

wh{3,5}y

matches `whhhy' or `whhhhy' or `whhhhhy', only.

wh{2,}y

matches `whhy' or `whhhy', and so on.

Interval expressions were not traditionally available in awk. As part of the POSIX standard they were added, to make awk and egrep consistent with each other.

However, since old programs may use `{' and `}' in regexp constants, by default gawk does not match interval expressions in regexps. If either `--posix' or `--re-interval' are specified (see section Command Line Options), then interval expressions are allowed in regexps.

In regular expressions, the `*', `+', and `?' operators, as well as the braces `{' and `}', have the highest precedence, followed by concatenation, and finally by `|'. As in arithmetic, parentheses can change how operators are grouped.

If gawk is in compatibility mode (see section Command Line Options), character classes and interval expressions are not available in regular expressions.

The next node discusses the GNU-specific regexp operators, and provides more detail concerning how command line options affect the way gawk interprets the characters in regular expressions.

4.4 Additional Regexp Operators Only in gawk

GNU software that deals with regular expressions provides a number of additional regexp operators. These operators are described in this section, and are specific to gawk; they are not available in other awk implementations.

Most of the additional operators are for dealing with word matching. For our purposes, a word is a sequence of one or more letters, digits, or underscores (`_').

\w

This operator matches any word-constituent character, i.e. any letter, digit, or underscore. Think of it as a short-hand for [[:alnum:]_].

\W

This operator matches any character that is not word-constituent. Think of it as a short-hand for [^[:alnum:]_].

\<

This operator matches the empty string at the beginning of a word. For example, /\<away/ matches `away', but not `stowaway'.

\>

This operator matches the empty string at the end of a word. For example, /stow\>/ matches `stow', but not `stowaway'.

\y

This operator matches the empty string at either the beginning or the end of a word (the word boundary). For example, `\yballs?\y' matches either `ball' or `balls' as a separate word.

\B

This operator matches the empty string within a word. In other words, `\B' matches the empty string that occurs between two word-constituent characters. For example, /\Brat\B/ matches `crate', but it does not match `dirty rat'. `\B' is essentially the opposite of `\y'.

There are two other operators that work on buffers. In Emacs, a buffer is, naturally, an Emacs buffer. For other programs, the regexp library routines that gawk uses consider the entire string to be matched as the buffer.

For awk, since `^' and `$' always work in terms of the beginning and end of strings, these operators don't add any new capabilities. They are provided for compatibility with other GNU software.

\`

This operator matches the empty string at the beginning of the buffer.

\'

This operator matches the empty string at the end of the buffer.

In other GNU software, the word boundary operator is `\b'. However, that conflicts with the awk language's definition of `\b' as backspace, so gawk uses a different letter.

An alternative method would have been to require two backslashes in the GNU operators, but this was deemed to be too confusing, and the current method of using `\y' for the GNU `\b' appears to be the lesser of two evils.

The various command line options (see section Command Line Options) control how gawk interprets characters in regexps.

No options

In the default case, gawk provides all the facilities of POSIX regexps and the GNU regexp operators described in Regular Expression Operators. However, interval expressions are not supported.

--posix

Only POSIX regexps are supported, the GNU operators are not special (e.g., `\w' matches a literal `w'). Interval expressions are allowed.

--traditional

Traditional Unix awk regexps are matched. The GNU operators are not special, interval expressions are not available, and neither are the POSIX character classes ([[:alnum:]] and so on). Characters described by octal and hexadecimal escape sequences are treated literally, even if they represent regexp metacharacters.

--re-interval

Allow interval expressions in regexps, even if `--traditional' has been provided.

4.5 Case-sensitivity in Matching

Case is normally significant in regular expressions, both when matching ordinary characters (i.e. not metacharacters), and inside character sets. Thus a `w' in a regular expression matches only a lower-case `w' and not an upper-case `W'.

The simplest way to do a case-independent match is to use a character list: `[Ww]'. However, this can be cumbersome if you need to use it often; and it can make the regular expressions harder to read. There are two alternatives that you might prefer.

One way to do a case-insensitive match at a particular point in the program is to convert the data to a single case, using the tolower or toupper built-in string functions (which we haven't discussed yet; see section Built-in Functions for String Manipulation). For example:

tolower($1) ~ /foo/ { ... }

converts the first field to lower-case before matching against it. This will work in any POSIX-compliant implementation of awk.

Another method, specific to gawk, is to set the variable IGNORECASE to a non-zero value (see section 10. Built-in Variables). When IGNORECASE is not zero, all regexp and string operations ignore case. Changing the value of IGNORECASE dynamically controls the case sensitivity of your program as it runs. Case is significant by default because IGNORECASE (like most variables) is initialized to zero.

x = "aB" if (x ~ /ab/) ... # this test will fail IGNORECASE = 1 if (x ~ /ab/) ... # now it will succeed

In general, you cannot use IGNORECASE to make certain rules case-insensitive and other rules case-sensitive, because there is no way to set IGNORECASE just for the pattern of a particular rule. To do this, you must use character lists or tolower. However, one thing you can do only with IGNORECASE is turn case-sensitivity on or off dynamically for all the rules at once.

IGNORECASE can be set on the command line, or in a BEGIN rule (see section Other Command Line Arguments; also see section Startup and Cleanup Actions). Setting IGNORECASE from the command line is a way to make a program case-insensitive without having to edit it.

Prior to version 3.0 of gawk, the value of IGNORECASE only affected regexp operations. It did not affect string comparison with `==', `!=', and so on. Beginning with version 3.0, both regexp and string comparison operations are affected by IGNORECASE.

Beginning with version 3.0 of gawk, the equivalences between upper-case and lower-case characters are based on the ISO-8859-1 (ISO Latin-1) character set. This character set is a superset of the traditional 128 ASCII characters, that also provides a number of characters suitable for use with European languages.

The value of IGNORECASE has no effect if gawk is in compatibility mode (see section Command Line Options). Case is always significant in compatibility mode.

4.6 How Much Text Matches?

Consider the following example:

echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }'

This example uses the sub function (which we haven't discussed yet, see section Built-in Functions for String Manipulation) to make a change to the input record. Here, the regexp /a+/ indicates "one or more `a' characters," and the replacement text is `<A>'.

The input contains four `a' characters. What will the output be? In other words, how many is "one or more"---will awk match two, three, or all four `a' characters?

The answer is, awk (and POSIX) regular expressions always match the leftmost, longest sequence of input characters that can match. Thus, in this example, all four `a' characters are replaced with `<A>'.

$ echo aaaabcd | awk '{ sub(/a+/, "<A>"); print }' -| <A>bcd

For simple match/no-match tests, this is not so important. But when doing text matching and substitutions with the match, sub, gsub, and gensub functions, it is very important. See section Built-in Functions for String Manipulation, for more information on these functions. Understanding this principle is also important for regexp-based record and field splitting (see section How Input is Split into Records, and also see section Specifying How Fields are Separated).

4.7 Using Dynamic Regexps

The right hand side of a `~' or `!~' operator need not be a regexp constant (i.e. a string of characters between slashes). It may be any expression. The expression is evaluated, and converted if necessary to a string; the contents of the string are used as the regexp. A regexp that is computed in this way is called a dynamic regexp. For example:

BEGIN { identifier_regexp = "[A-Za-z_][A-Za-z_0-9]*" } $0 ~ identifier_regexp { print }

sets identifier_regexp to a regexp that describes awk variable names, and tests if the input record matches this regexp.

Caution: When using the `~' and `!~' operators, there is a difference between a regexp constant enclosed in slashes, and a string constant enclosed in double quotes. If you are going to use a string constant, you have to understand that the string is in essence scanned twice; the first time when awk reads your program, and the second time when it goes to match the string on the left-hand side of the operator with the pattern on the right. This is true of any string valued expression (such as identifier_regexp above), not just string constants.

What difference does it make if the string is scanned twice? The answer has to do with escape sequences, and particularly with backslashes. To get a backslash into a regular expression inside a string, you have to type two backslashes.

For example, /\*/ is a regexp constant for a literal `*'. Only one backslash is needed. To do the same thing with a string, you would have to type "\\*". The first backslash escapes the second one, so that the string actually contains the two characters `\' and `*'.

Given that you can use both regexp and string constants to describe regular expressions, which should you use? The answer is "regexp constants," for several reasons.

1. String constants are more complicated to write, and more difficult to read. Using regexp constants makes your programs less error-prone. Not understanding the difference between the two kinds of constants is a common source of errors.

2. It is also more efficient to use regexp constants: awk can note that you have supplied a regexp and store it internally in a form that makes pattern matching more efficient. When using a string constant, awk must first convert the string into this internal form, and then perform the pattern matching.

3. Using regexp constants is better style; it shows clearly that you intend a regexp match.

5. Reading Input Files

In the typical awk program, all input is read either from the standard input (by default the keyboard, but often a pipe from another command) or from files whose names you specify on the awk command line. If you specify input files, awk reads them in order, reading all the data from one before going on to the next. The name of the current input file can be found in the built-in variable FILENAME (see section 10. Built-in Variables).

The input is read in units called records, and processed by the rules of your program one record at a time. By default, each record is one line. Each record is automatically split into chunks called fields. This makes it more convenient for programs to work on the parts of a record.

On rare occasions you will need to use the getline command. The getline command is valuable, both because it can do explicit input from any number of files, and because the files used with it do not have to be named on the awk command line (see section Explicit Input with getline).

5.1 How Input is Split into Records		Controlling how data is split into records.
5.2 Examining Fields		An introduction to fields.
5.3 Non-constant Field Numbers		Non-constant Field Numbers.
5.4 Changing the Contents of a Field		Changing the Contents of a Field.
5.5 Specifying How Fields are Separated		The field separator and how to change it.
5.6 Reading Fixed-width Data		Reading constant width data.
5.7 Multiple-Line Records		Reading multi-line records.
5.8 Explicit Input with getline		Reading files under explicit program control using the getline function.

5.1 How Input is Split into Records

The awk utility divides the input for your awk program into records and fields. Records are separated by a character called the record separator. By default, the record separator is the newline character. This is why records are, by default, single lines. You can use a different character for the record separator by assigning the character to the built-in variable RS.

You can change the value of RS in the awk program, like any other variable, with the assignment operator, `=' (see section Assignment Expressions). The new record-separator character should be enclosed in quotation marks, which indicate a string constant. Often the right time to do this is at the beginning of execution, before any input has been processed, so that the very first record will be read with the proper separator. To do this, use the special BEGIN pattern (see section The BEGIN and END Special Patterns). For example:

awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list

changes the value of RS to "/", before reading any input. This is a string whose first character is a slash; as a result, records are separated by slashes. Then the input file is read, and the second rule in the awk program (the action with no pattern) prints each record. Since each print statement adds a newline at the end of its output, the effect of this awk program is to copy the input with each slash changed to a newline. Here are the results of running the program on `BBS-list':

$ awk 'BEGIN { RS = "/" } ; { print $0 }' BBS-list -| aardvark 555-5553 1200 -| 300 B -| alpo-net 555-3412 2400 -| 1200 -| 300 A -| barfly 555-7685 1200 -| 300 A -| bites 555-1675 2400 -| 1200 -| 300 A -| camelot 555-0542 300 C -| core 555-2912 1200 -| 300 C -| fooey 555-1234 2400 -| 1200 -| 300 B -| foot 555-6699 1200 -| 300 B -| macfoo 555-6480 1200 -| 300 A -| sdace 555-3430 2400 -| 1200 -| 300 A -| sabafoo 555-2127 1200 -| 300 C -|

Note that the entry for the `camelot' BBS is not split. In the original data file (see section Data Files for the Examples), the line looks like this:

camelot 555-0542 300 C

It only has one baud rate; there are no slashes in the record.

Another way to change the record separator is on the command line, using the variable-assignment feature (see section Other Command Line Arguments).

awk '{ print $0 }' RS="/" BBS-list

This sets RS to `/' before processing `BBS-list'.

Using an unusual character such as `/' for the record separator produces correct behavior in the vast majority of cases. However, the following (extreme) pipeline prints a surprising `1'. There is one field, consisting of a newline. The value of the built-in variable NF is the number of fields in the current record.

$ echo | awk 'BEGIN { RS = "a" } ; { print NF }' -| 1

Reaching the end of an input file terminates the current input record, even if the last character in the file is not the character in RS (d.c.).

The empty string, "" (a string of no characters), has a special meaning as the value of RS: it means that records are separated by one or more blank lines, and nothing else. See section Multiple-Line Records, for more details.

If you change the value of RS in the middle of an awk run, the new value is used to delimit subsequent records, but the record currently being processed (and records already processed) are not affected.

After the end of the record has been determined, gawk sets the variable RT to the text in the input that matched RS.

The value of RS is in fact not limited to a one-character string. It can be any regular expression (see section Regular Expressions). In general, each record ends at the next string that matches the regular expression; the next record starts at the end of the matching string. This general rule is actually at work in the usual case, where RS contains just a newline: a record ends at the beginning of the next matching string (the next newline in the input) and the following record starts just after the end of this string (at the first character of the following line). The newline, since it matches RS, is not part of either record.

When RS is a single character, RT will contain the same single character. However, when RS is a regular expression, then RT becomes more useful; it contains the actual input text that matched the regular expression.

The following example illustrates both of these features. It sets RS equal to a regular expression that matches either a newline, or a series of one or more upper-case letters with optional leading and/or trailing white space (see section Regular Expressions).

$ echo record 1 AAAA record 2 BBBB record 3 | > gawk 'BEGIN { RS = "\n|( *[[:upper:]]+ *)" } > { print "Record =", $0, "and RT =", RT }' -| Record = record 1 and RT = AAAA -| Record = record 2 and RT = BBBB -| Record = record 3 and RT = -|

The final line of output has an extra blank line. This is because the value of RT is a newline, and then the print statement supplies its own terminating newline.

See section A Simple Stream Editor, for a more useful example of RS as a regexp and RT.

The use of RS as a regular expression and the RT variable are gawk extensions; they are not available in compatibility mode (see section Command Line Options). In compatibility mode, only the first character of the value of RS is used to determine the end of the record.

The awk utility keeps track of the number of records that have been read so far from the current input file. This value is stored in a built-in variable called FNR. It is reset to zero when a new file is started. Another built-in variable, NR, is the total number of input records read so far from all data files. It starts at zero but is never automatically reset to zero.

5.2 Examining Fields

When awk reads an input record, the record is automatically separated or parsed by the interpreter into chunks called fields. By default, fields are separated by whitespace, like words in a line. Whitespace in awk means any string of one or more spaces, tabs or newlines;(5) other characters such as formfeed, and so on, that are considered whitespace by other languages are not considered whitespace by awk.

The purpose of fields is to make it more convenient for you to refer to these pieces of the record. You don't have to use them--you can operate on the whole record if you wish--but fields are what make simple awk programs so powerful.

To refer to a field in an awk program, you use a dollar-sign, `$', followed by the number of the field you want. Thus, $1 refers to the first field, $2 to the second, and so on. For example, suppose the following is a line of input:

This seems like a pretty nice example.

Here the first field, or $1, is `This'; the second field, or $2, is `seems'; and so on. Note that the last field, $7, is `example.'. Because there is no space between the `e' and the `.', the period is considered part of the seventh field.

NF is a built-in variable whose value is the number of fields in the current record. awk updates the value of NF automatically, each time a record is read.

No matter how many fields there are, the last field in a record can be represented by $NF. So, in the example above, $NF would be the same as $7, which is `example.'. Why this works is explained below (see section Non-constant Field Numbers). If you try to reference a field beyond the last one, such as $8 when the record has only seven fields, you get the empty string.

$0, which looks like a reference to the "zeroth" field, is a special case: it represents the whole input record. $0 is used when you are not interested in fields.

Here are some more examples:

$ awk '$1 ~ /foo/ { print $0 }' BBS-list -| fooey 555-1234 2400/1200/300 B -| foot 555-6699 1200/300 B -| macfoo 555-6480 1200/300 A -| sabafoo 555-2127 1200/300 C

This example prints each record in the file `BBS-list' whose first field contains the string `foo'. The operator `~' is called a matching operator (see section How to Use Regular Expressions); it tests whether a string (here, the field $1) matches a given regular expression.

By contrast, the following example looks for `foo' in the entire record and prints the first field and the last field for each input record containing a match.

$ awk '/foo/ { print $1, $NF }' BBS-list -| fooey B -| foot B -| macfoo A -| sabafoo C

5.3 Non-constant Field Numbers

The number of a field does not need to be a constant. Any expression in the awk language can be used after a `$' to refer to a field. The value of the expression specifies the field number. If the value is a string, rather than a number, it is converted to a number. Consider this example:

awk '{ print $NR }'

Recall that NR is the number of records read so far: one in the first record, two in the second, etc. So this example prints the first field of the first record, the second field of the second record, and so on. For the twentieth record, field number 20 is printed; most likely, the record has fewer than 20 fields, so this prints a blank line.

Here is another example of using expressions as field numbers:

awk '{ print $(2*2) }' BBS-list

awk must evaluate the expression `(2*2)' and use its value as the number of the field to print. The `*' sign represents multiplication, so the expression `2*2' evaluates to four. The parentheses are used so that the multiplication is done before the `$' operation; they are necessary whenever there is a binary operator in the field-number expression. This example, then, prints the hours of operation (the fourth field) for every line of the file `BBS-list'. (All of the awk operators are listed, in order of decreasing precedence, in Operator Precedence (How Operators Nest).)

If the field number you compute is zero, you get the entire record. Thus, $(2-2) has the same value as $0. Negative field numbers are not allowed; trying to reference one will usually terminate your running awk program. (The POSIX standard does not define what happens when you reference a negative field number. gawk will notice this and terminate your program. Other awk implementations may behave differently.)

As mentioned in Examining Fields, the number of fields in the current record is stored in the built-in variable NF (also see section 10. Built-in Variables). The expression $NF is not a special feature: it is the direct consequence of evaluating NF and using its value as a field number.

5.4 Changing the Contents of a Field

You can change the contents of a field as seen by awk within an awk program; this changes what awk perceives as the current input record. (The actual input is untouched; awk never modifies the input file.)

Consider this example and its output:

$ awk '{ $3 = $2 - 10; print $2, $3 }' inventory-shipped -| 13 3 -| 15 5 -| 15 5 ...

The `-' sign represents subtraction, so this program reassigns field three, $3, to be the value of field two minus ten, `$2 - 10'. (See section Arithmetic Operators.) Then field two, and the new value for field three, are printed.

In order for this to work, the text in field $2 must make sense as a number; the string of characters must be converted to a number in order for the computer to do arithmetic on it. The number resulting from the subtraction is converted back to a string of characters which then becomes field three. See section Conversion of Strings and Numbers.

When you change the value of a field (as perceived by awk), the text of the input record is recalculated to contain the new field where the old one was. Therefore, $0 changes to reflect the altered field. Thus, this program prints a copy of the input file, with 10 subtracted from the second field of each line.

$ awk '{ $2 = $2 - 10; print $0 }' inventory-shipped -| Jan 3 25 15 115 -| Feb 5 32 24 226 -| Mar 5 24 34 228 ...

You can also assign contents to fields that are out of range. For example:

$ awk '{ $6 = ($5 + $4 + $3 + $2) > print $6 }' inventory-shipped -| 168 -| 297 -| 301 ...

We've just created $6, whose value is the sum of fields $2, $3, $4, and $5. The `+' sign represents addition. For the file `inventory-shipped', $6 represents the total number of parcels shipped for a particular month.

Creating a new field changes awk's internal copy of the current input record--the value of $0. Thus, if you do `print $0' after adding a field, the record printed includes the new field, with the appropriate number of field separators between it and the previously existing fields.

This recomputation affects and is affected by NF (the number of fields; see section Examining Fields), and by a feature that has not been discussed yet, the output field separator, OFS, which is used to separate the fields (see section 6.3 Output Separators). For example, the value of NF is set to the number of the highest field you create.

Note, however, that merely referencing an out-of-range field does not change the value of either $0 or NF. Referencing an out-of-range field only produces an empty string. For example:

if ($(NF+1) != "") print "can't happen" else print "everything is normal"

should print `everything is normal', because NF+1 is certain to be out of range. (See section The if-else Statement, for more information about awk's if-else statements. See section Variable Typing and Comparison Expressions, for more information about the `!=' operator.)

It is important to note that making an assignment to an existing field will change the value of $0, but will not change the value of NF, even when you assign the empty string to a field. For example:

$ echo a b c d | awk '{ OFS = ":"; $2 = "" > print $0; print NF }' -| a::c:d -| 4

The field is still there; it just has an empty value. You can tell because there are two colons in a row.

This example shows what happens if you create a new field.

$ echo a b c d | awk '{ OFS = ":"; $2 = ""; $6 = "new" > print $0; print NF }' -| a::c:d::new -| 6

The intervening field, $5 is created with an empty value (indicated by the second pair of adjacent colons), and NF is updated with the value six.

Finally, decrementing NF will lose the values of the fields after the new value of NF, and $0 will be recomputed. Here is an example:

$ echo a b c d e f | ../gawk '{ print "NF =", NF; > NF = 3; print $0 }' -| NF = 6 -| a b c

5.5 Specifying How Fields are Separated

This section is rather long; it describes one of the most fundamental operations in awk.

5.5.1 The Basics of Field Separating		How fields are split with single characters or simple strings.
5.5.2 Using Regular Expressions to Separate Fields		Using regexps as the field separator.
5.5.3 Making Each Character a Separate Field		Making each character a separate field.
5.5.4 Setting FS from the Command Line		Setting FS from the command line.
5.5.5 Field Splitting Summary		Some final points and a summary table.

5.5.1 The Basics of Field Separating

The field separator, which is either a single character or a regular expression, controls the way awk splits an input record into fields. awk scans the input record for character sequences that match the separator; the fields themselves are the text between the matches.

In the examples below, we use the bullet symbol "*" to represent spaces in the output.

If the field separator is `oo', then the following line:

moo goo gai pan

would be split into three fields: `m', `*g' and `*gai*pan'. Note the leading spaces in the values of the second and third fields.

The field separator is represented by the built-in variable FS. Shell programmers take note! awk does not use the name IFS which is used by the POSIX compatible shells (such as the Bourne shell, sh, or the GNU Bourne-Again Shell, Bash).

You can change the value of FS in the awk program with the assignment operator, `=' (see section Assignment Expressions). Often the right time to do this is at the beginning of execution, before any input has been processed, so that the very first record will be read with the proper separator. To do this, use the special BEGIN pattern (see section The BEGIN and END Special Patterns). For example, here we set the value of FS to the string ",":

awk 'BEGIN { FS = "," } ; { print $2 }'

Given the input line,

John Q. Smith, 29 Oak St., Walamazoo, MI 42139

this awk program extracts and prints the string `*29*Oak*St.'.

Sometimes your input data will contain separator characters that don't separate fields the way you thought they would. For instance, the person's name in the example we just used might have a title or suffix attached, such as `John Q. Smith, LXIX'. From input containing such a name:

John Q. Smith, LXIX, 29 Oak St., Walamazoo, MI 42139

the above program would extract `*LXIX', instead of `*29*Oak*St.'. If you were expecting the program to print the address, you would be surprised. The moral is: choose your data layout and separator characters carefully to prevent such problems.

Normally, fields are separated by whitespace sequences (spaces, tabs and newlines), not by single spaces: two spaces in a row do not delimit an empty field. The default value of the field separator FS is a string containing a single space, " ". If this value were interpreted in the usual way, each space character would separate fields, so two spaces in a row would make an empty field between them. The reason this does not happen is that a single space as the value of FS is a special case: it is taken to specify the default manner of delimiting fields.

If FS is any other single character, such as ",", then each occurrence of that character separates two fields. Two consecutive occurrences delimit an empty field. If the character occurs at the beginning or the end of the line, that too delimits an empty field. The space character is the only single character which does not follow these rules.

5.5.2 Using Regular Expressions to Separate Fields

The previous node discussed the use of single characters or simple strings as the value of FS. More generally, the value of FS may be a string containing any regular expression. In this case, each match in the record for the regular expression separates fields. For example, the assignment:

FS = ", \t"

makes every area of an input line that consists of a comma followed by a space and a tab, into a field separator. (`\t' is an escape sequence that stands for a tab; see section 4.2 Escape Sequences, for the complete list of similar escape sequences.)

For a less trivial example of a regular expression, suppose you want single spaces to separate fields the way single commas were used above. You can set FS to "[ ]" (left bracket, space, right bracket). This regular expression matches a single space and nothing else (see section Regular Expressions).

There is an important difference between the two cases of `FS = " "' (a single space) and `FS = "[ \t\n]+"' (left bracket, space, backslash, "t", backslash, "n", right bracket, which is a regular expression matching one or more spaces, tabs, or newlines). For both values of FS, fields are separated by runs of spaces, tabs and/or newlines. However, when the value of FS is " ", awk will first strip leading and trailing whitespace from the record, and then decide where the fields are.

For example, the following pipeline prints `b':

$ echo ' a b c d ' | awk '{ print $2 }' -| b

However, this pipeline prints `a' (note the extra spaces around each letter):

$ echo ' a b c d ' | awk 'BEGIN { FS = "[ \t]+" } > { print $2 }' -| a

In this case, the first field is null, or empty.

The stripping of leading and trailing whitespace also comes into play whenever $0 is recomputed. For instance, study this pipeline:

$ echo ' a b c d' | awk '{ print; $2 = $2; print }' -| a b c d -| a b c d

The first print statement prints the record as it was read, with leading whitespace intact. The assignment to $2 rebuilds $0 by concatenating $1 through $NF together, separated by the value of OFS. Since the leading whitespace was ignored when finding $1, it is not part of the new $0. Finally, the last print statement prints the new $0.

5.5.3 Making Each Character a Separate Field

There are times when you may want to examine each character of a record separately. In gawk, this is easy to do, you simply assign the null string ("") to FS. In this case, each individual character in the record will become a separate field. Here is an example:

$ echo a b | gawk 'BEGIN { FS = "" } > { > for (i = 1; i <= NF; i = i + 1) > print "Field", i, "is", $i > }' -| Field 1 is a -| Field 2 is -| Field 3 is b

Traditionally, the behavior for FS equal to "" was not defined. In this case, Unix awk would simply treat the entire record as only having one field (d.c.). In compatibility mode (see section Command Line Options), if FS is the null string, then gawk will also behave this way.

5.5.4 Setting FS from the Command Line

FS can be set on the command line. You use the `-F' option to do so. For example:

awk -F, 'program' input-files

sets FS to be the `,' character. Notice that the option uses a capital `F'. Contrast this with `-f', which specifies a file containing an awk program. Case is significant in command line options: the `-F' and `-f' options have nothing to do with each other. You can use both options at the same time to set the FS variable and get an awk program from a file.

The value used for the argument to `-F' is processed in exactly the same way as assignments to the built-in variable FS. This means that if the field separator contains special characters, they must be escaped appropriately. For example, to use a `\' as the field separator, you would have to type:

# same as FS = "\\" awk -F\\\\ '...' files ...

Since `\' is used for quoting in the shell, awk will see `-F\\'. Then awk processes the `\\' for escape characters (see section 4.2 Escape Sequences), finally yielding a single `\' to be used for the field separator.

As a special case, in compatibility mode (see section Command Line Options), if the argument to `-F' is `t', then FS is set to the tab character. This is because if you type `-F\t' at the shell, without any quotes, the `\' gets deleted, so awk figures that you really want your fields to be separated with tabs, and not `t's. Use `-v FS="t"' on the command line if you really do want to separate your fields with `t's (see section Command Line Options).

For example, let's use an awk program file called `baud.awk' that contains the pattern /300/, and the action `print $1'. Here is the program:

/300/ { print $1 }

Let's also set FS to be the `-' character, and run the program on the file `BBS-list'. The following command prints a list of the names of the bulletin boards that operate at 300 baud and the first three digits of their phone numbers:

$ awk -F- -f baud.awk BBS-list -| aardvark 555 -| alpo -| barfly 555 ...

Note the second line of output. In the original file (see section Data Files for the Examples), the second line looked like this:

alpo-net 555-3412 2400/1200/300 A

The `-' as part of the system's name was used as the field separator, instead of the `-' in the phone number that was originally intended. This demonstrates why you have to be careful in choosing your field and record separators.

On many Unix systems, each user has a separate entry in the system password file, one line per user. The information in these lines is separated by colons. The first field is the user's logon name, and the second is the user's encrypted password. A password file entry might look like this:

arnold:xyzzy:2076:10:Arnold Robbins:/home/arnold:/bin/sh

The following program searches the system password file, and prints the entries for users who have no password:

awk -F: '$2 == ""' /etc/passwd

5.5.5 Field Splitting Summary

According to the POSIX standard, awk is supposed to behave as if each record is split into fields at the time that it is read. In particular, this means that you can change the value of FS after a record is read, and the value of the fields (i.e. how they were split) should reflect the old value of FS, not the new one.

However, many implementations of awk do not work this way. Instead, they defer splitting the fields until a field is actually referenced. The fields will be split using the current value of FS! (d.c.) This behavior can be difficult to diagnose. The following example illustrates the difference between the two methods. (The sed(6) command prints just the first line of `/etc/passwd'.)

sed 1q /etc/passwd | awk '{ FS = ":" ; print $1 }'

will usually print

root

on an incorrect implementation of awk, while gawk will print something like

root:nSijPlPhZZwgE:0:0:Root:/:

The following table summarizes how fields are split, based on the value of FS. (`==' means "is equal to.")

FS == " "

Fields are separated by runs of whitespace. Leading and trailing whitespace are ignored. This is the default.

FS == any other single character

Fields are separated by each occurrence of the character. Multiple successive occurrences delimit empty fields, as do leading and trailing occurrences. The character can even be a regexp metacharacter; it does not need to be escaped.

FS == regexp

Fields are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty fields.

FS == ""

Each individual character in the record becomes a separate field.

5.6 Reading Fixed-width Data

(This section discusses an advanced, experimental feature. If you are a novice awk user, you may wish to skip it on the first reading.)

gawk version 2.13 introduced a new facility for dealing with fixed-width fields with no distinctive field separator. Data of this nature arises, for example, in the input for old FORTRAN programs where numbers are run together; or in the output of programs that did not anticipate the use of their output as input for other programs.

An example of the latter is a table where all the columns are lined up by the use of a variable number of spaces and empty fields are just spaces. Clearly, awk's normal field splitting based on FS will not work well in this case. Although a portable awk program can use a series of substr calls on $0 (see section Built-in Functions for String Manipulation), this is awkward and inefficient for a large number of fields.

The splitting of an input record into fixed-width fields is specified by assigning a string containing space-separated numbers to the built-in variable FIELDWIDTHS. Each number specifies the width of the field including columns between fields. If you want to ignore the columns between fields, you can specify the width as a separate field that is subsequently ignored.

The following data is the output of the Unix w utility. It is useful to illustrate the use of FIELDWIDTHS.

10:06pm up 21 days, 14:04, 23 users User tty login idle JCPU PCPU what hzuo ttyV0 8:58pm 9 5 vi p24.tex hzang ttyV3 6:37pm 50 -csh eklye ttyV5 9:53pm 7 1 em thes.tex dportein ttyV6 8:17pm 1:47 -csh gierd ttyD3 10:00pm 1 elm dave ttyD4 9:47pm 4 4 w brent ttyp0 26Jun91 4:46 26:46 4:41 bash dave ttyq4 26Jun9115days 46 46 wnewmail

The following program takes the above input, converts the idle time to number of seconds and prints out the first two fields and the calculated idle time. (This program uses a number of awk features that haven't been introduced yet.)

BEGIN { FIELDWIDTHS = "9 6 10 6 7 7 35" } NR > 2 { idle = $4 sub(/^ */, "", idle) # strip leading spaces if (idle == "") idle = 0 if (idle ~ /:/) { split(idle, t, ":") idle = t[1] * 60 + t[2] } if (idle ~ /days/) idle *= 24 * 60 * 60 print $1, $2, idle }

Here is the result of running the program on the data:

hzuo ttyV0 0 hzang ttyV3 50 eklye ttyV5 0 dportein ttyV6 107 gierd ttyD3 1 dave ttyD4 0 brent ttyp0 286 dave ttyq4 1296000

Another (possibly more practical) example of fixed-width input data would be the input from a deck of balloting cards. In some parts of the United States, voters mark their choices by punching holes in computer cards. These cards are then processed to count the votes for any particular candidate or on any particular issue. Since a voter may choose not to vote on some issue, any column on the card may be empty. An awk program for processing such data could use the FIELDWIDTHS feature to simplify reading the data. (Of course, getting gawk to run on a system with card readers is another story!)

Assigning a value to FS causes gawk to return to using FS for field splitting. Use `FS = FS' to make this happen, without having to know the current value of FS.

This feature is still experimental, and may evolve over time. Note that in particular, gawk does not attempt to verify the sanity of the values used in the value of FIELDWIDTHS.

5.7 Multiple-Line Records

In some data bases, a single line cannot conveniently hold all the information in one entry. In such cases, you can use multi-line records.

The first step in doing this is to choose your data format: when records are not defined as single lines, how do you want to define them? What should separate records?

One technique is to use an unusual character or string to separate records. For example, you could use the formfeed character (written `\f' in awk, as in C) to separate them, making each record a page of the file. To do this, just set the variable RS to "\f" (a string containing the formfeed character). Any other character could equally well be used, as long as it won't be part of the data in a record.

Another technique is to have blank lines separate records. By a special dispensation, an empty string as the value of RS indicates that records are separated by one or more blank lines. If you set RS to the empty string, a record always ends at the first blank line encountered. And the next record doesn't start until the first non-blank line that follows--no matter how many blank lines appear in a row, they are considered one record-separator.

You can achieve the same effect as `RS = ""' by assigning the string "\n\n+" to RS. This regexp matches the newline at the end of the record, and one or more blank lines after the record. In addition, a regular expression always matches the longest possible sequence when there is a choice (see section How Much Text Matches?). So the next record doesn't start until the first non-blank line that follows--no matter how many blank lines appear in a row, they are considered one record-separator.

There is an important difference between `RS = ""' and `RS = "\n\n+"'. In the first case, leading newlines in the input data file are ignored, and if a file ends without extra blank lines after the last record, the final newline is removed from the record. In the second case, this special processing is not done (d.c.).

Now that the input is separated into records, the second step is to separate the fields in the record. One way to do this is to divide each of the lines into fields in the normal manner. This happens by default as the result of a special feature: when RS is set to the empty string, the newline character always acts as a field separator. This is in addition to whatever field separations result from FS.

The original motivation for this special exception was probably to provide useful behavior in the default case (i.e. FS is equal to " "). This feature can be a problem if you really don't want the newline character to separate fields, since there is no way to prevent it. However, you can work around this by using the split function to break up the record manually (see section Built-in Functions for String Manipulation).

Another way to separate fields is to put each field on a separate line: to do this, just set the variable FS to the string "\n". (This simple regular expression matches a single newline.)

A practical example of a data file organized this way might be a mailing list, where each entry is separated by blank lines. If we have a mailing list in a file named `addresses', that looks like this:

Jane Doe 123 Main Street Anywhere, SE 12345-6789 John Smith 456 Tree-lined Avenue Smallville, MW 98765-4321 ...

A simple program to process this file would look like this:

# addrs.awk --- simple mailing list program # Records are separated by blank lines. # Each line is one field. BEGIN { RS = "" ; FS = "\n" } { print "Name is:", $1 print "Address is:", $2 print "City and State are:", $3 print "" }

Running the program produces the following output:

$ awk -f addrs.awk addresses -| Name is: Jane Doe -| Address is: 123 Main Street -| City and State are: Anywhere, SE 12345-6789 -| -| Name is: John Smith -| Address is: 456 Tree-lined Avenue -| City and State are: Smallville, MW 98765-4321 -| ...

See section Printing Mailing Labels, for a more realistic program that deals with address lists.

The following table summarizes how records are split, based on the value of RS. (`==' means "is equal to.")

RS == "\n"

Records are separated by the newline character (`\n'). In effect, every line in the data file is a separate record, including blank lines. This is the default.

RS == any single character

Records are separated by each occurrence of the character. Multiple successive occurrences delimit empty records.

RS == ""

Records are separated by runs of blank lines. The newline character always serves as a field separator, in addition to whatever value FS may have. Leading and trailing newlines in a file are ignored.

RS == regexp

Records are separated by occurrences of characters that match regexp. Leading and trailing matches of regexp delimit empty records.

In all cases, gawk sets RT to the input text that matched the value specified by RS.

5.8 Explicit Input with getline

So far we have been getting our input data from awk's main input stream--either the standard input (usually your terminal, sometimes the output from another program) or from the files specified on the command line. The awk language has a special built-in command called getline that can be used to read input under your explicit control.

5.8.1 Introduction to getline		Introduction to the getline function.
5.8.2 Using getline with No Arguments		Using getline with no arguments.
5.8.3 Using getline Into a Variable		Using getline into a variable.
5.8.4 Using getline from a File		Using getline from a file.
5.8.5 Using getline Into a Variable from a File		Using getline into a variable from a file.
5.8.6 Using getline from a Pipe		Using getline from a pipe.
5.8.7 Using getline Into a Variable from a Pipe		Using getline into a variable from a pipe.
5.8.8 Summary of getline Variants		Summary Of getline Variants.

5.8.1 Introduction to getline

This command is used in several different ways, and should not be used by beginners. It is covered here because this is the chapter on input. The examples that follow the explanation of the getline command include material that has not been covered yet. Therefore, come back and study the getline command after you have reviewed the rest of this Info file and have a good knowledge of how awk works.

getline returns one if it finds a record, and zero if the end of the file is encountered. If there is some error in getting a record, such as a file that cannot be opened, then getline returns -1. In this case, gawk sets the variable ERRNO to a string describing the error that occurred.

In the following examples, command stands for a string value that represents a shell command.

5.8.2 Using getline with No Arguments

The getline command can be used without arguments to read input from the current input file. All it does in this case is read the next input record and split it up into fields. This is useful if you've finished processing the current record, but you want to do some special processing right now on the next record. Here's an example:

awk '{ if ((t = index($0, "/*")) != 0) { # value will be "" if t is 1 tmp = substr($0, 1, t - 1) u = index(substr($0, t + 2), "*/") while (u == 0) { if (getline <= 0) { m = "unexpected EOF or error" m = (m ": " ERRNO) print m > "/dev/stderr" exit } t = -1 u = index($0, "*/") } # substr expression will be "" if */ # occurred at end of line $0 = tmp substr($0, t + u + 3) } print $0 }'

This awk program deletes all C-style comments, `/* ... */', from the input. By replacing the `print $0' with other statements, you could perform more complicated processing on the decommented input, like searching for matches of a regular expression. This program has a subtle problem--it does not work if one comment ends and another begins on the same line.

This form of the getline command sets NF (the number of fields; see section Examining Fields), NR (the number of records read so far; see section How Input is Split into Records), FNR (the number of records read from this input file), and the value of $0.

Note: the new value of $0 is used in testing the patterns of any subsequent rules. The original value of $0 that triggered the rule which executed getline is lost (d.c.). By contrast, the next statement reads a new record but immediately begins processing it normally, starting with the first rule in the program. See section The next Statement.

5.8.3 Using getline Into a Variable

You can use `getline var' to read the next record from awk's input into the variable var. No other processing is done.

For example, suppose the next line is a comment, or a special string, and you want to read it, without triggering any rules. This form of getline allows you to read that line and store it in a variable so that the main read-a-line-and-check-each-rule loop of awk never sees it.

The following example swaps every two lines of input. For example, given:

wan tew free phore

it outputs:

tew wan phore free

Here's the program:

awk '{ if ((getline tmp) > 0) { print tmp print $0 } else print $0 }'

The getline command used in this way sets only the variables NR and FNR (and of course, var). The record is not split into fields, so the values of the fields (including $0) and the value of NF do not change.

5.8.4 Using getline from a File

Use `getline < file' to read the next record from the file file. Here file is a string-valued expression that specifies the file name. `< file' is called a redirection since it directs input to come from a different place.

For example, the following program reads its input record from the file `secondary.input' when it encounters a first field with a value equal to 10 in the current input file.

awk '{ if ($1 == 10) { getline < "secondary.input" print } else print }'

Since the main input stream is not used, the values of NR and FNR are not changed. But the record read is split into fields in the normal manner, so the values of $0 and other fields are changed. So is the value of NF.

According to POSIX, `getline < expression' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `getline < dir "/" file' is ambiguous because the concatenation operator is not parenthesized, and you should write it as `getline < (dir "/" file)' if you want your program to be portable to other awk implementations.

5.8.5 Using getline Into a Variable from a File

Use `getline var < file' to read input the file file and put it in the variable var. As above, file is a string-valued expression that specifies the file from which to read.

In this version of getline, none of the built-in variables are changed, and the record is not split into fields. The only variable changed is var.

According to POSIX, `getline var < expression' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `getline < dir "/" file' is ambiguous because the concatenation operator is not parenthesized, and you should write it as `getline < (dir "/" file)' if you want your program to be portable to other awk implementations.

For example, the following program copies all the input files to the output, except for records that say `@include filename'. Such a record is replaced by the contents of the file filename.

awk '{ if (NF == 2 && $1 == "@include") { while ((getline line < $2) > 0) print line close($2) } else print }'

Note here how the name of the extra input file is not built into the program; it is taken directly from the data, from the second field on the `@include' line.

The close function is called to ensure that if two identical `@include' lines appear in the input, the entire specified file is included twice. See section Closing Input and Output Files and Pipes.

One deficiency of this program is that it does not process nested `@include' statements (`@include' statements in included files) the way a true macro preprocessor would. See section An Easy Way to Use Library Functions, for a program that does handle nested `@include' statements.

5.8.6 Using getline from a Pipe

You can pipe the output of a command into getline, using `command | getline'. In this case, the string command is run as a shell command and its output is piped into awk to be used as input. This form of getline reads one record at a time from the pipe.

For example, the following program copies its input to its output, except for lines that begin with `@execute', which are replaced by the output produced by running the rest of the line as a shell command:

awk '{ if ($1 == "@execute") { tmp = substr($0, 10) while ((tmp | getline) > 0) print close(tmp) } else print }'

The close function is called to ensure that if two identical `@execute' lines appear in the input, the command is run for each one. See section Closing Input and Output Files and Pipes.

Given the input:

foo bar baz @execute who bletch

the program might produce:

foo bar baz arnold ttyv0 Jul 13 14:22 miriam ttyp0 Jul 13 14:23 (murphy:0) bill ttyp1 Jul 13 14:23 (murphy:0) bletch

Notice that this program ran the command who and printed the result. (If you try this program yourself, you will of course get different results, showing you who is logged in on your system.)

This variation of getline splits the record into fields, sets the value of NF and recomputes the value of $0. The values of NR and FNR are not changed.

According to POSIX, `expression | getline' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `"echo " "date" | getline' is ambiguous because the concatenation operator is not parenthesized, and you should write it as `("echo " "date") | getline' if you want your program to be portable to other awk implementations. (It happens that gawk gets it right, but you should not rely on this. Parentheses make it easier to read, anyway.)

5.8.7 Using getline Into a Variable from a Pipe

When you use `command | getline var', the output of the command command is sent through a pipe to getline and into the variable var. For example, the following program reads the current date and time into the variable current_time, using the date utility, and then prints it.

awk 'BEGIN { "date" | getline current_time close("date") print "Report printed on " current_time }'

In this version of getline, none of the built-in variables are changed, and the record is not split into fields.

According to POSIX, `expression | getline var' is ambiguous if expression contains unparenthesized operators other than `$'; for example, `"echo " "date" | getline var' is ambiguous because the concatenation operator is not parenthesized, and you should write it as `("echo " "date") | getline var' if you want your program to be portable to other awk implementations. (It happens that gawk gets it right, but you should not rely on this. Parentheses make it easier to read, anyway.)

5.8.8 Summary of getline Variants

With all the forms of getline, even though $0 and NF, may be updated, the record will not be tested against all the patterns in the awk program, in the way that would happen if the record were read normally by the main processing loop of awk. However the new record is tested against any subsequent rules.

Many awk implementations limit the number of pipelines an awk program may have open to just one! In gawk, there is no such limit. You can open as many pipelines as the underlying operating system will permit.

An interesting side-effect occurs if you use getline (without a redirection) inside a BEGIN rule. Since an unredirected getline reads from the command line data files, the first getline command causes awk to set the value of FILENAME. Normally, FILENAME does not have a value inside BEGIN rules, since you have not yet started to process the command line data files (d.c.). (See section The BEGIN and END Special Patterns, also see section Built-in Variables that Convey Information.)

The following table summarizes the six variants of getline, listing which built-in variables are set by each one.

getline

sets $0, NF, FNR, and NR.

getline var

sets var, FNR, and NR.

getline < file

sets $0, and NF.

getline var < file

sets var.

command | getline

sets $0, and NF.

command | getline var

sets var.

6. Printing Output

One of the most common actions is to print, or output, some or all of the input. You use the print statement for simple output. You use the printf statement for fancier formatting. Both are described in this chapter.

6.1 The print Statement		The print statement.
6.2 Examples of print Statements		Simple examples of print statements.
6.3 Output Separators		The output separators and how to change them.
6.4 Controlling Numeric Output with print		Controlling Numeric Output With print.
6.5 Using printf Statements for Fancier Printing		The printf statement.
6.6 Redirecting Output of print and printf		How to redirect output to multiple files and pipes.
6.7 Special File Names in gawk		File name interpretation in gawk.

                                gawk allows access to inherited file
                                descriptors.

6.8 Closing Input and Output Files and Pipes

Closing Input and Output Files and Pipes.

6.1 The print Statement

The print statement does output with simple, standardized formatting. You specify only the strings or numbers to be printed, in a list separated by commas. They are output, separated by single spaces, followed by a newline. The statement looks like this:

print item1, item2, ...

The entire list of items may optionally be enclosed in parentheses. The parentheses are necessary if any of the item expressions uses the `>' relational operator; otherwise it could be confused with a redirection (see section Redirecting Output of print and printf).

The items to be printed can be constant strings or numbers, fields of the current record (such as $1), variables, or any awk expressions. Numeric values are converted to strings, and then printed.

The print statement is completely general for computing what values to print. However, with two exceptions, you cannot specify how to print them--how many columns, whether to use exponential notation or not, and so on. (For the exceptions, see section 6.3 Output Separators, and Controlling Numeric Output with print.) For that, you need the printf statement (see section Using printf Statements for Fancier Printing).

The simple statement `print' with no items is equivalent to `print $0': it prints the entire current record. To print a blank line, use `print ""', where "" is the empty string.

To print a fixed piece of text, use a string constant such as "Don't Panic" as one item. If you forget to use the double-quote characters, your text will be taken as an awk expression, and you will probably get an error. Keep in mind that a space is printed between any two items.

Each print statement makes at least one line of output. But it isn't limited to one line. If an item value is a string that contains a newline, the newline is output along with the rest of the string. A single print can make any number of lines this way.

6.2 Examples of print Statements

Here is an example of printing a string that contains embedded newlines (the `\n' is an escape sequence, used to represent the newline character; see section 4.2 Escape Sequences):

$ awk 'BEGIN { print "line one\nline two\nline three" }' -| line one -| line two -| line three

Here is an example that prints the first two fields of each input record, with a space between them:

$ awk '{ print $1, $2 }' inventory-shipped -| Jan 13 -| Feb 15 -| Mar 15 ...

A common mistake in using the print statement is to omit the comma between two items. This often has the effect of making the items run together in the output, with no space. The reason for this is that juxtaposing two string expressions in awk means to concatenate them. Here is the same program, without the comma:

$ awk '{ print $1 $2 }' inventory-shipped -| Jan13 -| Feb15 -| Mar15 ...

To someone unfamiliar with the file `inventory-shipped', neither example's output makes much sense. A heading line at the beginning would make it clearer. Let's add some headings to our table of months ($1) and green crates shipped ($2). We do this using the BEGIN pattern (see section The BEGIN and END Special Patterns) to force the headings to be printed only once:

awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, $2 }' inventory-shipped

Did you already guess what happens? When run, the program prints the following:

Month Crates ----- ------ Jan 13 Feb 15 Mar 15 ...

The headings and the table data don't line up! We can fix this by printing some spaces between the two fields:

awk 'BEGIN { print "Month Crates" print "----- ------" } { print $1, " ", $2 }' inventory-shipped

You can imagine that this way of lining up columns can get pretty complicated when you have many columns to fix. Counting spaces for two or three columns can be simple, but more than this and you can get lost quite easily. This is why the printf statement was created (see section Using printf Statements for Fancier Printing); one of its specialties is lining up columns of data.

As a side point, you can continue either a print or printf statement simply by putting a newline after any comma (see section awk Statements Versus Lines).

6.3 Output Separators

As mentioned previously, a print statement contains a list of items, separated by commas. In the output, the items are normally separated by single spaces. This need not be the case; a single space is only the default. You can specify any string of characters to use as the output field separator by setting the built-in variable OFS. The initial value of this variable is the string " ", that is, a single space.

The output from an entire print statement is called an output record. Each print statement outputs one output record and then outputs a string called the output record separator. The built-in variable ORS specifies this string. The initial value of ORS is the string "\n", i.e. a newline character; thus, normally each print statement makes a separate line.

You can change how output fields and records are separated by assigning new values to the variables OFS and/or ORS. The usual place to do this is in the BEGIN rule (see section The BEGIN and END Special Patterns), so that it happens before any input is processed. You may also do this with assignments on the command line, before the names of your input files, or using the `-v' command line option (see section Command Line Options).

The following example prints the first and second fields of each input record separated by a semicolon, with a blank line added after each line:

$ awk 'BEGIN { OFS = ";"; ORS = "\n\n" } > { print $1, $2 }' BBS-list -| aardvark;555-5553 -| -| alpo-net;555-3412 -| -| barfly;555-7685 ...

If the value of ORS does not contain a newline, all your output will be run together on a single line, unless you output newlines some other way.

6.4 Controlling Numeric Output with print

The built-in variable OFMT contains the default format specification that print uses with sprintf when it wants to convert a number to a string for printing. The default value of OFMT is "%.6g". By supplying different format specifications as the value of OFMT, you can change how print will print your numbers. As a brief example:

$ awk 'BEGIN { > OFMT = "%.0f" # print numbers as integers (rounds) > print 17.23 }' -| 17

According to the POSIX standard, awk's behavior will be undefined if OFMT contains anything but a floating point conversion specification (d.c.).

6.5 Using printf Statements for Fancier Printing

If you want more precise control over the output format than print gives you, use printf. With printf you can specify the width to use for each item, and you can specify various formatting choices for numbers (such as what radix to use, whether to print an exponent, whether to print a sign, and how many digits to print after the decimal point). You do this by supplying a string, called the format string, which controls how and where to print the other arguments.

6.5.1 Introduction to the printf Statement		Syntax of the printf statement.
6.5.2 Format-Control Letters		Format-control letters.
6.5.3 Modifiers for printf Formats		Format-specification modifiers.
6.5.4 Examples Using printf		Several examples.

6.5.1 Introduction to the printf Statement

The printf statement looks like this:

printf format, item1, item2, ...

The entire list of arguments may optionally be enclosed in parentheses. The parentheses are necessary if any of the item expressions use the `>' relational operator; otherwise it could be confused with a redirection (see section Redirecting Output of print and printf).

The difference between printf and print is the format argument. This is an expression whose value is taken as a string; it specifies how to output each of the other arguments. It is called the format string.

The format string is very similar to that in the ANSI C library function printf. Most of format is text to be output verbatim. Scattered among this text are format specifiers, one per item. Each format specifier says to output the next item in the argument list at that place in the format.

The printf statement does not automatically append a newline to its output. It outputs only what the format string specifies. So if you want a newline, you must include one in the format string. The output separator variables OFS and ORS have no effect on printf statements. For example:

BEGIN { ORS = "\nOUCH!\n"; OFS = "!" msg = "Don't Panic!"; printf "%s\n", msg }

This program still prints the familiar `Don't Panic!' message.

6.5.2 Format-Control Letters

A format specifier starts with the character `%' and ends with a format-control letter; it tells the printf statement how to output one item. (If you actually want to output a `%', write `%%'.) The format-control letter specifies what kind of value to print. The rest of the format specifier is made up of optional modifiers which are parameters to use, such as the field width.

Here is a list of the format-control letters:

c

This prints a number as an ASCII character. Thus, `printf "%c", 65' outputs the letter `A'. The output for a string value is the first character of the string.

d

i

These are equivalent. They both print a decimal integer. The `%i' specification is for compatibility with ANSI C.

e

E

This prints a number in scientific (exponential) notation. For example,

printf "%4.3e\n", 1950

prints `1.950e+03', with a total of four significant figures of which three follow the decimal point. The `4.3' are modifiers, discussed below. `%E' uses `E' instead of `e' in the output.

f

This prints a number in floating point notation. For example,

printf "%4.3f", 1950

prints `1950.000', with a total of four significant figures of which three follow the decimal point. The `4.3' are modifiers, discussed below.

g

G

This prints a number in either scientific notation or floating point notation, whichever uses fewer characters. If the result is printed in scientific notation, `%G' uses `E' instead of `e'.

o

This prints an unsigned octal integer. (In octal, or base-eight notation, the digits run from `0' to `7'; the decimal number eight is represented as `10' in octal.)

s

This prints a string.

u

This prints an unsigned decimal number. (This format is of marginal use, since all numbers in awk are floating point. It is provided primarily for compatibility with C.)

x

X

This prints an unsigned hexadecimal integer. (In hexadecimal, or base-16 notation, the digits are `0' through `9' and `a' through `f'. The hexadecimal digit `f' represents the decimal number 15.) `%X' uses the letters `A' through `F' instead of `a' through `f'.

%

This isn't really a format-control letter, but it does have a meaning when used after a `%': the sequence `%%' outputs one `%'. It does not consume an argument, and it ignores any modifiers.

When using the integer format-control letters for values that are outside the range of a C long integer, gawk will switch to the `%g' format specifier. Other versions of awk may print invalid values, or do something else entirely (d.c.).

6.5.3 Modifiers for printf Formats

A format specification can also include modifiers that can control how much of the item's value is printed and how much space it gets. The modifiers come between the `%' and the format-control letter. In the examples below, we use the bullet symbol "*" to represent spaces in the output. Here are the possible modifiers, in the order in which they may appear:

-

The minus sign, used before the width modifier (see below), says to left-justify the argument within its specified width. Normally the argument is printed right-justified in the specified width. Thus,

printf "%-4s", "foo"

prints `foo*'.

space

For numeric conversions, prefix positive values with a space, and negative values with a minus sign.

+

The plus sign, used before the width modifier (see below), says to always supply a sign for numeric conversions, even if the data to be formatted is positive. The `+' overrides the space modifier.

#

Use an "alternate form" for certain control letters. For `%o', supply a leading zero. For `%x', and `%X', supply a leading `0x' or `0X' for a non-zero result. For `%e', `%E', and `%f', the result will always contain a decimal point. For `%g', and `%G', trailing zeros are not removed from the result.

0

A leading `0' (zero) acts as a flag, that indicates output should be padded with zeros instead of spaces. This applies even to non-numeric output formats (d.c.). This flag only has an effect when the field width is wider than the value to be printed.

width

This is a number specifying the desired minimum width of a field. Inserting any number between the `%' sign and the format control character forces the field to be expanded to this width. The default way to do this is to pad with spaces on the left. For example,

printf "%4s", "foo"

prints `*foo'.

The value of width is a minimum width, not a maximum. If the item value requires more than width characters, it can be as wide as necessary. Thus,

printf "%4s", "foobar"

prints `foobar'.

Preceding the width with a minus sign causes the output to be padded with spaces on the right, instead of on the left.

.prec

This is a number that specifies the precision to use when printing. For the `e', `E', and `f' formats, this specifies the number of digits you want printed to the right of the decimal point. For the `g', and `G' formats, it specifies the maximum number of significant digits. For the `d', `o', `i', `u', `x', and `X' formats, it specifies the minimum number of digits to print. For a string, it specifies the maximum number of characters from the string that should be printed. Thus,

printf "%.4s", "foobar"

prints `foob'.

The C library printf's dynamic width and prec capability (for example, "%*.*s") is supported. Instead of supplying explicit width and/or prec values in the format string, you pass them in the argument list. For example:

w = 5 p = 3 s = "abcdefg" printf "%*.*s\n", w, p, s

is exactly equivalent to

s = "abcdefg" printf "%5.3s\n", s

Both programs output `**abc'.

Earlier versions of awk did not support this capability. If you must use such a version, you may simulate this feature by using concatenation to build up the format string, like so:

w = 5 p = 3 s = "abcdefg" printf "%" w "." p "s\n", s

This is not particularly easy to read, but it does work.

C programmers may be used to supplying additional `l' and `h' flags in printf format strings. These are not valid in awk. Most awk implementations silently ignore these flags. If `--lint' is provided on the command line (see section Command Line Options), gawk will warn about their use. If `--posix' is supplied, their use is a fatal error.

6.5.4 Examples Using printf

Here is how to use printf to make an aligned table:

awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list

prints the names of bulletin boards ($1) of the file `BBS-list' as a string of 10 characters, left justified. It also prints the phone numbers ($2) afterward on the line. This produces an aligned two-column table of names and phone numbers:

$ awk '{ printf "%-10s %s\n", $1, $2 }' BBS-list -| aardvark 555-5553 -| alpo-net 555-3412 -| barfly 555-7685 -| bites 555-1675 -| camelot 555-0542 -| core 555-2912 -| fooey 555-1234 -| foot 555-6699 -| macfoo 555-6480 -| sdace 555-3430 -| sabafoo 555-2127

Did you notice that we did not specify that the phone numbers be printed as numbers? They had to be printed as strings because the numbers are separated by a dash. If we had tried to print the phone numbers as numbers, all we would have gotten would have been the first three digits, `555'. This would have been pretty confusing.

We did not specify a width for the phone numbers because they are the last things on their lines. We don't need to put spaces after them.

We could make our table look even nicer by adding headings to the tops of the columns. To do this, we use the BEGIN pattern (see section The BEGIN and END Special Patterns) to force the header to be printed only once, at the beginning of the awk program:

awk 'BEGIN { print "Name Number" print "---- ------" } { printf "%-10s %s\n", $1, $2 }' BBS-list

Did you notice that we mixed print and printf statements in the above example? We could have used just printf statements to get the same results:

awk 'BEGIN { printf "%-10s %s\n", "Name", "Number" printf "%-10s %s\n", "----", "------" } { printf "%-10s %s\n", $1, $2 }' BBS-list

By printing each column heading with the same format specification used for the elements of the column, we have made sure that the headings are aligned just like the columns.

The fact that the same format specification is used three times can be emphasized by storing it in a variable, like this:

awk 'BEGIN { format = "%-10s %s\n" printf format, "Name", "Number" printf format, "----", "------" } { printf format, $1, $2 }' BBS-list

See if you can use the printf statement to line up the headings and table data for our `inventory-shipped' example covered earlier in the section on the print statement (see section The print Statement).

6.6 Redirecting Output of print and printf

So far we have been dealing only with output that prints to the standard output, usually your terminal. Both print and printf can also send their output to other places. This is called redirection.

A redirection appears after the print or printf statement. Redirections in awk are written just like redirections in shell commands, except that they are written inside the awk program.

There are three forms of output redirection: output to a file, output appended to a file, and output through a pipe to another command. They are all shown for the print statement, but they work identically for printf also.

print items > output-file

This type of redirection prints the items into the output file output-file. The file name output-file can be any expression. Its value is changed to a string and then used as a file name (see section 7. Expressions).

When this type of redirection is used, the output-file is erased before the first output is written to it. Subsequent writes to the same output-file do not erase output-file, but append to it. If output-file does not exist, then it is created.

For example, here is how an awk program can write a list of BBS names to a file `name-list' and a list of phone numbers to a file `phone-list'. Each output file contains one name or number per line.

$ awk '{ print $2 > "phone-list" > print $1 > "name-list" }' BBS-list $ cat phone-list -| 555-5553 -| 555-3412 ... $ cat name-list -| aardvark -| alpo-net ...

print items >> output-file

This type of redirection prints the items into the pre-existing output file output-file. The difference between this and the single-`>' redirection is that the old contents (if any) of output-file are not erased. Instead, the awk output is appended to the file. If output-file does not exist, then it is created.

print items | command

It is also possible to send output to another program through a pipe instead of into a file. This type of redirection opens a pipe to command and writes the values of items through this pipe, to another process created to execute command.

The redirection argument command is actually an awk expression. Its value is converted to a string, whose contents give the shell command to be run.

For example, this produces two files, one unsorted list of BBS names and one list sorted in reverse alphabetical order:

awk '{ print $1 > "names.unsorted" command = "sort -r > names.sorted" print $1 | command }' BBS-list

Here the unsorted list is written with an ordinary redirection while the sorted list is written by piping through the sort utility.

This example uses redirection to mail a message to a mailing list `bug-system'. This might be useful when trouble is encountered in an awk script run periodically for system maintenance.

report = "mail bug-system" print "Awk script failed:", $0 | report m = ("at record number " FNR " of " FILENAME) print m | report close(report)

The message is built using string concatenation and saved in the variable m. It is then sent down the pipeline to the mail program.

We call the close function here because it's a good idea to close the pipe as soon as all the intended output has been sent to it. See section Closing Input and Output Files and Pipes, for more information on this. This example also illustrates the use of a variable to represent a file or command: it is not necessary to always use a string constant. Using a variable is generally a good idea, since awk requires you to spell the string value identically every time.

Redirecting output using `>', `>>', or `|' asks the system to open a file or pipe only if the particular file or command you've specified has not already been written to by your program, or if it has been closed since it was last written to.

Many awk implementations limit the number of pipelines an awk program may have open to just one! In gawk, there is no such limit. You can open as many pipelines as the underlying operating system will permit.

6.7 Special File Names in gawk

Running programs conventionally have three input and output streams already available to them for reading and writing. These are known as the standard input, standard output, and standard error output. These streams are, by default, connected to your terminal, but they are often redirected with the shell, via the `<', `<<', `>', `>>', `>&' and `|' operators. Standard error is typically used for writing error messages; the reason we have two separate streams, standard output and standard error, is so that they can be redirected separately.

In other implementations of awk, the only way to write an error message to standard error in an awk program is as follows:

print "Serious error detected!" | "cat 1>&2"

This works by opening a pipeline to a shell command which can access the standard error stream which it inherits from the awk process. This is far from elegant, and is also inefficient, since it requires a separate process. So people writing awk programs often neglect to do this. Instead, they send the error messages to the terminal, like this:

print "Serious error detected!" > "/dev/tty"

This usually has the same effect, but not always: although the standard error stream is usually the terminal, it can be redirected, and when that happens, writing to the terminal is not correct. In fact, if awk is run from a background job, it may not have a terminal at all. Then opening `/dev/tty' will fail.

gawk provides special file names for accessing the three standard streams. When you redirect input or output in gawk, if the file name matches one of these special names, then gawk directly uses the stream it stands for.

`/dev/stdin'

The standard input (file descriptor 0).

`/dev/stdout'

The standard output (file descriptor 1).

`/dev/stderr'

The standard error output (file descriptor 2).

`/dev/fd/N'

The file associated with file descriptor N. Such a file must have been opened by the program initiating the awk execution (typically the shell). Unless you take special pains in the shell from which you invoke gawk, only descriptors 0, 1 and 2 are available.

The file names `/dev/stdin', `/dev/stdout', and `/dev/stderr' are aliases for `/dev/fd/0', `/dev/fd/1', and `/dev/fd/2', respectively, but they are more self-explanatory.

The proper way to write an error message in a gawk program is to use `/dev/stderr', like this:

print "Serious error detected!" > "/dev/stderr"

gawk also provides special file names that give access to information about the running gawk process. Each of these "files" provides a single record of information. To read them more than once, you must first close them with the close function (see section Closing Input and Output Files and Pipes). The filenames are:

`/dev/pid'

Reading this file returns the process ID of the current process, in decimal, terminated with a newline.

`/dev/ppid'

Reading this file returns the parent process ID of the current process, in decimal, terminated with a newline.

`/dev/pgrpid'

Reading this file returns the process group ID of the current process, in decimal, terminated with a newline.

`/dev/user'

Reading this file returns a single record terminated with a newline. The fields are separated with spaces. The fields represent the following information:

$1

The return value of the getuid system call (the real user ID number).

$2

The return value of the geteuid system call (the effective user ID number).

$3

The return value of the getgid system call (the real group ID number).

$4

The return value of the getegid system call (the effective group ID number).

If there are any additional fields, they are the group IDs returned by getgroups system call. (Multiple groups may not be supported on all systems.)

These special file names may be used on the command line as data files, as well as for I/O redirections within an awk program. They may not be used as source files with the `-f' option.

Recognition of these special file names is disabled if gawk is in compatibility mode (see section Command Line Options).

Caution: Unless your system actually has a `/dev/fd' directory (or any of the other above listed special files), the interpretation of these file names is done by gawk itself. For example, using `/dev/fd/4' for output will actually write on file descriptor 4, and not on a new file descriptor that was dup'ed from file descriptor 4. Most of the time this does not matter; however, it is important to not close any of the files related to file descriptors 0, 1, and 2. If you do close one of these files, unpredictable behavior will result.

The special files that provide process-related information will disappear in a future version of gawk. See section Probable Future Extensions.

6.8 Closing Input and Output Files and Pipes

If the same file name or the same shell command is used with getline (see section Explicit Input with getline) more than once during the execution of an awk program, the file is opened (or the command is executed) only the first time. At that time, the first record of input is read from that file or command. The next time the same file or command is used in getline, another record is read from it, and so on.

Similarly, when a file or pipe is opened for output, the file name or command associated with it is remembered by awk and subsequent writes to the same file or command are appended to the previous writes. The file or pipe stays open until awk exits.

This implies that if you want to start reading the same file again from the beginning, or if you want to rerun a shell command (rather than reading more output from the command), you must take special steps. What you must do is use the close function, as follows:

close(filename)

or

close(command)

The argument filename or command can be any expression. Its value must exactly match the string that was used to open the file or start the command (spaces and other "irrelevant" characters included). For example, if you open a pipe with this:

"sort -r names" | getline foo

then you must close it with this:

close("sort -r names")

Once this function call is executed, the next getline from that file or command, or the next print or printf to that file or command, will reopen the file or rerun the command.

Because the expression that you use to close a file or pipeline must exactly match the expression used to open the file or run the command, it is good practice to use a variable to store the file name or command. The previous example would become

sortcom = "sort -r names" sortcom | getline foo ... close(sortcom)

This helps avoid hard-to-find typographical errors in your awk programs.

Here are some reasons why you might need to close an output file:

• To write a file and read it back later on in the same awk program. Close the file when you are finished writing it; then you can start reading it with getline.

• To write numerous files, successively, in the same awk program. If you don't close the files, eventually you may exceed a system limit on the number of open files in one process. So close each one when you are finished writing it.

• To make a command finish. When you redirect output through a pipe, the command reading the pipe normally continues to try to read input as long as the pipe is open. Often this means the command cannot really do its work until the pipe is closed. For example, if you redirect output to the mail program, the message is not actually sent until the pipe is closed.

• To run the same program a second time, with the same arguments. This is not the same thing as giving more input to the first run!

  For example, suppose you pipe output to the mail program. If you output several lines redirected to this pipe without closing it, they make a single message of several lines. By contrast, if you close the pipe after each line of output, then each line makes a separate message.

close returns a value of zero if the close succeeded. Otherwise, the value will be non-zero. In this case, gawk sets the variable ERRNO to a string describing the error that occurred.

If you use more files than the system allows you to have open, gawk will attempt to multiplex the available open files among your data files. gawk's ability to do this depends upon the facilities of your operating system: it may not always work. It is therefore both good practice and good portability advice to always use close on your files when you are done with them.

7. Expressions

Expressions are the basic building blocks of awk patterns and actions. An expression evaluates to a value, which you can print, test, store in a variable or pass to a function. Additionally, an expression can assign a new value to a variable or a field, with an assignment operator.

An expression can serve as a pattern or action statement on its own. Most other kinds of statements contain one or more expressions which specify data on which to operate. As in other languages, expressions in awk include variables, array references, constants, and function calls, as well as combinations of these with various operators.

7.1 Constant Expressions		String, numeric, and regexp constants.
7.2 Using Regular Expression Constants		When and how to use a regexp constant.
7.3 Variables		Variables give names to values for later use.
7.4 Conversion of Strings and Numbers		The conversion of strings to numbers and vice versa.
7.5 Arithmetic Operators		Arithmetic operations (`+', `-', etc.)
7.6 String Concatenation		Concatenating strings.
7.7 Assignment Expressions		Changing the value of a variable or a field.
7.8 Increment and Decrement Operators		Incrementing the numeric value of a variable.
7.9 True and False in awk		What is "true" and what is "false".
7.10 Variable Typing and Comparison Expressions		How variables acquire types, and how this affects comparison of numbers and strings with

                                `<', etc.

7.11 Boolean Expressions

Combining comparison expressions using boolean operators `||' ("or"), `&&'

                                ("and") and `!' ("not").

7.12 Conditional Expressions		Conditional expressions select between two subexpressions under control of a third subexpression.
7.13 Function Calls		A function call is an expression.
7.14 Operator Precedence (How Operators Nest)		How various operators nest.

7.1 Constant Expressions

The simplest type of expression is the constant, which always has the same value. There are three types of constants: numeric constants, string constants, and regular expression constants.

7.1.1 Numeric and String Constants		Numeric and string constants.
7.1.2 Regular Expression Constants		Regular Expression constants.