Awk: Difference between revisions
Dubiousjim (talk | contribs) m (→Awk and Lua: tweak awk backref) |
Dubiousjim (talk | contribs) m (add some line breaks) |
||
(30 intermediate revisions by 4 users not shown) | |||
Line 1: | Line 1: | ||
This page compares BusyBox's implementation of awk (using [ | {{DISPLAYTITLE:awk}} | ||
This page compares BusyBox's implementation of awk (using [https://git.alpinelinux.org/cgit/aports/tree/main/busybox/busyboxconfig?id=v2.3.6 this] config file) with gawk (versions >= 3.1.8) and FreeBSD 9's nawk, which is based on [https://ia903404.us.archive.org/0/items/pdfy-MgN0H1joIoDVoIC7/The_AWK_Programming_Language.pdf Bell Labs/Brian Kernighan's 2007 version of awk]. | |||
It's not intended as a tutorial, but rather as a technical summary and list of "gotchas" (places where different implementations may behave in different or unexpected ways). | It's not intended as a tutorial, but rather as a technical summary and list of "gotchas" (places where different implementations may behave in different or unexpected ways). | ||
{{note|This summary was drawn up back when we were using uClibc. Some of the behavior of Alpine's version of BusyBox awk reported here may have changed with the switch to musl.}} | |||
__TOC__ | __TOC__ | ||
Line 7: | Line 11: | ||
== Invoking awk == | == Invoking awk == | ||
[ | awk [-F char_or_regex] [-v var=value ...] [--] 'awk script...' [ ARGV[1] ... ARGV[ARGC-1] ] | ||
awk [-F char_or_regex] [-v var=value ...] -f scriptfile ... [--] [ ARGV[1] ... ARGV[ARGC-1] ] | |||
Gawk makes a third invocation pattern possible, which mixes <code>-f scriptfile</code> options with an <code>'awk script...'</code> on the command-line: | |||
gawk [-F char_or_regex] [-v var=value ...] -f scriptfile ... -e 'awk script...' [--] [ ARGV[1] ... ARGV[ARGC-1] ] | |||
< | [https://github.com/dubiousjim/awkenough Awkenough] is a small set of Awk utility routines and a C stub that makes it easier to write shell scripts with awk shebang lines. <span style="color:gray"> (I'll make an Alpine package for this soon.)</span> It also permits mixing <code>-f scriptfile</code> and command-line scripts, using the same <code>-e</code> option gawk provides. In both cases, the long-form option <code>--source</code> works the same as <code>-e</code>. | ||
;Notes | |||
* | * all implementations honor <code>\t</code> and so on in <code>-v FS=expr</code>. BusyBox ''doesn't'' honor <code>\t</code> in <code>-F expr</code>; perhaps this is a bug. You can use the <code>$'\t'</code> escapes of BusyBox ash to work around this. | ||
* | * <code>nawk</code> and <code>gawk --traditional</code> (but not <code>gawk --posix</code>) interpret <code>-F t</code> as <code>-F '\t'</code>. This is a weird special-case preserved for historical compatibility. | ||
* | * scriptfile can be <code>-</code> for stdin | ||
* | * ARGVs can be of any form, but awk only knows how to ''automatically'' process arguments whose form is: | ||
** <code>var=unquoted_value</code>: assignments will be made when the main loop reaches that ARGV | |||
** <code>""</code>: will be skipped | |||
** <code>filename</code> | |||
** <code>-</code>: will use stdin | |||
: If no files are processed (nor <code>-</code>), stdin will be processed after all command-line assignments. | |||
* | * standalone scripts should look like this: | ||
:{{Cat|foo.awk| | |||
#!/usr/bin/awk -f | |||
# will receive the script's ARGC, ARGV with ARGV[0]{{=}}awk | |||
BEGIN { ... } | |||
... | |||
}} | |||
:or like this: | |||
:{{Cat|bar.awk| | |||
#!/usr/bin/runawk -f /usr/share/awkenough/library.awk | |||
... | |||
}} | |||
* gawk places any unrecognized short options that precede <code>--</code> into ARGV; other implementations (and <code>gawk --traditional</code>) fail/warn | |||
* some gawk-only options: | |||
** <code>--lint</code>: warns about nonportable constructs <!-- --lint=fatal, --lint=invalid --> | |||
** <code>--re-interval</code>: enables <code>/pat{1,3}/</code> regexes; some versions of gawk enable by default, others don't | |||
** <code>--traditional</code> or <code>--compat</code>: makes gawk behave like nawk | |||
** <code>--posix</code>: like <code>--traditional --re-interval</code>, with some further restrictions | |||
** <code>--exec scriptfile</code>: stops option processing and disables any further <code>-v</code>s; meant to be used in shebang line for scripts in untrusted environments (cgi) | |||
** <code>--sandbox</code>: disable <code>system("foo")</code>, <code>"foo"|getline</code>, <code>getline <"file"</code>, <code>print|"foo"</code>, <code>print >"file"</code>, and gawk's <code>extension(obj, func)</code>; so only local resources explicitly specified in ARGV will be accessible. This feature is only available on recent versions of gawk. | |||
</ | * when handling <code>-f <var>path</var></code> options where <var>path</var> contains no <code>/</code>s, gawk will first search in directories specified in an AWKPATH environment variable | ||
== Awk grammar == | == Awk grammar == | ||
=== Blocks === | === Blocks === | ||
; function definitions | |||
* Functions have global scope, and they can be called from positions that syntactically precede their definitions. | |||
* Scalars (that is, non-arrays) are passed by value, arrays are passed by reference. | |||
* Function calls can be nested or recursive. Each invocation of the function has its own local environment: so upon return, values of the scalar parameters the caller supplied shall be unchanged (array parameters may have been mutated). | |||
<ul><li> | |||
Consider functions defined like this: | |||
<pre> | |||
function foo(x,y) { | function foo(x,y) { | ||
... | ... | ||
bar() | |||
return | return "result" | ||
} | |||
function bar(z) { | |||
y += 1 | |||
return | |||
} | } | ||
</pre> | |||
Here the variables <code>x</code> and <code>y</code> are local to foo, and the variable <code>z</code> is local to bar. All other variable references are ''global'': contrast the shell, where non-local variable references are instead ''dynamic''. That is, in the shell, when foo calls bar, bar will be incrementing foo's local variable <code>y</code>. In awk, on the other hand, bar always only increments a global variable <code>y</code>. | |||
Note also that the number of arguments in a function call needn't match the number of parameters in the function's definition; if more arguments are supplied, they will be ignored; if fewer are supplied (as in foo's call of bar), the missing variables are assigned either "" or an empty array, depending on how they're used in the function. | |||
Note also the the <code>return</code> statement needn't be given an explicit return value (it will default to ""). The <code>return</code> statement can also be omitted altogether. | |||
</ul> | |||
; BEGIN and END | |||
BEGIN { ... } ... | |||
END { ... } ... | |||
BEGIN | * Multiple BEGIN blocks are merged; so too multiple END blocks. | ||
END | |||
* If there are only BEGIN blocks (no main loop or END blocks), and getline isn't used, then POSIX requires awk to terminate without reading stdin or any file operands. The implementations I checked honor this, but some historical implementations discarded input in this case. For portability, you can explicitly append <code>< /dev/null</code> to an invocation of awk you know shouldn't consume stdin. | |||
* If there is a main loop or END blocks, and awk's processing of stdin hasn't been suppressed by encountering files in its ARGV list, then stdin will be processed before executing any END block(s). | |||
* If there are END block(s), but <code>exit <var>status</var></code> is invoked outside them, then the main loop stops and control passes immediately to the first END block. If <code>exit <var>status</var></code> is invoked within an END block, or if there are no end blocks, the script terminates with result <var>status</var>. | |||
* POSIX requires that inside END, all of <code>FILENAME NR FNR NF</code> retain their most-recent values, so long as <code>getline</code> is not invoked. My implementations comply, and also retain the most-recent fields <code>$0 ...</code>, but earlier versions of nawk did not; and Darwin's awk may not. | |||
; main loop blocks | |||
These are of the form: | |||
<var>pattern</var> { <var>action</var> ... } | |||
or: | |||
<var>pattern</var>,<var>pattern</var> { <var>action</var> ... } | |||
where <var>action</var> defaults to <code>print $0</code>; and <var>pattern</var> defaults to <code>1</code> (that is, to true). | |||
<ul><li> | |||
'''Patterns''' can take any of the forms: | |||
* <code>/<var>regex</var>/</code> | |||
* <code><var>relational</var></code> | |||
* <code><var>pattern</var> && <var>pattern</var></code> | |||
* <code><var>pattern</var> || <var>pattern</var></code> | |||
* <code>! <var>pattern</var></code> | |||
* <code>( <var>pattern</var> )</code> | |||
* <code><var>pattern</var> ? <var>pattern</var> : <var>pattern</var></code> | |||
The first form is interpreted the same as the relational <code>$0 ~ /<var>regex</var>/</code>. | |||
<li> | |||
'''Relationals''' can take any of the forms: | |||
* <code><var>expr</var> ~ <var>RE</var></code> | |||
* <code><var>expr</var> !~ <var>RE</var></code> | |||
* <code><var>expr</var> <var>eqop</var> <var>expr</var></code> | |||
* <code><var>expr</var> in <var>arrayname</var></code> | |||
* <code>(<var>expr</var>, ...) in <var>arrayname</var></code> | |||
* <code><var>expr</var></code> | |||
The schemas <var>RE</var> can be regex literals like <code>/a[bc]+/</code> or any expression that evaluates to a string, like <code>"a[bc]+"</code>. Note that strings require an extra level of <code>\</code>s: you can write <code>/\w+/</code> or <code>"\\w+"</code>. | |||
The scemas <var>eqop</var> can be any of: <code>== != <= < > >=</code>. | |||
A bare expression, as in the last form, is interpreted as false when the expression evaluates to <code>0</code> or <code>""</code>, else to true. | |||
</ul> | |||
; other block forms | |||
BEGINFILE { ... } | |||
ENDFILE { ... } | |||
These are only available in (recent versions of) gawk. | |||
@include "filename" | |||
This is only available in some versions of gawk. See also Aleksey Cheusov's [https://runawk.sourceforge.net/ Runawk], which uses a different approach. | |||
=== Statements === | === Statements === | ||
func(expr,...) | Lines can be broken after any of: <code>\ , { && || ? :</code> (The last two only in BusyBox and gawk, and not in <code>gawk --posix</code>.) | ||
; function calls | |||
<var>func</var>(<var>expr</var>, ...) | |||
always evaluates to a value, so can be used as an expression, but also can occur in statement contexts. (Some awks like nawk ''don't'' allow arbitrary expressions to occur in statement contexts; but BusyBox awk does.) | |||
When calling user-defined functions, no space can separate the function name and the <code>(</code>. | |||
; assignments | |||
<var>lvalue</var> = <var>expr</var> | |||
<var>lvalue</var> += 1 (similarly for -= *= /= %= ^=) | |||
<var>lvalue</var> ++ (similarly for --) | |||
++ <var>lvalue</var> (similarly for --) | |||
<var>lvalue</var> can take any of the forms: | |||
* <code><var>var</var></code> | |||
* <code><var>arrayname</var>[<var>expr</var>]</code> | |||
* <code><var>arrayname</var>[<var>expr</var>, ...]</code> | |||
* <code>$<var>expr</var></code> | |||
Like function calls, assignments always evaluate to a value, so can be used as expressions, but also can occur in statement contexts, even in nawk. | |||
delete <var>arrayname</var>[<var>expr</var>] | |||
delete <var>arrayname</var>[<var>expr</var>, ...] | |||
delete <var>arrayname</var> | |||
The last form is not available in <code>gawk --traditional</code>. More portably, you can get the same effect with: | |||
split("", <var>arrayname</var>) | |||
After an array has been deleted, it no longer has any elements; however, the array name remains unavailable for use as a scalar variable. | |||
; control operators | |||
if (<var>test</var>) <var>action</var> | |||
if (<var>test</var>) <var>action</var>; else <var>action</var> | |||
if (<var>test</var>) <var>action</var>; else if (<var>test</var>) <var>action</var> | |||
if (<var>test</var>) <var>action</var>; else if (<var>test</var>) <var>action</var>; else <var>action</var> | |||
if (<var>test</var>) { ... } | |||
if (<var>test</var>) { ... } else <var>action</var> | |||
and so on. | |||
while (<var>test</var>) <var>action</var> | |||
... | |||
do <var>action</var> while (<var>test</var>) | |||
... | |||
The do...while form will execute <var>action</var> at least once. | |||
for (i=1; i<=NF; i++) <var>action</var> | |||
... | |||
for (k in <var>arrayname</var>) <var>action</var> | |||
... | |||
The last form has an unspecified iteration order. Also, the effect of adding or removing elements from an array while iterating over it is undefined. | |||
break | |||
continue | |||
Historic awk implementations interpreted <code>break</code> and <code>continue</code> outside of while/do/for-loops as <code>next</code>. BusyBox supports this usage; as do some versions of <code>gawk --traditional</code>. | |||
next | |||
This reads the next line of input and restarts the main loop from the first rule. For example, to treat the first line of awk's input specially, you can do this: | |||
NR==1 {...<var>handle first line</var>...; next } | |||
{...<var>handle other lines</var>...} | |||
POSIX doesn't specify the behavior of <code>next</code> in BEGIN or END blocks. | |||
nextfile | |||
This aborts processing the current file and starts processing the next file (if any) from the first rule. It is not available in <code>gawk --traditional</code>. | |||
return [<var>value</var>] | |||
<var>value</var> defaults to <code>""</code>. POSIX doesn't specify the behavior of a <code>return</code> statement outside of a function. | |||
exit <code>status</code> | |||
This will begin executing END rules, or exit immediately if invoked inside an END block. <var>status</var> defaults to <code>0</code>. | |||
In gawk and nawk, sequences like this: | |||
{ ... exit n } | |||
END { exit } | |||
will preserve the <code>n</code> status code. In BusyBox awk, the second exit will instead revert to the default of <code>0</code>. | |||
; printing | |||
Parentheses are optional for <code>print</code> and <code>printf</code>, but are mandatory for <code>sprintf</code>. Also, parentheses should be used if any of the arguments to <code>print</code> or <code>printf</code> contains a <code>></code>. | |||
Though <code>print</code> and <code>printf</code> accepts parenthesized argument lists, their invocations are statements not expressions. They return no value and cannot appear in non-statement contexts. (<code>sprintf</code> on the other hand is a function; and its invocations can appear in both statement and expression contexts.) | |||
The first of these: | |||
print "a" "b" | |||
print "a","b" | |||
invokes <code>print</code> with a single argument, which is the simple concatenation <code>"ab"</code> of the expressions <code>"a"</code> and <code>"b"</code>. On the other hand, the second form invokes <code>print</code> with two arguments. They will be separated in the output by the current value of OFS---which may, but need not, be "". The default value of OFS is a single space. | |||
print | |||
is interpreted the same as <code>print $0</code>. (Note also that a block of the form <code><var>pattern</var></code> with no <code>{...}</code> is interpreted as having the body <code>{ print $0 }</code>.) | |||
print "" | |||
print("") | |||
both print a newline. | |||
print (<var>expr</var>, ...) > "path" | |||
print (<var>expr</var>, ...) >> "path" | |||
print to the designated path. (See also [[#Built-in filenames]], below.) The output file is not closed until <code>close "path"</code> is executed, or awk exits. | |||
print (<var>expr</var>, ...) | "shell pipeline" | |||
prints to the designated pipeline. The pipeline is not closed until <code>close "shell pipeline"</code> is executed, or awk exits. | |||
printf ( <var>format</var>, <var>expr</var>, ... ) | |||
can be followed by <code>> "path"</code> or <code>>> "path"</code> or <code>| "shell pipeline"</code>, just like <code>print</code>. | |||
This usually implements a subset of the functonality of printf(3). The <var>format</var> string can contain: | |||
<ul><li> | |||
normal Awk string escape codes, such as: | |||
* <code>\b</code> (0x8) which backspaces one space | |||
* <code>\r</code> (0xd) which on Unix backspaces to the start of the current line | |||
* <code>\v</code> and <code>\f</code> (0xb and 0xc) which on Unix stay in the current column but go to the next screen line | |||
* <code>\n</code> and <code>\t</code> and <code>\"</code> and <code>\\</code> | |||
* <code>\a</code> (0x7) | |||
* <code>\c</code> which ignores the rest of the string (none of my awk implementations honored this) | |||
* <code>\000</code> for up to three octal digits | |||
Of the implementations I tried, only gawk would printf or sprintf <code>"\0"</code>; others would take it to terminate the string. (A different way to print <code>"\0"</code> is to use <code>printf "%c", 0</code>. This works with gawk's <code>printf</code> and <code>sprintf</code> and nawk's <code>printf</code>.) | |||
print | |||
printf | |||
<li> | |||
formatting codes, of the form: | |||
<pre> | |||
%[n$][#][-|0][+| ]['][minwidth][.precision][wordsize specifier][format specifier] | |||
</pre> | |||
where: | |||
* <code><var>n</var>$</code> means use argv[<var>n</var>] instead of the next argv in sequence. The count is 1-based, and only gawk honors this. | |||
* <code>#</code> forces octals to have an initial <code>0</code>, hex to have initial <code>0x</code>, floats to always have a decimal point. | |||
* <code>-</code> means align to the left | |||
* <code>0</code> means right-align as usual but pad with zeros instead of spaces | |||
* <code>+</code> forces the inclusion of a sign | |||
* <code> </code> leads with a sign if negative, or space if positive | |||
* <code>'</code> means write a million as <code>1,000,000</code>. None of my awks honor this. | |||
* <code><var>minwidth</var></code> or <code><var>precision</var></code> can be <code>*</code>, in which case the values are supplied by the next argv. Only gawk and nawk honor this. | |||
* <code><var>precision</var></code> gives the number of digits after the decimal for a number, or the maxwidth for a string. | |||
* <code><var>wordsize specifier</var></code> only nawk honors these, and then only: <code>hh h l ll</code> | |||
* <code><var>format specifier</var></code> can be any of the signed formats <code>fegdi</code> (the last two are equivalent), or any of the unsigned formats <code>xuo</code>, or any of the text formats <code>sc</code>. On BusyBox, the format <code>%c</code> only processes values up to 0x7fff, and the results are always masked to the range "\x00"..."\xff". | |||
</ | None of my awk implementations honored <code>%b</code> (which is like <code>echo -e</code>) or any other format. | ||
</ul> | |||
=== Expressions === | === Expressions === | ||
{{Draft|}} | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
Line 251: | Line 377: | ||
The expression `k in array` doesn't create an array entry, but the reference `array[k]` will create an entry with an uninitialized value. (`k in array` will then be true.) | The expression `k in array` doesn't create an array entry, but the reference `array[k]` will create an entry with an uninitialized value. (`k in array` | ||
will then be true.) | |||
References to fields > NF, on the other hand, don't create new fields. Assignments to these fields do create them (increasing NF and setting any intermediate fields to uninitialized values). Assignments to any field (even <= NF) cause $0 to be recomputed using OFS, but do not cause $0 to be reparsed. (So the modified/new field may contain embedded FS characters.) | References to fields > NF, on the other hand, don't create new fields. Assignments to these fields do create them (increasing NF and setting any | ||
intermediate fields to uninitialized values). Assignments to any field (even <= NF) cause $0 to be recomputed using OFS, but do not cause $0 to be | |||
reparsed. (So the modified/new field may contain embedded FS characters.) | |||
Assignments to $0 do cause NF, $1, ... to be recomputed. | Assignments to $0 do cause NF, $1, ... to be recomputed. | ||
Line 261: | Line 390: | ||
=== String vs numeric === | === String vs numeric === | ||
{{Draft|}} | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
Uninitialized values include unset variables and array elements, invalid fields (> NF) or valid fields of length 0, and unassigned function parms (which can be used as scalars or arrays). Unititalized scalars have value "", which math operators treat as 0. | Uninitialized values include unset variables and array elements, invalid fields (> NF) or valid fields of length 0, and unassigned function parms | ||
(which can be used as scalars or arrays). Unititalized scalars have value "", which math operators treat as 0. | |||
Can force string interpretaton by `var ""`, or force numeric interpretation by `var + 0`. "1text" is coerced as 1. | Can force string interpretaton by `var ""`, or force numeric interpretation by `var + 0`. "1text" is coerced as 1. | ||
In boolean contexts: false when expr evaluates to 0 or "", else true. | In boolean contexts: false when expr evaluates to 0 or "", else true. | ||
Line 297: | Line 429: | ||
=== Arrays === | === Arrays === | ||
{{Draft|}} | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
Line 311: | Line 445: | ||
</nowiki></div> | </nowiki></div> | ||
=== | === Built-in functions === | ||
{{Draft|}} | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
Line 338: | Line 474: | ||
# nawk and gawk support length without () | # nawk and gawk support length without () | ||
# nawk and gawk (but not gawk --posix) support length(array) | # nawk and gawk (but not gawk --posix) support length(array) | ||
gawk has length(A), returns count of keys rather than maxkey | |||
substr("string", 1-based-start, maxlength=until_end) ~~> "substring" | substr("string", 1-based-start, maxlength=until_end) ~~> "substring" | ||
index("haystack","needle") ~~> 1-based, or 0 if fails | index("haystack","needle") ~~> 1-based, or 0 if fails | ||
Line 371: | Line 509: | ||
close("file" or "pipe commnd") ~~> 0 if successful | close("file" or "pipe commnd") ~~> 0 if successful | ||
# awk doesn't automatically close files, pipes, sockets, or co-processes when they return EOF. | # awk doesn't automatically close files, pipes, sockets, or co-processes when they return EOF. | ||
# the return value of close() is unspecified; gawk uses value from fclose(3) or pclose(3), or -1 if the named file, piple, or co-process was not opened with a redirection. | # the return value of close() is unspecified; gawk uses value from fclose(3) or pclose(3), or -1 if the named file, piple, or co-process was not | ||
opened with a redirection. | |||
{ print ... | "sort>tmpfile" } END { close("sort>tmpfile"); while ((getline < "tmpfile")>0) ... close("tmpfile") } | { print ... | "sort>tmpfile" } END { close("sort>tmpfile"); while ((getline < "tmpfile")>0) ... close("tmpfile") } | ||
Line 382: | Line 521: | ||
</nowiki></div> | </nowiki></div> | ||
=== | === Built-in filenames === | ||
< | |||
Only gawk handles these filenames internally: | |||
* <code>"/dev/tty"</code> | |||
* <code>"/dev/stdin"</code> | |||
* <code>"/dev/stdout"</code> | |||
* <code>"/dev/stderr"</code> | |||
* <code>"/dev/fd/<var>n</var>"</code> | |||
but in many Unix systems they're available externally anyway. As an alternative to <code>getline < "/dev/stdin"</code>, you can also use <code>getline < "-"</code>. As an alternative to <code>print ... > "/dev/stderr"</code>, you can also use: <code>print ... | "cat 1>&2"</code>. | |||
=== Built-in variables === | |||
{{Draft|}} | |||
; ENVIRON | |||
: This is an associative array holding the current environment variables. The array is mutable, but POSIX does not specify whether changes to it must be visible to child processes spawned from awk. (In none of the implementations I checked were such changes visible.) | |||
; ARGC and ARGV | |||
: The number of command-line arguments to awk, and an array holding them. ARGV[0] will usually be "awk", and ARGV[1]...ARGV[ARGC-1] will be the arguments. Awk handles empty arguments specially, and also arguments of the form <code><var>var</var>=<var>value</var></code>. | |||
: If your script only has a BEGIN block, and no main-loop rules or END block, then awk won't attempt to automatically read or process the ARGV arguments, or stdin, in any way. | |||
: You can decrement ARGC; awk then won't process any arguments that have been lost. You can also set <code>ARGV[<var>n</var>] = ""</code>, or alternatively, <code>delete ARGV[<var>n</var>]</code>. Similarly, you can add additional ARGV entries, though if you do so, you need to increment ARGC manually. | |||
: Arguments that awk does process should not refer to directories or nonexisting files (unless you are going to to scan the ARGV list manually and remove such). Some versions of some awk implementations will warn if asked to process a directory; others will raise an error. | |||
; ARGIND | |||
: Indicates which ARGV entry was last processed by awk. This is present only in gawk and BusyBox awk. The variable is mutable; but changes to it have no side-effect in gawk. In BusyBox, changes to it affect which ARGV entry will be processed next, after the current one is finished. | |||
; FILENAME | |||
: Indicates the name of the ARGV entry currently being processed. When stdin is being processed, this will be <code>"-"</code>. | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
ERRNO In gawk and BusyBox, contains string error message after getline or close fails | ERRNO In gawk and BusyBox, contains string error message after getline or close fails | ||
NR Number of current record | NR Number of current record | ||
Line 435: | Line 587: | ||
POSIX requires that assigning a new value to FS has no effect on the current input line; it only affects the next input line. | POSIX requires that assigning a new value to FS has no effect on the current input line; it only affects the next input line. | ||
Only gawk conforms; BusyBox and FreeBSD awk will also use the new FS for current line if no fields have yet been referenced. | Only gawk conforms; BusyBox and FreeBSD awk will also use the new FS for current line if no fields have yet been referenced. | ||
FS=space (default): strips leading and trailing space/tabs, fields are separated by spans of space/tabs/newlines (in gawk --posix, only spans of spaces/tabs) | FS=space (default): strips leading and trailing space/tabs, fields are separated by spans of space/tabs/newlines (in gawk --posix, only spans of | ||
spaces/tabs) | |||
FS=":" or "\t": each occurrence of the char separates another field | FS=":" or "\t": each occurrence of the char separates another field | ||
FS="pat": separator is leftmost longest non-null and non-overlapping match of pattern | FS="pat": separator is leftmost longest non-null and non-overlapping match of pattern | ||
Line 462: | Line 615: | ||
== Gawk-only extensions == | == Gawk-only extensions == | ||
{{Draft|}} | |||
<div style="white-space:pre; font-family:monospace;"><nowiki> | <div style="white-space:pre; font-family:monospace;"><nowiki> | ||
Line 522: | Line 677: | ||
These links may also be of interest: | These links may also be of interest: | ||
* | * https://www.pement.org/awk/awk1line.txt | ||
* | * https://www.catonmat.net/series/awk-one-liners-explained | ||
* http://awk.freeshell.org/HomePage | * http://awk.freeshell.org/HomePage | ||
* http://awk.freeshell.org/AwkFeatureComparison | * http://awk.freeshell.org/AwkFeatureComparison | ||
Line 579: | Line 734: | ||
gsplit returns as iterseq, split as multivalue --> | gsplit returns as iterseq, split as multivalue --> | ||
|<code>split(str,ITEMS,[seppat],[gawk's SEPS])</code> --> nitems <br /> | |<code>split(str,ITEMS,[seppat],[gawk's SEPS])</code> --> nitems <br /> | ||
<code style="color:brown">gsplit(str,ITEMS,[seppat],[ | <code style="color:brown">gsplit(str,ITEMS,[seppat],[SEPS])</code> --> nitems <br /> | ||
<code style="color:brown">asplit(str, PAIRS, ["="], [" "])</code> --> nitems | <code style="color:brown">asplit(str, PAIRS, ["="], [" "])</code> --> nitems | ||
|- valign=top | |- valign=top | ||
Line 628: | Line 783: | ||
|} | |} | ||
< | <!-- | ||
Lua args ~-> table: table.pack | |||
Lua iterseq ~-> tables: table.[i]grab | |||
args | --> | ||
[[Category:Shell]] [[Category:Lua]] | [[Category:Shell]] [[Category:Lua]] |
Latest revision as of 07:42, 9 November 2024
This page compares BusyBox's implementation of awk (using this config file) with gawk (versions >= 3.1.8) and FreeBSD 9's nawk, which is based on Bell Labs/Brian Kernighan's 2007 version of awk.
It's not intended as a tutorial, but rather as a technical summary and list of "gotchas" (places where different implementations may behave in different or unexpected ways).
Invoking awk
awk [-F char_or_regex] [-v var=value ...] [--] 'awk script...' [ ARGV[1] ... ARGV[ARGC-1] ] awk [-F char_or_regex] [-v var=value ...] -f scriptfile ... [--] [ ARGV[1] ... ARGV[ARGC-1] ]
Gawk makes a third invocation pattern possible, which mixes -f scriptfile
options with an 'awk script...'
on the command-line:
gawk [-F char_or_regex] [-v var=value ...] -f scriptfile ... -e 'awk script...' [--] [ ARGV[1] ... ARGV[ARGC-1] ]
Awkenough is a small set of Awk utility routines and a C stub that makes it easier to write shell scripts with awk shebang lines. (I'll make an Alpine package for this soon.) It also permits mixing -f scriptfile
and command-line scripts, using the same -e
option gawk provides. In both cases, the long-form option --source
works the same as -e
.
- Notes
- all implementations honor
\t
and so on in-v FS=expr
. BusyBox doesn't honor\t
in-F expr
; perhaps this is a bug. You can use the$'\t'
escapes of BusyBox ash to work around this.
nawk
andgawk --traditional
(but notgawk --posix
) interpret-F t
as-F '\t'
. This is a weird special-case preserved for historical compatibility.
- scriptfile can be
-
for stdin
- ARGVs can be of any form, but awk only knows how to automatically process arguments whose form is:
var=unquoted_value
: assignments will be made when the main loop reaches that ARGV""
: will be skippedfilename
-
: will use stdin
- If no files are processed (nor
-
), stdin will be processed after all command-line assignments.
- standalone scripts should look like this:
Contents of foo.awk
#!/usr/bin/awk -f # will receive the script's ARGC, ARGV with ARGV[0]=awk BEGIN { ... } ...- or like this:
Contents of bar.awk
#!/usr/bin/runawk -f /usr/share/awkenough/library.awk ...
- gawk places any unrecognized short options that precede
--
into ARGV; other implementations (andgawk --traditional
) fail/warn
- some gawk-only options:
--lint
: warns about nonportable constructs--re-interval
: enables/pat{1,3}/
regexes; some versions of gawk enable by default, others don't--traditional
or--compat
: makes gawk behave like nawk--posix
: like--traditional --re-interval
, with some further restrictions--exec scriptfile
: stops option processing and disables any further-v
s; meant to be used in shebang line for scripts in untrusted environments (cgi)--sandbox
: disablesystem("foo")
,"foo"|getline
,getline <"file"
,print|"foo"
,print >"file"
, and gawk'sextension(obj, func)
; so only local resources explicitly specified in ARGV will be accessible. This feature is only available on recent versions of gawk.
- when handling
-f path
options where path contains no/
s, gawk will first search in directories specified in an AWKPATH environment variable
Awk grammar
Blocks
- function definitions
- Functions have global scope, and they can be called from positions that syntactically precede their definitions.
- Scalars (that is, non-arrays) are passed by value, arrays are passed by reference.
- Function calls can be nested or recursive. Each invocation of the function has its own local environment: so upon return, values of the scalar parameters the caller supplied shall be unchanged (array parameters may have been mutated).
-
Consider functions defined like this:
function foo(x,y) { ... bar() return "result" } function bar(z) { y += 1 return }
Here the variables
x
andy
are local to foo, and the variablez
is local to bar. All other variable references are global: contrast the shell, where non-local variable references are instead dynamic. That is, in the shell, when foo calls bar, bar will be incrementing foo's local variabley
. In awk, on the other hand, bar always only increments a global variabley
.Note also that the number of arguments in a function call needn't match the number of parameters in the function's definition; if more arguments are supplied, they will be ignored; if fewer are supplied (as in foo's call of bar), the missing variables are assigned either "" or an empty array, depending on how they're used in the function.
Note also the the
return
statement needn't be given an explicit return value (it will default to ""). Thereturn
statement can also be omitted altogether.
- BEGIN and END
BEGIN { ... } ... END { ... } ...
- Multiple BEGIN blocks are merged; so too multiple END blocks.
- If there are only BEGIN blocks (no main loop or END blocks), and getline isn't used, then POSIX requires awk to terminate without reading stdin or any file operands. The implementations I checked honor this, but some historical implementations discarded input in this case. For portability, you can explicitly append
< /dev/null
to an invocation of awk you know shouldn't consume stdin.
- If there is a main loop or END blocks, and awk's processing of stdin hasn't been suppressed by encountering files in its ARGV list, then stdin will be processed before executing any END block(s).
- If there are END block(s), but
exit status
is invoked outside them, then the main loop stops and control passes immediately to the first END block. Ifexit status
is invoked within an END block, or if there are no end blocks, the script terminates with result status.
- POSIX requires that inside END, all of
FILENAME NR FNR NF
retain their most-recent values, so long asgetline
is not invoked. My implementations comply, and also retain the most-recent fields$0 ...
, but earlier versions of nawk did not; and Darwin's awk may not.
- main loop blocks
These are of the form:
pattern { action ... }
or:
pattern,pattern { action ... }
where action defaults to print $0
; and pattern defaults to 1
(that is, to true).
-
Patterns can take any of the forms:
/regex/
relational
pattern && pattern
pattern || pattern
! pattern
( pattern )
pattern ? pattern : pattern
$0 ~ /regex/
. -
Relationals can take any of the forms:
expr ~ RE
expr !~ RE
expr eqop expr
expr in arrayname
(expr, ...) in arrayname
expr
/a[bc]+/
or any expression that evaluates to a string, like"a[bc]+"
. Note that strings require an extra level of\
s: you can write/\w+/
or"\\w+"
. The scemas eqop can be any of:== != <= < > >=
. A bare expression, as in the last form, is interpreted as false when the expression evaluates to0
or""
, else to true.
- other block forms
BEGINFILE { ... } ENDFILE { ... }
These are only available in (recent versions of) gawk.
@include "filename"
This is only available in some versions of gawk. See also Aleksey Cheusov's Runawk, which uses a different approach.
Statements
Lines can be broken after any of: \ , { && || ? :
(The last two only in BusyBox and gawk, and not in gawk --posix
.)
- function calls
func(expr, ...)
always evaluates to a value, so can be used as an expression, but also can occur in statement contexts. (Some awks like nawk don't allow arbitrary expressions to occur in statement contexts; but BusyBox awk does.)
When calling user-defined functions, no space can separate the function name and the (
.
- assignments
lvalue = expr lvalue += 1 (similarly for -= *= /= %= ^=) lvalue ++ (similarly for --) ++ lvalue (similarly for --)
lvalue can take any of the forms:
var
arrayname[expr]
arrayname[expr, ...]
$expr
Like function calls, assignments always evaluate to a value, so can be used as expressions, but also can occur in statement contexts, even in nawk.
delete arrayname[expr] delete arrayname[expr, ...] delete arrayname
The last form is not available in gawk --traditional
. More portably, you can get the same effect with:
split("", arrayname)
After an array has been deleted, it no longer has any elements; however, the array name remains unavailable for use as a scalar variable.
- control operators
if (test) action if (test) action; else action if (test) action; else if (test) action if (test) action; else if (test) action; else action if (test) { ... } if (test) { ... } else action
and so on.
while (test) action ... do action while (test) ...
The do...while form will execute action at least once.
for (i=1; i<=NF; i++) action ... for (k in arrayname) action ...
The last form has an unspecified iteration order. Also, the effect of adding or removing elements from an array while iterating over it is undefined.
break continue
Historic awk implementations interpreted break
and continue
outside of while/do/for-loops as next
. BusyBox supports this usage; as do some versions of gawk --traditional
.
next
This reads the next line of input and restarts the main loop from the first rule. For example, to treat the first line of awk's input specially, you can do this:
NR==1 {...handle first line...; next } {...handle other lines...}
POSIX doesn't specify the behavior of next
in BEGIN or END blocks.
nextfile
This aborts processing the current file and starts processing the next file (if any) from the first rule. It is not available in gawk --traditional
.
return [value]
value defaults to ""
. POSIX doesn't specify the behavior of a return
statement outside of a function.
exit status
This will begin executing END rules, or exit immediately if invoked inside an END block. status defaults to 0
.
In gawk and nawk, sequences like this:
{ ... exit n } END { exit }
will preserve the n
status code. In BusyBox awk, the second exit will instead revert to the default of 0
.
- printing
Parentheses are optional for print
and printf
, but are mandatory for sprintf
. Also, parentheses should be used if any of the arguments to print
or printf
contains a >
.
Though print
and printf
accepts parenthesized argument lists, their invocations are statements not expressions. They return no value and cannot appear in non-statement contexts. (sprintf
on the other hand is a function; and its invocations can appear in both statement and expression contexts.)
The first of these:
print "a" "b" print "a","b"
invokes print
with a single argument, which is the simple concatenation "ab"
of the expressions "a"
and "b"
. On the other hand, the second form invokes print
with two arguments. They will be separated in the output by the current value of OFS---which may, but need not, be "". The default value of OFS is a single space.
is interpreted the same as print $0
. (Note also that a block of the form pattern
with no {...}
is interpreted as having the body { print $0 }
.)
print "" print("")
both print a newline.
print (expr, ...) > "path" print (expr, ...) >> "path"
print to the designated path. (See also #Built-in filenames, below.) The output file is not closed until close "path"
is executed, or awk exits.
print (expr, ...) | "shell pipeline"
prints to the designated pipeline. The pipeline is not closed until close "shell pipeline"
is executed, or awk exits.
printf ( format, expr, ... )
can be followed by > "path"
or >> "path"
or | "shell pipeline"
, just like print
.
This usually implements a subset of the functonality of printf(3). The format string can contain:
-
normal Awk string escape codes, such as:
\b
(0x8) which backspaces one space\r
(0xd) which on Unix backspaces to the start of the current line\v
and\f
(0xb and 0xc) which on Unix stay in the current column but go to the next screen line\n
and\t
and\"
and\\
\a
(0x7)\c
which ignores the rest of the string (none of my awk implementations honored this)\000
for up to three octal digits
"\0"
; others would take it to terminate the string. (A different way to print"\0"
is to useprintf "%c", 0
. This works with gawk'sprintf
andsprintf
and nawk'sprintf
.) -
formatting codes, of the form:
%[n$][#][-|0][+| ]['][minwidth][.precision][wordsize specifier][format specifier]
where:
n$
means use argv[n] instead of the next argv in sequence. The count is 1-based, and only gawk honors this.#
forces octals to have an initial0
, hex to have initial0x
, floats to always have a decimal point.-
means align to the left0
means right-align as usual but pad with zeros instead of spaces+
forces the inclusion of a sign'
means write a million as1,000,000
. None of my awks honor this.minwidth
orprecision
can be*
, in which case the values are supplied by the next argv. Only gawk and nawk honor this.precision
gives the number of digits after the decimal for a number, or the maxwidth for a string.wordsize specifier
only nawk honors these, and then only:hh h l ll
format specifier
can be any of the signed formatsfegdi
(the last two are equivalent), or any of the unsigned formatsxuo
, or any of the text formatssc
. On BusyBox, the format%c
only processes values up to 0x7fff, and the results are always masked to the range "\x00"..."\xff".
None of my awk implementations honored
%b
(which is likeecho -e
) or any other format.
Expressions
This material is work-in-progress ...
|
String vs numeric
This material is work-in-progress ...
|
Arrays
This material is work-in-progress ...
|
Built-in functions
This material is work-in-progress ...
|
Built-in filenames
Only gawk handles these filenames internally:
"/dev/tty"
"/dev/stdin"
"/dev/stdout"
"/dev/stderr"
"/dev/fd/n"
but in many Unix systems they're available externally anyway. As an alternative to getline < "/dev/stdin"
, you can also use getline < "-"
. As an alternative to print ... > "/dev/stderr"
, you can also use: print ... | "cat 1>&2"
.
Built-in variables
This material is work-in-progress ...
|
- ENVIRON
- This is an associative array holding the current environment variables. The array is mutable, but POSIX does not specify whether changes to it must be visible to child processes spawned from awk. (In none of the implementations I checked were such changes visible.)
- ARGC and ARGV
- The number of command-line arguments to awk, and an array holding them. ARGV[0] will usually be "awk", and ARGV[1]...ARGV[ARGC-1] will be the arguments. Awk handles empty arguments specially, and also arguments of the form
var=value
.
- If your script only has a BEGIN block, and no main-loop rules or END block, then awk won't attempt to automatically read or process the ARGV arguments, or stdin, in any way.
- You can decrement ARGC; awk then won't process any arguments that have been lost. You can also set
ARGV[n] = ""
, or alternatively,delete ARGV[n]
. Similarly, you can add additional ARGV entries, though if you do so, you need to increment ARGC manually.
- Arguments that awk does process should not refer to directories or nonexisting files (unless you are going to to scan the ARGV list manually and remove such). Some versions of some awk implementations will warn if asked to process a directory; others will raise an error.
- ARGIND
- Indicates which ARGV entry was last processed by awk. This is present only in gawk and BusyBox awk. The variable is mutable; but changes to it have no side-effect in gawk. In BusyBox, changes to it affect which ARGV entry will be processed next, after the current one is finished.
- FILENAME
- Indicates the name of the ARGV entry currently being processed. When stdin is being processed, this will be
"-"
.
Gawk-only extensions
This material is work-in-progress ...
|
Useful links
These links may also be of interest:
- https://www.pement.org/awk/awk1line.txt
- https://www.catonmat.net/series/awk-one-liners-explained
- http://awk.freeshell.org/HomePage
- http://awk.freeshell.org/AwkFeatureComparison
Awk and Lua
See summary of Lua regex.
Commands in brown are from awkenough.
in lua | in awk |
---|---|
string.find(str, pattern, [startpos]) --> (start,stop) or nil
|
match(str, pat) --> returns and sets RSTART, also sets RLENGTH
|
string.match(str, pattern, [start=1]) --> (%0) or nil or (%1,%2,...)
|
matchstr(str, pat, [nth=1]) --> \\0, sets RSTART and RLENGTH gawk's |
string.gmatch(str, pattern) --> an iterator over all matches or sets of matching groups The iteration sequence will look like: (%0), (%0), (%0) ...; or like: (%1,%2...), (%1,%2...),... |
gmatch(str, pat, MATCHES, STARTS) --> nmatches
|
string.gsub(str, pattern, replacement, [max#repls]) --> (newstr,nrepls)
|
gensub(pat, repl, nth/"g", str) : is closest to Lua's gsub
|
string:len
|
length(str)
|
string:lower
|
tolower
|
string.rep(str, count,[5.2 adds sep])
|
rep(str,count,[sep])
|
string.sub(str, start,[stop])
|
substr(str,start,[len])
|
split(str,ITEMS,[seppat],[gawk's SEPS]) --> nitems
| |
table.concat(tbl,[sep],[start],[stop]) --> string
|
concat([start=1], [len=to_end], [fs=OFS], [A]) --> string if you want to preserve existing FS, need to use |
string:reverse
|
reverse([A])
|
table.remove(tbl, [pos=from end]) --> value formerly at tbl[pos]
|
pop([start=from_end], [len=to_end], [A]) --> values sep by SUBSEP
|
table.insert(tbl,[valpos=insert at end],value) --> nil
|
insert(value, [start=after_end], [A]) --> new length of array
|
table.sort(tbl, [lessthan])
|
sort(A)
|
isempty(A)
| |
includes(A, B, [onlykeys?]) : is B <= A?
|