This first edition was written for Lua 5.0. While still largely relevant for later versions, there are some differences.
The fourth edition targets Lua 5.3 and is available at Amazon and other bookstores.
By buying the book, you also help to support the Lua project.

7.2 – The Semantics of the Generic for

One drawback of those previous iterators is that we need to create a new closure for each new loop. For most situations, this is not a real problem. For instance, in the allwords iterator, the cost of creating one single closure is negligible compared to the cost of reading a whole file. However, in a few situations this overhead can be undesirable. In such cases, we can use the generic for itself to keep the iteration state.

We saw that the generic for keeps the iterator function internally, during the loop. Actually, it keeps three values: The iterator function, an invariant state, and a control variable. Let us see the details now.

The syntax for the generic for is as follows:

    for <var-list> in <exp-list> do
where <var-list> is a list of one or more variable names, separated by commas, and <exp-list> is a list of one or more expressions, also separated by commas. More often than not, the expression list has only one element, a call to an iterator factory. For instance, in the code
    for k, v in pairs(t) do
      print(k, v)
the list of variables is k, v; the list of expressions has the single element pairs(t). Often the list of variables has only one variable too, as in
    for line in io.lines() do
      io.write(line, '\n')
We call the first variable in the list the control variable. Its value is never nil during the loop, because when it becomes nil the loop ends.

The first thing the for does is to evaluate the expressions after the in. These expressions should result in the three values kept by the for: the iterator function, the invariant state, and the initial value for the control variable. Like in a multiple assignment, only the last (or the only) element of the list can result in more than one value; and the number of values is adjusted to three, extra values being discarded or nils added as needed. (When we use simple iterators, the factory returns only the iterator function, so the invariant state and the control variable get nil.)

After this initialization step, the for calls the iterator function with two arguments: the invariant state and the control variable. (Notice that, for the for structure, the invariant state has no meaning at all. It only gets this value from the initialization step and passes it when it calls the iterator function.) Then the for assigns the values returned by the iterator function to variables declared by its variable list. If the first value returned (the one assigned to the control variable) is nil, the loop terminates. Otherwise, the for executes its body and calls the iteration function again, repeating the process.

More precisely, a construction like

    for var_1, ..., var_n in explist do block end
is equivalent to the following code:
      local _f, _s, _var = explist
      while true do
        local var_1, ... , var_n = _f(_s, _var)
        _var = var_1
        if _var == nil then break end
So, if our iterator function is f, the invariant state is s, and the initial value for the control variable is a0, the control variable will loop over the values a1 = f(s, a0), a2 = f(s, a1), and so on, until ai is nil. If the for has other variables, they simply get the extra values returned by each call to f.