Technical Note 11

Require revisited: Import

This LTN depends on "loadfile," introduced in Lua 5.0

Abstract

Lua 4.1 introduced the "require" function that loads and runs a file unless it already loaded. Lua 5.0 offers require as a built-in function in its base lib. The require command together with LTN 7 "Modules & packages" offers a basis for simple module support in Lua. This technical note proposes an improved version of require, dubbed "import." The proposed import scheme avoids direct access to the globals, corrects a globals related security loophole and handles cyclic module dependencies gracefully.

The problem

The module approach of LTN 7 proposes that a package should publish its public interface (wrapped in a table) in the globals table. This has the following drawbacks:

The global name used by the package might already hold data (a name clash.)
Users of the package must know to which global name the package assigns its interface.
Metamethods set on the globals table might interfere with the loading of a module.

The current implementation of require has also some shortcomings:

Require relies on the LTN 7 system of global public interfaces as described above and provides no further package management.
The require call in Lua 5.0 passes its own table of globals on to the required package, thereby offering the package a severe security loophole. Because require is a built-in C function, a "setglobals" call does not apply to require to prevent this.
If modules require each other (i.e. are cyclically dependendent) then a require call will recurse indefinitely, resulting in a stack overflow.

The solution

The proposed import scheme addresses the problems posed by global package names, globals security loopholes and cyclic dependencies. Import can be completely implemented in vanilla Lua 5. The main points:

A package returns a "package install" function (PIF) that is in turn called by import.
A table is passed to the PIF into which the package's public interface should be inserted. This table is then returned as the result of the import call. A package should no longer install a global interface.
The package name and the full package path are passed as the second and third parameter to the PIF.
Import imposes its caller's globals on the imported package.
Import will report an error if a package is used before it is fully imported. This could happen during import of cyclic dependent packages. (A package is "used" if its public interface is accessed.) Packages can be cyclically dependendent without using each other during import. In this case import will succeed without error. (An example is given in section Explanation below.)

The import function could be implemented with the following Lua 5.0 code.

local imported = {}

local function package_stub(name)
  local stub = {}
  local stub_meta = {
    __index = function(_, index)
      error(string.format("member `%s' is accessed before package `%s' is fully imported", index, name))
    end,
    __newindex = function(_, index, _)
      error(string.format("member `%s' is assigned a value before package `%s' is fully imported", index, name))
    end,
  }
  setmetatable(stub, stub_meta)
  return stub
end

local function locate(name)
  local path = LUA_PATH
  if type(path) ~= "string" then
    path = os.getenv "LUA_PATH" or "./?.lua"
  end
  for path in string.gfind(path, "[^;]+") do
    path = string.gsub(path, "?", name)
    local chunk = loadfile(path)
    if chunk then return chunk, path end
  end
  return nil, path
end

function import(name)
  local package = imported[name]
  if package then return package end
  local chunk, path = locate(name)
  if not chunk then
    error(string.format("could not locate package `%s' in `%s'", name, path))
  end
  package = package_stub(name)
  imported[name] = package
  setglobals(chunk, getglobals(2))
  chunk = chunk()
  setmetatable(package, nil)
  if type(chunk) == "function" then
    chunk(package, name, path)
  end
  return package
end

Typical use of import is as follows:

-- import the complex package
local complex = import "complex"

-- complex now holds the public interface
local x = 5 + 3*complex.I

A package should be structured as follows:

-- first import all other required packages.
local a = import "a"
local b = import "b"

-- then define the package install function.
-- the PIF more or less contains the code of a
-- LTN 7 package.
local function pif(Public, path)

local Private = {}

function Public.fun()
  -- public function
end

-- etc.
end

-- return the package install function
return pif

Explanation

Setting a "package stub" just before the package is loaded must trap any access to the stub (invoked by a nested import.) In order for this to work, additional imports should be placed in the global scope of each package involved, typically as the first calls. Note that the stub (stripped from its access restrictions) will later hold the package's public interface. In particular it is safe to refer to an imported interface (e.g. through upvalues) even in cyclic dependencies, as long as the interface is not actually accessed.

Import is almost backward compatible with require. Import will however not define the _REQUIREDNAME global during loading. An "old style" package that does not return a PIF will still be loaded and run but import returns an empty public interface. This will not impact old style code because require has no return values.

Here is an example of two packages mutually importing each other. Because neither one actually uses the other during import, this will not be a problem.

Package "a.lua":

local b = import "b"

local function pif(pub, name, path)

function pub.show()
  -- use a message from package b
  print("in " .. name .. ": " .. b.message)
end

pub.message = "this is package " .. name .. " at " .. path

end

return pif

Package "b.lua":

local a = import "a"

local function pif(pub, name, path)

function pub.show()
  -- use a message from package a
  print("in " .. name .. ": " .. a.message)
end

pub.message = "this is package " .. name .. " at " .. path

end

return pif

And some code importing and running both:

local a = import "a"
local b = import "b"

a.show() -- prints "in a: this is package b at ./b.lua"
b.show() -- prints "in b: this is package a at ./a.lua"

Weaknesses

The import function assumes that the packages it imports are "well-behaved." A package can of course still access and update the globals so care should be taken. Proper structuring of a package (import calls in its global scope, return a PIF, etc.) is not enforced.

Conclusion

The require function has proved itself to be very useful. The proposed import scheme builds on this success. It provides more controlled package visibility and supports cyclic dependencies whenever possible. The import functionality is lightweight and can be completely defined in vanilla Lua 5.

Last update: Wed Feb 19 09:25:05 EST 2003