Originally Published: Wednesday, 25 July 2001 Author: Ken McDonald and Darryl Harms
Published to: develop_articles/Development Articles Page: 3/4 - [Printable]

Chapter 11, The Quick Python Book: Modules and Scoping rules

Linux.com is proud to feature the following article on Python modules and scooping from the book "The Quick Python Book", published by Manning Press. Python, many people say is one of the most elegant programming languages in existence: read on to find out why. Also don't miss Linux.com's companion Live chat with Ken McDonald and Darryl Harms today at 11:00 am PDT on IRC in channel #live on irc.openprojects.net.

Module Locations  << Page 3 of 4  >>

11. 4 The module search path

Exactly where Python looks for modules is defined in a variable called path, which is accessible to the programmer through a module called sys. Do the following:
>>> import sys 
>>> sys. path 
==> _list of directories in the search path_ 

The value shown in place of where I've said ...list of directories in the search path... will depend on the configuration of your system. Regardless of the details, the string indicates a list of directories that are searched by Python (in order), when attempting to execute an import statement. The first module found which satisfies the import request is used. If there is no satisfactory module in the module search path, an ImportError exception is raised.

If you are using IDLE, you can graphically look at the search path and the modules on it using the Path Browser window, which you can start from File menu of the Python Shell window.

The sys. path variable is initialized from the value of the environment (operating system) variable PYTHONPATH, if it exists, or from a default value which is dependent on your installation. In addition, whenever a Python script is run, the sys.path variable for that script will have the directory containing the script inserted as its first element-this provides a convenient way of determining where the executing Python program is located. In an interactive session such as the one just above, the first element of sys.path will be set to the empty string, which Python takes as meaning that it should first look for modules in the current directory.

11. 4.1 Where to place your own modules

In the example that started this chapter, the mymath module was accessible to Python because: (1) when you execute Python interactively, the first element of sys. path is "", telling Python to look for modules in the current directory; and (2) you were executing Python in the directory which contained the mymath.py file. In a production program, neither of these conditions will typically be true. You will not be running Python interactively, and Python code files will not be located in your current directory. In order to ensure that modules coded by you can be used by your programs, you need to do one of the following:
  • Place your modules into one of the directories that Python normally searches for modules.
  • Place all of the modules used by a Python program into the same directory as the program.
  • Create a directory (or directories) which will hold your modules, and modify the sys.path variable so that it includes this new directory.

Of these three options, the first is apparently the easiest, and is also an option that should never be chosen because it can cause trouble. For example, what if you place new modules in a standard Python directory, and then you or someone else installs a new version of Python on top of that? Your modules would disappear and your programs would stop working, even if a new installation were done more carefully and you still had the old directory with your modules. You would still have to remember which ones are yours and copy them to their new residence.

Note that it's possible that your version of Python includes local code directories in its default module search path. Such directories are specifically intended for site-specific code, and are not in danger of being overwritten by a new Python install, because they are not part of the Python installation. If your sys.path refers to such directories, put your modules there.

The second option is a good choice for modules that are associated with a particular program. Just keep them with the program.

The third option is the right choice for site-specific modules that will be used in more than one program at that site. You can modify sys.path in various ways. You can assign to it in your code, which is easy, but hard-codes directory locations right into your program code; you can set the PYTHONPATH environment variable, which is relatively easy, but may not apply to all users at your site; or you can add to the default search path using using a .pth file.

See the section on environment variables in the appendix for examples of how to set PYTHONPATH. The directory or directories you set it to are prepended to the sys.path variable. If you use it be careful that you do not define a module with the same name as one of the existing library modules that you are using or is being used for you. Your module will be found before the library module. In some cases this may be what you want, but probably not often.

You can avoid this issue using the .pth method. In this case, the directory or directories you added will be appended to sys.path. The last of these mechanisms is best illustrated by a quick example. On Windows you can place this in the directory pointed to by sys.prefix. Assume your sys.prefix is c:\program files\python, and you place the following file in that directory .

File myModules.pth

c:\My Documents\python\modules 

Then the next time a Python interpreter is started, sys. path will have c:\program files\python\mymodules and c:\My Documents\python\modules added to it, if they exist. You can now place your modules in these directories. Note that the mymodules directory still runs the danger of being overwritten with a new installation. The modules directory is safer. You also may have to move or create a mymodules. pth file when you upgrade Python. See the description of the site module in the Python Library Reference if you want more details on using .pth files

11. 5 Private names in modules

We mentioned that you could say from module import * to import almost all names from a module. The exception to this is that names in the module beginning with an underscore cannot be imported in this manner. The primary use of this intentional feature in Python is to permit people to write modules which are intended for importation with from module import *. By leading off all internal names (i. e., names which should not be accessed outside the module) with an underscore, the module writer can ensure that from module import * brings in only those names which the user will want to access.

To see this in action let's assume we have a file called modtest. py, containing the following code:

File modtest. Py
""" modtest: our test module""" 
def f( x): 
return x
def _g( x): 
    return x 
a = 4 
_b = 2 

Now, start up an interactive session, and try the following:

>>> from modtest import * 
>>> f(3) 
==> 3 
>>> _g(3) 
Traceback (innermost last): 
  File "<stdin>", line 1, in ? 
NameError: _g 
>>> a
==> 4
>>> _b 
Traceback (innermost last): 
  File "< stdin>", line 1, in ? 
NameError: _b 

As you can see, the names f and a were imported, but the names _g and _b remain hidden outside of modtest. Note that this behavior occurs only with from ... import *. We can do the following to access _g or _b:


>>> import modtest 
>>> modtest._b 
==> 2 
>>> from modtest import _g
>>> _g(5) 
==> 5

The convention of leading underscores to indicate private names is used throughout Python, and not just in modules. You'll encounter it in classes and packages, later in the book.

11. 6 Library and third-party modules

It was mentioned at the beginning of this chapter that the standard Python distribution is itself split into modules, to make it more manageable. Once Python has been installed, all of the functionality in these library modules is available to the Python programmer. All that is needed is to import the appropriate modules, functions, classes, and so forth explicitly, before using them.

Many of the most common and useful standard modules are discussed throughout this book. However, the standard Python distribution includes far more than what this book describes. At the very least, you should browse through the table of contents of the Python Library Reference.

In IDLE you can also easily browse to and look at those that are written in Python using the Path Browser window. You can also search for example code which uses them with the Find in Files dialog, which can be opened from the Edit menu of the Python Shell window. You can search through your own modules as well in this way.

Available third-party modules, and links to them, are identified on the Python home page. These simply need to be downloaded and placed in a directory in your module search path in order to make them available for import into your programs.

Module Locations  << Page 3 of 4  >>