"""
:mod:`disco.worker.modutil` -- Parse and find module dependencies
=========================================================================

.. versionadded:: 0.2.3

This module provides helper functions to be used with the ``required_modules``
parameter in :class:`~disco.worker.classic.worker.Worker`.
These functions are needed when
your job functions depend on external Python modules and the default value
for ``required_modules`` does not suffice.

By default, Disco tries to find out which modules are required by job functions
automatically. If the found modules are not included in the Python standard library
or other package that is installed system-wide, it sends them to nodes so they
can be used by the Disco worker process.

Sometimes Disco may fail to detect all required modules. In this case,
you can override the default value either by providing a list of requirements manually,
or by generating the list semi-automatically using the functions in this module.

.. _modspec:

How to specify required modules
-------------------------------

The ``required_modules`` parameter accepts a list of module definitions.
A module definition may be either a module name, e.g. ``"PIL.Image"``,
or a tuple that specifies both the module name and its path,
e.g. ``("mymodule", "lib/mymodule.py")``.
In the former case,
the :class:`disco.worker.classic.worker.Worker` only imports the module,
assuming that it has been previously installed to the node.
In the latter case,
Disco sends the module file to nodes before importing it,
and no pre-installation is required.

For instance, the following is a valid list for
``required_modules``::

        required_modules = ["math", "random", ("mymodule", "lib/mymodule.py")]

This expression imports the standard modules ``math`` and ``random`` and sends
a custom module ``lib/mymodule.py`` to nodes before importing it.

Note that Disco sends only the files that can be found in your `PYTHONPATH`. It
is assumed that files outside ``PYTHONPATH`` belong either to the Python standard
library or to another package that is installed system-wide. Make sure
that all modules that require automatic distribution can be found in your `PYTHONPATH`.

Automatic distribution works only for individual modules and not
for packages nor modules that require a specific directory hierarchy. You need
to install packages and modules with special requirements manually to your nodes.

Typical use cases
-----------------

The following list describes some typical use cases for ``required_modules``.
The list helps you decide when to use the :func:`find_modules` and
:func:`locate_modules` functions.

 - (**default**) If you want to find and send all required modules used by
   your job functions recursively (i.e. also modules that depend on other modules are included),
   you don't need to specify ``required_modules`` at all. This equals to::

        required_modules = modutil.find_modules(job_functions)

   where *job_functions* is a list of all job functions: *map*, *map_init*,
   *combiner* etc.

 - If you want to find and import all required modules, but not send them, or
   you want to disable recursive analysis, use :func:`find_modules`
   explicitly with the ``send_modules`` and ``recursive`` parameters.

 - If you want to send a known set of modules (possible recursively) but you
   don't know their paths, use :func:`locate_modules`.

 - If you want to send a known set of modules. provide a list of *(module name,
   module path)* tuples.

 - If you just want to import specific modules, or sub-modules in a
   pre-installed package (e.g. ``PIL.Image``), provide a list of module names.

Any combinations of the above are allowed. For instance::

        required_modules = find_modules([fun_map]) + [("mymodule", "/tmp/mymodule.py"), "time", "random"]

is a valid expression.

Functions
---------
"""
import sys, os
import imp, modulefinder, struct, dis
import functools
from os.path import abspath, dirname

from disco.compat import bytes_of_int, int_of_byte
from disco.error import DiscoError

class ModUtilImportError(DiscoError, ImportError):
    """Error raised when a module can't be found by :mod:`disco.worker.modutil`."""
    def __init__(self, error, function):
        self.error    = error
        self.function = function

    def __str__(self):
        # XXX! Add module name below
        return ("{0}: Could not find module defined in {1}. Maybe it is a typo. "
                "See documentation of the required_modules parameter for details "
                "on how to include modules.".format(self.error, self.function.__name__))

def user_paths():