Compiler Hacker's Guide to the new makelib
========================================

Last change: 12/5/2000

* Libraries
-----------

The new way of building the compiler is heavily library-oriented.
Aside from a tiny portion of code that is responsible for defining the
pervasive environment, _everything_ lives in libraries.  Building the
compiler means compiling and building these libraries first.  Some
of the libraries exist just for reasons of organizing the code,
others are potentially useful in their own right.  Therefore, as a
beneficial side-effect of compiling the compiler, you will end up with
compiled versions of these libraries.

At the moment, the following libraries are constructed when compiling
the compiler ("*" means that I consider the library potentially useful
in its own right):

  * $ROOT/src/lib/std/standard.lib			Default library (pre-loaded)
  * $ROOT/src/lib/html/html.lib    	        HTML Library
  * $ROOT/src/lib/prettyprint/big/prettyprinter.lib	Prettyprinting library

  * $ROOT/src/app/yacc/lib/mythryl-yacc-lib.lib	Mythryl-Yacc runtime library

  * $ROOT/src/lib/core/compiler/{pwrpc32,sparc32,intel32}.lib
					cross-compiler libraries, exporting
					package <Arch>Compiler
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib	package compiler (current arch)
  * $ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib		all cross-compilers and all cross-CMBs

  * $ROOT/src/lib/core/makelib/full.lib		package makelib (see manual)
  * $ROOT/src/lib/core/makelib/tools.lib		makelib tools library

  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-pwrpc32-posix.lib
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-sparc32-posix.lib
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-intel32-posix.lib
					cross-bootstrap-compilers for Unix
					(package <Arch>UnixCMB)
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-pwrpc32-macos.lib		...for Mac (package pwrpc32MacosCMB)
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-intel32-win32.lib		...for Windows (package mythryl_compiler_compiler_for_intel32_win32)
  * $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib		package make_compiler (current arch/os)

  * $ROOT/src/lib/core/compiler.lib
  * $ROOT/src/lib/core/cm.lib

  * $comp-lib.lib/comp-lib.lib			Utility library for compiler

  - $ROOT/src/lib/core/viscomp/core.sublib		Compiler core functionality
  - $ROOT/src/lib/core/viscomp/{pwrpc32,sparc32,intel32}.lib
					Machine-specific parts of compiler

  - $ROOT/src/lib/core/internal/{interactive-system,cm-lib,host-compiler-0}.lib
					Glue that holds the interactive system
					together

  * $LOWER/{LOWHALF,control,Lib,PWRPC32,SPARC32,INTEL32}.lib
					Various LOWHALF bits
					(Other LOWHALF libraries such as
					 graph, Visual, etc. do not currently
					 take part in the build.)

  * $/{mlyacc,mllex,mlburg}-tool.lib	makelib plug-in libraries for common tools
  * $/{grammar,lex,burg}-ext.lib	makelib plug-in libraries for common file
					extensions

Paths of the form $/foo/<more> are shorthands for $foo/foo/<more>.

A more complete explanation of the $-notation can be found later in
this document or in the makelib manual.

To learn about the definitions of the $-anchors (and, thus, where in
the source tree the above libraries are defined), consult the
"pathconfig" file here in this directory.

* Before you can use the bootstrap compiler (make_compiler)...
----------------------------------------------------

To be able to use make_compiler at all, you must first say

	makelib::autoload "$ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib";

after you start mythryld.  Alternatively -- and perhaps more conveniently --
you can provide "$ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib" as a command-line argument to mythryld:

      % mythryld '$ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib'

(Be sure to protect the dollar symbol which usually has its own
special meaning to the shell.)

* Compiling the compiler
------------------------

We are now back to the old scheme where a call to make_compiler.make() suffices to
build a bootable set of files (libraries in our case).  make_compiler.make maintains
two parallel hierarchies of derived files:

  1. The .compiled file hierarchy, containing compilation results for
     each individual source file; this hierarchy is rooted at
      <prefix>.compiledfiles.<arch>-<opsys>

  2. The seed library hierarchy, containing copies of all
     libraries that participate in building the compiler.
     This hierarchy is rooted at
       build7.seed-libraries

(The default "mythryld" prefix may be changed to whatever you like
by using make_compiler.make' with the new <prefix> as the optional string argument.)

make_compiler.make reuses existing seed libraries after it has verified that
they are consistent with their corresponding .compiled files.

Seed libraries do not need to be deleted in order
for make_compiler.make to work correctly.

To bootstrap a new system (using the runtime system loader),
the seed libraries _must_ be present.

The .compiled files need not be present, but their presence
does not hurt either.

You can reduce the number of extra files compiled and libraries
built during make_compiler.make at the expense of not building any
cross-compilers.  For that, say
	.set (make_compiler.symval "LIGHT") (THE 1);
before running make_compiler.make.

* Making the executable image
-----------------------------

The bin/mythryld "executable" (heap image) is made by doing
   make compiler
in the toplevel install directory.



* Installing an executable (heap image) for more permanent use
--------------------------------------------------------------

You can "install" a newly generated heap image by replacing the old
image with the new one AND AT THE SAME TIME replacing the old
libaries with the new ones.  To do this, run the "installml" script.

Like "run-new-build", "installml" also expects the <prefix> as its first
argument.  <prefix> defaults to "mythryld" if no argument is specified.

"installml" patches the ../../lib/pathconfig file to reflect any
changes or additions to the path name mapping.  (I say "patches"
because entries unrelated to the compiler build process are retained in
their original form.)  If you want to use a destination directory that
is different from ../../lib, then you must do this by hand (i.e.,
installml does not have an option for that).


* Cross-compiling
-----------------

All cross-compilers live in the "$ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib" library.
You must first say

	makelib::autoload "$ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib";

before you can access them.  (This step corresponds to the old
make_compiler.retarget call.)  After that, _all_ cross-compilers are available
at the same time.  However, the ones that you are not using don't take
up any undue space because they only get loaded once you actually
mention them at top level.  The names of the packages currently
exported by $ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib are:

	mythryl_compiler_compiler_for_pwrpc32_macos
	mythryl_compiler_compiler_for_pwrpc32_posix
	mythryl_compiler_compiler_for_sparc32_posix
	mythryl_compiler_compiler_for_intel32_posix
	mythryl_compiler_compiler_for_intel32_win32

	mythryl_compiler_for_pwrpc32
	mythryl_compiler_for_sparc32
	mythryl_compiler_for_intel32_win32
	mythryl_compiler_for_intel32_posix

(mythryl_compiler_compiler_for_pwrpc32_macos is not very useful at the moment because there is no
implementation of the basis library for the MacOS.)

Alternatively, you can select just the one single package that you
are interested in by auto-loading $ROOT/src/lib/core/compiler/<arch>.lib or
$ROOT/src/etc/mythryl-compiler-compiler/<arch>-<os>.lib.
<arch> currently ranges over "pwrpc32", "sparc32" and "intel32".
<os> can be "posix" or "macos" or "win32".
(Obviously, not all combinations are valid.)

Again, as with $ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib, you can specify the .lib file as an
argument to the mythryld command:

    % mythryld '$ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib'

or

    % mythryld '$ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-sparc32-posix.lib'

[Note: The command line for the "mythryld" command accepts configuration
parameters of the form "--runtime-xxx...", mode switches of the form "--make"
and "--autoload", names of source files -- which are passed to "use" -- and
arguments suitable for makelib::make or makelib::autoload.  makelib::autoload is the
default; the "--make" and "--autoload" mode switches can be used to change the
default -- even several times within the same command line.
A single argument "--compile-server" is also accepted, but it should not be
used directly as it is intended for use by the parallel compilation
facility within makelib.]

* Path configuration
--------------------

+ Basics:

One of the new features of makelib is its handling of path names.  In the
old makelib, one particular point of trouble was the autoloader.  It
analyzes a group or library and remembers the locations of associated
files.  Later, when the necessity arises, those files will be read.
Therefore, one was asking for trouble if the current working directory
was changed between analysis- and load-time, or, worse, if files
actually moved about (as is often the case if build- and
installation-directories are different, or, to put it more generally,
if makelib's state is frozen into a heap image and used in a different
environment).

Maybe it would have been possible to work around most of these
problems by fixing the path-lookup mechanism in the old makelib and using
it extensively.  But path-lookup (as in the Unix-shell's "PATH") is
inherently dangerous because one can never be too sure what it will be
that is found on the path.  A new file in one of the directories early
in the path can upset the program that hopes to find something under
the same name later on the path.  Even when ignoring security-issues
like trojan horses and such, this definitely opens the door for
various unpleasant surprises.  (Who has never named a test version
of a program "test" an found that it acts strangely only to discover
later that /bin/test was run instead?)

Thus, the new scheme used by makelib is a fixed mapping of what I call
"configuration anchors" to corresponding directories.  The mapping can
be changed, but one must do so explicitly.  In effect, it does not
depend on the contents of the file system.  Here is how it works:

If I specify a pathname that starts with a "$", then the first arc
between "$" and the first "/" is taken as the name of a so-called
"anchor".  makelib knows a mapping from anchor names to directory names and
replaces the prefix $<anchor> with the name of the corresponding
directory.  Therefore, an anchored path has the general form

   $<anchor>/<path>

It is important that there is at least one arc in <path>.  In other
words, the form $<anchor> is NOT valid.

(Actually, under certain circumstances it _is_ valid -- and means what
it seems to mean, namely the directory denoted by the name that
<anchor> is mapped to.  However, since directory names do not usually
occur by themselves, you can think of this form as being invalid.
There is one exception to this: "bind" specifications for .lib files.
See the makelib manual for more details.)

Examples:

   $ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib
   $ROOT/src/lib/std/standard.lib
   $LOWER/control.lib

The special case where <anchor> coincides with the first arc of <path>
can be abbreviated by ommitting <anchor>.  This leads to the shorthand

  $/<anchor>/<more>...

for the longer

  $<anchor>/<anchor>/<more>...

Examples:

  $/foo/bar/baz.lib      #  same as $foo/foo/bar/baz.lib 
  $/standard.lib            #  same as $ROOT/src/lib/std/standard.lib 

There used to be a notion of "implicit" anchors where in the case that
<anchor> is a known anchor, paths of the form

   <anchor>/<more>...

were interpreted as if they had been written

   $<anchor>/<anchor>/<more>...

This is no longer the case. <foo>/<bar>... now always means what it
seems to mean: a relative path starting with an arc named <foo>.

+ Why anchored paths?

The important point is that one can change the mapping of the anchor,
and the tranlation of the (anchored) path name -- together will all
file names derived from it -- will also change accordingly -- even
very late in the game.  makelib avoids "elaborating" path names until it
really needs them when it is time to open files.  makelib is also willing
to re-elaborate the same names if there is reason to do so. Thus, the
"standard.lib" library that was analyzed "here" but then moved "there"
will also be found "there" if the anchor has been re-set accordingly.

The anchor mapping is (re-)initialized at startup time by reading two
configuration files.  Normally, those are the "../../lib/pathconfig"
file and the ".lib7-pathconfig" file in your home directory (if such
exists).  During an ongoing session, function makelib::anchor::anchor can be
used to query and modify the anchor mapping.

+ Different configurations at different times:

During compilation of the compiler, make_compiler uses a path configuration that
is read from the file "pathconfig" located here in this directory.

At bootstrap time,
the same anchors are mapped to the corresponding sub-directory of
the "seed-libraries" directory: standard.lib is mapped to
    build7.seed-libraries.<arch>-<os>/standard.lib
which means that makelib will look for a library named
    build7.seed-libraries.<arch>-<os>/standard.lib/standard.lib
and so forth.

[Note, there are some anchors in "pathconfig" that have no
corresponding sub-directory of the boot director.  Examples are
"root.lib", "cm", and so on.  The reason is that there are no stable
libraries whose description files are named using these anchors;
everything anchored at "$cm" is a group but not a library.]

By the way, you will perhaps notice that there is no file
	build7.seed-directories.<arch>-<os>/standard.lib/standard.lib
but there _is_ the corresponding freezefile
	build7.seed-directories.<arch>-<os>/standard.lib/CM/<arch>-<os>/standard.lib
makelib always looks for freezefiles first.

This mapping (from anchors to names in the boot directory) is the one
that will get frozen into the generated heap image at boot time.
Thus, unless it is changed, makelib will look for its libraries in the boot
directory.  The aforementioned "run-new-build" script will make sure (by
setting the environment variable CM_PATHCONFIG) that the mapping be
changed to the one specified in a new "pathconfig" file and
placed into the test library directory.  It points all anchors to the
corresponding entry in the test library directory.
  Thus, "run-new-build" will let a new heap image run with its
corresponding new libraries.

Normally, however, makelib consults other pathconfig files at startup --
files that live in standard locations.  These files are used to modify
the path configuration to let anchors point to their "usual" places.
The names of the files that are read (if present) are configurable via
environment variables.  At the moment they default to
	/usr/lib/lib7-pathconfig
and
	$HOME/.lib7-pathconfig
The first one is configurable via CM_PATHCONFIG (and the default is
configurable at boot time, also via CM_PATHCONFIG); the last is
configurable via CM_LOCAL_PATHCONFIG.
For example, I always keep my "good" libraries in `pwd`/../../lib --
where both the main build script (in sh/build) and the
"installml" script (see above) also put them -- so I don't have to do
anything special about my pathconfig file.

Once I have new heap image and libraries working, I replace the old
"good" image with the new one:

  mv <image>.<arch>-<osvariant> ../../heap/mythryld.<arch>-<osvariant>

After this I must also move all libraries from <image>.libs/* to their
corresponding position in ../../lib.

Since this is cumbersome to do by hand, there is a script called
"installml" that automates this task.  Using the script has the added
advantage that it will not clobber libraries that belong to other than
the current architecture.  (A rather heavy-handed "rm/mv" approach
will delete all stable libraries for all architectures.)
"installml" also patches the ../../lib/pathconfig file as necessary.

Of course, you can organize things differently for yourself -- the
path configuration mechanism should be sufficiently flexible.  If you
do so, you will have to set CM_PATHCONFIG.  This must be done before
you start mythryld.  If you want to change the pathname mapping at the time
mythryld is already running, then use the functions in makelib::anchor.

* Libraries vs. Groups
----------------------

With the old makelib, "group" was the primary concept while "library" and
"building" could be considered afterthoughts.  This has changed.
Now "library" is the primary concept, "building" is semantically
significant, and "groups" are a secondary mechanism.

Libraries are used to "package the world"; groups are used to give
package to libraries.  Each group can be used either in precisely
one library (in which case it cannot be used at the interactive
toplevel) or at the toplevel (in which case it cannot be used in any
library).  In other words, if you count the toplevel as a library,
then each group has a unique "owner" library.  Of course, there still
is no limit on how many times a group can be mentioned as a member of
other groups -- as long as all these other groups belong to the same
owner library.

Normally, collections of files that belong together should be made
into proper makelib libraries.  makelib groups (aka "library components") should
be used only when there are namespace problems within a library.

Aside from the fact that I find this design quite natural, there is
actually a technical reason for it: when you build a library
(groups cannot be built), then all its sub-groups (not
sub-libraries!)  get "sucked into" the stable archive of the library.
In other words, even if you have n+1 makelib description files (1 for the
library, n for n sub-groups), there will be just one file representing
the one stable archive (per architecture/os) for the whole thing.  For
example, I structured the standard basis into one library with two
sub-groups, but once you compile it (make_compiler.make) there is only one
stable file that represents the whole basis library.  If groups were
allowed to appear in more than one library, then library building would
duplicate the group (its code, its environment data structures, and
even its dynamic state).

There is a small change to the syntax of group description files: they
must explicitly state which library they belong to. makelib will verify
that.  The owner library is specified in parentheses after the "group"
keyword.  If the specification is missing (that's the "old" syntax),
then the the owner will be taken to be the interactive toplevel.

* Pervasive environment, core environment, the init library "init.cmi"
-------------------------------------------------------------------------

make_compiler.make starts out by building and compiling the
"init library".  This library cannot be described in the "usual" way
because it uses "magic" in three ways:
 - it is used to later tie in the runtime system
 - it binds the "_Core" package
 - it exports the "pervasive" environment

The pervasive environment no longer includes the entire basis library
but only non-modular namings (top-level namings of variables and
types).

makelib cannot automatically determine dependencies (or exports) for the
init library source files, but it still does use its regular cutoff
recompilation mechanism.  Therefore, dependencies must be given
explicitly.  This is done by a special description file which
currently lives in core/init/init.cmi (as an anchored path:
"$ROOT/src/lib/core/init/init.cmi").  See the long comment at the beginning of
that file for more details.

After it is built, $ROOT/src/lib/core/init/init.cmi can be used as an "ordinary"
library by other libraries.  (This is done, for example, by the
implementation of the Basis library.)  Access to
"$ROOT/src/lib/core/init/init.cmi" is protected by the privilege named
"primitive".  Also, note that the .cmi-file is not automatically
recognized as as makelib description file. ("cmi" should remind you of "CM
- Initial library".)  Therefore, it must be given an explicit member
class:

     $ROOT/src/lib/core/init/init.cmi : cm

* Autoloader
------------

The new system heavily relies on the autoloader.  As a result, almost
no symbot tables need to get unpickled at bootstrap time.  The
construction of such tables is deferred until they become
necessary.  Thanks to this, it was possible to reduce the size of the
heap image by more than one megabyte (depending on the architecture).
The downside (although not really terribly bad) is that there is a
short wait when you first touch an identifier that hasn't been touched
before.  (I acknowledge that the notion of "short" may depend on your
sense of urgency. :-)

The reliance on the autoloader (and therefore makelib's library mechanism)
means that in order to be able to use the system, your paths must be
properly configured.

Several libraries get pre-registered at bootstap time.  Here, at least
the following two should be included: the basis library ($ROOT/src/lib/std/standard.lib)
and makelib itself ($ROOT/src/lib/core/cm.lib).  Currently, we also pre-register the
library exporting package compiler ($ROOT/src/lib/core/compiler.lib) and the
core library ($ROOT/src/lib/std/standard.lib).

Here are some other useful libraries that are not pre-registered but
which can easily be made accessible via makelib::autoload (or, non-lazily,
via makelib::make):

	$ROOT/src/lib/core/mythryl-compiler-compiler/mythryl-compiler-compiler-for-this-platform.lib
	$ROOT/src/lib/core/compiler/mythryl-compiler-compilers-for-all-supported-platforms.lib	- provides "package <Arch>Compiler" and
				  	"package <Arch><OS>make_compiler" for various
				  	values of <Arch> and <OS>

* Internal sharing
------------------

Dynamic values of loaded modules are shared.  This is true even for
those modules that are used by the interactive compiler itself.  If
you load a module from a library that is also used by the interactive
compiler, then "loading" means "loading the static environmnent" -- it
does not mean "loading the code and linking it".  Instead, you get to
share the compiler's dynamic values (and therefore the executable
code as well).

Of course, if you load a module that hasn't been loaded before and
also isn't used by the interactive system, then makelib will get the code
and link (execute) it.

* Access control
----------------

In some places, you will find that the "group" and "library" keywords
in description files are preceeded by certain strings, sometimes in
parentheses.  These strings are the names of "privileges".  Don't
worry about them too much at the moment.  For the time being, access
control is not enforced, but the infrastructure is in place.

* Preprocessor
--------------

The syntax of expressions in and.if #elif clauses is now more ML-ish
instead of C-ish.  (Hey, this is ML after all!)  In particular, you
must use "and", "or", and "not" instead of "&&", "||" and "!".
Unary minus is "~".

A more interesting change is that you can now query the exports of
sources/subgroups/sublibraries:

  - Within the "members" section of the description (i.e., after "is"):
    The expression
	defined(<namespace> <name>)
    is true if any of the included members preceeding this clause exports
    a symbol "<namespace> <name>".
  - Within the "exports" section of the description (i.e., before "is"):
    The same expression is true if _any_ of the members exports the
    named symbol.
    (It would be more logical if the exports section would follow the
     members section, but for esthetic reasons I prefer the exports
     section to come first.)

Example:

	+--------------------------+
	|Library		   |
	|  package Foo	           |
	|defined.if(package Bar)   |
	|  package Bar	           |
	|#endif			   |
	|is			   |
	|MYTHRYL_COMPILER_MAJOR_VERSION.if > 110     |
	|  new-foo.sml		   |
	|#else			   |
	|  old-foo.sml		   |
	|#endif			   |
	|defined.if(package Bar)   |
	|  bar-client.sml	   |
	|#else			   |
	|  no-bar-so-far.sml	   |
	|#endif			   |
	+--------------------------+

Here, the file bar-client.sml gets included if MYTHRYL_COMPILER_MAJOR_VERSION is
greater than 110 and new-foo.sml exports a package Bar _or_ if
MYTHRYL_COMPILER_MAJOR_VERSION <= 110 and old-foo.sml exports package Bar. Otherwise
no-bar-so-far.sml gets included instead.  In addition, the export of
package Bar is guarded by its own existence.  (Package Bar could
also be defined by no-bar-so-far.sml in which case it would get
exported regardless of the outcome of the other "defined" test.)

Some things to note:

  - For the purpose of the pre-processor, order among members is
    significant.  (For the purpose of dependency analysis,
    order continues to be not significant).

  - As a consequence, in some cases pre-processor dependencies and
    compilation-dependencies may end up to be opposites of each other.
    (This is not a problem; it may very well be a feature.)

* The Basis Library is no longer built-in
-----------------------------------------

The SML'97 basis is no longer built-in.  If you want to use it, you
must specify $standard.lib/standard.lib as a member of your group/library.

* No more aliases
-----------------

The "alias" feature is no longer with us.  At first I thought I could
keep it, but it turns out that it causes some fairly fundamental
problems with the autoloader.  However, I don't think that this is a
big loss because path anchors make up for most of it.  Moreover,
stable libraries can now easily be moved to convenient locations
without having to move large source trees at the same time. (See my
new sh/build script for examples of that.)

It is possible to simulate aliases (in a way that is safer than the
original alias mechanism).  For example, the root.lib file (which is the
root of the whole system as far as make_compiler.make is concerned) acts as an
alias for $ROOT/src/lib/core/internal/interactive-system.lib.
In this case, root.lib is a group to avoid having a (trivial) stable
library file built for it.

A library can act as an "alias" for another library if it has a
verbatim copy of the export list and mentions the other library as its
only member.

* Don't use relative or absolute pathnames to refer to libraries
----------------------------------------------------------------

Don't use relative or absolute pathnames to refer to libraries.  If
you do it anyway, you'll get an appropriate warning at the time when
you do make_compiler::make_compiler().  If you use relative or absolute pathnames to
refer to library B from library A, you will be committed to keeping B
in the same relative (to A) or absolute location.  This, clearly,
would be undesirable in many situations (although perhaps not always).
