Parallel compilation from the command lineSunday, March 29th, 2009
I often need to compile a bunch of generated C++ files from the command line to make sure everything compiles cleanly and to run a quick test. These are C++ data binding files generated by XSD and XSD/e from various XML schemas. There can be quite a few files (like a thousand) or there can be a handful of them but each is quite large and takes long to compile. I used to run
g++ from the command line which compiles one file at a time:
g++ -c *.cxx
This was slow. Annoyingly, I also had three more CPU cores idling while they could have been used to speed things up. If it were a longer-term project, then I would have created a makefile for it and run GNU
make in parallel (
-j 4). But these were just quick tests and creating a makefile for each set of schemas that I test seemed like a chore (after the initial test the schemas are added to a test repository where a set of special scripts and makefiles are used to automatically check for regressions before each release).
What would then be nice is a single, generic makefile that I could use to compile a set of files in parallel. Ideally, it shouldn’t be much more typing than the simple
g++ invocation above. It is quite easy to come up with such a makefile for GNU
make:
ifeq ($(src),) src := *.cxx endif obj := $(patsubst %.cxx,%.o,$(wildcard $(src))) .PHONY: all clean all: $(obj) clean: rm -f $(obj) %.o: %.cxx $(CXX) $(CPPFLAGS) $(CXXFLAGS) -c $< -o $@
Now I can compile all the C++ files in a directory in parallel:
make -f ~/makefile -j 4
I can also specify a custom set of C++ files using the
src variable:
make -f ~/makefile -j 4 src="test*.cxx driver.cxx"