Installation guide
From Diracwiki
The DIRAC developers team would like to provide the user with 'almost' complete installation know-how. The user is welcome to send a report in the case of installation problems. Answers are going to appear in this web-space.
Supported hardware and software
DIRAC can be run on a number of different UNIX operating systems. The DIRAC release has been tested on Linux, SGI, IBM AIX, Apple OS X, Cray, HP, Sun, Dec, and others (see here).
Most of the program is written in Fortran 77, although the main program and some utility routines are written in C. More and more functionalities are being written in module oriented Fortran 90. Machine-dependent (compiler-dependent) features are handled with the C-preprocessor. All floating point computations are performed in double precision (64-bit).
DIRAC may work in other UNIX environments, and the user is encouraged to report back any successful porting of the program.
DIRAC has not been ported to any Windows operating system, but could presumably run in the Cygwin environment, for example.
Compiler specific issues
GFortran
- If you get an error message such as Fatal Error: Can't open module file 'memory_tracker.mod' for reading at (1): No such file or directory and you are using a GFortran version older than 4.3 please update to at least version 4.3.
Ifort
- Certain versions of the Intel Fortran compiler implements "dummy reads" in a very stupid way. This makes seeking very slow in large files, which can affect calculations on large molecules. The issue is reported to Intel, but affect Ifort at least up to version 10. As far as we know there is no solution except to use a different compiler.
Compilation examples
See also how to do Compilation on various platforms.
32 and 64 bit compatibility
DIRAC is intended to run on both 32 bit and 64 bit platforms[1][2][3]. The advantage of 64 bit computers is that DIRAC can use more than 2 GB of memory on these machines. Note however, that Dirac safely uses and indexes much larger sets of integrals on file, so it is only the in-core memory that is affected by the choice of 32 and 64 bit platforms.
Certain platforms (for example x86_64) allow the use of both 32 and 64 bit code. In this case the user can choose the appropriate memory model. Since DIRAC is using very few integers and pointers there is no real speed or space advantage when employing 32-bit code. It is however important that the user keeps in mind that a 32-bit program cannot safely allocate more than 2 GB of memory (about 260 million double precision words).
For safe 64 bit operation the program has be compiled with 64-bit Fortran integers (not C-integers). This requires that all supporting libraries - both numerical (BLAS,LAPACK) and parallel (MPI) support these integers, i.e. these must also be built in ILP64 manner[4][5]. This can be achieved by using Fortran compiler specific 'integer*8' flag for their buildup.
In certain cases it may be possible to use 32 bit integers in combination with a 64 bit memory model (LP64). Then DIRAC cannot access more than 16 GB of memory. Currently, LUCITA module is not working with LP64.
When configuring DIRAC on 64-bit architectures (like x86_64), it assigns as default 64-bit Fortran integers. User can see that when appropriate Fortran flag (-i8 for ifort, -fdefault-integer8 for gfortran, etc) is appearing together with -DINT_STAR8 preprocessor flag in the Makefile.config configuration file.
We present some cases for adapting common libraries (mathematical and parallel) on 64-bit architectures.
Internal lapack and blas mathematical libraries in DIRAC
DIRAC needs in fact two basic libraries lapack and blas. These may be replaced by many others available to the user, for example Intel MKL library, or Atlas.
However, DIRAC is distributed with own lapack and blas libraries. One of the reasons is that not all 64-bit architectures (x86_64) are providing ILP64 data type libraries (but almost all are providing I32LP64 libraries - lib64). The user gets these libraries compiled within the DIRAC suite, if this is activated by the configure setting. Another reason for own DIRAC libraries is that not all external libraries provide all subroutines of the lapack suite.
The internal (netlib) version of LAPACK is compiled when flags
-DUSE_DIRAC_DLAPACK -DUSE_DIRAC_ZLAPACK -DUSE_DIRAC_DLAMCH
are present in 'Makefile.config'.
For both BLAS and LAPACK the preprocessor flags are:
-DUSE_DIRAC_BLAS -DUSE_DIRAC_DLAMCH -DUSE_DIRAC_DLAPACK -DUSE_DIRAC_ZLAPACK
Usually, the internal lapack is used when a linked external library does not provide all required subroutines.
Normally, the user would enable the combined set via configure where he specifies internal lapack together with an external blas-type library. This is the case of following ATLAS library.
ATLAS library on 64-bit platforms
ATLAS library [6] is the most popular and free of charge choice for the DIRAC program suite, especially when there are no commercial substitutes available, or are broken (like some versions INTEL MKL).
How to install the ILP64 clone of this library ? First, according to ATLAS installation manual run ./configure. Afterwards do following changes in the generated "Make.inc" configuration file.
Assuming the default GNU set of compilers (gfortran,gcc) for building up the library, the user has to specify [7]
-DF77_INTEGER=long
in F2CDEFS (instead of default int) and also to add
-fdefault-integer-8
to F77FLAGS in the mentioned Make.inc file, which is generated after configuring the library. Not changing these parameters shall lead either to LP64 data type (on 64-bit platforms), or to ILP32 data type module (on 32-bit architectures).
Installation is described in the INSTALL.txt file of the ATLAS library package. We recommend the user to do the checking procedures as well described in this file (make check...).
However, optimized ATLAS does not have all lapack routines[8] needed to satisfy the RELADC module of DIRAC.
The user can therefore pick the DIRAC internal lapack library during the configuring step. This shouldn't matter so much for performance, since the DIRAC's own lapack is utilizing ATLAS for blas routines.
Preparing ATLAS with full LAPACK library in the ILP64 data model
Even better is installing the LAPACK extension to ATLAS as described here [9].
To summarize, it is important to make LAPACK compilation flags identical with ATLAS counterparts. Then create the LAPACK library and in the second step make ATLAS configure script pointing to the prepared LAPACK library file. Then initiate the 'make ATLAS' machinery.
Part of LAPACK make.inc configuration file will look like:
FORTRAN = <the_same_as_in_ATLAS - F77> OPTS = <the_same_as_in_ATLAS with '-i8' - F77FLAGS > DRVOPTS = $(OPTS) NOOPT = LOADER = <the_same_as_in_ATLAS - F77>
More detailed description of the LAPACK-ATLAS preparation is in the next paragraph.
ATLAS library with full LAPACK for -i4 integers on x86_64-redhat-linux (I32ILP64 model)
How to make (I32LP64) ATLAS-LAPACK library.
On our x86_64-redhat-linux cluster I (radovan) had problems with the MKL 10.0.011 library and DIRAC with -i4 integers. Therefore I decided to build the ATLAS library with the full LAPACK instead of the mentioned MKL.
In the hope that this can be useful to somebody one day. I describe here the steps, following the nice example ATLAS with full LAPACK on Linux/AMD64.
I have downloaded the latest ATLAS and LAPACK and extracted them in the directory $HOME/math
so now there are two directories
<$HOME/math>ls atlas-3.8.2 lapack-3.1.1
first I go into atlas-3.8.2 and create a directory for a dry-run configuration which will give me options for the LAPACK compilation (options in ATLAS and LAPACK have to match)
<$HOME/math/atlas-3.8.2> mkdir delete_me <$HOME/math/atlas-3.8.2> cd delete_me
using
cat /proc/cpuinfo
I got the info for the dry-run configure and choose a target directory
<$HOME/math/atlas-3.8.2/delete_me> ../configure -b 64 -D c -DPentiumCPS=3000 -Fa alg -fPIC --prefix=$HOME/atlas ... lot of output ...
I need to extract F77 and F77FLAGS
<$HOME/math/atlas-3.8.2/delete_me> fgrep "F77 =" Make.inc F77 = g77 <$HOME/math/atlas-3.8.2/delete_me> fgrep "F77FLAGS =" Make.inc F77FLAGS = -O -fPIC -m64
with this I can go into the LAPACK directory, copy the most similar make.inc example
<$HOME/math/lapack-3.1.1> cp INSTALL/make.inc.LINUX make.inc
and adapt it, the relevant lines are (and they match ATLAS)
... FORTRAN = g77 OPTS = -O -fPIC -m64 DRVOPTS = $(OPTS) NOOPT = LOADER = g77 LOADOPTS = ...
and build it
<$HOME/math/lapack-3.1.1> make lib ... lot of output ...
now back to ATLAS, this time, instead of dry-run, I create the real directory
<$HOME/math/atlas-3.8.2> mkdir stallo-64-i4 <$HOME/math/atlas-3.8.2> cd stallo-64-i4
and configure pointing to the generated LAPACK library
<$HOME/math/atlas-3.8.2/stallo-64-i4> ../configure -b 64 -D c -DPentiumCPS=3000 -Fa alg -fPIC --prefix=$HOME/atlas-64-i4 \
--with-netlib-lapack=$HOME/math/lapack-3.1.1/lapack_LINUX.a
... lot of output ...
and I am ready to run make
<$HOME/math/atlas-3.8.2/stallo-64-i4> make ... lot of output and time for a coffee ...
after this is done, few checks
<$HOME/math/atlas-3.8.2/stallo-64-i4> make check ... lot of output ... you want to see: ... 0 skipped, 0 failed <$HOME/math/atlas-3.8.2/stallo-64-i4> make ptcheck ... lot of output ... you want to see: ... 0 skipped, 0 failed <$HOME/math/atlas-3.8.2/stallo-64-i4> make time ... lot of output ...
and finally
<$HOME/math/atlas-3.8.2/stallo-64-i4> make install
Now ATLAS is built with full the LAPACK in I32LP64 manner and I only need to adapt Makefile.config to link to the ATLAS library:
LIBS = -L$HOME/atlas-64-i4/lib -llapack -lcblas -lf77blas -latlas -L/usr/lib/gcc/x86_64-redhat-linux/3.4.3 -lg2c
This step can be also performed using '-i8' flags in appropriate ATLAS and LAPACK positions in Make.inc.
Intel MKL on 64 bit platforms
When DIRAC is compiled to use 64 bit integers (i.e. with the '-i8' flag for ifort compiler) it is very important to link BLAS and LAPACK libraries of this kind.
In the case of the Intel MKL library this is the ILP64 version of the library [10]. A typical library list (for MKL 10) can be
-lmkl_lapack -lmkl_intel_ilp64 -lmkl_sequential -lmkl_core
or (for higher MKL version of 10.0.3.020 with typical installation path)
-L/opt/intel/mkl/10.0.3.020/lib/em64t -lmkl_lapack -lguide -lpthread -lmkl_intel_ilp64 -lmkl
If you do not link to the ILP64, but to the 32-bit-integer (I32LP64) version, you get DIRAC error stop already at the beginning; if discard this initial tests, you may get very strange bugs and crashes.
Note also that MKL may have bugs[11] that affect DIRAC. In doubt use the built-in LAPACK version bundled with DIRAC.
However, not all MKL installations contain the ILP64 prepared library ! You should check whether this library is available on your system. If not, we recommend combine ILP64 build ATLAS with the internal lapack.
Parallel MPI libraries on 64 bit platforms
When DIRAC is compiled to use 64 bit integers (i.e. with the '-i8' flag for ifort compiler and -DINT_STAR8 directive) it is also required to link an appropriate MPI library.
Open MPI
If you use e.g. the open source library Open MPI [12] on your cluster, you need a 64-bit version that is configured as (assuming you use the Intel compiler suite for compilation)
./configure --prefix=/people/disk2/ilias/bin/openmpi_ifort CXX=icpc CC=icc F77=ifort FC=ifort FFLAGS=-i8 FCFLAGS=-i8
or for GNU set of compilers
./configure CXX=g++ CC=gcc F77=gfortran FC=gfortran FFLAGS=-fdefault-integer-8 FCFLAGS=-fdefault-integer-8
Up to now Open MPI 1.2.5 to 1.2.9, 1.3, 1.3.1rc4, and 1.3.2 have been tested to compile with the above given configurations.
NOTE: The GNU compiler set and Open MPI 1.3 does not work out of the box, we recommend to upgrade to the forthcoming Open MPI 1.3.1.
When you are using GNU C/C++ compilers and Intel Fortran compiler, you can configure with the specified location for installation as
./configure --prefix=/home/miro/bin/openmpi CXX=c++ CC=gcc F77=ifort FC=ifort FFLAGS=-i8 FCFLAGS=-i8
and type
make all install
You can check the correct configuration of your MPI library by using
ompi_info -a
lots of output ... Fort integer size: 8 (correct) more output
or simply by
ompi_info -a | grep 'Fort integer size'
After the successful compilation make sure that MPIRUN in pam and Makefile.config points to the new mpirun.
Notes
- when you are using Intel Compiler Fortranr/ILP64 OpenMPI, you might get this message when linking {dirac} objects with libraries:
/opt/intel/fce/9.1.036/lib/libimf.so: warning: warning: feupdateenv is not implemented and will always fail
You can fix it by adding '-i_dynamic' to LDFLGAS in Makefile.config.
- mpirun launching in the pam script (and elsewhere) must be without host file (-machinefile) parameter for OpenMPI ILP64/Intel Compilers. On Odense cluster [] I (MI) got error message of this type:
[s41p04:24210] pls:tm: failed to poll for a spawned proc, return status = 17002 [s41p04:24210] [0,0,0] ORTE_ERROR_LOG: In errno in file rmgr_urm.c at line 462 [s41p04:24210] mpirun: spawn failed with errno=-11
On Odense cluster one has to resupply parallel dirac.x with these Intel Fortran library files (from /opt/intel//fce/9.1.036/lib): libifcoremt.so.5, libifcore.so.5, libifport.so.5. Copy them into directory accessible for all nodes and add the path into LD_LIBRARY_PATH (in the case these are not reachable from original installation directory /opt/intel/... by all nodes).
Intel MPI
No check needed. The compiler wrapper should pick up the correct MPI library. More...
MPICH2
This parallel library compiled with corresponding 'Integer*8' Fortran flag does not work for the DIRAC.
MPICH2 library is not (yet?) adapted for handling different definitions of Fortran integers on 64bit machines.
On 32-bit architectures (ILP32 model) MPICH2 works.
Using 32-bit integers on 64-bit architectures (I32LP64 data model)
Not all numerical/parallel libraries on 64-bit architectures are provided with the ILP64 subset (i.e. obtained with '-i8' Fortran compiler flag). I32LP64 is the default model for system libraries.
Therefore during the DIRAC installation on 64-bit machines using the configure script the user chooses 32-bit integers. In this case DIRAC can not use LUCITA/LUCIAREL modules (to be actualized). This disadvantage is partially compensated by the fact that DIRAC can be linked against default /usr/lib64 system libraries which do not need any modification as for ILP64 model.
Installing the program
The program system is supplied as a tar-file compressed with gzip.
Unpack the program system:
tar -zxvf DIRAC08.tar.gz
or, on some machines
gunzip DIRAC08.tar.gz tar -xvf DIRAC08.tar
and a directory DIRAC08 has been created with the source code.
cd DIRAC08
Check the content of the README file.
Now run the configure script:
./configure
The script should automatically guess your machine/architecture. configure creates the file Makefile.config with system-dependent information (as which compilers, libraries, compiler flags etc. to use) for make and the pam shell script for starting DIRAC calculations.
For help on configure:
configure --help
Note that configure can be invoked with flags, e.g. for choosing compilers or paths for mathematical libraries.
Check Makefile.config and edit if needed.
Run (optional):
make depend
Note: for the compilation on an IBM system run
gmake depend
This finds out dependencies for each file, that is, if you change a header file, then all Fortran and C files using this header file will be recompiled. This is very useful for programmers, or if you wish to change some dimensions in a common block.
Finally, run:
make
or on an IBM system
gmake
for compilation/installation.
See also Examples_of_various_platforms/compilers for further hints and tips for the installation on various platforms.
DIRAC on a PC
Nowadays it is very easy to install DIRAC on personal computers or notebooks. The only necessity is a Linux operating system, which is also providing gfortran/gcc package.
Most of today's PCs are 64-bit, therefore follow installation instructions given above. For PCs with more than two cores, we recommend to prepare the corresponding openmpi set to have it parallel.
Note that one does not need a multi core computer to run DIRAC in parallel, for testing purposes one can run many processes on a single core (although it will be a bit slow of course). One can even try to run 8 processes on a common dual core machine (2CPU) to test parallelization (if the memory permits). As
Modifying the Makefile.config file
The Makefile.config file is created by the configure script and contains info about Fortran/C compilers, their flags, the amount of dynamically allocated memory and linked libraries.
The user may modify it to obtain code of desired properties: quickly compilable code for debugging, or fully optimized (serial/parallel) code.
To learn about supported platforms/compilers see here.
