simulation | Dr. Ravindra Shinde

GPAW v20 installation on the comet cluster at XSEDE, San Diego

Sun, 21 Feb 2021 00:00:00 +0000

The following set of instructions will guide you through the steps involved in the installation of GPAW on the Comet cluster, located at the XSEDE, San Diego.

This will be an Intel Python-based installation. You may have to install it separately in your home directory. I will be installing an MPI enabled version sans scalapack.

Make sure that you load the following modules:

intel/2018.1.163
mvapich2_ib/2.3.2
gsl/2.5
mkl/2018.1.163
fftw/3.3.8

Copy siteconfig_example.py file to siteconfig.py

Make the necessary changes in the path such that the final file looks like this:

compiler = './icc.py'
mpicompiler = './icc.py'
mpilinker = 'MPICH_CC=gcc mpicc'

compiler = 'gcc'
mpicompiler = 'mpicc'
mpilinker = 'mpicc'
scalapack = True

library_dirs += ['/opt/intel/oneapi/mkl/latest/lib/intel64']

include_dirs = ['/opt/intel/oneapi/mpi/latest/include']

libraries = ['mkl_rt', 'pthread', 'm', 'dl']

libraries += ['xc']
# change this to your installation directory
LIBXCDIR='/home/neelravi/libxc/libxc-4.3.4/install/'
library_dirs += [LIBXCDIR + 'lib']
include_dirs += [LIBXCDIR + 'include']

define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]
define_macros += [("GPAW_ASYNC",1)]
define_macros += [("GPAW_MPI2",1)]


# FFTW3:
fftw = False
if fftw:
    libraries += ['fftw3']

After making the necessary changes, install the gpaw in the home directory of the user.

python3 setup.py install --user

This will compile the C code and copy the python files in appropriate locations.

You may have to add the PYTHONPATH in you .bashrc file.

export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/home/user/libxc/libxc-4.3.4/install/lib
export PATH=/home/user/.local/bin:$PATH
export PYTHONPATH=/home/user/ase:/home/user/gpaw

CP2K wrap atoms back to the home unit cell in VMD

Tue, 16 Jun 2020 00:00:00 +0000

The following set of commands in the VMD console will wrap back the atoms in the neighboring unit cells back to the home unit cell.

a, b, and c are the lattice constants in x,y and z directions.

pbc set {a b c} -all
pbc box
pbc wrap -all

Octopus TDDFT code installation on the comet cluster at XSEDE, San Diego

Thu, 21 Nov 2019 18:46:00 +0000

Load Modules:

I chose Intel compiler and intelmpi to compile the mpi version of octopus on Comet (after the Rocks OS upgrade to Rocks 7).

The following modules need to be loaded first.

intel/2018.1.163
intelmpi/2018.1.163
gsl/2.5
mkl/2018.1.163

Install dependencies/libraries:

Install libxc-4.3.4 or above

Download the latest package of libxc. Use the following configure line:

./configure --prefix=/home/neelravi CC=mpicc FC=mpif90 FCFLAGS=-O3 CFLAGS=-O3
make 
make install
make check

Install fftw-3.3.8 or above

Do not use the pre-installed fftw package. Download the latest fftw package. Use the following configure line to compile.

I chose fftw-3.3.8.

./configure --prefix=/home/neelravi CC=mpicc CFLAGS=-O3 F77=mpif90 F77FLAGS=-O3
make 
make install
make check

Install octopus code

Download the stable version of the octopus. (i.e. octopus 9.1) Use the following configure script.

./configure --prefix=$HOME \
  CC=mpicc FC=mpif90 CFLAGS="-O3" FCFLAGS="-O3" \
--enable-mpi --enable-openmp \
--with-blas="-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm -ldl" \
--with-libxc-prefix=$HOME \
--with-fftw-prefix=$HOME \
--with-blacs="-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm -ldl" \
--with-scalapack="-L${MKLROOT}/lib/intel64 -lmkl_rt -lpthread -lm -ldl" \
--with-gsl-prefix="/opt/gsl/2.5/intel"

make 
make install
make check

The last command make check should either pass or fail all the tests. If some tests pass and some fail, then there is something wrong with the installation. Please note that a test corresponding to the glib may fail (probably this is used when Species are read from the picture file). You may ignore it.

Run a sample calculation with more than one node. The output file should show the parallelization and anticipated resource wastage.

Your octopus is ready!

Octopus TDDFT code installation on the XC40 Cray supercomputer

Thu, 06 Jul 2017 13:00:00 +0000

Load Modules in the PrgEnv-gnu:

I managed to install octopus in gnu programming environment with the following settings:

Modules loaded in the PrgEnv-gnu

modules/3.2.10.5
nodestat/2.2-1.0502.60539.1.31.ari
sdb/1.1-1.0502.63652.4.25.ari
alps/5.2.4-2.0502.9822.32.1.ari
lustre-cray_ari_s/2.5_3.0.101_0.46.1_1.0502.8871.20.1-1.0502.21538.30.1
udreg/2.3.2-1.0502.10518.2.17.ari
ugni/6.0-1.0502.10863.8.29.ari
gni-headers/4.0-1.0502.10859.7.8.ari
dmapp/7.0.1-1.0502.11080.8.76.ari
xpmem/0.1-2.0502.64982.5.3.ari
hss-llm/7.2.0
Base-opts/1.0.2-1.0502.60680.2.4.ari
gcc/6.1.0
craype/2.5.7
nodehealth/5.1-1.0502.65826.9.1.ari
cray-libsci/16.09.1
pmi/5.0.10-1.0000.11050.0.0.ari
atp/2.0.2
PrgEnv-gnu/5.2.82
pbs/12.2.404.152084
cray-parallel-netcdf/1.7.0
cray-netcdf-hdf5parallel/4.4.1
cray-hdf5-parallel/1.8.16
fftw/3.3.4.10
craype-network-aries
craype-haswell
cce/8.5.3
cray-shmem/7.4.3
cray-mpich/7.4.3

The LD LIBRARY PATH had following things in the path:

/opt/gcc/6.1.0/snos/lib64:/opt/cray/nodehealth/5.1-1.0502.65826.9.1.ari/lib64 /opt/cray/fftw/3.3.4.10/haswell/lib:/opt/cray/hdf5-parallel/1.8.16/GNU/5.1/lib:/opt/cray/netcdf-hdf5parallel/4.4.1/GNU/5.1/lib:/opt/cray/parallel-netcdf/1.7.0/GNU/5.1/lib:/opt/cray/pmi/5.0.10-1.0000.11050.0.0.ari/lib64:/opt/cray/libsci/16.09.1/GNU/5.1/x86_64/lib:/opt/cray/xpmem/0.1-2.0502.64982.5.3.ari/lib64:/opt/cray/dmapp/7.0.1-1.0502.11080.8.76.ari/lib64:/opt/cray/ugni/6.0-1.0502.10863.8.29.ari/lib64:/opt/cray/udreg/2.3.2-1.0502.10518.2.17.ari/lib64:/opt/cray/alps/5.2.4-2.0502.9822.32.1.ari/lib64

the configure script looks like:

set LD_LIBRARY_PATH=($LD_LIBRARY_PATH $CRAY_LD_LIBRARY_PATH)
setenv LIBS_BLAS /opt/cray/libsci/16.09.1/GNU/5.1/x86_64/lib/libsci_gnu_mpi_mp.a
setenv LIBS_LAPACK /opt/cray/libsci/16.09.1/GNU/5.1/x86_64/lib/libsci_gnu_mpi_mp.a
setenv LIBS_FFT /opt/cray/fftw/3.3.4.10/haswell/lib/libfftw3.a

./configure  --prefix=/mnt/lustre/mrc2/mrcravi/devel-basedir-test  \
--with-libxc-prefix=/mnt/lustre/mrc2/mrcravi/devel-basedir  \
--with-libxc-include=/mnt/lustre/mrc2/mrcravi/devel-basedir/include \
--with-gsl-prefix=/mnt/lustre/mrc2/mrcravi/devel-basedir \
--with-fftw-prefix=/opt/cray/fftw/3.3.4.10/haswell \
--enable-openmp --enable-mpi --host=cray

make -j 24
make install

The above script is for minimum working example. I managed to couple etsf, netcdf, hdf5 etc to octopus.

GPAW v20 installation on a workstation or desktop

Sat, 10 Jun 2017 00:00:00 +0000

I tried following configuration for the installation of GPAW on a standalone machine with python installed from miniconda2.

Install openmpi with intel compilers to get mpicc etc

install libxc in static mode with -fPIC flag

./configure FC=ifort CC=icc --disable-shared --enable-static --prefix=/usr/local CFLAGS=-fPIC

After that, we need mkl blas lapack, blacs and scalapack libraries, Following is the customize.py file for appropriate options.

extra_compile_args += ['-fPIC']


libraries = [ 'xc', 'mkl_blas95_lp64', 
              'mkl_lapack95_lp64', 
              'mkl_scalapack_lp64',
              'mkl_intel_lp64', 
              'mkl_sequential',
              'mkl_core', 
              'mkl_blacs_openmpi_lp64',
              'pthread', 'm', 'dl']

library_dirs += ['/usr/local/lib', 
                 '/opt/intel/composer_xe_2013.2.146/mkl/lib/intel64']
include_dirs += ['/usr/local/include', 
                 '/opt/intel/composer_xe_2013.2.146/mkl/include']

mpi_libraries = ['mpi']
mpi_library_dirs += ['/usr/local/lib']
mpi_include_dirs += ['/usr/local/include']

compiler = 'mpicc'
mpicompiler = 'mpicc'  
mpilinker = 'mpicc'

scalapack = True

if scalapack:
    define_macros += [('GPAW_NO_UNDERSCORE_CBLACS', '1')]
    define_macros += [('GPAW_NO_UNDERSCORE_CSCALAPACK', '1')]

# - static linking:
if 0:
    include_dirs += ['/usr/local/include']
    extra_link_args += ['/usr/local/lib/libxc.a']
    if 'xc' in libraries:
        libraries.remove('xc')


# Build MPI-interface into _gpaw.so:
if 0:
    compiler = 'mpicc'
    define_macros += [('PARALLEL', '1')]
    mpicompiler = 'mpicc'