computer science | Dr. Ravindra Shinde

Install and manage Intel oneAPI compiler and component tools using environment modules

Fri, 03 Dec 2021 08:41:00 +0000

Intel has recently changed their business model. Their suite of compilers, debuggers, toolkits, and libraries are given free of cost for personal or academic use. In an earlier post, I explained how to install them on Ubuntu linux and activate all the softwares using a single source command.

In this post, I will show you how to manage these compilers, libraries, and debuggers using the environment modules. So just using “module load icc” you could activate a single “icc” compoment. All the dependent modules will also be loaded automatically, if you follow the these steps.

Pre-requisites

Make sure that you have the following packages already installed on your ubuntu machine.

environment modules (i.e. the module command)
Intel oneapi basekit and/or hpckit (I am repeating the steps below)

Intel Repository Public Key

Get the Intel Repository public key and add it to your apt sources keyring so the packages will be trusted by ubuntu.

# use wget to fetch the Intel repository public key
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# add to your apt sources keyring so that archives signed with this key will be trusted.
sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# remove the public key
rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB

Configure apt client to use Intel repository

echo "deb https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list

sudo add-apt-repository "deb https://apt.repos.intel.com/oneapi all main"

Install all the packages or selected packages.

All basic compilers (Fortran, C, C++, and Python)

sudo apt install intel-basekit

HPC toolkits

sudo apt install intel-hpckit

IoT (internet of things) toolkits

sudo apt install intel-iotkit

AI (Artificial Intelligence) analytics toolkits

sudo apt install intel-aikit

Rendering toolkit

sudo apt install intel-renderkit

After Installation

After you have installed the oneAPI kits, they can be activated using the following command written in the ~/.bashrc file.

Option 1: Activate all the components at once for all users (root access)

. /opt/intel/oneapi/setvars.sh

Option 2: Activate individual components using the environment modules

cd /opt/intel/oneapi/
You will see a modulefiles-setup.sh file. If we execute this file using bash it will create a folder named modulefiles in the same /opt/intel/oneapi/ folder.
Put the following line into your $HOME/.bashrc file
1. export MODULEPATH=${MODULEPATH}:/opt/intel/oneapi/modulefiles
2. . $HOME/.bashrc or close and reopen the terminal
If you type module avail, you will see all the components of Intel oneapi available for use.
You can load or unload individual components using module load icc or module load compiler etc.
You can create your own module file and put it in the modulefiles folder. Let us create a modulefile for intel python.
Copy the following content into a file /opt/intel/oneapi/modulefiles/python/latest

    #%Module1.0###################################################################
    set min_tcl_ver 8.4
    if { $tcl_version < $min_tcl_ver } {
        puts stderr " "
        puts stderr "ERROR: This modulefile requires tcl $min_tcl_ver or greater."
        puts stderr "Your system reports that tclsh version $tcl_version is installed."
        exit 1
    }

    proc ModulesHelp { } {
            puts stderr "\tAdds Intel Python 3 to your PATH environment variable\n"
    }

    module-whatis   "adds Intel Python 3 to your PATH environment variable"

    prepend-path    PATH    /opt/intel/oneapi/intelpython/latest/bin

Check the installed packages using module avail, check the loaded packages using module list, unload packages using module unload packagename

How to predict size of proposal/wedding ring from a photo

Sun, 17 Oct 2021 20:00:00 +0000

After searching a lot over internet, I couldn’t find a code which could predict the size of the proposal ring from a photograph. So, I decided to write one. Here in the version 1.0, I am using python computer vision module to do the job.

I bought a ring based on this code’s predictions and voila, she said yes!

Pre-requisites

Make sure that you have the following packages already installed on your ubuntu machine.

python 3
opencv module of python; use pip install opencv-python

Github Gist

Run the following Github Gist code using python. It reads a file called hand1.png. Make sure that the photograph file you are providing has this name, or change the name inside the code.

How to use Intel oneapi compilers on the Github actions workflow

Sat, 03 Jul 2021 11:51:00 +0000

Intel has recently changed their business model. Their suite of compilers, debuggers, toolkits, and libraries are given free of cost for personal or academic use. We can also use these compilers on the Github Actions workflow. This post gives details of the steps and a yml file which you can directly use on your Github repository to kick the automatic compilation and testing.

Step 1

Create a file with any name (but with a .yml extention) in the .github/workflows directory in your repository. If you want to use the Github website to create the action’s file, then

click actions
create new workflow
setup a workflow by yourself
Copy paste the following content

name: Intel OneAPI build

on:
  push:
    branches: [main, master]
  pull_request:
    branches: [main, master]
jobs:
  build_intel_champ_fparser:
    runs-on: ubuntu-20.04
    defaults:
      run:
        shell: bash --noprofile --norc {0}
    steps:
    - uses: actions/checkout@v2
    - name: setup repo
      run: |
        wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
        sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
        rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
        sudo echo "deb https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list
        sudo apt-get update
    - name: install
      run: |
        sudo apt-get install -y intel-oneapi-common-vars
        sudo apt-get install -y intel-oneapi-compiler-fortran
        sudo apt-get install -y intel-oneapi-mkl
        sudo apt-get install -y intel-oneapi-mpi
        sudo apt-get install -y intel-oneapi-mpi-devel

    - name: Compile
      run: |
        source /opt/intel/oneapi/setvars.sh
        printenv >> $GITHUB_ENV
        ifort test.f90 -o test_program
        mpiifort test2.f90 -o mpitest

    - name: "Test the code"
      run: |
        echo "Running the tests using the Intel oneapi fortran compiler"
        cd tests/test01
        mpirun -np 1  $HOME/work/reponame/reponame/mpitest
        ./$HOME/work/reponame/reponame/test_program

Note that entire oneapi-basekit is not needed (and exceeds github’s docker size alloted to each user). Install the components as needed for your project and don’t forget to use “-y” option while installing using apt install

Installation of Intel oneAPI suite of compilers on Ubuntu Linux

Wed, 26 May 2021 22:51:00 +0000

Intel has recently changed their business model. Their suite of compilers, debuggers, toolkits, and libraries are given free of cost for personal or academic use. Here I explain how to install them on Ubuntu linux. All of these softwares are activated using a single source command.

Pre-requisites

Make sure that you have the following packages already installed on your ubuntu machine.

cmake
pkg-config
build-essential

If they are not installed, you can install using the following command:

sudo apt update
sudo apt -y install cmake pkg-config build-essential

Intel Repository Public Key

Get the Intel Repository public key and add it to your apt sources keyring so the packages will be trusted by ubuntu.

# use wget to fetch the Intel repository public key
wget https://apt.repos.intel.com/intel-gpg-keys/GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# add to your apt sources keyring so that archives signed with this key will be trusted.
sudo apt-key add GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB
# remove the public key
rm GPG-PUB-KEY-INTEL-SW-PRODUCTS.PUB

Configure apt client to use Intel repository

echo "deb https://apt.repos.intel.com/oneapi all main" | sudo tee /etc/apt/sources.list.d/oneAPI.list

sudo add-apt-repository "deb https://apt.repos.intel.com/oneapi all main"

Install all the packages or selected packages.

All basic compilers (Fortran, C, C++, and Python)

sudo apt install intel-basekit

HPC toolkits

sudo apt install intel-hpckit

IoT (internet of things) toolkits

sudo apt install intel-iotkit

AI (Artificial Intelligence) analytics toolkits

sudo apt install intel-aikit

Rendering toolkit

sudo apt install intel-renderkit

After Installation

After you have installed the oneAPI kits, they can be activated using the following command written in the ~/.bashrc file.

For all users (root access)

. /opt/intel/oneapi/setvars.sh

For single user

. ~/intel/oneapi/setvars.sh

Create a tag from a particular commit git / github

Sun, 04 Apr 2021 00:00:00 +0000

Create a tag from a particular commit and give some message to the tag

git tag -a v1.0.0 fg3ab1b -m "Tagged the first commit with v1.0.0"

Verify that the tag is created and all the previous tags

git tag -n

push the tag to remote

git push origin v1.0.0

GPU Programming : Matrix Multiplication using CUDA

Thu, 25 Mar 2021 00:00:00 +0000

Generate the Nsight report using :

nsys profile application.o
open the report file in nsight systems on the local machine to visualize the report.

CUDA fortran

Compile instructions:: CUDA fortran is based on the PGI compiler There are two ways in which CUDA can be used in the Fortran code.

using cudafortran directly
using C bindings

First Way using only cublas API

pgfortran -Mcudalib=cublas -Mcuda test01_dgemm.f90 -o test_dgemm

Program::

        program cublas_dgemm_test
        use cublas
        implicit none
        real*8, allocatable         :: a(:,:),b(:,:),c(:,:)
        real*8, device, allocatable :: a_d(:,:),b_d(:,:),c_d(:,:)
        integer                     :: n=20480, i
        real*8                      :: alpha=1.0d0, beta=2.0d0

        allocate(a(n,n), b(n,n), c(n,n), a_d(n,n), b_d(n,n), c_d(n,n))

        a = 0.0d0; forall(i = 1:n) a(i,i) = 1.0 
        a_d = a

        b = 0.0d0; forall(i = 1:n) b(i,i) = 1.0
        b_d = b

        c = 0.0d0; forall(i = 1:n) c(i,i) = 1.0
        c_d = c


        call dgemm('N','N',n,n,n,alpha,a_d,n,b_d,n,beta,c_d,n) 
        c=c_d
        write(*,*) "done; printing first element ", c(1,1) 
        
        deallocate (a,b,c,a_d,b_d,c_d)
        
        end program cublas_dgemm_test

Submit script for Juwels booster system

#!/bin/bash -x
#SBATCH --account=training2105
#SBATCH --partition=develbooster
#SBATCH --reservation=gpu-hack-2021-day3
#SBATCH --gres=gpu:1
#SBATCH --nodes=1
#SBATCH --ntasks=1
#SBATCH --ntasks-per-node=1
#SBATCH --cpus-per-task=1
#SBATCH --output=test_dgemm_ravindra.out.%j
#SBATCH --error=test_dgemm_ravindra.err.%j
#SBATCH --time=00:20:00


module load CUDA/11.0 NVHPC/20.11-GCC-9.3.0 
./test_dgemm

Non-thunking (user defined data allocation, movement, and deallocation)

        program example_sgemm
        real, dimension(:,:),allocatable:: A(:,:),B(:,:),C(:,:)
        integer*8:: devPtrA, devPtrB, devPtrC
        integer:: n=20480, size_of_real=16
        allocate (A(n,n),B(n,n),C(n,n))

        call cublas_Alloc(n*n,size_of_real, devPtrA)
        call cublas_Alloc(n*n,size_of_real, devPtrB)
        call cublas_Alloc(n*n,size_of_real, devPtrC)

        ! Initialize A, B and C
        A = 0.0d0; forall(i = 1:n) A(i,i) = 1.0 
        B = 0.0d0; forall(i = 1:n) B(i,i) = 1.0
        C = 0.0d0; forall(i = 1:n) C(i,i) = 1.0


        ! Copy data to GPU
        call cublas_Set_Matrix(n,n,size_of_real,A,n,devPtrA,n)
        call cublas_Set_Matrix(n,n,size_of_real,B,n,devPtrB,n)
        call cublas_Set_Matrix(n,n,size_of_real,C,n,devPtrC,n)

        ! Call SGEMM in CUBLAS library
        call cublas_SGEMM('n','n', n,n,n,1.,devPtrA,n,devPtrB,n,1.,devPtrC,n)

        ! Copy data from GPU
        call cublas_Get_Matrix(n,n,size_of_real,devPtrC,n,C,n)

        print *,c(1,1)

        call cublas_Free(devPtrA)
        call cublas_Free(devPtrB)
        call cublas_Free(devPtrC)

        end program example_sgemm

Conditional execution of jobs using PBS or slurm

Wed, 16 Nov 2016 00:00:00 +0000

It is possible to start a job on the condition that another one completes beforehand; this may be necessary for instance if the input to one job is generated by another job. Job dependency is defined in PBS using the -W flag.

To illustrate with an example, suppose you need to start a job using the script second_job.sh after another job finished successfully. Assume the first job is started using script first_job.sh and the command to start the first job

qsub first_job.sh

returns the job ID 7777. Then, the command to start the second job is

qsub -W depend=afterok:7777 second_job.sh

This job dependency can be further automated (possibly to be included in a bash script) using environment variables:

JOB_ID_1=qsub first_job.sh

JOB_ID-2=qsub -W depend=afterok:$JOB_ID_1 second_job.sh

Find and delete all files with specific extension linux

Thu, 09 Jun 2016 00:00:00 +0000

To list all the files with an extention .log

find . -name "*.log" -type f -print0 | xargs -0 /bin/ls

To remove all the files with an extention .log

find . -name "*.log" -type f -print0 | xargs -0 /bin/rm