In addition to running BioDynaMo on standard computers, users can also utilise the BioDynaMo frame work on high performance computers (HPCs). Due to many HPCs having higher security than standard computers however it is likely that a user wishing to do this will need to build BioDynaMo within a container on the chosen HPC.
For this we have chosen to Singularity. Singularity is an open source programme for operating-system-level virtualization. Singularity works similar to other popular container programmes such as Docker. However, due to its higher level of security it has seen more widespread uptake by HPCs and we thus believe the best choice for users.
To create our singularity image for BioDynaMo we can use on of two methods.
Method 1, directly building on the HPC.
For the first method, we directly import the biodynamo Singularity file from our local PC to the desired HPC as follows.
Firstly, we must scp into the desired HPC from our local PC. This can be done directly through the terminal as follows.
ssh -x [HPC.ADDRESS]We then must secure copy the BioDynaMo singularity file from the chosen PC to the local HPC.
scp [user@]SRC_LOCALPC/SingularityFileLocation:]Singularity ~/From here we can build BioDynaMo utilising the fake root feature of singularity.
singularity build --fakeroot Singularity.sif SingularityFakeroot is not needed if you have sudo rights on the HPC but this is highly unlikely for most individuals. In that case however we simply amend the above to:
sudo singularity build Singularity.sif SingularityThen simply run the BioDynaMo singularity container using:
singularity run --cleanenv Singularity.sifwhere the --cleanenv option clears out all the environment variables set in the host machine to avoid conflicts.
Once the container is running we can source Root and the BioDynaMo environment:
source /biodynamo/build/third_party/root/bin/thisroot.shsource /biodynamo/build/bin/thisbdm.shMethod 2, Export image from local PC to HPC.
If you cannot utilise either sudo or fakeroot on the HPC, we can instead build the BioDynaMo image on our local PC and export the image. This works similar to method 1 but with a change in order. Firstly, we must build our image on the local PC with sudo rights:
sudo singularity build Singularity.sif SingularityThen ssh into the HPC.
ssh -x [HPC.ADDRESS]secure copy the image from our PC to the local HPC.
scp [user@]SRC_LOCALPC/SingularityFileLocation:]Singularity.sif ~/Finally we once again run the Singularity image on the HPC:
singularity run --cleanenv Singularity.sifwhere the --cleanenv option clears out all the environment variables set in the host machine to avoid conflicts.
Once the container is running we can source Root and the BioDynaMo environment:
source /biodynamo/build/third_party/root/bin/thisroot.shsource /biodynamo/build/bin/thisbdm.shIf you wish to read further about Singularity, you can find a substantial amount of information on the singularity home website: https://sylabs.io/guides/3.5/user-guide/introduction.html. Including many tutorials and more complex use cases.
Exporting visualisations on HPC's
By default, BioDynaMo uses ParaView for visualization, which needs an X-display server to function. Since the Singularity image does not provide a graphical desktop environment and hence no X-server, there are two workarounds.
Option 1: Using xvfb
xvfb stands for X virtual frame buffer and emulates the required display server. Start a virtual frame buffer with:
/biodynamo/util/xvfb-initd.sh startThe buffer can be managed with the arguments start|stop|restart
Note: Some systems may require display forwarding before starting the frame buffer. This can be achived through export DISPLAY=0.
Option 2: Setting Paraview to work in a headless mode using the CMAKE flag
Paraview with the osmesa or EGL backend does not require an X-server. To change the backend, you have to edit the cmake options in [1], build paraview, and replace the ParaView version that BioDynaMo installs. See also [2].
[2] https://discourse.paraview.org/t/pvpython-offscreen-rendering-segfault/246/3