It can be quite annoying to install software on a hpc without having sudo rights. I’ve had the experience installing packages for R can be especially be frustrating, as you’ll get error messages like:

* installing *source* package ‘V8’ ...
** package ‘V8’ successfully unpacked and MD5 sums checked
** using staged installation
Using PKG_CFLAGS=-I/usr/include/v8 -I/usr/include/v8-3.14
Using PKG_LIBS=-lv8 -lv8_libplatform
-----------------------------[ ANTICONF ]-------------------------------
Configuration failed to find the libv8 engine library. Try installing:
 * deb: libv8-dev or libnode-dev (Debian / Ubuntu)
 * rpm: v8-devel (Fedora, EPEL)
 * brew: v8 (OSX)
 * csw: libv8_dev (Solaris)
To use a custom libv8, set INCLUDE_DIR and LIB_DIR manually via:
R CMD INSTALL --configure-vars='INCLUDE_DIR=... LIB_DIR=...'
---------------------------[ ERROR MESSAGE ]----------------------------
<stdin>:1:16: fatal error: v8.h: No such file or directory
compilation terminated.
------------------------------------------------------------------------
ERROR: configuration failed for package ‘V8’
* removing ‘/home/pol.van-rijn/R/x86_64-redhat-linux-gnu-library/3.6/V8’

or

Error: package or namespace load failed for ‘reshape2’ in dyn.load(file, DLLpath = DLLpath, ...):
 unable to load shared object '/opt/conda/lib/R/library/stringi/libs/stringi.so':
  libicui18n.so.64: cannot open shared object file: No such file or directory

Those messages initially made me believe you need to install those packages system wide, however the trick here is to install them using Conda.

Here are the steps to do it:

1. Make sure anaconda is available: module load conda

2. Create a new environment: conda create --name r-3-6-3

3. Activate you environment conda activate r-3-6-3

4. Install R: conda install -c r r

   It is very important to install the packages using Conda and not in R, as you can get errors like these

   installing via 'install.libs.R' to /home/mpg08/pvanrij/.conda/envs/v8/lib/R/library/00LOCK-stringi/00new/stringi                                                                                                       
   ** R                                                                                                                                                                                                                   
   ** inst                                                                                                                                                                                                                
   ** byte-compile and prepare package for lazy loading                                                                                                                                                                   
   ** help                                                                                                                                                                                                                
   *** installing help indices                                                                                                                                                                                            
   ** building package indices                                                                                                                                                                                            
   ** testing if installed package can be loaded from temporary location                                                                                                                                                  
   Error: package or namespace load failed for ‘stringi’ in dyn.load(file, DLLpath = DLLpath, ...):                                                                                                                       
    unable to load shared object '/home/mpg08/pvanrij/.conda/envs/v8/lib/R/library/00LOCK-stringi/00new/stringi/libs/stringi.so':                                                                                         
     /home/mpg08/pvanrij/.conda/envs/v8/lib/R/library/00LOCK-stringi/00new/stringi/libs/stringi.so: undefined symbol: _ZTIN6icu_588ByteSinkE                                                                              
   Error: loading failed                                                                                                                                                                                                  
   Execution halted                                                                                                                                                                                                       
   ERROR: loading failed                                                                                                                                                                                                  
   * removing ‘/home/mpg08/pvanrij/.conda/envs/v8/lib/R/library/stringi’ 
   

5. Install v8: conda install -c conda-forge r-v8

6. Install brms: conda install -c conda-forge r-brms

7. Now start R and run a very simple model

   (r-3-6-3) bash-4.2$ R                                                                                                                                                                                                       
                                                                                                                                                                                                                             
   R version 3.6.3 (2020-02-29) -- "Holding the Windsock"                                                                                                                                                                 
   Copyright (C) 2020 The R Foundation for Statistical Computing                                                                                                                                                          
   Platform: x86_64-conda_cos6-linux-gnu (64-bit)                                                                                                                                                                         
                                                                                                                                                                                                                             
   R is free software and comes with ABSOLUTELY NO WARRANTY.                                                                                                                                                              
   You are welcome to redistribute it under certain conditions.                                                                                                                                                           
   Type 'license()' or 'licence()' for distribution details.                                                                                                                                                              
                                                                                                                                                                                                                             
     Natural language support but running in an English locale                                                                                                                                                            
                                                                                                                                                                                                                             
   R is a collaborative project with many contributors.                                                                                                                                                                   
   Type 'contributors()' for more information and                                                                                                                                                                         
   'citation()' on how to cite R or R packages in publications.                                                                                                                                                           
                                                                                                                                                                                                                             
   Type 'demo()' for some demos, 'help()' for on-line help, or                                                                                                                                                            
   'help.start()' for an HTML browser interface to help.                                                                                                                                                                  
   Type 'q()' to quit R.                                                                                                                                                                                                  
                                                                                                                                                                                                                             
   > library(brms)                                                                                                                                                                                                        
   Loading required package: Rcpp                                                                                                                                                                                         
   Loading 'brms' package (version 2.13.5). Useful instructions                                                                                                                                                           
   can be found by typing help('brms'). A more detailed introduction                                                                                                                                                      
   to the package is available through vignette('brms_overview').                                                                                                                                                         
                                                                                                                                                                                                                             
   Attaching package: ‘brms’                                                                                                                                                                                              
                                                                                                                                                                                                                             
   The following object is masked from ‘package:stats’:                                                                                                                                                                   
                                                                                                                                                                                                                             
       ar                                                                                                                                                                                                                 
      
   

   Do not only load the arms package, but also make sure it runs. You can test it using the example:

   fit1 <- brm(count ~ zAge + zBase * Trt + (1|patient), 
               data = epilepsy, family = poisson())
   

8. And repeat the same steps on the compute node to double check if it works on there too.

It can happen you get errors, because Rcpp and rstan need to be compiled with the same compiler. If so, install Rcpp and rstan again using install.packages('Rcpp') and install.packages('rstan')

If you get errors about igraph, install it via Conda: conda install -c conda-forge igraph.

Click here to close

In this tutorial, we’ll cover how to setup remote debugging with the Pycharm IDE.

Motivation

First of all, why should you bother setting this up? When writing code, it can be practical to the code debug in an interactive fashion. This debugging can be hard if you run code remotely and need some special hardware like GPUs. Using interactive slurm sessions you can request such resources (e.g. srun –pty -p gpu -N 1 -n 4 /bin/bash requests access to a GPU and opens a bash, partition name depends on your hpc). You will be able to debug existing code by starting a python session and copy paste all lines of code. However, this approach comes with some downsides:

• you will need to copy paste every line,
• your code will not stop executing if an error occurs,
• you’ll have to take care of re-indenting intended code (see example below),

• line breaks in code that will cause errors,
• and many more annoying things…

A remote debugger will allow you to execute code until a certain point (aka breakpoint). From this moment on you’ll be able to interact with the code and inspect variables conveniently from your IDE.

Setting up the remote debugger

The first step is to open up PyCharm and go to Edit configurations…, see screenshot below

Now you need to add a new configuration, select Python remote debug

Now you need to fill in the following fields:

1. Give it a name, I called it HPC
2. Note the Pycharm version you are using, I have version 193.6494.30
3. Select localhost
4. Now pick a port, I use 8223 in this example. You need to make sure that this port is available
5. Finally click on (5) and make sure the mapping between your local files and remote files is correct

Now open a new terminal window (A) and

• connect to the HPC, e.g. ssh GWDG (or whatever alias or login you have to the hpc, e.g. ssh user@domain.com)
• load conda module load conda (you might need to load conda first)
• create a new environment if it does not already exist, conda create –name pycharm_demo
• activate the environment, e.g. source activate pycharm_demo
• install the right version of pydevd-pycharm (you noted down the Pycharm version in the last step). In my case, I run pip install pydevd-pycharm~=193.6494.30

Set a breakpoint

Go to a file you want to pause execution. On the top of the file insert

import pydevd_pycharm

In the line you want to stop, add the following line. Make sure you set the correct port here (I use the port 8223 here):

pydevd_pycharm.settrace('localhost', port=8223, stdoutToServer=True, stderrToServer=True)

Make sure the file is saved on the HPC server. In my case I made a file called test.py that contains the following contents:

import pydevd_pycharm
a = 1
pydevd_pycharm.settrace('localhost', port=8223, stdoutToServer=True, stderrToServer=True)
raise Exception()

Request required resources

The next step is to go back to terminal A and request for the required resources, e.g.:

srun --pty /bin/bash

The next step is to note the host name, you find this by running:

hostname

This will give you a host name, in this example it is gwdd173.

Setting up the ssh tunnel

Open a new terminal window (B) and type:

HOST=gwdd173 # add your host name
PORT=8223 # add your port
USER=pvanrij # change your username
ssh -J GWDG -R $PORT:localhost:$PORT $USER@$HOST

If the -J GWDG option does not work, you may have an older version of the ssh client. In that case you can try -o “ProxyCommand ssh GWDG -W %h:%p”.

Start the debugger

Go to Pycharm and click on the bug to start the debugger

Execute the program

Go back to terminal A and make sure you still have the right Conda environment activated (in my case pycharm_demo). Navigate to the folder that contains the file you want to pause and run:

python test.py

Now your code should stop and you can nicely debug your code

Click here to close

English version: Disclaimer

Last updated: September 07, 2018

The information contained on http://www.pol.works website (the “Service”) is for general information purposes only.

Pol van Rijn assumes no responsibility for errors or omissions in the contents on the Service.

In no event shall Pol van Rijn be liable for any special, direct, indirect, consequential, or incidental damages or any damages whatsoever, whether in an action of contract, negligence or other tort, arising out of or in connection with the use of the Service or the contents of the Service. Pol van Rijn reserves the right to make additions, deletions, or modification to the contents on the Service at any time without prior notice. This Disclaimer for Pol van Rijn is managed by TermsFeed Disclaimer Generator

Pol van Rijn does not warrant that the website is free of viruses or other harmful components.

External links disclaimer http://www.pol.works website may contain links to external websites that are not provided or maintained by or in any way affiliated with Pol van Rijn

Please note that the Pol van Rijn does not guarantee the accuracy, relevance, timeliness, or completeness of any information on these external websites.

Deutsche Fassung: Impressum

Angaben gemäß § 5 TMG:

Pol van Rijn

Damaschkeanger 28

60488 Frankfurt

Kontakt

E-Mail: hello@polvanrijn.de

Haftung für Inhalte

Als Diensteanbieter sind wir gemäß § 7 Abs.1 TMG für eigene Inhalte auf diesen Seiten nach den allgemeinen Gesetzen verantwortlich. Nach §§ 8 bis 10 TMG sind wir als Diensteanbieter jedoch nicht verpflichtet, übermittelte oder gespeicherte fremde Informationen zu überwachen oder nach Umständen zu forschen, die auf eine rechtswidrige Tätigkeit hinweisen. Verpflichtungen zur Entfernung oder Sperrung der Nutzung von Informationen nach den allgemeinen Gesetzen bleiben hiervon unberührt. Eine diesbezügliche Haftung ist jedoch erst ab dem Zeitpunkt der Kenntnis einer konkreten Rechtsverletzung möglich. Bei Bekanntwerden von entsprechenden Rechtsverletzungen werden wir diese Inhalte umgehend entfernen.

Haftung für Links

Unser Angebot enthält Links zu externen Websites Dritter, auf deren Inhalte wir keinen Einfluss haben. Deshalb können wir für diese fremden Inhalte auch keine Gewähr übernehmen. Für die Inhalte der verlinkten Seiten ist stets der jeweilige Anbieter oder Betreiber der Seiten verantwortlich. Die verlinkten Seiten wurden zum Zeitpunkt der Verlinkung auf mögliche Rechtsverstöße überprüft. Rechtswidrige Inhalte waren zum Zeitpunkt der Verlinkung nicht erkennbar. Eine permanente inhaltliche Kontrolle der verlinkten Seiten ist jedoch ohne konkrete Anhaltspunkte einer Rechtsverletzung nicht zumutbar. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Links umgehend entfernen.

Urheberrecht

Die durch die Seitenbetreiber erstellten Inhalte und Werke auf diesen Seiten unterliegen dem deutschen Urheberrecht. Die Vervielfältigung, Bearbeitung, Verbreitung und jede Art der Verwertung außerhalb der Grenzen des Urheberrechtes bedürfen der schriftlichen Zustimmung des jeweiligen Autors bzw. Erstellers. Downloads und Kopien dieser Seite sind nur für den privaten, nicht kommerziellen Gebrauch gestattet. Soweit die Inhalte auf dieser Seite nicht vom Betreiber erstellt wurden, werden die Urheberrechte Dritter beachtet. Insbesondere werden Inhalte Dritter als solche gekennzeichnet. Sollten Sie trotzdem auf eine Urheberrechtsverletzung aufmerksam werden, bitten wir um einen entsprechenden Hinweis. Bei Bekanntwerden von Rechtsverletzungen werden wir derartige Inhalte umgehend entfernen. Quelle

Click here to the disclaimer