Harvard Research Computing Notes

June 2022

Overall intro docs

https://docs.rc.fas.harvard.edu/kb/quickstart-guide/

To create an RC ticket:
go to https://portal.rc.fas.harvard.edu/rcrt/submit_ticket

To use the computing cluster, need to be on their VPN, which is vpn.rc.fas.harvard.edu

user=stubbs@fasrc with appropriate FARC password

pwd plus Google authenticator dual factor from iphone

then there is a web interface:
https://vdi.rc.fas.harvard.edu/pun/sys/dashboard

username stubbs

same FASARC password

about modules:
https://docs.rc.fas.harvard.edu/kb/modules-intro/

installing software:

SELF SERVICE: Please be aware that in many cases you can install software for your own use on the cluster, either directly to your home directory/lab storage or via a container.

See our documentation:
Installing for Yourself: https://docs.rc.fas.harvard.edu/kb/installing-software-yourself/
Singularity Containers: https://docs.rc.fas.harvard.edu/kb/singularity-on-the-cluster/
Python/Anaconda: https://docs.rc.fas.harvard.edu/kb/python/
R Packages: https://docs.rc.fas.harvard.edu/kb/r-packages/
Module Search for Existing Software: https://portal.rc.fas.harvard.edu/apps/modules

searching for modulew:
https://portal.rc.fas.harvard.edu/apps/modules

moving data with scp
https://docs.rc.fas.harvard.edu/kb/copying-data-to-and-from-cluster-using-scp/

from summit at Pachon, need both tunnelblik and fasrc VPNs to be active.

scp foo.dat stubbs@login.rc.fas.harvard.edu:~/

The best answer depends on a couple of factors:
If you have small(ish) amounts of data that you wish to scp/rsync/sftp you can connect directly to a login node, preferably one in the same data center as your storage, so:

So for /n/holystore01/LABS/stubbs_lab you'd connect to one of these:
holylogin01.rc.fas.harvard.edu
holylogin02.rc.fas.harvard.edu
holylogin03.rc.fas.harvard.edu
holylogin04.rc.fas.harvard.edu

If, on the other hand, you have a 3rd party asking for an IP range to whitelist, you should tell them:
140.247.111.0/28
This is probably not the case, but if it is please let me know.
For large amounts of data scp/rsync are fraught with potential pitfalls and now your best option. For anything over, say, 50-100GB we do not recommend using direct connects like that over the internet, but to use Globus instead. See: https://docs.rc.fas.harvard.edu/kb/globus-file-transfer/
This can be done using either endpoints at [remote institution which supports Globus] and FASRC endpoint, or using a personal endpoint to FASRC endpoint.

/n/holystore01/LABS/stubbs_lab is accessible from Globus under the 'Harvard FAS RC Holyoke' collection.

Storage

scratch space is at
/n/holyscratch01/stubbs_lab

backed-up storage (40 TB) is at
/n/holystore01/LABS/stubbs_lab

even better quality storage (4 TB) at
/n/stubbs_lab

The holystore01 storage is Tier 0 and the /n/stubbs_lab storage is Tier 1 and is more secure: https://www.rc.fas.harvard.edu/services/data-storage/

9/15/2022: holystore01 is full (45 TB in a 40 TB drive), /n/stubbs_lab has about 3.6 TB left.

These are the commands used to get information on space:

[root@holy7c22501 ~]# lfs quota -hg stubbs_lab /n/holystore01
Disk quotas for grp stubbs_lab (gid 34726):
      Filesystem    used   quota   limit   grace   files   quota limit
grace
  /n/holystore01 45.14T*    40T     40T       - 80749572* 35651584
35651584       -
[root@holy7c22501 ~]# df -h /n/stubbs_lab
Filesystem                              Size Used Avail Use% Mounted on
holy-nfsisilon:/ifs/rc_labs/stubbs_lab 4.0T 430G 3.6T 11%
/net/holy-nfsisilon/ifs/rc_labs/stubbs_lab

Common toolkit (Updated 02/22/2023)

New write up: (One time setup)

VPN onto the FASRC system
ssh onto the login server USERNAME@login.rc.fas.harvard.edu
load Anaconda: module load Anaconda3
load other modules needed: module load fftw bzip2 cmake cfitsio
open file : nano ~/.conda/environments.txt
add line: /n/holylabs/LABS/stubbs_lab/Lab/python/LSST_DESC
save the file and exit from nano
activate the existing conda environment source activate /n/holylabs/LABS/stubbs_lab/Lab/python/LSST_DESC

You should now be in a conda environment that has 'pyccl' and 'galsim' installed

Accessing the JupyterLab via the VDI interface (Have to do every time you open a new instance)

click on 'Jupyter notebook/jupyterlab' (WARNING: don't click on "JupyterLab (scipy-notebook)")
Before pressing launch find the place where it says Full path of script to be executed before launching jupyter (Optional), here add /n/holylabs/LABS/stubbs_lab/Lab/python/desc.sh
Launch the notebook, once you get into the notebook you should be able to select a kernel called something with LSST_DESC

cut-and-pastable:

module load Anaconda3

nano ~/.conda/environments.txt

/n/holylabs/LABS/stubbs_lab/Lab/python/LSST_DESC

go to https://vdi.rc.fas.harvard.edu/

click on jupyternotebook / JupyterLab
Before pressing launch find the place where it says Full path of script to be executed before launching jupyter (Optional) here add /n/holylabs/LABS/stubbs_lab/Lab/python/desc.sh
Launch the notebook, once you get into the notebook there should hopefully be an option to select a conda environment related to LSST_DESC.

module load sextractor/2.19.5-fasrc01

To run Source Extractor on cluster:

module load centos6/0.0.1-fasrc01 intel/17.0.4-fasrc01 gcc/8.2.0-fasrc01 intel/18.0.5-fasrc01 intel/19.0.5-fasrc01

module load centos6/0.0.1-fasrc01 intel/17.0.4-fasrc01 openmpi/4.0.1-fasrc01 gcc/8.2.0-fasrc01 openmpi/4.0.1-fasrc01