Table of Contents

1. General introduction

...

/n/sunderland_lab/Lab/MITgcm/

Note: Do NOT copy the verification folder, it takes up huge disk space.

In your ~username home directory make an MITgcm directory and copy all of the folders except verification into your MITgcm directory from the Lab copies. For example:

cd

mkdir MITgcm

cd MITgcm

cp -r /n/sunderland_lab/Lab/MITgcm/bin/ .

cp -r /n/sunderland_lab/Lab/MITgcm/doc/ .

cp -r /n/sunderland_lab/Lab/MITgcm/eesupp/ .

...etc.!

For users outside this group, we are currently working on a Github site.

...

1. Need to comment out all calls to ALLOW_CAL in pcb_fields_load.F
2. In gchem_init_fixed.F, you need to make sure you have the line: CALL PCB_PARM. Yanxu got rid of his hg_parms.F file, so a CALL HG_PARM line is missing from his gchem_initi_fixed.F file. The PCB simulation still has a pcb_parms.F file and if it isn't "turned on" by calling it from gchem_init_fixed.F, then your output will be all NaNs. 
3. Use online wind, ice, and solar radiation information from ECCOv4. In ECCOv1, we read wind, ice, and radiation from an offline file (e.g., archived from MERRA or GEOS-5). Now those variables are generated online. You need to do two things to activate this capability:
   1. Add "#ifdef USE_EXFIWR" statements to your package. The easiest way to do this is to search "USE_EXFIWR" in the HG code (

      /n/sunderland_lab/MITgcm/pkg/hg/) and copy these to your own code. 

   2. After adding the "#ifdef USE_EXFIWR" statements to your package, you need to update the names of your ice, wind, and radiation variables. You probably need to do this if your code has air-sea exchange, ice interactions, or photochemistry. In pcba_surfforcing.F, which handles air-sea exchange, I had to replace wind(i,j,bi,bj)with windo and fIce(i,j,bi,bj) with ice. If you haven't done this properly, your PTRACER output might have big blocks missing, like this:

...

Here we will set up the following directories within your ~username/MITgcm/verification/global_hg_llc90/ directory:

code: Header/option or other files that are often modified.

...

Before compiling the code, you need to obtain the content of code/ directory. Copy all the files in /n/sunderland_lab/Lab/MITgcm/verification/global_hg_llc90/code/ :

cd ~username/MITgcm/verification/global_hg_llc90/

cp -r /n/sunderland_lab/Lab/MITgcm/verification/global_hg_llc90/code/ ./

Lastly, make empty build/ and run/ directories within your ~username/MITgcm/verification/global_hg_llc90/ directory:

cd ~username/MITgcm/verification/global_hg_llc90/ 

mkdir build

mkdir run

If you are running the Hg simulation, you should be all set. If you are running a different simulation (e.g., PCBs or PFOS) and only using Hg as a template, you need to modify:

...

Before compiling, you also need to load the proper compilers for the code (MPI, intel Fortran etc.). You can also add this to your ~/.bashrc file (see /n/home09/hmh/.bashrc_sunderland for an example). IMPORTANT: Odyssey is constantly updating its "modules" and will be phasing out the "old" module system that is used below. For each set of modules (intel Fortran compiler, MPI, netcdf) loaded, you will need to refer to a specific file within your "build_options" directory, and may need to edit the file paths within it to match the location of the modules on Odyssey. See Appendix for lists of combinations of modules that work with MITgcm ECCOv4 and their corresponding build_options files (which will be attached to this wiki).

*****Also, a WARNING: the MITgcm Hg simulation has NOT successfully run with this module configuration below on Odyssey. the error is described in section 7, here. This page will be updated as more is understood, but in the meantime please use the modules and optfile listed here, under section (b) ('used by Chris Horvat'), which have been tested and do run on Odyssey. *****

module load hpc/openmpi-intel-latest

module load hpc/netcdf-3.6.3

Then let's go to the build/ directory and build your Makefile:

cd ~username/MITgcm/verification/global_hg_llc90/build

First, build the Makefile. Note: the "-optfile" filename below (and contents) will need to be changed if you have to load different module versions than the specific ones listed above.

make clean        Note: this is needed if you change which modules are loaded and/or the optfile

../../../tools/genmake2 -mods=../code -optfile=../../../tools/build_options/linux_ia64_ifort+mpi_harvard3      

...

Copy these folders to ~username/MITgcm/verification/

cd ~username/MITgcm/verification/

cp -r /n/sunderland_lab/Lab/MITgcm/verification/global_oce_cs32/ ./

cp -r /n/sunderland_lab/Lab/MITgcm/verification/global_oce_input_fields/ ./

...

cd ~username/MITgcm/verification/global_hg_llc90/run

ln -s /n/sunderland_lab/Lab/eccov4_input/controls/* .

ln -s /n/sunderland_lab/Lab/eccov4_input/MITprof/* .

ln -s /n/sunderland_lab/Lab/eccov4_input/pickups/* .

ln -s /n/sunderland_lab/Lab/eccov4_input/era-interim/* .

...

Still in your run/ directory, make a control/ subdirectory:

mkdir control

Move all the control files into this folder

mv xx_* control/

cp /n/sunderland_lab/Lab/MITgcm/verification/global_hg_llc90/run/control/wt_* control/

6.6 data* files

If you're running an Hg simulation, copy data* files to your run/ directory from here:

cp /n/sunderland_lab/Lab/MITgcm/verification/global_hg_llc90/run/data* .

...

You should keep all your optfiles within your directory: ~username/MITgcm/tools/build_options/ .

1. Old module system:

a. Standard, as written in instructions above. optfile is already in your build_options directory if you follow the instructions to copy folders from sunderland_lab

module load hpc/openmpi-intel-latest

module load hpc/netcdf-3.6.3

../../../tools/genmake2 -mods=../code -optfile=../../../tools/build_options/linux_ia64_ifort+mpi_harvard3

Note: this will load the following versions: intel compiler 13.0.079; openmpi 1.6.2. 

          b. used by Chris Horvat. Download optfile by clicking here. use scp to copy this file to Odyssey, then mv into build_options directory.  

module load centos6/openmpi-1.7.2_intel-13.0.079

module load centos6/netcdf-4.3.0_intel-13.0.079

../../../tools/genmake2 -mods=../code -optfile=../../../tools/build_options/linux_amd64_ifort_mpi_odyssey2 -mpi -enable=mnc

2. New module system:

Download optfile by clicking here.

...

1.Load Lmod, Odyssey's new module system. at the command line, in any directory, enter:

source new-modules.sh

2. Load intel compiler:

module load intel/13.0.079-fasrc01

3. Find out which modules are compatible with this intel version:

module avail

Right now, the list looks something like this:

openmpi/1.6.5-fasrc01 

openmpi/1.8.1-fasrc01

openmpi/1.8.3-fasrc01

netcdf/3.6.3-fasrc01

This means you can choose any of the 3 openmpi versions, but there is only one compatible netCDF version.

 4. Load your openmpi module of choice and netCDF module. As an example, here we'll choose openmpi 1.6.5.

module load openmpi/1.6.5-fasrc01

module load netcdf/3.6.3-fasrc01

 5. Find out what the filepaths are for these modules:

printenv

Now look for  "LD_LIBRARY_PATH" and "CPATH" (search within the terminal window). For the modules above, it should look something like this:

LD_LIBRARY_PATH=/n/sw/fasrcsw/apps/Comp/intel/13.0.079-fasrc01/netcdf/3.6.3-fasrc01/lib64:/n/sw/fasrcsw/apps/Comp/intel/13.0.079-fasrc01/openmpi/1.6.5-fasrc01/lib:/n/sw/intel_cluster_studio-2013/lib/intel64:/lsf/7.0/linux2.6-glibc2.3-x86_64/lib

CPATH=/n/sw/fasrcsw/apps/Comp/intel/13.0.079-fasrc01/netcdf/3.6.3-fasrc01/include:/n/sw/fasrcsw/apps/Comp/intel/13.0.079-fasrc01/openmpi/1.6.5-fasrc01/include:/n/sw/intel_cluster_studio-2013/composerxe/include/intel64:/n/sw/intel_cluster_studio-2013/composerxe/include

6. Create a new optfile, by making a copy of a previous one, within your ~username/MITgcm/tools/build_options/ directory.

cd ~username/MITgcm/tools/build_options/

cp linux_ia64_ifort+mpi_harvard3 linux_ia64_ifort+mpi_harvard_test             (just an example, can change filename to whatever you want)

7. Open the file you've just copied (e.g., with emacs, nano, vi, or whatever text editor), and look for the following lines, which you will want to edit (Note, they may be slightly different, this is an example): 

INCLUDES='-I/n/sw/openmpi-1.6.2_intel-13.0.079/include -I/n/sw/intel_cluster_studio-2013/mkl/include'

...

cd /n/sw/fasrcsw/apps/Comp/intel/13.0.079-fasrc01/openmpi/1.6.5-fasrc01/include

5. More information on Odyssey modules & useful commands:

https://rc.fas.harvard.edu/resources/documentation/software-on-odyssey/modules/

module purge - clears all loaded modules

module list - shows currently loaded modules