Running the MITgcm PCB Simulation on Odyssey

Step 0: Initial Setup

Sign up for a free bitbucket.org account. Once you have an account, email Colin Thackray (thackray @ g.harvard.edu) to be added to the Harvard BGC group on Bitbucket. All users of the Harvard BGC group will automatically have access to the MITgcm_code repository.

If you have not done so already, download the set of Unix environment scripts that are customized for use with the MITgcm. To obtain the scripts, type the following commands:

cd /n/sunderland_lab/Lab/environment
./install

This will copy a number of startup files to your home folder (~/) and your ~/bin folder. Files that you already have will not be over-written.

Step 1: Set up an interactive session

You should always use an interactive session for any work when working on Odyssey rather than work on the login node. Working on the login node reduces your available memory and generally slows down the network for other users.

You can use the "interactive" script (downloaded to your bin/ directory during initial setup) to request an interactive session. The requested number of CPUS, nodes, memory (MB), time duration (min), and partition are all configurable. For example, to start an 8-hr interactive session with 8 CPUs, 1 node, and 4 GB on the interact partition, type:

interactive 8 1 4000 480 interact  

You should request at least 3 hours for your interactive session when first setting up with MITgcm since the initial input download will take over an hour. If you plan on running MITgcm interactively rather than submit your job to slurm, you should request a multiple of 13 cores.

Once your interactive session starts up, we recommend that you also run the awake.sh script in the background (type 'awake.sh &'). This will periodically print. a blank line to keep your terminal from freezing. Alternatively, place awake.sh into your .my_personal_settings file and configure your .bashrc to source that file. It will then run every time you start an interactive session.

Step 2: Download the "MITgcm_code" repository

 First, navigate to  your /n/regal/sunderland_lab/your_user_name folder since you must store the MITgcm_code directory within your regal directory for maximum performance of the model. Next, navigate to whatever subdirectory in your regal directory you want to work in. Finally, type the following commands to download the MITgcm_code repository.

git clone https://bitbucket.org/harvard-bgc/mitgcm_code MITgcm_code


Step 3: Set up the initial PCB run directory 

 Change directories into MITgcm_code which you just cloned. Execute shell script make_initial_pcb_rundir.sh. Note that downloading input files from the Sunderland_lab space to your regal directory will take a while. For this reason, it is only done once and additional run directories that you set up later in the last step below will symbolically link to files in this directory.

Pay attention to the text output to learn what the script is doing. After it is complete instructions are printed out to clone the pkg/ source code repository developed by Helen Amos. Follow these instructions before proceeding.  

The MITgcm_code directory will serve as the local storage location for all MITgcm input files, including those used in additional MITgcm directories that you create. For this reason, once you clone it and set it up using the instructions above, do not rename it or move it. Due to the 90-day storage retention on Odyssey, you will need to periodically perform this setup step again in the future to prevent the files from being deleted.

Step 4: Download the "hamos_mitgcm_pkg" repository

Instructions for downloading Helen's Amos' source code used for PCB simulations are printed at the end of make_initial_pcb_rundir.sh executation. If you did not follow those instructions, do them now. To clone the repository, which you should store in MITgcm_code/pkg, type the below code. All PCB-specific source code is in the pkg/pcb/ directory.

git clone https://bitbucket.org/harvard-bgc/hamos_mitgcm_pkg pkg

Step 5: Set the PCB congener

Execute python script change_pcb_congener.py to set your congener in config files, source code (pcb_param.F), and run scripts. For example, "./change_PCB_congener.py PCB101". If you add any congener-specific source code outside of pcb_param.F, make sure that you incorporate setting the correct settings into change_PCB_congener.py.

Step 6: Compile the source code

Execute shell script compile_pcb.sh and follow the instructions at the prompt. For fast runs you should choose 96 CPUs and accept copying Helen Amos' files.

Step 7: Create a run script or edit an existing run script

Run scripts are provided for Helen Amos' stages 1, 2, and 3. Open the run script and edit line 9 to use your own email address. Look over the "User-specific input" section to make sure the settings are as you wish. Note that the three stages are meant to be run in series. If you are not starting at stage 1 then you must manually copy the ptracers pickup files into the MITgcm_code/verification/global_pcb_llc90/run/ directory. You should use the pickup files with the largest number (representing timestep) found in the stage 2 run 15 results directory of your choosing. When you copy the files (one ending in .data and another ending in .meta), rename them as pickup_ptracers.000000001.data and pickup_ptracers.0000000001.meta.

Step 8: Submit the run to SLURM

Submit your run by typing "sbatch your_run_script".

NOTE: If you decide to run MITgcm interactively rather than submitting a job to SLURM then you must source the .bashrc_mitgcm file prior to running.

IMPORTANT NOTE: Files on the /n/regal/ disk are purged every 90 days to save disk space. You therefore should keep a backup copy of all MITgcm results you wish to keep long-term in your Sunderland group lab space folder (/n/sunderland_lab/Lab/Users/YOUR-USER-NAME).

Step 9: Create additional PCB run directories

Once you have one run directory you can very quickly create new ones without having the re-download all of the input files. This saves a lot of time and resources. Go back to step 2 and clone the repository to a new directory with a different name than your first one. Make sure it is stored in the same directory as your original clone "MITgcm_code". Change directories into your new clone, and execute the shell script make_new_pcb_rundir.sh to set it up. Additional directories can have descriptive names of what you plan on doing with them, e.g. MITgcm_PCB28_{update_description}. Follow the instructions prompted by the script to complete the setup. Then continue through steps above to get the source code, change congeners, compile, and run.

Step 10: Clean an existing run directory for re-use

If you would like to re-use a run directory it is a good idea to remove all files leftover from previous runs. You can do this by executing the clean_pcb_rundir.sh script located in the top-level of any of your MITgcm_code cloned repositories after run directory setup. All output and log files will be removed with the exception of files and directories in the results_pcb/ directory.