Step 0: Initial Setup
Sign up for a free bitbucket.org account. Once you have an account, email yantosca@seas.harvard.edu or elundgren@seas.harvard.edu to be added to the Harvard BGC group on Bitbucket. All users of the Harvard BGC group will automatically have access to the MITgcm_code repository.
If you have not done so already, download the set of Unix environment scripts that are customized for use with the MITgcm. To obtain the scripts, type the following commands:
cd /n/sunderland_lab/Lab/environment
./install
This will copy a number of startup files to your home folder (~/) and your ~/bin folder. Files that you already have will not be over-written.
Step 1: Set up an interactive session
You should always use an interactive session for any work when working on Odyssey rather than work on the login node. You can use the "interactive" script (downloaded to your bin/ directory during initial setup) to request an interactive session. The requested number of CPUS, nodes, memory (MB), time duration (min), and partition are all configurable. For example, to start an 8-hr interactive session with 13 CPUs, 1 node, and 4 GB on the interact partition, type:
interactive 13 1 4000 480 interact
You should request at least 3 hours for your interactive session when first setting up with MITgcm since the initial input download will take over an hour.
Once your interactive session starts up, we recommend that you also run the awake.sh script in the background (type 'awake.sh &'). This will periodically print. a blank line to keep your terminal from freezing. Alternatively, place awake.sh into your .my_personal_settings file and configure your .bashrc to source that file. It will then run every time you start an interactive session.
Step 2: Download the "MITgcm_code" repository
The MITgcm_code directory will serve as the local storage location for all MITgcm input files, including those used in additional MITgcm run directories that you create.
First, navigate to your /n/regal/sunderland_lab/your_user_name folder since you must store the MITgcm_code directory within your regal directory for maximum performance of the model. Next, navigate to whatever subdirectory in your regal directory you want to work in. Finally, type the following commands to download the MITgcm_code repository.
git clone https://bitbucket.org/harvard-bgc/mitgcm_code MITgcm_code
Step 3: Set up the initial PCB run directory
Change directories into MITgcm_code which you just cloned. Execute shell script make_initial_pcb_rundir.sh. Note that downloading input files from the Sunderland_lab space to your regal directory will take a while. For this reason, it is only done once and additional run directories that you set up later in the last step below will symbolically link to files in this directory.
Pay attention to the text output to learn what the script is doing. After it is complete instructions are printed out to clone the pkg/ source code repository developed by Helen Amos. Follow these instructions before proceeding.
Step 4: Download the "hamos_mitgcm_pkg" repository
Instructions for downloading Helen's Amos' source code used for PCB simulations are printed at the end of make_initial_pcb_rundir.sh executation. If you did not follow those instructions, do them now. To clone the repository, which you should store in MITgcm_code/pkg, type the below code. All PCB-specific source code is in the pkg/pcb/ directory.
git clone https://bitbucket.org/harvard-bgc/hamos_mitgcm_pkg pkg
Step 5: Set the PCB congener
Execute python script change_PCB_congener.py to set your congener in config files, source code (pcb_param.F), and run scripts. For example, "./change_PCB_congener.py PCB101". If you add any congener-specific source code outside of pcb_param.F, make sure that you incorporate setting the correct settings into change_PCB_congener.py.
Step 6: Compile the source code
Execute shell script compile_pcb.sh and follow the instructions at the prompt. For fast runs you should choose 96 CPUs and accept copying Helen Amos' files.
Step 7: Create a run script or edit an existing run script
Run scripts are provided for Helen Amos' stages 1, 2, and 3. Open the run script and edit line 9 to use your own email address. Look over the "User-specific input" section to make sure the settings are as you wish. Note that the three stages are meant to be run in series. If you are not starting at stage 1 then you must manually copy the ptracers pickup files into the MITgcm_code/verification/global_pcb_llc90/run/ directory. You should use the pickup files with the largest number (representing timestep) found in the stage 2 run 15 results directory of your choosing. When you copy the files (one ending in .data and another ending in .meta), rename them as pickup_ptracers.000000001.data and pickup_ptracers.0000000001.meta.
Step 8: Submit the run to SLURM
Submit your run by typing "sbatch your_run_script".
NOTE: If you decide to run MITgcm interactively rather than submitting a job to SLURM then you must source the .bashrc_mitgcm file prior to running.
IMPORTANT NOTE: Files on the /n/regal/ disk are purged every 90 days to save disk space. You therefore should keep a backup copy of all MITgcm results you wish to keep long-term in your Sunderland group lab space folder (/n/sunderland_lab/Lab/Users/YOUR-USER-NAME).
Step 9: Create additional PCB run directories
Once you have one run directory you can very quickly create new ones without having the re-download all of the input files. This saves a lot of time and resources. Go back to step 2 and clone the repository to a new directory with a different name than your first one. Make sure it is stored in the same directory as your original clone "MITgcm_code". Change directories into your new clone, and execute the shell script make_new_pcb_rundir.sh to set it up. Additional directories can have descriptive names of what you plan on doing with them, e.g. MITgcm_PCB28_{update_description}. Follow the instructions prompted by the script to complete the setup. Then continue through steps above to get the source code, change congeners, compile, and run.
Step 10: Clean an existing run directory for re-use
If you would like to re-use a run directory it is a good idea to remove all files leftover from previous runs. You can do this by executing the clean_pcb_rundir.sh script located in the top-level of any of your MITgcm_code cloned repositories after run directory setup. All output and log files will be removed with the exception of files and directories in the results_pcb/ directory.