...
Now that you have compiled the MITgcm and copied the executable to the run directory, you can start a MITgcm simulation. We will look at the filefiles contained in each of the Hg, PFOS, and PCB simulation run directories.
Run
...
directory files
In the global_hg_llc90/run, pfos/run, and pcb/run directory, you will find everal several sample scripts that you can use to run MITgcm jobs.
| Simulation type | Run script | data.exch file | data file |
|---|---|---|---|
| 13 CPUs, debug run (10 hours) | run.mitgcm.13np.debug | data.exch2.13np | data.debug.run |
| 13 CPUs, 1-month run | run.mitgcm.13np.1month | data.exch2.13np | data.1month_run |
| 96 CPUs, debug run (10 hours) | run.mitgcm.96np.debug | data.exch2.96np | data.debug_run |
| 96 CPUs, 20 year run | run.mitgcm.96np.20yr | data.exch2.96np | data.20yr_run |
We will look at each of these scripts in more detail below.
The run.mitgcm scripts
The run.mitgcm* scripts are used to start a MITgcm simulation with 13 CPUs (for debugging) or 96 CPUs. Each run script also copies the following files before starting For example, the run
...
.
...
mitgcm.
...
data.debug_run or data.1month_run or data.20yr_run ---> data
The data.exch2.13np and data.exch2.96np files
The data.exch2.13np contains the following namelist data declaration. This is used to set up the horizontal grid for 13 CPus.
...
13np.debug scripts look like this:
| Code Block |
|---|
#!/bin/bash
#SBATCH -n 13
#SBATCH -N 1
#SBATCH -t 60
#SBATCH -p regal
#SBATCH --mem-per-cpu=3750
#SBATCH --mail-type=ALL
#EOC
#------------------------------------------------------------------------------
# Harvard Biogeochemistry of Global Pollutants Group !
#------------------------------------------------------------------------------
#BOP
#
# !IROUTINE: run.mitgcm.13np.debug
#
# !DESCRIPTION: Script to run a debug MITgcm simulation with 13 CPUs.
#\\
#\\
# !CALLING SEQUENCE:
# sbatch run.mitgcm.13np.debug # To submit a batch job
# ./run.mitgcm.hg.13np.debug # To run in an interactive session
#
# !REMARKS:
# Consider requesting an entire node (-N 16 -n 1), which will prevent
# outside jobs from slowing down your simulation.
#
# Also note: Make your timestep edits in "data.debug_run", which will
# automatically be copied to "data" by this script.
#
# !REVISION HISTORY:
# 17 Feb 2015 - R. Yantosca - Initial version
#EOP
#------------------------------------------------------------------------------
#BOC
# Make sure we apply the .bashrc_mitgcm settings
source ~/.bashrc_mitgcm
# Copy run-time parameter input files for the 13 CPU run
cp -f data.debug_run data
cp -f data.exch2.13np data.exch2
# Remove old output files
rm -f STDOUT.*
rm -f STDERR.*
rm -f PTRACER*
# Run MITgcm with 13 CPUs
time -p ( mpirun -np 13 ./mitgcmuv )
exit 0
#EOC |
The run.mitgcm.96np.20yr is similar except for the fact that it requests SLURM to give it more CPUs and a longer amount of run time:
| Code Block |
|---|
#!/bin/bash
#SBATCH -n 96
#SBATCH -N 6
#SBATCH -t 1-00:00
#SBATCH -p regal
#SBATCH --mem-per-cpu=3750
#SBATCH --mail-type=ALL
#EOC
#------------------------------------------------------------------------------
# Harvard Biogeochemistry of Global Pollutants Group !
#------------------------------------------------------------------------------
#BOP
#
# !MODULE: run.mitgcm.96np.20yr
#
# !DESCRIPTION: Script to submit a long (e.g. 20 yr) MITgcm simulation
# using 96 CPUs.
#\\
#\\
# !CALLING SEQUENCE:
# sbatch run.mitgcm.96np.20yr # To submit a batch job
#
# !REMARKS:
# Consider requesting 6 entire nodes (-n 96 -N 6), which will prevent
# outside jobs from slowing down your simulation.
#
# Also note: Make your timestep edits in "data.20yr_run", which will
# automatically be copied to "data" by this script.
#
# !REVISION HISTORY:
# 17 Feb 2015 - R. Yantosca - Initial version
#EOP
#------------------------------------------------------------------------------
#BOC
# Make sure we apply the .bashrc_mitgcm settings
source ~/.bashrc_mitgcm
# Copy run-time parameter input files for the 96 CPU run
cp -f data.20yr_run data
cp -f data.exch2.96np data.exch2
# Remove old output files
rm -f STDOUT.*
rm -f STDERR.*
rm -f PTRACER.*
# Run MITgcm with 96 CPUs
time -p ( mpirun -np 96 ./mitgcmuv )
exit 0
#EOC |
The run.mitgcm* scripts all do the following things:
Gets the proper compiler and library settings from your
~/.bashrc_mitgcmfile.Reserves CPUs for the MITgcm run.
- NOTE: For MITgcm production runs, we recommend that you request 96 CPUs (i.e. 6 entire nodes). This will reserve all nodes exclusively for your MITgcm simulation, and will prevent other Odyssey jobs from running on the same node and competing for resources.
- NOTE: For MITgcm production runs, we recommend that you request 96 CPUs (i.e. 6 entire nodes). This will reserve all nodes exclusively for your MITgcm simulation, and will prevent other Odyssey jobs from running on the same node and competing for resources.
Creates the proper
datafile for your simulation from a template. This file contains basic information for the simulation, including- The number of timesteps for the simulation to run;
- How frequently diagnostics are saved to disk (i.e.
dumpFreq); - How frequenlty statistics are written to the log file (i.e.
monitorFreq)
Creates the proper
data.exchfile for your simulation from a template.The data.exch file, which is described below, contains information about the tiles used for the horizontal grid specification.
- Runs the MITgcm simulation and prints the user, cpu, and system time in seconds.
The data.exch2 files
The data.exch2 input file specifies tiling information for the number of CPUs used. This tells the MITgcm to place a certain number of grid boxes on each CPU. For your convenience, we have created two separate data.exch files, one that can be used with 13 CPU simulations (data.exch2.13np) and one for 96 CPU simulations (data.exch2.96np). The run.mitgcm script that you use will select the proper data.exch file for your simulation.
data.exch2.13np
The data.exch2.13np contains the following namelist data declaration. This is used to set up the horizontal grid for 13 CPus.
| Code Block |
|---|
&W2_EXCH2_PARM01
W2_printMsg = 0 ,
W2_mapIO = 1 ,
preDefTopol = 0 ,
#==============================================================================
#-- 5 facets llc_120 topology (drop facet 6 and its connection):
#==============================================================================
dimsFacets(1:10) = 90, 270, 90, 270, 90, 90, 270, 90, 270, 90 ,
facetEdgeLink(1:4,1) = 3.4, 0. , 2.4, 5.1 ,
facetEdgeLink(1:4,2) = 3.2, 0. , 4.2, 1.3 ,
facetEdgeLink(1:4,3) = 5.4, 2.1, 4.4, 1.1 ,
facetEdgeLink(1:4,4) = 5.2, 2.3, 0. , 3.3 ,
facetEdgeLink(1:4,5) = 1.4, 4.1, 0. , 3.1 ,
/ |
data.exch2.96np
The data,exch2.96np is used to set up the horizontal grid for 96 CPUs. It contains the same namelist variables as does data.exch2.13np, with an additional variable named blanklist. This is used to set certain tiles to zero.
| Code Block |
|---|
&W2_EXCH2_PARM01
W2_printMsg = 0 ,
W2_mapIO = 1 ,
preDefTopol = 0 ,
#==============================================================================
#-- 5 facets llc_120 topology (drop facet 6 and its connection):
#==============================================================================
dimsFacets(1:10) = 90, 270, 90, 270, 90, 90, 270, 90, 270, 90 ,
facetEdgeLink(1:4,1) = 3.4, 0. , 2.4, 5.1 ,
facetEdgeLink(1:4,2) = 3.2, 0. , 4.2, 1.3 ,
facetEdgeLink(1:4,3) = 5.4, 2.1, 4.4, 1.1 ,
facetEdgeLink(1:4,4) = 5.2, 2.3, 0. , 3.3 ,
facetEdgeLink(1:4,5) = 1.4, 4.1, 0. , 3.1 ,
#==============================================================================
#-- 30x30 nprocs = 96 : Blank out certain tiles
#==============================================================================
blankList(1:21) = 1,2,3,5,6,28,29,30,31,32,33,49,50
52,53,72,81,90,99,108,117
/ |
The run.mitgcm* scripts will copy data.exch2.13np or data.exch2.96np to a file named data.exch2, so that you won't forget to do this yourself.
NOTE: You should not have to touch the data.exch2* input files because they are already set up for the 13 CPU and 96 CPU runs. The only time you would have to modify these input files if you were changing the horizontal grid specification and the number of CPUs that you wanted to use.
The data files
The main input file for each MITgcm simulation is called data. This file contains various physical parameters, file names, and time stepping settings for the simulation. Most of the time you will only have to change the time stepping parameters, which determine how long your MITgcm will run and how often it will save out diagnostic quantities.
For your convenience, we provide several data files that you can use to schedule MITgcm simulations of different lengths. Each of these files are identical, except for the time stepping parameters:
| Parameter | Units | Description |
|---|---|---|
nIter0 | 1 | Index of the first iteration. If these variables are set to nonzero values, the model will look for a ''pickup'' file |
nTimeSteps | 1 | Number of time steps that you want the MITgcm simulation to run. |
| s | The model ''clock'' timestep. This determines the IO frequencies and is used in tagging output. |
deltaTmom | s | Timestep for momentum equations. This can be set to the same value as deltaTclock. |
deltaTtracer | s | Timestep for tracer equations. This can be set to the same value as deltaTclock. |
deltaTfreesurf | s | Timestep for free surface equations. This can be set to the same value as deltaTclock. |
| s | Control the output frequency (in seconds) of permanent checkpoint files. See MITgcm manual section 1.5.1. |
| s | Control the output frequency (in seconds) of rolling checkpoint files. See MITgcm manual section 1.5.1. |
taveFreq | s | Controls the frequency (in seconds) of saving time-averaged diagnostic quantities. |
dumpFreq | s | Controls the frequency (in seconds) with which the instantaneous state of the model is saved. |
monitorFreq | s | Sets the the interval between diagnostics written out to the text stdout stream (i.e. to the terminal or the files STDOUT.*). It supplies statistics on model variables (max,mean,etc.) and also checks the CFL values. It can be quite expensive and so should not be done every time-step but perhaps every 10-50 timesteps. |
data.debug_run
The data.debug_run file is used to submit a 10-hour MITgcm simulation. The time stepping settings are as follows.
| Code Block |
|---|
&PARM03
nIter0 = 1 ,
nTimeSteps = 10 ,
#
forcing_In_AB = .FALSE. ,
momDissip_In_AB = .FALSE. ,
#
# Set 1-hour timesteps
#
deltaTmom = 3600. ,
deltaTtracer = 3600. ,
deltaTfreesurf = 3600. ,
deltaTClock = 3600. ,
#
#when using ab3:
doAB_onGtGs = .FALSE. ,
alph_AB = 0.5 ,
beta_AB = 0.281105 ,
#
# Time averaging and dumping parameters
#
pChkptFreq = 315576000.0 ,
chkptFreq = 315576000.0 ,
taveFreq = 360000.0 ,
dumpFreq = 360000.0 ,
monitorFreq = 360000.0 ,
dumpInitAndLast = .TRUE. ,
adjDumpFreq = 3155760000.0 ,
adjMonitorFreq = 360000.0 ,
pickupStrictlyMatch = .FALSE. ,
# pickupSuff ='0000166548' ,
/ |
As you can see, we set the basic timestep (deltaTclock) to 3600 seconds = 1 hour, and then run for 10 timesteps = 10 hours total. Diagnostics (taveFreq, dumpFreq, monitorFreq) are saved out at the end of the run (after 360000 seconds).
data.1month_run
This file contains the proper time settings to submit a 1-month MITgcm simulation:
| Code Block |
|---|
&PARM03
nIter0 = 1 ,
nTimeSteps = 744 ,
#
forcing_In_AB = .FALSE. ,
momDissip_In_AB = .FALSE. ,
#
#when using the cd scheme:
#epsAB_CD = 0.25 ,
#tauCD = 172800.0, ,
#
# Set 1-hour timesteps
#
deltaTmom = 3600. ,
deltaTtracer = 3600. ,
deltaTfreesurf = 3600. ,
deltaTClock = 3600. ,
#
#when using ab2:
#abEps = 0.1 ,
#
#when using ab3:
doAB_onGtGs = .FALSE. ,
alph_AB = 0.5 ,
beta_AB = 0.281105 ,
#
# Time averaging and dumping parameters
#
pChkptFreq = 315576000.0 ,
chkptFreq = 315576000.0 ,
taveFreq = 360000.0 ,
dumpFreq = 360000.0 ,
monitorFreq = 360000.0 ,
dumpInitAndLast = .TRUE. ,
adjDumpFreq = 3155760000.0 ,
adjMonitorFreq = 360000.0 ,
pickupStrictlyMatch = .FALSE. ,
# pickupSuff ='0000166548' ,
/ |
This will run the MITgcm for 744 1-hour timesteps, or 1 month of simulation. We archive diagnostics (taveFreq, dumpFreq, monitorFreq) every 10 timesteps.
data.20yr_run
This file contains the proper time stepping parameters to schedule a long 20-year MITgcm simulation.
| Code Block |
|---|
&PARM03 nIter0 = 1 , nTimeSteps = 175332 , # forcing_In_AB = .FALSE. , momDissip_In_AB = .FALSE. , # #when using the cd scheme: #epsAB_CD = 0.25 , #tauCD = 172800.0, , # # Set 1-hour timesteps # deltaTmom = 3600. , deltaTtracer = 3600. , deltaTfreesurf = 3600. , deltaTClock = 3600. , # #when using ab2: #abEps = 0.1 , # #when using ab3: doAB_onGtGs = .FALSE. , alph_AB = 0.5 , beta_AB = 0.281105 , # # Time averaging and dumping parameters # pChkptFreq = 315576000.0 , chkptFreq = 315576000.0 , taveFreq = 31557600.0 , dumpFreq = 3155760000.0 , monitorFreq = 864000.0 , dumpInitAndLast = .TRUE. , adjDumpFreq = 3155760000.0 = 0 , adjMonitorFreq = 360000.0 , W2_mapIO = 1 , pickupStrictlyMatch = .FALSE. , # pickupSuff preDefTopol = 0 ='0000166548' , / , #============================================================================== #-- 5 facets llc_120 topology (drop facet 6 and its connection): #============================================================================== dimsFacets(1:10) = 90, 270, 90, 270, 90, 90, 270, 90, 270, 90 , facetEdgeLink(1:4,1) = 3.4, 0. , 2.4, 5.1 |
This will run MITgcm for 175332 hours = 20 years. Diagnostic files will be written to disk once per year (315576000 seconds) and statistics will be printed to the screen every 10 days (864000 seconds).
If you need to run MITgcm for a different amount of time, we recommend that you copy one of the existing data* files and edit the time stepping parameters accordingly.
Submitting a MITgcm simulation
10-hour debug run
To submit a 10-hour debugging run (on 13 CPUs), type the following commands:
| Code Block |
|---|
#### 10-hr debug Hg simulation #### cd MITgcm_code/ # Switch to main code directory cd verification/global_hg_llc90/run # Change to the Hg run directory sbatch run.mitgcm.13np.1month # Sets the proper data and data.exch files; submits job #### 10-hr debug PFOS simulation #### cd MITgcm_code/ , facetEdgeLink(1:4,2) = 3.2, 0. , 4.2, 1.3 # Switch to main code directory cd verification/pfos/run # Change to the PFOS run directory sbatch run.mitgcm.13np.1month , facetEdgeLink(1:4,3) = 5.4, 2.1, 4.4, 1.1 # Sets the proper data and data.exch files; submits job #### 10-hr debug PCB simulation #### cd MITgcm_code/ , facetEdgeLink(1:4,4) = 5.2, 2.3, 0. , 3.3# Switch to main code directory cd verification/pfos/run # Change to the PCB run directory sbatch run.mitgcm.13np.1month ,# Sets facetEdgeLink(1:4,5) = 1.4, 4.1, 0. , 3.1 the proper data and data.exch files; submits job |
1-month run
To submit a 1-month MITgcm run (on 13 CPUs), type the following commands:
| Code Block |
|---|
#### 1-month Hg simulation ### cd MITgcm_code/ , / |
The data,exch2.96np is used to set up the horizontal grid for 96 CPUs. It contains the same namelist variables as does data.exch2.13np, with an additional variable named blanklist. This is used to set certain tiles to zero.
| Code Block |
|---|
&W2_EXCH2_PARM01 W2_printMsg = 0 # Switch to main code directory cd verification/global_hg_llc90/run # Change to the Hg run directory sbatch run.mitgcm.13np.1month # Sets the proper data and data.exch files; submits job #### 1-month PFOS simulation ### cd MITgcm_code/ , W2_mapIO # Switch to =main 1code directory cd verification/pfos/run # Change to the PFOS run directory sbatch run.mitgcm.13np.1month # Sets the proper data and data.exch files; submits job #### 1-month PCB ,simulation ### cd preDefTopolMITgcm_code/ = 0 # Switch to main code directory cd verification/pfos/run # Change to the PCB run directory sbatch run.mitgcm.13np.1month , #============================================================================== #-- 5 facets llc_120 topology (drop facet 6 and its connection): #============================================================================== dimsFacets(1:10) = 90, 270, 90, 270, 90, 90, 270, 90, 270, 90 , facetEdgeLink(1:4,1) = 3.4, 0. , 2.4, 5.1 # Sets the proper data and data.exch files; submits job |
20-year run
To submit a 20-year MITgcm simulation (on 96 CPUs), type the following commands:
| Code Block |
|---|
#### To run a debug Hg simulation ### cd MITgcm_code/ # Switch to main code directory cd verification/global_hg_llc90/run # Change to the Hg run directory sbatch run.mitgcm.13np.1month # Sets the proper data and data.exch files; submits job #### To run a debug PFOS simulation ### cd MITgcm_code/ , facetEdgeLink(1:4,2) = 3.2, 0. , 4.2, 1.3 # Switch to main code directory cd verification/pfos/run # Change ,to the facetEdgeLink(1:4,3) = 5.4, 2.1, 4.4, 1.1PFOS run directory sbatch run.mitgcm.13np.1month # Sets the proper data and data.exch files; submits job #### To run a debug PCB simulation ### cd MITgcm_code/ , facetEdgeLink(1:4,4) = 5.2, 2.3, 0. , 3.3 # Switch to main code directory cd verification/pcb/run , # facetEdgeLink(1:4,5) = 1.4, 4.1, 0. , 3.1Change to the PCB run directory sbatch run.mitgcm.96np.20yr # Sets the proper data and data.exch files; submits job |
Runs for other lengths of time
To run a MITgcm simulation for a different length of time:
| Code Block |
|---|
cp data.20yr_run , #============================================================================== #-- 30x30 nprocs = 96 : Blank out certain tiles #============================================================================== blankList(1:21) = 1,2,3,5,6,28,29,30,31,32,33,49,50 52,53,72,81,90,99,108,117 / |
The run.mitgcm* scripts will copy data.exch2.13np or data.exch2.96np to a file named data.exch2, so that you won't forget to do this yourself.
Debug run
Hello
1-month run
Hello
20 year run
Hello
Other runs
...
data.myrun # Copy an existing data template file to a new file
cp run.mitgcm.96np.20yr run.mitgcm.96np.myrun # Copy an existing run.mitgcm* file to a new file |
Then in data.myrun, you can change the time stepping parameters (number of timesteps and frequency of diagnostic output) accordingly.
Lastly, in the new run file run.mitgcm.96np.myrun, change this line:
| Code Block |
|---|
Change this line from:
cp -f data.20yr_run data
To:
cp -f data.myrun data |
and if necessary, also increase the amount of run time that you are requesting (e.g.the #SBATCH -t setting).
Then submit the job to the queue:
| Code Block |
|---|
sbatch run.mitgcm.96np.myrun |