Page Comparison

...

Now that you have compiled the MITgcm and copied the executable to the run directory, you can start a MITgcm simulation. We will look at the filefiles contained in each of the Hg, PFOS, and PCB simulation run directories.

Run directory files

In the global_hg_llc90/run, pfos/run, and pcb/run directory, you will find several sample scripts that you can use to run MITgcm jobs.

...

Code Block

#!/bin/bash 
#SBATCH -n 13
#SBATCH -N 1
#SBATCH -t 60
#SBATCH -p regal
#SBATCH --mem-per-cpu=3750
#SBATCH --mail-type=ALL 
#EOC
#------------------------------------------------------------------------------
#              Harvard Biogeochemistry of Global Pollutants Group             !
#------------------------------------------------------------------------------
#BOP
#
# !IROUTINE: run.mitgcm.13np.debug
#
# !DESCRIPTION: Script to run a debug MITgcm simulation with 13 CPUs.
#\\
#\\
# !CALLING SEQUENCE:
#  sbatch run.mitgcm.13np.debug       # To submit a batch job
#  ./run.mitgcm.hg.13np.debug         # To run in an interactive session
#
# !REMARKS:
#  Consider requesting an entire node (-N 6416 -n 1), which will prevent
#  outside jobs from slowing down your simulation.
#
#  Also note: Make your timestep edits in "data.debug_run", which will
#  automatically be copied to "data" by this script.
#
# !REVISION HISTORY:
#  17 Feb 2015 - R. Yantosca - Initial version
#EOP
#------------------------------------------------------------------------------
#BOC

# Make sure we apply the .bashrc_mitgcm settings
source ~/.bashrc_mitgcm

# Copy run-time parameter input files for the 13 CPU run
cp -f data.debug_run   data
cp -f data.exch2.13np  data.exch2

# Remove old output files
rm -f STDOUT.*
rm -f STDERR.*
rm -f PTRACER*

# Run MITgcm with 13 CPUs
time -p ( mpirun -np 13  ./mitgcmuv )
exit 0
#EOC

The run.mitgcm.96np.debug 20yr is similar except for the fact that it requests SLURM to give it more CPUs and a longer amount of run time:

Code Block

#!/bin/bash 
#SBATCH -n 12896
#SBATCH -N 26
#SBATCH -t 601-00:00
#SBATCH -p regal
#SBATCH --mem-per-cpu=3750
#SBATCH --mail-type=ALL 

#EOC
#------------------------------------------------------------------------------
#              Harvard Biogeochemistry   GEOS-Chemof Global ChemicalPollutants TransportGroup Model                  !
#------------------------------------------------------------------------------
#BOP
#
# !IROUTINEMODULE: run.mitgcm.96np.debug20yr
#
# !DESCRIPTION: Script to runsubmit a debuglong (e.g. 20 yr) MITgcm simulation Hg
# simulation withusing 96 CPUs.
#\\
#\\
# !CALLING SEQUENCE:
#  sbatch run.mitgcm.96np.debug20yr    # To submit a batch job
#
# !REMARKS:
#  Consider requesting 26 entire nodes (-n 12896 -N 26), which will prevent
#  outside jobs from slowing down your simulation.
#
#  Also note: Make your timestep edits in "data.debug20yr_run", which will
#  automatically be copied to "data" by this script.
#
# !REVISION HISTORY:
#  17 Feb 2015 - R. Yantosca - Initial version
#EOP
#------------------------------------------------------------------------------
#BOC

# Make sure we apply the .bashrc_mitgcm settings
source ~/.bashrc_mitgcm

# Copy run-time parameter input files for the 96 CPU run
cp -f data.debug.20yr_run    data
cp -f data.exch2.96np  data.exch2

# Remove old output files
rm -f STDOUT.*
rm -f STDERR.*
rm -f PTRACER.*

# Run MITgcm with 96 CPUs
time -p ( mpirun -np 96  ./mitgcmuv )
exit 0
#EOC

...

Gets the proper compiler and library settings from your ~/.bashrc_mitgcm file.
Reserves CPUs for the MITgcm run.
- NOTE: For MITgcm production runs, we recommend that you request 128 96 CPUs (i.e. 2 6 entire nodes) even though the MITgcm only uses 96. This will reserve both all nodes exclusively for your MITgcm simulation, and will prevent other Odyssey jobs from running on the same node and competing for resources.
Creates the proper data file for your simulation from a template. This file contains basic information for the simulation, including
- The number of timesteps for the simulation to run;
- How frequently diagnostics are saved to disk (i.e. dumpFreq);
- How frequenlty statistics are written to the log file (i.e. monitorFreq)
Creates the proper data.exch file for your simulation from a template.
- The data.exch file, which is described below, contains information about the tiles used for the horizontal grid specification.
Runs the MITgcm simulation and prints the user, cpu, and system time in seconds.

...

If you need to run MITgcm for a different amount of time, we recommend that you copy one of the existing data* files and edit the time stepping parameters accordingly.

Submitting a MITgcm

...

simulation

10-hour debug run

To submit a 10-hour debugging run (on 13 CPUs), type the following commands:

Code Block

#### 10-hr debug Hg simulation ####
cd MITgcm_code/                         # Switch to main code directory
cd verification/global_hg_llc90/run     # Change to the Hg run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### 10-hr debug PFOS simulation ####
cd MITgcm_code/                         # Switch to main code directory
cd verification/pfos/run                # Change to the PFOS run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### 10-hr debug PCB simulation ####
cd MITgcm_code/                         # Switch to main code directory
cd verification/pfos/run                # Change to the PCB run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

...

1-month run

To submit a 1-month MITgcm run (on 13 CPUs), type the following commands:

Code Block

#### 1-month Hg simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/global_hg_llc90/run     # Change to the Hg run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### 1-month PFOS simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/pfos/run                # Change to the PFOS run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### 1-month PCB simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/pfos/run                # Change to the PCB run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

...

20-year run

To submit a 20-year MITgcm simulation (on 96 CPUs), type the following commands:

Code Block

#### To run a debug Hg simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/global_hg_llc90/run     # Change to the Hg run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### To run a debug PFOS simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/pfos/run                # Change to the PFOS run directory
sbatch run.mitgcm.13np.1month           # Sets the proper data and data.exch files; submits job

#### To run a debug PCB simulation ###
cd MITgcm_code/                         # Switch to main code directory
cd verification/pcb/run                 # Change to the PCB run directory
sbatch run.mitgcm.96np.20yr             # Sets the proper data and data.exch files; submits job

...

Runs for other lengths of time

To run a MITgcm simulation for a different length of time:

Code Block
cp data.20yr_run data.myrun # Copy an existing data template file to a new file cp run.mitgcm.96np.20yr run.mitgcm.96np.myrun # Copy an existing run.mitgcm* file to a new file

Then in data.myrun, you can change the time stepping parameters (number of timesteps and frequency of diagnostic output) accordingly.

Lastly, in the new run file run.mitgcm.96np.myrun, change this line:

Code Block
Change this line from: cp -f data.20yr_run data To: cp -f data.myrun data

and if necessary, also increase the amount of run time that you are requesting (e.g.the #SBATCH -t setting).

Then submit the job to the queue:

Code Block
sbatch run.mitgcm.96np.myrun

Previous | Up | Next

Versions Compared

Old Version 12

New Version Current

Key

Run directory files

Submitting a MITgcm

simulation

10-hour debug run

1-month run

20-year run

Runs for other lengths of time