What is CESM / CAM / SPCAM?

(Please add more)

CESM is earth system model.

CAM is the atmospheric part of CESM.

CCSM is an older version of CESM.

SPCAM is super-parameterized CAM.

POP is cesm's ocean model.

Running CESM on Harvard cluster

Latest release on official GitHub:

CESM2 with SPCAM (Jun 2018 or later): instructions to download yourself
CESM1.1.1 with SPCAM (Feb 2013) is downloaded in ~pchan/spcam2_0-cesm1_1_1/
CESM1.2.2 (Jun 2014) is downloaded in /n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS/cesm1_2_2/
SPCAM3.5.0 (Feb 2009) is downloaded in ~kuang/Model/origSPCAM/

1. One-time setup

Register in http://www.cesm.ucar.edu/models/register/register.html

If you do not yet have "huce file access", request it from RC. If you see groups=...10047(huce)... when you run id, then you already have the access.

Edit ~/.bashrc to include following lines (**updated June 2023 for rocky8**):

module load intel/23.0.0-fasrc01 intelmpi/2021.8.0-fasrc01

module load netcdf-fortran/4.6.0-fasrc03

module load netcdf-c/4.9.2-fasrc02

module load ncf/1.0.0-fasrc01 perl_modules/5.10.1-ncf

export UCX_TLS=ib

If you never use cpan (package manager for Perl) before, after loading perl-modules, enter cpan, respond: yes, blank, yes.., then exit cpan. (takes several minutes)

If you need POP2 (e.g. compset B), enter cpan, key in: install Switch, then exit cpan.

**Rocky8 update**

The netcdf software is now in two locations (netcdf-c and netcdf-fortran). CESM will break unless you put them in the same folder. One option is to copy the files for netcdf-c and netcdf-fortran in /n/sw/ into one folder in your home directory and combine the files in bin, lib, and include into the same folder. You can also copy

/n/home04/sweidman/netcdffolder into your home directory.

Then, edit your .bashrc file to include the lines:

export NETCDF_HOME='/n/home04/sweidman/netcdffolder/' (replace with the location of your combined netcdf folder)

export NETCDF=${NETCDF_HOME}

** **

cp /n/home05/pchan/spcam2_0-cesm1_1_1/scripts/create_new_build.sh ~/  #please copy a newer version after May 2018 (the CentOS7 update)

2. Edit create_new_build.sh

Edit ~/create_new_build.sh  # look at TODO’s. For details, see "about create_new_build.sh".

Partition should default to be huce_cascade or huce_ice. (huce_amd was decomissioned, 2/3/2020.) For more about HUCE's partitions, see Harvard cluster job queues.

3. (Advanced user): prepare namelist and SourceMods

If you already know what namelist (user_nl_*) or SourceMods you are going to use, cd to the directory with user_nl_* and/or SourceMods. create_new_build.sh should be run in this directory.

4. Run create_new_build.sh (~12 mins)

Make sure you have modules loaded as step 1.

~/create_new_build.sh CASENAME RESOLUTION COMPSET NCORES

E.g. ~/create_new_build.sh c1test f19_f19 F_2000 64

5. Final check

**Update Rocky 8**

Open your $case.run file and comment out the line that looks like "srun --mpi=pmi2 --ntasks=64 --cpu_bind=sockets --cpu_bind=verbose --kill-on-bad-exit $EXEROOT/cesm.exe >&! ccsm.log.$LID"

Instead, uncomment the line that looks like "mpirun -np 64 $EXEROOT/cesm.exe >&! ccsm.log.$LID" just above it.

** **

You can modify SourceMods here, but if you do so, you will have to do $CASE.clean_build and rebuild the model with

CASE=`./xmlquery -valonly -silent CASE`
srun -p huce_intel,test -c 4 -t 100 --mem=4000 ./${CASE}.build

You can modify namelist here. Often you don't have to rebuild the model.

create_new_build.sh should give you a sentence like this:

2018-06-15T13:30:20 TEST-ONLY-sbatch: Job 46319755 to start at 2018-06-15T13:30:23 using 64 processors on holy2c[12315-12316]

If you are not satisfied with the queue time, or any sbatch setting, you can modify $CASE.run, and run this command to test the setting:

echo -n `date +%FT%T` TEST-ONLY-; sbatch --test-only ${CASE}.run

6. Submit the model

Make sure you have modules loaded as step 1.

Follow instructions from printout of create_new_build.sh, for example:

cd /n/home05/pchan/cesm_caseroot/c1test

./c1test.submit

About create_new_build.sh

This script combines create_newcase, cesm_setup and build. It is originally written by Andy Rhines, and further developed by Wanying Kang, Minmin Fu and Packard Chan. You do not have to use this script if you want to do each step separately.

How to modify:

OUTPUT is the path of large file storage where all Build/Run and Short Term Archive will be located. Default is kuanglfs, which works only for Kuang's group. "$SCRATCH/`id -gn`/$USER/cesm_output/${CASE}" will work well if you want to use $SCRATCH.

Partition is default to be huce_amd. To use huce_intel, uncomment the 3rd-4th lines.

Email notification is default off. Turn on by uncommenting the relevant line.

DOUT_S is now default true. This switch turns on the short term archive procedure, which moves things out of RUNDIR when job completes, i.e. (1)-(3) in the following figure. If you want to leave everything in RUNDIR, set to false.

STOP_N and STOP_OPTION set the length of the model run.

Setting GET_REFCASE to false may cause problem for some compset (e.g. B_1850_CN). Now using default from compset.

REST_N determines how often restart files are saved. Default is only once, after the run completes. For a long run, uncomment this line and change to something like 1 (nyears) so you can restart your run if the job fails.

DOUT_S_SAVE_INT_REST_FILES is a switch that can stop "short term archive" from deleting your intermediate restart files. However, those restart files will not be put in one folder for restart, but distributed across components.

BRANCH is one way of restarting a model from some restart files, disabled by default.

Not much speed up can be obtained when GMAKE_J >4, based on some tests with F_2000. This number should be same as srun -c argument for build.

In general, the above parameters and more can be changed after you initially build the model by using ./xmlchange <VARIABLE NAME>=<new value>. Often, the variables you want to change are listed in env_run.xml. You will need to rebuild the model if you make any changes here - see step 5.

What to expect:

Right after create_newcase, copying user_nl_* and SourceMods might results in "cp: No match". These errors can be safely ignored.

Right after build, there are two lines that print out all non-empty, uncommented lines in user_nl_*; and list files in SourceMods. You might see "ls: No match" if no files are in SourceMods, please ignore.

Test-only sbatch is submitted to test sbatch settings. See step 5.

In case directory, a link named "scratch" will point you to the Build/Run/Short-term Archive directory.

The current create_new_build.sh is copied to the case directory. (xxx: bugs identified, to be fixed)

More about CESM

Directories and files in CESM

The above figure is adapted from http://www2.cesm.ucar.edu/events/tutorials/2016/practical1-bertini.pdf p.48.

There is no need to copy the CESM download folder. (for CESM1, which someone already downloaded for you)

The inputdata directory is located in /n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS/cesm_2_1/inputdata. Seb has opened write access so everyone should be able to download inputdata when needed. Please only download data that you need, as the whole data set is >20T in size.

CASE directory is default to be under ~/cesm_caseroot/$CASE

Build/Run directory and Short Term Archive is changed to use the same directory. That is, atm, lnd, etc. from short term archive, and bld and run, will now locate under the same folder in the same level.

Short term archive is procedure when the job completes, move files out of run directory, which are (1)-(3) described in the above figure.

Long-term archive is not needed on Harvard cluster, as there is no HPSS disk.

What to expect?

When the job is running, or the job got error and didn't reach the short-term archive, expect to see log files and output (e.g. history) files in run directory.

Naming conventions of nc files: http://www2.cesm.ucar.edu/models/cesm1.1/filename_conventions_cesm.html

*.h* files are history files. Similar for r (restart), i (initial), d (diagnostic).

Default for CAM is to only output monthly history files. If your run is less than one month, you might get no history files. (Jan has 31 days.) To increase output, see namelist.

Namelist in CESM

xxx: migrating from https://docs.google.com/document/d/1ap8b3lsA9f89Li5dKICReYuz8bAf1yYACfavivX9joU/edit

What is namelist?

How namelists are built in CESM?

$RUNDIR/*_in are namelists directly read by Fortran program. They are recommended NOT to be edited directly, but instead, indirectly through user_nl_* and env_*.xml. The processes of indirectly creating namelists are as follows:

Before submitting job:

• create_new_build.sh calls cesm_setup and $CASE.build
• Both cesm_setup and $CASE.build call preview_namelists

• preview_namelists 
  • calls Buildconf/$comp.buildnml.csh
  • copies *_in to CaseDocs/

• Buildconf/$comp.buildnml.csh 
  • call $CCSMROOT/models/**/bld/build-namelist, which create namelist Buildconf/*conf/*_in
  • copy *_in to $RUNDIR (overwrite)

• build-namelist 
  • takes “infile” from user_nl_*
  • writes out Buildconf/*conf/*_in and Buildconf/*.input_data_list

When job is submitted:

• $CASE.submit submits $CASE.run
• $CASE.run calls preview_namelists
• Same things happen again.

How to modify namelists in CESM?

If possible, always use user_nl_* and env_*.xml.

E.g. Change the path to an input data file: ./xmlchange SSTICE_DATA_FILENAME=/path/to/new_sst.nc (this can also be listed as a line in user_nl_cam).

E.g. To customize CAM output, edit user_nl_cam according to userguide, and master field list: v1.1.1, v1.2.2.

If impossible, manually change $RUNDIR/*_in, and disable copying (overwriting) in Buildconf/$comp.buildnml.csh

After modifying the namelist, run ./preview_namelists to submit the changes.

Tips:

1. grep -Ev '^\s*$|^\s*!' user_nl_*
   can display all non-empty uncommented lines in user_nl_*

2. ./preview_namelists can update namelists (and input_data_list)

3. grep '' Buildconf/*.input_data_list
   can list out all input data.
   

4. DIN_LOC_ROOT=`./xmlquery -valonly -silent DIN_LOC_ROOT`
   ./check_input_data -inputdata $DIN_LOC_ROOT -check
   can check input data

5. Section &seq_infodata_inparm in drv_in can be checked in $RUNDIR/cpl.log.$LID

6. Always compare default namelist and new namelist

7. Sometimes, user_nl_* are not interpreted literally. E.g. specifying prescribed_ozone_file could remove lines for prescribed_ozone_cycle_yr, prescribed_ozone_name, etc.

8. Never edit namelist in CaseDocs/ or Buildconf/, they are overwritten every time and NOT read by Fortran program.

References:

http://jules-lsm.github.io/vn4.2/namelists/intro.html

http://www.lahey.com/docs/lfpro73help/f95arnameliststmt.htm

http://www.codezone.co.uk/fortran/namelist.shtml

http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c1128.html

http://www.cesm.ucar.edu/events/tutorials/2016/practical4-hannay.pdf

Compset

Compset is a short form for "component set", which specifies component models, forcing scenarios and physics options for those models.

Commonly used compset are: F_2000 (atmosphere and land only for 2000), F_2000_SPCAM_sam1mom (SPCAM), FC4AQUAP (aquaplanet, only CESM1.2+)

List all available compset by ~pchan/spcam2_0-cesm1_1_1/scripts/create_newcase -list, or browse http://www2.cesm.ucar.edu/models/cesm1.1/cesm/doc/modelnl/compsets.html

You can view technical details about compset in README.case under case root directory, after you have created a case with that compset.

To create a new compset (e.g. Aquaplanet in SP-CESM), use the -compset_file argument in create_newcase:

(from ~pchan/spcam2_0-cesm1_1_1/scripts/create_newcase -help)

Make a copy of ~pchan/spcam2_0-cesm1_1_1/scripts/sample_compset_file.xml to your directory and edit it. And edit create_new_build.sh to add '-compset_file' argument when it calls create_newcase.

Details coming

SourceMod related

http://www2.cgd.ucar.edu/cms/bundy/cam_interface.pdf

Technical Note (283 pp.): http://www2.cesm.ucar.edu/models/cesm1.1/cam/docs/description/cam5_desc.pdf

http://www2.cgd.ucar.edu/cms/eaton/cam/atm_desdoc/node2.html#SECTION00024000000000000000

Reference Manual: http://www2.cesm.ucar.edu/models/cesm1.2/cam/docs/rm5_3/rm.html#subcol_intro

Userguide?: http://www2.cesm.ucar.edu/models/cesm1.0/cam/docs/ug5_1_1/ug.pdf

CAM uses dry static energy as prognostic variable (instead of temperature). Detailed discretization is implemented in geopotential.F90:geopotential_dse, and described in 4.11.5 in cam4_desc.pdf.

Parallelization and indexing

https://docs.google.com/document/d/1_y2u4V3E1j6hGUROtLpzVzzv0-dswTJg_s7t2ADmJC4

Restart and looping

Based on CESM1.1.1. Following link is now public (9/27/2019).

https://docs.google.com/document/d/1pev8G4OmJGacKhtK8Bh3t_B4yh4DFc_pvGc6I6BtWl8/edit?usp=sharing

Bugs and technical notes

/n/holylfs & /n/kuanglfs are near end of life. Both disks store inputdata. Please transition to /n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS/cesm_2_1.
For CESM1, the command is ./xmlchange DIN_LOC_ROOT="/n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS/cesm_2_1/inputdata"
For CESM2 (who follows our GoogleDoc instructions), the command is ln -sfT /n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS/cesm_2_1/inputdata $CESMDATAROOT/inputdata
A temporary link has been created pointing from /n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS to /n/holystore01/INTERNAL_REPOS/CLIMATE_MODELS. This temporary link will be removed when /n/holylfs decommission. Please run the above commands before the decommission.
The new place is limited to huce group. Updates (if any) will be posted here.
(last update: 7/30/2020)

Technical notes on how the model is ported to Harvard cluster: https://github.com/PackardChan/spcam2_0-cesm1_1_1-port

Starting CESM1.2.2, cesm_postrun_setup is problematic.

CESM1.2.2 obtained from NCAR is missing a few directories since google apps no longer works.  If you are running in to this problem:

1. Missing pio directory in /models/utils/pio
Solution:
svn checkout https://github.com/NCAR/ParallelIO.git/tags/pio1_8_12/pio ./cesm1_2_2/models/utils/pio

2. Missing genf90.pl file
Solution:
svn checkout https://github.com/PARALLELIO/genf90/tags/genf90_140121 ./cesm1_2_2/tools/cprnc/genf90

Unconfirmed: aquaplanet - src.aquap do not take effect.

References

http://www2.cesm.ucar.edu/events/tutorials/2016/coursework.html (look for practical sessions)

spcam-workshop-jul2013.pdf