...

Edit ~/.bashrc to include following lines (May 2018, also known as CentOS7**updated June 2023 for rocky8**):

module load intel/1723.0.40-fasrc01 impiintelmpi/20172021.28.1740-fasrc01

module load netcdf-fortran/4.6.0-fasrc03

module load netcdf-c/4.19.32-fasrc02

module load perl-ncf/1.0.0-fasrc01 perl_modules/5.10.1-fasrc13ncf

export UCX_TLS=ib

export I_MPI_HYDRA_IFACE="ib0"

If you never use cpan (package manager for Perl) before, after loading perl-modules, enter cpan, respond: yes, blank, yes.., then exit cpan. (takes several minutes)

If you need POP2 (e.g. compset B), enter cpan, key in: install Switch, then exit cpan.

cp /n/home05/pchan/spcam2_0-cesm1_1_1/scripts/create_new_build.sh ~/  #please copy a newer version after May 2018 (the CentOS7 update)

2. Edit create_new_build.sh

Edit ~/create_new_build.sh  # look at TODO’s. For details, see "about create_new_build.sh".

Partition is default to be huce_amd, which works in most of the case. (huce_amd decomissioned, 2/3/2020) For more about HUCE's partitions, see Harvard cluster job queues.

3. (Advanced user): prepare namelist and SourceMods

If you already know what namelist (user_nl_*) or SourceMods you are going to use, cd to the directory with user_nl_* and/or SourceMods. create_new_build.sh should be run in this directory.

4. Run create_new_build.sh (~12 mins)

Make sure you have modules loaded as step 1.

~/create_new_build.sh CASENAME RESOLUTION COMPSET NCORES

E.g. ~/create_new_build.sh c1test f19_f19 F_2000 64

...

** Cluster update May 2024: CESM1 had compilation issues. If you need to run CESM1, please email sweidman at g dot harvard dot edu for instructions (until I update this wiki page). **

**Rocky8 update**

The netcdf software is now in two locations (netcdf-c and netcdf-fortran). CESM will break unless you put them in the same folder. One option is to copy the files for netcdf-c and netcdf-fortran in /n/sw/ into one folder in your home directory and combine the files in bin, lib, and include into the same folder. You can also copy

/n/home04/sweidman/netcdffolder into your home directory.

Then, edit your .bashrc file to include the lines:

export NETCDF_HOME='/n/home04/sweidman/netcdffolder/' (replace with the location of your combined netcdf folder)

export NETCDF=${NETCDF_HOME}

** **

cp /n/home05/pchan/spcam2_0-cesm1_1_1/scripts/create_new_build.sh ~/  #please copy a newer version after May 2018 (the CentOS7 update)

2. Edit create_new_build.sh

Edit ~/create_new_build.sh  # look at TODO’s. For details, see "about create_new_build.sh".

Partition should default to be huce_cascade or huce_ice. (huce_amd was decomissioned, 2/3/2020.) For more about HUCE's partitions, see Harvard cluster job queues.

3. (Advanced user): prepare namelist and SourceMods

If you already know what namelist (user_nl_*) or SourceMods you are going to use, cd to the directory with user_nl_* and/or SourceMods. create_new_build.sh should be run in this directory.

4. Run create_new_build.sh (~12 mins)

Make sure you have modules loaded as step 1.

~/create_new_build.sh CASENAME RESOLUTION COMPSET NCORES

E.g. ~/create_new_build.sh c1test f19_f19 F_2000 64

5. Final check

**Update Rocky 8**

Open your $case.run file and comment out the line that looks like "srun --mpi=pmi2 --ntasks=64 --cpu_bind=sockets --cpu_bind=verbose --kill-on-bad-exit $EXEROOT/cesm.exe >&! ccsm.log.$LID"

Instead, uncomment the line that looks like "mpirun -np 64 $EXEROOT/cesm.exe >&! ccsm.log.$LID" just above it.

** **

You can modify SourceMods here, but if you do so, you will have to do $CASE.clean_build and rebuild the model with

...

Setting GET_REFCASE to false may cause problem for some compset (e.g. B_1850_CN). Now using default from compset.). Now using default from compset.

REST_N determines how often restart files are saved. Default is only once, after the run completes. For a long run, uncomment this line and change to something like 1 (nyears) so you can restart your run if the job fails.

DOUT_S_SAVE_INT_REST_FILES is a switch that can stop "short term archive" from deleting your intermediate restart files. However, those restart files will not be put in one folder for restart, but distributed across components.

...

Not much speed up can be obtained when GMAKE_J >4, based on some tests with F_2000. This number should be same as srun -c argument for build >4, based on some tests with F_2000. This number should be same as srun -c argument for build.

In general, the above parameters and more can be changed after you initially build the model by using ./xmlchange <VARIABLE NAME>=<new value>. Often, the variables you want to change are listed in env_run.xml. You will need to rebuild the model if you make any changes here - see step 5.

What to expect:

Right after create_newcase, copying user_nl_* and SourceMods might results in "cp: No match". These errors can be safely ignored.

...

The above figure is adapted from http://wwwwww2.cesm.ucar.edu/events/tutorials/2016/practical1-bertini.pdf p.48.

There is no need to copy the CESM download folder. (for CESM1, which someone already downloaded for you)

...

Naming conventions of nc files: http://wwwwww2.cesm.ucar.edu/models/cesm1.1/filename_conventions_cesm.html

*.h* files are history files. Similar for r (restart), i (initial), d (diagnostic).

...

How namelists are built in CESM?

$RUNDIR/*_in are namelists directly read by Fortran program. They are recommended NOT to be edited directly, but instead, indirectly through user_nl_* and env_*.xml. The processes of indirectly creating namelists are as follows:

...

• preview_namelists 
  • calls Buildconf/$comp.buildnml.csh
  • copies *_in to CaseDocs/

• Buildconf/$comp.buildnml.csh 
  • call $CCSMROOT/models/**/bld/build-namelist, which create namelist Buildconf/*conf/*_in
  • copy *_in to $RUNDIR (overwrite)

• build-namelist 
  • takes “infile” from user_nl_*
  • writes out Buildconf/*conf/*_in and Buildconf/*.input_data_list

When job is submitted:

• $CASE.submit submits $CASE.run
• $CASE.run calls preview_namelists
• Same things happen again.

...

How to modify namelists in CESM?

If possible, always use user_nl_* and env_*.xml.

E.g. Change the path to an input data file: ./xmlchange SSTICE_DATA_FILENAME=/path/to/new_sst.nc (this can also be listed as a line in user_nl_cam).

E.g. To customize CAM output, edit user_nl_cam according to userguide, and master field list: v1.1.1, v1.2.2.

If impossible, manually change $RUNDIR/*_in, and disable copying (overwriting) in Buildconf/$comp.buildnml.csh

...

1. grep -Ev '^\s*$|^\s*!' user_nl_*
   can display all non-empty uncommented lines in user_nl_*

2. ./preview_namelists can update namelists (and input_data_list)

3. grep '' Buildconf/*.input_data_list
   can list out all input data.
   

4. DIN_LOC_ROOT=`./xmlquery -valonly -silent DIN_LOC_ROOT`
   ./check_input_data -inputdata $DIN_LOC_ROOT -check
   can check input data

5. Section &seq_infodata_inparm in drv_in can be checked in $RUNDIR/cpl.log.$LID

6. Always compare default namelist and new namelist

7. Sometimes, user_nl_* are not interpreted literally. E.g. specifying prescribed_ozone_file could remove lines for prescribed_ozone_cycle_yr, prescribed_ozone_name, etc.

8. Never edit namelist in CaseDocs/ or Buildconf/, they are overwritten every time and NOT read by Fortran program.

References:

http://jules-lsm.github.io/vn4.2/namelists/intro.html

http://www.lahey.com/docs/lfpro73help/f95arnameliststmt.htm

http://www.codezone.co.uk/fortran/namelist.shtml

http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c1128.html

http://www.cesm.ucar.edu/events/tutorials/2016/practical4-hannay.pdf

Compset

Compset is a short form for "component set", which specifies component models, forcing scenarios and physics options for those models.

...

List all available compset by ~pchan/spcam2_0-cesm1_1_1/scripts/create_newcase -list, or browse http://wwwwww2.cesm.ucar.edu/models/cesm1.1/cesm/doc/modelnl/compsets.html

You can view technical details about compset in README.case under case root directory, after you have created a case with that compset.

...

Details coming

SourceMod related

http://wwwwww2.cgd.ucar.edu/cms/bundy/cam_interface.pdf

Technical Note (283 pp.): http://wwwwww2.cesm.ucar.edu/models/cesm1.1/cam/docs/description/cam5_desc.pdf

http://wwwwww2.cgd.ucar.edu/cms/eaton/cam/atm_desdoc/node2.html#SECTION00024000000000000000

Reference Manual: http://wwwwww2.cesm.ucar.edu/models/cesm1.2/cam/docs/rm5_3/rm.html#subcol_intro

Userguide?: http://wwwwww2.cesm.ucar.edu/models/cesm1.0/cam/docs/ug5_1_1/ug.pdf

CAM uses dry static energy as prognostic variable (instead of temperature). Detailed discretization is implemented in geopotential.F90:geopotential_dse, and described in 4.11.5 in cam4_desc.pdf.

...

Unconfirmed: aquaplanet - src.aquap do not take effect.

References

http://wwwwww2.cesm.ucar.edu/events/tutorials/2016/coursework.html (look for practical sessions)

...