...

Edit ~/.bashrc to include following lines (May 2018, also known as CentOS7**updated June 2023 for rocky8**):

module load intel/1723.0.40-fasrc01 impiintelmpi/20172021.28.1740-fasrc01

module load netcdf-fortran/4.1.36.0-fasrc03

module load netcdf-c/4.9.2-fasrc02

module load perl-ncf/1.0.0-fasrc01 perl_modules/5.10.1-fasrc13ncf

export UCX_TLS=ib

export I_MPI_HYDRA_IFACE="ib0"

If you never use cpan (package manager for Perl) before, after loading perl-modules, enter cpan, respond: yes, blank, yes.., then exit cpan. (takes several minutes)

If you need POP2 (e.g. compset B), enter cpan, key in: install Switch, then exit cpan.

cp /n/home05/pchan/spcam2_0-cesm1_1_1/scripts/create_new_build.sh ~/  #please copy a newer version after May 2018 (the CentOS7 update)

2. Edit create_new_build.sh

** Cluster update May 2024: CESM1 had compilation issues. If you need to run CESM1, please email sweidman at g dot harvard dot edu for instructions (until I update this wiki page). **

**Rocky8 update**

The netcdf software is now in two locations (netcdf-c and netcdf-fortran). CESM will break unless you put them in the same folder. One option is to copy the files for netcdf-c and netcdf-fortran in /n/sw/ into one folder in your home directory and combine the files in bin, lib, and include into the same folder. You can also copy

/n/home04/sweidman/netcdffolder into your home directory.

Then, edit your .bashrc file to include the lines:

export NETCDF_HOME='/n/home04/sweidman/netcdffolder/' (replace with the location of your combined netcdf folder)

export NETCDF=${NETCDF_HOME}

** **

cp /n/home05/pchan/spcam2_0-cesm1_1_1/scripts/create_new_build.sh ~/  #please copy a newer version after May 2018 (the CentOS7 update)

2. Edit create_new_build.sh

Edit ~/create_new_build.sh  # look at TODO’s. For details, see "about create_new_build.sh".

Partition is should default to be huce_amd, which works in most of the casecascade or huce_ice. (huce_amd was decomissioned, 2/3/2020.) For more about HUCE's partitions, see Harvard cluster job queues.

...

If you already know what namelist (user_nl_*) or SourceMods you are going to use, cd to the directory with user_nl_* and/or SourceMods. _nl_* and/or SourceMods. create_new_build.sh should be run in this directory.

4. Run create_new_build.sh (~12 mins)

Make sure you have modules loaded as step 1.

~/create_new_build.sh CASENAME RESOLUTION COMPSET NCORES

E.g. ~/create_new_build.sh should be run in this directory.

4. Run create_new_build.sh (~12 mins)

Make sure you have modules loaded as step 1.

~/create_new_build.sh CASENAME RESOLUTION COMPSET NCORES

E.g. ~/create_new_build.sh c1test f19_f19 F_2000 64

...

c1test f19_f19 F_2000 64

5. Final check

**Update Rocky 8**

Open your $case.run file and comment out the line that looks like "srun --mpi=pmi2 --ntasks=64 --cpu_bind=sockets --cpu_bind=verbose --kill-on-bad-exit $EXEROOT/cesm.exe >&! ccsm.log.$LID"

Instead, uncomment the line that looks like "mpirun -np 64 $EXEROOT/cesm.exe >&! ccsm.log.$LID" just above it.

** **

You can modify SourceMods here, but if you do so, you will have to do $CASE.clean_build and rebuild the model with

...

In general, the above parameters and more can be changed after you initially build the model by using ./xmlchange <VARIABLE NAME>=<new value>. Often, the variables you want to change are listed in env_run.xml. You will need to rebuild the model if you make any changes here - see step 5.

What to expect:

Right after create_newcase, copying user_nl_* and SourceMods might results in "cp: No match". These errors can be safely ignored.

...

The above figure is adapted from http://wwwwww2.cesm.ucar.edu/events/tutorials/2016/practical1-bertini.pdf p.48.

There is no need to copy the CESM download folder. (for CESM1, which someone already downloaded for you)

...

Naming conventions of nc files: http://wwwwww2.cesm.ucar.edu/models/cesm1.1/filename_conventions_cesm.html

*.h* files are history files. Similar for r (restart), i (initial), d (diagnostic).

...

How namelists are built in CESM?

$RUNDIR/*_in are namelists directly read by Fortran program. They are recommended NOT to be edited directly, but instead, indirectly through user_nl_* and env_*.xml. The processes of indirectly creating namelists are as follows:

...

• preview_namelists 
  • calls Buildconf/$comp.buildnml.csh
  • copies *_in to CaseDocs/

• Buildconf/$comp.buildnml.csh 
  • call $CCSMROOT/models/**/bld/build-namelist, which create namelist Buildconf/*conf/*_in
  • copy *_in to $RUNDIR (overwrite)

• build-namelist 
  • takes “infile” from user_nl_*
  • writes out Buildconf/*conf/*_in and Buildconf/*.input_data_list

When job is submitted:

• $CASE.submit submits $CASE.run
• $CASE.run calls preview_namelists
• Same things happen again.

...

How to modify namelists in CESM?

If possible, always use user_nl_* and env_*.xml.

E.g. Change the path to an input data file: ./xmlchange SSTICE_DATA_FILENAME=/path/to/new_sst.nc (this can also be listed as a line in user_nl_cam).

E.g. To customize CAM output, edit user_nl_cam according to userguide, and master field list: v1.1.1, v1.2.2.

If impossible, manually change $RUNDIR/*_in, and disable copying (overwriting) in Buildconf/$comp.buildnml.csh

...

1. grep -Ev '^\s*$|^\s*!' user_nl_*
   can display all non-empty uncommented lines in user_nl_*

2. ./preview_namelists can update namelists (and input_data_list)

3. grep '' Buildconf/*.input_data_list
   can list out all input data.
   

4. DIN_LOC_ROOT=`./xmlquery -valonly -silent DIN_LOC_ROOT`
   ./check_input_data -inputdata $DIN_LOC_ROOT -check
   can check input data

5. Section &seq_infodata_inparm in drv_in can be checked in $RUNDIR/cpl.log.$LID

6. Always compare default namelist and new namelist

7. Sometimes, user_nl_* are not interpreted literally. E.g. specifying prescribed_ozone_file could remove lines for prescribed_ozone_cycle_yr, prescribed_ozone_name, etc.

8. Never edit namelist in CaseDocs/ or Buildconf/, they are overwritten every time and NOT read by Fortran program.

References:

http://jules-lsm.github.io/vn4.2/namelists/intro.html

http://www.lahey.com/docs/lfpro73help/f95arnameliststmt.htm

http://www.codezone.co.uk/fortran/namelist.shtml

http://www.cesm.ucar.edu/models/cesm1.1/cesm/doc/usersguide/c1128.html

http://www.cesm.ucar.edu/events/tutorials/2016/practical4-hannay.pdf

Compset

Compset is a short form for "component set", which specifies component models, forcing scenarios and physics options for those models.

...

List all available compset by ~pchan/spcam2_0-cesm1_1_1/scripts/create_newcase -list, or browse http://wwwwww2.cesm.ucar.edu/models/cesm1.1/cesm/doc/modelnl/compsets.html

You can view technical details about compset in README.case under case root directory, after you have created a case with that compset.

...

Details coming

SourceMod related

http://wwwwww2.cgd.ucar.edu/cms/bundy/cam_interface.pdf

Technical Note (283 pp.): http://wwwwww2.cesm.ucar.edu/models/cesm1.1/cam/docs/description/cam5_desc.pdf

http://wwwwww2.cgd.ucar.edu/cms/eaton/cam/atm_desdoc/node2.html#SECTION00024000000000000000

Reference Manual: http://wwwwww2.cesm.ucar.edu/models/cesm1.2/cam/docs/rm5_3/rm.html#subcol_intro

Userguide?: http://wwwwww2.cesm.ucar.edu/models/cesm1.0/cam/docs/ug5_1_1/ug.pdf

CAM uses dry static energy as prognostic variable (instead of temperature). Detailed discretization is implemented in geopotential.F90:geopotential_dse, and described in 4.11.5 in cam4_desc.pdf.

...

Unconfirmed: aquaplanet - src.aquap do not take effect.

References

http://wwwwww2.cesm.ucar.edu/events/tutorials/2016/coursework.html (look for practical sessions)

...