Table of Contents

This page documents the steps to run WRF and WRF-Chem on Harvard cluster.

...

Exit datamover01 node. You can use login node for this default case.
cd /n/holylfs/LABS/`id -gn`/$USER/new-wrf-root/410-WPS/
source source_wrf # 25s on holylogin03
vim namelist.wps # no edits
./geogrid.exe # 8s on holylogin03, v410, Jan00
ln -sf ungrib/Variable_Tables/Vtable.GFS-PRMSL Vtable # for ungrib

...

./ungrib.exe # 2s on holylogin03, v410, Jan00
./metgrid.exe # 1s on holylogin03, v410, Jan00

...

cd /n/holylfs/LABS/`id -gn`/$USER/new-wrf-root/412-WRF/
cp -ai run 00run # 5s on holylogin03
cd 00run/
# make sure you have sourced source_wrf
ln -s ../../410-WPS/met_em.d01.2000-01-2* ./
vim namelist.input # no edits
./real.exe # 3s on holylogin03
tail rsl.error.0000 # expect to see "SUCCESS COMPLETE REAL_EM INIT"
vim job_wrf.sbatch # no required edits
sbatch job_wrf.sbatch # 2m36s with 4 huce_intel cpus
tail rsl.error.0000* # expect to see "SUCCESS COMPLETE WRF"

Running REAL cases

PART I. Setting up/Configuration/Compilation.

Instructions in this part assume you want to compile and run your own version of WRF. However, note that a compiled usable version of WRF/WRF-Chem v3.6.1 including all external utilities and supplementary geography datasets that you can copy to your preferred run directory is already located at:

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1

This folder (hereafter $CLIMATE_MODELS) contains the WRF-ARW model, the WRF Pre-processing system (WPS; used for real test cases), the chemistry module add-on, the complete WRF geography dataset (for use with WPS and WRF-Chem), and other utilities needed for WRF-Chem. Note: WPS, WRF-Chem not relevant for idealized cases.

...

Read the user's guide:
WRF (ARW) User's Guides: v3, v4
WRF-Chem: https://ruc.noaa.gov/wrf/wrf-chem/Users_guide.pdf #This is for a different WRF-Chem version (3.9), but it's still a relevant guide.
https://ruc.noaa.gov/wrf/wrf-chem/Emission_guide.pdf #This is a separate supplementary WRF-Chem guide to chemical input data processing.
https://ruc.noaa.gov/wrf/wrf-chem/wrf_tutorial_nepal/talks/Setup.pdf #Some helpful WRF-Chem slides from NOAA

PART I. Setting up/Configuration/Compilation.

Instructions in this part assume you want to compile and run your own version of WRF. However, note that a compiled usable version of WRF/WRF-Chem v3.6.1 including all external utilities and supplementary geography datasets that you can copy to your preferred run directory is already located at:
```
/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1
```
folder to a location you are going to run it from. Soft link to the geography data set in the $CLIMATE_MODELS folder.
.If you want to use another version, you will first need to register as a user to download WRF source codes
WRF: http://www2.mmm.ucar.edu/wrf/users/download/get_source.html
Once you've registered, you will be able to navigate to the downloads section. Select the WRF components relevant to you. For example:
http://www2.mmm.ucar.edu/wrf/src/WPSV3.6.1.TAR.gz #WPS 3.6.1
http://www2.mmm.ucar.edu/wrf/src/WRFV3.6.1.TAR.gz #WRF 3.6.1
http://www2.mmm.ucar.edu/wrf/src/WRFV3-Chem-3.6.1.TAR.gz #Chem module add-on for WRF 3.6.1
Be sure to read the online user manuals:
WRF-ARW
```
This folder (hereafter $CLIMATE_MODELS) contains the WRF-ARW model, the WRF Pre-processing system (WPS; used for real test cases), the chemistry module add-on, the complete WRF geography dataset (for use with WPS and WRF-Chem), and other utilities needed for WRF-Chem. Note: WPS, WRF-Chem not relevant for idealized cases.
Note 2: With the exception of the geography data set which is really big, copy the WRF_CHEM_3-6-1 folder to a location you are going to run it from. Soft link to the geography data set in the $CLIMATE_MODELS folder.
.
```
If you want to use another version, you will first need to register as a user to download WRF source codes
WRF: http://www2.mmm.ucar.edu/wrf/users/docsdownload/user_guide_V3/contentsget_source.html
WRF-Chem: https://ruc.noaa.gov/wrf/wrf-chem/Users_guide.pdf #This is for a different WRF-Chem version (3.9), but it's still a relevant guide.
https://ruc.noaa.gov/wrf/wrf-chem/Emission_guide.pdf #This is a separate supplementary WRF-Chem guide to chemical input data processing.
https://ruc.noaa.gov/wrf/wrf-chem/wrf_tutorial_nepal/talks/Setup.pdf #Some helpful WRF-Chem slides from NOAA
Once you've registered, you will be able to navigate to the downloads section. Select the WRF components relevant to you. For example:
http://www2.mmm.ucar.edu/wrf/src/WPSV3.6.1.TAR.gz #WPS 3.6.1
http://www2.mmm.ucar.edu/wrf/src/WRFV3.6.1.TAR.gz #WRF 3.6.1
http://www2.mmm.ucar.edu/wrf/src/WRFV3-Chem-3.6.1.TAR.gz #Chem module add-on for WRF 3.6.1
If you're going to be using WRF meteorology output to drive the STILT LPDM (http://stilt-model.org/index.php/Main/HomePage) you will need to make some changes to the Registry file before compiling. Some basic instructions for this are located in $CLIMATE_MODELS folder as a README_WRF-STILT_modifications.txt file.

Once you've downloaded the necessary tar.gz files, confirm your bashrc file looks something like:

# (1) Load required modules (here we use Intel and Intel MPI)

   module load intel/17.0.4-fasrc01
   module load impi/2017.2.174-fasrc01
   module load netcdf/4.1.3-fasrc02 
   module load libpng/1.6.25-fasrc01
   module load jasper/1.900.1-fasrc02
   module load intel/17.0.4-fasrc01 impi/2017.2.174-fasrc01 ncview/2.1.7-fasrc01 
   module load ncl_ncarg/6.4.0-fasrc01

# (2) Define required environment variables

   export NETCDF=${NETCDF_FORTRAN_HOME:-${NETCDF_HOME}}
   export JASPERLIB=${JASPER_LIB}
   export JASPERINC=${JASPER_INCLUDE}
   export WRFIO_NCD_LARGE_FILE_SUPPORT=1
   export HDF5=${HDF5_HOME}
   unset MPI_LIB  #unset for WPS, where WPS is used for real WRF simulations

### ...... For WRF-Chem: ...... ###

   export WRF_EM_CORE=1
   export WRF_NMM_CORE=0
   export WRF_CHEM=1

Now, configure and compile. WRF must be compiled before WPS.

...

./compile &> compile.output

...

#(1)-(8)

...

are adapted

...

from

...

Plamen's

...

advice

...

for

...

v3.9.1

...

(6/8/2018)

Now that the main WRF and WPS programs are compiled, it's time to think about utilities specific WRF-Chem. If you are planning on running WRF-Chem, you will additionally need to compile some utilities to make sure your chemical data is in the right format. Specifically, you will likely need to compile convert_emiss.exe; prep-chem-src, and mozbc. These are just examples; you may find that you may need additional ones.

...

#(10) Compile PREP-CHEM-SOURCES (available HERE), a mapping utility for converting raw anthropogenic chemical fields to binary intermediates that are then fed into convert_emiss.exe. Unzip the tar.gz file into your main WRF folder. There are some typos, and missing details in the pdf guide above, so a modified version of the instructions (and Paul Edmon's help rebuilding HDF5 to fix an error message) enabled successful compilation of the utility. The modified instructions are located here:

...

Regardless of whether you are running WRF or WRF-Chem, it is important that you do the following first and in this order (detailed instructions follow, including in the examples in Part III and IV):

(1) Run WPS (geogrid.exe, ungrib.exe, metgrid.exe) to create real data-based initialization files with of form met_em.d0X.YYYY-MM-DD_hh:mm:ss.nc

(2) Run real.exe to generate input and boundary files of form wrfinput_d0*, wrfbdy_d01 (and optionally wrffdda_d0*) to initialize WRF model

...

3. You should see met_em.d01(or d02 or d03....).YYYY-MM-DD_hh:mm:ss.nc files created. These are your WPS final output files that real.exe ingests.

Step 2. Running WRF

Navigate to your WRF/WRFV3/run folder
Step 2b. PROGRAM REAL: real data initialization program to set up the model domain and initialize WRF

...

Now get ready to run ungrib. Since we will be using NCEP fnl data (GRIB2 format) for this run, we do some sleuthing to confirm the correct Vtable to use. Following the discussion here we download the appropriate Vtable separately (this one isn't included with the standard set of Vtables in WRFv3.6.1 because they recently updated the dataset). Note: I have already downloaded it and saved it: ungrib/Variable_Tables/Vtable.GFS_new

...

Create a new text file called anthro_emis.inp. This will be your namelist file. Check out the README file for detailed instructions. Your namelist file should look like the following. This is constructed based on a hybrid of following instructions in the README file and the input file located in the anthro_data/EDGAR-HTAP/INP-Examples folder. Make sure you understand the purpose of the entries.

&CONTROL
anthro_dir

=

'/n/holylfs/LABS/kuang_lab/adayalu/WRF/ANTHRO/src'
domains
domains =

3
wrf_dir

=

'/n/holylfs/LABS/kuang_lab/adayalu/WRF/UTILINP'
src_file_prefix

=

'EDGAR_HTAP_emi_'
src_file_suffix

=

'_2010.0.1x0.1.nc'
src_names

=

'PM2.5(1)'
sub_categories

=

'emis_tot'
serial_output

=

.false.
start_output_time

=

'2010-12-01_00:00:00'
emissions_zdim_stag

=

1
emis_map

=

'PM_25(a)
->PM2
→PM2.5',
/

Note that the src_file_prefix and src_file suffix are concatenated with whatever is specified in src_names (here, PM2.5) to generate the full filename string.
Now run anthro_emis.

...

In any case, this exercise should at least get you familiarity with the process of running WRF-Chem and set you up for being able to do second-order troubleshooting (like the more important question of why these values are unrealistic!).

PART IV. Running WRF-Chem for real cases in Large Eddy Simulation (LES) Mode: A Beijing PM2.5 Case Study

Running IDEALIZED cases

Miscellaneous links

WRF (ARW) User's Guides: v3, v4

Google Docs: https://docs.google.com/document/d/1Jls4FlWIOIhMlCzMPWm6_aBZqx_Axxe8RMcKjdILDFg/

...

Version	Old Version 93	New Version 94
Changes made by	Packard Chan (Unlicensed)	Packard Chan (Unlicensed)
Saved on	Nov 14, 2019	Nov 15, 2019

Versions Compared

Key

Running REAL cases

PART I. Setting up/Configuration/Compilation.

PART I. Setting up/Configuration/Compilation.

Step 2. Running WRF

PART IV. Running WRF-Chem for real cases in Large Eddy Simulation (LES) Mode: A Beijing PM2.5 Case Study

Running IDEALIZED cases

Miscellaneous links

Content Comparison

Versions Compared

Key

Running REAL cases

PART I. Setting up/Configuration/Compilation.

PART I. Setting up/Configuration/Compilation.

Step 2. Running WRF

PART IV. Running WRF-Chem for real cases in Large Eddy Simulation (LES) Mode: A Beijing PM2.5 Case Study

Running IDEALIZED cases

Miscellaneous links