...

We invite Lee Miller, Jiahua Guo, and Archana Dayalu to co-edit this page. 

Running WRF (and/or WRF-Chem) on Harvard Odyssey: REAL cases

PART I. Setting up/Configuration/Compilation.

• Instructions in this part assume you want to compile and run your own version of WRF. However, note that a compiled usable version of WRF/WRF-Chem v3.6.1 including all external utilities and supplementary geography datasets that you can copy to your preferred run directory is already located at: 

  /n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF

  This folder (hereafter $CLIMATE_MODELS) contains the WRF-ARW model, the WRF Pre-processing system (WPS; used for real test cases), the chemistry module add-on, the complete WRF geography dataset (for use with WPS and WRF-Chem), and other utilities needed for WRF-Chem. Note: WPS, WRF-Chem not relevant for idealized cases.

  Note 2: With the exception of the geography data set which is really big, copy the WRF_CHEM_3-6-1 folder to a location you are going to run it from. Soft link to the geography data set in the $CLIMATE_MODELS folder.
  .

• If you want to use another version, you will first need to register as a user to download WRF source codes
  WRF: http://www2.mmm.ucar.edu/wrf/users/download/get_source.html
  
  
• Once you've registered, you will be able to navigate to the downloads section. Select the WRF components relevant to you. For example:
  http://www2.mmm.ucar.edu/wrf/src/WPSV3.6.1.TAR.gz #WPS 3.6.1
  http://www2.mmm.ucar.edu/wrf/src/WRFV3.6.1.TAR.gz #WRF 3.6.1
  http://www2.mmm.ucar.edu/wrf/src/WRFV3-Chem-3.6.1.TAR.gz #Chem module add-on for WRF 3.6.1
  
  
• Be sure to read the online user manuals:
  WRF-ARW: http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/contents.html
  WRF-Chem: https://ruc.noaa.gov/wrf/wrf-chem/Users_guide.pdf #This is for a different WRF-Chem version (3.9), but it's still a relevant guide.
  https://ruc.noaa.gov/wrf/wrf-chem/Emission_guide.pdf #This is a separate supplementary WRF-Chem guide to chemical input data processing.
  https://ruc.noaa.gov/wrf/wrf-chem/wrf_tutorial_nepal/talks/Setup.pdf  #Some helpful WRF-Chem slides from NOAA
  
  
• If you're going to be using WRF meteorology output to drive the STILT LPDM (http://stilt-model.org/index.php/Main/HomePage) you will need to make some changes to the Registry file before compiling. Some basic instructions for this are located in $CLIMATE_MODELS folder as a README_WRF-STILT_modifications.txt file. 
  
  
• Once you've downloaded the necessary tar.gz files, confirm your bashrc file looks something like:

  # (1) Load required modules (here we use Intel and Intel MPI) 

     module load intel/17.0.4-fasrc01

     module load impi/2017.2.174-fasrc01

     module load netcdf/4.1.3-fasrc02 

     module load libpng/1.6.25-fasrc01

     module load jasper/1.900.1-fasrc02

     module load intel/17.0.4-fasrc01 impi/2017.2.174-fasrc01 ncview/2.1.7-fasrc01 

     module load ncl_ncarg/6.4.0-fasrc01

  # (2) Define required environment variables

     export NETCDF=${NETCDF_FORTRAN_HOME:-${NETCDF_HOME}}

     export JASPERLIB=${JASPER_LIB}

     export JASPERINC=${JASPER_INCLUDE}

     export WRFIO_NCD_LARGE_FILE_SUPPORT=1

     export HDF5=${HDF5_HOME}

     unset MPI_LIB  #unset for WPS, where WPS is used for real WRF simulations

  ### ...... For WRF-Chem: ...... ###

     export WRF_EM_CORE=1

     export WRF_NMM_CORE=0

     export WRF_CHEM=1

   

• Now, configure and compile. WRF must be compiled before WPS.

...

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/anthro_data

PART II. Running WPS and WRF: Overview/General Steps

...

Now that you have a compiled version of WRF and WPS, you are ready to set up your model runs. Since these are for real cases, you need access to initialization data sets. You will need to figure out the initialization data set best suited to your domain and purposes. For the examples for China provided here, we are using GRIB2 NCEP FNL Operational Model Global Tropospheric Analyses, continuing from July 1999 (http://dx.doi.org/10.5065/D6M043C6). For these examples, the files are already downloaded. Instructions to link to them are noted where necessary.

Regardless of whether you are running WRF or WRF-Chem, it is important that you do the following first and in this order (detailed instructions follow, including in the examples in Part III and IV):

(1) Run WPS (geogrid.exe, ungrib.exe, metgrid.exe) to create real data-based initialization files with of form met_em.d0X.YYYY-MM-DD_hh:mm:ss.nc

(2) Run real.exe to generate input and boundary files of form wrfinput_d0*, wrfbdy_d01 (and optionally wrffdda_d0*) to initialize WRF model 

If you are running WRF without chemistry, you can go ahead and run the main WRF model at this point. If you are running WRF-Chem, this is the point at which you run your chemistry data prep program (i.e., prep-chem-src, anthro_emis, and/or convert_emiss) which requires wrfinput_d0* files to actually work. Once you have your correctly formatted chemical data (they should be in the form wrfchemi_00z_d01 and wrfchemi_12z_d01). Once you are done with this and have all your requisite chem data in the right format stored or linked in the WRFV3/test/em_real folder, you can run the wrf.exe model.

...

Step 1: Running WPS (WRF Pre-processing System)

The pre-processing to create a real data-based initialization file. Read the README file in the parent WPS folder for a quick guide on what the programs do.

...

      3. You should see met_em.d01(or d02 or d03....).YYYY-MM-DD_hh:mm:ss.nc files created. These are your WPS final output files that real.exe ingests.

Step 2. Running WRF

• Navigate to your WRF/WRFV3/run folder

• Step 2b. PROGRAM REAL: real data initialization program to set up the model domain and initialize WRF

...

wrfout_d0#_YYYY-MM-DD_hh:mm:ss

PART III. Running WRF With Chemistry: A PM2.5 example

...

The purpose of this example is to take the general steps listed above and actually run a three nested domain WRF-Chem PM2.5 simulation over Beijing during the January 2013 severe haze event and compare with observations. We are going to run WRF-Chem for a total of 10 days from Jan 6 2013 00:00UTC to Jan 16 2013 00:00UTC. We establish 5 days for model spin-up such that the usable simulation time period is 5 days. Make sure you have a local copy of the /n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF directory and contents. You don't need to copy the geography data set.

...

• Get in the habit of using ncview for sanity-check visuals
• Use the dust map from the WPS high-res geographical data set (namelist.input dust_opt=3), although this is less important for this winter exercise.
• Use prep-chem-sources and convert_emis to format a gridded anthropogenic PM2.5 emissions file to the WRF forecast domain
• Use gridded chemical data for biogenic emissions and chemical boundary conditions
• Run WRF-Chem with PM2.5-radiation feedbacks to simulate PM2.5 concentrations in Beijing in January 2013.
• Extract relevant components from wrfout netcdf files and examine/visualize using NCL.

...

Step 0.1.

At any point where you want to check the contents of a netcdf file as a sanity-check use ncview! This is an excellent habit to develop. Just type 

...

and navigate through the variables and panes and make sure things look realistic.

Step 0.2

For wrf-chem, it's good practice to create a folder for use with various external utilities that you link your intermediate wrf files to. This will become clearer, but for now make sure you have a directory in your $CLIMATE_MODELS location that's entitled "UTILINP".

Step 1. Run WPS

• Navigate to the WRF/WPS folder.
• First customize the WPS namelist.wps file such that it looks like the following. Note that with the exception of the geog_data_path, all path variables should point to a writable dump directory of your choice. It is recommended you write to regal or a similar scratch space you have access to; the dump files take up space and aren't needed again after the run so they don't have to be stored in a permanent location. Consult the WRF user guide for detailed explanation of the namelist variables.

...

mpirun -np 1 ./metgrid.exe

Step 2. Run real.exe to get necessary intermediate files for WRF-Chem utilities (chem_opt off).

• Navigate to WRFV3/test/em_real
• Copy namelist.input_pm25Beijing to namelist.input. Take a look at the namelist and confirm the dates and domain match your wps namelist
• Make sure to turn off chemistry (chem_opt = 0) for this step.
• Change the filepath in history_outname to reflect where you want your wrfchem output files to be saved.
• Link the metgrid files to the test/em_real directory and make sure the namelist num_metgrid_levels and num_metgrid_soil_levels match what's in the metgrid files.

...

• Create soft links of all the met_em.<domain>, wrfinput_<domain>, wrfbd_d01 files in your UTILINP folder. These will be used by the external utilities in the next steps.

Step 3. Generate your bio emissions using MEGAN

• You will need access to the relevant MEGAN initial files, from https://www.acom.ucar.edu/wrf-chem/download.shtml
• Make sure you are still using an interactive shell ('srun -n 1 --mem=10000 --pty --x11=first -p test -t 200 bash' should be sufficient). Navigate to the MEGAN directory in the $CLIMATE_MODELS directory (or wherever you have saved your local copy of everything)
• Create a new text file called megan_bio_emiss.inp. This is your MEGAN namelist file.Note that as the README instructs, the leaf area index (lai) months requires the simulation month and the previous month such that for January (as our example here is) we have to simulate all months.  Following the instructions in the README, you should populate to look like follows ... with the paths obviously reflecting where your WRF + external utility directories are located.

...

wrfbiochemi_d01

wrfbiochemi_d02

wrfbiochemi_d03

Step 4. OPTION 1 (USE THIS FOR NOW): Prep your anthropogenic emissions using anthro_emis

• Now that you have your bio emissions, it's time to get your anthropogenic emissions in the right format. We're going to use the EDGAR-HTAP PM2.5 emissions on a 0.1x0.1 deg annual global grid.
• This option has been successfully tested if you're using EDGAR-HTAP emissions or IPCC emissions (for use with CAM-Chem) and seems a little less complicated than prep-chem-sources. In these examples, we will go this route. If you are starting with other emissions, you probably need to go the prep-chem route. I haven't spent much time figuring out how to use it; the anthro_emis option seemed most straightforward for now because I already had some familiarity with utilities in the same family (MEGAN, MOZBC). 
•  Navigate to your ANTHRO folder

...

wrfchemi_00z_<domain> 

wrfchemi_12z_<domain> 

Step 4. OPTION 2 (FYI for now): Prep your anthropogenic emissions using prep-chem-sources

• TBD. Watch this space.
• Use this utility if you're using something other than EDGAR-HTAP or things not listed in the anthro_emis README/instructions. In theory, anthro_emis could work for other emissions fields, but they have just not been tested.
• Now that you have your bio emissions, it's time to get your anthropogenic emissions in the right format. Navigate to your PREP-CHEM-SRC-1.5 folder.

...

• Edit the prep_chem_sources.inp namelist file. Check out the README file for detailed instructions. 

Step 5. Run real.exe again, with chem_opt turned on

• Navigate to your WRFV3/test/em_real/ directory
• Link your bio and anthro files to your WRFV3/test/em_real/ directory.
• Open up the the namelist.input file and turn chem_opt back on (set it to 10).
• Make sure kemit=1 (vertical levels in anthro emissions files...in this case it is 1...surface data).
• Call up an interactive shell if you don't have one running. Run real.exe again. This incorporates the chem variables into the initial and boundary condition files so that MOZBC can populate them.

...

• Check that the tail of the rsl.error.0000 file says "SUCCESS COMPLETE REAL_EM INIT"

Step 6. Prep the chemical data initial and boundary conditions using MOZBC

• This step modifies the wrf initial and boundary condition files that now have space for chemical data (since we re-ran real.exe with chem_opt on and anthro/bio fields)
• Go to https://www.acom.ucar.edu/wrf-chem/mozart.shtml and submit a data request. This can take some time to process depending on your domain size request. Submit your bounding box, and times. For this exercise you can use what was already downloaded. This is available in the mozbc directory in the $CLIMATE_MODELS folder as a mozart4geos5-ZZZZ.nc file.
• Navigate to your mozbc folder from the $CLIMATE_MODELS folder.

...

./mozbc < mozbc.inp > mozbc.out

#tail mozbc.out should have a final line that reads:

bc_wrfchem completed successfully

Step 7. You're (FINALLY) ready to run WRF-Chem

• Navigate back to your WRFV3/test/em_real directory. Check that you have the preferred path set to your WRFOUT directory (recommend regal or some scratch space with a lot of space) in your history_outname field. If you change the filename, note the only part of the filename that you should mess with is the part before '_d<domain>_d<date>'!
• I liked to keep my frames hourly (a file is written out every simulation hour. More helpful with debugging if run fails. Up to you.)
• Open up the script run_wrf.sh and make sure it looks ok.
• Submit your wrf job

...

Step 8. Post-Processing and data visualization

• I primarily use NCL, and/or some combination of NCL and R. Pick whatever you're used to for processing netcdf files. If you're going to use NCL, I recommend looking at the very well documented examples on their website. For example, start with the one about how to open and read netcdf files and go from there:

https://www.ncl.ucar.edu/Document/Functions/Built-in/addfile.shtml

• If you use ncview, you can quickly get a sense of how your simulation worked. Take a look at the PM2.5_DRY variable (3D Vars). You will notice that while the run completed successfully from a technical standpoint, it's actually way off. The PM2.5 values are unrealistic – two orders of magnitude lower than observations in the d03 domain – and this is most likely due to the choice of anthropogenic inventory. We used EDGAR-HTAP from 2010, which may be an inappropriate choice for this particular case study. In the past I have run a test of this using a more specialized inventory from 2010 processed using prep-chem-sources which led to a far more realistic PM2.5 simulation. In any case, this exercise should at least get you familiarity with the process of running WRF-Chem and set you up for being able to do second-order troubleshooting (like the more important question of why these values are unrealistic!).

PART IV. Running WRF-Chem for real cases in Large Eddy Simulation (LES) Mode: A Beijing PM2.5 Case Study

Running WRF (and/or WRF-Chem) on Harvard Odyssey: IDEALIZED cases

Miscellaneous links

Google Docs: https://docs.google.com/document/d/1Jls4FlWIOIhMlCzMPWm6_aBZqx_Axxe8RMcKjdILDFg/

...