this page document the steps to run WRF and WRF-Chem on Odyssey.  

We invite Lee Miller, Jiahua Guo, and Archana Dayalu to co-edit this page. 

Running WRF (and/or WRF-Chem) on Harvard Odyssey: REAL cases

PART I. Setting up/Configuration/Compilation.

• Instructions in this part assume you want to compile and run your own version of WRF. However, note that a compiled usable version of WRF/WRF-Chem v3.6.1 that you can copy to your preferred run directory is already located at: 

  /n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF

  This folder (hereafter $CLIMATE_MODELS) contains the WRF-ARW model, the WRF Pre-processing system (WPS; used for real test cases), the chemistry module add-on, the complete WRF geography dataset (for use with WPS and WRF-Chem), and other utilities needed for WRF-Chem. Note: WPS, WRF-Chem not relevant for idealized cases.

  Note 2: With the exception of the geography data set which is really big, it's recommended that you copy the WRF_CHEM_3-6-1 folder to a location you are going to run it from. Soft link to the geography data set in the $CLIMATE_MODELS folder.
  .

• If you want to use another version, you will first need to register as a user to download WRF source codes
  WRF: http://www2.mmm.ucar.edu/wrf/users/download/get_source.html
  
  
• Once you've registered, you will be able to navigate to the downloads section. Select the WRF components relevant to you. For example:
  http://www2.mmm.ucar.edu/wrf/src/WPSV3.6.1.TAR.gz #WPS 3.6.1
  http://www2.mmm.ucar.edu/wrf/src/WRFV3.6.1.TAR.gz #WRF 3.6.1
  http://www2.mmm.ucar.edu/wrf/src/WRFV3-Chem-3.6.1.TAR.gz #Chem module add-on for WRF 3.6.1
  
  
• Be sure to read the online user manuals:
  WRF-ARW: http://www2.mmm.ucar.edu/wrf/users/docs/user_guide_V3/contents.html
  WRF-Chem: https://ruc.noaa.gov/wrf/wrf-chem/Users_guide.pdf #This is for a different WRF-Chem version (3.9), but it's still a relevant guide.
  https://ruc.noaa.gov/wrf/wrf-chem/Emission_guide.pdf #This is a separate supplementary WRF-Chem guide to chemical input data processing.
  https://ruc.noaa.gov/wrf/wrf-chem/wrf_tutorial_nepal/talks/Setup.pdf  #Some helpful WRF-Chem slides from NOAA
  
  
• Once you've downloaded the necessary tar.gz files, confirm your bashrc file looks something like:

  # (1) Load required modules (here we use Intel and Intel MPI) 

     module load intel/17.0.4-fasrc01

     module load impi/2017.2.174-fasrc01

     module load netcdf/4.1.3-fasrc02 

     module load libpng/1.6.25-fasrc01

     module load jasper/1.900.1-fasrc02

     module load ncview/2.1.7-fasrc01 #Note, as of 6/28/18 this is not yet available on odyssey after CentOS7 upgrade

     module load ncl_ncarg/6.4.0-fasrc01

  # (2) Define required environment variables

     export NETCDF=${NETCDF_FORTRAN_HOME:-${NETCDF_HOME}}

     export JASPERLIB=${JASPER_LIB}

     export JASPERINC=${JASPER_INCLUDE}

     export WRFIO_NCD_LARGE_FILE_SUPPORT=1

     export HDF5=${HDF5_HOME}

     unset MPI_LIB  #unset for WPS, where WPS is used for real WRF simulations

  ### ...... For WRF-Chem: ...... ###

     export WRF_EM_CORE=1

     export WRF_NMM_CORE=0

     export WRF_CHEM=1

   

• Now, configure and compile. WRF must be compiled before WPS.

#(3) Configure WRF

 ./configure

#(4) Choose 15. (dmpar) to compile MPI version with Intel compilers

              **Note! Do not use dm+sm or smpar options with WRF-Chem!! Choose either serial or dmpar**

#(5) Modify the file “configure.wrf” (around lines 149-150) to read the following. Note that you have to do this each time you run ./configure, because the configure.wrf script is overwritten each time.

 DM_FC = mpiifort -f90=$(SFC)

 DM_CC = mpiicc -cc=$(SCC) -DMPI2_SUPPORT

#(6) Compile WRF before WPS!! Compilation will take a while. If you're on an interactive shell, remove the "&" to avoid timing out.

      # For real cases:  

./compile em_real &> compile.log

  #(7) Configure WPS

      #Likely you will need GRIB2 Support...so Choose option #19: 

      #Linux x86_64, Intel compiler    (dmpar)

./configure

#(8) Compile WPS

./compile &> compile.output

   #(1)-(8) is adapted from Plamen's advice for v3.9.1

• Now that the main WRF and WPS programs are compiled, it's time to think about utilities specific WRF-Chem. If you are planning on running WRF-Chem, you will additionally need to compile some utilities to make sure your chemical data is in the right format. Specifically, you will likely need to compile convert_emiss.exe; prep-chem-src, and mozbc. These are just examples; you may find that you may need additional ones.

      #(9) Compile convert_emiss.exe, a program that converts binary intermediate chemical fields into netcdf WRF-compatible format. Navigate to your WRFV3 folder and type:

./compile emi_conv

     If compilation was successful, you should see convert_emiss.exe in the WRFV3/chem folder.

      #(10) Compile Prep-chem-sources (available HERE), a mapping utility for converting raw chemical fields to binary intermediates that are then fed into convert_emiss.exe. Unzip the tar.gz file into your main WRF folder. There are some typos, and missing details in the pdf guide above, so a modified version of the instructions (and Paul Edmon's help rebuilding HDF5 to fix an error message) enabled successful compilation of the utility. The modified instructions are located here:

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF/README_prepchem_modifications

In any case, a compiled version of prep chem sources utility using the instructions above is located here:

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF/PREP-CHEM-SRC-1.5/bin/prep_chem_sources_RADM_WRF_FIM_SIMPLE.exe

#(11)  The mozbc utility for mapping chemical boundary conditions to your WRF-Chem  domain has already been compiled and saved in the $CLIMATE_MODELS WRF-Chem folder, following the instructions in the README_mozbc file. You can use that, or if you wanted, download and compile MOZBC on your own.

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/mozbc

#(12) MEGAN

#(13) Other potentially useful utilities: EDGAR-HTAP, ANTHRO. Read the discussion here as to what each of these is:

https://www2.acom.ucar.edu/wrf-chem/wrf-chem-tools-community

And download here as needed for your purposes. Otherwise you can link to what has already been downloaded for the examples on this page.

https://acom.ucar.edu/wrf-chem/download.shtml

PART II. Running WPS and WRF: Overview/General Steps

Now that you have a compiled version of WRF and WPS, you are ready to set up your model runs. Since these are for real cases, you need access to initialization data sets. You will need to figure out the initialization data set best suited to your domain and purposes. For the examples for China provided here, we are using GRIB2 NCEP FNL Operational Model Global Tropospheric Analyses, continuing from July 1999 (http://dx.doi.org/10.5065/D6M043C6). For these examples, the files are already downloaded. Instructions to link to them are noted where necessary.

Regardless of whether you are running WRF or WRF-Chem, it is important that you do the following first and in this order (detailed instructions follow, including in the examples in Part III and IV):

(1) Run WPS (geogrid.exe, ungrib.exe, metgrid.exe) to create real data-based initialization files with of form met_em.d0X.YYYY-MM-DD_hh:mm:ss.nc

(2) Run real.exe to generate input and boundary files of form wrfinput_d0*, wrfbdy_d01 (and optionally wrffdda_d0*) to initialize WRF model 

If you are running WRF without chemistry, you can go ahead and run the main WRF model at this point. If you are running WRF-Chem, this is the point at which you run your chemistry data prep program (i.e., prep-chem-src, anthro_emis, and/or convert_emiss) which requires wrfinput_d0* files to actually work. Once you have your correctly formatted chemical data (they should be in the form wrfchemi_00z_d01 and wrfchemi_12z_d01). Once you are done with this and have all your requisite chem data in the right format stored or linked in the WRFV3/test/em_real folder, you can run the wrf.exe model.

Step 1: Running WPS (WRF Pre-processing System)

The pre-processing to create a real data-based initialization file. Read the README file in the parent WPS folder for a quick guide on what the programs do.

• Navigate to your WRF/WPS folder

• Step 1a. PROGRAM GEOGRID: define simulation domains, interpolate terrestrial data to model grids

1. Define model domains with geogrid. Edit the namelist.wps file, specifically:

 &share (e.g., core='ARW', max_dom, io_form_geogrid=2, opt_output_from_geogrid_path='/n/your_scratch_dir_path/GEOGRID_DUMP/’)

 &geogrid

      2. View domain configuration, confirm it's correct.

 cp util/plotgrids_new.ncl . ; ncl plotgrids.ncl

      3. Run geogrid.

 mpirun -np 1 ./geogrid.exe

• Step 1b. PROGRAM UNGRIB: get gribbed met data into usable format to ingest into WRF

      1.Examine GRIB data (most likely GRIB2).

./util/g1print.exe /your_GRIB1_data_path/filename | more

./util/g2print.exe /your_GRIB2_data_path/filename | more

      2. Edit namelist.wps as needed.

&share (start date, end date, interval seconds=21600)

&ungrib (outformat='WPS',prefix='/n/your_scratch_dir_path/METEM_FILEDUMP/FILE')

               #note: be aware of 90-day retention policy of scratch data. Make sure you've emptied any files from previous runs in the dump dirs.

      3. link Vtable (it's like a Rosetta Stone) to appropriate GRIB data type to prepare for ungribbing. e.g., for GFS data. WRF has a bunch of Vtables already in the Variable_Table subfolder of ungrib, but sometimes you will need to do some sleuthing and download the one that you actually need into the Variable_Tables folder. For the examples that follow, this is exactly what we'll need to do. 

ln -sf ungrib/Variable_Tables/Vtable.theVtableYouWantToUse Vtable

      4. link the GRIB data that you are going to use

./link_grib.csh /your_GRIB_data_path/XXX*

             #This should update GRIBFILE.AAA, .AAB, etc links in pwd.

      5. Run ungrib

mpirun -np 1 ./ungrib.exe >& ungrib.output

            #You should see FILE:YYYY-MM-DD_hh in your folder prescribed by “prefix”

• Step 1c. PROGRAM METGRID: horizontally interpolate ungribbed met data onto geogrid domains

1. Edit namelist.wps as needed.

&share (this should be exactly as you need it already)

&metgrid (fg_name = &ungrib prefix; io_form_metgrid=2, outpath)

      2. Run metgrid

mpirun -np 1 ./metgrid.exe

      3. You should see met_em.d01(or d02 or d03....).YYYY-MM-DD_hh:mm:ss.nc files created. These are your WPS final output files that real.exe ingests.

Step 2. Running WRF

• Navigate to your WRF/WRFV3/run folder

• Step 2b. PROGRAM REAL: real data initialization program to set up the model domain and initialize WRF

1. First link your met files from metgrid.exe

cd run

ln -s /n/your_scratch_dir_path/METEM_FILEDUMP/met_em* .

      2. Edit namelist.input file.

&time #obviously should match namelist.wps.

&domains #obviously should match namelist.wps

#Customize other sections as needed.

#Particularly examine num_metgrid_levels and num_metgrid soil levels so they match with the input specified by the met_em* files. At command line:

ncdump -h met_em_d0XXXX.nc | more

num_metgrid_levels is typically in the header of the file

num_metgrid_soil_levels is close to the footer of the file

#Execute real.exe ; no real benefit to running as a parallel job so…

mpirun -np 1 ./real.exe >& run_real.log #make sure namelist.wrf chem_opt = 0 for this step for now.

      3. Examine the run_real.log file. You should see SUCCESS EM_REAL INIT printed at the end.

      4. Make sure you have the following files output from this step:

wrfinput_d0* (one for each domain)

wrfbdy_d01 (just for d01)

wrffdda_d01 (just for d01 since nudging is happening only in this domain)

• This is the point at which you have some separate additional steps (in red font, below) if you are running WRF-Chem. This is because successfully running these utilities for your simulation domain(s) requires the wrfinput (and wrfbdy, if using mozbc) files from real.exe. If you are not running WRF-Chem, skip this subsection and continue with Step 10 below.

       5. Process gridded chemical data (biogenic, boundary conditions, anthropogenic) using specified utilities (e.g., MEGAN bio, prep-chem-src, mozbc). Note: there are other options, like fire emissions ... you need to figure out what is relevant to your question.

       6. Make sure your binary emissions files are saved in WRFV3/test/em_real either as hard or soft links. They MUST be in this directory in some form.

       7. Turn your chemistry back on (chem_opt=XXX) in the namelist.wrf file.

       8. Run convert_emiss.exe to convert the chemistry data from binary intermediate to WRF input form

       9. You should have your relevant chem data in  WRFV3/test/em_real at this point, ready for ingestion by the wrf model. At this point, the key files input files that WRF-Chem expects in order to run successfully may now include, but is not limited, to the following:

           wrfbiochemi_<domain> #if you planned to include biogenic chem

           wrfchemi_00z_<domain> #if you planned to use anthropogenic chem, for two sets of time 00z and 12z

           wrfchemi_12z_<domain> 

           wrfbdy_d01 #Boundary file, should include chemical boundary conditions from mozbc for example if you chose to go that route.

           wrfinput_<domain> #your standard initialization file from real.exe

           wrffdda_<domain> #for the domain that you're nudging in (if you are nudging).

     10. If everything looks in order, run the wrf model. This is the only step that has significant benefit from running in parallel!

sbatch run_wrf.sh #Edit this script as needed; a template is provided.

     11. You should now have netcdf files of format, stored in /n/your_scratch_dir_path/WRFOUT/. If you are planning to drive an LPDM like STILT with these met files, these need to eventually be converted to .arl format using WRF2ARL directory contents. This is something that will be treated in a separate page. It is highly recommended that you get familiar with NCL (and have access to ncview) for post-processing and visualization of the wrfout files.

wrfout_d0#_YYYY-MM-DD_hh:mm:ss

PART III. Running WRF With Chemistry: A PM2.5 example

The purpose of this example is to take the general steps listed above and actually run a three nested domain WRF-Chem PM2.5 simulation over Beijing during the January 2013 severe haze event and compare with observations. We are going to run WRF-Chem for a total of 10 days from Jan 6 2013 00:00UTC to Jan 16 2013 00:00UTC. We establish 5 days for model spin-up such that the usable simulation time period is 5 days. Make sure you have a local copy of the /n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF directory and contents. You don't need to copy the geography data set.

Thanks to Meng Gao at Harvard China Project and Thomas Lauvaux at PSU for help with this and for providing their WRF-Chem namelist templates for this example!

At the end of this example you will have learned how to:

• Use the dust map from the WPS high-res geographical data set.
• Use prep-chem-sources and convert_emis to format a gridded anthropogenic PM2.5 emissions file to the WRF forecast domain
• Use gridded chemical data for biogenic emissions and chemical boundary conditions
• Run WRF-Chem with PM2.5-radiation feedbacks to simulate PM2.5 concentrations in Beijing in January 2013.
• Extract relevant components from wrfout netcdf files and examine/visualize using NCL.

Step 1. Run WPS

• Navigate to the WRF/WPS folder.
• First customize the WPS namelist.wps file such that it looks like the following. Note that with the exception of the geog_data_path, all path variables should point to a writable dump directory of your choice. It is recommended you write to regal or a similar scratch space you have access to; the dump files take up space and aren't needed again after the run so they don't have to be stored in a permanent location. Consult the WRF user guide for detailed explanation of the namelist variables.

&share
 wrf_core = 'ARW',
 max_dom = 3,
 start_date = '2013-01-06_00:00:00','2013-01-06_00:00:00','2013-01-06_00:00:00'
 end_date = '2013-01-16_00:00:00','2013-01-16_00:00:00','2013-01-16_00:00:00'
 interval_seconds = 21600,
 opt_output_from_geogrid_path = '/n/regal/wofsy_lab/adayalu/GEOGRID_DUMP/WRF01/'
 io_form_geogrid = 2,

&geogrid
 parent_id = 1, 1, 2,
 parent_grid_ratio = 1, 3, 3,
 i_parent_start = 1, 40, 17,
 j_parent_start = 1, 26, 21,
 e_we = 81, 49, 55,
 e_sn = 57, 49, 55,
 geog_data_res = 'usgs_30s+5m','usgs_30s+2m','usgs_30s+30s',
 dx = 81000,
 dy = 81000,
 map_proj = 'lambert',
 ref_lat = 35,
 ref_lon = 110,
 truelat1 = 30,
 truelat2 = 60,
 stand_lon = 116.397,
 geog_data_path = '/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF_GEOG_COMPLETE/geog'
/

&ungrib
 out_format = 'WPS',
 prefix = '/n/regal/wofsy_lab/adayalu/METEM_FILEDUMP/WRF01/FILE',
/

&metgrid
 fg_name = '/n/regal/wofsy_lab/adayalu/METEM_FILEDUMP/WRF01/FILE'
 io_form_metgrid = 2,
 opt_output_from_metgrid_path = '/n/regal/wofsy_lab/adayalu/METEM_FILEDUMP/WRF01/'
/

• Type:

ncl plotgrids_new.ncl

• Your domain should look like the following.

  

• Clear dump directories

rm -f /n/regal/wofsy_lab/adayalu/GEOGRID_DUMP/WRF01/*

rm -f /n/regal/wofsy_lab/adayalu/METEM_FILEDUMP/WRF01/*

• First call an interactive shell, then start running geogrid, ungrib, metgrid etc.

srun -n 1 --mem=10000 --pty --x11=first -p test -t 200 bash

mpirun -np 1 ./geogrid.exe 

And you should see something like:

Parsed 22 entries in GEOGRID.TBL
Processing domain 1 of 3
 Processing XLAT and XLONG
 Processing MAPFAC

[...etc for other variables, domains ...] 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of geogrid. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

• Now get ready to run ungrib. Since we will be using NCEP fnl data (GRIB2 format) for this run, we do some sleuthing to confirm the correct Vtable to use. Following the discussion here we download the appropriate Vtable separately (this one isn't included with the standard set of Vtables in WRFv3.6.1 because they recently updated the dataset). Note: I have already downloaded it and saved it: ungrib/Variable_Tables/Vtable.GFS_new

ln -sf ungrib/Variable_Tables/Vtable.GFS_new Vtable

./link_grib.csh /n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130106* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130107* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130108* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130109* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130110* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130111* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130112* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130113* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130114* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130115* \

/n/holylfs/LABS/kuang_lab/adayalu/WRF_GRIB2_NCEP_FNL/fnl_20130116* 

 

• Examine a sample GRIB2 files using the g2print.exe tool 

./util/g2print.exe /your_GRIB2_data_path/filename | more

mpirun -np 1 ./ungrib.exe > ungrib.out

• Examine the end of the ungrib.out file and make sure you see the following. Also, check out the contents of the file to get an idea of what ungrib does.

tail -3 ungrib.out 

!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!
! Successful completion of ungrib. !
!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!

• Run metgrid and make sure you get the "successful completion" message.

mpirun -np 1 ./metgrid.exe

Step 2. Run real.exe to get necessary intermediate files for WRF-Chem utilities.

• Navigate to WRFV3/test/em_real
• Copy namelist.input_pm25Beijing to namelist.input. Take a look at the namelist and confirm the dates and domain match your wps namelist
• Make sure chem_opt = 0 for this step.
• Change the filepath in history_outname to reflect where you want your wrfchem output files to be saved.
• Link the metgrid files to the test/em_real directory and make sure the namelist num_metgrid_levels and num_metgrid_soil_levels match what's in the metgrid files.

ln -s /n/your_scratch_dir_path/METEM_FILEDUMP/met_em* .

• Run real.exe to output the necessary intermediate files for chem data processing. Check that the tail of the file says SUCCESS COMPLETE REAL_EM INIT 

mpirun -np 1 ./real.exe >& run_real.log

• You should have five files that real.exe produced. These currently only contain meteorological info, which is fine for running plain WRF-ARW. But – since we are running WRF-Chem – we need to run the additional utilities that either use the info contained in these files to map chemical data correctly (e.g., prep-chem-src) or modify the contents directly (e.g., mozbc for inputs of chemical boundary conditions into wrfbdy_d01).

wrfbdy_d01 #the outermost domain parameter boundary condition file

wrffdda_d01 #nudging file, we requested nudging in the outer domain

wrfinput_d<domain> #initial condition files for each of your study domains.

Step 3. Prep the gridded chemical data initial and boundary conditions using MOZBC

• Navigate to the mozbc folder from the $CLIMATE_MODELS folder.

cd mozbc

• Recall the MOZBC utility is used to modify the wrfinput/wrfbdy files to contain chemical boundary conditions.
• From the namelist.input file for this exercise in WRFV3/test/em_real, we see that chem_opt = 10. Referring to the WRF-Chem user manual for more details. This chemistry option uses the CBMZ  chemical mechanism and MOSAIC using 8 sectional aerosol bins. We match this with what MOZBC provides as sample namelists and we pick CBMZ-MOSAIC_8bins.inp as our mozbc namelist file. Copy this to a new file:

cp CBMZ-MOSAIC_8bins.inp mozbc.inp

Step 4. Run WRF

Step XX. Prep the anthropogenic PM2.5 emissions data.

WRF-Chem only: Processing Gridded Chemical Data

In order to run WRF-Chem you need gridded chemical data related to the chemistry of interest. The gridded chemical data must be converted to a WRF-Chem compatible format and MUST be located in 

WRFV3/test/em_real

prior to running the model. You should have gridded chemical data at the resolution for each of the domains in your simulation. Read the WRF-Chem emissions guide for detailed guidelines.

To summarize, you will typically require the following sources of gridded chemical data: 

(1) Chemical boundary conditions (mozbc utility)

(2) Biogenic contributions (e.g., MEGAN)

(3) Anthropogenic emissions (processed via convert_emiss.exe; prep-chem-sources utility...; these are discussed below.). For non-contiguous US regions, you would likely need to use the prep-chem-sources grid mapping utility which converts the gridded emissions to your WRF forecast domain.

Note: Prep chem sources can be tricky to compile and run. You can download it here if you want to start from scratch: ftp://aftp.fsl.noaa.gov/divisions/taq/global_emissions. This guide is helpful: 

ftp://ftp.cptec.inpe.br/brams/BRAMS/documentation/guide-PREP-CHEM-SRC-1.5.pdf

There are some typos, and missing details in the pdf guide above, so a modified version of the instructions (and Paul Edmon's help rebuilding HDF5 to fix an error message) enabled successful compilation of the utility. The modified instructions are located here:

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF/README_prepchem_modifications

In any case, a compiled version of prep chem sources utility using the instructions above is located here:

/n/holylfs/INTERNAL_REPOS/CLIMATE_MODELS/WRF_CHEM_3-6-1/WRF/PREP-CHEM-SRC-1.5/bin/prep_chem_sources_RADM_WRF_FIM_SIMPLE.exe

• We will use the 2012 December MEIC emissions, provided in netcdf format (http://www.meicmodel.org/) to approximate Jan 2013 emissions. Note: these MEIC files are not for external distribution. If you are going to use it outside of this tutorial example, please first contact Qiang Zhang (qiangzhang@tsinghua.edu.cn). 
• The MEIC emissions are from 5 sectors: Agriculture, Residential, Industrial, Transportation, Power. The quantities are in Mg/Month. Let's first examine the file contents. (Note: this assumes basic familiarity with NCL...https://www.ncl.ucar.edu/Document/Functions/Built-in/addfile.shtml).

> ncl_filedump -c 2012_12__transportation__PM2.5.nc

filename: 2012_12__transportation__PM2.5
path: 2012_12__transportation__PM2.5.nc
 file global attributes:
 title : 
 source : MEIC Core gridding - GMTEngine
 dimensions:
 side = 2
 xysize = 64000
 variables:
 double x_range ( side ) #reading this in NCL -- range is 70E to 150E

double y_range ( side )  #reading this in NCL-- range is 10N to 60N

double spacing ( side ) #reading this in NCL-- spacing is 0.25 degrees

integer dimension ( side ) #reading this in NCL-- X is 320 gridcells, Y is 200 gridcells.

float z ( xysize ) #Values. Mg/month.
 scale_factor : 1
 add_offset : 0
 node_offset : 1
 nodata_value : -9999

• Now navigate to your PREP-CHEM-SRC-1.5 folder. At this point, we are referring to page 6 of the prep chem user guide. 

cd bin

ln -s /yourpath/PREP-CHEM-SRC-1.5/extra/UserData #make a symbolic link to the chemical data in your prep chem bin directory

• The sample MEIC emissions we will use here have already been copied into this directory. 

PART V. Running WRF-Chem for real cases in Large Eddy Simulation (LES) Mode: A Beijing PM2.5 Case Study

Running WRF (and/or WRF-Chem) on Harvard Odyssey: IDEALIZED cases

Miscellaneous links

Google Docs: https://docs.google.com/document/d/1Jls4FlWIOIhMlCzMPWm6_aBZqx_Axxe8RMcKjdILDFg/

Ding's notes: global_WRF_on_Odyssey.pdf

Optimizing performance: https://www2.cisl.ucar.edu/software/community-models/optimizing-wrf-performance

Running on Cheyenne: http://www2.mmm.ucar.edu/wrf/users/cheyenne-note.html