========== 20180301: 1. For the gmax control and experiments, Randy said there is no need to spin up and just run for a couple of years first to see. Also, write out monthly mean total conductance (=AR1/rcsat+AR2/rcuns) and unstressed conductance (=1/rcsat). Need to add this to the code for the control run (and for the experiment later as well) and recompile. See /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/process_cn.F90.gmax_ctrl /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/gmax_ctrl/. cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/run nedit lenkf.0.j & deleted "qsub lenkf.1.j" Check the executable: ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69763390 2018-03-01 11:04 /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/build/Linux/bin/LDASsaCN_mpi.x* ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 90 2018-02-28 15:52 catchcn_internal_rst -> /discover/nobackup/fzeng/offline_restarts/DC0288xPC0181_DE0360xPE0180/catchcn_internal_rst Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/run ./lenkf.0.i It's running. Stopped it at date_time_new 19800101_070730z. qsub lenkf.0.j 2. Created ~/Catchment/grid_restore_gmax.f90 and compiled it. 3. For the gmax experiment: Modified and used process_cn.F90.gmax_exp and compute_rc.F90.gmax_exp01. Compiled. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/gmax_exp01/ 4. Reviewed my 20171215 and 20171218 notes on how to run original CLM4.5 (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily) 5. Set up the stomatal conductance experiment. Modified DC0288xPC0181.bat and DC0288xPC0181_CN.exe in /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/gmax_exp01/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0004 /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run/DC0288xPC0181_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run/DC0288xPC0181.bat --runmodel --monthsperjob 36 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_exp01/run nedit lenkf.0.j & changed "-restart_path ../input/restart/DC0288xPC0181_DE0360xPE0180 \" to "-restart_path ../input/restart/ \" cd ../input/restart ln -s $NOBACKUP/offline_restarts/DC0288xPC0181_DE0360xPE0180/catchcn_internal_rst /bin/rm DC0288xPC0181_DE0360xPE0180 cd ../../run Check the executable: ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_exp01/build/Linux/bin/LDASsaCN_mpi.x Check the restart file: ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 90 2018-03-01 16:56 catchcn_internal_rst -> /discover/nobackup/fzeng/offline_restarts/DC0288xPC0181_DE0360xPE0180/catchcn_internal_rst Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_exp01/run ./lenkf.0.i It crashed at date_time_new 19800101_043000z forrtl: error (72): floating overflow Image PC Routine Line Source libirc.so 00002AAAAB590961 Unknown Unknown Unknown libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2B3392 Unknown Unknown Unknown libc.so.6 00002AAAABA16910 Unknown Unknown Unknown LDASsaCN_mpi.x 00000000017D1607 Unknown Unknown Unknown LDASsaCN_mpi.x 000000000075EFAD compute_rc_mod_mp 657 compute_rc.F90 LDASsaCN_mpi.x 000000000075A760 compute_rc_mod_mp 361 compute_rc.F90 L361 compute_rc.F90: call stomata(nch, nveg, ei, ea, o2, co2, rb, dayl_fac, pbot, ityp, tm, tc, & btran, parsun, slasun, rssun, psnsun, sifsun) ! sunlit L657 compute_rc.F90: vcmx = lnc * flnr(ivt) * fnr * act / f2(tc) * btran(n) * dayl_factor(n) Why? Recompiled with "gmake clean" and "gmake install BOPT=g". Used Allinea DDT to debug. Again it crashed at date_time_new 19800101_043000z forrtl: error (72): floating overflow L657 compute_rc.F90: vcmx = lnc * flnr(ivt) * fnr * act / f2(tc) * btran(n) * dayl_factor(n) tc = -274.91217 ivt = 6 n = 12620 act = 2.3722091e-10 fnr = 7.15999985 lnc = 1.21197033 tc ! leaf temperature (degree celsius) It should not go below -273.15**! Why is it so low? Can I narrow down a small region where the problematic tile is located? n = 12620 The 12620th tile in gmax_exp01.ldas_domain.txt has tile id 12618. However, the land cover in tile 12618 is (see /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0288xPC0181_DE0360xPE0180/clsm/CLM_veg_typs_fracs) 12618 8009 18 19 13 13 49.20 16.88 33.93 0.00 16 13 66.07 33.93 12610 8006 13 13 18 19 57.45 0.00 31.88 10.67 13 16 57.45 42.55 12611 8006 13 13 18 19 50.70 0.00 37.18 12.12 13 16 50.70 49.30 12612 8007 13 13 18 19 77.41 0.00 16.64 5.95 13 16 77.41 22.59 12613 8007 18 19 13 13 47.89 15.75 36.36 0.00 16 13 63.64 36.36 12614 8008 13 13 18 19 68.35 0.00 23.50 8.15 13 16 68.35 31.65 12615 8008 13 13 18 19 68.42 0.00 24.25 7.33 13 16 68.42 31.58 12616 8008 18 19 13 13 53.88 15.49 30.63 0.00 16 13 69.37 30.63 12617 8009 13 13 18 19 77.78 0.00 16.54 5.69 13 16 77.78 22.22 12618 8009 18 19 13 13 49.20 16.88 33.93 0.00 16 13 66.07 33.93 12619 8009 18 19 13 13 40.61 12.94 46.45 0.00 16 13 53.55 46.45 12620 8009 18 19 13 13 45.55 14.85 39.60 0.00 16 13 60.40 39.60 12621 8010 18 19 13 13 44.90 15.50 39.60 0.00 16 13 60.40 39.60 12622 8010 18 19 13 13 41.25 12.61 46.14 0.00 16 13 53.86 46.14 12623 8010 18 19 13 13 45.09 13.72 41.18 0.00 16 13 58.82 41.18 12624 8011 18 19 14 15 44.40 41.40 7.35 6.85 16 14 85.80 14.20 12625 8012 18 19 14 15 49.29 43.42 3.88 3.41 16 14 92.71 7.29 None of these has vegetation type 6. Modified the code to identify the problematic tile. date_time_new 19800101_042230z Canopy temperature tl is below 0K: -2.700226 tile_id,ivt: 110897 1 n, nv: 2483 1 Canopy temperature tl is below 0K: -6.314983 tile_id,ivt: 104065 2 n, nv: 439 1 Canopy temperature tl is below 0K: -6.555436 tile_id,ivt: 100921 19 n, nv: 2082 2 Canopy temperature tl is below 0K: -5.369931 tile_id,ivt: 25330 11 n, nv: 1892 2 Canopy temperature tl is below 0K: -6.575928 tile_id,ivt: 29003 19 n, nv: 881 2 Canopy temperature tl is below 0K: -7.193978 tile_id,ivt: 121892 2 n, nv: 3902 1 Canopy temperature tl is below 0K: -3.0228279E-03 tile_id,ivt: 116314 19 n, nv: 3112 2 Canopy temperature tl is below 0K: -0.4784725 tile_id,ivt: 202816 4 n, nv: 1067 1 Canopy temperature tl is below 0K: -2.623298 tile_id,ivt: 200679 4 n, nv: 3912 1 Canopy temperature tl is below 0K: -1.752160 tile_id,ivt: 182068 6 n, nv: 206 1 Canopy temperature tl is below 0K: -0.3496678 tile_id,ivt: 194897 4 n, nv: 3098 1 Do small regional run on DDT for tile 110897. land cover: 110897 67723 1 1 2 2 62.59 0.00 37.41 0.00 2 3 62.59 37.41 lat and lon: 100 0.208721015543E-05 0.996422513940E+02 0.282416575756E+02 225 119 0.620829029974E-02 34209 67723 67555 0.999999999998E+00 67555 lon = 99.64 lat = 28.24 ========== 20180302: 1. Continue working on 5 on 20180301. 2. Feb report 3. Processed March forecast Can't scp the plots to my laptop. Will scp them to my desktop on Monday and send them to Randy. Hi Randy, Attached are the soil moisture percentile and precipitation & 2m air temperature anomaly figures from the March 2018 GEOS5 ensemble forecasts (initialized 00z 25 February). The initialization in general agrees with the current top 1M soil moisture percentile product on http://www.emc.ncep.noaa.gov/mmb/nldas/drought/ and the root zone soil moisture percentile product on http://drought.unl.edu/MonitoringTools/NASAGRACEDataAssimilation.aspx. The exception is that we are too wet in the regions near the Great Lakes. These files are large, so I uploaded them to OpenDrive. Please click on the link below to view or download the files. You have 10 days from 02-09-2018 to download the file(s) from the following URL: https://opendrive.gsfc.nasa.gov/shortauth/c/OvRhxjET Please let me know if you have any questions. Thanks, Fanwei 4. Checked if the gmax parameters are read in to the run correctly: What's written to the /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/gmax_params_DC0288xPC0181 file: 0.5415040 0.5645073 0.6104813 0.9491952 0.8124695 -0.9694139 5.0298616E-02 0.4616311 -7.399760 -366.2322 -336.9393 -95.60806 -117.4296 -128.1884 -234.0207 -96.42458 -189.1814 -43.56734 -30.22899 What's read: 0.5415040 0.5645073 0.6104813 0.9491952 0.8124695 -0.9694139 5.0298616E-02 0.4616311 -7.399760 -366.2322 -336.9393 -95.60806 -117.4296 -128.1884 -234.0207 -96.42458 -189.1814 -43.56734 -30.22899 They are identical. Good. 5. Completed BUSINESS ETHICS - WHAT EMPLOYEES NEED TO KNOW 5 required by SSAI. ========== 20180305: 1. Sent the forecast plots to Randy 2. Continued working on 5 on 20180301. Did a small regional run on DDT for tile 110897. land cover: 110897 67723 1 1 2 2 62.59 0.00 37.41 0.00 2 3 62.59 37.41 lat and lon: 100 0.208721015543E-05 0.996422513940E+02 0.282416575756E+02 225 119 0.620829029974E-02 34209 67723 67555 0.999999999998E+00 67555 lon = 99.64 lat = 28.24 cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_exp01/run driver_inputs_DC0288xPC0181_GLOBAL.nml: minlon = 99. ! min longitude maxlon = 100. ! max longitude minlat = 28. ! min latitude maxlat = 29. ! max latitude ./lenkf.0.ddt Crashed: date_time_new 19800101_042230z Canopy temperature tl is below 0K: -2.700226 tile_id,ivt: 110897 1 n, nv: 5 1 Talked to Randy about this. Randy emailed Alex Konings. 3. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_47 and clm4.5_DE720_48. 4. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 27, 28 clm4_IPAR_DE720: 33, 34 clm4_IPAR_DReqDF_DE720: 32, 33 clm4_pIPAR_DE720: 20, 21 clm4_pTair_DE720: 20, 21 5. Sanity check: Plot Princeton IPAR climatology Plot IPAR climatology from the pIPAR experiment Plot Princeton Tair climatology Plot Tair climatology from the pTair experiment Can use these scripts below: ~/Catchment/M2n5P_m0001/princetonVSmerra2/compare_princeton_merra2.m ~/Catchment/matlab/m_map/fz_plot_princeton_merra2_diff.m I expect to see difference close to 0. However, the difference is quite large. Why? Plotted Tair from clm4_DE720 model outputs and TLML from monthly MERRA2 files for Jul 1980 and Jul 2012. They look the same. ========== 20180306: 1. Investigated why the difference in Tair between p0007s_67 and clm4_pTair_DE720 is not close to zeros, and same for the difference in IPAR between p0007s_67 and clm4_pIPAR_DE720. Checked if I computed the climatology correctly. Didn't see anything wrong. Checked model code modifications. Couldn't find anything wrong. It turned out to be the mismatch in the land mask between the two. Modified the plotting script to shift one of them to match the other as much as possible. Now the difference in Tair is pretty small. 2. Met with Eunjee about the CO2 experiment analysis. ========== 20180307: 1. Using the updated plotting script, the IPAR difference is now pretty small. 2. Randy wants these 4 fields from MERRA2: -- EVLAND (evaporation from the land portion of the grid cell) -- total runoff -- net radiation (longwave+shortwave) incident on the surface -- saturated specific humidity near the surface Look for them in MERRA2 file specification: EVLAND: tavg1_2d_lnd_Nx RUNOFF: overland runoff including throughflow, kg/m2/s, tavg1_2d_lnd_Nx BASEFLOW: baseflow flux, kg m-2 s-1, tavg1_2d_lnd_Nx LWGNT: surface net downward longwave flux, W m-2, tavg1_2d_rad_Nx SWGNT: surface net downward shortwave flux, W m-2, tavg1_2d_rad_Nx total runoff = RUNOFF + BASEFLOW net radition = LWGNT + SWGNT Can't find saturated specific humidity near the surface. Skip it for now. Modified ~/merra2/extract_daily.gs and ran it to extract a total of 12 fields. Modified merra2.ctl and plot_merra2.gs under ~/merra2 to include the 3 new fields. Plot the data and check. 3. Ran the original CLM4.5 to see if livestemn could go negative. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180308: 1. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 29 clm4_IPAR_DE720: 35 clm4_IPAR_DReqDF_DE720: 34 clm4_pIPAR_DE720: 22 clm4_pTair_DE720: 22 2. Checked the MERRA2 data 3. Prepare for meeting with Randy CLM4.5: GPP and NEE Add print statement to original CLM4.5 and ran it 4. Met with Randy and Eunjee ========== 20180309: 1. Prepared for meeting with Randy 2. Met with Randy and Eunjee 3. Met with Eunjee about hindcast 4. Ran Sarith's updated process_seasonal_forecast.csh. Got this error message. % Program caused arithmetic error: Floating underflow convert: No match. convert: No match. /bin/mv: cannot stat `w2_perc_conus.png': No such file or directory /bin/mv: cannot stat `w2_perc_global.png': No such file or directory /bin/mv: cannot stat `anom_PRECTOT_conus.png': No such file or directory /bin/mv: cannot stat `anom_PRECTOT_global.png': No such file or directory /bin/mv: cannot stat `anom_T2M_conus.png': No such file or directory /bin/mv: cannot stat `anom_T2M_global.png': No such file or directory 5. Set up a run using MERRA2 forcing, 5min dtstep and 5min DTCN, and write 5min output: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -pr clm4_DE720_template clm4_DE720_5min cd clm4_DE720_5min cd run nedit lenkf.0.j & nedit driver_inputs_0.5D_GLOBAL.nml & cd ../input/restart/ ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_29/output /bin/rm rst_clm4 cd ../../run Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_5min/run ./lenkf.0.i Finished 5 days of simulation successfully. 6. Created GrADS ~/Catchment/CLM4/clm4_DE720_5min.ctl for the model output. ========== 20180312: 1. Emailed Randy to let him know that the MERRA2 data is ready on land01. 2. Used Sarith's updated scripts to produce the soil moisture percentile plots, and precipitation and temperature plots, and sent them to Randy. 3. Updated higgins rain data through 11 Mar 2018. 4. Read Eunjee's CO2 manuscript. 5. Talked with Justin about the negative livestemn issue. He showed me how to check the experiment log files and how to make the run use a particular restart file. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180313: 1. Read Eunjee's CO2 manuscript and sent her my comments. 2. Met with Eunjee about the hindcast fields needed to run Catchment-CN. 3. Do offline runs starting from 1 Aug of 2010, 2011 and 2012 and just run for 1 day each, using restart files in /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Yehui_03 Then plot global LAI. (1) The run starting from 1 Aug 2010: /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C > rm CF0090x6C [remove the symbolic link CF0090x6C -> CF0090x6C_03] (1.1) Preparation for ldsetup: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/run nedit CF0090x6C.bat CF0090x6C_CN.exe driver_inputs_CF0090x6C_GLOBAL.nml & (1.2) Run ldsetup: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/CF0090x6C/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/run/CF0090x6C_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/run/CF0090x6C.bat --runmodel --monthsperjob 36 --landmodel catchCN (1.3) Get the restart file ready: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/CF0090x6C_GLOBAL/rs/ens0000 mkdir -p Y2010/M08 cd Y2010/M08 cp -p /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Yehui_03/CF0090x6C.ens0000.catchcn_ldas_rst.20100801_2100z . (1.4) Run for 3 hours first from 20100801_2100z through 20100802_0000z: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run nedit lenkf.0.j & (a) change "-start_year 2010 -start_month 8 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \" to "-start_year 2010 -start_month 8 -start_day 1 -start_hour 21 -start_min 0 -start_sec 0 \" (b) change "-end_year 2010 -end_month 9 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \" to "-end_year 2010 -end_month 8 -end_day 2 -end_hour 0 -end_min 0 -end_sec 0 \" (c) make sure "-restart_id CF0090x6C \" make sure it's x-hourly output in driver_inputs_CF0090x6C_GLOBAL.nml ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69767604 2017-07-21 11:20 /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_20100801/build/Linux/bin/LDASsaCN_mpi.x* /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 70 2018-03-14 09:59 output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/ Run it interactively: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.0.i > log.lenkf.0.i (1.5) Run for 1 day from 20100802_0000z through 20100806_0000z. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run cp lenkf.0.i lenkf.1.i nedit lenkf.1.i & (a) change "-start_year 2010 -start_month 8 -start_day 1 -start_hour 21 -start_min 0 -start_sec 0 \" to "-start_year 2010 -start_month 8 -start_day 2 -start_hour 0 -start_min 0 -start_sec 0 \" (b) change "-end_year 2010 -end_month 8 -end_day 2 -end_hour 0 -end_min 0 -end_sec 0 \" to "-end_year 2010 -end_month 8 -end_day 6 -end_hour 0 -end_min 0 -end_sec 0 \" (c) change "-restart_path ../input/restart/output \" to "-restart_path ../output \" make sure it's daily output in driver_inputs_CF0090x6C_GLOBAL.nml interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.1.i > log.lenkf.1.i Crashed!! Why? Will investigate this tomorrow. ========== 20180314: 1. Repeated 3(1) on 20180313 to investigate the issue. Didn't change anything in the procedures above but this time it works! Continued from 3(1) above. (1.6) Post-processing: convert from tile-space to grid-space for visualization. ~/Catchment/CF0090x6C > tile2grid_CF0090x6C_daily CF0090x6C (1.7) Plot on GrADS using ~/Catchment/CF0090x6C/plot_lai.gs. (2) The run starting from 1 Aug 2011: (2.1) Get the restart file ready: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/CF0090x6C_GLOBAL/rs/ens0000 mkdir -p Y2011/M08 cd Y2011/M08 cp -p /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Yehui_03/CF0090x6C.ens0000.catchcn_ldas_rst.20110801_2100z . (2.2) Run for 3 hours first from 20110801_2100z through 20110802_0000z: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run cp lenkf.0.i lenkf.2.i nedit lenkf.2.i & (a) set -start_year 2011 -start_month 8 -start_day 1 -start_hour 21 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 8 -end_day 2 -end_hour 0 -end_min 0 -end_sec 0 \ (c) make sure "-restart_id CF0090x6C \" make sure it's x-hourly output in driver_inputs_CF0090x6C_GLOBAL.nml ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69767604 2017-07-21 11:20 /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_20100801/build/Linux/bin/LDASsaCN_mpi.x* /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 70 2018-03-14 09:59 output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/ Run it interactively: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.2.i > log.lenkf.2.i (2.3) Run for 1 day from 20110802_0000z through 20110806_0000z. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run cp lenkf.1.i lenkf.3.i nedit lenkf.3.i & (a) set -start_year 2011 -start_month 8 -start_day 2 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 8 -end_day 6 -end_hour 0 -end_min 0 -end_sec 0 \ (c) set "-restart_path ../output \" make sure it's daily output in driver_inputs_CF0090x6C_GLOBAL.nml interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.3.i > log.lenkf.3.i (2.4) Post-processing: convert from tile-space to grid-space for visualization. ~/Catchment/CF0090x6C > tile2grid_CF0090x6C_daily CF0090x6C (2.5) Plot on GrADS using ~/Catchment/CF0090x6C/plot_lai.gs. (3) The run starting from 1 Aug 2012: (3.1) Get the restart file ready: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/CF0090x6C_GLOBAL/rs/ens0000 mkdir -p Y2012/M08 cd Y2012/M08 cp -p /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Yehui_03/CF0090x6C.ens0000.catchcn_ldas_rst.20120801_2100z . (3.2) Run for 3 hours first from 20120801_2100z through 20120802_0000z: cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run cp lenkf.0.i lenkf.4.i nedit lenkf.4.i & (a) set -start_year 2012 -start_month 8 -start_day 1 -start_hour 21 -start_min 0 -start_sec 0 \ -end_year 2012 -end_month 8 -end_day 2 -end_hour 0 -end_min 0 -end_sec 0 \ (c) make sure "-restart_id CF0090x6C \" make sure it's x-hourly output in driver_inputs_CF0090x6C_GLOBAL.nml ls -l /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69767604 2017-07-21 11:20 /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_20100801/build/Linux/bin/LDASsaCN_mpi.x* /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 70 2018-03-14 09:59 output -> /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C_03/output/ Run it interactively: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.4.i > log.lenkf.4.i (3.3) Run for 1 day from 20120802_0000z through 20120806_0000z. cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run cp lenkf.1.i lenkf.5.i nedit lenkf.5.i & (a) set -start_year 2012 -start_month 8 -start_day 2 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2012 -end_month 8 -end_day 6 -end_hour 0 -end_min 0 -end_sec 0 \ (c) set "-restart_path ../output \" make sure it's daily output in driver_inputs_CF0090x6C_GLOBAL.nml interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C/run ./lenkf.5.i > log.lenkf.5.i (3.4) Post-processing: convert from tile-space to grid-space for visualization. ~/Catchment/CF0090x6C > tile2grid_CF0090x6C_daily CF0090x6C (3.5) Plot on GrADS using ~/Catchment/CF0090x6C/plot_lai.gs. 2. Two IDS meetings at UMD (11am through ~4pm) ========== 20180315: 1. Sent the global LAI plots from the offline simulations on 20180314 to Randy. 2. Made two difference maps of Aug. 2 LAI for Randy. 1) 2011-2010 2) 2012-2010 3. Helped Lei Ma (at UMD, IDS project) with the CT CO2 data pre-processing procedures. 4. Re-made the Jan 1980 Princeton SWdown plot 5. Plot SWdown from clm4_DE720_5min (ran on 20180309). Found that SWdown is 0 in the first day. It's because I set lat to be 77.25N and lon to be 23.5E. Correct this in driver_inputs_0.5D_GLOBAL.nml. See below: minlon = 77. ! min longitude maxlon = 77.5 ! max longitude minlat = 23. ! min latitude maxlat = 23.5 ! max latitude Re-do the run. interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_5min/run ./lenkf.0.i > log.lenkf.0.i Plot 5min SWdown output using ~/Catchment/CLM4/verify_exp.gs The 5min SWdown interpolated from hourly MERRA2 SWdown looks reasonable. Compared subroutine interpolate_to_timestep in clsm_ensdrv_drv_routines.F90 between /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp and /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp. They are the same. 5. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_49 and clm4.5_DE720_50. 6. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 30, 31, 32 clm4_IPAR_DE720: 36, 37, 38 clm4_IPAR_DReqDF_DE720: 35, 36, 37 clm4_pIPAR_DE720: 23, 24, 25 clm4_pTair_DE720: 23, 24, 25 ========== 20180319: 1. Continued working on 6 of 20180315 above. Discover nobackup was down most of the day. 2. Contacted Sarith about the restart file for GEOSldas. 3. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_51. 4. Checked the original CLM4.5 output: GPP in Jul 1901, 1910 and 1919 looks correct. ========== 20180320: 1. Create a restart file for GEOSldas following Sarith's instructions: Check out: cd /discover/nobackup/fzeng/offline_code cvs co -r GEOSldas_m4-17_6 GEOSldas_m4 mv GEOSldas GEOSldas_m4-17_6 Build: cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src source g5_modules make install The executable is Linux/bin/GEOSldas.x. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ln -s e0004s_transientCO2_05 e0004s_transientCO2 cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M03 ln -s e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150301_0000z e0004s_transientCO2.catchcn_internal_rst.20150301_0000 Set up a run: cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/Applications/LDAS_App ./ldas_setup sample --exeinp > create_restart.exec ./ldas_setup sample --batinp > create_restart.bat nedit create_restart.exec create_restart.bat & (following /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run/) ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ ./create_restart.exec ./create_restart.bat (ldas_setup can also be run from /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/Linux/bin) 2. Prepared for meeting with Randy. 3. Met with Randy and Eunjee. 4. Set up a run using Princeton forcing, 5min dtstep and 5min DTCN, and write 5min output: cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -pr clm4_DE720_template clm4_DE720_5min_P cd clm4_DE720_5min_P cd run nedit lenkf.0.j & cp -p ../../clm4_DE720_5min/run/driver_inputs_0.5D_GLOBAL.nml . cd ../input/restart/ ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_32/output /bin/rm rst_clm4 cd ../../run Did an interactive run to make sure it works: interactive.py -A sp3 -n 1 -a g0620 -X --debug cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_5min_P/run ./lenkf.0.i > log_lenkf.0.i date_time_new 19800101_000500z LDAS ERROR (3000) from get_Princeton_netcdf: timing ERROR!! Will take a look tomorrow. ========== 20180321: 1. Continued working on 4 of 20180320. Changed "-force_dtstep 3600 \" to "-force_dtstep 10800 \" It ran successfully. Plotted SWdown using ~/Catchment/CLM4/verify_exp.gs. There are spikes in the interpolated 5min SWdown, confirming what I saw earlier. 2. Completed GSFC PROPERTY ACCOUNTABILITY TRAINING, Environmental Awareness Training, and FY2018 CYBERSECURITY AND SENSITIVE UNCLASSIFIED INFORMATION AWARENESS TRAINING on Satern. ========== 20180322: 1. Followed Sarith's instructions (see email) to convert my M09 20150301 restart file for GEOSldas: (1) already done on 20180320: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M03 ln -s e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150301_0000z e0004s_transientCO2.catchcn_internal_rst.20150301_0000 (2) [NO NEED. The reordered file /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts/catchcn_internal_rst.20150301 is ready.] run the command to reorder the M09 restart file cd /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts mpirun -np 1 bin/mk_LDASsaRestarts -b BCSDIR -d 20150301 -e EXPNAME -l EXPDIR -m MODEL -s SURFLAY(20/50) -r Y -t TILFILE At this time: /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts > ls -l bin/mk_LDASsaRestarts -rwxr-xr-x 1 fzeng g0620 27268133 2018-03-20 11:47 bin/mk_LDASsaRestarts* (3) change the L66 in /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ mk_LDASsaRestarts.F90: from InCNRestart = '/gpfsm/dnb42/projects/p16/ssd/land/l_data/LandRestarts_for_Regridding/CatchCN/M09/20151231/catchcn_internal_rst', & to InCNRestart = '/discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts/catchcn_internal_rst.20150301', & i.e. the reordered file, (4) from /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ mk_LDASsaRestarts.F90: setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/ source $ESMADIR/src/g5_modules gmake install At this time: /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts > ls -l bin/mk_LDASsaRestarts -rwxr-xr-x 1 fzeng g0620 27268133 2018-03-22 11:00 bin/mk_LDASsaRestarts* (5) then run equivalent command mpirun -map-by core --mca btl ^vader -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m ${MODEL} -s 50 -j Y What I did: interactive.py -A sp3 -n 56 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts source /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/g5_modules mpirun -map-by core --mca btl ^vader -np 56 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ -t CF0090x6C_DE0360xPE0180-Pfafstetter.til -m catchcn -s 50 -j Y but got some error messages: ------------------------------------------------------- Primary job terminated normally, but 1 process returned a non-zero exit code.. Per user-direction, the job has been aborted. ------------------------------------------------------- bin/mk_LDASsaRestarts: error while loading shared libraries: libmpi_usempif08.so.20: cannot open shared object file: No such file or directory bin/mk_LDASsaRestarts: error while loading shared libraries: libmpi_usempif08.so.20: cannot open shared object file: No such file or directory bin/mk_LDASsaRestarts: error while loading shared libraries: libmpi_usempif08.so.20: cannot open shared object file: No such file or directory bin/mk_LDASsaRestarts: error while loading shared libraries: libmpi_usempif08.so.20: cannot open shared object file: No such file or directory ... After adding "source /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/g5_modules": forrtl: No such file or directory forrtl: severe (29): file not found, unit 11, file /gpfsm/dnb31/fzeng/Catchment/hindcast/create_restart/mk_restarts/OutData1/OutTileFile Image PC Routine Line Source mk_LDASsaRestarts 00000000012C9913 Unknown Unknown Unknown mk_LDASsaRestarts 00000000012E511E Unknown Unknown Unknown mk_LDASsaRestarts 00000000004671A2 mk_ldassarestarts 3440 mk_LDASsaRestarts.F90 2. Showed Randy the plot of SWdown from clm4_DE720_5min_P. Will talk about this with him tomorrow. 3. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). ========== 20180323: 1. Continued working on 1 above on 20180322. 2. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). ========== 20180326: 1. Asked Sarith to help me process the April 2018 forecast and generate the soil moisture percentile maps, as well as precipitation & 2m air temperature anomaly maps. 2. Updated higgins data through 25 March 2018. 3. Found that all my runs stoppe a week ago!! Resumed them. (1) clm4.5_DE720: stopped at 19990717. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5/clm4.5_DE720/output/global > ls -l rs/ens0000/Y1999/M07 total 397824 -rw-r--r-- 1 fzeng g0620 407366395 2018-03-19 15:57 clm4.5_DE720.ens0000.catchcn_ldas_rst.19990701_0000z (2) clm4_DE720: stopped at 20100829. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720/output/global > ls -l rs/ens0000/Y2010/M08 total 284640 -rw-r--r-- 1 fzeng g0620 291445863 2018-03-19 09:17 clm4_DE720.ens0000.catchcn_ldas_rst.20100801_0000z (3) clm4_IPAR_DE720: stopped at 20030516. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_IPAR_DE720/output/global > ls -l rs/ens0000/Y2003/M05 total 284640 -rw-r--r-- 1 fzeng g0620 291445863 2018-03-17 01:00 clm4_IPAR_DE720.ens0000.catchcn_ldas_rst.20030501_0000z (4) clm4_IPAR_DReqDF_DE720: change the directory name of the last cycle before the current cycle started, so the current cycle couldn't find the restart file. cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_IPAR_DReqDF_DE720/input/restart /bin/rm output ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_IPAR_DReqDF_DE720_37/output cd ../../run/ qsub lenkf.0.j (5) clm4_pIPAR_DE720: same as (4) above cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/input/restart /bin/rm output ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720_25/output cd ../../run/ qsub lenkf.0.j (6) clm4_pTair_DE720: same as (4) above cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/input/restart /bin/rm output ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720_25/output 4. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). ========== 20180327: 1. Continued on 1(5) on 20180322. Sarith helped me resolve the issue: the steps below can be found in Applications/LDAS_App/process_rst.csh. 999 12:09 cd /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts 1000 12:11 setenv ESMADIR /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_6/ 1001 12:12 source $ESMADIR/src/g5_modules 1003 12:16 mkdir OutData1 1004 12:16 mkdir OutData2 1006 12:16 cd OutData1/ 1008 12:17 ln -s /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/CF0090x6C_DE0360xPE0180-Pfafstetter.til OutTileFile 1009 12:18 cd ../OutData2/ 1011 12:18 /bin/rm clsm # This clsm is for the previous step -- reordering 1012 12:18 /bin/rm OutTileFile # This OutTileFile is for the previous step -- reordering 1014 12:18 ln -s /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/CF0090x6C_DE0360xPE0180-Pfafstetter.til OutTileFile 1015 12:18 ln -s /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/clsm/ 1016 12:18 cd .. 1018 12:23 /bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst 1020 12:24 mpirun -map-by core --mca btl ^vader -np 56 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ -t CF0090x6C_DE0360xPE0180-Pfafstetter.til -m catchcn -s 50 -j Y 1021 12:32 bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 /discover/nobackup/fzeng/Catchment/hindcast/create_restart/mk_restarts/catchcn_internal_rst is the restart file I need to run GEOSldas later. 2. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). ========== 20180328: 1. Copied some GEOS_S2S data to my nobackup for code modification next. cd /discover/nobackup/fzeng/forcing/data/rm dec12.geosgcm_3hrlnd.daily.201512.nc4.tar*/seasonal/Y2015/dec12/ens1/geosgcm_3hrlnd cp -p /archive/u/gmaofcst/GEOS_S2S/seasonal/Y2015/dec12/ens1/geosgcm_3hrlnd/dec12.geosgcm_3hrlnd.daily.201512.nc4.tar . tar -xvf dec12.geosgcm_3hrlnd.daily.201512.nc4.tar /bin/rm dec12.geosgcm_3hrlnd.daily.201512.nc4.tar* 2. Modified LDAS_Forcing.F90 for use of hindcast data. 3. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). 4. Noticed that I missed PS (surface pressure) in the list of the hindcast forcing variables. Have to interpolate from daily to 3-hourly later. 5. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_51. 6. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 33, 34 clm4_IPAR_DE720: 39 clm4_IPAR_DReqDF_DE720: 38 clm4_pIPAR_DE720: 26 clm4_pTair_DE720: 26 ========== 20180329: 1. Preparing for meeting with Randy. 2. Met with Randy and Eunjee. 3. Continued investigating the negative livestemn issue (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018). ========== 20180330: 1. March report.