========== 20180201: 1. Continued on literature search for in situ FPAR measurements 2. Met with Kristi: asked her how they downscale daily forcing to hourly 3. Processed the clm4_IPAR_DE720_23 and clm4_IPAR_DReqDF_DE720_22 output using /home/fzeng/Catchment/M2n5P_m0001/princetonVSmerra2/compare_ts.m 4. Jan report ========== 20180202: 1. Jan report 2. Computed global annual GPP and NEE of clm4.5_DE720_41 3. Analyzed burned area and fire carbon loss in clm4.5_DE720_41. Compared them to clm4.5_DE720_40 (before nc4_grass bug fix) and Li et al., 2013. 4. Met with Eunjee 5. For CLM4.5 Catchment-CN, the bug fix in late Jan removed the large burned area in the Sahel region. However, burned area is still too large in SHSA, MIDE and TENA. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180205: 1. Computed and plotted global annual GPP and NEE time series of clm4_IPAR_DE720_24, clm4_IPAR_DReqDF_DE720_23, and clm4_pIPAR_DE720_20. 2. Continued debugging the fire model. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180206: 1. Computed global annual GPP and NEE of clm4.5_DE720_42. 2. Preparing for meeting with Randy and Eunjee tomorrow. 3. Continued debugging the fire model. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180207: 1. Computed and plotted global annual GPP and NEE time series of clm4_IPAR_DE720_25, clm4_IPAR_DReqDF_DE720_24, and clm4_pIPAR_DE720_21. 2. Preparing for meeting with Randy and Eunjee this afternoon. 3. Continued debugging the fire model. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. 4. Met with Randy and Eunjee to update my work. 5. Resumed clm4_DE720 following the procedures in /home/fzeng/Catchment/CLM4/submit_next_batch_DE720. set exp = clm4_DE720 set num = 20 cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -r ${exp}_template ${exp} cd ${exp}/input/restart ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num}/output /bin/rm rst_clm4 cd ../../run qsub lenkf.0.j ========== 20180208: 1. Investigated why global annual GPP and NEE computed from the same clm4_DE720_20 are different between the plots comparing CLM4 to CLM4.5 and the plots comparing CLM4 to the experiments using the PAR assumptions on MERRA2 forcing. It's because the former is just rough calculation, and the latter is more accurate in matching the GPP and NEE data to the land mask and in accounting for different number of days in different months of the year. Improved the former to match the latter. Re-computed global annual GPP and NEE of clm4.5_DE720 for the 33-42 cycles. 2. Made plots showing the 2002 and 2006 US droughts. 3. Tried to run Sarith's updated process_seasonal_forecast.csh but was having some issue. Sarith said he can take a look tomorrow. ========== 20180209: 1. Continued debugging the fire model. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. ========== 20180212: 1. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_43. GPP: 88.3045 NEE: -0.3605 2. Calculated 1980-2016 mean global annual GPP and NEE of these runs. GPP NEE clm4_DE720_21 149.2616 -0.1233 clm4_IPAR_DE720_26 155.2199 -0.2510 clm4_IPAR_DReqDF_DE720_25 153.2335 -0.3570 clm4_pIPAR_DE720_22 151.1257 -0.2440 clm4_IPAR_DE720_27 155.6860 -0.2298 clm4_IPAR_DReqDF_DE720_26 153.9988 -0.3243 clm4_pIPAR_DE720_23 151.6517 -0.2254 3. Helped Eunjee manually set up the 5th cyle of her transient CO2 run. cd /discover/nobackup/elee15/offline/sims/co2var_transCO2/M2.n5P.HRv2.transCO2/ cat year_co2.txt 1981 [Good!] ls -l CN_restart -rw-r--r-- 1 elee15 s1583 1530820456 2018-02-11 02:58 CN_restart ls -l RUN/rs/ens0000/Y1981/M01/ total 40512 -rw-r--r-- 1 elee15 s1583 41453776 2018-02-11 02:58 mM2.n5P.HRv2.transCO2.ens0000.catch_ldas_rst.19810101_0000z.bin The CN and land restarts match. Good. Need to modify lenkf.j. See email. 4. Updated higgins rain data through Feb 11. 5. Extracted rcuns (unstressed stomatal conductance) for Randy from /discover/nobackup/elee15/offline/sims/co2var_agu2016/M2.n5P.3hco2.glb/mM2.n5P.3hco2.glb_%y4%m2.gdat using /home/fzeng/Catchment/extract_rcuns.gs. The GrADS control file is /home/fzeng/Catchment/M2.n5P.3hco2.glb_rcuns.ctl. Copied the files to land01:/land/fzeng/rcuns for Randy. ========== 20180213: 1. Added the 1980-2016 mean global annual GPP and NEE values in 1&2 on 20180212 to the plots. 2. Continued debugging the fire model. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily. 3. Re-calculated and plotted annual burned area using the method Randy suggested. See ~/Catchment/CLM4.5/check_output.gs 4. Computed Pearson correlation coefficients for each month between Catchment-CN FPAR and GIMMS FPAR. ~/Catchment/M2n5P_m0002/corr_fpar_catchCN_GIMMS.m: converts 0.5x0.625 deg Catchment-CN FPAR to 0.5x0.5 deg grid, and computes Pearson correlation coefficients for each of the 12 months ~/Catchment/matlab/m_map/fz_plot_r_fpar.m: plot the correlation coefficients for the 12 months 5. CLM4.5 N fertilization code. ========== 20180214: 1. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_44. 2. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720_22 clm4_IPAR_DE720_27 clm4_IPAR_DReqDF_DE720_26 clm4_pIPAR_DE720_23 clm4_IPAR_DReqDF_DE720_27 clm4_pIPAR_DE720_24 3. Prepared for meeting with Randy and Eunjee. 4. Met with Randy and Eunjee. ========== 20180220: 1. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_45. 2. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 23, 24 clm4_IPAR_DE720: 28, 29 clm4_IPAR_DReqDF_DE720: 28 clm4_pIPAR_DE720: 25, 26 3. Met with Randy and Eunjee. Randy told us to look for some variable that is related to stomatal conductance and is only dependent on PFT. Eunjee and I read the Tech Note and model code. The rcuns I extracted for Randy is actually local noon stomatal conductance of CN zone 2 (the unstressed zone). Stomatal resistance is a function of psn (photosynthesis), so it's not solely dependent on PFT. The relationship between stomatal resistance and psn is not explicit so it's hard, or maybe impossible, to create a variable=f(stomatal resistance, psn) that is solely dependent on PFT. Corrected the description of rcuns in all the .ctl files in ~/Catchment/M2n5P_m0001. Added notes to compute_rc.F90 of all m3-15 and m3-16 tags I have to specify that rc is resistance instead of conductance. ========== 20180221: 1. Re-visited CLM4 Tech Note. m in the stomatal conductance calculation is PFT dependent. Would it be the one we are looking for? Talked to Randy. Randy suggests to extract tile area, PFT and canopy height. 2. Helped Eunjee set up the CO2 experiments. 3. Modified /home/fzeng/merra2/extract_daily.gs and ran it to extract some fields in MERRA2 for Randy. 4. Created and ran /home/fzeng/Catchment/extract_area_PFT_canopyHeight.f90 to extract area_, PFT and canopyHeight data which will be used to scale stomatal conductance later. ========== 20180222: 1. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 25 clm4_IPAR_DE720: 30 clm4_IPAR_DReqDF_DE720: 29 2. Prepared for meeting with Randy this afternoon. 3. Met with Randy and Eunjee. 4. Read the materials about stomatal conductance from Alex Konings 5. Reviewed my 20171215 and 20171218 notes on how to run original CLM4.5 (see /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily) ========== 20180223: 1. Copied the MERRA2 data to land01 for Randy. Then archive the data and remove it from NOBACKUP. /discover/nobackup/fzeng/merra2 > mkdir 9var_1980_2017 /discover/nobackup/fzeng/merra2 > mv *.gdat 9var_1980_2017 2. Modified /home/fzeng/Catchment/princeton_monthly_cli.f90 so that it works for both Princeton v2 and Princeton v2.2. Computed 1980-2014 January climatology of dswrf for Princeton v2.2 using the modified princeton_monthly_cli.f90. It's identical to the one computed using the original princeton_monthly_cli.f90. Computed the 1980-2012 climatology of air temperature for Princeton v2 using the modified princeton_monthly_cli.f90. 3. Modified /home/fzeng/Catchment/merra2_monthly_cli.f90 so that it works for any variable in MERRA2. Computed 1980-2014 DRPAR January climatology using the modified version. It's identical to the one computed using the original version. Computed the 1980-2012 climatology of air temperature (TLML) for MERRA2 using the modified version. 4. Modified /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/clsm_ensdrv_force_routines.F90 so that for the new experiment which uses MERRA2 forcing but Tair is adjusted to match Princeton (v2) climatology. See clsm_ensdrv_force_routines.F90.pTair in the same directory. Compiled. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4_pTair/. 5. Set up and automate the clm4_pTair_DE720 spinup run: (1) Create the template: Modified D0.5_cn.exe and D0.5.bat in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run. cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/clm4_pTair/Linux/bin source /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5_cn.exe /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/run/D0.5.bat --runmodel --monthsperjob 96 --landmodel catchCN cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/ mv clm4_pTair_DE720 clm4_pTair_DE720_template cd clm4_pTair_DE720_template/run nedit lenkf.0.j & changed "-restart_path ../input/restart/rst_clm4 \" to "-restart_path ../input/restart/output \" cp -p ../../clm4_DE720_24/run/edit_lenkf.csh . cp -p ../../clm4_DE720_24/run/LSM_Calib.rc . mkdir ../lenkf_orig cp -p lenkf.*.j ../lenkf_orig/. ./edit_lenkf.csh Change the last line to "/home/fzeng/Catchment/CLM4/submit_next_batch_DE720 clm4_pTair_DE720" in lenkf.4.j (2) Set up the clm4_pTair_DE720_20 cycle following steps in /home/fzeng/Catchment/CLM4/submit_next_batch_DE720: set exp = clm4_pTair_DE720 cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -r ${exp}_template ${exp} cd ${exp}/input/restart ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output /bin/rm rst_clm4 cd ../../run Check the executable: ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69951053 2018-02-23 15:46 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/build/Linux/bin/LDASsaCN_mpi.x* Check the restart file: /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/input/restart > ls -l lrwxrwxrwx 1 fzeng g0620 76 2018-02-23 16:01 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/run nedit lenkf.0.j & change "-restart_id clm4_pTair_DE720 \" to "-restart_id clm4_DE720 \" Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/run ./lenkf.0.i Crashed: Read Princeton Tair and MERRA2 Tair climatology: 01 Read Princeton Tair and MERRA2 Tair climatology: 02 forrtl: severe (174): SIGSEGV, segmentation fault occurred done reading ALBvf ############################### iMERRA2 out of bound, iMERRA2: ========== 20180226: 1. All my runs except clm4.5_DE720 stopped on Sunday evening around 8pm or afternoon around 2:30pm. (1) clm4_DE720: time out at date_time_new 19870522_040730z. The last restart file: -rw-r--r-- 1 fzeng g0620 291445863 2018-02-25 19:47 clm4_DE720.ens0000.catchcn_ldas_rst.19870501_0000z Resumed it from 19870501_0000z. (2) clm4_IPAR_DE720: time out at date_time_new 19950702_140730z. The last restart file: -rw-r--r-- 1 fzeng g0620 291445863 2018-02-25 19:51 clm4_IPAR_DE720.ens0000.catchcn_ldas_rst.19950701_0000z Resumed it from 19950701_0000z. (3) clm4_IPAR_DReqDF_DE720: time out at date_time_new 19950702_140730z The last restart file: -rw-r--r-- 1 fzeng g0620 291445863 2018-02-25 19:51 clm4_IPAR_DReqDF_DE720.ens0000.catchcn_ldas_rst.19950701_0000z Resumed it from 19950701_0000z. (4) clm4_pIPAR_DE720: time out at date_time_new 20031206_230730z The last restart file: -rw-r--r-- 1 fzeng g0620 291445863 2018-02-25 14:28 clm4_pIPAR_DE720.ens0000.catchcn_ldas_rst.20031201_0000z Resumed it from 20031201_0000z. 2. Continued to investigate the crash of the clm4_pTair_DE720 test run on in 5 on 20180223 above. It's because I didn't compute the longitude indices for both Princeton and MERRA2 correctly. Fixed the bug. ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69957669 2018-02-26 11:19 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/build/Linux/bin/LDASsaCN_mpi.x* /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/input/restart > ls -l lrwxrwxrwx 1 fzeng g0620 76 2018-02-23 16:01 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output/ cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/run qsub lenkf.0.j 3. Updated higgins data through Feb 25, 2018. However, the data for Feb 20 and Feb 21 are missing. I re-downloaded the netcdf file (ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/RT/precip.V1.0.2018.nc) another time but found the same issue. Hopefully this issue will be resolved by NOAA ESRL at a later time. 4. Fixed the bug (same as 3 above) in the pIPAR run and re-do it. This time use Princeton v2? YES, use this old version of Princeton data. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/clm4_pIPAR/. Followed 2(2) on 20180130 to set up the clm4_pIPAR_DE720_20 cycle: set exp = clm4_pIPAR_DE720 cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -r ${exp}_template ${exp} cd ${exp}/input/restart ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output /bin/rm rst_clm4 cd ../../run Check the executable: ls -l /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69957632 2018-02-26 16:57 /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/build/Linux/bin/LDASsaCN_mpi.x* Check the restart file: /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/input/restart > ls -l lrwxrwxrwx 1 fzeng g0620 76 2018-02-26 17:17 output -> /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720_19/output/ cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/run nedit lenkf.0.j & change "-restart_id clm4_pIPAR_DE720 \" to "-restart_id clm4_DE720 \" Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pIPAR_DE720/run ./lenkf.0.i It's running and read the right restart file. Stopped it. qsub lenkf.0.j 5. Double-checked the IPAR and IPAR_DReqDF run: IPAR run: IPAR/SWdown = 0.5, good! IPAR_DReqDF run: PARdir=PARdif=0.25, good! 6. Started working on ~/Catchment/compute_gmax_params.f90. ========== 20180227: 1. The clm4_pTair_DE720 run crashed again when it's mapping from grid-space to tile-space for the 2nd month. forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source libirc.so 00002AAAAB590961 Unknown Unknown Unknown libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2B2BA8 Unknown Unknown Unknown libc.so.6 00002AAAABA16910 Unknown Unknown Unknown LDASsaCN_mpi.x 00000000006AF367 clsm_ensdrv_force 496 clsm_ensdrv_force_routines.F90 LDASsaCN_mpi.x 00000000004A6FE1 MAIN__ 1868 cnlsm_ensdrv_main.F90 L496 clsm_ensdrv_force_routines.F90: Tair_cli_MERRA2_tile(n) = Tair_cli_MERRA2_grid( iMERRA2(n), jMERRA2(n)) Why? iPrinceton, jPrinceton, iMERRA2 and jMERRA2 are already declared as save. Added a print statement (see below) right before mapping from grid-space to tile-space to print out information and stop if any of the indices is out of bound. ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ! map from grid-space to tile-space do n=1,N_catd ! Make sure the indices are not out of bounds if(iPrinceton(n)<1 .or. iPrinceton(n)>Princeton_grid_N_lon) then print *, '##############################################################' print *, 'iPrinceton out of bound, iPrinceton(n): ',iPrinceton(n) print *, '##############################################################' stop end if if(jPrinceton(n)<1 .or. jPrinceton(n)>Princeton_grid_N_lat) then print *, '##############################################################' print *, 'jPrinceton out of bound, jPrinceton(n): ',jPrinceton(n) print *, '##############################################################' stop end if if(iMERRA2(n)<1 .or. iMERRA2(n)>MERRA2_grid_N_lon) then print *, '##############################################################' print *, 'iMERRA2 out of bound, iMERRA2(n): ',iMERRA2(n) print *, '##############################################################' stop end if if(jMERRA2(n)<1 .or. jMERRA2(n)>MERRA2_grid_N_lat) then print *, '##############################################################' print *, 'jMERRA2 out of bound, jMERRA2(n): ',jMERRA2(n) print *, '##############################################################' stop end if Tair_cli_Princeton_tile(n) = Tair_cli_Princeton_grid(iPrinceton(n),jPrinceton(n)) Tair_cli_MERRA2_tile(n) = Tair_cli_MERRA2_grid( iMERRA2(n), jMERRA2(n)) end do ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ It crashed at the same place at the same time. See below the error message: Reading Princeton Tair and MERRA2 Tair climatology: 02 forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source libirc.so 00002AAAAB590961 Unknown Unknown Unknown libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2B2BA8 Unknown Unknown Unknown libc.so.6 00002AAAABA16910 Unknown Unknown Unknown LDASsaCN_mpi.x 00000000006AF88C clsm_ensdrv_force 525 clsm_ensdrv_force_routines.F90 LDASsaCN_mpi.x 00000000004A6FE1 MAIN__ 1868 cnlsm_ensdrv_main.F90 It's not because the index is out of bound because it didn't print out any information or stop. So the indices are good and the clm4_pIPAR_DE720 run is fine. Tried this: Changed real, allocatable, dimension(:,:) :: Tair_cli_Princeton_grid ! Princeton Tair monthly climatology, grid-space real, allocatable, dimension(:,:) :: Tair_cli_MERRA2_grid ! MERRA2 Tair monthly climatology, grid-space real, allocatable, dimension(:) :: Tair_cli_Princeton_tile ! Princeton Tair monthly climatology, tile-space real, allocatable, dimension(:) :: Tair_cli_MERRA2_tile ! MERRA2 Tair monthly climatology, tile-space to real, allocatable, dimension(:,:), save :: Tair_cli_Princeton_grid ! Princeton Tair monthly climatology, grid-space real, allocatable, dimension(:,:), save :: Tair_cli_MERRA2_grid ! MERRA2 Tair monthly climatology, grid-space real, allocatable, dimension(:), save :: Tair_cli_Princeton_tile ! Princeton Tair monthly climatology, tile-space real, allocatable, dimension(:), save :: Tair_cli_MERRA2_tile ! MERRA2 Tair monthly climatology, tile-space It passed the beginning of February and still running!! Stopped it. Submitted the full job. However, it still crashed at the beginning of February. Why?!!! lenkf.0.i and lenkf.0.j only differ in the number of nodes. Did an interactive run. It again passed the beginning of Feb. Let it continue. Stopped it when it finished simulating March. Modified lenkf.0.j to start from 19800401. It crashed at the beginning of 19800401, saying floating divided by 0 in catchmentCN.F90. Did an interactive run starting from 19800401. It again crashed at the beginning of 19800401, saying floating divided by 0 in catchmentCN.F90. Tried start it from 19800301 interactively. It runs. Waited until it finished simulating March and April. Stopped it at date_time_new 19800504_073000z. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_pTair_DE720/run > ls -l ../output/global/rs/ens0000/Y1980/M05 -rw-r--r-- 1 fzeng g0620 291445863 2018-02-28 17:19 clm4_pTair_DE720.ens0000.catchcn_ldas_rst.19800501_0000z Modified lenkf.0.j to start from 19800501. 2. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 26 clm4_IPAR_DE720: 31 clm4_IPAR_DReqDF_DE720: 30 3. Calculated 1980-2016 mean global annual GPP and NEE of clm4.5_DE720_46. 4. Continued working on ~/Catchment/compute_gmax_params.f90. Ran it to compute area-weighted gmax parameters for each of the 19 PFTs in CLM4. Data in /discover/nobackup/fzeng/bcs/Icarus-NL/Icarus-NL_Reynolds/gmax_params_DC0288xPC0181. 5. Met with Eunjee about the updated CO2 variation experiment results. 6. The clm4_DE720 crashed saying divided by zero. It could be because the 19870501_0000z restart file was not written out correctly. Tried starting from 19870401_0000z. It worked. 7. Met with Randy and Eunjee. ========== 20180228: 1. Regridded my 20010101 e0004s_transient CO2 restart file to /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0288xPC0181_DE0360xPE0180/DC0288xPC0181_DE0360xPE0180-Pfafstetter.til 2001 is a normal year, so the restart file should be good to use. (1.1) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: cd /discover/nobackup/fzeng/offline_restarts/DC0288xPC0181_DE0360xPE0180 cp /discover/nobackup/fzeng/offline_restarts/DC0576xPC0361_DE0360xPE0180/mk_LDASsaRestarts.F90 . nedit mk_LDASsaRestarts.F90 & InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2001/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20010101_0000z', & cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ source $ESMADIR/src/g5_modules gmake install (1.2) Run mk_LDASsaRestarts to convert to desired tile system: cd /discover/nobackup/fzeng/offline_restarts/DC0288xPC0181_DE0360xPE0180 ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin mpirun -np 1 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/DC0288xPC0181_DE0360xPE0180/ -m catchcn -s 50 -t DC0288xPC0181_DE0360xPE0180-Pfafstetter.til sbatch mkLDASsa.j use the final catchcn_internal_rst (for CLM4.5, use OutData2/catchcn_internal_clm45 as of 22 August, 2017) 2. Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_IPAR_DE720: 32 clm4_IPAR_DReqDF_DE720: 31 3. Compiled the code for the 1 deg (i.e. DC0288xPC0181_DE0360xPE0180) control run. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp ls *restore clsm_ensdrv_glob_param.F90.restore clsm_ensdrv_init_routines.F90.restore clsm_ensdrv_out_routines.F90.restore ens_driver_routines.F90.restore process_cn.F90.restore xdiff clsm_ensdrv_glob_param.F90.restore clsm_ensdrv_glob_param.F90 & xdiff clsm_ensdrv_init_routines.F90.restore clsm_ensdrv_init_routines.F90 & xdiff clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90 & xdiff ens_driver_routines.F90.restore ens_driver_routines.F90 & xdiff process_cn.F90.restore process_cn.F90 & The 5 files are all the same as the *restore ones, i.e. they are all the default ones. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp ls *restore compute_rc.F90.restore xdiff compute_rc.F90.restore compute_rc.F90 & compute_rc.F90 is currently for varying CO2. mv compute_rc.F90 compute_rc.F90.varCO2 cp compute_rc.F90.restore compute_rc.F90 xdiff compute_rc.F90.restore compute_rc.F90 & Now compute_rc.F90 is the default one. Compiled. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > diff Linux/bin/LDASsaCN_mpi.x exec/CLM4_ctrl/Linux/bin/LDASsaCN_mpi.x Files Linux/bin/LDASsaCN_mpi.x and exec/CLM4_ctrl/Linux/bin/LDASsaCN_mpi.x differ [Don't know where it differs in the code] /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/gmax_ctrl/. 4. Set up and automate the 1 deg (i.e. DC0288xPC0181_DE0360xPE0180) control spinup run. (1) Create the template: Created DC0288xPC0181.bat, DC0288xPC0181_CN.exe, and driver_inputs_DC0288xPC0181_GLOBAL.nml in /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/gmax_ctrl/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/M2n5P/m0004 /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run/DC0288xPC0181_CN.exe /discover/nobackup/fzeng/Catchment/M2n5P/m0004/run/DC0288xPC0181.bat --runmodel --monthsperjob 36 --landmodel catchCN cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004 mv gmax_ctrl gmax_ctrl_template cd gmax_ctrl_template/run nedit lenkf.0.j & changed "-restart_path ../input/restart/DC0288xPC0181_DE0360xPE0180 \" to "-restart_path ../input/restart/output \" cp -p /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4/clm4_DE720/run/edit_lenkf.csh . mkdir ../lenkf_orig cp -p lenkf.*.j ../lenkf_orig/. ./edit_lenkf.csh Change the last line to "/home/fzeng/Catchment/CLM4/submit_next_batch_gmax gmax_ctrl" in lenkf.12.j (2) Set up the gmax_ctrl_01 cycle following steps in /home/fzeng/Catchment/CLM4/submit_next_batch_gmax: set exp = gmax_ctrl cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004 /bin/cp -r ${exp}_template ${exp} cd ${exp}/input/restart ln -s $NOBACKUP/offline_restarts/DC0288xPC0181_DE0360xPE0180/catchcn_internal_rst /bin/rm DC0288xPC0181_DE0360xPE0180 cd ../../run Check the executable: ls -l /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 69771582 2018-02-28 11:37 /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/build/Linux/bin/LDASsaCN_mpi.x* Check the restart file: cd ../input/restart/ ls -l lrwxrwxrwx 1 fzeng g0620 90 2018-02-28 15:52 catchcn_internal_rst -> /discover/nobackup/fzeng/offline_restarts/DC0288xPC0181_DE0360xPE0180/catchcn_internal_rst cd ../../run/ nedit lenkf.0.j & change "-restart_path ../input/restart/output \" to "-restart_path ../input/restart/ \" Did an interactive run to make sure it works: interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/M2n5P/m0004/gmax_ctrl/run ./lenkf.0.i It's running. Stopped it at date_time_new 19800101_112230z. qsub lenkf.0.j Cancelled the job because we need to write out something new. See notes on 20180301. 5. Created ~/Catchment/submit_next_batch_gmax