==========
20171002:

1. September report.

2. For test_DE720 which used original LDAS_TileCorrdRoutines.F90 and the modified DE_00720x00360_PE_0720x0360.til:

Did an interactively run and it runs. Stopped it at 1980-01-07T22:00:00.

Submitted the full job.

3. Met with Randy and Eunjee to talk about Eunjee's AGCM experiment results.  

==========
20171003:

1. The test_DE720 run finished simulating 19800101-19871231. Looked at GPP, NPP and NEE of Jan and Jul 1980. The patterns look correct.

Created a shell script to delete the daily output and the temporary files. 

Sarith said the temporary files are supposed to be deleted by the system itself. He is investigating why I still have the temporary files. 

discover05:/discover/nobackup/fzeng/Catchment/M2n5P/m0003 > ls -l test_DE720/
total 160
lrwxrwxrwx 1 fzeng g0620    48 2017-09-29 15:42 build -> /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_0/
-rwxr-xr-x 1 fzeng g0620  1148 2017-10-03 13:31 delete_daily_output*
drwxr-xr-x 4 fzeng g0620   512 2017-09-29 15:42 input/
drwxr-xr-x 4 fzeng g0620   512 2017-09-29 16:10 mk_restarts/
drwxr-xr-x 3 fzeng g0620   512 2017-09-29 15:42 output/
drwxr-xr-x 2 fzeng g0620 32768 2017-10-03 13:06 run/
drwxr-xr-x 2 fzeng g0620 32768 2017-10-03 13:05 scratch/
discover05:/discover/nobackup/fzeng/Catchment/M2n5P/m0003 > ls -l test_DE720_mod/
total 0
lrwxrwxrwx 1 fzeng g0620  48 2017-09-29 16:34 build -> /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_0/
drwxr-xr-x 4 fzeng g0620 512 2017-09-29 16:34 input/
drwxr-xr-x 4 fzeng g0620 512 2017-09-29 18:48 mk_restarts/
drwxr-xr-x 3 fzeng g0620 512 2017-09-29 16:34 output/
drwxr-xr-x 2 fzeng g0620 512 2017-09-29 18:48 run/

test_DE720_mod was set up after test_DE720, and to set up test_DE720_mod I re-compiled the code using the modified LDAS_TileCorrdRoutines.F90. Therefore, my test_DE720 was actually using the modified LDAS_TileCorrdRoutines.F90.

About the temporary files not removed issue, Sarith told me to update my Applications/LDAS_App and try resuming the run from 19880101. 

Made sure the original LDAS_TileCorrdRoutines.F90 is used.

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/src/Applications/LDAS_App
cvs upd -r GEOSldas_m4-17_0

Compiled: "make install" in (1) GEOSldas_GridComp/Shared and (2) Applications/LDAS_App.
-rwxr-xr-x 1 fzeng g0620 48991065 2017-10-03 15:01 GEOSldas.x

cd /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/run

Checked the restart file:
/discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/run > cat cap_restart 
19880101 000000
/discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/run > ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 138 2017-10-03 13:05 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/output/global/rs/ens0000/Y1988/M01/test_DE720.catchcn_internal_rst.19880101_0000
lrwxrwxrwx 1 fzeng g0620 113 2017-09-29 16:10 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/output/global/rs/ens0000/test_DE720.vegdyn_internal_rst

Checked the executable:
/discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/run > ls -l ../build/Linux/bin/GEOSldas.x
-rwxr-xr-x 1 fzeng g0620 48991065 2017-10-03 15:01 ../build/Linux/bin/GEOSldas.x*

CAP.rc: 
  Change END_DATE to "19880301 000000".
  
lenkf.j:
  #SBATCH --time=01:00:00
  #SBATCH --qos=debug

qsub lenkf.j

Still see the temporary files in /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/output/global/cat/ens0000/Y1988/M01. Do I need to run ldas_setup again to set up a new experiment? Asked Sarith. He said yes.

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/src/Applications/LDAS_App
mv GEOSldas_test.exec GEOSldas_test.exec.old
mv GEOSldas_test.bat GEOSldas_test.bat.old
mv GEOSldas_test.def GEOSldas_test.def.old
./ldas_setup sample --exeinp > GEOSldas_test.exec
./ldas_setup sample --batinp > GEOSldas_test.bat
./ldas_setup sample --domaininp > GEOSldas_test.def
nedit GEOSldas_test.exec GEOSldas_test.bat GEOSldas_test.def &
./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/m0003/ ./GEOSldas_test.exec ./GEOSldas_test.bat
(ldas_setup can also be run from /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/Linux/bin) 

cd /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720_ext1/run/

Check restart file:
cat cap_restart 
  19880101 000000
ls -l ../input/restart/
  total 0
lrwxrwxrwx 1 fzeng g0620 148 2017-10-03 16:10 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720_ext1/output/global/rs/ens0000/Y1988/M01/test_DE720_ext1.catchcn_internal_rst.19880101_0000
lrwxrwxrwx 1 fzeng g0620 123 2017-10-03 16:10 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720_ext1/output/global/rs/ens0000/test_DE720_ext1.vegdyn_internal_rst

Check executable:
ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 48991065 2017-10-03 15:01 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the END_DATE.
Check HISTORY.rc:
  - Corrected EXP ID

qsub lenkf.j  

It's running. When it finished 1 month of simulation, check output/cat and found that the daily output and the temporary files are not there. Only monthly output is there, which is what I want. Great. Told Sarith about this.

2. Checked the clm4.5_DE720 output on GrADS.

Got the error message below when trying to run /home/fzeng/Catchment/CLM4.5/check_output.gs which I have used many times before:

I/O Error writing to print metafile.

Asked David. As David suspected, it's because I am over my home directory quota. 

Delete some files from my home directory to release some space. 

==========
20171004:

1. Sarith suggests to check out the released tag and start fresh. Check out the tag in the afternoon again because of the need to update ldas_setup and tutorial (see Rolf's email in the morning).

Check out:

cd /discover/nobackup/fzeng/offline_code
cvs co -r GEOSldas_m4-17_0 GEOSldas_m4
mv GEOSldas GEOSldas_m4-17_0

Compile:

cd GEOSldas_m4-17_0/src
setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/
source $ESMADIR/src/g5_modules
make install

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/
mkdir -p exec/test_DE720
/bin/cp -pr Linux exec/test_DE720/. 

#Update the tutorial: no need with the tag checked out in the afternoon

#cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/src/Applications/LDAS_App/doc
#cvs upd -r GEOSldas_m4-17_0 GEOSldas-tutorial_09.11.2017.pptx 

#Update ldas_setup: no need with the tag checked out in the afternoon

#cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/src/Applications/LDAS_App
#cvs upd -r GEOSldas_m4-17_0 ldas_setup 

Set up a run:

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_0/exec/test_DE720/Linux/bin
./ldas_setup sample --exeinp > GEOSldas_test.exec
./ldas_setup sample --batinp > GEOSldas_test.bat
nedit GEOSldas_test.exec GEOSldas_test.bat &
mv GEOSldas_test.exec /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run
mv GEOSldas_test.bat /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run
./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/m0003/ /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run/GEOSldas_test.exec /discover/nobackup/fzeng/Catchment/M2n5P/m0003/run/GEOSldas_test.bat

cd /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/run/

Check restart file:
cat cap_restart 
  19800101 000000
ls -l ../input/restart/
  total 0
lrwxrwxrwx 1 fzeng g0620 138 2017-10-04 15:00 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/output/global/rs/ens0000/Y1980/M01/test_DE720.catchcn_internal_rst.19800101_0000
lrwxrwxrwx 1 fzeng g0620 113 2017-10-04 15:00 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/m0003/test_DE720/output/global/rs/ens0000/test_DE720.vegdyn_internal_rst

Check executable:
ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 48823327 2017-10-04 13:40 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the END_DATE, JOB_SGMT and NUM_SGMT.
Modify HISTORY.rc for desired output:
(1)
  COLLECTIONS:
            'tavg1_tile_lnd_Nx'
            'tavg1_tile_mf-st'
            
Became:
  COLLECTIONS:
#           'tavg1_tile_lnd_Nx'
#           'tavg1_tile_mf-st'
            'tavg1_2D_lnd_Nx'
            'tavg1_2D_mf-st'            

(2)
  tavg1_2D_mf-st.regrid_name:'GRIDNAME'
  tavg1_2D_mf-st.resolution: 720 361,
Became:
  tavg1_2D_mf-st.regrid_name:'PE720x360-DE'
  tavg1_2D_mf-st.resolution: 720 360,
Same for tavg1_2D_lnd_Nx.

Did an interactive run:

In lenkf.j:
  change ntasks to 96 in L12
  add "exit" in L173, i.e. after "mpirun -map-by core --mca btl ^vader  -np $numprocs $GEOSBIN/GEOSldas.x" (so even if you delete lines nothing would happen)

./lenkf.j

It's running. Stopped it at 1980-01-01T15:00:00

Restore lenkf.j and qsub lenkf.j  
 
2. Improved converge.gs in ~/Catchment/CLM4.5.

==========
20171005:

1. Modified and ran calculate_clim_tile_monthly_Tair.f90 and calculate_clim_tile_monthly_precip.f90 in ~/Catchment/M2n5P_m0002 to compute the September climatology of the temperature and precipition forcing data used in the offline simulations.

~/Catchment/M2n5P_m0002 > calculate_clim_tile_monthly_Tair CTco2_3hrly
~/Catchment/M2n5P_m0002 > calculate_clim_tile_monthly_precip CTco2_3hrly

2. Modified and ran grid2tile_monthly.f90 (was for Apr-Aug, now for Apr-Sep) to pre-process the input data (Eunjee's AGCM output):

/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mkdir old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv T2M_ctrl_80yravg* old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv PRCP_ctrl_80yravg* old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv verify_T2M_ctrl_80yravg* old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv verify_PRCP_ctrl_80yravg* old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv *PRCP_dryL_80yravg* old
/discover/nobackup/fzeng/Catchment/M2n5P/m0002/input_data/climatology > mv *T2M_dryL_80yravg* old

~/Catchment/M2n5P_m0002 > grid2tile_monthly deltaCO2SC_80yravg_dryS_ctrl
~/Catchment/M2n5P_m0002 > grid2tile_monthly T2M_dryS_80yravg
~/Catchment/M2n5P_m0002 > grid2tile_monthly PRCP_dryS_80yravg
~/Catchment/M2n5P_m0002 > grid2tile_monthly T2M_ctrl_80yravg
~/Catchment/M2n5P_m0002 > grid2tile_monthly PRCP_ctrl_80yravg

3. Modified and ran ~/Catchment/M2n5P_m0002/tile2grid_DC0576xPC0361_monthly_clim.f90 to convert the AGCM T2M_ctrl_80yravg, PRCP_ctrl_80yravg, deltaCO2SC_80yravg_dryS_ctrl, T2M_dryS_80yravg and PRCP_dryS_80yravg back to grid-space for verification purpose.

~/Catchment/M2n5P_m0002 > tile2grid_DC0576xPC0361_monthly_clim T2M_ctrl_80yravg
~/Catchment/M2n5P_m0002 > tile2grid_DC0576xPC0361_monthly_clim PRCP_ctrl_80yravg

4. Verified the AGCM deltaCO2SC_80yravg_dryS_ctrl, T2M_dryS_80yravg and PRCP_dryS_80yravg using ~/Catchment/M2n5P_m0002/verify_AGCM_clim.gs 

5. Need to re-do the control run because Eunjee's AGCM control run is re-done.

discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mkdir old 
discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mv CTco2_3hrly_ctrl old
discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mv CTco2_3hrly_dco2sc old 
discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mv CTco2_3hrly_dryL old
discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mv CTco2_3hrly_dco2sc_dryL old

/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mkdir CTco2_3hrly_ctrl

Modified the code for the new control run, see compute_rc.F90, ens_driver_routines.F90.ctrl (changed "8" to "9" to include September) and process_cn.F90.3hrlyco2 in /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_ctrl/.

6. Followed (2) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the new control run for the offline drought impact experiments: CTco2_3hrly_ctrl_YYYY. YYYY is each year from 2001 through 2014.

Did an interactive run and stopped it at date_time_new 20010407_183730z

==========
20171006:

1. Processed Oct seasonal forecast.

2. Processed the output of CTco2_3hrly_ctrl_YYYY:

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_ctrl CTco2_3hrly_ctrl_2001

Verified the run using ~/Catchment/M2n5P_m0002/compare_met_AGCMvsOffline.gs.

3. Modified and compiled the code for CTco2_3hrly_dco2sc_S_YYYY runs. Double-checked with Eunjee -- dco2sc_S data is good.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_S ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_S/.

Followed (7) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the CTco2_3hrly_dco2sc_S_YYYY runs (2001-2014).

==========
20171010:

1. Updated Higgins rain data through Oct 9, 2017.

2. GMAO Seminar by Manuela. 

3. Processed the output of CTco2_3hrly_dco2sc_S_YYYY:

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_S CTco2_3hrly_dco2sc_S_2001 

Analyzed the output using ~/Catchment/M2n5P_m0002/compare_control_dco2sc.gs 

4. Modified and compiled the code for CTco2_3hrly_T2M_S_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.T2M_S ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_T2M_S/.

Followed (8) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

5. Modified and compiled the code for CTco2_3hrly_PRCP_S_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.PRCP_S ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_PRCP_S/.

Followed (9) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

6. Modified and compiled the code for CTco2_3hrly_dco2sc_dryS_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_S ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_dryS/.
Followed (10) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

==========
20171011:

1. Processed the output

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_T2M_S CTco2_3hrly_T2M_S_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_PRCP_S CTco2_3hrly_PRCP_S_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryS CTco2_3hrly_dco2sc_dryS_2001

2. Analyzed the output using compare_control_dryMET.gs and compare_control_dco2scNdryMET.gs in ~/Catchment/M2n5P_m0002

==========
20171012:

1. Continued analyzing the output using compare_all.gs in ~/Catchment/M2n5P_m0002

2. Helped Melanie to verify the CLM4.5 fire output.

3. GMAO Science Theme Meeting

==========
20171013:

1. Meeting with Randy, Eunjee, Lesley and Brad.

2. Updated my CV for SSAI's annual assessment.

3. SSAI's annual assessment.

==========
20171016: followed steps on 20171005, 20171006 and 20171010 above.

1. Ran grid2tile_monthly.f90 to pre-process the input data (Eunjee's AGCM output):

~/Catchment/M2n5P_m0002 > grid2tile_monthly deltaCO2SC_80yravg_dryL_ctrl
~/Catchment/M2n5P_m0002 > grid2tile_monthly T2M_dryL_80yravg
~/Catchment/M2n5P_m0002 > grid2tile_monthly PRCP_dryL_80yravg

2. Ran ~/Catchment/M2n5P_m0002/tile2grid_DC0576xPC0361_monthly_clim.f90 to convert deltaCO2SC_80yravg_dryL_ctrl, T2M_dryL_80yravg and PRCP_dryL_80yravg back to grid-space for verification purpose.

~/Catchment/M2n5P_m0002 > tile2grid_DC0576xPC0361_monthly_clim deltaCO2SC_80yravg_dryL_ctrl
~/Catchment/M2n5P_m0002 > tile2grid_DC0576xPC0361_monthly_clim T2M_dryL_80yravg
~/Catchment/M2n5P_m0002 > tile2grid_DC0576xPC0361_monthly_clim PRCP_dryL_80yravg

3. Verified the AGCM deltaCO2SC_80yravg_dryL_ctrl, T2M_dryL_80yravg and PRCP_dryL_80yravg using ~/Catchment/M2n5P_m0002/verify_AGCM_clim.gs

4. Modified and compiled the code for CTco2_3hrly_dco2sc_L_YYYY runs. 

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_L ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_L/.

Followed (3) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the CTco2_3hrly_dco2sc_L_YYYY runs (2001-2014). 

5. Modified and compiled the code for CTco2_3hrly_T2M_L_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.T2M_L ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_T2M_L/.

Followed (4) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

6. Modified and compiled the code for CTco2_3hrly_PRCP_L_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.PRCP_L ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_PRCP_L/.

Followed (5) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

7. Modified and compiled the code for CTco2_3hrly_dco2sc_dryL_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.dryL ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_L ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_dryL/.
Followed (6) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

==========
20171017:

1. Worked on Melanie's fire project. 

Looked through the plots Melanie made (see her email on 2017-10-16). Trying to understand why our burned area is so high and fire carbon emission is so low.

Modified ~/Catchment/matlab/calculate_land_frac.m (was in ~/Catchment/M2n5P_m0001) to compute land fraction for Melanie.

Tried using ~/Catchment/matlab/clm4-to-clm4.5/read_fields.m to extract the burned area data. Still working on it. 

2. Python class.

3. Processed the output.

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_L CTco2_3hrly_dco2sc_L_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_T2M_L CTco2_3hrly_T2M_L_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_PRCP_L CTco2_3hrly_PRCP_L_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryL CTco2_3hrly_dco2sc_dryL_2001

4. Analyzed the output using compare_control_dco2sc.gs, compare_control_dryMET.gs and compare_control_dco2scNdryMET.gs in ~/Catchment/M2n5P_m0002

==========
20171018:

1. Continued working on ~/Catchment/matlab/clm4-to-clm4.5/read_fields.m to extract the burned area data. Discussed with Melanie (via email).

2. Further analyzed the output using compare_all.gs in ~/Catchment/M2n5P_m0002

==========
20171019:

1. Continued analyzing the output using compare_all.gs in ~/Catchment/M2n5P_m0002

2. Showed results to Randy and Eunjee.

3. Provided comments on Melanie's poster.

==========
20171020:

1. Continued analyzing the output using compare_all.gs in ~/Catchment/M2n5P_m0002

2. Python homework #1. 

==========
20171023:

1. Updated Higgins rain data through Oct 22.

2. Copied Sarith's progam: see Sarith's email on 10/23/2017

/discover/nobackup/fzeng/Sarith > cp -p /discover/nobackup/smahanam/edit_carbon_rest.F90 .

Yehui is doing a catchcn experiment in which he wants to change initial carbon restarts only while hydrological restarts remain unchanged. I wrote the below fortran program for him. Since I am not going to check in that code to the repository, if you want you can keep it with you for future use.

Hi Yehui,

Put this:
/discover//nobackup/smahanam/edit_carbon_rest.F90 

inside
/discover/nobackup/ychang2/coupled/GEOS-S2S-2_1_CN/GEOSodas/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts.

Edit GNUmakefile : add the new file
       edit_carbon_rest

then 
setenv ESMADIR /discover/nobackup/ychang2/coupled/GEOS-S2S-2_1_CN/GEOSodas/
source $ESMADIR/g5_modules

gmake install

Now in your 
$ESMADIR/Linux/bin/edit_carbon_rest file1 file2 file3

is the command
where 
Suppose you read carbon restarts from file1, and all other restarts from file2 (or unchanged)
! file3 is the final merged restart file  

3. Checked the clm4.5_DE720 output.

4. Created ~/Catchment/matlab/clm4-to-clm4.5/LandGridArea.m referring to Eunjee's /discover/nobackup/elee15/analysis/offline/landgridarea.m. Used it to compute land area for 0.5x0.5 deg.

5. Created ~/Catchment/matlab/clm4-to-clm4.5/calculate_global.m and used it to compute global annual GPP for CLM4 and CLM4.5. CLM4.5 GPP is only half of CLM4 GPP. 

==========
20171024:

1. Used ~/Catchment/matlab/clm4-to-clm4.5/calculate_global.m to compute global annual NEE for CLM4 and CLM4.5. CLM4 is a weak sink, while CLM4.5 is a small source due to low GPP. Need to investigate the low CLM4.5 GPP issue.  

2. GMAO seminar.

3. Think about how to modify the offline code to do the experiment which uses carbon states (excluding water and energy states) from the daily restart files of the control. Emailed Sarith for help.

Met with Sarith and he gave me suggestions on code modification.

4. Modified and compiled the code for the control run to write restart files daily.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.ctrl ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.3hrlyco2 ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp clsm_ensdrv_out_routines.F90.w_daily_rst ../clsm_ensdrv_out_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled the code.
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/M2n5P_m0002CTco2_ctrl_w_daily_rst/.

5. Re-do the control run on region S only (as opposed to global as the control run done on 20171005).

/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mkdir CTco2_3hrly_ctrl_S

Followed (11) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the regional control run for the offline drought impact experiments: CTco2_3hrly_ctrl_S_YYYY. YYYY is each year from 2001 through 2014.

Did an interactive run and stopped it at date_time_new 20010407_183730z

==========
20171025:

1. Discover was down the entire day.

2. Commented on Eunjee's poster for Young Scientist Forum.

3. Worked on the presentation for GMAO Science Theme Meeting.

==========
20171026:

1. Checked the restart files written out in the CTco2_3hrly_ctrl_S_YYYY runs. They are daily. Good.

Restore the offline code:
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90

2. Followed Sarith's suggestions to modify the offline code to read the daily restart files from the CTco2_3hrly_ctrl_S_YYYY runs. See clsm_ensdrv_init_routines.F90.r_daily_rst and cnlsm_ensdrv_main.F90.r_daily_rst in /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2.

3. Compiled the code for CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp clsm_ensdrv_init_routines.F90.r_daily_rst ../clsm_ensdrv_init_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp cnlsm_ensdrv_main.F90.r_daily_rst_S /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/cnlsm_ensdrv_main.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.dryS ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_S ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Save a copy of the Linux directory:
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_dryS_ctrl_d_rst/.

Followed (12) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

==========
20171027:

1. Processed the output: modified grid_restore_DC0576xPC0361_monthly.f90 for region S and ran it.

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_ctrl_S CTco2_3hrly_ctrl_S_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryS_ctrl_d_rst CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_2001

2. Created ~/Catchment/M2n5P_m0002/compare_ts.gs to analyze the output.

3. Restored cnlsm_ensdrv_main.F90, clsm_ensdrv_init_routines, ens_driver_routines.F90, and process_cn.F90. 

4. Similar to 5 on 20171024, re-do the control run on region L only (as opposed to global as the control run done on 20171005).

/discover/nobackup/fzeng/Catchment/M2n5P/m0002 > mkdir CTco2_3hrly_ctrl_L

Followed (13) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the regional control run for the offline drought impact experiments: CTco2_3hrly_ctrl_L_YYYY. YYYY is each year from 2001 through 2014.

Did an interactive run and stopped it at date_time_new 20010407_183730z

==========
20171030:

1. Discover was down until 3pm.

2. When discover was down, clm4.5_DE720 was simulating date_time_new 19860714_120730z. Resumed it from 19860701.

3. Checked the CTco2_3hrly_ctrl_L_YYYY runs. They finished and wrote out daily restart files. Good.

4. Modified and compiled the code for CTco2_3hrly_dco2sc_dryL_ctrl_d_rst_YYYY runs.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp clsm_ensdrv_init_routines.F90.r_daily_rst ../clsm_ensdrv_init_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp cnlsm_ensdrv_main.F90.r_daily_rst_L /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/cnlsm_ensdrv_main.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp ens_driver_routines.F90.dryL ../ens_driver_routines.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp process_cn.F90.dCO2SC_L ../process_cn.F90
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/M2n5P_m0002CTco2 > cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Save a copy of the Linux directory:
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux/ exec/M2n5P_m0002CTco2_dco2sc_dryL_ctrl_d_rst/.

Followed (14) in /discover/nobackup/fzeng/Catchment/M2n5P/m0002/README to set up the runs (2001-2014).

5. Young Scientist Forum.

==========
20171031:

1. Processed the output: modified grid_restore_DC0576xPC0361_monthly.f90 for region L and ran it.

~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_ctrl_L CTco2_3hrly_ctrl_L_2001
~/Catchment/M2n5P_m0002 > grid_restore_DC0576xPC0361_monthly CTco2_3hrly_dco2sc_dryL_ctrl_d_rst CTco2_3hrly_dco2sc_dryL_ctrl_d_rst_2001

2. Made plots using ~/Catchment/M2n5P_m0002/compare_ts.gs. The LAI plots are not as expected. Investigating ...

Talked to Sarith. He couldn't find anything wrong with my modifications, but he think there is a bug somewhere. His suggestions:
(1) Make plots for a single point
(2) Compare the first restart file of every month between the CTco2_3hrly_ctrl_S_YYYY and CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY

For (1):
    lon_min = -93.5
    lon_max = -93
    lat_min = 33
    lat_max = 33.5
The lines for CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY and CTco2_3hrly_dco2sc_dryS_YYYY almost identical. 
Same for this point:
    lon_min = -91.75
    lon_max = -91.25
    lat_min = 31.25
    lat_max = 31.75

For (2):
Used ~/Catchment/matlab/read_catchcn_internal_rst.m
Found that for each year, the two restart files are identical for 0401_0000z, but differ for the other months 0x01_0000z. This may be expected. Because, for example, the 0501_0000z restart file in the CTco2_3hrly_dco2sc_dryS_ctrl_d_rst_YYYY run is written out at the end of the time step where date_time_old=20010430_235230z and date_time_new=20010501_000000z. And then in the next time step where date_time_old=20010501_000000 and date_time_new=20010501_000730z the daily restart file from the control run CTco2_3hrly_ctrl_S_YYYY is read in but not saved. So we may not be able to find out what's wrong using this method.