==========
20170801:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170802:

1. Processed GMAO GEOS5 seasonal forecast for August.

2. July report.

3. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

4. On sick leave for half day.

==========
20170803:

1. The CF0090x6C has finished the 3rd cycle.

(1) Checked and processed the output:

discover07:/discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C > ls -l output/CF0090x6C_GLOBAL/rs/ens0000/Y2016/M01/
total 2076544
-rw-r--r-- 1 fzeng g0620 2129526247 2017-08-03 04:51 CF0090x6C.ens0000.catchcn_ldas_rst.20160101_0000z
discover07:/discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/CF0090x6C > ls -l output/CF0090x6C_GLOBAL/cat/ens_avg/Y2015/M12
total 148544
-rw-r--r-- 1 fzeng g0620 152106240 2017-08-03 04:51 CF0090x6C.ens_avg.ldas_tile_monthly_out.201512.bin

Process the output (covert from tile-space to grid-space):
~/Catchment/CF0090x6C > tile2grid_CF0090x6C_monthly CF0090x6C

Check the output on GrADS. Looks good.

cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C
mv CF0090x6C CF0090x6C_03
mv rst_Yehui rst_Yehui_03
mv rst_Eunjee rst_Eunjee_03
cd CF0090x6C_03
rm -rf lenkf_orig

(2) Check carbon convergence. The carbon states look stable enough. 

(3) Reorder the restart files for Eunjee. Emailed Yehui, Sarith and Eunjee.

2. Randy wants the daily MERRA-2 data to be extended to the latest available. 

Followed my notes on 20151203: 
Modified the year and month in /home/fzeng/merra2/extract_daily.gs and ran it:
nohup grads -lbc "run extract_daily.gs" >& grads.out &

4. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170804:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

2. Met with Randy and Eunjee: Eunjee showed results from her AGCM experiments.

3. Moved Sarith's point driver to /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS and re-arrange the directories.

/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS > mv /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4 .
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4 > mv LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ .
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4 > rmdir LDAS
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4 > mv exec/ LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
/discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/clm4/run > cp -pr DE_00720x00360_PE_0720x0360 driver_inputs_0.5D_10tiles.nml ../../tests/clm4/run/.

4. The SMAP M09 transient CO2 run finished the 5th cycle. CO2 years: 1981-2016; meteorology years: 1981-2016.

Process the output (covert from tile-space to grid-space):
~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2

Check the July 2016 CO2, LAI, GPP, NPP and NEE output on GrADS. Looks good.

Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/
mv e0004s_transientCO2 e0004s_transientCO2_05
cd e0004s_transientCO2_05
rm -rf lenkf_orig

==========
20170807:

1. Saved another copy of the restart files from e0004s_transientCO2_05 and archive it.
/archive/u/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05_rst.tgz

2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

3. Met with Huisheng. Huisheng needs restart files for 1981 and 1994. 

(1) Reorder:

  (1.1) Create path:

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 
  mkdir -p rst_Huisheng/SMAP_EASEv2_M09_reordered
  
  (1.2) Create shell script "reorder_restarts" for reordering:

  cd rst_Huisheng/SMAP_EASEv2_M09_reordered
  nedit reorder_restarts &
  
  reorder_restarts reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
  #!/bin/csh

  set YYYY = 1994
  set YYYYMMDD=`echo ${YYYY}0101`          
          
  mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
    
  (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  cp /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/mk_LDASsaRestarts.F90 .

  nedit mk_LDASsaRestarts.F90 &
    In SUBROUTINE  reorder_LDASsa_restarts:
    (a)
       tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin'
    (b)
           rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//&
                '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_0000z'

  cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (1.4) Run "reorder_restarts" to reorder the offline restart file:
 
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  reorder_restarts

(2) regrid from SMAP_EASEv2_M09 to Huisheng's AGCM grid/resolution:

  (2.1) Create path:

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng
  mkdir Icarus-NL_Reynolds_CF90

  (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  cd Icarus-NL_Reynolds_CF90
  cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 .
  nedit mk_LDASsaRestarts.F90 &
       InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.19940101', &
       InCNTilFile = '/discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til',         &
  cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (2.3) Run mk_LDASsaRestarts to convert to desired tile system:
  
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  mpirun -np 1 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/smahanam/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ -m catchcn -s 50 -t CF0090x6C_DE0360xPE0180-Pfafstetter.til       
  sbatch mkLDASsa.j       

==========
20170808:

1. The mkLDASsa.j created has some issue, so the run crashed. Sarith helped me fix the bug. Processed the 19940101 and 19810101 restart files for Huisheng.

2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170809:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170810:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170811:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170814:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170815:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170816:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170817:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170818:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170821:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170822:

1. Met with Huisheng.

Using the catchcn_internal_rst I provided, Huisheng's AGCM run crashed at L3171 "if(associated( TH)) TH  = TH /CHX" in GEOS_CatchCNGridComp.F90. The error message is "floating invalid". CHX is computed in L3154 "CHX     = CHX + CH(:,N)*FR(:,N)". In the catchcn_internal_rst she copied from my directory, CH and FR are both 0 everywhere, so CHX is 0 everywhere and caused the crash. 

Checked the CH and FR values in the restart files from my SMAP_EASEv2_M09 and CF0090x6C offline runs, and the reordered and/or regridded ones. 

(1) SMAP_EASEv2_M09 offline simulations:

Offline restart files in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000:
CH: 0 everywhere.
FR: 0 everywhere.

Reordered restart files in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/SMAP_EASEv2_M09_reordered:
CH: 0.001 everywhere.
FR: 0.25 everywhere.

Regridded (to CF0090x6C) restart files in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90:
CH: 0 everywhere.
FR: 0 everywhere.

(2) CF0090x6C offline simulations:

Offline restart files in /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Eunjee_03:
CH: 0 everywhere.
FR: 0 everywhere.

Reordered restart files in /discover/nobackup/fzeng/Catchment/M2n5P/CF0090x6C/rst_Eunjee_03_AGCM:
CH: 0.001 everywhere.
FR: 0.25 everywhere.

Do the reordered ones look correct? There must be something wrong with the regridded ones. Is there anything wrong in the mk_LDASsaRestarts.F90 files I used?

Emailed Sarith for help. See our emails back and forth. 

Briefly, I need to modify mk_LDASsaRestarts.F90 to add values for CH, CM, CQ, FR and WW, because the current mk_LDASsaRestarts.F90 is for LDAS and doesn't provide valid values for these variables.

Modified /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90/mk_LDASsaRestarts.F90  

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90 > cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/.

cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
gmake install

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng/Icarus-NL_Reynolds_CF90
ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
mpirun -np 1 bin/mk_LDASsaRestarts -a g0620 -b /discover/nobackup/smahanam/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ -m catchcn -s 50 -t CF0090x6C_DE0360xPE0180-Pfafstetter.til       
sbatch mkLDASsa.j

Huisheng did a test run and it worked. This catchcn_internal_rst is for 19940101. 

Created catchcn_internal_rst for 19810101 as well. 

==========
20170823-20170825:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170828-20170830:

1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. 

See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily.

==========
20170831:

1. Met with Randy, Eunjee and Lesley about Eunjee's drought simulation results.

2. Found both my clm4_DE720 and clm4.5_DE720 stopped. The log files show that they stopped when they were trying to read the tile file /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til. It seems that the tile file is not accessible to others now. 

Looks like Sarith removed the entire directory /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/.

/discover/nobackup/smahanam/bcs/Heracles-4_3 > ls
Heracles-NL_Reynolds/  Heracles-NL_Tripolar-Ocean/

3. Processed the output of M2.n5P.3hco2+1ppm.glb and M2.n5P.3hco2+10ppm.glb.

~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2+1ppm.glb
~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2+10ppm.glb

Made maps showing the CO2 fertilization effect on GPP.

4. Discussed with Eunjee and Randy about the offline experimental design.