==========
20180601:

1. The e0004s_transientCO2_2012 run finished simulating 201202. It took 07:15:57 (too slow)!!

2. Asked Sarith again yesterday about the difference in the BCs file mosaic_veg_typs_fracs between /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/ (call it A here) used in e0004s_transientCO2_05 and that in /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE/ (call it B here) used in e0004s_transientCO2_2011 and e0004s_transientCO2_2012. 

He said other than the 8th column which is in B but not in A, the 7th columns do differ occasionally at the 3rd decimal point. 

He also said since I run on M09, I can try to use Heracles-NL BCs from Larry. I asked whether the BCs files in Larry's Heracles-NL/SMAP_EASEv2_M09 are exactly the same as those I used for my e0004s_transientCO2_05 (they were once in /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/). Sarith said "I believe so". 

3. Re-do the e0004s_transientCO2_2012 run using Heracles-NL (HNL) BCs and the executable used for e0004s_transientCO2_05.

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09
mv e0004s_transientCO2_2011 e0004s_transientCO2_2011_INL (INL means using Icarus-NL BCs)
mv e0004s_transientCO2_2012 e0004s_transientCO2_2012_INL

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec
mv e0004s_transientCO2 e0004s_transientCO2_INL_20180502
mv e0004s_transientCO2_20170427 e0004s_transientCO2_HNL (HNL means using Heracles-NL BCs. This is the one used for e0004s_transientCO2_01through05)

Following notes on 20180530:

(1) Prepare files for experiment setup: 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run
cp M09_CN_e0004s_transientCO2_2011.exe M09_CN_e0004s_transientCO2_2012.exe 
nedit M09_CN_e0004s_transientCO2_2012.exe &
    exp_id                  = e0004s_transientCO2_2012
    start_time              = 2012-01-01-00-00-00
    end_time                = 2012-02-01-00-00-00
    restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
    restart_id              = e0004s_transientCO2
    driver_inputs_file      = driver_inputs_M09_GLOBAL_HNL.nml   

mv driver_inputs_M09_GLOBAL.nml driver_inputs_M09_GLOBAL_INL.nml
cp driver_inputs_M09_GLOBAL_INL.nml driver_inputs_M09_GLOBAL_HNL.nml
nedit driver_inputs_M09_GLOBAL_HNL.nml &
    veg_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
    alb_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
    soil_path            = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
    top_path             = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'

    mwRTM_param_path     = ''

    tile_coord_path      = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'
    catchment_def_path   = '/discover/nobackup/ltakacs/bcs/Heracles-NL/'

    tile_coord_file      = 'SMAP_EASEv2_M09_3856x1624.til'
    catchment_def_file   = 'catchment.def'

(2) Run ldsetup:

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_HNL/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_2012.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

(3) Check the executable, restart file and create year_co2.txt:

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012
ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
    -rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
ls -l input/restart/ (to make sure the restart file is the right one)
    lrwxrwxrwx 1 fzeng g0620 80 2018-06-01 10:08 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/

(4) Create year_co2.txt:
cd run
echo 2012 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)
    2012 [Correct!]
    
(5) Check lenkf.0.j and namelist:

lenkf.0.j: 

    #SBATCH --time=12:00:00
    #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120101_0000.txt
    #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120101_0000.txt
     -start_year 2012 -start_month  1 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
     -end_year   2012 -end_month    2 -end_day    1 -end_hour    0 -end_min    0 -end_sec    0 \
     -restart_path ../input/restart/output \             [Need modification for lenkf.1.j and lenkf.2.j]
     -restart_id e0004s_transientCO2 \                   [Need modification for lenkf.1.j and lenkf.2.j]
     -driver_inputs_file driver_inputs_M09_GLOBAL_HNL.nml    [Make sure it's monthly output now. Need modification for lenkf.1.j and lenkf.2.j]

Do an interactive run to make sure it runs:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run
/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*  
./lenkf.0.i

It's running. Stopped it at date_time_new 20120101_040730z.

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
qsub lenkf.0.j

4. The e0004s_transientCO2_2012 finished 1 month of simulation. It took only 01:04:42 (same as the  e0004s_transientCO2_05)!! Great!

Start the 201202 simulation.

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run
cat year_co2.txt 
2012  [Correct!]
cp lenkf.0.j lenkf.1.j
nedit lenkf.1.j &
    #SBATCH --time=03:00:00
    #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120201_0000.txt
    #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120201_0000.txt
    -start_year 2012 -start_month  2 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
    -end_year   2012 -end_month    3 -end_day    1 -end_hour    0 -end_min    0 -end_sec    0 \
    -restart_path ../output \
    -restart_id e0004s_transientCO2_2012 \
NOTE: remember to change driver_inputs_M09_GLOBAL_HNL.nml for the next two job segments!!

Do an interactive run to make sure it runs:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run 
/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* 
./lenkf.1.i

It's running. Stop it at date_time_new 20120201_040730z.

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* 
qsub lenkf.1.j

5. Processed the 201201 output. 

~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2012

Compared the output to that in e0004s_transientCO2_05. They are identical! Great!!

6. May report. 

7. Randy wants two more fields from MERRA-2:

Snowfall: PRECSNO (snowfall, kg m-2 s-1) in tavg1_2d_flx_Nx
Snowmelt: SMLAND (Snowmelt flux land, kg m-2 s-1) in tavg1_2d_lnd_Nx

Modified ~/merra2/extract_daily.gs to add these two fields, and ran it to extract all the fields from 19800101 through 20180430.

~/merra2 > grads -lbc "run extract_daily.gs"

8. The e0004s_transientCO2_2012 finished simulating Feb 2012. It took only 00:53:14. 

~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2012

Checked the 201202 output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2_2012.gs.
It's identical to that from e0004s_transientCO2_05.

Make it run from 20120301_00z through 20120331_00z.

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run
cat year_co2.txt 
    2012 [Correct!]
cp driver_inputs_M09_GLOBAL_HNL.nml driver_inputs_M09_GLOBAL_HNL_daily.nml
nedit driver_inputs_M09_GLOBAL_HNL_daily.nml &
    out_select_ensavg%tile%daily   = .true.
    out_select_ensavg%tile%pentad  = .false.	
    out_select_ensavg%tile%monthly = .false.  
cp lenkf.1.j lenkf.2.j
nedit lenkf.2.j &
    #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120301_0000.txt
    #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120301_0000.txt
     -start_year 2012 -start_month  3 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
     -end_year   2012 -end_month    3 -end_day   31 -end_hour    0 -end_min    0 -end_sec    0 \
     -driver_inputs_file driver_inputs_M09_GLOBAL_HNL_daily.nml

Do an interactive run to make sure it runs:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run  
/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* 
./lenkf.2.i

It works. Stopped it at date_time_new 20120301_033730z. 

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
qsub lenkf.2.j

9. Repeat the e0004s_transientCO2_2011 run using Larry's Heracles-NL (HNL) BCs and the executable used for e0004s_transientCO2_2011_INL, to see how long it will take to finish one month of simulation.

Following notes earlier today:

(1) Prepare files for experiment setup: 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run 
nedit M09_CN_e0004s_transientCO2_2011.exe &
    exp_id                  = e0004s_transientCO2_2011
    start_time              = 2011-01-01-00-00-00
    end_time                = 2011-02-01-00-00-00
    restart_path            = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
    restart_id              = e0004s_transientCO2
    driver_inputs_file      = driver_inputs_M09_GLOBAL_HNL.nml   

(2) Run ldsetup:

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_INL_20180502/Linux/bin
source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules
./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_2011.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN
(NOTE: don't use the "--submit" option because we need to modify the job scripts, see below)

(3) Check the executable, restart file and create year_co2.txt:

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011
ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one)
    -rwxr-xr-x 1 fzeng g0620 90093587 2018-05-02 14:04 build/Linux/bin/LDASsaCN_mpi.x [Correct!]
ls -l input/restart/ (to make sure the restart file is the right one)
    lrwxrwxrwx 1 fzeng g0620 80 2018-06-01 17:14 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/
    
(4) Create year_co2.txt:
cd run
echo 2011 > year_co2.txt
cat year_co2.txt (to make sure the year of CO2 is correct)
    2011 [Correct!]
    
(5) Check lenkf.0.j and namelist:

lenkf.0.j: 

    #SBATCH --time=12:00:00
    #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_log.20110101_0000.txt
    #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_out.20110101_0000.txt
     -start_year 2011 -start_month  1 -start_day  1 -start_hour  0 -start_min  0 -start_sec  0 \
     -end_year   2011 -end_month    2 -end_day    1 -end_hour    0 -end_min    0 -end_sec    0 \
     -restart_path ../input/restart/output \             [Need modification for lenkf.1.j and lenkf.2.j]
     -restart_id e0004s_transientCO2 \                   [Need modification for lenkf.1.j and lenkf.2.j]
     -driver_inputs_file driver_inputs_M09_GLOBAL_HNL.nml    [Make sure it's monthly output now. Need modification for lenkf.1.j and lenkf.2.j]

Do an interactive run to make sure it runs:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run
/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*  
./lenkf.0.i

It's running. Stopped it at date_time_new 20110101_034500z. [It's slower than the e0004s_transientCO2_2012 run that uses the same executable as e0004s_transientCO2_05 and Larry's Heracles-NL BCs.]

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
qsub lenkf.0.j

==========
20180604:

1. The e0004s_transientCO2_2012 finished simulating 20120301_00z through 20120331_00z. It took only 01:02:51. 

Make it run from 20120331_00z through 20120331_21z.

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run
cat year_co2.txt 
    2012 [Correct!]
cp driver_inputs_M09_GLOBAL_HNL.nml driver_inputs_M09_GLOBAL_HNL_3hourly.nml
nedit driver_inputs_M09_GLOBAL_HNL_3hourly.nml &
    out_dtstep%xhourly   = 10800
    out_dtstep%inst      = 10800

    ! ensemble average output choices

    out_select_ensavg%tile%inst    = .false. 
    out_select_ensavg%tile%xhourly = .true.
    out_select_ensavg%tile%daily   = .false.
    out_select_ensavg%tile%pentad  = .false.	
    out_select_ensavg%tile%monthly = .false. 
cp lenkf.2.j lenkf.3.j
nedit lenkf.3.j &
    #SBATCH --time=01:00:00
    #SBATCH --qos=debug
    #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120331_0000.txt
    #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120331_0000.txt
     -start_year 2012 -start_month  3 -start_day 31 -start_hour  0 -start_min  0 -start_sec  0 \
     -end_year   2012 -end_month    3 -end_day   31 -end_hour   21 -end_min    0 -end_sec    0 \
     -driver_inputs_file driver_inputs_M09_GLOBAL_HNL_3hourly.nml 

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
qsub lenkf.3.j

It finished. 

2. The e0004s_transientCO2_2011 run repeated in 9 on 20180601 using Larry's Heracles-NL (HNL) BCs and the executable used for e0004s_transientCO2_2011_INL finished simulating 201101. It took 07:46:51! So the problem is in the executable. 

Try to recompile again and make sure I don't use BOPT=g.

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
xdiff clsm_ensdrv_init_routines.F90 clsm_ensdrv_init_routines.F90.restore & [Identical. Good!]
nedit clsm_ensdrv_init_routines.F90 & [No modifications made.]
    L1659: "case (7.8)" 
cd e0004s_transientCO2
cp clsm_ensdrv_out_routines.F90 ../.
cp process_cn.F90 ../.
cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/.

Compiled:
setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/
source $ESMADIR/src/g5_modules
"gmake clean" and "gmake install" in GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp, GEOSlana_GridComp, and Applications/LDAS_App. [NOTE: I didn't do "gmake clean" when I compiled on 20180502.]

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec
mkdir e0004s_transientCO2_INL
cd ..
/bin/cp -pr Linux exec/e0004s_transientCO2_INL

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011
mv build build2
ln -s /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_INL build
    lrwxrwxrwx 1 fzeng g0620    89 2018-06-04 11:46 build -> /gpfsm/dnb31/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_INL/
cd output/SMAP_EASEv2_M09_GLOBAL/rc_out 
/bin/rm *
/bin/rm Y2011/M01/*

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run/
cat year_co2.txt 
    2011 [Correct!]
rm *ts.txt
ls -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/build/Linux/bin/LDASsaCN_mpi.x
    -rwxr-xr-x 1 fzeng g0620 69819400 2018-06-04 11:43 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/build/Linux/bin/LDASsaCN_mpi.x* [Correct!]
    
Do an interactive run to make sure it runs:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run
/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*  
./lenkf.0.i

It's running. Stopped it at date_time_new 20110101_040730z. [It seems that it's as fast as the e0004s_transientCO2_2012 run that uses the same executable as e0004s_transientCO2_05 and Larry's Heracles-NL BCs.]

/bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*
qsub lenkf.0.j    

The new executable is smaller. I must have used BOPT=g for the old one.

NEW:

discover06:/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > ls -l e0004s_transientCO2_INL/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 69819400 2018-06-04 11:43 e0004s_transientCO2_INL/Linux/bin/LDASsaCN_mpi.x*

OLD:

discover06:/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec > ls -l e0004s_transientCO2_INL_20180502/Linux/bin/LDASsaCN_mpi.x
-rwxr-xr-x 1 fzeng g0620 90093587 2018-05-02 14:04 e0004s_transientCO2_INL_20180502/Linux/bin/LDASsaCN_mpi.x*

Sarith said "you don't get a zero-diff if you had BOPT=g".

3. Restored the 3 files:    

cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp
cp clsm_ensdrv_out_routines.F90.restore clsm_ensdrv_out_routines.F90
cp process_cn.F90.restore process_cn.F90
cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/
cp compute_rc.F90.restore compute_rc.F90

4. The new e0004s_transientCO2_2011 run finished 1 month of simulation. It took only 01:02:21. 

Processed the 201101 output. 

~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2011

Checked the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2_2011.gs
There is some small difference. Why? It also used HNL. Why is it different from e0004s_transientCO2_05?

This file is newer than the executable (20170427) for e0004s_transientCO2_05:
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared:
-rwxr-xr-x 1 fzeng g0620  87731 2017-07-14 13:28 lsm_routines.F90*

What did I do about it? Here is my notes about it on 20170714.

~~~~~~~~~~~~~~~~~~~~~~~
Modified offline code for the CF0090x6C run: 19800101-20151231.
See /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Component/GEOSlana_GridComp/CF0090x6C:
clsm_ensdrv_out_routines.F90
tile_coord.F90
Also fixed a bug in lsm_routines.F90 (L422):
Was: bflow(n)=amin1(cond(n),bflow(n))
Now: bflow(n)=amin1(1000.*cond(n),bflow(n))
Sarith said:
I remember this change, it is a fix to a decade old bug in the code, that has pretty much zero-impact, but you sure make the change.
I think there is nothing else that I remember.

Compiled: (1) GEOSland_GridComp/Shared, (2) GEOSland_GridComp/GEOScatchCN_GridComp, (3) GEOSlana_GridComp, and (4) Applications/LDAS_App.
~~~~~~~~~~~~~~~~~~~~~~~

Would this bug fix be the reason for the difference? Sarith said "that has pretty much zero-impact" on 20170714. Couldn't find any other files that are modified after 20170427. Will ask Randy. 

5. Reorder and regrid /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2012/M03/e0004s_transientCO2_2012.ens0000.catchcn_ldas_rst.20120331_2100z to CF90 for Eunjee, following notes on 20180515. 

(1) Reorder:

  (1.1) Create path: 

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 
  mkdir -p rst_Eunjee/SMAP_EASEv2_M09_reordered

  (1.2) Create shell script "reorder_restarts" for reordering:

  cd rst_Eunjee/SMAP_EASEv2_M09_reordered
  nedit reorder_restarts &
  
  reorder_restarts reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
  #!/bin/csh

  set YYYY = 2012
  set YYYYMMDD=`echo ${YYYY}0331`          
          
  mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2_2012 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:
  
  cp /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 .
  
  nedit mk_LDASsaRestarts.F90 &
    In SUBROUTINE  reorder_LDASsa_restarts:
    (a) L298:
       tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin'
    (b) L540:
       rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//&
            '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_2100z' [NOTE: remember to set it to "2100z" to use the right restart file!!]

    cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (1.4) Run "reorder_restarts" to reorder the offline restart file:
 
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/SMAP_EASEv2_M09_reordered
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  reorder_restarts
  
(2) Regrid from SMAP_EASEv2_M09 to Eunjee's AGCM grid/resolution: 

  (2.1) Create path:

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee
  mkdir -p Heracles-NL_Reynolds_CF90
  
  (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  cd Heracles-NL_Reynolds_CF90
  mv catchcn_internal_rst catchcn_internal_rst.20110331_21z
  mv OutData1 OutData1_20110331_21z
  mv OutData2 OutData2_20110331_21z
  rm mkLDAS.e mkLDAS.o
  
  cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 .
  nedit mk_LDASsaRestarts.F90 &
       L67: InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20120331', &
       L68: InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til',         &
       [Do "xdiff mk_LDASsaRestarts.F90 ../SMAP_EASEv2_M09_reordered/. &" for more modifications made.]
  cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (2.3) Run mk_LDASsaRestarts to convert to desired tile system:
  
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/Heracles-NL_Reynolds_CF90
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  nedit mkLDASsa.j & [Create a job file]

  mkLDASsa.j reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
#!/bin/csh -f
 
#SBATCH --account=g0620
#SBATCH --time=1:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=56
#SBATCH --job-name=mkLDAS
#SBATCH --constraint=hasw
#SBATCH --output=mkLDAS.o
#SBATCH --error=mkLDAS.e
 
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
limit stacksize unlimited

setenv SPONSORID g0620
setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/
setenv TILFILE CF0090x6C_DE0360xPE0180-Pfafstetter.til

mkdir -p OutData1/
mkdir -p OutData2/
ln -s $BCSDIR/$TILFILE OutData1/OutTileFile
ln -s $BCSDIR/$TILFILE OutData2/OutTileFile
ln -s $BCSDIR/clsm OutData2/clsm

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

/bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50  
  
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
           
  chmod 755 mkLDASsa.j
  sbatch mkLDASsa.j
  
Finished.

mv catchcn_internal_rst catchcn_internal_rst.20120331_21z
mv OutData1 OutData1_20120331_21z
mv OutData2 OutData2_20120331_21z

6. Processed June forecast.

==========
20180605:

1. Sent the June forecast images to Randy.

2. Copied the MERRA-2 data to land01 and emailed Randy.

3. Met with Randy and Eunjee.

4. Documented why we decided to use the land cover map that we have been used for CLM4 Catchment-CN in CLM4.5 Catchment-CN. Also documented the procedures to run the original CLM4.5 with CROP ON and OFF. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes. 

5. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180606:

1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180607:

1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

2. Extracted Greg's runs that created initial conditions for the forecast (published in 2015) from Archive. This is to compare Catchment-CN June LAI anomaly for each year from 2003 to 2015 to that of MODIS LAI. 

cd /discover/nobackup/fzeng/gkwalker/Catchment/princeton

cp /archive/u/gkwalker/Catchment/princeton/p0036g_42-CN-may31_21z.tar .
tar -xf p0036g_42-CN-may31_21z.tar
There are subdirectories from 1948 through 2010. 

Each subdirectory only has the restart files, no model output. 
/discover/nobackup/fzeng/gkwalker/Catchment/princeton/p0036g_42-CN-may31_21z/2003 > ls
CN_restart_0531.gz  lnd_restart_0531

mkdir p0036g_42-daily_LAI
cd p0036g_42-daily_LAI
cp /archive/u/gkwalker/Catchment/princeton/p0036g_42-daily_LAI/lai_2003.tgz .
tar -xf lai_2003.tgz

There are 256 (NOT 365) files inside and the filenames are:
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat000001
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat000254
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat000507
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat000760
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat001013
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat001266
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat001519
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat001772
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat002025
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat002278
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat002531
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat002784
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat003037
...
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat062745
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat062998
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat063251
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat063504
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat063757
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat064010
-rw-r--r-- 1 fzeng g0620 8868040 2014-09-09 17:25 lai_2003.gdat064263
-rw-r--r-- 1 fzeng g0620 8938120 2014-09-09 17:25 lai_2003.gdat064516

Don't understand which day each of these files is for. 

Talked to Randy.

Randy found that he has monthly FPAR data from Greg. He will use the FPAR data and I don't need to do anything about this.

rm -rf Catchment

==========
20180608:

1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180611:

1. Pre-processing CT CO2 data for the new experiments: 

M2.n5P.maco2.glb
M2.n5P.magco2.glb
M2.n5P.magtco2.glb
M2.n5P.cco2.glb [use different number]

2. Modified LDASsa_m3-15_2-CN for the M2.n5P.maco2.glb case. 

Make sure the other routines are the default ones. 

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> ls *restore
    ens_driver_routines.F90.restore  process_cat.F90.restore
> xdiff process_cat.F90 process_cat.F90.restore &      [Identical. Good.]
> xdiff ens_driver_routines.F90 ens_driver_routines.F90.restore &        [Identical. Good.]

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/
> ls *restore
    CNAllocationMod.F90.restore  CNGRespMod.F90.restore  compute_rc.F90.restore  unified_rc3f_matrix_calc.F90.restore
> xdiff CNAllocationMod.F90 CNAllocationMod.F90.restore &       [Identical. Good.] 
> xdiff CNGRespMod.F90 CNGRespMod.F90.restore &      [Identical. Good.]
> xdiff compute_rc.F90 compute_rc.F90.restore &      [Identical. Good.]
> xdiff unified_rc3f_matrix_calc.F90 unified_rc3f_matrix_calc.F90.restore &      [Identical. Good.]

Compiled the driver with the routines (process_cat.F90 and compute_rc.F90) specific for the case.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp 
> ls -l M2n5P_m0001CTco2/process_cat.F90.annualCO2
    -rw-r--r-- 1 fzeng g0620 92774 2018-06-11 12:44 M2n5P_m0001CTco2/process_cat.F90.annualCO2
> /bin/cp M2n5P_m0001CTco2/process_cat.F90.annualCO2 process_cat.F90
> /bin/cp M2n5P_m0001CTco2/compute_rc.F90 ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/.

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/
> source $ESMADIR/src/g5_modules 
> gmake clean
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake clean
> gmake install

> cd ../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec
> cp -p ../Linux/bin/LDASsa_mpi.x LDASsa_mpi_M2n5P_m0001CTco2_annual.x

3. Set up the M2.n5P.maco2.glb run:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001
> mkdir -p M2.n5P.maco2.glb/RUN/rs/ens0000/Y2001/M01
> cd M2.n5P.maco2.glb
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/CN_restart .
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/mM2.n5P.REFco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.maco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.maco2.glb > ls -l CN_restart
-rw-r--r-- 1 fzeng g0620 1530820456 2018-02-18 16:27 CN_restart
/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.maco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/
-rw-r--r-- 1 fzeng g0620 41453776 2018-02-18 16:27 mM2.n5P.maco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

> cp ../m0001CTco2_one/lenkf.j .
> nedit lenkf.j &

> ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annual.x
-rwxr-xr-x 1 fzeng g0620 49558868 2018-06-11 13:17 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annual.x

> cp lenkf.j lenkf.i
> nedit lenkf.i &  
  changed "mpirun -np ${SLURM_NTASKS} \" to "mpirun -np 96 \"
> chmod 755 lenkf.i
  
Do an interactive run to make sure it runs:

> ssh -XY discover-sp3
> interactive.py -A sp3 -n 96 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.maco2.glb  
> ./lenkf.i 

It's running. Finished almost 3 months of simulation within 1 hour.

> cd ~/Catchment/M2n5P_m0001
> ifort tile2grid_FV_144x91_monthly.f90
> a.out M2.n5P.maco2.glb

Created M2n5P_m0001CTco2_annual.ctl and checked CO2 on GrADS. Jan 2001 and Feb 2001 CO2 looks identical to me. Great!

Emailed Eunjee to let her know and she can set up the run. 

4. Restored the two files:

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp
> cp compute_rc.F90.restore compute_rc.F90
> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> cp process_cat.F90.restore process_cat.F90

==========
20180612:

1. Modified LDASsa_m3-15_2-CN for the M2.n5P.magco2.glb case. 

Make sure the other routines are the default ones.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> xdiff process_cat.F90 process_cat.F90.restore &      [Identical. Good.]

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/
> xdiff compute_rc.F90 compute_rc.F90.restore &      [Identical. Good.]

Compiled the driver with the routines (only process_cat.F90) specific for the case.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> ls -l M2n5P_m0001CTco2/process_cat.F90.annualGlobalCO2
    -rw-r--r-- 1 fzeng g0620 92808 2018-06-12 11:15 M2n5P_m0001CTco2/process_cat.F90.annualGlobalCO2
> /bin/cp M2n5P_m0001CTco2/process_cat.F90.annualGlobalCO2 process_cat.F90

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp
> setenv ESMADIR /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/
> source $ESMADIR/src/g5_modules
> xdiff compute_rc.F90 compute_rc.F90.restore &    [Identical. Good.] 
> gmake clean
> gmake install

> cd ../../../GEOSlana_GridComp/
> gmake clean
> gmake install

> cd ../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec
> cp -p ../Linux/bin/LDASsa_mpi.x LDASsa_mpi_M2n5P_m0001CTco2_annualGlobal.x

2. Set up the M2.n5P.magco2.glb run:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001
> mkdir -p M2.n5P.magco2.glb/RUN/rs/ens0000/Y2001/M01
> cd M2.n5P.magco2.glb
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/CN_restart .
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/mM2.n5P.REFco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.magco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magco2.glb > ls -l CN_restart
-rw-r--r-- 1 fzeng g0620 1530820456 2018-02-18 16:27 CN_restart
/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/
-rw-r--r-- 1 fzeng g0620 41453776 2018-02-18 16:27 mM2.n5P.magco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

> cp ../M2.n5P.maco2.glb/lenkf.j .
> nedit lenkf.j &

> ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annualGlobal.x
-rwxr-xr-x 1 fzeng g0620 49554596 2018-06-12 11:34 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annualGlobal.x

> cp lenkf.j lenkf.i
> nedit lenkf.i &  
  changed "mpirun -np ${SLURM_NTASKS} \" to "mpirun -np 96 \"
> chmod 755 lenkf.i
  
Do an interactive run to make sure it runs:

> ssh -XY discover-sp3
> interactive.py -A sp3 -n 96 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magco2.glb  
> ./lenkf.i 

It's running. The 2001 CO2 concentration printed out is correct. Finished almost 3 months of simulation within 1 hour.

> cd ~/Catchment/M2n5P_m0001
> ifort tile2grid_FV_144x91_monthly.f90
> a.out M2.n5P.magco2.glb

Created M2n5P_m0001CTco2_annualGlobal.ctl and checked CO2 on GrADS. Jan 2001 and Feb 2001 CO2 looks correct to me - close to constant 378.727 ppm (with some difference in the 3rd decimal point, originally 378.7220ppm) everywhere and same between Jan and Feb. Great!  
This difference is due to the unit conversion from ppm to fraction, 378.7220ppm --> 0.000378722, causing data truncation. This is confirmed by Sarith.

Emailed Eunjee to let her know and she can set up the run. 

3. Restored the file:

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> cp process_cat.F90.restore process_cat.F90

4. Modified LDASsa_m3-15_2-CN for the M2.n5P.magtco2.glb case. 

Make sure the other routines are the default ones.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> xdiff process_cat.F90 process_cat.F90.restore &      [Identical. Good.]

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/
> xdiff compute_rc.F90 compute_rc.F90.restore &      [Identical. Good.]

Compiled the driver with the routines (only process_cat.F90) specific for the case.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> ls -l M2n5P_m0001CTco2/process_cat.F90.annualGlobalTrendCO2
    -rw-r--r-- 1 fzeng g0620 92794 2018-06-12 13:11 M2n5P_m0001CTco2/process_cat.F90.annualGlobalTrendCO2
> /bin/cp M2n5P_m0001CTco2/process_cat.F90.annualGlobalTrendCO2 process_cat.F90

> setenv ESMADIR /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/
> source $ESMADIR/src/g5_modules
> gmake clean
> gmake install

> cd ../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec
> cp -p ../Linux/bin/LDASsa_mpi.x LDASsa_mpi_M2n5P_m0001CTco2_annualGlobalTrend.x

5. Set up the M2.n5P.magtco2.glb run:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001
> mkdir -p M2.n5P.magtco2.glb/RUN/rs/ens0000/Y2001/M01
> cd M2.n5P.magtco2.glb
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/CN_restart .
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/mM2.n5P.REFco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.magtco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magtco2.glb > ls -l CN_restart
-rw-r--r-- 1 fzeng g0620 1530820456 2018-02-18 16:27 CN_restart
/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magtco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/
-rw-r--r-- 1 fzeng g0620 41453776 2018-02-18 16:27 mM2.n5P.magtco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

> cp ../M2.n5P.magco2.glb/lenkf.j .
> nedit lenkf.j &

> ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annualGlobalTrend.x
-rwxr-xr-x 1 fzeng g0620 49554596 2018-06-12 13:23 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_annualGlobalTrend.x

> cp lenkf.j lenkf.i
> nedit lenkf.i &  
  changed "mpirun -np ${SLURM_NTASKS} \" to "mpirun -np 96 \"
> chmod 755 lenkf.i
  
Do an interactive run to make sure it runs:

> ssh -XY discover-sp3
> interactive.py -A sp3 -n 96 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.magtco2.glb  
> ./lenkf.i 

It's running. The 2001 CO2 concentration printed out is correct. Finished almost 3 months of simulation within 1 hour.

> cd ~/Catchment/M2n5P_m0001
> ifort tile2grid_FV_144x91_monthly.f90
> a.out M2.n5P.magtco2.glb

Created M2n5P_m0001CTco2_annualGlobalTrend.ctl and checked CO2 on GrADS. Jan 2001 and Feb 2001 CO2 looks correct to me - close to constant 378.855 ppm (with some difference in the 2nd decimal point, originally 378.8721ppm) everywhere and same between Jan and Feb. Great!
This difference is due to the unit conversion from ppm to fraction, 378.8721ppm --> 0.0003788721, causing data truncation. This is confirmed by Sarith. 

Emailed Eunjee to let her know and she can set up the run. 

6. Restored the file:

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> cp process_cat.F90.restore process_cat.F90

7. Modified LDASsa_m3-15_2-CN for the new M2.n5P.cco2.glb case. 

Make sure the other routines are the default ones.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> xdiff process_cat.F90 process_cat.F90.restore &      [Identical. Good.]

> cd ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/
> xdiff compute_rc.F90 compute_rc.F90.restore &      [Identical. Good.]

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/M2n5P_m0001CTco2
> mv process_cat.F90.oneCO2 process_cat.F90.oneCO2.old
> cp process_cat.F90.oneCO2.old process_cat.F90.oneCO2
> nedit process_cat.F90.oneCO2 &

Compiled the driver with the routines (only process_cat.F90) specific for the case.

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> ls -l M2n5P_m0001CTco2/process_cat.F90.oneCO2
    -rw-r--r-- 1 fzeng g0620 91744 2018-06-12 16:36 M2n5P_m0001CTco2/process_cat.F90.oneCO2
> /bin/cp M2n5P_m0001CTco2/process_cat.F90.oneCO2 process_cat.F90

> setenv ESMADIR /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/
> source $ESMADIR/src/g5_modules
> gmake clean
> gmake install

> cd ../../Applications/LDAS_App/
> gmake clean
> gmake install

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec
> cp -p ../Linux/bin/LDASsa_mpi.x LDASsa_mpi_M2n5P_m0001CTco2_oneCO2.x

8. Set up the new M2.n5P.cco2.glb run:

> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001
> mkdir -p M2.n5P.cco2.glb/RUN/rs/ens0000/Y2001/M01
> cd M2.n5P.cco2.glb
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/CN_restart .
> cp -p /discover/nobackup/elee15/offline/sims/co2var_exp_c14/mM2.n5P.REFco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.cco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.cco2.glb > ls -l CN_restart
-rw-r--r-- 1 fzeng g0620 1530820456 2018-02-18 16:27 CN_restart
/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.cco2.glb > ls -l RUN/rs/ens0000/Y2001/M01/
-rw-r--r-- 1 fzeng g0620 41453776 2018-02-18 16:27 mM2.n5P.cco2.glb.ens0000.catch_ldas_rst.20010101_0000z.bin

> cp ../M2.n5P.magco2.glb/lenkf.j .
> nedit lenkf.j &

> ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_oneCO2.x
-rwxr-xr-x 1 fzeng g0620 49553198 2018-06-12 16:45 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_oneCO2.x

> cp lenkf.j lenkf.i
> nedit lenkf.i &  
  changed "mpirun -np ${SLURM_NTASKS} \" to "mpirun -np 96 \"
> chmod 755 lenkf.i
  
Do an interactive run to make sure it runs:

> ssh -XY discover-sp3
> interactive.py -A sp3 -n 96 -a g0620 -X --debug
> cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.cco2.glb  
> ./lenkf.i 

It's running. The constant CO2 concentration 392.3408ppm printed out is correct. Finished almost 3 months of simulation within 1 hour.

> cd ~/Catchment/M2n5P_m0001
> ifort tile2grid_FV_144x91_monthly.f90
> a.out M2.n5P.cco2.glb

Created M2n5P_m0001CTco2_oneCO2.ctl and checked CO2 on GrADS. Jan 2001 and Feb 2001 CO2 looks correct to me - close to constant 392.325 ppm (with some difference in the 2nd decimal point, originally 392.3408ppm) everywhere and same between Jan and Feb. Great!
This difference is due to the unit conversion from ppm to fraction, 392.3408ppm --> 0.0003923408, causing data truncation. This is confirmed by Sarith. 

Emailed Eunjee to let her know and she can set up the run. 

6. Restored the file:

> cd /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp
> cp process_cat.F90.restore process_cat.F90

==========
20180613:

1. Randy asked me which is more trustable, MODIS albedo or MODIS FPAR? I was thinking about asking Yanmin but have to check if she is in China or in the US now. Emailed her.

Randy talked to Jim Tucker. Jim said MODIS albedo might be more trustworthy than MODIS FPAR is because the latter depends on some estimation of PFT. See Randy's email "notes from meeting" on 6/14/2018.

2. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018.

==========
20180614:

1. Met with Randy, Sarith and Eunjee. See Randy's email "notes from meeting".

2. Met with Sarith. 

My GEOSldas_m4-17_6 is up-to-date. No need to update.

Sarith modified my GEOS_CatchCNGridComp.F90.

Compiled:

setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6
source ESMADIR/src/g5_modules
cd GEOSsurface_GridComp 
gmake clean
gmake install FOPT='-DLAND UPD'    # the FOPT='-DLAND UPD' is to make sure to include some updates (see tutorial)
cd ../../Applications/LDAS_App/
gmake install

But more modifications are needed. See notes in the next day.

==========
20180615:

1. Met with Pete.

2. Met with Sarith to talk about the strategy to implement transient CO2 in GEOSldas, and to save daily elai and esai. 

==========
20180618:

1. Read Joanna's paper.

2. Modeling small group meeting.

3. Seminar on Quantifying the Socioeconomic Benefits of Earth Observations in Decision-Making.

==========
20180619:

1. Continued reading Joanna's paper. Provided my comments.

2. Irrigation meeting.

3. Used /home/fzeng/matlab/plot_catchcn_internal_rst_vars.m to compute the % of tiles having negative livestemn. For both catchcn_internal_rst and catchcn_internal_clm45 in /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/restarts/0.5D: 

% of negative values: 7.63e-03
i.e. < 0.01%

% of tiles having negative values: 9.16e-02
i.e. < 0.1%

Negative livestemn (gN/m2) is mostly found in zone 3. See below:

Negative PFT level variable in tile id, nz, ityp, value: 1077 3 10 -1.44e-06
Negative PFT level variable in tile id, nz, ityp, value: 1077 3 11 -1.19e-05
Negative PFT level variable in tile id, nz, ityp, value: 1141 3 11 -2.89e-06
Negative PFT level variable in tile id, nz, ityp, value: 2118 2 11 -1.37e-06
Negative PFT level variable in tile id, nz, ityp, value: 3407 3 11 -4.81e-10
Negative PFT level variable in tile id, nz, ityp, value: 4067 3 11 -9.68e-08
Negative PFT level variable in tile id, nz, ityp, value: 4076 3 11 -2.88e-08
Negative PFT level variable in tile id, nz, ityp, value: 5851 3 11 -2.12e-09
Negative PFT level variable in tile id, nz, ityp, value: 5851 3 6 -8.89e-10
Negative PFT level variable in tile id, nz, ityp, value: 6400 2 6 -3.95e-07
Negative PFT level variable in tile id, nz, ityp, value: 6733 3 11 -1.57e-06
Negative PFT level variable in tile id, nz, ityp, value: 6918 3 11 -2.23e-06
Negative PFT level variable in tile id, nz, ityp, value: 7037 3 11 -3.70e-10
Negative PFT level variable in tile id, nz, ityp, value: 7134 3 6 -1.84e-06
Negative PFT level variable in tile id, nz, ityp, value: 7232 3 11 -1.29e-07
Negative PFT level variable in tile id, nz, ityp, value: 7917 3 4 -2.09e-06
Negative PFT level variable in tile id, nz, ityp, value: 11814 3 11 -1.85e-05
Negative PFT level variable in tile id, nz, ityp, value: 12833 3 11 -4.34e-06
Negative PFT level variable in tile id, nz, ityp, value: 13257 3 11 -2.75e-06
Negative PFT level variable in tile id, nz, ityp, value: 13284 3 4 -5.18e-07
Negative PFT level variable in tile id, nz, ityp, value: 13895 3 11 -1.50e-06
Negative PFT level variable in tile id, nz, ityp, value: 14435 3 11 -1.34e-05
Negative PFT level variable in tile id, nz, ityp, value: 17293 2 11 -3.47e-05
Negative PFT level variable in tile id, nz, ityp, value: 18739 3 6 -1.40e-06
Negative PFT level variable in tile id, nz, ityp, value: 19880 2 6 -7.36e-07
Negative PFT level variable in tile id, nz, ityp, value: 20359 3 6 -1.80e-06
Negative PFT level variable in tile id, nz, ityp, value: 21887 3 5 -7.08e-07
Negative PFT level variable in tile id, nz, ityp, value: 22940 2 6 -2.87e-05
Negative PFT level variable in tile id, nz, ityp, value: 22948 3 5 -6.16e-06
Negative PFT level variable in tile id, nz, ityp, value: 22989 3 1 -1.13e-07
Negative PFT level variable in tile id, nz, ityp, value: 23788 2 11 -2.06e-05
Negative PFT level variable in tile id, nz, ityp, value: 23788 3 11 -1.01e-06
Negative PFT level variable in tile id, nz, ityp, value: 24357 3 4 -1.18e-05
Negative PFT level variable in tile id, nz, ityp, value: 25015 3 1 -3.06e-09
Negative PFT level variable in tile id, nz, ityp, value: 25254 3 1 -1.20e-07
Negative PFT level variable in tile id, nz, ityp, value: 25255 3 5 -1.38e-06
Negative PFT level variable in tile id, nz, ityp, value: 25337 3 6 -1.19e-05
Negative PFT level variable in tile id, nz, ityp, value: 26902 3 1 -1.64e-07
Negative PFT level variable in tile id, nz, ityp, value: 27223 3 1 -4.83e-07
Negative PFT level variable in tile id, nz, ityp, value: 29845 3 1 -5.10e-08
Negative PFT level variable in tile id, nz, ityp, value: 29858 3 10 -3.88e-06
Negative PFT level variable in tile id, nz, ityp, value: 30166 3 11 -2.31e-06
Negative PFT level variable in tile id, nz, ityp, value: 30969 3 1 -2.05e-07
Negative PFT level variable in tile id, nz, ityp, value: 31245 2 11 -1.46e-05
Negative PFT level variable in tile id, nz, ityp, value: 31801 3 11 -6.06e-07
Negative PFT level variable in tile id, nz, ityp, value: 32103 2 1 -3.74e-08
Negative PFT level variable in tile id, nz, ityp, value: 32192 3 11 -5.88e-08
Negative PFT level variable in tile id, nz, ityp, value: 33489 3 2 -1.51e-08
Negative PFT level variable in tile id, nz, ityp, value: 33825 2 1 -2.74e-06
Negative PFT level variable in tile id, nz, ityp, value: 34398 1 11 -6.69e-06
Negative PFT level variable in tile id, nz, ityp, value: 34542 3 2 -2.86e-08
Negative PFT level variable in tile id, nz, ityp, value: 34900 3 2 -6.90e-09
Negative PFT level variable in tile id, nz, ityp, value: 35262 3 2 -6.60e-09
Negative PFT level variable in tile id, nz, ityp, value: 35610 3 2 -4.49e-09
Negative PFT level variable in tile id, nz, ityp, value: 40341 3 8 -2.72e-05
Negative PFT level variable in tile id, nz, ityp, value: 45113 3 8 -3.52e-06
Negative PFT level variable in tile id, nz, ityp, value: 45542 3 2 -1.58e-06
Negative PFT level variable in tile id, nz, ityp, value: 65516 3 12 -7.75e-09
Negative PFT level variable in tile id, nz, ityp, value: 66701 3 12 -1.53e-08
Negative PFT level variable in tile id, nz, ityp, value: 67284 3 8 -4.99e-08
Negative PFT level variable in tile id, nz, ityp, value: 67287 3 8 -4.40e-09
Negative PFT level variable in tile id, nz, ityp, value: 67288 3 8 -1.39e-08

Made plots to show the locations of the grid cells with negative livestemn using /home/fzeng/Catchment/matlab/m_map/fz_plot_rst_var.m

==========
20180620:

1. Creating a subroutine to read hindcast data in Catchment-CN.

2. IDS meeting. 

==========
20180621:

1. Creating a subroutine to read hindcast data in Catchment-CN.

==========
20180622:

1. Creating subroutine get_GEOS_hindcast in LDAS_Forcing.F90 to read hindcast data in Catchment-CN. Was able to compile (GEOSmetforce_GridComp and then Applications/LDAS_App).

Next week:
Check if they match MERRA2
Ask Lena:
    0130z is the average over 0000z through 0030z?
    Does precipitation include snow or just rain?

Do a test run next week. 

2. On 20180628 Sarith said I need to do these below to compile the code. 

# ----------------------------
#  Activate V24_C05 updates
# ----------------------------
# (i) Build the model; (ii) go to GEOSsurface_GridComp and gmake clean (iii) gmake install FOPT='-DLAND_UPD' (iv) goto Applications/GEOSgcm_App and gmake install

Read the tutorial. The tutorial was last modified on 3 Nov 2017, after my test run on 20171004. The way to compile the code and to set up a run is slightly different 

So recompiled the code as below. This was done on 20180628. 

Build:

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src
setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6
source g5_modules
make -j 8 install |& tee tmpfile.txt

Verify build:

./Config/gmh.pl tmpfile.txt
    All 44 packages compiled successfully.

Activate V24_C05 updates:

cd GEOSldas_GridComp/GEOSsurface_GridComp/
gmake clean 
gmake install FOPT='-DLAND_UPD'
cd ../../Applications/LDAS_App/ 
gmake install
    
cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
mkdir -p exec/hindcast
/bin/cp -pr Linux exec/hindcast/. 
  

==========
20180625: 

1. Emailed Lena for the questions.

2. Followed my notes on 20171004 to do a test run. Also see notes on 20180628. 

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
mkdir -p exec/hindcast
/bin/cp -pr Linux exec/hindcast/.  

Set up a run:

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
source $ESMADIR/src/g5_modules

./ldas_setup sample --exeinp > M2n5P.exec
./ldas_setup sample --batinp > M2n5P.bat

# Need to create a restart file before doing the steps below. See 1 on 20180626.

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M03 
ln -s /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/Heracles-NL_Reynolds_CF90/catchcn_internal_rst.20150301_0000z e0004s_transientCO2_05.catchcn_internal_rst.20150301_0000
cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09//e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL//rc_out/
ln -s e0004s_transientCO2.ldas_tilecoord.bin e0004s_transientCO2_05.ldas_tilecoord.bin

nedit M2n5P.exec M2n5P.bat &
mkdir -p /discover/nobackup/fzeng/Catchment/hindcast/run
mv M2n5P.exec /discover/nobackup/fzeng/Catchment/hindcast/run/
mv M2n5P.bat /discover/nobackup/fzeng/Catchment/hindcast/run/
./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/M2n5P.exec /discover/nobackup/fzeng/Catchment/hindcast/run/M2n5P.bat

This doesn't work because there is no ndvi_clim_*.data in /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/.

Sarith said I have to use /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180 instead. He said it will be zero-diff against Heracles-NL
/discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/.

# Need to regrid to /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180. See 2 on 20180626. 

cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M03
ln -s /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/Icarus-NL_Reynolds_CF90/catchcn_internal_rst.20150301_0000z e0004s_transientCO2_05.catchcn_internal_rst.20150301_0000

cd /discover/nobackup/fzeng/Catchment/hindcast/run/
gedit M2n5P.exec &
    BCS_PATH                : /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/exec/hindcast/Linux/bin
setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
source $ESMADIR/src/g5_modules
./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/hindcast/ /discover/nobackup/fzeng/Catchment/hindcast/run/M2n5P.exec /discover/nobackup/fzeng/Catchment/hindcast/run/M2n5P.bat

cd /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/run/

Check restart file:
cat cap_restart 
  20150301 000000
ls -l ../input/restart/
lrwxrwxrwx 1 fzeng g0620 136 2018-06-28 16:04 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rs/ens0000/Y2015/M03/2015Mar01.catchcn_internal_rst.20150301_0000
lrwxrwxrwx 1 fzeng g0620 111 2018-06-28 16:04 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/output/CF0090x6C/rs/ens0000/2015Mar01.vegdyn_internal_rst

Check executable:
ls -l ../build/Linux/bin/GEOSldas.x 
-rwxr-xr-x 1 fzeng g0620 50708160 2018-06-28 13:32 ../build/Linux/bin/GEOSldas.x*

Check CAP.rc about the END_DATE, JOB_SGMT and NUM_SGMT. No change needed.
Modify the output resolution in HISTORY.rc to match Eunjee's AGCM output (no more modification needed):
  tavg1_2D_mf-st.resolution: 360 181,
  tavg1_2D_lnd_Nx.resolution: 360 181,  
  
Did an interactive run:

ssh -XY discover-sp3
interactive.py -A sp3 -n 96 -a g0620 -X --debug
cd /discover/nobackup/fzeng/Catchment/hindcast/2015Mar01/run

cp lenkf.j lenkf.j.orig
nedit lenkf.j &

In lenkf.j:
  change ntasks to 96 in L11

In LDAS.rc:
  change NY to 96  

./lenkf.j

Crashed. 

3. Sarith has created 1/8 BCs and modified the offline code for the simulation to write out daily elai and esai. Copied the 1/8 BCs files.

cd /discover/nobackup/fzeng/bcs
mkdir QL
/bin/cp -pr /gpfsm/dnb02/smahanam/QL/DE_02880x01440 QL/.

==========
20180626:

1. Reorder and regrid /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M03/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150301_0000z to CF90 for the offline simulations using hindcast as forcing data set, following notes on 20180604. 

(1) Reorder:

  (1.1) Create path: 

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 
  mkdir -p rst_hindcast/SMAP_EASEv2_M09_reordered

  (1.2) Create shell script "reorder_restarts" for reordering:

  cd rst_hindcast/SMAP_EASEv2_M09_reordered
  nedit reorder_restarts &
  chmod 755 reorder_restarts
  
  reorder_restarts reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
  #!/bin/csh

  set YYYY = 2015
  set YYYYMMDD=`echo ${YYYY}0301`          
          
  mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

  (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:
  
  cp /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 .
  
  nedit mk_LDASsaRestarts.F90 &
    In SUBROUTINE  reorder_LDASsa_restarts:
    (a) L298:
       tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin'
    (b) L540:
       rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//&
            '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_0000z' 

    cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (1.4) Run "reorder_restarts" to reorder the offline restart file:
 
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/SMAP_EASEv2_M09_reordered
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  reorder_restarts
  
(2) Regrid from SMAP_EASEv2_M09 to /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180: 

  (2.1) Create path:

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast
  mkdir -p Heracles-NL_Reynolds_CF90
  
  (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  cd Heracles-NL_Reynolds_CF90
  
  cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 .
  nedit mk_LDASsaRestarts.F90 &
       L67: InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20150301', &
       L68: InCNTilFile = '/discover/nobackup/ltakacs/bcs/Heracles-NL/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til',         &
       L145: print *,'(1.1) mpirun -np 1 bin/mk_LDASsaRestarts -a SPONSORCODE -b BCSDIR -m MODEL -s SURFLAY(20/50) -t TILFILE'
       L147: print *,'(1.2) sbatch mkLDASsa.j'
       L214-223:
        !tmpstring = "set BINDIR=`ls -l bin | cut -d'>' -f2`"
        !write(10,'(a)')trim(tmpstring)
        !tmpstring = "setenv ESMADIR `echo $BINDIR | sed 's/Linux\/bin//g'`"
        write(10,'(a)')'setenv ESMADIR '//trim(ESMADIR)
        write(10,'(a)')'source bin/g5_modules'
        write(10,'(a)')'setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs'
        write(10,'(a)')'setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2'
        write(10,'(a)')'limit stacksize unlimited'
        write(10,'(a)')' ' 
        write(10,'(a)')'mpirun -np 56 '//trim(cmd)//' -j Y'
       L227: if(clm45) write(10,'(a)')'bin/Scale_CatchCN OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45 catchcn_internal_clm45 '//trim(SFL)
       Add this block below to L2047-2062:
        VAR_DUM = 0.1
        do i = 1,4
           STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'WW'), (/1,i/), (/NTILES,1 /),VAR_DUM(:))
        end do
        
        VAR_DUM = 0.25
        do i = 1,4
           STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'FR'), (/1,i/), (/NTILES,1 /),VAR_DUM(:))
        end do
        
        VAR_DUM = 0.001
        do i = 1,4
           STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CH'), (/1,i/), (/NTILES,1 /),VAR_DUM(:))
           STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CM'), (/1,i/), (/NTILES,1 /),VAR_DUM(:))
           STATUS = NF_PUT_VARA_REAL(OutID,VarID(OUTID,'CQ'), (/1,i/), (/NTILES,1 /),VAR_DUM(:))
        end do
       L3437: if(clm45) call system('/bin/cp OutData1/catchcn_internal_clm45 OutData2/catchcn_internal_clm45')       
       [Do "xdiff mk_LDASsaRestarts.F90 ../SMAP_EASEv2_M09_reordered/. &" for more modifications made.]
  cp mk_LDASsaRestarts.F90  /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. 
  cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts
  setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/
  source $ESMADIR/src/g5_modules
  gmake install
  
  (2.3) Run mk_LDASsaRestarts to convert to desired tile system:
  
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/Heracles-NL_Reynolds_CF90
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  nedit mkLDASsa.j & [Create a job file]

  mkLDASsa.j reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
#!/bin/csh -f
 
#SBATCH --account=g0620
#SBATCH --time=1:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=56
#SBATCH --job-name=mkLDAS
#SBATCH --constraint=hasw
#SBATCH --output=mkLDAS.o
#SBATCH --error=mkLDAS.e
 
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
limit stacksize unlimited

setenv SPONSORID g0620
setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/
setenv TILFILE CF0090x6C_DE0360xPE0180-Pfafstetter.til

mkdir -p OutData1/
mkdir -p OutData2/
ln -s $BCSDIR/$TILFILE OutData1/OutTileFile
ln -s $BCSDIR/$TILFILE OutData2/OutTileFile
ln -s $BCSDIR/clsm OutData2/clsm

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

/bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50  
  
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
           
  chmod 755 mkLDASsa.j
  sbatch mkLDASsa.j
  
Finished.

mv catchcn_internal_rst catchcn_internal_rst.20150301_0000z
mv OutData1 OutData1_20150301_0000z
mv OutData2 OutData2_20150301_0000z

This doesn't work because there is no ndvi_clim_*.data in /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/.

Sarith said I have to use /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180 instead. He said it will be zero-diff against Heracles-NL
/discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/. (see notes on 20180725)

2. Regrid from SMAP_EASEv2_M09 to /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180: 

  (2.1) Create path:

  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast
  mkdir -p Icarus-NL_Reynolds_CF90
  
  (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it:

  cd Icarus-NL_Reynolds_CF90
  
  cp ../Heracles-NL_Reynolds_CF90/mk_LDASsaRestarts.F90 .  [No need to modify.]
  
  (2.3) Run mk_LDASsaRestarts to convert to desired tile system:
  
  cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_hindcast/Icarus-NL_Reynolds_CF90
  ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin
  nedit mkLDASsa.j & [Create a job file]

  mkLDASsa.j reads:
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  
#!/bin/csh -f
 
#SBATCH --account=g0620
#SBATCH --time=1:00:00
#SBATCH --qos=debug
#SBATCH --ntasks=56
#SBATCH --job-name=mkLDAS
#SBATCH --constraint=hasw
#SBATCH --output=mkLDAS.o
#SBATCH --error=mkLDAS.e
 
setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05
source $ESMADIR/src/g5_modules
setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs
setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2
limit stacksize unlimited

setenv SPONSORID g0620
setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/
setenv TILFILE CF0090x6C_DE0360xPE0180-Pfafstetter.til

mkdir -p OutData1/
mkdir -p OutData2/
ln -s $BCSDIR/$TILFILE OutData1/OutTileFile
ln -s $BCSDIR/$TILFILE OutData2/OutTileFile
ln -s $BCSDIR/clsm OutData2/clsm

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

/bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst

mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y 

bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50  
  
  ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~  
           
  chmod 755 mkLDASsa.j
  sbatch mkLDASsa.j
  
Finished.

mv catchcn_internal_rst catchcn_internal_rst.20150301_0000z
mv OutData1 OutData1_20150301_0000z
mv OutData2 OutData2_20150301_0000z

==========
20180627:

1. Continued working on 2 of 20180625.

==========
20180628:

1. Worked on 2 of 20180622. 

2. Read the tutorial for the current way to compile the code and set up a run. 

3. Continued working on 2 of 20180625.

Checked M2n5P.bat and M2n5P.exec to see if they agree with the tutorial.

4. Checked out the tag that Sarith created for writing out daily elai and esai:

cd /discover/nobackup/fzeng/offline_code
mkdir tmp
cd tmp
cvs co -r GEOSldas_m4-17_pLAI GEOSldas_m4
mv GEOSldas GEOSldas_m4-17_pLAI
mv GEOSldas_m4-17_pLAI ../

Modified LDAS_TileCoordRoutines.F90 to read these 1/8 BCs:

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_pLAI/src/GEOSldas_GridComp/Shared
cp -p LDAS_TileCoordRoutines.F90 LDAS_TileCoordRoutines.F90.orig
nedit LDAS_TileCoordRoutines.F90 &

==========
20180629:

1. The 2015Mar01 run crashed. Talked to Weiyuan. Weiyuan found that some updates need to be made to GEOS_CatchCNGridComp.F90. He told me to follow GEOS_CatchGridComp.F90 to make the updates.

Modified GEOS_CatchCNGridComp.F90.

Followed notes on 2 of 20180622 to compile.

# ----------------------------
#  Activate V24_C05 updates
# ----------------------------
# (i) Build the model; (ii) go to GEOSsurface_GridComp and gmake clean (iii) gmake install FOPT='-DLAND_UPD' (iv) goto Applications/GEOSgcm_App and gmake install

Build:

cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src
setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6
source g5_modules
make -j 8 install |& tee tmpfile.txt

Verify build:

./Config/gmh.pl tmpfile.txt
    All 44 packages compiled successfully.

Activate V24_C05 updates:

cd GEOSldas_GridComp/GEOSsurface_GridComp/
gmake clean 
gmake install FOPT='-DLAND_UPD'


Stopped here on Friday 20180629. 

cd ../../Applications/LDAS_App/ 
gmake install
    
cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/
mkdir -p exec/hindcast
/bin/cp -pr Linux exec/hindcast/.