20160401:

1. Mar 2016 report.

2. Understanding MODIS FPAR data files. 

MODIS FPAR
Contacted the curator on the webpage (http://landweb.nascom.nasa.gov/cgi-bin/QA_WWW/newPage.cgi?fileName=mod15cm&subdir=modland_v2_metadata) where MOD15CM (MODIS CMG Monthly LAI and FPAR Product) is listed but with little information. The curator, Dr. Demin Feng in the MODIS LDOPE (Land Data Operational Product Evaluation) group at Goddard, told me that this climate modeling grid monthly product was never created. 

Is MCD15A2 (available starting from 20020704 on http://reverb.echo.nasa.gov) better than MOD15A2?
Read the user guide and the literature. 
No luck on this so far. This may not matter.

MODIS/Terra+Aqua LAI/FPAR 8-day L4 Global 1km SIN grid V005 product (MCD15A2) each at a 10°x10° tile (1200 rows x 1200 columns)
size of each file: 2 MB
number of files: 36*18*(365/8)*(2012-2001+1) = 229,897,440
data format: HDF-EOS
total size: 2 * 229,897,440 / 1024 = 449,018 GB (for 1km resolution)
Will be 449,018/(60*60) = 124.7 GB for 0.5-deg resolution

After downloading a set of (36*18) granules for one 8-day global coverage (how to do so?), before proceeding to download the next set:
reduce spatial resolution from 1km to 0.5deg
aggregate to one data file (how? see below)
delete the 1km granules

How to convert from sinusoidal projection to climate modeling grid?
Can we just put the granules one next to each other?
No. See Fig. 1 in the MODIS-LAI-FPAR-User-Guide.pdf.

How to convert from 8-day to monthly? (learn from Yasuko)


=============
20160404:

1. Ran process-smap to make e0004s and e0005s run through 20160331 and to save land and CN restarts for 20160401.

2. Archived the 9km SMAP land and CN restarts files:

/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > scp -p *20151231* /archive/u/fzeng/Catchment/SMAP_EASEv2_M09/.
/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > scp -p *20160201* *20160301* /archive/u/fzeng/Catchment/SMAP_EASEv2_M09/.

3. Created scripts to analyze the NEE difference between e0004s and e0005s.
gs6101-land01:/land/fzeng/projects/fluxes/code

=============
20160405:

1. Continued 3 on 20160405. Created scripts to analyze the NEE difference between e0004s and e0005s.
gs6101-land01:/land/fzeng/projects/fluxes/code

2. Met with Randy to discuss how we can find monthly CGM MODIS FPAR data.


=============
20160406:

1. Processed GMAO GEOS5 seasonal forecast for April.

2. Continued 3 on 20160405. Created scripts to analyze the NEE difference between e0004s and e0005s.
gs6101-land01:/land/fzeng/projects/fluxes/code


=============
20160407: 

1. Continued 3 on 20160405. Created scripts to analyze the NEE difference between e0004s and e0005s.
gs6101-land01:/land/fzeng/projects/fluxes/code

Verified that the NEE I extracted is identical to Eunjee's.

Verified that the zonal mean plot of p0008se_63_p3 by comparing with Brad's .

Discussed with Eunjee about the undefined values in the model output. The undefined values in the offline model output is -999 as in the control files. When I use the "fwrite" function to extract one or more variables and write out a new .gdat file (output), by defaulte the undefined value in the output is -9.99e8 (see http://gradsusr.org/pipermail/gradsusr/2009-August/009219.html). I can add a line "set undef -999" after the line of "fwrite" to set the undefined value in the output back to -999 to be consistent with the input. 

Discussed with Brad about the zonal mean plots of e0004s vs. e0005s. He is concerned that the precipitation data in the forcing (/discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing/d591_fpit) that I am using for my e0004s and e0005s runs have deficiency in Africa (it's too low in the Sahel region and too high in the region right south of the Sahel region - region names not exactally correct here). 

 
=============
20160408:

1. Review the code for my transient CO2 run without fire. What I did was:

(1) Ran the offline model at 280ppm atmospheric CO2 concentration until the carbon state reached equilibrium. process_cat.F90 was modified to use co2v=280 instead of the original 390. See /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/p0008s/process_cat.F90
The output of the last cycle is in /archive/u/fzeng/Catchment/princeton/p0008s_63.tgz.

(2) Starting from the end of p0008s_63, ran the offline model using increasing co2v. I named this p0008se_63 ("e" means "extension). Again, modifications were made to process_cat.F90, see
/discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/p0008se/process_cat.F90
Also, 
(a) a file year_co2.txt which contains just one number "1850" needs to be created by "echo 1850 > year_co2.txt" in the directory where lenkf.j and CN_restart is located. 
(b) the job script "lenkf.j" needs to be modified. 
(c) Need to create "submit_next_batch_princeton_co2v"

In (2), co2v increased from 280ppm in 1850 to 311ppm in 1950, and then to 391ppm in 2012. Since there are 163 years from 1850 to 2012, it requires 3 cycles when using Princeton forcing which is 65 years per cycle. The 2nd cycle is not a full 65-year cycle. This is to make sure that p3 is using the right forcing for each CO2 year from 1948 to 2012. I gave them these names: p0008se_63_p1, p0008se_63_p2, and p0008se_63_p3 ("p" means "part").

                CO2 year       forcing year       CO2 concentration (ppm)            
p0008se_63_p1:  1850-1914      1948-2012                280.00-299.84
p0008se_63_p2:  1915-1947      1948-1980                300.15-310.07
p0008se_63_p3:  1948-2012      1948-2012                310.38-391.00

Procedures for (2):

(2.1) See my notes on 20151021 (below) to set up p0008se_63_p1. It's named p0008se at the beginning, and then renamed to p0008se_63_p1 when the run is finished.
~~~~~~~~~~~~~~~~~~~
6. Set up the extended run for p0008s using carbon and land restarts from p0008s_63. This is for 1850-1914 using 1948-2012 forcing.

/discover/nobackup/fzeng/Catchment/princeton > mkdir -p p0008se/RUN/rs/ens0000/Y1948/M01
/discover/nobackup/fzeng/Catchment/princeton/p0008se > echo 1850 > year_co2.txt
/discover/nobackup/fzeng/Catchment/princeton/p0008se > cp -p ../p0008s_63/CN_restart .
/discover/nobackup/fzeng/Catchment/princeton/p0008se > cp -p ../p0008s_63/RUN/rs/ens0000/Y2013/M01/p0008.ens0000.catch_ldas_rst.20130101_0000z.bin RUN/rs/ens0000/Y1948/M01/p0008.ens0000.catch_ldas_rst.19480101_0000z.bin
/discover/nobackup/fzeng/Catchment/princeton/p0008se > cp -p ../p0008s_63/lenkf.j .
/discover/nobackup/fzeng/Catchment/princeton/p0008se > nedit lenkf.j &

Make sure that carbon restart and land restart have the same time stamp:
dali16:/discover/nobackup/fzeng/Catchment/princeton/p0008se > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 1530820456 2015-10-21 07:09 CN_restart
dali16:/discover/nobackup/fzeng/Catchment/princeton/p0008se > ls -l RUN/rs/ens0000/Y1948/M01/
total 40512
-rw-r--r-- 1 fzeng g0620 41453776 2015-10-21 07:09 p0008.ens0000.catch_ldas_rst.19480101_0000z.bin

Make sure that the beginning year is 1850:
dali16:/discover/nobackup/fzeng/Catchment/princeton/p0008se > cat year_co2.txt 
1850

Submit a debug job:
/discover/nobackup/fzeng/Catchment/princeton/p0008se > sbatch lenkf_test.j
Running. 

Submit the full job:
/discover/nobackup/fzeng/Catchment/princeton/p0008se > sbatch lenkf.j
~~~~~~~~~~~~~~~~~~~

(2.2) When this first cycle/part is finished, manually process the output in p0008se using grid_restore_princeton, and then set up the run for the 2nd cycle/part (my notes on 20151023):

dali16:/discover/nobackup/fzeng/Catchment/princeton > mv p0008se p0008se_63_p1
dali16:/discover/nobackup/fzeng/Catchment/princeton > mkdir -p p0008se/RUN/rs/ens0000/Y1948/M01
/discover/nobackup/fzeng/Catchment/princeton/p0008se_63_p1 > cp -p CN_restart lenkf.j year_co2.txt ../p0008se/.
/discover/nobackup/fzeng/Catchment/princeton > cp -p p0008se_63_p1/RUN/rs/ens0000/Y2013/M01/p0008.ens0000.catch_ldas_rst.20130101_0000z.bin p0008se/RUN/rs/ens0000/Y1948/M01/p0008.ens0000.catch_ldas_rst.19480101_0000z.bin

Make sure that carbon restart and land restart have the same time stamp:
dali16:/discover/nobackup/fzeng/Catchment/princeton/p0008se > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 1530820456 2015-10-23 09:23 CN_restart
dali16:/discover/nobackup/fzeng/Catchment/princeton/p0008se > ls -l RUN/rs/ens0000/Y1948/M01
total 40512
-rw-r--r-- 1 fzeng g0620 41453776 2015-10-23 09:23 p0008.ens0000.catch_ldas_rst.19480101_0000z.bin

Make sure that the start year is 1915:
/discover/nobackup/fzeng/Catchment/princeton/p0008se > cat year_co2.txt 
        1915
        
/discover/nobackup/fzeng/Catchment/princeton/p0008se > nedit lenkf.j & (change stopyear to 1981)
/discover/nobackup/fzeng/Catchment/princeton/p0008se > sbatch lenkf.j

Note that the "lenkf.j" in p0008se_63_p2 is different from those in the other two parts. 

(2.3) Repeat (2.2) when the 2nd cycle is finished. 

2. LAI is too low in SE US and the Amazon for June in Sarith's run output. Compared his process_cn.F90 to my process_cat.F90. 


=============
20160411:

1. Continued 2 on 20160408. Sarith found that the vegetation type/fraction, which should be constant, changes over time in zones 2 and 3. Why? Investigated the code.


=============
20160412:

1. Processed and analyzed MODIS FPAR data from Taejin Park at BU.

Found FPAR=0.89 for July of 2005 and 2009-2012, and for June of 2010-2011 in (100,550) hdeg /land/fzeng/projects/fpar/output/fpar_modis_c5_bu_hdeg_2001_2012.mat
This pixel is located in the desert area of NW China. Why is the FPAR so high for July there?
Checked the original data from Taejin Park using gs6101-land01:/land/fzeng/projects/fpar/code/check_modis_fpar_bu.m.
Yes. 0.89 is from the original data.
Pixel (100,550) at hdeg resolution is centered at (40.25N,94.75E). Google Earth shows that this area is mostly cropland, so the FPAR values are correct.


=============
20160413:

1. Updated Higgins precipitation data through 20160411.

2. Compared our FPAR to GIMMS FPAR and MODIS FPAR. Got similar results in the two comparisons for both maximum FPAR and month of maximum FPAR, suggesting that GIMMS FPAR and MODIS FPAR are close to each other and our FPAR does have some biases.
 
gs6101-land01:/land/fzeng/projects/fpar/code/compare_maxfpar_4panels.m, compare_mon_maxfpar_3panel.m

3. Met with Eunjee to talk about scientific ideas. 


=============
20160414:

1. Analyzed e0004s NEE to take a look at the 2015 anomalies.

/discover/nobackup/fzeng/Catchment/e0004s_27 > scp -p e0004_??????.gdat gs6101-land01:/land/fzeng/projects/fluxes/data/e0004s_com/.
/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s > scp -p e0004_??????.gdat gs6101-land01:/land/fzeng/projects/fluxes/data/e0004s_com/.
(Note: /discover/nobackup/fzeng/Catchment/e0004s_27 was extracted from /archive/u/gkwalker/Catchment/SMAP_EASEv2_M09/e0004s_27.tgz)

Also note that Greg's e0004s_27 and my e0004s use different forcing datasets:

e0004s_27: (200101-201503)
met_tag: cross_RPFPIT__precCPCUG5RPFPITv2 
met_path: /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing/

e0004s and e0005s: (201504-201603/present)
met_tag: cross_FP__precCPCUG5FPv2 
met_path: /discover/nobackup/projects/gmao/merra/iau/merra_land/GEOS5_land_forcing/

2. Checked out the updated SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 tag. 
/discover/nobackup/fzeng > cvs co -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 LDASsa_m3

Modified tile_coord.F90 and clsm_ensdrv_init_routines.F90 in LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 so that it can run on the hdeg tile coordinate system that Sarith created for me to use in my m2_m0008hdeg run.

/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > cp -p /discover/nobackup/fzeng/LDASsa_m3-15_2-CN/src/cleanup .
/discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > nedit cleanup &
(changed exp to "m3-16_0_p2_CatchCatchCN_for_MERRA3")

Compiled.

3. Added 3 diagnostics (9:30 local time gpp, tsurf and rzmc) to merra2_m0008hdeg for Joanna and Yasuko.

Modifications needed for these 4 routines:
catch_types.F90: done
clsm_ensdrv_init_routines.F90: done
clsm_ensdrv_out_routines.F90: done
process_cat.F90: done

=============
20160415:

1. Continued 3 on 20160414. 

Compiled. 

2. Set up the new m0008hdeg_21_5min simulation (following notes on 20160329).

/discover/nobackup/fzeng/Catchment/merra2 > mv m0008hdeg_21_5min m0008hdeg_21_5min_old
/discover/nobackup/fzeng/Catchment/merra2 > mkdir -p m0008hdeg_21_5min/RUN/rs/ens0000/Y2007/M01
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_20 > cp -p CN_restart ../m0008hdeg_21_5min/.
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_20/RUN/rs/ens0000/Y2016/M01/m0008.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_21_5min_old/lenkf.j .

dali14:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 280023744 2016-03-29 09:19 CN_restart
dali14:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l RUN/rs/ens0000/Y2007/M01
total 7424
-rw-r--r-- 1 fzeng g0620 7583072 2016-03-29 09:19 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin

Submitted a debug run. It's working. Submitted the full run.

3. Sarith helped me set up two simulations using the new offline tag. 

####Organize the notes####
See /discover/nobackup/fzeng/notes/set_up_consolidated_catchcn_simulations

4. Updated the number of variables from 85 to 88 in grid_restore_merra_hdeg_daily.f90, recompiled and processed the output in merra2/m0008hdeg_21_5min:

~/Catchment > nedit grid_restore_merra_hdeg_daily.f90 &
~/Catchment > ifort -o grid_restore_merra_hdeg_daily grid_restore_merra_hdeg_daily.f90
~/Catchment > grid_restore_merra_hdeg_daily 0008 hdeg_21_5min

Finished 20071109 so far. 

Checked the data using ~/Catchment/verify_m0008hdeg_21_5min_2.gs.
sif, lai, apar, ipar, psn, sifl, aparl, iparl, psnl, gpp, rzmc, and tsurf: new m0008hdeg_21_5min = old m0008hdeg_21_5min (which was sent to Yasuko earlier), which is good.
lail : new m0008hdeg_21_5min differs from old m0008hdeg_21_5min, which is bad. Big negative numbers in lail, e.g. -9720 to -9690.
gppl, rzmcl and tsurfl: big negative numbers, which is bad.

Fixed the problem for lail. In process_cat.F90:

This block 

! LAI and GPP snapshot at 09:30 local time
! -------------------------------- 
    do n = 1,N_cat
      local_time = date_time%hour + date_time%min/60. + date_time%sec/3600. + 12.*tile_coord(n)%com_lon/180.
      if(local_time <   0.) local_time = local_time + 24.
      if(local_time >= 24.) local_time = local_time - 24.

      ldiff = abs(local_time - 9.5) ! find time closest to 9:30am
      if(ldiff < ldiff2_save(n)) then
        lai_save(n)    = cat_diagS(n)%lai
        gpp_save(n)    = cat_diagF(n)%gpp
        ldiff2_save(n) = ldiff
      endif
    end do
    
and this block

! LAI and GPP snapshot at 09:30 local time
! Can not put this block right after the above "LAI and GPP snapshot at 09:30 local time" block
! ---------------------------------------------------------------------------------------------

    if(date_time%hour==0 .and. date_time%min==0 .and. date_time%sec==0) then
      cat_diagS(:)%lail  = lai_save(:)  * 86400/dtstep
      cat_diagF(:)%gppl  = gpp_save(:)  * 86400/dtstep
      ldiff2_save = 1.e20
      lai_save    = -1.e20
      gpp_save    = -1.e20
     else
      cat_diagS(:)%lail  = 0.
      cat_diagF(:)%gppl  = 0.
    endif 
    
can not be put together. The 2nd block has to be after this block below:

! scale CN diags
! --------------
    scale = 1./scale

    cat_diagS(:)%lai      = scale * cat_diagS(:)%lai
    cat_diagS(:)%sai      = scale * cat_diagS(:)%sai
    cat_diagS(:)%lai11    = scale * cat_diagS(:)%lai11
    cat_diagS(:)%lai12    = scale * cat_diagS(:)%lai12
    cat_diagS(:)%lai13    = scale * cat_diagS(:)%lai13
    cat_diagS(:)%lai21    = scale * cat_diagS(:)%lai21
    cat_diagS(:)%lai22    = scale * cat_diagS(:)%lai22
    cat_diagS(:)%lai23    = scale * cat_diagS(:)%lai23
    cat_diagS(:)%totcolc  = scale * cat_diagS(:)%totcolc
    cat_diagS(:)%totcolc1 = scale * cat_diagS(:)%totcolc1
    cat_diagS(:)%totcolc2 = scale * cat_diagS(:)%totcolc2
    cat_diagS(:)%totcolc3 = scale * cat_diagS(:)%totcolc3
    cat_diagF(:)%npp      = scale * cat_diagF(:)%npp
    cat_diagF(:)%gpp      = scale * cat_diagF(:)%gpp
    cat_diagF(:)%sr       = scale * cat_diagF(:)%sr
    cat_diagF(:)%nee      = scale * cat_diagF(:)%nee
    cat_diagF(:)%padd     = scale * cat_diagF(:)%padd
    cat_diagS(:)%root     = scale * cat_diagS(:)%root
    cat_diagS(:)%vegc     = scale * cat_diagS(:)%vegc
!!  cat_diagS(:)%xsmr     = scale * cat_diagS(:)%xsmr
    cat_diagF(:)%burn     = scale * cat_diagF(:)%burn
    cat_diagF(:)%closs    = scale * cat_diagF(:)%closs
    cat_diagS(:)%fsel     = scale * cat_diagS(:)%fsel

    else ! CN diags set to zero

    cat_diagS(:)%lai      = 0.
    cat_diagS(:)%sai      = 0.
    cat_diagS(:)%lai11    = 0.
    cat_diagS(:)%lai12    = 0.
    cat_diagS(:)%lai13    = 0.
    cat_diagS(:)%lai21    = 0.
    cat_diagS(:)%lai22    = 0.
    cat_diagS(:)%lai23    = 0.
    cat_diagS(:)%totcolc  = 0.
    cat_diagS(:)%totcolc1 = 0.
    cat_diagS(:)%totcolc2 = 0.
    cat_diagS(:)%totcolc3 = 0.
    cat_diagF(:)%npp      = 0.
    cat_diagF(:)%gpp      = 0.
    cat_diagF(:)%sr       = 0.
    cat_diagF(:)%nee      = 0.
    cat_diagF(:)%padd     = 0.
    cat_diagS(:)%root     = 0.
    cat_diagS(:)%vegc     = 0.
!!  cat_diagS(:)%xsmr     = 0.
    cat_diagF(:)%burn     = 0.
    cat_diagF(:)%closs    = 0.
    cat_diagS(:)%fsel     = 0.

    endif
    
However, the problem for gppl, rzmcl and tsurfl still exists. 


=============
20160418: 

1. Continued 4 on 20160415. 

It took a long time to get the debug simulation to run, so tried to do the simulation interactively.

Followed notes from meeting with Sarith before (/discover/nobackup/fzeng/notes/sarith) and did this:
dali17:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > interactive.py -A sp3 -n 96 -a g0620 -X -t 3:00:00  

But got this error message:
Running: /usr/bin/ssh -XYqt -p2255 dali17 salloc --ntasks=96 --constraint=sp3 --time=3:00:00 --account=g0620 --mail-type=BEGIN
Traceback (most recent call last):
  File "/home/fzeng/bin/interactive.py", line 163, in <module>
    main()
  File "/home/fzeng/bin/interactive.py", line 129, in main
    sp.check_call(cmd.split())
  File "/usr/local/other/SLES11/SIVO-PyD/1.10.0/lib/python2.7/subprocess.py", line 504, in check_call
    raise CalledProcessError(retcode, cmd)
subprocess.CalledProcessError: Command '['/usr/bin/ssh', '-XYqt', '-p2255', 'dali17', 'salloc', '--ntasks=96', '--constraint=sp3', '--time=3:00:00', '--account=g0620', '--mail-type=BEGIN']' returned non-zero exit status 255 

This doesn't work on dali. Needs to be on discover-sp3. The following worked:
discover05:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > interactive.py -A sp3 -n 96 -a g0620 -X -t 3:00:00


=============
20160419:

1. Continued 1 on 20160418. Asked Sarith. He pointed out that I should modify scalar2cat_diagS/F, cat_diagS/F_div_scalar and add_cat_diagS/F in catch_type.F90 as well, which I forgot. Fixed this. Compiled. 

Did a test run. Checked the data using ~/Catchment/verify_m0008hdeg_21_5min_2.gs. The rzmcl and tsurf look good now. gppl also look ok, but with some discontinuity probably because of 3-hour DTCN.

Tried using 5min DTCN. Change DTCN in process_cat.F90. Recompiled. 

/discover/nobackup/fzeng/Catchment/merra2 > mv m0008hdeg_21_5min m0008hdeg_21_5min_3hrDTCN
/discover/nobackup/fzeng/Catchment/merra2 > mkdir -p m0008hdeg_21_5min/RUN/rs/ens0000/Y2007/M01
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_20/CN_restart .
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_20/RUN/rs/ens0000/Y2016/M01/m0008.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2007/M01/m0008.ens0000.catch_ldas_rst.20070101_0000z.bin
/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > cp -p ../m0008hdeg_21_5min_3hrDTCN/lenkf.j .

dali17:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 280023744 2016-03-29 09:19 CN_restart
dali17:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l RUN/rs/ens0000/Y2007/M01
total 7424
-rw-r--r-- 1 fzeng g0620 7583072 2016-03-29 09:19 m0008.ens0000.catch_ldas_rst.20070101_0000z.bin

Submitted a debug run. The gppl looks good now. Submitted a full run. 

NOTE that the m0008hdeg_21_5min simulation output that I sent Yasuko used 5min dtstep and 3hr DTCN.
The new m0008hdeg_21_5min simulation output uses 5min for both dtstep and DTCN. 

2. My simulations m0001hdeg_cat and m0001hdeg_cn finished 1988-1995. Submitted next batch to run from 1996 to 2003. 

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/run > qsub lenkf.2.j
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/run > qsub lenkf.2.j 


=============
20160420:

1. In the lenkf.j of m0008hdeg_21_5min, I set ntasks as 84 and made it run 5 years in 12 hours. It actually only finished 2 years and 4 months within 12 hours (ended somewhere in May 2009). Changed ntasks to 168 and make it run 4 years in 12 hours. 

/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min/RUN/rs/ens0000/Y2009 > /bin/rm -rf M0[2-5]

dali14:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l CN_restart 
-rw-r--r-- 1 fzeng g0620 280023744 2016-04-20 06:39 CN_restart
dali14:/discover/nobackup/fzeng/Catchment/merra2/m0008hdeg_21_5min > ls -l RUN/rs/ens0000/Y2009/M01
total 7424
-rw-r--r-- 1 fzeng g0620 7583072 2016-04-20 06:39 m0008.ens0000.catch_ldas_rst.20090101_0000z.bin

Submitted the job to continue the simulation. 

2. My simulations m0001hdeg_cat and m0001hdeg_cn finished 1996 to 2003. Submitted next batch to run from 2004 to 2011.

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/run > ls -l lenkf_job_completed.txt
-rw-r--r-- 1 fzeng g0620 11 2016-04-20 07:07 lenkf_job_completed.txt
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/run > cat lenkf_job_completed.txt
 SUCCEEDED
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/run > qsub lenkf.3.j
 
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/run > ls -l lenkf_job_completed.txt
-rw-r--r-- 1 fzeng g0620 11 2016-04-20 10:31 lenkf_job_completed.txt
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/run > cat lenkf_job_completed.txt
 SUCCEEDED
/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/run > qsub lenkf.3.j 

=============
20160422:

1. The simulation m0008hdeg_21_5min finished. Looked at the output. Found large differences in NW India for 5min vs 3hr DTCN. 

=============
20160425:

1. Talked to Randy about the issue on the differences between 5min and 3hr DTCN. Randy suggested me to use the single point driver the investigate the issue. 

2. Met with Eunjee to help her set up the transient CO2 simulation. 

3. Checked the two simulations Catchment/merra3/m0001hdeg_cat and m0001hdeg_cn. They finished successfully.

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/output/hdeg/cat/ens_avg > ls -l Y2015/M12
total 12192
-rw-r--r-- 1 fzeng g0620 12458016 2016-04-21 11:49 m0001hdeg_cat.ens_avg.ldas_grid_monthly_out.201512.bin

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cat/output/hdeg/rs/ens0000 > ls -l Y2016/M01
total 6624
-rw-r--r-- 1 fzeng g0620 6770600 2016-04-21 11:49 m0001hdeg_cat.ens0000.catch_ldas_rst.20160101_0000z.bin

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/output/hdeg/cat/ens_avg > ls -l Y2015/M12
total 21696
-rw-r--r-- 1 fzeng g0620 22207680 2016-04-21 13:17 m0001hdeg_cn.ens_avg.ldas_grid_monthly_out.201512.bin

/discover/nobackup/fzeng/Catchment/merra3/m0001hdeg_cn/output/hdeg/rs/ens0000 > ls -l Y2016/M01
total 284352
-rw-r--r-- 1 fzeng g0620 291174095 2016-04-21 13:17 m0001hdeg_cn.ens0000.catchcn_ldas_rst.20160101_0000z

However, Sarith said the cn code is buggy. See 4 below.

4. Met with Sarith to update the routines that he modified to fix the bug in the CN model code. Also, model code was modified to be consistent to the m3-15_p2-CN and then re-built. A simulation will be set up to see if there is only minor difference between the output of this new tag and that of my existing simulation that based on the m3-15_p2-CN code. 

Sarith will create catchcn_internal_rst from the CN_restart and land restart of my m0008hdeg_19.


=============
20160426:

1. Copied Jung's GPP data to discover for Eunjee. 

2. Organized the notes from meeting with Sarith on 20160415 to set up catchment and carbon simulations based on the new tag.

3. Safety Awareness Campaign.

=============
20160427:

1. Archived princeton_hourly to release some space on nobackup:

/discover/nobackup/fzeng > tarem princeton_hourly
/discover/nobackup/fzeng > ls /archive/u/fzeng/Catchment/
merra/  merra2/  princeton/  princeton_hourly.tgz  SMAP_EASEv2_M09/
/discover/nobackup/fzeng > /bin/rm -rf princeton_hourly

2. Updated Higgins precipitation data through 20160426.

3. Safety Awareness Campaign.

4. Met with Sarith to set up a carbon simulation that can be compared to my m0008hdeg_20. However, the run crashed. The problem is in the catchcn_internal_rst. Give up for now. 

Updated the routines back to what they are in SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 to remove the modifications made in 4 on 20160425.

(1) Compare:

(1a) LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3
These are the routines that differ:
- Components/GEOSlana_GridComp/process_cn.F90
- Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/catchmentCN.F90
- Components/GEOSsurface_GridComp/Shared/StieglitzSnow.F90

(1b) LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > ~ltakacs/cvstools/cvsxdiff -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 GEOSlana_GridComp/process_cn.F90

(2) Update the routines listed in 1(a):

LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > /bin/rm process_cn.F90
LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 process_cn.F90
LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > module avai other/tkcvs (or module avail other/tkcvs)
LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > module load other/tkcvs-8.2.3
LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > tkcvs process_cn.F90 & (to see what modifications were made)

(3) Check and make sure process_cn.F90 has been updated:
LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3

(4) Repeat (1)-(3) for catchmentCN.F90 and StieglitzSnow.F90.
 
5. Investigated the issue about 5min vs 3hr DTCN. 

gs6101-land01:~/point_driver_LDASsa/projects> mkdir dtcn
gs6101-land01:~/point_driver_LDASsa/projects/dtcn> cp -pr ../../point_driver_LDASsa_m3-15_2-CN_20150814 point_driver_LDASsa_m3-15_2-CN

princeton data can be extracted using NCCS:/home/gkwalker/Catchment/extract_princeton_point_pexp_hires.f90

model_dtstep is set in conus_force_loop.F90.
dtcn and firefac is set in process_cat.F90.


=============
20160428:

1. Continued 5 on 20160427. 

conus_force_loop.F90:

L302-308:     
    if(iter == niter) call outfld(N_cat,cat_diagS_avg,cat_diagF_avg,ctile) ! output monthly means if final cycle

    fpar = cat_diagF_avg(1)%apar/cat_diagF_avg(1)%ipar
    write(6,800) iter,date_time%dofyr,date_time%year,date_time%month,cat_progn%tg1,fpar,cat_diagS_avg%lai,cat_diagS_avg%totcolc1, &
                 cat_diagS_avg%totcolc2,cat_diagS_avg%totcolc3,cat_diagS_avg%vegc,1./cat_diagS_avg%xsmr, &
		 cat_diagS_avg%root,86400*cat_diagF_avg%padd
800 format(2i4,i5,i3,f7.2,f9.6,f9.5,3f9.1,2f8.1,2es10.3)

L313-315:
! write land model restart at end of each cycle
    rst_name = 'restart/lnd_restart'//ctile
    call io_rstrt('w', rst_name, def_name, N_cat, cat_progn, cat_param)
    
In process_cat.F90:
set dtcn to 10800, firefac to 0.1.

Plan: 
(1) First spin up with 20min model time step, 3hr dtcn, 0.1 firefac, write output monthly, 33 cycles (65 years/cycle).
(2) Then starting from (1), set up the following two simulations:
(2a) 5min model time step, 3hr dtcn, write output every 3hr;
(2b) 5min model time step, 5min dtcn, write output every 5min;

Step 1: used ~/Catchment/princeton_tiles_location.f90 to locate tiles of interest. 
   
Tiles of interest:
NW India: (23N, 77.3E)             
SE US:    (35N, 86.75W)
SE China: (25N, 113.2E)
Amazon:   (0,   73.45W)
Central W Africa: (6.25N, 10.64E)

Pick the one that has a relatively high fraction (the 6th column below).

NW India: (23N, 77.3E)
  22730 104  58   77.34   23.17 0.01249 18 19 16 17 0.0000 0.9948 0.0000 0.0052   

SE US:    (35N, 86.75W) 
  222607  38  64  -86.77   35.15 0.01488  7  7 18 19 0.9350 0.0000 0.0205 0.0445  
 
SE China: (25N, 113.2E)
  89694 118  59  112.82   25.12 0.01274 18 19 10 11 0.0000 0.7200 0.0000 0.2800

Amazon:   (0,   73.45W)
  312960  44  46  -73.33    0.07 0.01250  4  4 18 19 0.9937 0.0000 0.0000 0.0063

Central W Africa: (6.25N, 10.64E)
  166868  77  49   10.72    6.15 0.01283 10 11 18 19 0.0000 0.7193 0.0000 0.2807

Step 2: used ~/Catchment/extract_princeton_point_pexp_hires.f90 to extract forcing data for these tiles.
This step takes quite some time. 

Step 3: After step 2 was done, copied the forcing data to land01. Spun up with 20min model time step, 3hr dtcn, 0.1 firefac, write output monthly, 33 cycles (65 years/cycle).

Step 4: to set up the 2a simulation above,
Modified conus_force_loop.F90:
(a) changed model_dtstep from 1200 to 300;
(b) changed niter (number of cycle) from 33 to 1;
(c) 

=============
20160429:

1. Met with Joanna and Joe Berry.

2. Modified conus_force_loop.F90 to write 3hr output for 3hr DTCN. The output is all undefined values. Asked Sarith for help. He showed me how to use totalview to debug.