========== 20160801: 1. The test run e0004s submitted on 20160729 failed. Debug. 2. Tried using the met tag and path Sarith uses. Changed the met tag and met path in M09_CN.exe. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/Catchment/merra3 /discover/nobackup/fzeng/Catchment/merra3/run/M09_CN.exe /discover/nobackup/fzeng/Catchment/merra3/run/M09.bat --runmodel --monthsperjob 12 --landmodel catchCN /discover/nobackup/fzeng/Catchment/merra3/e0004s/input/restart > /bin/rm regridded_restarts /discover/nobackup/fzeng/Catchment/merra3/e0004s/input/restart > cp -p /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts/catchcn_internal_rst . Modified lenkf.0.j: removed "regridded_restarts" from the restart_path. Made sure that the setting in driver_inputs_M09_GLOBAL.nml is what desired. Complained about L349 in StieglitzSnow.F90. Tried using Sarith's M09 catchcn_internal_rst (/discover/nobackup/smahanam/MERRA3/regridded_restarts/M09/catchcn_internal_rst). It worked. So the problem is in hydrology parameters in my catchcn_internal_rst instead of cn variables. 3. July monthly report. 4. Sarith found a bug in mk_LDASsaRestarts.F90 and fixed it. mk_LDASsaRestarts.F90 originally used Qing's LDAS restart and had no problem. However here I use my own LDAS restart which has one more variable Tg. Therefore mk_LDASsaRestarts.F90 needs to be modified. Recompiled it. Created a symbolic link under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts: mk_restarts -> /discover/nobackup/fzeng/tmp/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/ borg01y103:~ > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > setenv ESMADIR /discover/nobackup/fzeng/tmp/Heracles-5_2/ borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > source $ESMADIR/src/g5_modules borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > limit stacksize unlimited borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > mpirun -np 96 mk_restarts/mk_LDASsaRestarts M09 50 borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 SURFLAY: 50.00000 Total Tiles: 1684725 Scaled Tiles: 1655410 (98%) Performing Sanity Check ... CatDef Tiles: 0 (00%) SrfExc Tiles: 0 (00%) SrfExc Tiles: 0 (00%) borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > ls -l bin/mk_LDASsaRestarts -rwxr-xr-x 1 fzeng g0620 27581377 2016-08-01 15:32 bin/mk_LDASsaRestarts* borg01y103:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > ls -l mk_restarts/mk_LDASsaRestarts -rwxr-xr-x 1 fzeng g0620 27582082 2016-08-01 14:37 mk_restarts/mk_LDASsaRestarts* Why do I have a newer executable mk_LDASsaRestarts in /Linux/bin than the mk_restarts/mk_LDASsaRestarts that I actually used? 5. Met with Sarith. For debugging purpose Sarith compiled my m3-16 driver with BOPT="-g" option. This will slow down the run. I should recompile the driver without this option. Tried "gmake clean" and "gmake install" in GEOSland_GridComp and GEOSlana_GridComp but got error messages. Compiled the whole driver instead. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > /bin/cp -p /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/bin/* . 6. Did a test run: /discover/nobackup/fzeng/Catchment/merra3/e0004s/input/restart > /bin/rm catchcn_internal_rst-save catchcn_internal_rst /discover/nobackup/fzeng/Catchment/merra3/e0004s/input/restart > ln -s /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts/catchcn_internal_rst ssh discover-sp3 interactive.py -A sp3 -n 112 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/merra3/e0004s/run ./lenkf.0.i (same as lenkf.0.j but without the "SBATCH" top part) It's running! /discover/nobackup/fzeng/Catchment/merra3/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2001/M01 > /bin/rm * /discover/nobackup/fzeng/Catchment/merra3/e0004s/output/SMAP_EASEv2_M09_GLOBAL/route > /bin/rm Pfaf_domain.dat /discover/nobackup/fzeng/Catchment/merra3/e0004s/run > nedit lenkf.0.j & (to add "sbatch lenkf.1.j" to the end) Submitted the full job. It's running. 7. Organized: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > /bin/rm catchcn_internal_rst_old /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > /bin/rm bin /discover/nobackup/fzeng/tmp/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > /bin/cp -p mk_LDASsaRestarts.F90 mk_LDASsaRestarts /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/. ========== 20160802: 1. The e0004s run failed due to time out. It can only finish 4 full months of simulation within 12 hours! 2. Updated higgins rain data through 20160731. 3. Sarith told me that there is another problem in my mk_LDASsaRestarts.F90, so my catchcn_internal_rst is not correct. The LDASsa_m3-15_2-CN driver produced ldas_catparam.20090101_0000z.bin file has more variables than Qing's ldas_param file (that is from the Catchment Model). Following Sarith's instructions to add the following two blocks to my mk_LDASsaRestarts.F90. ! Fanwei_change read (10) (int_get(n), n = 1,NTILES_IN) ! End_Fanwei_change ! Fanwei_change read (10) (var_get(n), n = 1,NTILES_IN) read (10) (var_get(n), n = 1,NTILES_IN) ! End_Fanwei_change Compiled it. /discover/nobackup/fzeng/tmp/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts > /bin/cp -p mk_LDASsaRestarts.F90 mk_LDASsaRestarts /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/. Then run mk_LDASsaRestarts interactively: ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X -t 02:00:00 borgu135:~ > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts borgu135:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > setenv ESMADIR /discover/nobackup/fzeng/tmp/Heracles-5_2/ borgu135:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > source $ESMADIR/src/g5_modules borgu135:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > limit stacksize unlimited borgu135:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > mpirun -np 96 mk_restarts/mk_LDASsaRestarts M09 50 borgu135:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts > mk_restarts/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 SURFLAY: 50.00000 Total Tiles: 1684725 Scaled Tiles: 342823 (20%) Performing Sanity Check ... CatDef Tiles: 0 (00%) SrfExc Tiles: 0 (00%) SrfExc Tiles: 0 (00%) 4. Set up a new e0004s run (now under Catchment/SMAP_EASEv2_M09 instead of Catchment/merra3): /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > mv e0004s e0004s_old /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > mkdir run /discover/nobackup/fzeng/Catchment/merra3/run > cp -p *M09* /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ln -s /discover/nobackup/smahanam/MERRA3/regridded_restarts Run ldsetup to set up the experiment directory: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 4 --landmodel catchCN Use the right restart file: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/input/restart > /bin/rm regridded_restarts /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/input/restart > ln -s /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/restarts/catchcn_internal_rst /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit driver_inputs_M09_GLOBAL.nml LDASsa_DEFAULT_inputs_driver.nml & In LDASsa_DEFAULT_inputs_driver.nml: set CN_DTSTEP to 5400. (Somehow the default value 14400 is not override by the 5400 in driver_inputs_M09_GLOBAL.nml, so have to set this manually here.) In driver_inputs_M09_GLOBAL.nml: make sure that the model_dtstep, out_collection_ID, ensemble average output choices, CN_DTSTEP and MERRA2_T2m are set correctly. Modified lenkf.0.j: removed "regridded_restarts" from the restart_path. changed the path to the log and out files. Did a test run interactively: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > cp lenkf.0.j lenkf.0.i /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit lenkf.0.i & ssh discover-sp3 interactive.py -A sp3 -n 112 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run ./lenkf.0.i (same as lenkf.0.j but without the "SBATCH" top part, note that the -np has to be 112 here, i.e. same as requested above) It's running! 5. Sarith updated a few routines so that we can choose to run with or without the routing model (set in drivers). Update these routines. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The following files are locally MODIFIED: - process_cn.F90 The following files need to be UPDATED for tag SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 |-------------|------------------------------------------------------------ | Newest | |-------------| File Name | Here | Tag | |------|------|------------------------------------------------------------ | | x | LDASsa_DEFAULT_inputs_driver.nml | | x | clsm_ensdrv_init_routines.F90 | - | x | process_ldas_output.pro |------|------|------------------------------------------------------------ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 LDASsa_DEFAULT_inputs_driver.nml /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 clsm_ensdrv_init_routines.F90 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 process_ldas_output.pro /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The following files are locally MODIFIED: - GNUmakefile The following files need to be UPDATED for tag SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 |-------------|------------------------------------------------------------ | Newest | |-------------| File Name | Here | Tag | |------|------|------------------------------------------------------------ | | x | clsm_ensdrv_main.F90 | | x | cnlsm_ensdrv_main.F90 |------|------|------------------------------------------------------------ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 clsm_ensdrv_main.F90 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 cnlsm_ensdrv_main.F90 6. Re-compiled the driver. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > source $ESMADIR/src/g5_modules /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > make install /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > gmake install /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/bin > /bin/cp -p * /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin ###NOTE: in the future copy the whole Linux, not just Linux/bin!#### 7. Set up a new e0004s run: NOTE: In Catchment/SMAP_EASEv2_M09/run/driver_inputs_M09_GLOBAL.nml, set "RIVER_ROUTING = 1" to run with river routing model on in the first cycle. Turn it OFF in the following cycles!! Made sure the settings in driver_inputs_M09_GLOBAL.nml, M09.bat and M09_CN.exe are correct. Run ldsetup to set up the experiment directory: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 4 --landmodel catchCN In Catchment/SMAP_EASEv2_M09/e0004s/run/driver_inputs_M09_GLOBAL.nml: make sure that the model_dtstep, out_collection_ID, ensemble average output choices, CN_DTSTEP, MERRA2_T2m and RIVER_ROUTING are set correctly. Modified lenkf.0.j: add "/" after "-restart_path ../input/restart/restarts" (so now it looks "-restart_path ../input/restart/restarts/"). Did a test run interactively: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > cp lenkf.0.j lenkf.0.i /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit lenkf.0.i & (same as lenkf.0.j but without the "SBATCH" top part) ssh discover-sp3 interactive.py -A sp3 -n 140 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run ./lenkf.0.i It's running! Add "sbatch lenkf.N+1.j" to the end of lenkf.N.j. Submitted lenkf.0.j. 8. Used the single point driver to conduct the following five experiments. nocld_TaerAODpt1 nocld_TaerAOD1 DeirC1-COD10_TaerAOD0 DeirC1-COD50_TaerAOD0 DeirC1-COD10_TaerAODpt1 remove process_cat.mod and process_cat.o before each compile: /bin/rm process_cat.mod process_cat.o 9. Processed GMAO GEOS5 seasonal forecast for August. ========== 20160803: 1. The e0004s run finished simulating 1/2001-4/2001 successfully but then stopped on 1/5/2001. It's because at the end the routing restart file for 20010501 was not written out and the model was looking for this file. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > mv e0004s e0004s_route_on Keep it in case Sarith needs it later. 2. Set up the new e0004s with route model OFF. Changed RIVER_ROUTING to 0 in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/driver_inputs_M09_GLOBAL.nml. Copied the entire Linux which not only includes bin but also others such as etc. The setting for RIVER_ROUTING is in the Linux/etc/LDASsa_DEFAULT_inputs_driver.nml. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s > /bin/rm -rf Linux/ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s > cp -pr /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux . Run ldsetup to set up the experiment directory: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 8 --landmodel catchCN In Catchment/SMAP_EASEv2_M09/e0004s/run/driver_inputs_M09_GLOBAL.nml: make sure that the model_dtstep, out_collection_ID, ensemble average output choices, CN_DTSTEP, MERRA2_T2m and RIVER_ROUTING are set correctly. Modified lenkf.0.j: add "/" after "-restart_path ../input/restart/restarts" (so now it looks "-restart_path ../input/restart/restarts/"). Did a test run interactively: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > cp lenkf.0.j lenkf.0.i /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit lenkf.0.i & (same as lenkf.0.j but without the "SBATCH" top part) ssh discover-sp3 interactive.py -A sp3 -n 140 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run ./lenkf.0.i It's running! /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2001/M01 > /bin/rm * Add "sbatch lenkf.N+1.j" to the end of lenkf.N.j. Submitted lenkf.0.j. 3. Think about how to process the output from e0004s: ~/Catchment > cp -p tile2grid_ease_spinup_monthly.f90 tile2grid_ease_spinup_monthly_m3-15.f90 ~/Catchment > mv tile2grid_ease_spinup_monthly tile2grid_ease_spinup_monthly_m3-15 Information on what varibales are written out is in clsm_ensdrv_out_routines.F90 (L1048-). Modified tile2grid_ease_spinup_monthly.f90 for post-processing the output of the new e0004s. It's now ready to be used. ~/Catchment > cp -p e0004s_spinup_monthly.ctl e0004s_spinup_monthly_m3-15.ctl ~/Catchment > nedit e0004s_spinup_monthly.ctl & (following clsm_ensdrv_out_routines.F90) Noticed that t2m and q2m are saved twice in the output. Also, rcuns (unstressed stomatal conductance) which actually is cond_save = 1/rcuns according to process_cat.F90 was saved in the previous e0004s (according to the clsm_ensdrv_out_routines.F90 in LDASsa_m3-15_2-CN, it seems that xsmr was borrowed to save this rcuns) but not in this new e0004s. Do I need to save this in the new e0004s run and in standard offline Catchment-CN run? Asked Greg and see what he says. Stopped the e0004s. It has to be re-done due to saving t2m and q2m twice. I should set out_collection_ID to 1 instead of 10. Changed out_collection_ID to 1 in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/driver_inputs_M09_GLOBAL.nml. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > cp -p M09_CN.exe M09_CN_e0004s_spinup.exe ###Need to modify M09_CN_e0004s_spinup.exe later### ###Also, out_wetness is set to .false. in clsm_ensdrv_out_routines.F90. ###Need to set it to true so that sfmc, rzmc and prmc are fraction of capacity. ========== 20160804: 1. Set out_wetness to .true. in clsm_ensdrv_out_routines.F90. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The following files are locally MODIFIED: - GEOSlana_GridComp/clsm_ensdrv_out_routines.F90 - GEOSlana_GridComp/process_cn.F90 The following files need to be UPDATED for tag SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 |-------------|------------------------------------------------------------ | Newest | |-------------| File Name | Here | Tag | |------|------|------------------------------------------------------------ | x | - | GEOSlana_GridComp/process_ldas_output.pro | - | x | GEOSlana_GridComp/build_river_network.pro | | x | GEOSlana_GridComp/process_route.F90 | - | x | GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_RouteRestarts.F90 |------|------|------------------------------------------------------------ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 GEOSlana_GridComp/build_river_network.pro /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 GEOSlana_GridComp/process_route.F90 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > cvs upd -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_RouteRestarts.F90 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components > /bin/rm GEOSlana_GridComp/process_ldas_output.pro (Sarith said it's renamed to build_river_network.pro) Recompiled: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > source $ESMADIR/src/g5_modules /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > gmake clean /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp > gmake install /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > gmake clean /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > gmake install /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > gmake clean /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > gmake install Copied the entire Linux to exec/e0004s: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s > /bin/rm -rf Linux/ /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s > cp -pr /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux . 2. Set up the new e0004s spinup run: Changed the restart_path to /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_route_on/output/ in M09_CN_e0004s_spinup.exe. Added "--submit" option to the end of the ldsetup command in Catchment/SMAP_EASEv2_M09/run/M09_setup. Run ldsetup to set up the experiment directory: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin > ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 8 --landmodel catchCN --submit In Catchment/SMAP_EASEv2_M09/e0004s/run/driver_inputs_M09_GLOBAL.nml: make sure that the model_dtstep (=450), out_collection_ID (=1), ensemble average output choices (out_select_ensavg%tile%monthly = .true.), CN_DTSTEP (=5400), MERRA2_T2m (=.true.) and RIVER_ROUTING (=0) are set correctly. Did a test run interactively: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > cp lenkf.0.j lenkf.0.i /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run > nedit lenkf.0.i & (same as lenkf.0.j but without the "SBATCH" top part) ssh discover-sp3 interactive.py -A sp3 -n 112 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/run ./lenkf.0.i It's running! /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2001/M01 > /bin/rm * Added "sbatch lenkf.1.j" to the end of lenkf.0.j and submitted lenkf.0.j. Found that all 22 jobs are in the queue. Does this look right? Checked with Sarith and Qing. It's supposed to be like this. However, I put "sbatch lenkf.1.j" at the end of lenkf.0.j. Would lenkf.1.j be submitted twice? Killed all the jobs, removed "sbatch lenkf.1.j" at the end of lenkf.0.j and submitted lenkf.0.j again, but this time there is only one job in the queue. Killed the job, added "sbatch lenkf.1.j" back to the end of lenkf.0.j and submitted lenkf.0.j again, but still there is only one job in the queue. Deleted the e0004s run directory and re-ran ldsetup to set up the experiment. Didn't change anything in lenkf.0.j. Submitted it. Now there are all 22 jobs in the queue. 3. Sarith showed me how to run build_river_network.pro: /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > nedit build_river_network.pro & changed exp_path (L17), exp_name (L22), exp_domain (L23), BCSDIR (L28) and til_file (L29) to match my e0004s run. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > module load tool/idl-8.5 /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp > idl IDL> .compile build_river_network IDL> MY_TEST The basin areas will be printed out on screen and two files below are generated: (1) /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_route_on/output/SMAP_EASEv2_M09_GLOBAL/route/Riverflow_Station_Information.nc (2) /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_route_on/output/SMAP_EASEv2_M09_GLOBAL/route/MY-basins.jpg xview /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_route_on/output/SMAP_EASEv2_M09_GLOBAL/route/MY-basins.jpg & to take a look at the global river basins. 4. CLM4.5. Read the subroutine photosynthesis. See notes on land01. 5. Found that two jobs are running simultaneously. Is this correct? Req'd Req'd Elap Job id Username Queue Name SessID NDS TSK Memory Time Use S Time -------------------- -------- -------- -------------------- ------ ----- ----- ------ ----- - ----- 9927020 fzeng compute e0004s -- 9 144 -- 12:00 R 02:32 9927021 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927022 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927023 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927024 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927025 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927026 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927027 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927028 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927029 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927030 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927031 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927032 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927033 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927034 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927035 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927036 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927037 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927038 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927039 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927040 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927041 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 9927050 fzeng compute e0004s -- 5 140 -- 12:00 R 01:57 Talked to Qing. She doesn't know what this 23th job is either (totally I have 22 job scripts). It may be some job moving files or archiving files. The output (finished 2 months so far) looks correct, so no need to worry about this now. ========== 20160805: 1. The e0004s has been running successfully so far. It takes about 9 hours to finish 8 months of simulation. It's now simulating August 2002. Don't see any job number added to the next job script when one job is finished though. Still don't understand what Qing said. However, it seems that when lenkf.N.j is done, lenkf.N+1.j is the next one to run. Sarith suspects that the ldsetup is programmed so that the SLURM sends message to the next lenkf.*j file to start once the the current job is done 2. The restart file catchcn_internal_rst is written out and saved every month. This takes a lot of disk space. Only need to keep the 12 restart files of the last year (2014 in this case) in the last spinup cycle (2001-2014). When it's simulating year N, I can delete all the restart files before year N, e.g. all the restart files for 2001 can be deleted when it is simulating 2002 like now. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > /bin/rm -rf Y2001 3. /home/fzeng/Catchment/tile2grid_ease_spinup_monthly.F90 can be used to convert the monthly output of e0004s spinup run from tile-space to half-degree grid-space for view on GrADS. Run this program like this: ~/Catchment > tile2grid_ease_spinup_monthly e0004s The GrADS control file is /home/fzeng/Catchment/e0004s_spinup_monthly.ctl. Compared the jul2001 and jul2002 output here to that of e0004s_32. For jul2002 FPAR: new e0004s (which will be e0004s_33) < e0004s_32 in >50N, eastern US, SE China, India, SE Asia, 20S-10N of Africa and northern Australia. ###Do the comparison again at the end of spinup. ========== 20160808: 1. PFTs in CLM4.5. There are 10 crop types. Checked their parameters. Looked for the land cover map for CLM4.5. This is needed to convert from ESA to CLM. See notes on land01. ========== 20160809: 1. Read the draft of Huisheng's proposal and provided comments. 2. PFTs in CLM4.5. See notes on land01. 3. Read the ESA documentation (from Sarith) to see the crop types in ESA land cover and to understand how mapping from ESA to CLM4.5 will work. ========== 20160810: 1. Read the CLM4.5 code to find (1) how are different crop types treated differently? (2) what other parameters are different for different crop types? 2. Talked to Randy. Decided to use all 25 PFTs (Don't split them to moisture and moisture+deciduous yet). Randy suggested to obtain the raw data file mksrf_24pftNT_landuse_rc2000_c121207.nc. Emailed Keith to ask how to obtain this file. ========== 20160811: 1. The e0004s spinup run finished one cycle. Check and make sure the job finished: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M01 > ls e0004s.ens0000.catchcn_ldas_rst.20150101_0000z Process the output (covert from tile-space to grid-space): ~/Catchment > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_33 cd e0004s_33/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that the cycle number (here e0004s_33) in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_33/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit The jobs are now in the queue: Req'd Req'd Elap Job id Username Queue Name SessID NDS TSK Memory Time Use S Time -------------------- -------- -------- -------------------- ------ ----- ----- ------ ----- - ----- 10087729 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087730 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087731 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087732 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087733 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087734 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087735 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087736 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087737 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087738 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087739 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087740 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087741 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087742 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087743 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087744 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 10087745 fzeng compute e0004s -- 5 140 -- 12:00 Q 00:00 It's running. Checked the log file and yes the right restart file is used. 2. Obtained the raw data file mksrf_24pftNT_landuse_rc2000_c121207.nc that was used to create the PFT maps in surfdata_0.125x0.125_mp24_simyr2000_c150114.nc and surfdata_1.9x2.5_mp24_simyr2000_c130419.nc from Keith Oleson. It's now in land01:/terra/fzeng/clm. This file has the fractions of 25 PFTs (the first one is bare) at 3minx3min spatial resolution (3600x7200), so 25 maps total. I did a quick look and it seems that the fraction of winter temperate cereal (both rainfed and irrigated, 21 and 22 below) have 0 fractions everywhere. I will look more closely on these maps later and will work with Sarith on mapping from ESA land cover to CLM4.5 after I upgrade the point driver. Here is the table showing the number and description of the PFTs. 3. Need to upgrade the single point driver to reflect Sarith's latest tag. Freshly checked out the LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 to /discover/nobackup/fzeng/tmp and take a look at what files differ from LDASsa_m3-15_2-CN: cvs co -r SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 LDASsa_m3 mv LDASsa LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 tar cvf LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3.tar LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 mv LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3.tar .. tar xf ../LDASsa_m3-15_2-CN.tar diff --brief -r LDASsa_m3-15_2-CN LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 ========== 20160812: OFF. ========== 20160815: 1. The e0004s spinup run stopped because my nobackup reached the disk space limit. The last restart file is 20090101. Removed the restart files in Y2001 through Y2007. Modified lenkf.9.j and re-submitted the job. It's running. Deleted the restart files in Y2008 as well. However, when post-processing the output, found that the output for 200812 was only partly written out although the restart files for 20090101 was written out. Since the restart files before 20090101 were all deleted, there is no way to start from 20081101 to get the output for 200812 written out. Deleted this e0004s cycle and started all over from the end of e0004s_33: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run Make sure that the cycle number (here e0004s_33) in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_33/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit ###In the future, other than checking the restart file, also need to check the output in cat/ to make sure that the output is written out completely before deleting anything!!!### 2. Need to upgrade the single point driver to reflect Sarith's latest tag. Started from Greg's last version (cn.txt: "The size and time stamp of this file is: 462888960 2015-08-14 20:41 point_driver_LDASsa_m3-15_2-CN.tar"): land01:~/point_driver_LDASsa> cp -p /land/gkwalker/point_driver_LDASsa_m3-15_2-CN.tar . 462888960 2015-08-14 20:41 point_driver_LDASsa_m3-15_2-CN.tar Added the snowdp correction. Added Eunjee's HR modification. Compiled and did a test run. The latest point driver for the LDASsa_m3-15_2-CN tag is: land01:/terra/fzeng/point_driver/point_driver_LDASsa_m3-15_2-CN.tar land01:/home/fzeng/point_driver_LDASsa/point_driver_LDASsa_m3-15_2-CN.tar /discover/nobackup/fzeng/point_driver/point_driver_LDASsa_m3-15_2-CN.tar The size and time stamp of this file is: 420341760 2016-08-15 14:52 point_driver_LDASsa_m3-15_2-CN.tar The point driver for the LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 tag is: land01:/home/fzeng/point_driver_LDASsa/point_driver_LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The upgrading process from LDASsa_m3-15_2-CN to LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 is on-going. 3. Upgrade the point driver from LDASsa_m3-15_2-CN to LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3. (1) The "**.dat" files are no longer needed. Read *** directly. (2) unified_rc3f_matrix_calc.F90 and GEOS_CatchGridComp.F90 no longer exist. (3) process_cat.F90 is replaced by process_cn.F90. (4) What else? 4. Updated higgins rain data through 20160813. ========== 20160816: 1. Upgrade the point driver from LDASsa_m3-15_2-CN to LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3. See notes land01:/home/fzeng/point_driver_LDASsa/notes/m3-15_2-CN_TO_m3-16_0_p2_CatchCatchCN 2. Modified these two files on NCCS to use Sarith's new tile: /home/fzeng/Catchment/princeton_tiles_location.f90 /home/fzeng/Catchment/extract_princeton_point_pexp_hires.f90 Ran the modified princeton_tiles_location.f90 using /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/DC0144xPC0091_DE0360xPE0180/DC0144xPC0091_DE0360xPE0180-Pfafstetter.til and pick this tile below. Central Amazon: 310316 48 45 -62.39 -2.67 0.00938 4 4 6 6 0.9983 0.0000 0.0017 0.0000 This same tile was used in the par_effect experiments for Huisheng. For comparison, this is from the old tile which has 370121 land tiles (see notes on 20160712): Central Amazon: 309736 48 45 -62.39 -2.67 0.00938 4 4 6 6 0.9983 0.0000 0.0017 0.0000 They are identical. So just use the meterology forcing and restart files from the par_effect experiments. ========== 20160817: 1. After talking with Sarith, decided to just use Sarith's single point driver. The good things about Sarith's point driver is: (1) It directly uses the LDASsa code (of course it has it's own point driver), so updating the point driver source code is easier. (2) It can use desired meterology forcing (not just Princeton) as long as the met path and met tag are specified. Therefore, stopped this effort (at least for now) of upgrading the point driver from LDASsa_m3-15_2-CN to LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3. 2. Organized nobackup: /discover/nobackup/fzeng > mkdir point_driver /discover/nobackup/fzeng > mv point_driver_LDASsa_m3-15_2-CN.tar point_driver /discover/nobackup/fzeng > mv point_driver_LDASsa_m3-15_2-CN point_driver /discover/nobackup/fzeng > mkdir offline_code /discover/nobackup/fzeng > mv LDASsa_m2-14_2_p1 offline_code /discover/nobackup/fzeng > mv LDASsa_m2-14_2_p1.tgz offline_code /discover/nobackup/fzeng > mv LDASsa_m2-14_2_p1_unified-SMAP_TestBed offline_code /discover/nobackup/fzeng > mv LDASsa_m2-14_2_p1_unified-SMAP_TestBed.tgz offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2 offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2-CN offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2_CN_1_0.tar offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2-CN_1_1 offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2-CN-photosynthesis offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2-CN.tar offline_code /discover/nobackup/fzeng > mv LDASsa_m3-15_2.tar offline_code /discover/nobackup/fzeng > mv LDASsa_m3-SMAP_Nature_v04_beta offline_code /discover/nobackup/fzeng > mv LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3.tar offline_code /discover/nobackup/fzeng > mv LDASsa_m3-16_0_p2 offline_code /discover/nobackup/fzeng > mv LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3_keep offline_code ###NOTE###: DO NOT move Catchment or LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 in /discover/nobackup/fzeng to anywhere else!! My e0004s and Sarith's runs are using executable or restart files from these directories. /discover/nobackup/fzeng > mkdir geos5_code /discover/nobackup/fzeng > mv Ganymed-4_1 geos5_code /discover/nobackup/fzeng > mv Ganymed-4_1-CN geos5_code /discover/nobackup/fzeng > mv Ganymed-4_1.tgz geos5_code /discover/nobackup/fzeng > mv Heracles-4_0 geos5_code /discover/nobackup/fzeng > mv Heracles-4_0-CN geos5_code /discover/nobackup/fzeng > mv Heracles-4_0-CN-photosynthesis geos5_code /discover/nobackup/fzeng > mv Heracles-4_0-CN.tar geos5_code /discover/nobackup/fzeng > mv Heracles-5_2 geos5_code /discover/nobackup/fzeng > mv Heracles-5_2-catchcn geos5_code /discover/nobackup/fzeng > mkdir geos5_sims /discover/nobackup/fzeng > mv h0000 geos5_sims /discover/nobackup/fzeng > mv h0006p geos5_sims /discover/nobackup/fzeng > mv h4-0000 geos5_sims /discover/nobackup/fzeng > mv h4cn-0000 geos5_sims /discover/nobackup/fzeng > mv h4cn-0001 geos5_sims /discover/nobackup/fzeng > mv h4cn-0001-cont geos5_sims /discover/nobackup/fzeng > mv h52_test geos5_sims /discover/nobackup/fzeng > mv h52_test2 geos5_sims /discover/nobackup/fzeng > mv h52_testHRv2 geos5_sims /discover/nobackup/fzeng > mv hcn0000 geos5_sims /discover/nobackup/fzeng > mv cn0045 geos5_sims /discover/nobackup/fzeng > mv cn0045_monthly.gdat geos5_sims /discover/nobackup/fzeng > mkdir geos5_forecast /discover/nobackup/fzeng > mv pt_anom* w2_percentile* geos5_forecast /discover/nobackup/fzeng > mkdir -p projects/others /discover/nobackup/fzeng > mv converge* compare* diff_* fire_effect* verify* lai_npp* projects/others /discover/nobackup/fzeng > mv p0007s* rzmc_smap* projects/others 3. Created a whitelist for using Sarith's point driver following /discover/nobackup/fzeng/notes/point_driver_from_Sarith. /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/run/DE_00720x00360_PE_0720x0360/whitelist_10tiles_split_types_clm4.txt (1) /discover/nobackup/fzeng/point_driver > mkdir -p clm4_to_45/clm4/run/DE_00720x00360_PE_0720x0360 (2) From /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/DE_00720x00360_PE_0720x0360/clsm/CLM_veg_typs_fracs and /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/DE_00720x00360_PE_0720x0360/DE_00720x00360_PE_0720x0360.til, pick a few tiles that have split types. Record the tile IDs in the whitelist. ! 10 broadleaf deciduous temperate shrub [moisture + deciduous] ! 11 broadleaf deciduous temperate shrub [moisture stress only] ! 14 cool c3 grass [moisture + deciduous] ! 15 cool c3 grass [moisture stress only] ! 16 warm c4 grass [moisture + deciduous] ! 17 warm c4 grass [moisture stress only] ! 18 crop [moisture + deciduous] ! 19 crop [moisture stress only] 100 277659 149.2500 -36.2500 659 108 1.000000000000 (from tile file) 1224 277659 7 7 10 11 60.59 0.00 16.75 22.66 8 11 60.59 39.41 (from PFT file, the first number is tile id) 100 263660 -67.7500 -32.7500 225 115 1.000000000000 1790 263660 10 11 16 17 5.29 65.21 2.21 27.29 11 15 70.50 29.50 100 181079 -106.2500 34.2500 148 249 1.000000000000 30475 181079 10 11 14 15 18.72 64.49 3.78 13.01 11 14 83.22 16.78 100 209535 36.2500 35.2500 433 251 1.000000000000 31240 209535 18 19 10 11 30.82 64.01 1.68 3.49 16 11 94.83 5.17 100 59843 128.2500 35.2500 617 251 0.991111099720 31413 59843 7 7 14 15 56.59 0.00 14.11 29.31 8 14 56.59 43.41 100 62003 117.7500 37.2500 596 255 1.000000000000 32761 62003 18 19 16 17 52.26 47.28 0.24 0.22 16 15 99.55 0.45 100 7148 90.2500 38.2500 541 257 0.999722242355 33390 7148 14 15 18 19 58.26 34.96 4.24 2.54 14 16 93.22 6.78 100 210185 30.2500 40.2500 421 261 1.000000000000 34664 210185 10 11 1 1 42.37 8.99 48.65 0.00 11 2 51.35 48.65 100 7409 66.2500 40.2500 493 261 0.998055577278 34733 7409 14 15 18 19 46.96 9.96 35.54 7.54 14 16 56.92 43.08 100 61796 106.7500 40.2500 574 261 1.000000000000 34814 61796 18 19 16 17 42.99 9.12 39.51 8.38 16 15 52.11 47.89 ========== 20160818: 1. Obtained and modified Sarith's point driver following /discover/nobackup/fzeng/notes/point_driver_from_Sarith. /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS > cp -pr /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 . /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The following files are locally MODIFIED: - Applications/LDAS_App/GNUmakefile - Components/GEOSlana_GridComp/build_river_network.pro - Components/GEOSlana_GridComp/clsm_ensdrv_out_routines.F90 - Components/GEOSlana_GridComp/process_cn.F90 The following files need to be UPDATED for tag SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 |-------------|------------------------------------------------------------ | Newest | |-------------| File Name | Here | Tag | |------|------|------------------------------------------------------------ | | x | Components/GEOSlana_GridComp/process_route.F90 | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/GNUmakefile | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/GNUmakefile_openmp | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/GNUmakefile | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_CatchRestarts.F90 | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_Restarts | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared/GNUmakefile |------|------|------------------------------------------------------------ Updated the routines in the above table. Sarith updated a few routines that he has not checked in yet. Copy these files from his sand box: /gpfsm/dnb02/smahanam/ROUTING_MODEL/science/SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/LDASsa/src/Components/GEOSlana_GridComp > cp -p GNUmakefile_parallel process_cn.F90 LSM_Calib.rc lsm_calib_routines.F90 /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/. /gpfsm/dnb02/smahanam/ROUTING_MODEL/science/SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/LDASsa/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared > cp -p GNUmakefile update_model_paras.F90 /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared/. /gpfsm/dnb02/smahanam/ROUTING_MODEL/science/SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/LDASsa/src/Applications/LDAS_App > cp -p GNUmakefile ldsetup lsm_calib_driver.F90 /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/. 2. Compile: setenv ESMADIR /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 source $ESMADIR/src/g5_modules then "gmake clean" and "gmake install" in this order: (1) src/Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared (2) src/Components/GEOSlana_GridComp (3) src/Applications/LDAS_App /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/get_forcing > cp -pr /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux . 3. Run: /discover/nobackup/fzeng/Catchment/merra3/run > cp -p D0.5.bat D0.5_cn.exe driver_inputs_0.5D_GLOBAL.nml /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/run/. Renamed driver_inputs_0.5D_GLOBAL.nml to driver_inputs_0.5D_10tiles.nml and modified the three files as needed. Under /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App: (1) source /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules (2) made sure the part below in ldsetup is uncommented: for outtyp in ['indata']: for y4m2 in y4m2_list: mydir = _getDirName_noens(outtyp, y4m2) self._mkdir_p(mydir) (3) ran ldsetup to set up the run: ./ldsetup setup /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims/ /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/run/D0.5_cn.exe /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/run/D0.5.bat --runmodel --monthsperjob 120 --landmodel catchCN input/restart > ln -s /discover/nobackup/smahanam/MERRA3/regridded_restarts/0.5D/catchcn_internal_rst input/restart > /bin/rm regridded_restarts run > nedit lenkf.0.j & (to delete regridded_restarts in "restart_path") Did an interactive run to make sure it runs. add "qsub lenkf.1.j" at the end of "lenkf.0.j"; add "qsub lenkf.2.j" at the end of "lenkf.1.j"; add "qsub lenkf.3.j" at the end of "lenkf.2.j". Sbatch lenkf.0.j. ========== 20160819: 1. /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > cp -p /gpfsm/dnb02/smahanam/ROUTING_MODEL/science/SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/LDASsa/src/Applications/LDAS_App/lsm_calib_driver.F90 . Since we ran ldsetup under Application/LDAS_App to set up the first run yesterday, the executable used in the first run is under /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/bin. If I do gmake clean and gmake install to compile lsm_calib_driver.F90 under Application/LDAS_App, the ongoing first run will be affected. 2. /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src > ~ltakacs/cvstools/cvscmp SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 The following files are locally MODIFIED: - Components/GEOSlana_GridComp/GNUmakefile_parallel - Components/GEOSlana_GridComp/build_river_network.pro - Components/GEOSlana_GridComp/clsm_ensdrv_out_routines.F90 - Components/GEOSlana_GridComp/tile_coord.F90 - Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared/GNUmakefile The following files are CONFLICTING: - Applications/LDAS_App/GNUmakefile - Applications/LDAS_App/ldsetup - Components/GEOSlana_GridComp/process_cn.F90 The following files need to be UPDATED for tag SM-LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 |-------------|------------------------------------------------------------ | Newest | |-------------| File Name | Here | Tag | |------|------|------------------------------------------------------------ | | x | Applications/LDAS_App/lsm_calib_driver.F90 | | x | Components/GEOSlana_GridComp/LSM_Calib.rc | | x | Components/GEOSlana_GridComp/lsm_calib_routines.F90 | | x | Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared/update_model_paras.F90 |------|------|------------------------------------------------------------ Updated LSM_Calib.rc and update_model_paras.F90. For some reason, couldn't update lsm_calib_driver.F90 and lsm_calib_routines.F90. See below: cvs update: move away `Applications/LDAS_App/lsm_calib_driver.F90'; it is in the way C Applications/LDAS_App/lsm_calib_driver.F90 cvs update: move away `Components/GEOSlana_GridComp/lsm_calib_routines.F90'; it is in the way C Components/GEOSlana_GridComp/lsm_calib_routines.F90 Deleted these files from my sandbox and updated fresh. 3. Killed the first run job. It finished 20 years (1980-2000) of simulation. Keep a copy of Linux in exec/get_forcing and make build point to it. Currently /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims/get_forcing/build -> /gpfsm/dnb31/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/get_forcing > /bin/rm -rf Linux/ /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/get_forcing > cp -pr /gpfsm/dnb31/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/ . /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims/get_forcing > /bin/rm build /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims/get_forcing > ln -s /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/get_forcing build 4. Modified the vegetation fractions in lsm_calib_driver.F90 (see /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/lsm_calib_driver.F90.NOsplitting). Under /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App: gmake clean and gmake install to compile /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/NOsplitting > cp -pr /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/ . ssh discover-sp3 interactive.py -A sp3 -n 1 -a g0620 -X --debug cd /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/exec/NOsplitting/Linux/bin ./lsm_calib_driver.x ./lsm_calib_driver.x: error while loading shared libraries: libmpi_usempif08.so.11: cannot open shared object file: No such file or directory Same thing when ran it under /LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App/ ========== 20160822: 1. The e0004s spinup run finished another cycle. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01/ -rw-r--r-- 1 fzeng g0620 7251104247 2016-08-22 03:18 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali10:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 -rw-r--r-- 1 fzeng g0620 539112640 2016-08-22 03:18 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_34 cd e0004s_34/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z Make sure that the cycle number (here e0004s_34) in the restart path in M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_34/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit 2. Talked to Sarith: I couldn't run lsm_calib_driver.x probably because I didn't source g5_modules. Sarith helped me fix some bugs. Now it runs. But need some modifications for my purpose (e.g. add write statement to write output to file). If the 10 tiles do not show the issue in LAI after combining the splitted types, do a similar run but on regional scale (e.g. conus, see Sarith's M36_conus run). (1) In the namelist: minlon = -125 maxlon = -65 minlat = 25 maxlat = 50 (2) Modify the fvg in process_cn.F90. 3. Sarith's suggestions on upgrading the CN model from CLM4 to CLM4.5: Like what he did for GEOScatchCN_GridComp and GEOScatch_GridComp, use directives to control whether to use CLM4 or CLM4.5. Keep the original CLM4 and CLM4.5 code in separate directories, and make as few modifications as possible. Write CN4.5_DiverMod.F90. In process_cn.F90, use something like this: if CLM4 call CN_Driver() else call CN4.5_Driver() endif 4. /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims > mkdir ityp_not_splitting_10tiles /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App: nedit lsm_calib_driver.F90 & source /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules gmake install ./lsm_calib_driver.x ========== 20160823: 1. Sarith modified these routines so that I can use one executable to run with or without splitting the vegetation types on selected tiles: - Applications/LDAS_App/lsm_calib_driver.F90 - Components/GEOSlana_GridComp/LSM_Calib.rc - Components/GEOSlana_GridComp/lsm_calib_routines.F90 - Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/CNPhenologyMod.F90 - Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared/update_model_paras.F90 To compile, gmake install in this order: setenv ESMADIR /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 source $ESMADIR/src/g5_modules (1) Components/GEOSsurface_GridComp/GEOSland_GridComp/Shared (2) Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp (3) Components/GEOSlana_GridComp (4) Applications/LDAS_App To run: Testing by Sarith: mkdir /discover/nobackup/fzeng/testing Applications/LDAS_App > /bin/cp ../../Components/GEOSlana_GridComp/LSM_Calib.rc /discover/nobackup/fzeng/testing/. cd /discover/nobackup/fzeng/testing ln -s $ESMADIR/Linux/bin/lsm_calib_driver.x nohup ./lsm_calib_driver.x What I did: Under /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/sims: cp -p /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp/LSM_Calib.rc . ln -s /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/Linux/bin/lsm_calib_driver.x setenv ESMADIR /discover/nobackup/fzeng/point_driver/clm4_to_45/clm4/LDAS/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 source $ESMADIR/src/g5_modules For the control run: Comment out these two lines in LSM_Calib.rc L18: "NVARS : 1" L42: "SPLIT : 0 0 0" nohup ./lsm_calib_driver.x For the run without splitting vegetation types: Uncomment these two lines in LSM_Calib.rc L18: "NVARS : 1" L42: "SPLIT : 0 0 0" nohup ./lsm_calib_driver.x 2. Sarith modified these routines so that I can use one executable to run with or without splitting the vegetation types on regional scale: Components/GEOSlana_GridComp/process_cn.F90 Applications/LDAS_App/cnlsm_ensdrv_main.F90 Compiled. NOTE: When running ldsetup to setup the experiments, don't use the "--submit" option! After creating the experiment/run directory where the job files are located, need to copy the LSM_Calib.rc files to this directory and make sure the setting is correct. Then sbatch the first job files (add "sbatch lenkf.N+1.j" to lenkf.N.j before submitting the first job). Tomorrow: Copy Linux bin and set up two regional runs: (1) control (with splitting types) and (2) without splitting types ========== 20160824: 1. /discover/nobackup/fzeng/clm4-to-clm4.5 > mkdir catchcn4 /discover/nobackup/fzeng/clm4-to-clm4.5 > /bin/cp -pr ../LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/* catchcn4/. 2. Checked and made sure that the modifications made to CNPhenologyMod.F90 to turn off splitting are correct. 3. Met with Randy and Eunjee about refining the CO2 project. 4. Talked to people in ORNL about the FLUXNET data. ========== 20160825: 1. For the NOAA CO2 project, interpolated year-specific monthly mean CO2 to daily CO2, and interpolated monthly means (i.e. mean seasonal cycle) CO2 to daily CO2. See /discover/nobackup/fzeng/noaaCO2/code/monthly2monthly_interpolated_tile_CO2.f90 and /discover/nobackup/fzeng/noaaCO2/code/monthlymean2monthlymean_interpolated_tile_CO2.f90 ========== 20160826: 1. Verified the daily CO2 interpolated from year-specific monthly mean CO2 and monthly means (i.e. mean seasonal cycle) CO2. See /discover/nobackup/fzeng/noaaCO2/code/comparison_diff_timescale.gs 2. Met with Randy, Yehui and Sarith: Set up a run parallel to my SMAP run and save restart files for Yehui. Yehui will be doing forecast with the CatchCN at 0.5 degree (C180). We need to provide him with the land states for the corresponding land tile distribution. Initial of the forecast is July 1 for each year in 1981-present. The forecast is only ~6 weeks. Meteorology forcing for this offline run: exactly the same as my SMAP run. What I need to do: (1) Regrid my SMAP restart file to this C180 (CF0180x6C_DE1440xPE0720) tile; (2) Modify offline code so that it can run on this tile and it writes out restart files at 21z instead of the traditional 00z. 3. Regrid my SMAP restart file to the C180 (CF0180x6C_DE1440xPE0720) tile: /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C: cp -p /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/mk_LDASsaRestarts.F90 . mkdir restarts Sarith showed me how to modify mk_LDASsaRestarts.F90. To compile, copy the modified mk_LDASsaRestarts.F90 to /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts (This Heracles-5_2 has to be already compiled before.) Under this mk_restarts directory: setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2/ source $ESMADIR/src/g5_modules gmake install Run mk_LDASsaRestarts interactively following the procedures in mk_LDASsaRestarts.F90: ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X -t 12:00:00 setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2 source $ESMADIR/src/g5_modules limit stacksize unlimited cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/restarts (removed everything under this directory if any) ln -s $ESMADIR/Linux/bin mkdir -p OutData1/ OutData2/ setenv BCSDIR /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/CF0180x6C_DE1440xPE0720/ ln -s $BCSDIR/CF0180x6C_DE1440xPE0720-Pfafstetter.til OutData1/OutTileFile ln -s $BCSDIR/CF0180x6C_DE1440xPE0720-Pfafstetter.til OutData2/OutTileFile ln -s $BCSDIR/clsm OutData2/clsm mpirun -np 96 bin/mk_LDASsaRestarts M09 50 bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 Failed (dw = 0.5): 1st time: Time out. 2nd time: APPLICATION TERMINATED WITH THE EXIT STRING: Bus error (signal 7). ========== 20160829: 1. The e0004s spinup run finished another cycle. dali10:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l Y2015/M01 total 7064800 -rw-r--r-- 1 fzeng g0620 7251104247 2016-08-28 23:15 e0004s.ens0000.catchcn_ldas_rst.20150101_0000z dali10:/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000 > ls -l ../../cat/ens_avg/Y2014/M12 total 526496 -rw-r--r-- 1 fzeng g0620 539112640 2016-08-28 23:15 e0004s.ens_avg.ldas_tile_monthly_out.201412.bin Process the output (covert from tile-space to grid-space): ~/Catchment > tile2grid_ease_spinup_monthly e0004s Check the output on GrADS. Looks good. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mv e0004s e0004s_35 cd e0004s_35/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y200[1-9] Y201[0-4] mkdir -p Y2001/M01 cd Y2001/M01 ln -s ../../Y2015/M01/e0004s.ens0000.catchcn_ldas_rst.20150101_0000z e0004s.ens0000.catchcn_ldas_rst.20010101_0000z Make sure that the cycle number (here e0004s_35) in the restart path in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe is correct (here "restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_35/output/"). cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_spinup.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09.bat --runmodel --monthsperjob 10 --landmodel catchCN --submit 2. Updated higgins rain data through 20160828. 3. For spinning up the model using 3hrly CO2, modified process_cat.F90 for use with 3hrly NOAA CO2 data. Now borrow sifl (instead of sai) to save co2v. /discover/nobackup/fzeng/Catchment/M2n5P > mkdir -p m0001CTco2_3hrly/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_3hrly > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_3hrly > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_3hrly > cp -p ../m0001CTco2_daily/lenkf.j . dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_3hrly > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_3hrly > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin ~/Catchment: grid_restore_M2n5P_3hrly 0001 CTco2_3hrly (for 3hrly output) grid_restore_M2n5P_daily 0001 CTco2_3hrly (for daily output) grid_restore_M2n5P 0001 CTco2_3hrly (for monthly output) Checked the CO2 output (which is sifl in the output) in GrADS. Looks correct. 4. Run mk_LDASsaRestarts.F90 using totalview for debugging: /discover/nobackup/fzeng/geos5_code/Heracles-5_2/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts: gmake clean gmake install BOPT=g ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2 source $ESMADIR/src/g5_modules limit stacksize unlimited cd /discover/nobackup/fzeng/Catchment/M2n5P/CF0180x6C/restarts (removed everything under this directory if any) ln -s $ESMADIR/Linux/bin mkdir -p OutData1/ OutData2/ setenv BCSDIR /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/CF0180x6C_DE1440xPE0720/ ln -s $BCSDIR/CF0180x6C_DE1440xPE0720-Pfafstetter.til OutData1/OutTileFile ln -s $BCSDIR/CF0180x6C_DE1440xPE0720-Pfafstetter.til OutData2/OutTileFile ln -s $BCSDIR/clsm OutData2/clsm module load tool/tview-8.15.0-15 totalview bin/mk_LDASsaRestarts processes -> startup parameter: Parallel: choose "Intel MPI_Hydra", tasks = 1, nodes = 1 Auguments: put "M09 50" in command-line argument Make it run to L997 "TILES : do n = low_ind (myid + 1), upp_ind (myid + 1)", error message: 5. Met with Sarith about mk_LDASsaRestarts.F90: Sarith tried: Run mk_LDASsaRestarts to write HYD and BCS first. Comment out "call regrid_hyd_vars" and "call regrid_carbon_vars" Compile with BOPT=g (already on discover) interactive.py -A sp3 -n 96 -a g0620 -X --debug setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Heracles-5_2 source $ESMADIR/src/g5_modules limit stacksize unlimited module load tool/allinea_tools_6.0.4 ddt bin/mk_LDASsaRestarts & Arguments: M09 50 No. of process: 28 (maximum is 28) Then run. He found two problems in mk_LDASsaRestarts.F90: (1) The tile order in the .til file is different from that in the xx.catchcn_ldas_rst.xx file (the latter is re-ordered). (2) The fvg is all 0 in the xx.catchcn_ldas_rst.xx file. The offline reads ity and fvg from bcs files instead of this restart file. catchcn_internal_rst in OutData1: without bcs catchcn_internal_rst in OutData2: with bcs ========== 20160830: 1. Set up a test run for use with daily CO2 interpolated from monthly CO2 (labeled monthly interpolated). Now borrow sifl (instead of sai) to save co2v. /discover/nobackup/fzeng/Catchment/M2n5P > mkdir -p m0001CTco2_monthly_interpolated/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthly_interpolated > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthly_interpolated > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthly_interpolated > cp -p ../m0001CTco2_3hrly/lenkf.j . dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthly_interpolated > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthly_interpolated > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin ~/Catchment: grid_restore_M2n5P_daily 0001 CTco2_monthly_interpolated grid_restore_M2n5P 0001 CTco2_monthly_interpolated Checked the CO2 (which is sifl in the output), sai, lai, gpp and npp in GrADS. Looks correct. 2. Set up a test run for use with daily CO2 interpolated from monthly mean CO2 (labeled monthly mean interpolated). Now borrow sifl (instead of sai) to save co2v. /discover/nobackup/fzeng/Catchment/M2n5P > mkdir -p m0001CTco2_monthlymean_interpolated/RUN/rs/ens0000/Y2001/M01 /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthlymean_interpolated > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/CN_restart . /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthlymean_interpolated > cp -p /discover/nobackup/elee15/offline/sims/rst_agu_prelim/mM2.n5P.HRv2.tCO2.ens0000.catch_ldas_rst.20160101_0000z.bin RUN/rs/ens0000/Y2001/M01/m0001.ens0000.catch_ldas_rst.20010101_0000z.bin /discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthlymean_interpolated > cp -p ../m0001CTco2_monthly_interpolated/lenkf.j . dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthlymean_interpolated > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-05-21 15:09 CN_restart dali17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001CTco2_monthlymean_interpolated > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-05-21 15:09 m0001.ens0000.catch_ldas_rst.20010101_0000z.bin ~/Catchment: grid_restore_M2n5P_daily 0001 CTco2_monthlymean_interpolated grid_restore_M2n5P 0001 CTco2_monthlymean_interpolated Checked the CO2 (which is sifl in the output), sai, lai, gpp and npp in GrADS. Looks correct. 3. Think about how to pre-process the precipitation data for the reducing time scale project. ========== 20160831: 1. Think about how to pre-process the precipitation data for the reducing time scale project.