========== 20180501: 1. Talked to Qing. Following her suggestion, I was able to resolve the issue -- need to release some space from my nobackup (which is almost full) because MATLAB needs to write some big files to my nobackup (symbolic link from Home directory). Calculated 1980-2016 mean global annual GPP and NEE of these runs. clm4_DE720: 35, 36, 37, 38, 39, 40, 41, 42 clm4_IPAR_DE720: 40, 41, 42, 43, 44, 45, 46, 47 clm4_IPAR_DReqDF_DE720: 39, 40, 41, 42, 43, 44, 45 clm4_pIPAR_DE720: 27, 28, 29, 30, 31, 32, 33 clm4_pTair_DE720: 27, 28, 29, 30, 31, 32, 33 2. April report. 3. Processed May forecast. ========== 20180502: 1. Organized files: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp mkdir gmax_ctrl mkdir gmax_exp01 mv process_cn.F90.gmax_ctrl gmax_ctrl/process_cn.F90 mv process_cn.F90.gmax_exp gmax_exp01/process_cn.F90 mv ../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/compute_rc.F90.gmax_exp01 gmax_exp01/compute_rc.F90 2. Step #1 for creating a 20110331_21z CF90 restart file for Eunjee: set up a SMAP M09 run starting from the 20110101_00z restart from e0004s_transientCO2_05 and run through 20110331_21z. By doing so there will be a reasonable sink (NEE) since the carbon pools went through pre-industrial equilibrium and transient CO2 period. To use the same executable as e0004s_transientCO2_05, after setting up the run, will need to do the following 3 job segments (a) run from 20110101_00z through 20110301_00z, save monthly output (b) run from 20110301_00z through 20110331_00z, save daily output (c) run from 20110331_00z through 20110331_21z, save 3-hourly output Following the notes on 20170427: (1) Prepare files for experiment setup: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run cp M09_CN_e0004s_transientCO2.exe M09_CN_e0004s_transientCO2_2011.exe nedit M09_CN_e0004s_transientCO2_2011.exe & exp_id = e0004s_transientCO2_2011 start_time = 2011-01-01-00-00-00 end_time = 2011-03-01-00-00-00 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_id = e0004s_transientCO2 (2) Run ldsetup: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_2011.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN (NOTE: don't use the "--submit" option because we need to modify the job scripts, see below) (3) Check the executable, restart file and create year_co2.txt: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011 ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x* [Correct!] ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng g0620 80 2018-05-02 11:23 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4) Create year_co2.txt: cd run echo 2011 > year_co2.txt cat year_co2.txt (to make sure the year of CO2 is correct) 2011 [Correct!] (5) Check lenkf.0.j and namelist: lenkf.0.j: #SBATCH --time=03:00:00 #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_log.20110101_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_out.20110101_0000.txt -start_year 2011 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 3 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../input/restart/output \ [Need modification for lenkf.1.j and lenkf.2.j] -restart_id e0004s_transientCO2 \ [Need modification for lenkf.1.j and lenkf.2.j] -driver_inputs_file driver_inputs_M09_GLOBAL.nml [Need modification for lenkf.1.j and lenkf.2.j] Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* ./lenkf.0.i Crashed: reading Catchment model parameters Reading vegetation class and, if available, height Reading from: /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs Reading from: /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs LDAS ERROR (3000) from read_cat_param: unknown number of columns in /mosaic_veg _typs_fracs -------------------------------------------------------------------------- MPI_ABORT was invoked on rank 0 in communicator MPI_COMM_WORLD with errorcode 3000. NOTE: invoking MPI_ABORT causes Open MPI to kill all MPI processes. You may or may not see output from other processes, depending on exactly when Open MPI kills them. -------------------------------------------------------------------------- forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libirc.so 00002AAAAB590961 Unknown Unknown Unknown libirc.so 00002AAAAB58F0B7 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E3362 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2E31B6 Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2D1A0C Unknown Unknown Unknown libmpi_usempif08. 00002AAAAC2B2B73 Unknown Unknown Unknown libc.so.6 00002AAAABA16910 Unknown Unknown Unknown mca_btl_vader.so 00002AAAB2844C46 Unknown Unknown Unknown libopen-pal.so.13 00002AAAAD1FC88B Unknown Unknown Unknown mca_pml_ob1.so 00002AAAB3DC51B0 Unknown Unknown Unknown libmpi.so.12 00002AAAACA9B9A1 Unknown Unknown Unknown libmpi_mpifh.so.1 00002AAAAC80C582 Unknown Unknown Unknown LDASsaCN_mpi.x 00000000004A0194 MAIN__ 1211 cnlsm_ensdrv_main.F90 ls -l /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs -r--r--r-- 1 ltakacs g0609 106137675 2017-09-01 22:13 /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE//SMAP_EASEv2_M09/clsm//mosaic_veg_typs_fracs e0004s_wet2 was set up on 20171122 but it didn't have this problem. driver_inputs_M09_GLOBAL.nml is identical between e0004s_transientCO2_2011 and e0004s_wet2 (the last M09 simulation I did). Is there any code difference between e0004s_transientCO2 and e0004s_wet2 related to this? I don't see any modification to e0004s_wet2 is related to number of columns in /mosaic_veg_typs_fracs. Why did this issue not happen to e0004s_wet2? The executable for e0004s_transientCO2 was created on 2017-04-27, before the mosaic_veg_typs_fracs file was modified the last time. Any change to the driver after this mosaic_veg_typs_fracs file was modified on 2017-09-01? /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Applications/LDAS_App > ls -l -rw-r--r-- 1 fzeng g0620 143907 2017-11-02 10:25 cnlsm_ensdrv_main.F90 -rw-r--r-- 1 fzeng g0620 143907 2017-10-24 14:14 cnlsm_ensdrv_main.F90.1.1.2.7 -rw-r--r-- 1 fzeng g0620 143907 2016-09-02 15:40 cnlsm_ensdrv_main.F90.restore But the three are identical. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp: -rw-r--r-- 1 fzeng g0620 147751 2017-09-06 16:13 clsm_ensdrv_init_routines.F90.restore This file clsm_ensdrv_init_routines.F90.restore was updated to fit the updated mosaic_veg_typs_fracs file. That's why e0004s_wet2 had no problem. Need to use this clsm_ensdrv_init_routines.F90.restore to recompile for e0004s_transientCO2 to be able to run with the updated mosaic_veg_typs_fracs file. cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec mv e0004s_transientCO2 e0004s_transientCO2_20170427 cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/Components/GEOSlana_GridComp xdiff clsm_ensdrv_init_routines.F90 clsm_ensdrv_init_routines.F90.restore & [identical, GOOD] cd e0004s_transientCO2 cp clsm_ensdrv_out_routines.F90 ../. cp process_cn.F90 ../. cp compute_rc.F90 ../../GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp/. Compiled: setenv ESMADIR /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/ source $ESMADIR/src/g5_modules "gmake install" in GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp, GEOSlana_GridComp and then Applications/LDAS_App. /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3 > /bin/cp -pr Linux exec/e0004s_transientCO2/. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011 ls -l build/Linux/bin/LDASsaCN_mpi.x -rwxr-xr-x 1 fzeng g0620 90093587 2018-05-02 14:04 build/Linux/bin/LDASsaCN_mpi.x* Tried to run interactively again. It works now, using the right restart file and CO2 year and concentration. using LDASsa off-line restart file(s) restart and experiment domains are identical Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2011/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20110101_0000z Writing restart (or incr) file ../output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2011/M01/e0004s_transientCO2_2011.ens0000.catchcn_ldas_rst.20110101_0000z Transient period:: forcing year, CO2 year, EEA global average CO2 (ppm), co2 sc alar: 2011 2011 389.7097 0.9995378 Stopped the interactive run and submitted the full job. cat year_co2.txt 2011 rm *ts.txt qsub lenkf.0.j Restored the three files: clsm_ensdrv_out_routines.F90, process_cn.F90 and compute_rc.F90 ###Compare the monthly output to that in e0004s_transientCO2_05 and see if they are identical. ========== 20180503: 1. Continued working on slides for meeting with Randy and Eunjee this afternoon. 2. Helped Melanie with extracting burn and closs data from the clm4.5_DE720_52 output. 3. Met with Randy and Eunjee. 4. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180504: 1. The e0004s_transientCO2_2011 stopped at 7:13pm last night due to time out. date_time_new 20110112_124500z It used to be able to finish 10 months of simulation within 12 hours, but now it couldn't finish 1 month of simulation within 3 hours! Have to start from 20110101 all over again! cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/* rm *ts.txt cat year_co2.txt 2011 [correct] nedit lenkf.0.j & #SBATCH --time=12:00:00 [request for 12 hours run time for 2 months of simulation!] qsub lenkf.0.j 2. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180507: 1. The e0004s_transientCO2_2011 stopped during the weekend due to time out. It was simulating 20110216 when it stopped. Why does it take so long? Checked the output. Found that it wrote out daily output instead of monthly output. It's because driver_inputs_M09_GLOBAL.nml was set to write out daily output for e0004s_wet2 the last time it's used, and I forgot to change it back to monthly!! Corrected this!! Re-do the run: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/output/SMAP_EASEv2_M09_GLOBAL/cat/ens_avg/Y2011 /bin/rm M01/* /bin/rm M02/* cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/* /bin/rm *ts.txt cat year_co2.txt 2011 [Correct] qsub lenkf.0.j It's still very very slow. After 28 minutes it's still simulating date_time_new 20110102_184500z. It won't be able to finish 2 months of simulation within 12 hours. Had to cancel the run, modify lenkf.0.j to just run for 1 month within 12 hours, and re-submit lenkf.0.j. 2. Why is this e0004s_transientCO2_2011 so slow? These are the files modified after 20170427 when the executable was generated for the e0004s_transientCO2_01throught05. -rw-r--r-- 1 fzeng g0620 80553 2017-08-01 15:13 tile_coord.F90 -rw-r--r-- 1 fzeng g0620 147751 2017-09-06 16:13 clsm_ensdrv_init_routines.F90.restore -rw-r--r-- 1 fzeng g0620 19661 2017-11-02 10:24 ens_driver_routines.F90 -rw-r--r-- 1 fzeng g0620 147751 2017-12-19 14:53 clsm_ensdrv_init_routines.F90 -rw-r--r-- 1 fzeng g0620 6549 2017-12-19 14:54 clsm_ensdrv_glob_param.F90 -rw-r--r-- 1 fzeng g0620 71502 2018-03-02 09:13 process_cn.F90.restore -rw-r--r-- 1 fzeng g0620 89026 2018-05-02 14:27 clsm_ensdrv_out_routines.F90 -rw-r--r-- 1 fzeng g0620 71502 2018-05-02 14:28 process_cn.F90 (1) tile_coord.F90 was modified for CF0090x6C. The modification is additional information, should not affect this e0004s_transientCO2_2011 run. (2) clsm_ensdrv_init_routines.F90.restore (and clsm_ensdrv_init_routines.F90) was updated to fit the updated mosaic_veg_typs_fracs file. Since the e0004s_wet2 run used it but no obvious slowdown was found, the update should not be causing the slowdown in e0004s_transientCO2_2011. (3) ens_driver_routines.F90 is identical to ens_driver_routines.F90.restore (2016-04-14 15:26). It has a newer time stamp because it was modified for the M2n5P_m0002CTco2 runs but later restored. -rw-r--r-- 1 fzeng g0620 19661 2016-04-14 15:26 ens_driver_routines.F90.restore (4) clsm_ensdrv_glob_param.F90 is identical to clsm_ensdrv_glob_param.F90.restore (2016-03-29 16:28). It has a newer time stamp because it was modified for the e0004s_wet2 run but later restored. -rw-r--r-- 1 fzeng g0620 6549 2016-03-29 16:28 clsm_ensdrv_glob_param.F90.restore (5) process_cn.F90.restore: The change made is likely addition of "if(master_proc)" before "print *, '***LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3*** DTCN and FireFac : ',dtcn,firefac" (6) clsm_ensdrv_out_routines.F90: same as clsm_ensdrv_glob_param.F90. It's identical to clsm_ensdrv_out_routines.F90.restore (2016-08-04 11:33). (7) process_cn.F90: identical to process_cn.F90.restore Couldn't find anything wrong I did. 3. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180508: 1. The e0004s_transientCO2_2011 finished 1 month of simulation. It took about 4 hours, still too slow. Files e0004s_transientCO2_2011.ens_avg.ldas_tile_monthly_out.201101.bin and /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/cat/ens_avg/Y2011/M01/e0004s_transientCO2.ens_avg.ldas_tile_monthly_out.201101.bin differ So the two monthly output are not identical! Why? Would the difference in the boundary condition have caused this difference? /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run > xdiff ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/*domain.txt ../../e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rc_out/*domain.txt & They are identical. Modified and compiled ~/Catchment/SMAP_M09/tile2grid_ease_spinup_monthly.f90 to process the output. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2011 Plot the 201101 output difference on GrADS using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2_2011.gs GPP diff: <0.02 g/m2/d LAI diff: <0.02 Emailed Sarith to ask whether the change to mosaic_veg_typs_fracs would affect the output, and why is the simulation so slow! Sarith's reply: --------------- (1) Would the change to mosaic_veg_typs_fracs affect the diagnostics in the output? Is ASCATZ0 (the last column) the column that was recently added to this file? What is it? For now, we don't use it. so it cannot be the reason - ASCATZ0 is roughness length from ASCAT satellite data (2) Can you think of anything (including any possible mistake I made) that slows down the simulation so badly? I don't know, some days discover is just slow, I heard now they have lesser number of processors until some new nodes are added - I don't know --------------- Compared the log files. The two experiments used the same restart file. The CO2 concentrations printed out are identical. Don't know why their outputs are not identical, and why e0004s_transientCO2_2011 is so slow. 2. Start the e0004s_transientCO2_2011 run for February after talking to Eunjee. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run cat year_co2.txt 2011 [Correct!] cp lenkf.0.j lenkf.1.j nedit lenkf.1.j & #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_log.20110201_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_out.20110201_0000.txt -start_year 2011 -start_month 2 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 3 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../output \ -restart_id e0004s_transientCO2_2011 \ NOTE: remember to change driver_inputs_M09_GLOBAL.nml for the next two job segments!! Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run ./lenkf.1.i It complained at tile_coord.F90 because I didn't remove the files under ../output/SMAP_EASEv2_M09_GLOBAL/rc_out. But I need these files to be there to resume the run. Domain successfully defined from existing files above. forrtl: severe (408): fort: (3): Subscript #1 of the array TILE_COORD has value 0 which is less than the lower bound of 1 Image PC Routine Line Source libmpi_usempif08. 00002AAAAC2D2093 Unknown Unknown Unknown LDASsaCN_mpi.x 0000000000A54A74 tile_coord_routin 1310 tile_coord.F90 LDASsaCN_mpi.x 00000000006D74FF clsm_ensdrv_init_ 944 clsm_ensdrv_init_routines.F90 LDASsaCN_mpi.x 00000000004A0633 MAIN__ 828 cnlsm_ensdrv_main.F90 Skip it and just submit the full job. qsub lenkf.1.j The same error occurred and the job failed. 3. Updated current goals and set new goals in PEP on ADP. 4. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180509: 1. LDAS group meeting. 2. For e0004s_transientCO2_2011, did these: Made another copy of e0004s_transientCO2_2011, so the original on becomes e0004s_transientCO2_2011_orig. Then remove the three domain and tile files in rc_out: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run ./lenkf.1.i It works!! Compare the CO2 concentrations printed out on screen to those in /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2.ldas_log.20110101_0000.txt. CO2 concentrations printed out on screen: Transient period:: forcing year, CO2 year, EEA global average CO2 (ppm), co2 sc alar: 2011 2011 389.7097 0.9995378 calculate CT co2 at 2011 2 1 0 7 30 mean co2v (fraction): 3.9130994E-04 calculate CT co2 at 2011 2 1 0 7 30 mean co2v (fraction): 3.9130994E-04 calculate CT co2 at 2011 2 1 0 7 30 mean co2v (fraction): 3.9130994E-04 They are not the same as those in e0004s_transientCO2.ldas_log.20110101_0000.txt. This is likely due to that the e0004s_transientCO2_05 run used 140 processors while this interactive run used 96 processors, so the number of tiles in the master processor is different between the two runs. Submit the full job: /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* qsub lenkf.1.j The run has started. Checked the CO2 concentrations in the log file. They are identical to those in e0004s_transientCO2_05. Great! 3. IDS meeting 4. Meeting with Randy, Eunjee and Manuela on adding irrigation to Catchment. 5. Talked with Jana about winter FPAR in the deserts. ========== 20180510: 1. The e0004s_transientCO2_2011 finished simulating Feb 2011. It took about 7 hours, too slow!! Files e0004s_transientCO2_2011.ens_avg.ldas_tile_monthly_out.201102.bin and /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/cat/ens_avg/Y2011/M02/e0004s_transientCO2.ens_avg.ldas_tile_monthly_out.201102.bin differ This is expected because the outputs of 201101 are already different. Process the output. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2011 Plot the 201102 output difference on GrADS using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2_2011.gs CO2 diff: 0 everywhere The difference in GPP, LAI, vegc and totcolc are very small except some grid cells in tropical Asia which show large difference (up to 1.5% for GPP). Discussed with Eunjee. Decided to continue the run through Mar 31, 2011. 2. Set up the e0004s_transientCO2_2011 run for 20110301_00z through 20110331_00z. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > rm -rf e0004s_transientCO2_2011_orig cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run cp driver_inputs_M09_GLOBAL.nml driver_inputs_M09_GLOBAL_daily.nml nedit driver_inputs_M09_GLOBAL_daily.nml & out_select_ensavg%tile%daily = .true. out_select_ensavg%tile%pentad = .false. out_select_ensavg%tile%monthly = .false. cp lenkf.1.j lenkf.2.j nedit lenkf.2.j & #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_log.20110301_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_out.20110301_0000.txt -start_year 2011 -start_month 3 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 3 -end_day 31 -end_hour 0 -end_min 0 -end_sec 0 \ -driver_inputs_file driver_inputs_M09_GLOBAL_daily.nml Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run ./lenkf.2.i Again it complained about tile_coord.F90: forrtl: severe (408): fort: (3): Subscript #1 of the array TILE_COORD has value 0 which is less than the lower bound of 1 Image PC Routine Line Source libmpi_usempif08. 00002AAAAC2D2093 Unknown Unknown Unknown LDASsaCN_mpi.x 0000000000A54A74 tile_coord_routin 1310 tile_coord.F90 LDASsaCN_mpi.x 00000000006D74FF clsm_ensdrv_init_ 944 clsm_ensdrv_init_routines.F90 LDASsaCN_mpi.x 00000000004A0633 MAIN__ 828 cnlsm_ensdrv_main.F90 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > /bin/cp -pr e0004s_transientCO2_2011 e0004s_transientCO2_2011_orig cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* ./lenkf.2.i It works. /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* qsub lenkf.2.j So the conclusion is that: (1) It's safe to remove the three files in rc_out, and (2) It's required to removed these three files in rc_out to make it work!! 3. To investigate why the e0004s_transientCO2_2011 run is so slow, resumed one of my previous run clm4_pIPAR_DE720 and see if it's also very slow. The last cycle of clm4_pIPAR_DE720 actually finished the simulation, but stopped during post-processing due to disk quota exceeded. Processed the output and set up the next cycle, following /home/fzeng/Catchment/CLM4/submit_next_batch_DE720. set exp = clm4_pIPAR_DE720 set num = 34 ls $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}/output/global/rs/ens0000/Y2017/M01/${exp}.ens0000.catchcn_ldas_rst.20170101_0000z $HOME/Catchment/CLM4/grid_restore_hdeg clm4_pIPAR_DE720 cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 mv ${exp} ${exp}_${num} cd ${exp}_${num}/output/global/rs/ens0000 /bin/rm -rf Y19?? Y200? Y201[0-6] mkdir -p Y1980/M01 cd Y1980/M01 ln -s ../../Y2017/M01/${exp}.ens0000.catchcn_ldas_rst.20170101_0000z ${exp}.ens0000.catchcn_ldas_rst.19800101_0000z cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num} /bin/rm -rf lenkf_orig cd $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4 /bin/cp -r ${exp}_template ${exp} cd ${exp}/input/restart ln -s $NOBACKUP/clm4-to-clm4.5/LDAS/tests/clm4/${exp}_${num}/output /bin/rm rst_clm4 cd ../../run qsub lenkf.0.j 4. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180515: 1. Read Melanie's poster for the GMAO review and provided my comments. 2. Set up the e0004s_transientCO2_2011 run for 20110331_00z through 20110331_21z. /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > rm -rf e0004s_transientCO2_2011_orig cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run cp driver_inputs_M09_GLOBAL.nml driver_inputs_M09_GLOBAL_3hourly.nml nedit driver_inputs_M09_GLOBAL_3hourly.nml & out_dtstep%xhourly = 10800 out_dtstep%inst = 10800 ! ensemble average output choices out_select_ensavg%tile%inst = .false. out_select_ensavg%tile%xhourly = .true. out_select_ensavg%tile%daily = .false. out_select_ensavg%tile%pentad = .false. out_select_ensavg%tile%monthly = .false. cp lenkf.2.j lenkf.3.j nedit lenkf.3.j & #SBATCH --time=01:00:00 #SBATCH --qos=debug #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_log.20110331_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2011/M01/e0004s_transientCO2_2011.ldas_out.20110331_0000.txt -start_year 2011 -start_month 3 -start_day 31 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2011 -end_month 3 -end_day 31 -end_hour 21 -end_min 0 -end_sec 0 \ -driver_inputs_file driver_inputs_M09_GLOBAL_3hourly.nml Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* ./lenkf.3.i It works. /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* qsub lenkf.3.j Finished! 3. Reorder and regrid /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2011/M03/e0004s_transientCO2_2011.ens0000.catchcn_ldas_rst.20110331_2100z to CF90 for Eunjee, following notes on 20170807. (1) Reorder: (1.1) Create path: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 mkdir -p rst_Eunjee/SMAP_EASEv2_M09_reordered (1.2) Create shell script "reorder_restarts" for reordering: cd rst_Eunjee/SMAP_EASEv2_M09_reordered nedit reorder_restarts & reorder_restarts reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh set YYYY = 2011 set YYYYMMDD=`echo ${YYYY}0331` mpirun -np 1 bin/mk_LDASsaRestarts -b /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE/SMAP_EASEv2_M09/ -d ${YYYYMMDD} -e e0004s_transientCO2_2011 -l /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2011/output/SMAP_EASEv2_M09_GLOBAL/ -m catchcn -s 50 -r Y -t SMAP_EASEv2_M09_3856x1624.til ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ (1.3) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: cp /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_6/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/mk_LDASsaRestarts.F90 . nedit mk_LDASsaRestarts.F90 & In SUBROUTINE reorder_LDASsa_restarts: (a) L298: tile_coord = trim(EXPDIR)//'rc_out/'//trim(expname)//'.ldas_tilecoord.bin' (b) L540: rst_file = trim(EXPDIR)//'rs/ens0000/Y'//YYYYMMDD(1:4)//'/M'//YYYYMMDD(5:6)//'/'//trim(ExpName)//& '.ens0000.catchcn_ldas_rst.'//trim(YYYYMMDD)//'_2100z' [NOTE: remember to set it to "2100z" to use the right restart file!!] cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ source $ESMADIR/src/g5_modules gmake install (1.4) Run "reorder_restarts" to reorder the offline restart file: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/SMAP_EASEv2_M09_reordered ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin reorder_restarts (2) Regrid from SMAP_EASEv2_M09 to Eunjee's AGCM grid/resolution: can't follow what we did for /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Huisheng in this step -- was able to run but TILE_ID and NDEP in the final restart file is 0 everywhere. Need to follow the instructions below for this new mk_LDASsaRestarts.F90 from GEOSldas. (2.1) Create path: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee mkdir Heracles-NL_Reynolds_CF90 (2.2) Get a copy of mk_LDASsaRestarts.F90, modify and compile it: cd Heracles-NL_Reynolds_CF90 cp ../SMAP_EASEv2_M09_reordered/mk_LDASsaRestarts.F90 . nedit mk_LDASsaRestarts.F90 & L67: InCNRestart = '/discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/SMAP_EASEv2_M09_reordered/catchcn_internal_rst.20110331', & L68: InCNTilFile = '/discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_EASE/SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til', & [Do "xdiff mk_LDASsaRestarts.F90 ../SMAP_EASEv2_M09_reordered/. &" for more modifications made.] cp mk_LDASsaRestarts.F90 /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts/. cd /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/ source $ESMADIR/src/g5_modules gmake install (2.3) Run mk_LDASsaRestarts to convert to desired tile system: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/rst_Eunjee/Heracles-NL_Reynolds_CF90 ln -s /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/mk_restarts bin nedit mkLDASsa.j & [Create a job file] mkLDASsa.j reads: ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ #!/bin/csh -f #SBATCH --account=g0620 #SBATCH --time=1:00:00 #SBATCH --qos=debug #SBATCH --ntasks=56 #SBATCH --job-name=mkLDAS #SBATCH --constraint=hasw #SBATCH --output=mkLDAS.o #SBATCH --error=mkLDAS.e setenv ESMADIR /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/Heracles-5_4_p3-M3_V24_C05 source $ESMADIR/src/g5_modules setenv MKL_CBWR SSE4_2 # ensure zero-diff across archs setenv MV2_ON_DEMAND_THRESHOLD 8192 # MVAPICH2 limit stacksize unlimited setenv SPONSORID g0620 setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds/CF0090x6C_DE0360xPE0180/ setenv TILFILE CF0090x6C_DE0360xPE0180-Pfafstetter.til mkdir -p OutData1/ mkdir -p OutData2/ ln -s $BCSDIR/$TILFILE OutData1/OutTileFile ln -s $BCSDIR/$TILFILE OutData2/OutTileFile ln -s $BCSDIR/clsm OutData2/clsm mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y /bin/cp OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst mpirun -np 56 bin/mk_LDASsaRestarts -a ${SPONSORID} -b ${BCSDIR} -t ${TILFILE} -m catchcn -s 50 -j Y bin/Scale_CatchCN OutData1/catchcn_internal_rst OutData2/catchcn_internal_rst catchcn_internal_rst 50 ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ chmod 755 mkLDASsa.j sbatch mkLDASsa.j 4. Checked the clm4_pIPAR_DE720 run. It stopped at "date_time_new 19870212_020730z" due to time out. It used to take 9.5 to 10 hours to finish 8 years of simulation. So clm4_pIPAR_DE720 is also slower than usual. Don't continue it for now. ========== 20180516: 1. Continued working on the regridding step in 3 on 20180515. Asked Sarith for help. 2. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. Updated Sarith about CLM4.5 Catchment-CN and let him know that we might need to change the restart file (and the programs/routines that are used to create/read/write the restart file). ========== 20180517: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180518: 1. Read and provided comments on Melanie's MAP proposal Y1 annual report. 2. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180521: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180522: 1. GMAO seminar. 2. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180523: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180524: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180525: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. ========== 20180529: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. 2. Prepared slides for meeting with Randy. ========== 20180530: 1. Worked on CLM4.5 Catchment-CN. See /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2018. 2. Prepared slides for meeting with Randy and Eunjee. 3. Met with Randy and Eunjee. 4. Set up a SMAP M09 run starting from the 20120101_00z restart from e0004s_transientCO2_05 and run through 20120331_21z. By doing so there will be a reasonable sink (NEE) since the carbon pools went through pre-industrial equilibrium and transient CO2 period. This is to create a 20120331_21z CF90 restart file for Eunjee. (Earlier I created a 20110331_21z CF90 restart file for Eunjee, but Randy prefers year 2012 because his paper used 2012.) Step #1: To use the same executable as e0004s_transientCO2_05, after setting up the run, will need to do the following 3 job segments (a) run from 20120101_00z through 20120301_00z, save monthly output (b) run from 20120301_00z through 20120331_00z, save daily output (c) run from 20120331_00z through 20120331_21z, save 3-hourly output Following the notes on 20170502: (1) Prepare files for experiment setup: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run cp M09_CN_e0004s_transientCO2_2011.exe M09_CN_e0004s_transientCO2_2012.exe nedit M09_CN_e0004s_transientCO2_2012.exe & exp_id = e0004s_transientCO2_2012 start_time = 2012-01-01-00-00-00 end_time = 2012-02-01-00-00-00 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_id = e0004s_transientCO2 (2) Run ldsetup: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_2012.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN (NOTE: don't use the "--submit" option because we need to modify the job scripts, see below) (3) Check the executable, restart file and create year_co2.txt: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012 ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 90093587 2018-05-02 14:04 build/Linux/bin/LDASsaCN_mpi.x* [Correct!] ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng g0620 80 2018-05-30 16:13 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4) Create year_co2.txt: cd run echo 2012 > year_co2.txt cat year_co2.txt (to make sure the year of CO2 is correct) 2012 [Correct!] (5) Check lenkf.0.j and namelist: lenkf.0.j: #SBATCH --time=12:00:00 #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120101_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120101_0000.txt -start_year 2012 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2012 -end_month 2 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../input/restart/output \ [Need modification for lenkf.1.j and lenkf.2.j] -restart_id e0004s_transientCO2 \ [Need modification for lenkf.1.j and lenkf.2.j] -driver_inputs_file driver_inputs_M09_GLOBAL.nml [Make sure it's monthly output now. Need modification for lenkf.1.j and lenkf.2.j] Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* ./lenkf.0.i It's running. Stopped it at date_time_new 20120101_023000z. /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* qsub lenkf.0.j ========== 20180531: 1. The e0004s_transientCO2_2012 run finished simulating 201201. It took 07:48:26!! (Too slow!!) The 201201 output here differs from that in e0004s_transientCO2_05. This is not surprising because we saw earlier (on 20180508) that the 201101 output files also differ. Start the 201202 simulation. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run cat year_co2.txt 2012 [Correct!] cp lenkf.0.j lenkf.1.j nedit lenkf.1.j & #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_log.20120201_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2012/M01/e0004s_transientCO2_2012.ldas_out.20120201_0000.txt -start_year 2012 -start_month 2 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2012 -end_month 3 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ -restart_path ../output \ -restart_id e0004s_transientCO2_2012 \ NOTE: remember to change driver_inputs_M09_GLOBAL.nml for the next two job segments!! Do an interactive run to make sure it runs: ssh -XY discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_2012/run /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* ./lenkf.1.i It's running. Stop it at date_time_new 20120201_030730z. /bin/rm ../output/SMAP_EASEv2_M09_GLOBAL/rc_out/* qsub lenkf.1.j 2. Process the available output from e0004s_transientCO2_2012. ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2_2012 Checked CO2 concentration using /home/fzeng/Catchment/SMAP_M09/verify_e0004s_transientCO2_2012.gs. It's identical to e0004s_transientCO2_05. Good. 3. Met with Melanie.