========== 20170501: 1. M2.n5P.3hco2+200ppm.glb run has finished. Processed the output. 2. Checked the *_all.txt files written out in the e0004s_transientCO2 run. Found that the information for CO2 year 1855 was written out twice. e.g. In year_ts_all.txt: 1851 1852 1853 1854 1855 1855 1856 1857 1858 1859 In EEAco2g_ts_all.txt: 280.0000 280.3131 280.6263 280.9394 281.2525 281.2525 281.5657 281.8788 282.1919 282.5051 In co2_scalar_ts_all.txt: 0.7181515 0.7189546 0.7197577 0.7205608 0.7213640 0.7213640 0.7221671 0.7229702 0.7237734 0.7245765 Checked the log files. The meteorology years and the co2 years look correct. In process_cn.F90: ! write out co2 year, the corresponding EEA global average CO2 and scalar for verification, fz, 16 Sep 2016 ! --------------------------------------------------------------------------------------------------------- if(date_time%dofyr==1 .and. date_time%hour==1 .and. date_time%min==0 .and. date_time%sec==0) then open(1010,file='year_ts.txt',action='write',status='unknown') write(1010,*) yr_co2 open(1011,file='EEAco2g_ts.txt',action='write',status='unknown') write(1011,*) co2g*1e6 open(1012,file='co2_scalar_ts.txt',action='write',status='unknown') write(1012,*) co2_scalar endif Therefore, the year_ts.txt written out in each job segment is: Job scripts exp period in the job year_ts.txt written metorology year when the above "if" condition is met lenkf.0.j 19950101-19951101 1851 1995 lenkf.1.j 19951101-19960901 1852 1996 lenkf.2.j 19960901-19970701 1853 1997 lenkf.3.j 19970701-19980501 1854 1998 lenkf.4.j 19980501-19990301 1855 1999 lenkf.5.j 19990301-20000101 - - lenkf.6.j 20000101-20001101 1856 2000 In the job segment lenkf.5.j, the above "if" condition was never met, so year_ts.txt was not updated and the content remained 1855 (same as for EEAco2g_ts.txt and co2_scalar_ts.txt: the content in these files remained the same as that of co2 year 1855). At the end of this job, these command lines at the end of the job script "cat year_ts.txt >> year_ts_all.txt cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt" appended the info in year_ts.txt to the end of year_ts_all.txt, so there are two lines for co2 year 1855. Same thing will happen every five simulation years. The simulation is correct. Only the information written out in these *all.txt files is not perfect. The I/O method I used here should be improved to do other transient CO2 run(s) in the future. 3. Apr report. ========== 20170502: 1. Reviewed the Apr report and sent it to Qing. 2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. ========== 20170503: 1. Did a short (14 days of simulation) run of M2.n5P.3hco2+200ppm.glb and saved 3 hourly output to verify the CO2 diurnal cycle in the M2.n5P.3hco2+200ppm.glb run. cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mkdir -p M2.n5P.3hco2+200ppm.glb.3hrOUT/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.3hco2+200ppm.glb.3hrOUT cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2+200ppm.glb.3hrOUT.ens0000.catch_ldas_rst.20010101_0000z.bin discover07:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.glb.3hrOUT > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart discover07:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.glb.3hrOUT > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2+200ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../M2.n5P.3hco2+200ppm.glb/lenkf.j . nedit lenkf.j & ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm.x -rwxr-xr-x 1 fzeng g0620 49563596 2017-04-27 10:43 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm.x qsub lenkf.j NOTE: Restart id has to be the same as exp id. Otherwise it would fail. Processed the output using ~/Catchment/M2n5P_m0001/tile2grid_FV_144x91_3hrly.f90 Compared the 3hrly CO2 output from this run and the control run using ~/Catchment/M2n5P_m0001/verify_m0001CTco2_3hrly.gs ~/Catchment/M2n5P_m0001/verify_M2.n5P.3hco2+200ppm.glb.gs Emailed Randy and Eunjee the comparison plots. 2. Processed GMAO GEOS5 seasonal forecast for May. 3. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. ========== 20170504: 1. Further verified the transient CO2 run output. It's simulating 200812 (met time). Process the available output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2 Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs 2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. ========== 20170505: 1. Compared NPP at 4 FACE sites between M2.n5P.3hco2.glb (control) and M2.n5P.3hco2+200ppm.glb (experiment). ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90 Run it like this: extract_tile_at_ECsite exp ECname lon lat ityp_EC var_id (1) DukeFACE extract_tile_at_ECsite M2.n5P.3hco2.glb DukeFACE -79.083 35.967 1 61 extract_tile_at_ECsite M2.n5P.3hco2+200ppm.glb DukeFACE -79.083 35.967 1 61 read raster data: 1 456957 tile id for the EC site: 217894 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -79.08300 35.96700 longitude and latitude of the tile found: -78.99482 35.86514 area (km2) of the tile found: 844.7407 ityp of the EC site: 1 ityp and fveg of the tile found: 7 7 1 1 0.5958000 0.0000000E+00 0.4042000 0.0000000E+00 done writing npp for DukeFACE (2) AspenFACE extract_tile_at_ECsite M2.n5P.3hco2.glb AspenFACE -89.617 45.667 7 61 extract_tile_at_ECsite M2.n5P.3hco2+200ppm.glb AspenFACE -89.617 45.667 7 61 read raster data: 1 456957 tile id for the EC site: 224882 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -89.61700 45.66700 longitude and latitude of the tile found: -89.79707 45.76556 area (km2) of the tile found: 1471.679 ityp of the EC site: 7 ityp and fveg of the tile found: 7 7 1 1 0.6170000 0.0000000E+00 0.3830000 0.0000000E+00 done writing npp for AspenFACE (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3hco2.glb ORNL_FACE -84.333 35.9 7 61 extract_tile_at_ECsite M2.n5P.3hco2+200ppm.glb ORNL_FACE -84.333 35.9 7 61 read raster data: 1 456957 tile id for the EC site: 222716 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -84.33300 35.90000 longitude and latitude of the tile found: -84.15839 35.96996 area (km2) of the tile found: 501.0312 ityp of the EC site: 7 ityp and fveg of the tile found: 7 7 1 1 0.7487000 0.0000000E+00 0.2513000 0.0000000E+00 done writing npp for ORNL_FACE (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3hco2.glb POP_EUROFACE 11.8 42.367 7 61 extract_tile_at_ECsite M2.n5P.3hco2+200ppm.glb POP_EUROFACE 11.8 42.367 7 61 read raster data: 1 456957 tile id for the EC site: 276370 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: 11.80000 42.36700 longitude and latitude of the tile found: 11.82086 42.33916 area (km2) of the tile found: 225.9444 ityp of the EC site: 7 ityp and fveg of the tile found: 18 19 14 15 0.9316999 0.0000000E+00 6.8300001E-02 0.0000000E+00 done writing npp for POP_EUROFACE Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. Emailed Randy and Eunjee the results. ========== 20170508: 1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. 2. Updated higgins rain data through May 7. 3. Compared NPP at 4 FACE sites between M2.n5P.3hco2.glb (mean CO2 ~387ppm) and /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c1 (mean CO2 ~385ppm). Mean CO2 values are from Eunjee. Modified ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90 Run it like this: extract_tile_at_ECsite exp ECname lon lat ityp_EC var_id (1) DukeFACE extract_tile_at_ECsite M2.n5P.3htCO2.c1 DukeFACE -79.083 35.967 1 61 read raster data: 1 456957 tile id for the EC site: 217894 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -79.08300 35.96700 longitude and latitude of the tile found: -78.99482 35.86514 area (km2) of the tile found: 844.7407 ityp of the EC site: 1 ityp and fveg of the tile found: 7 7 1 1 0.5958000 0.0000000E+00 0.4042000 0.0000000E+00 done writing npp for DukeFACE (2) AspenFACE extract_tile_at_ECsite M2.n5P.3htCO2.c1 AspenFACE -89.617 45.667 7 61 read raster data: 1 456957 tile id for the EC site: 224882 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -89.61700 45.66700 longitude and latitude of the tile found: -89.79707 45.76556 area (km2) of the tile found: 1471.679 ityp of the EC site: 7 ityp and fveg of the tile found: 7 7 1 1 0.6170000 0.0000000E+00 0.3830000 0.0000000E+00 done writing npp for AspenFACE (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3htCO2.c1 ORNL_FACE -84.333 35.9 7 61 read raster data: 1 456957 tile id for the EC site: 222716 tile coordinate file land tiles: 370121 longitude and latitude of the EC site: -84.33300 35.90000 longitude and latitude of the tile found: -84.15839 35.96996 area (km2) of the tile found: 501.0312 ityp of the EC site: 7 ityp and fveg of the tile found: 7 7 1 1 0.7487000 0.0000000E+00 0.2513000 0.0000000E+00 done writing npp for ORNL_FACE (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3htCO2.c1 POP_EUROFACE 11.8 42.367 7 61 Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. ========== 20170509: 1. The SMAP M09 transient CO2 run finished the 1st cycle. CO2 years: 1851-1872; meteorology years: 1995-2016. Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2 Check the July 2016 CO2, LAI, GPP, NPP and NEE output on GrADS. Looks good. Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_transientCO2 e0004s_transientCO2_01 cd e0004s_transientCO2_01 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y199? Y200? Y201[0-6] mkdir -p Y1981/M01 cd Y1981/M01 ln -s ../../Y2017/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20170101_0000z e0004s_transientCO2.ens0000.catchcn_ldas_rst.19810101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run nedit M09_CN_e0004s_transientCO2.exe & start_time = 1981-01-01-00-00-00 end_time = 2017-01-01-00-00-00 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_01/output/ restart_id = e0004s_transientCO2 Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN (NOTE: don't use the "--submit" option because we need to modify the job scripts, see below) Check the executable and restart file: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2 ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x* ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng g0620 80 2017-05-09 10:48 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_01/output/ Copy year_co2.txt and the *all.txt files from previous cycle: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_01/run cp -p year_co2.txt *all.txt /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run/. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run Check if these files are correct, particularly year_co2.txt! cat year_co2.txt (to make sure the year of CO2 is correct) 1873 (correct!) nedit *all.txt & Modify the job scripts before submission: Under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run: mkdir ../lenkf_orig (Important: lenkf_orig has to be one level up. Otherwise the lenkf.*.j files in lenkf_orig will be counted by edit_lenkf.csh) cp -p lenkf.*.j ../lenkf_orig/. cp ../../e0004s_transientCO2_01/run/edit_lenkf.csh . ./edit_lenkf.csh Check the lenkf.*.j files and they look correct. qsub lenkf.0.j NOTE: for transientCO2 run, (1) Need to use edit_lenkf.csh to add these lines below to the end of each job file lenkf.N.j (as above): cat year_ts.txt >> year_ts_all.txt cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt qsub lenkf.N+1.j (2) Need to copy edit_lenkf.csh, year_co2.txt, year_ts_all.txt, EEAco2g_ts_all.txt and co2_scalar_ts_all.txt from previous cycle. 2. For the CO2 experiment that uses CT 3hrly CO2 + 1ppm: Modified process_cat.F90 from /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/M2n5P_m0001CTco2/process_cat.F90.3hrlyCO2+200ppm. Used this new process_cat.F90 and the compute_rc.F90 in this M2n5P_m0001CTco2 directory and compiled. Set up the experiment: cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mkdir -p M2.n5P.3hco2+1ppm.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.3hco2+1ppm.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2+1ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+1ppm.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+1ppm.glb > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2+1ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../M2.n5P.3hco2+200ppm.glb/lenkf.j . nedit lenkf.j & ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+1ppm.x -rwxr-xr-x 1 fzeng g0620 49563596 2017-05-09 11:43 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+1ppm.x* qsub lenkf.j It's running. Processed the available output: ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2+1ppm.glb Verified the output using ~/Catchment/M2n5P_m0001/verify_M2.n5P.3hco2+1ppm.glb.gs. Looks correct. 3. For the CO2 experiment that uses CT 3hrly CO2 + 10ppm: Modified process_cat.F90 from /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/M2n5P_m0001CTco2/process_cat.F90.3hrlyCO2+200ppm. Used this new process_cat.F90 and the compute_rc.F90 in this M2n5P_m0001CTco2 directory and compiled. Set up the experiment: cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mkdir -p M2.n5P.3hco2+10ppm.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.3hco2+10ppm.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2+10ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+10ppm.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart discover17:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+10ppm.glb > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2+10ppm.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../M2.n5P.3hco2+200ppm.glb/lenkf.j . nedit lenkf.j & ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+10ppm.x -rwxr-xr-x 1 fzeng g0620 49563596 2017-05-09 12:02 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+10ppm.x* qsub lenkf.j It's running. Processed the available output: ~/Catchment/M2n5P_m0001 > tile2grid_FV_144x91_monthly M2.n5P.3hco2+10ppm.glb Verified the output using ~/Catchment/M2n5P_m0001/verify_M2.n5P.3hco2+10ppm.glb.gs. Looks correct. 4. Calculate 2014 CO2 using the slope in transient CO2 run and compare it to 2014 CT CO2. ~/Catchment/SMAP_M09/e0004s_transientCO2_2014CO2.f90 co2_scalar: 1.009466 Plot the difference using /discover/nobackup/fzeng/noaaCO2/code/verify_transientCO2.gs ========== 20170510: 1. Organized my notes about CVS and now I understand CVS more. 2. Read Grayson Badgley's papers and talked with him. He is in Joe Berry's group. 3. Prepared for using Greg's point driver on land01 to understand how ci responds to ambient co2. (3.1) Located the tiles that are closet to the FACE sites using ~/Catchment/princeton_tiles_location.f90. To be consistent with global simulations, use this tile file: /discover/nobackup/fzeng/bcs/SMAP_TestBed/FV_144x91/DC0144xPC0091_DE0360xPE0180-Pfafstetter.til Pick the ones that match those found on 20170505 above. Here what we need from here is tile id. (3.1.1) DukeFACE: lon = -79.08, lat = 35.97 217894 41 64 -78.99 35.87 0.01689 7 7 1 1 0.0060 0.0000 0.0040 0.0000 (3.1.2) AspenFACE: lon = -89.62, lat = 45.67 224882 37 69 -89.80 45.77 0.03427 7 7 1 1 0.0062 0.0000 0.0038 0.0000 (3.1.3) ORNL-FACE: lon = -84.33, lat = 35.90 222716 39 64 -84.16 35.97 0.01002 7 7 1 1 0.0075 0.0000 0.0025 0.0000 (3.1.4) POP-EUROFACE: lon = 11.80, lat = 42.37 276370 78 67 11.82 42.34 0.00492 18 19 14 15 0.0093 0.0000 0.0007 0.0000 (3.2) Tried to extract the Princeton forcing for the 4 tiles found above using ~/Catchment/extract_princeton_point_pexp_hires.f90, but got the error message "error opening Princeton_netcdf file". Found that the Princeton data files on /discover/nobackup/projects/lis/MET_FORCING/PRINCETON/ have time stemps of 2017-04-24. Checked with David Mocko. He confirmed that the Princeton data was updated recently. The file format is a little different. David helped me modify extract_princeton_point_pexp_hires.f90 to make it run. Now, to compile extract_princeton_point_pexp_hires.f90, just "gmake" to get the executable extract_princeton_point_pexp_hires.exe. Saved the information about the data update in /discover/nobackup/fzeng/princeton (see emails, new.txt, and old.txt). ========== 20170511: 1. ACMAP meeting. 2. Continued working on 3.2 on 20170510 above. 3. Checked the point driver to see if we need to update pcli in /discover/nobackup/fzeng/princeton/cli3 before I do the point driver runs. In conus_force_loop.F90 (L119): forc(ic,3) = a*(pcli + b*(forc(ic,3)-pcli)) Therefore, as long as I use a=1 and b=1 (which I have been doing in all my runs so far), pcli is actually not used. So it's fine not to update it now. ========== 20170512: 1. Modify the point driver to write out ci values. Ongoing. 2. Repeated the M2.n5P.3hco2+200ppm.glb experiment but now used 30 iterations in the calculation of photosynthesis instead of using the default 3 iterations. Used process_cat.F90.3hrlyCO2+200ppm and compute_rc.F90.30iter in /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/src/Components/GEOSlana_GridComp/M2n5P_m0001CTco2. Set up the experiment M2.n5P.3hco2+200ppm.30iter.glb: cd /discover/nobackup/fzeng/Catchment/M2n5P/m0001 mkdir -p M2.n5P.3hco2+200ppm.30iter.glb/RUN/rs/ens0000/Y2001/M01 cd M2.n5P.3hco2+200ppm.30iter.glb cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/CN_restart . cp -p /discover/nobackup/elee15/offline/sims/M2.n5P.3htCO2.c5/RUN/rs/ens0000/Y2001/M01/mM2.n5P.3htCO2.ens0000.catch_ldas_rst.20010101_0000z.bin RUN/rs/ens0000/Y2001/M01/mM2.n5P.3hco2+200ppm.30iter.glb.ens0000.catch_ldas_rst.20010101_0000z.bin discover18:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.30iter.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2016-11-08 19:34 CN_restart discover18:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+200ppm.30iter.glb > ls -l RUN/rs/ens0000/Y2001/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2016-11-08 19:34 mM2.n5P.3hco2+200ppm.30iter.glb.ens0000.catch_ldas_rst.20010101_0000z.bin cp ../M2.n5P.3hco2+200ppm.glb/lenkf.j . nedit lenkf.j & ls -l /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm_30iter.x -rwxr-xr-x 1 fzeng g0620 49563596 2017-05-12 16:31 /discover/nobackup/fzeng/offline_code/LDASsa_m3-15_2-CN/exec/LDASsa_mpi_M2n5P_m0001CTco2_3hrly+200ppm_30iter.x qsub lenkf.j The run crashed because I forgot to put letter "m" at the beginning of "mM2.n5P.3hco2+200ppm.30iter.glb.ens0000.catch_ldas_rst.20010101_0000z.bin", i.e. it was "M2.n5P.3hco2+200ppm.30iter.glb.ens0000.catch_ldas_rst.20010101_0000z.bin" so the run can't find the land restart file. The above has been corrected. Re-submitted lenkf.j. 3. Compared NPP at 4 FACE sites between M2.n5P.3hco2.glb and M2.n5P.3hco2+10ppm.glb (only available for 2001-2010 so far, the run has not finished yet) Modified ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90 (changed 2014 to 2010) Run it like this: extract_tile_at_ECsite exp ECname lon lat ityp_EC var_id (1) DukeFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb DukeFACE -79.083 35.967 1 61 tile id for the EC site: 217894 (2) AspenFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb AspenFACE -89.617 45.667 7 61 tile id for the EC site: 224882 (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb ORNL_FACE -84.333 35.9 7 61 tile id for the EC site: 222716 (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb POP_EUROFACE 11.8 42.367 7 61 tile id for the EC site: 276370 Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. ========== 20170515: 1. The M2.n5P.3hco2+10ppm.glb run stopped at "date_time_new 20110101_011000z". It took 12 hours to run 1 hour of simulation!! Don't know why. Started: date_time_new 20110101_001000z Ended: date_time_new 20110101_011000z slurmstepd-borgw109: *** JOB 15623431 CANCELLED AT 2017-05-13T13:47:59 DUE TO TIME LIMIT on borgw109 *** mpirun: Forwarding signal 18 to job discover18:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+10ppm.glb > ls -l CN_restart -rw-r--r-- 1 fzeng g0620 1530820456 2017-05-12 14:10 CN_restart discover18:/discover/nobackup/fzeng/Catchment/M2n5P/m0001/M2.n5P.3hco2+10ppm.glb > ls -l RUN/rs/ens0000/Y2011/M01/ total 40512 -rw-r--r-- 1 fzeng g0620 41453776 2017-05-12 14:10 mM2.n5P.3hco2+10ppm.glb.ens0000.catch_ldas_rst.20110101_0000z.bin Modified lenkf.j to make it start running from 20110101. Submitted it. 2. The M2.n5P.3hco2+1ppm.glb run has finished. Extracted the NPP output for the 4 tiles using ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90. (1) DukeFACE extract_tile_at_ECsite M2.n5P.3hco2+1ppm.glb DukeFACE -79.083 35.967 1 61 tile id for the EC site: 217894 (2) AspenFACE extract_tile_at_ECsite M2.n5P.3hco2+1ppm.glb AspenFACE -89.617 45.667 7 61 tile id for the EC site: 224882 (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3hco2+1ppm.glb ORNL_FACE -84.333 35.9 7 61 tile id for the EC site: 222716 (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3hco2+1ppm.glb POP_EUROFACE 11.8 42.367 7 61 tile id for the EC site: 276370 Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. ========== 20170516: 1. David Mocko created a GrADS script pull_time-series.gs for me to extract the new (and old as well if needed) Princeton forcing data. ~/Catchment > cp -p /discover/nobackup/projects/lis/MET_FORCING/PRINCETON/grads_scripts/pull_time-series.gs . /discover/nobackup/fzeng/princeton > cp -p /discover/nobackup/projects/lis/MET_FORCING/PRINCETON/grads_scripts/prcp_climatology.xdf . Modified and ran pull_time-series.gs to extract the new Princeton forcing for the 4 FACE sites. Double checked the pull_time-series.gs like David did and confirmed that the script is running correctly. /discover/nobackup/fzeng/princeton_point > diff -q point_217706_old.gdat point_217706_pexp.gdat returned nothing. 2. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. 3. The M2.n5P.3hco2+200ppm.30iter.glb run crashed in 200211 due to time out. Extracted the available NPP output for the 4 tiles using ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90. (1) DukeFACE extract_tile_at_ECsite M2.n5P.3hco2+200ppm.30iter.glb DukeFACE -79.083 35.967 1 61 tile id for the EC site: 217894 (2) AspenFACE extract_tile_at_ECsite M2.n5P.3hco2+200ppm.30iter.glb AspenFACE -89.617 45.667 7 61 tile id for the EC site: 224882 (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3hco2+200ppm.30iter.glb ORNL_FACE -84.333 35.9 7 61 tile id for the EC site: 222716 (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3hco2+200ppm.30iter.glb POP_EUROFACE 11.8 42.367 7 61 tile id for the EC site: 276370 Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. ========== 20170517: 1. The M2.n5P.3hco2+10ppm.glb run has finished. Extracted the NPP output for the 4 tiles using ~/Catchment/M2n5P_m0001/extract_tile_at_ECsite.f90. (1) DukeFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb DukeFACE -79.083 35.967 1 61 tile id for the EC site: 217894 (2) AspenFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb AspenFACE -89.617 45.667 7 61 tile id for the EC site: 224882 (3) ORNL-FACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb ORNL_FACE -84.333 35.9 7 61 tile id for the EC site: 222716 (4) POP-EUROFACE extract_tile_at_ECsite M2.n5P.3hco2+10ppm.glb POP_EUROFACE 11.8 42.367 7 61 tile id for the EC site: 276370 Used ~/Catchment/M2n5P_m0001/comparison_site_level.m to analyze the tile-space NPP data extracted above and make plots. 2. Met with 618 semiar speaker. 3. Met with Randy and Eunjee about our CO2 fertilization results. 4. Read the N cycle part of the CN code. ========== 20170518: 1. Summarized the major N processes in the CN code and sent slides to Randy and Eunjee. 2. Met with Randy, Eunjee and Abhishek about the SUSMAP project. 3. Emailed Abhishek the output information of the old SMAP runs. 4. Create a script to extract sminn from restart files: ~/Catchment/M2n5P_m0001/extract_CN_restart_sminn.f90. ========== 20170519: 1. Continued working on 4 on 20170518. 2. Extracted sminn data from CN_restart files of Eunjee's 5 transient CO2 cycles using ~/Catchment/M2n5P_m0001/extract_CN_restart_sminn.f90. Made plots using ~/Catchment/M2n5P_m0001/check_CN_restart_sminn.gs. Emailed Randy and Eunjee the maps. 3. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. ========== 20170522: 1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. 2. Huisheng showed me her GEOS5 test run results. The Amazon GPP is too low. LAI is also too low. Decided to look into the restart file I provided. ========== 20170523: 1. Worked on upgrading the CN model in Catchment-CN to CLM4.5. See notes /discover/nobackup/fzeng/clm4-to-clm4.5/notes_daily. 2. Created /home/fzeng/geos5/check_rst_vars.m to look into the restart file I provided Huisheng for her GEOS5 test run. ========== 20170524: 1. Found that the e0004s_transientCO2 run stopped at 20080701 met time which is 1900 in CO2 year, because it was not able to read this file: /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3//SMAP_EASEv2_M09/SMAP_EASEv2_M09_3856x1624.til NCCS support noticed us that dnb02 was partially unavailable. This could be the reason. The CO2 year, scalar and concentration for 1900 were written out multiple times to the co2_scalar_ts_all.txt, EEAco2g_ts_all.txt, year_ts_all.txt files. Don't know why. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2008 mv M07 M07_old mkdir M07 cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run qsub lenkf.33.j Crashed: could not find file: ../output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2008/M07/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20080701_0000z Maybe the run didn't get to 20080701 before it stopped yesterday. The last monthly output file was written 2017-05-23 14:40. -rw-r--r-- 1 fzeng g0620 539112640 2017-05-23 14:40 e0004s_transientCO2.ens_avg.ldas_tile_monthly_out.200801.bin This was right before the run stopped yesterday. -rw-r--r-- 1 fzeng g0620 35763 2017-05-23 14:58 ../../../rc_out/Y2008/M07_old/e0004s_transientCO2.ldas_log.20080701_0000.txt Has to resume the run from 20080101. -rw-r--r-- 1 fzeng g0620 7547615847 2017-05-23 13:45 e0004s_transientCO2.ens0000.catchcn_ldas_rst.20080101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run cp lenkf.32.j lenkf.32p2.j nedit lenkf.32p2.j & #SBATCH --time=10:00:00 #SBATCH --output=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2007/M09/e0004s_transientCO2.ldas_log.20080101_0000.txt #SBATCH --error=../output/SMAP_EASEv2_M09_GLOBAL/rc_out/Y2007/M09/e0004s_transientCO2.ldas_out.20080101_0000.txt -start_year 2008 -start_month 1 -start_day 1 -start_hour 0 -start_min 0 -start_sec 0 \ -end_year 2008 -end_month 7 -end_day 1 -end_hour 0 -end_min 0 -end_sec 0 \ qsub lenkf.32p2.j It's running. 2. Continued working on /home/fzeng/geos5/check_rst_vars.m. Yesterday it didn't run correctly in reading the tile file, but now the same script runs as expected. The failure could be due to the same issue on dnb02 yesterday. ========== 20170525: 1. Checked out SM-Heracles-5_4_p3-M3_V24_C05 and set up an experiment to see if I can get correct GPP output. /discover/nobackup/fzeng/geos5_code/SM-Heracles-5_4_p3-M3_V24_C05/GEOSagcm/src/Applications/GEOSgcm_App > ./gcm_setup Enter the Experiment ID: SM-Heracles-5_4_p3-M3_V24_C05_test Enter a 1-line Experiment Description: SM-Heracles-5_4_p3-M3_V24_C05_test Do you wish to CLONE an old experiment? (Default: NO or FALSE) Enter the Atmospheric Horizontal Resolution code: ----------------------------------------------------------- Lat/Lon Cubed-Sphere ----------------------------------------------------------- b -- 2 deg c48 -- 2 deg c -- 1 deg c90 -- 1 deg d -- 1/2 deg c180 -- 1/2 deg (56-km) e -- 1/4 deg (35-km) c360 -- 1/4 deg (28-km) c720 -- 1/8 deg (14-km) c1440 - 1/16 deg ( 7-km) c90 Enter the Atmospheric Model Vertical Resolution: LM (Default: 72) Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE) Enter the Data_Ocean Horizontal Resolution code: o1 (1 -deg, 360x180 Reynolds) Default o2 (1/4-deg, 1440x720 MERRA-2) o3 (1/8-deg, 2880x1440 OSTIA) o2 Enter the choice of Land Surface Model: use 1 (Default: Catchment Model), 2 (CatchmentCN Model) 2 Do you wish to run the Runoff Routing Model? (Default: NO or FALSE) Do you wish to run GOCART with Actual or Climatological Aerosols? (Enter: A (Default) or C) Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS: Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use (To use HISTORY.AGCM.rc.tmpl from current build, Type: Current ) ------------------------------------------------------------------------- Hit ENTER to use Default Tag/Location: (Current) Enter Desired Location for the HOME Directory (to contain scripts and RC files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/tests/SM-Heracles-5_4_p3-M3_V24_C05_test /discover/nobackup/fzeng/geos5_sims/SM-Heracles-5_4_p3-M3_V24_C05_test Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/clm4-to-clm4.5/GEOS5/tests/SM-Heracles-5_4_p3-M3_V24_C05_test /discover/nobackup/fzeng/geos5_sims/SM-Heracles-5_4_p3-M3_V24_C05_test Enter Location for Build directory containing: src/ Linux/ etc... Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/geos5_code/SM-Heracles-5_4_p3-M3_V24_C05/GEOSagcm Current GROUPS: g0620 gmaoint m2val Enter your GROUP ID for Current EXP: (Default: g0620) ----------------------------------- sending incremental file list GEOSgcm.x sent 100153687 bytes received 31 bytes 200307436.00 bytes/sec total size is 100141366 speedup is 1.00 Creating gcm_run.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_post.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_archive.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_regress.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_convert.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_plot.tmpl for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_quickplot.csh for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_moveplot.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_stats.j for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_forecast.tmpl for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating gcm_forecast.setup for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating CAP.rc.tmpl for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating AGCM.rc.tmpl for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating HISTORY.rc.tmpl for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Creating fvcore_layout.rc for Experiment: SM-Heracles-5_4_p3-M3_V24_C05_test Done! ----- Build Directory: /discover/nobackup/fzeng/geos5_code/SM-Heracles-5_4_p3-M3_V24_C05/GEOSagcm ---------------- The following executable has been placed in your Experiment Directory: ---------------------------------------------------------------------- /discover/nobackup/fzeng/geos5_code/SM-Heracles-5_4_p3-M3_V24_C05/GEOSagcm/Linux/bin/GEOSgcm.x You must now copy your Initial Conditions into: ----------------------------------------------- /discover/nobackup/fzeng/geos5_sims/SM-Heracles-5_4_p3-M3_V24_C05_test Copying Sandbox Source Code (including non-committed files) into /discover/nobackup/fzeng/geos5_sims/SM-Heracles-5_4_p3-M3_V24_C05_test/src/GEOSagcm/src ... Enter CVS Tag to COMMIT this Experiment (Default: SM-Heracles-5_4_p3-M3_V24_C05_test__fzeng) Enter q or quit to QUIT or SKIP the CVS COMMIT: q cd /discover/nobackup/fzeng/geos5_sims/SM-Heracles-5_4_p3-M3_V24_C05_test mkdir restarts cd restarts cp -p /discover/nobackup/elee15/H52_CatCN_ctrl/restarts/ctrl_c1/restarts.e20001231_21z.tar tar -xf restarts.e20001231_21z.tar created move_rst to re-name the restart files mv H52_CatCN_ctrl.vegdyn.data.e20001231_21z.Heracles-5_2.Heracles-4_3_MERRA-3_CF0090x6C_DE1440xPE0720 vegdyn.data mv *rst ../ mv vegdyn.data ../ cd .. Create cap_restart (which has only this line below) 20001231 210000 Edit CAP.rc: END_DATE: 20010301 210000 Edit HISTORY.rc In COLLECTIONS, comment out everything other than "geosgcm_surf" [Line 58] geosgcm_surf.frequency: 240000, [Lines 207-243] Uncomment geosgcm_surf collection for carbon variables (e.g., CNLAI, ICESOI) Edit gcm_run.j [Line 161] setenv BCSDIR /discover/nobackup/ltakacs/bcs/Ganymed-4_0/Ganymed-4_0_MERRA-2 => setenv BCSDIR /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/ Comment out L1195 "qsub $FILE" in /discover/nobackup/fzeng/geos5_code/SM-Heracles-5_4_p3-M3_V24_C05/GEOSagcm/src/GMAO_Shared/GEOS_Util/post/gcmpost.script to avoid submitting the post plotting job. Noticed that the BCSDIR (in gcm_run.j) in Eunjee's run /discover/nobackup/smahanam/bcs/Heracles-4_3/Heracles-4_3_MERRA-3/ differs from that in Huisheng's run /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds/. Didn't submit the job. Talked to Huisheng and Sarith. Although both are 1 deg atmosphere resolution, the land mask is different between the two above. Icarus code is new but the land mask is old - has much smaller number of catchments. Huisheng has to run regrid.pl to create restart files that fit the resolution she wants. ========== 20170526: 1. Read Joanna Joiner's MS and provided a couple of minor comments. 2. Using the restarts generated from running regrid.pl yesterday, Huisheng's run crashed. Do this myself to see what happens. Checked out Icarus-1_0. Compiled. 3. Set up an experiment with CN: (1) Set up experiment directory: cd Applications/GEOSgcm_App ./gcm_setup Enter the Experiment ID: I10catcn Enter a 1-line Experiment Description: I10catcn Do you wish to CLONE an old experiment? (Default: NO or FALSE) Enter the Atmospheric Horizontal Resolution code: ----------------------------------------------------------- Lat/Lon Cubed-Sphere ----------------------------------------------------------- b -- 2 deg c48 -- 2 deg c -- 1 deg c90 -- 1 deg d -- 1/2 deg c180 -- 1/2 deg (56-km) e -- 1/4 deg (35-km) c360 -- 1/4 deg (28-km) c720 -- 1/8 deg (14-km) c1440 - 1/16 deg ( 7-km) c90 Enter the Atmospheric Model Vertical Resolution: LM (Default: 72) Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE) Enter the Data_Ocean Horizontal Resolution code: o1 (1 -deg, 360x180 Reynolds) Default o2 (1/4-deg, 1440x720 MERRA-2) o3 (1/8-deg, 2880x1440 OSTIA) CS (Cubed-Sphere OSTIA) o1 Enter the choice of Land Surface Model: use 1 (Default: Catchment Model), 2 (CatchmentCN Model) 2 Do you wish to run the Runoff Routing Model? (Default: NO or FALSE) Do you wish to run GOCART with Actual or Climatological Aerosols? (Enter: A (Default) or C) Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS: Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use (To use HISTORY.AGCM.rc.tmpl from current build, Type: Current ) ------------------------------------------------------------------------- Hit ENTER to use Default Tag/Location: (Current) Enter Desired Location for the HOME Directory (to contain scripts and RC files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/geos5_sims/I10catcn Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/geos5_sims/I10catcn Enter Location for Build directory containing: src/ Linux/ etc... Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/geos5_code/Icarus-1_0/GEOSagcm Current GROUPS: g0620 gmaoint m2val Enter your GROUP ID for Current EXP: (Default: g0620) ----------------------------------- sending incremental file list GEOSgcm.x sent 100463720 bytes received 31 bytes 66975834.00 bytes/sec total size is 100451363 speedup is 1.00 Creating gcm_run.j for Experiment: I10catcn Creating gcm_post.j for Experiment: I10catcn Creating gcm_archive.j for Experiment: I10catcn Creating gcm_regress.j for Experiment: I10catcn Creating gcm_convert.j for Experiment: I10catcn Creating gcm_plot.tmpl for Experiment: I10catcn Creating gcm_quickplot.csh for Experiment: I10catcn Creating gcm_moveplot.j for Experiment: I10catcn Creating gcm_stats.j for Experiment: I10catcn Creating gcm_forecast.tmpl for Experiment: I10catcn Creating gcm_forecast.setup for Experiment: I10catcn Creating CAP.rc.tmpl for Experiment: I10catcn Creating AGCM.rc.tmpl for Experiment: I10catcn Creating HISTORY.rc.tmpl for Experiment: I10catcn Creating fvcore_layout.rc for Experiment: I10catcn Done! ----- Build Directory: /discover/nobackup/fzeng/geos5_code/Icarus-1_0/GEOSagcm ---------------- The following executable has been placed in your Experiment Directory: ---------------------------------------------------------------------- /discover/nobackup/fzeng/geos5_code/Icarus-1_0/GEOSagcm/Linux/bin/GEOSgcm.x You must now copy your Initial Conditions into: ----------------------------------------------- /discover/nobackup/fzeng/geos5_sims/I10catcn Copying Sandbox Source Code (including non-committed files) into /discover/nobackup/fzeng/geos5_sims/I10catcn/src/GEOSagcm/src ... Enter CVS Tag to COMMIT this Experiment (Default: I10catcn__fzeng) Enter q or quit to QUIT or SKIP the CVS COMMIT: q (2) Create restart files: cd /discover/nobackup/fzeng/geos5_sims/I10catcn mkdir restarts_regrid cd restarts_regrid ln -s /discover/nobackup/fzeng/geos5_code/Icarus-1_0/GEOSagcm/Linux/bin bin/regrid.pl -catchcn Do you want script to select MERRA-2 input restarts for you (y/n)? [n] y Enter OUTPUT directory for new restarts: OUTPUT mkdir OUTPUT Enter ymd (yyyymmdd): 20030101 Enter hour (hh): 21 INPUT atmosphere grid: C180 Enter OUTPUT atmosphere grid: C90 INPUT atmosphere levels: 72 Enter OUTPUT atmosphere levels: [72] Ocean Grids ----------- c = 360x180 (Reynolds) e = 1440x720 (MERRA-2) f = 2880x1440 (OSTIA) CS = same as atmosphere (OSTIA cubed-sphere) INPUT ocean grid: e Enter OUTPUT ocean grid: [c] OUTPUT ocean grid: c INPUT tag: Ganymed-4_0_MERRA-2 Type 'bcs' to see BCS tags or Enter GCM or DAS tag for outputs: [Ganymed-4_0] Heracles-NL - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.rst.lcv.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.moist_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.fvcore_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.pchem_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.matrix_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.mam_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.stratchem_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.geosachem_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.tr_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.gocart_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.gmichem_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.agcm_import_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.agcm_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.catch_internal_rst.20030101_21z.bin NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.catchcn_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.lake_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.landice_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.saltwater_internal_rst.20030101_21z.bin - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg03_sfc_rst.20030101_21z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg03_eta_rst.20030101_21z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.cbkg03_eta_rst.20030101_21z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg06_sfc_rst.20030102_00z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg06_eta_rst.20030102_00z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.cbkg06_eta_rst.20030102_00z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg09_sfc_rst.20030102_03z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.bkg09_eta_rst.20030102_03z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.cbkg09_eta_rst.20030102_03z.nc4 - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.ana_satbang_rst.20030101_21z.txt - /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.ana_satbias_rst.20030101_21z.txt NOT FOUND: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.trak.GDA.rst.2003010118z.txt =============== REGRID RESTARTS =============== ESMABIN: /gpfsm/dnb31/fzeng/geos5_code/Icarus-1_0/GEOSagcm/Linux/bin ESMATAG: Icarus-1_0 logfile: /gpfsm/dnb31/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT/C90c_d5124_m2_jan00.20030101_21z.log upper-air: yes surface: yes - route: no - catch: yes - catchCN: yes bkg/satb: yes rst.lcv: yes namelabel: no # inputs #------- . CVS tag: Ganymed-4_0 . expid: d5124_m2_jan00 . date: 20030101 . hour: 21 . atmos grid: 180x1080 (C180) . atmos levs: 72 . ocean grid: 1440x720 (e) . BCS tag: Ganymed-4_0_MERRA-2 . surflay: 50 . rstdir: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01 # outputs #-------- . CVS tag: Heracles-NL . expid: C90c_d5124_m2_jan00 . date: 20030101 . hour: 21 . atmos grid: 90x540 (C90) . atmos levs: 72 . ocean grid: 360x180 (c) . BCS tag: Heracles-NL_Reynolds . surflay: 50 . rescale: yes . outdir: OUTPUT . workdir: OUTPUT/20030101_12290 =============== Continue (y/n)? [y] ----------------- Upperair Restarts ----------------- -------- Group ID -------- Sp Code| Org | Sponsor | Research -------+-------+--------------------+---------------------------------- g0620 | 610.1 | William Putman | GMAO - Systems and Data Synthesis select group: [g0620] copy /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.fvcore_internal_rst.20030101_21z.bin => OUTPUT/20030101_12290/d5124_m2_jan00.fvcore_internal_rst.20030101_21z.bin copy /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.moist_internal_rst.20030101_21z.bin => OUTPUT/20030101_12290/d5124_m2_jan00.moist_internal_rst.20030101_21z.bin copy /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.agcm_import_rst.20030101_21z.bin => OUTPUT/20030101_12290/d5124_m2_jan00.agcm_import_rst.20030101_21z.bin copy /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.gocart_internal_rst.20030101_21z.bin => OUTPUT/20030101_12290/d5124_m2_jan00.gocart_internal_rst.20030101_21z.bin copy /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.pchem_internal_rst.20030101_21z.bin => OUTPUT/20030101_12290/d5124_m2_jan00.pchem_internal_rst.20030101_21z.bin The CS regridding is MPI based; submitting job to PBS qsub -W block=true -o /gpfsm/dnb31/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT/C90c_d5124_m2_jan00.upperair.20030101_21z.log.o%j /gpfsm/dnb31/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT/20030101_12290/regrid.j Copied these restart files to the experiment dir and renamed using /gpfsm/dnb31/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT/copy_rst: catchcn_internal_rst catch_internal_rst fvcore_internal_rst gocart_internal_rst lake_internal_rst landice_internal_rst moist_internal_rst pchem_internal_rst saltwater_internal_rst (3) Modify settings and submit the job: cp AGCM.rc AGCM.rc.orig Edit AGCM.rc, added: # CN model time step (DTCN) # ------------------------- DTCN: 5400 # 1.5-hourly Create cap_restart 20030101 210000 Edit CAP.rc END_DATE: 20030103 210000 Edit HISTORY.rc In COLLECTIONS, comment out everything other than "geosgcm_surf" [L57] geosgcm_surf.frequency: 240000 [Lines 206-242] Uncomment geosgcm_surf collection for carbon variables (e.g., CNLAI, ICESOI) Edit gcm_run.j [Line 152] setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds => setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds [This depends on the resolution!] Comment out L1195 "qsub $FILE" in /discover/nobackup/fzeng/Heracles-5_2/src/GMAO_Shared/GEOS_Util/post/gcmpost.script to avoid submitting the post plotting job. sbatch gcm_run.j (4) Crashed: AGCM Date: 2003/01/02 Time: 10:00:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown GEOSgcm.x 00000000009E229F gw_drag_mp_gw_oro 601 gw_drag.F90 GEOSgcm.x 00000000009E1942 gw_drag_mp_gw_int 402 gw_drag.F90 GEOSgcm.x 00000000009CF157 geos_gwdgridcompm 1277 GEOS_GwdGridComp.F90 GEOSgcm.x 00000000009CCA4E geos_gwdgridcompm 705 GEOS_GwdGridComp.F90 4. Set up an experiment without CN: (1) Set up experiment directory: cd Applications/GEOSgcm_App ./gcm_setup Enter the Experiment ID: I10cat Enter a 1-line Experiment Description: I10cat Do you wish to CLONE an old experiment? (Default: NO or FALSE) Enter the Atmospheric Horizontal Resolution code: ----------------------------------------------------------- Lat/Lon Cubed-Sphere ----------------------------------------------------------- b -- 2 deg c48 -- 2 deg c -- 1 deg c90 -- 1 deg d -- 1/2 deg c180 -- 1/2 deg (56-km) e -- 1/4 deg (35-km) c360 -- 1/4 deg (28-km) c720 -- 1/8 deg (14-km) c1440 - 1/16 deg ( 7-km) c90 Enter the Atmospheric Model Vertical Resolution: LM (Default: 72) Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE) Enter the Data_Ocean Horizontal Resolution code: o1 (1 -deg, 360x180 Reynolds) Default o2 (1/4-deg, 1440x720 MERRA-2) o3 (1/8-deg, 2880x1440 OSTIA) CS (Cubed-Sphere OSTIA) o1 Enter the choice of Land Surface Model: use 1 (Default: Catchment Model), 2 (CatchmentCN Model) 1 Do you wish to run the Runoff Routing Model? (Default: NO or FALSE) Do you wish to run GOCART with Actual or Climatological Aerosols? (Enter: A (Default) or C) Enter the GOCART Emission Files to use: MERRA2 (Default), PIESA, CMIP, NR, MERRA2-DD or OPS: (2) Copy restart files from I10catcn: /discover/nobackup/fzeng/geos5_sims/I10catcn > cp -p *rst ../I10cat/. (3) Modify settings and submit the job: Create cap_restart 20030101 210000 Edit CAP.rc END_DATE: 20030103 210000 Edit HISTORY.rc In COLLECTIONS, comment out everything other than "geosgcm_surf" Edit gcm_run.j [Line 152] setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds => setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds [This depends on the resolution!] (4) Also crashed: AGCM Date: 2003/01/02 Time: 11:30:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown libimf.so 00002AAAAADD9ADC Unknown Unknown Unknown GEOSgcm.x 00000000028F61B6 dyn_core_mod_mp_g 2176 dyn_core.F90 GEOSgcm.x 00000000028D8FB0 dyn_core_mod_mp_d 666 dyn_core.F90 GEOSgcm.x 000000000286A8D9 fv_dynamics_mod_m 327 fv_dynamics.F90 GEOSgcm.x 00000000021A4D61 fv_statemod_mp_fv 1230 FV_StateMod.F90 GEOSgcm.x 00000000020AE1DE fvdycorecubed_gri 3760 DynCore_GridCompMod.F90 5. Try choosing Icarus as GCM or DAS tag for outputs: (1) Run regrid.pl: cd /discover/nobackup/fzeng/geos5_sims/I10catcn/restarts_regrid bin/regrid.pl -catchcn Do you want script to select MERRA-2 input restarts for you (y/n)? [n] y Enter OUTPUT directory for new restarts: OUTPUT2 mkdir OUTPUT2 Enter ymd (yyyymmdd): 20030101 Enter hour (hh): 21 ---------------- Atmosphere Grids ---------------- Lat/Lon Grids Cubed-Sphere Grids ------------- ------------------ a = 4 deg C12 C180 C1000 b = 2 deg C24 C360 C1440 c = 1 deg C48 C500 C2880 d = 1/2 deg C90 C720 e = 1/4 deg FVCORE: /archive/users/gmao_ops/MERRA2/gmao_ops/GEOSadas-5_12_4/d5124_m2_jan00/rs/Y2003/M01/d5124_m2_jan00.fvcore_internal_rst.20030101_21z.bin Getting atmosphere grid resolution from INPUT fvcore file ... DONE INPUT atmosphere grid: C180 Enter OUTPUT atmosphere grid: C90 Atmosphere levels ----------------- 72 132 137 144 INPUT atmosphere levels: 72 Enter OUTPUT atmosphere levels: [72] Ocean Grids ----------- c = 360x180 (Reynolds) e = 1440x720 (MERRA-2) f = 2880x1440 (OSTIA) CS = same as atmosphere (OSTIA cubed-sphere) INPUT ocean grid: e Enter OUTPUT ocean grid: [c] OUTPUT ocean grid: c Sample GCM tags --------------- F14 : Fortuna-1_4 ......... Fortuna-1_4_p1 F20 : Fortuna-2_0 ......... Fortuna-2_0 F21 : Fortuna-2_1 ......... Fortuna-2_5_pp2 G10 : Ganymed-1_0 ......... Ganymed-1_0_BETA4 G10p : Ganymed-1_0_p1 ...... Ganymed-1_0_p6 G20 : Ganymed-2_0 ......... Ganymed-2_1_p6 G30 : Ganymed-3_0 ......... Ganymed-3_0_p1 G40 : Ganymed-4_0 ......... Heracles-5_4_p3 HNL : Heracles-NL ......... Heracles-NL ICA : Icarus .............. Icarus Sample DAS tags --------------- 214 : GEOSdas-2_1_4 ....... GEOSdas-2_1_4-m4 540 : GEOSadas-5_4_0 ...... GEOSadas-5_5_3 561 : GEOSadas-5_6_1 ...... GEOSadas-5_7_3_p2 580 : GEOSadas-5_8_0 ...... GEOSadas-5_9_1 591p : GEOSadas-5_9_1_p1 ... GEOSadas-5_9_1_p9 5A0 : GEOSadas-5_10_0 ..... GEOSadas-5_10_0_p1 5B0 : GEOSadas-5_10_0_p2 .. GEOSadas-5_11_0 512 : GEOSadas-5_12_2 ..... GEOSadas-5_16_5 INPUT tag: Ganymed-4_0_MERRA-2 Type 'bcs' to see BCS tags or Enter GCM or DAS tag for outputs: [Ganymed-4_0] Icarus (2) Copy to /discover/nobackup/fzeng/geos5_sims/I10catcn and rename the restart files: cd /discover/nobackup/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT2 cp -p ../OUTPUT/copy_rst . copy_rst Files copied: agcm_import_rst ana_satbang_rst ana_satbias_rst catchcn_internal_rst catch_internal_rst fvcore_internal_rst gocart_internal_rst lake_internal_rst landice_internal_rst moist_internal_rst pchem_internal_rst saltwater_internal_rst NOTE that the previous runs above didn't copy agcm_import_rst, ana_satbang_rst and ana_satbias_rst. (3) In gcm_run.j, use "setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds". qsub gcm_run.j (4) Crashed: forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown GEOSgcm.x 00000000010481E2 geos_catchcngridc 2970 GEOS_CatchCNGridComp.F90 L2970 in GEOS_CatchCNGridComp.F90: where(ITY(:,1) > 0.) Check the values of ITY in /discover/nobackup/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT2/C90c_d5124_m2_jan00.catchcn_internal_rst.20030101_21z.bin. They are all 0!! Values of ITY in /discover/nobackup/fzeng/geos5_sims/I10catcn/restarts_regrid/OUTPUT/C90c_d5124_m2_jan00.catchcn_internal_rst.20030101_21z.bin look correct. OUTPUT: HNL OUTPUT2: Icarus 6. Try using the restart files generated on 5 above in the I10cat run: /discover/nobackup/fzeng/geos5_sims/I10catcn > /bin/cp -p *rst ../I10cat/. cd ../I10cat rm catchcn_internal_rst In gcm_run.j, use "setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds". qsub gcm_run.j Finished two days of simulation successfully. 7. Try using the full set of HNL (Heracles-NL) restarts in I10cat: In gcm_run.j, use "setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds". qsub gcm_run.j Crashed: SALTWATERMAPL_GenericInitialize 1480 SALTWATERInitialize 2739 SURFACEMAPL_GenericInitialize 1404 SURFACEInitialize 3015 PHYSICSMAPL_GenericInitialize 1404 PHYSICS::Initialize 1313 AGCMMAPL_GenericInitialize 1404 AGCMInitialize 1060 GCMMAPL_GenericInitialize 1404 GCMInitialize 1017 GCSMAPL_GenericInitialize 1404 MAPL_Cap 676 application called MPI_Abort(MPI_COMM_WORLD, -21896) - process 0 srun.slurm: Job step aborted: Waiting up to 2 seconds for job step to finish. slurmstepd-borgq127: *** JOB 15832359 CANCELLED AT 2017-05-26T15:42:28 *** on borgq127 forrtl: error (78): process killed (SIGTERM) Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07582C3 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown libmpi.so.12 00002AAAAF3E7572 Unknown Unknown Unknown 8. Try using the full set of HNL restarts in I10catcn: In gcm_run.j, use "setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds". qsub gcm_run.j AGCM Date: 2003/01/02 Time: 10:00:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown GEOSgcm.x 00000000009E229F gw_drag_mp_gw_oro 601 gw_drag.F90 GEOSgcm.x 00000000009E1942 gw_drag_mp_gw_int 402 gw_drag.F90 GEOSgcm.x 00000000009CF157 geos_gwdgridcompm 1277 GEOS_GwdGridComp.F90 GEOSgcm.x 00000000009CCA4E geos_gwdgridcompm 705 GEOS_GwdGridComp.F90 ========== 20170530: 1. The SMAP M09 transient CO2 run finished the 2nd cycle. CO2 years: 1873-1908; meteorology years: 1981-2016. Process the output (covert from tile-space to grid-space): ~/Catchment/SMAP_M09 > tile2grid_ease_spinup_monthly e0004s_transientCO2 Check the July 2016 CO2, LAI, GPP, NPP and NEE output on GrADS. Looks good. Verify the output using ~/Catchment/SMAP_M09/verify_e0004s_transientCO2.gs cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ mv e0004s_transientCO2 e0004s_transientCO2_02 cd e0004s_transientCO2_02 rm -rf lenkf_orig cd output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/ /bin/rm -rf Y198? Y199? Y200? Y201[0-6] (NOTE: DO NOT do this for the last cycle!!! Keep all the restart files!!!) mkdir -p Y1981/M01 cd Y1981/M01 ln -s ../../Y2017/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20170101_0000z e0004s_transientCO2.ens0000.catchcn_ldas_rst.19810101_0000z cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run nedit M09_CN_e0004s_transientCO2.exe & restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_02/output/ Run ldsetup to set up a new experiment: cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2/Linux/bin source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules ./ldsetup setup /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2.bat --runmodel --monthsperjob 10 --landmodel catchCN (NOTE: don't use the "--submit" option because we need to modify the job scripts, see below) Check the executable and restart file: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2 ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69819515 2017-04-27 10:10 build/Linux/bin/LDASsaCN_mpi.x* ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng g0620 80 2017-05-30 11:24 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_02/output/ Copy year_co2.txt and the *all.txt files from previous cycle: cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_02/run cp -p year_co2.txt *all.txt /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run/. cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run Check if these files are correct, particularly year_co2.txt! cat year_co2.txt (to make sure the year of CO2 is correct) 1909 (correct!) nedit *all.txt & Modify the job scripts before submission: Under /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2/run: mkdir ../lenkf_orig (Important: lenkf_orig has to be one level up. Otherwise the lenkf.*.j files in lenkf_orig will be counted by edit_lenkf.csh) cp -p lenkf.*.j ../lenkf_orig/. cp ../../e0004s_transientCO2_02/run/edit_lenkf.csh . ./edit_lenkf.csh Check the lenkf.*.j files and they look correct. qsub lenkf.0.j NOTE: for transientCO2 run, (1) Need to use edit_lenkf.csh to add these lines below to the end of each job file lenkf.N.j (as above): cat year_ts.txt >> year_ts_all.txt cat EEAco2g_ts.txt >> EEAco2g_ts_all.txt cat co2_scalar_ts.txt >> co2_scalar_ts_all.txt qsub lenkf.N+1.j (2) Need to copy edit_lenkf.csh, year_co2.txt, year_ts_all.txt, EEAco2g_ts_all.txt and co2_scalar_ts_all.txt from previous cycle. 2. More GEOS5 test runs: Catchment only (CN off) (1) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds HNL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst ICL rst: agcm_import_rst ana_satbang_rst ana_satbias_rst fvcore_internal_rst gocart_internal_rst moist_internal_rst pchem_internal_rst ssh discover-sp3 interactive.py -A sp3 -n 96 -a g0620 -X --debug setenv ESMADIR /discover/nobackup/fzeng/geos5_code/Icarus-1_0/GEOSagcm/ source $ESMADIR/src/g5_modules cd /discover/nobackup/fzeng/geos5_sims/I10cat ./gcm_run.j Error! Found 167757 tiles in saltwater. Expect to find 161364 tiles. Your restarts are probably for a different ocean. (2) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds rst files same as in (1) AGCM Date: 2003/01/02 Time: 11:45:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown libimf.so 00002AAAAADD9ADC Unknown Unknown Unknown GEOSgcm.x 00000000028F61B6 dyn_core_mod_mp_g 2176 dyn_core.F90 GEOSgcm.x 00000000028D8FB0 dyn_core_mod_mp_d 666 dyn_core.F90 GEOSgcm.x 000000000286A8D9 fv_dynamics_mod_m 327 fv_dynamics.F90 GEOSgcm.x 00000000021A4D61 fv_statemod_mp_fv 1230 FV_StateMod.F90 GEOSgcm.x 00000000020AE1DE fvdycorecubed_gri 3760 DynCore_GridCompMod.F90 (3) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds HNL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst agcm_import_rst ana_satbang_rst ana_satbias_rst pchem_internal_rst Huisheng's rst for her successful run (though wrong GPP output): fvcore_internal_rst gocart_internal_rst moist_internal_rst AGCM Date: 2003/01/02 Time: 09:00:00 Writing: 151 Slices ( 3 Nodes, 1 PartitionRoot) to File: I10cat.geosgcm_surf.20030102_0730z.nc4 AGCM Date: 2003/01/02 Time: 09:15:00 AGCM Date: 2003/01/02 Time: 09:30:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown libimf.so 00002AAAAADD9ADC Unknown Unknown Unknown GEOSgcm.x 00000000028F61B6 dyn_core_mod_mp_g 2176 dyn_core.F90 GEOSgcm.x 00000000028D8FB0 dyn_core_mod_mp_d 666 dyn_core.F90 GEOSgcm.x 000000000286A8D9 fv_dynamics_mod_m 327 fv_dynamics.F90 GEOSgcm.x 00000000021A4D61 fv_statemod_mp_fv 1230 FV_StateMod.F90 GEOSgcm.x 00000000020AE1DE fvdycorecubed_gri 3760 DynCore_GridCompMod.F90 Both (2) and (3) complained about fv_**. It seems that this has to do with fvcore_internal_rst and fvcore_layout.rc (in the exp dir). (4) This ran succesfully before. Now repeat it to confirm. gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds ICL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst agcm_import_rst ana_satbang_rst ana_satbias_rst fvcore_internal_rst gocart_internal_rst moist_internal_rst pchem_internal_rst Successful!! (5) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds HNL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst fvcore_internal_rst ICL rst: agcm_import_rst ana_satbang_rst ana_satbias_rst gocart_internal_rst moist_internal_rst pchem_internal_rst SALTWATERMAPL_GenericInitialize 1480 SALTWATERInitialize 2739 SURFACEMAPL_GenericInitialize 1404 SURFACEInitialize 3015 PHYSICSMAPL_GenericInitialize 1404 PHYSICS::Initialize 1313 AGCMMAPL_GenericInitialize 1404 AGCMInitialize 1060 GCMMAPL_GenericInitialize 1404 GCMInitialize 1017 GCSMAPL_GenericInitialize 1404 MAPL_Cap 676 application called MPI_Abort(MPI_COMM_WORLD, -22040) - process 0 (6) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds Huisheng: catch_internal_rst fvcore_internal_rst geosachem_internal_rst gocart_internal_rst irrad_internal_rst lake_internal_rst landice_internal_rst moist_internal_rst saltwater_internal_rst solar_internal_rst stratchem_internal_rst tr_internal_rst turb_internal_rst Successful! (7) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_Reynolds HNL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst Huisheng's rst for her successful run (though wrong GPP output): fvcore_internal_rst gocart_internal_rst moist_internal_rst not needed - bootstrappable: agcm_import_rst ana_satbang_rst ana_satbias_rst pchem_internal_rst AGCM Date: 2003/01/02 Time: 09:00:00 Writing: 151 Slices ( 3 Nodes, 1 PartitionRoot) to File: I10cat.geosgcm_surf.20030102_0730z.nc4 AGCM Date: 2003/01/02 Time: 09:15:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown libimf.so 00002AAAAADD9ADC Unknown Unknown Unknown GEOSgcm.x 00000000028F61B6 dyn_core_mod_mp_g 2176 dyn_core.F90 GEOSgcm.x 00000000028D8FB0 dyn_core_mod_mp_d 666 dyn_core.F90 GEOSgcm.x 000000000286A8D9 fv_dynamics_mod_m 327 fv_dynamics.F90 GEOSgcm.x 00000000021A4D61 fv_statemod_mp_fv 1230 FV_StateMod.F90 GEOSgcm.x 00000000020AE1DE fvdycorecubed_gri 3760 DynCore_GridCompMod.F90 (8) Done on 20170531. gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds ICL rst: catch_internal_rst lake_internal_rst landice_internal_rst saltwater_internal_rst Huisheng's rst for her successful run (though wrong GPP output): fvcore_internal_rst gocart_internal_rst moist_internal_rst Successful! 3. More GEOS5 test runs: Catchment-CN (CN on) (1) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Icarus/Icarus_Reynolds Huisheng: catch_internal_rst fvcore_internal_rst geosachem_internal_rst gocart_internal_rst irrad_internal_rst lake_internal_rst landice_internal_rst moist_internal_rst saltwater_internal_rst solar_internal_rst stratchem_internal_rst tr_internal_rst turb_internal_rst /discover/nobackup/fzeng/geos5_sims/h52_testHRv2/catchcn_internal_rst Successful, but wrong GPP output (like Huisheng got). ========== 20170531: 1. More GEOS5 test runs: Catchment-CN (CN on) (1) Ran gcm_setup; Ran regrid.pl, chose M2 ocean (i.e. e) gcm_run.j: setenv BCSDIR /discover/nobackup/ltakacs/bcs/Heracles-NL/Heracles-NL_MERRA-2/ HNL M2 ocean rst: catchcn_internal_rst fvcore_internal_rst gocart_internal_rst lake_internal_rst landice_internal_rst moist_internal_rst pchem_internal_rst saltwater_internal_rst AGCM Date: 2003/01/02 Time: 10:00:00 forrtl: error (65): floating invalid Image PC Routine Line Source libirc.so 00002AAAAF9E9961 Unknown Unknown Unknown libirc.so 00002AAAAF9E80B7 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA782 Unknown Unknown Unknown libifcore.so.5 00002AAAB07DA5D6 Unknown Unknown Unknown libifcore.so.5 00002AAAB074740C Unknown Unknown Unknown libifcore.so.5 00002AAAB07585D2 Unknown Unknown Unknown libc.so.6 00002AAAAFC6A910 Unknown Unknown Unknown GEOSgcm.x 00000000009E229F gw_drag_mp_gw_oro 601 gw_drag.F90 GEOSgcm.x 00000000009E1942 gw_drag_mp_gw_int 402 gw_drag.F90 GEOSgcm.x 00000000009CF157 geos_gwdgridcompm 1277 GEOS_GwdGridComp.F90 GEOSgcm.x 00000000009CCA4E geos_gwdgridcompm 705 GEOS_GwdGridComp.F90 The GEOS5 test runs above are all with Icarus-1_0. 2. Now update the code to Icarus-1_0_p1 and do test runs with this updated code: Catchment-CN (CN on) /discover/nobackup/fzeng/geos5_code > mv Icarus-1_0 Icarus-1_0_p1 /discover/nobackup/fzeng/geos5_code/Icarus-1_0_p1/GEOSagcm/src > cvs upd -r Icarus-1_0_p1 Compiled (chose realclean). (1) /discover/nobackup/fzeng/geos5_code/Icarus-1_0_p1/GEOSagcm > cp -p Linux/bin/GEOSgcm.x /discover/nobackup/fzeng/geos5_sims/I10catcnM2ocean/. /discover/nobackup/fzeng/geos5_sims > mv I10catcnM2ocean I10p1catcnM2ocean In gcm_run.j: Changed "Icarus-1_0" to "Icarus-1_0_p1", and "I10catcnM2ocean" to "I10p1catcnM2ocean". MPT ERROR: could not run executable. If this is a non-MPT application, you may need to set MPI_SHEPHERD=true. (SGI MPT 2.14 04/05/16 03:49:22) MPT ERROR: fork_slaves/fork: Cannot allocate memory MPT: shepherd terminated: borgq141 - job aborting GEOSgcm Run Status: -1 /bin/mv: No match. /bin/ls: No match. /bin/ls: No match. /bin/ls: No match. /bin/ls: No match. tar: No match. /bin/ls: No match. /bin/ls: No match. /bin/ls: No match. /bin/mv: missing destination file operand after `/discover/nobackup/fzeng/geos5_sims/I10p1catcnM2ocean/holding/geosgcm_surf' Try `/bin/mv --help' for more information. 15910646 forrtl: severe (259): sequential-access I/O to unit open for direct access, unit 10, file /gpfsm/dnb31/fzeng/geos5_sims/I10p1catcnM2ocean/scratch/saltwater_internal_rst Image PC Routine Line Source rs_numtiles.x 000000000136D4A4 Unknown Unknown Unknown rs_numtiles.x 0000000001394D5A Unknown Unknown Unknown rs_numtiles.x 000000000047EE02 MAIN__ 57 rs_numtiles.F90 rs_numtiles.x 000000000047E72E Unknown Unknown Unknown libc.so.6 00002AAAACA0AC36 Unknown Unknown Unknown rs_numtiles.x 000000000047E639 Unknown Unknown Unknown MPT ERROR: could not run executable. If this is a non-MPT application, you may need to set MPI_SHEPHERD=true. (SGI MPT 2.14 04/05/16 03:49:22) Error! Found tiles in saltwater. Expect to find 1078388 tiles. Your restarts are probably for a different ocean. (2) /discover/nobackup/fzeng/geos5_sims > mv I10p1catcnM2ocean I10p1catcnM2ocean_old Ran gcm_setup: /discover/nobackup/fzeng/geos5_code/Icarus-1_0_p1/GEOSagcm/src/Applications/GEOSgcm_App > ./gcm_setup Ran regrid.pl, chose M2 ocean (i.e. e): cd /discover/nobackup/fzeng/geos5_sims/I10p1catcnM2ocean mkdir M2_restarts cd M2_restarts ln -s /discover/nobackup/fzeng/geos5_code/Icarus-1_0_p1/GEOSagcm/Linux/bin bin/regrid.pl -catchcn # outputs #-------- . CVS tag: Heracles-NL . expid: C90e_d5124_m2_jan00 . date: 20030101 . hour: 21 . atmos grid: 90x540 (C90) . atmos levs: 72 . ocean grid: 1440x720 (e) . BCS tag: Heracles-NL_MERRA-2 . surflay: 50 . rescale: yes . outdir: OUTPUT . workdir: OUTPUT/20030101_29713 srun.slurm: Warning: can't run 1 processes on 4 nodes, setting nnodes to 1 forrtl: severe (259): sequential-access I/O to unit open for direct access, unit 10, file /gpfsm/dnb31/fzeng/geos5_sims/I10p1catcnM2ocean/scratch/saltwater_internal_rst Image PC Routine Line Source rs_numtiles.x 000000000136D4A4 Unknown Unknown Unknown rs_numtiles.x 0000000001394D5A Unknown Unknown Unknown rs_numtiles.x 000000000047EE02 MAIN__ 57 rs_numtiles.F90 3. Read Eunjee's manuscript.