========== 20190701: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. ========== 20190702: 1. Prepared meeting slides about the fire model. 2. Met with Eunjee to talk about the IDS simulation output. 3. June report. 4. Created a python script to read and plot GFED4 data. ========== 20190703: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. GEOSldas: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test2/run In GEOSldas_CF90_test2.exec: # Scale CATCHCN ALBEDO and FPAR # 0--NO # 1-- Scale albedo to match interannually varying MODIS NIRDF and VISDF anomaly # 2-- Scale albedo to match interannually varying MODIS NIRDF and VISDF plus FPAR to match MODIS FPAR CDF SCALE_ALBFPAR: 0 > nedit lenkf.j & #SBATCH --ntasks=672 > nedit LDAS.rc & SCALE_ALBFPAR: 0 DTCN: 10800 NY: 672 Check restart file: > ls -l ../input/restart/ total 0 lrwxrwxrwx 1 fzeng g0620 153 2019-05-29 18:45 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test2/output/CF0090x6C/rs/ens0000/Y1987/M08/GEOSldas_CF90_test2.catchcn_internal_rst.19870801_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-05-24 16:40 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test2/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test2.vegdyn_internal_rst > cat cap_restart 19870801 000000 > nedit CAP.rc & END_DATE: 19880201 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 > interactive.py -A sp3 -n 672 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test2/run > ./lenkf.j total tiles 475330 land_distribute: 949 1114 1367 707 790 878 846 945 1025 1078 1106 1236 1216 1139 1278 1386 1432 1448 1419 1418 1406 1379 1428 1526 1422 1536 1554 1577 1543 1580 1569 1500 1612 1391 1244 1241 1131 1166 1148 1119 1468 1746 1954 1907 1918 1904 2002 2007 2082 2073 2089 2228 2090 2141 2012 2136 2069 2019 1949 1943 1858 1978 1807 1752 1714 1682 1684 1549 1507 1291 953 852 1209 838 834 1032 625 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1005 861 962 750 863 1116 1127 816 756 753 726 825 711 881 975 1429 801 802 940 936 884 898 872 829 831 801 848 945 1051 1134 1053 1077 1192 1546 1625 1728 1824 2159 2358 2447 2487 2497 2744 2680 2651 2820 2692 2779 2778 2787 2760 2870 2880 2846 2927 2873 2763 2686 2707 2494 2486 2532 2589 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2529 2401 2409 2440 2648 2601 2660 2656 2684 2609 2475 2521 2500 2687 2895 3119 3060 3133 3137 3112 3183 3260 3149 3255 3168 3128 3096 2465 2292 2179 2115 2139 2152 2018 1692 1488 1311 1133 984 941 935 1201 1006 824 872 878 1096 1247 853 906 1206 1356 1443 1254 1142 1209 1271 1154 1224 1217 1229 1399 1461 1396 1438 1489 1568 1565 1513 1518 1550 1641 1551 1558 1500 1495 1415 1362 1327 1394 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 823 812 767 798 779 754 862 894 915 990 941 991 1007 971 988 1109 1236 1265 1391 1375 1207 1175 1001 1037 858 778 1208 709 748 198 0 0 0 0 0 0 0 0 0 0 750 974 915 1071 1345 1411 782 775 905 999 1047 1086 1063 1080 1102 1112 964 779 725 727 762 831 844 896 846 1341 756 792 712 745 775 867 890 1027 1228 1204 1280 1252 1412 1532 1797 2007 1917 1975 2082 1959 1962 1928 1880 1828 1846 1689 1585 1570 1624 1552 1417 1402 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 712 721 777 756 850 JMS.rc 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 5 5 6 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 65 3 3 4 15 In MAPL_Shmem: NumCores per Node = 28 NumNodes in use = 24 Total PEs = 672 Crashed using 672 cores. Runs when changing it back to 168 cores. ========== 20190705: 1. Got "Disk quota exceeded" error when trying to process GMAO July forecast data. Archived some data and released some disk space. Processed GMAO July forecast. /discover/nobackup/fzeng/clm4-to-clm4.5/LDAS/tests/clm4.5> rm clm4.5_DE720_04_burn.gdat Found that it's because the clm4.5_fire_DE720 I set up on 20190703 wrote output every 3 hours! 2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. ========== 20190708: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. GEOSldas: Set up GEOSldas_CF90_test3. > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > xdiff GEOSldas_CF90_test2.bat ../GEOSldas_CF90_test2/run/. & [Identical!] > xdiff GEOSldas_CF90_test2.exec ../GEOSldas_CF90_test2/run/. & [Identical!] > cp GEOSldas_CF90_test2.bat GEOSldas_CF90_test3.bat > cp GEOSldas_CF90_test2.exec GEOSldas_CF90_test3.exec > nedit GEOSldas_CF90_test3.bat & ntasks : 672 > nedit GEOSldas_CF90_test3.exec & EXP_ID : GEOSldas_CF90_test3 > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/tranCO2_test2/Linux/bin > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/ > source $ESMADIR/src/g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.bat 11 1980 1986 6 4 creating dir structure creating restart and bc Correct the tile file if it is an old EASE tile format... cmd: ./preprocess_ldas.x correctease /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rc_out//MAPL_CF0090x6C_DE0360xPE0180-Pfafstetter.til Creating f2g.txt.... cmd: ./preprocess_ldas.x c_f2g /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rc_out//CF0090x6C_DE0360xPE0180-Pfafstetter.til LDAS_domain_def.nml /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/clsm/catchment.def GEOSldas_CF90_test3 198101010000 Creating domain..., reading white and black lists if there have ones... Finish domain setup... Writing catparam file : /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_ test3/output/CF0090x6C/rc_out///Y1981/M01/GEOSldas_CF90_test3.ldas_catparam.198 10101_0000z.bin linking bcs... Creating and lining restart... cmd: ./process_rst.csh g0620 GEOSldas_CF90_test3 /discover/nobackup/fzeng/Catchment/M2n5P /discover/nobackup/ltakacs/bcs/Icarus-NL/Icarus-NL_Reynolds/CF0090x6C_DE0360xPE0180/ CF0090x6C_DE0360xPE0180-Pfafstetter.til 2 1 19810101 e0004s_transientCO2_05 SMAP_EASEv2_M09_GLOBAL /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/ 1 0 Please hold on for a while until the restart file is created ..... restart: 1 catcRstFile1: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y1981/M01/e0004s_transientCO2_05.catchcn_internal_rst.19810101_0000 catcRstFile: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 Updating restart path... creating RC Files Optimizing... decomposition of processes.... cmd: ./preprocess_ldas.x optimize /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/input/tile.data 672 total tiles 475330 land_distribute: 949 1114 1367 707 790 878 846 945 1025 1078 1106 1236 1216 1139 1278 1386 1432 1448 1419 1418 1406 1379 1428 1526 1422 1536 1554 1577 1543 1580 1569 1500 1612 1391 1244 1241 1131 1166 1148 1119 1468 1746 1954 1907 1918 1904 2002 2007 2082 2073 2089 2228 2090 2141 2012 2136 2069 2019 1949 1943 1858 1978 1807 1752 1714 1682 1684 1549 1507 1291 953 852 1209 838 834 1032 625 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1005 861 962 750 863 1116 1127 816 756 753 726 825 711 881 975 1429 801 802 940 936 884 898 872 829 831 801 848 945 1051 1134 1053 1077 1192 1546 1625 1728 1824 2159 2358 2447 2487 2497 2744 2680 2651 2820 2692 2779 2778 2787 2760 2870 2880 2846 2927 2873 2763 2686 2707 2494 2486 2532 2589 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2529 2401 2409 2440 2648 2601 2660 2656 2684 2609 2475 2521 2500 2687 2895 3119 3060 3133 3137 3112 3183 3260 3149 3255 3168 3128 3096 2465 2292 2179 2115 2139 2152 2018 1692 1488 1311 1133 984 941 935 1201 1006 824 872 878 1096 1247 853 906 1206 1356 1443 1254 1142 1209 1271 1154 1224 1217 1229 1399 1461 1396 1438 1489 1568 1565 1513 1518 1550 1641 1551 1558 1500 1495 1415 1362 1327 1394 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 823 812 767 798 779 754 862 894 915 990 941 991 1007 971 988 1109 1236 1265 1391 1375 1207 1175 1001 1037 858 778 1208 709 748 198 0 0 0 0 0 0 0 0 0 0 750 974 915 1071 1345 1411 782 775 905 999 1047 1086 1063 1080 1102 1112 964 779 725 727 762 831 844 896 846 1341 756 792 712 745 775 867 890 1027 1228 1204 1280 1252 1412 1532 1797 2007 1917 1975 2082 1959 1962 1928 1880 1828 1846 1689 1585 1570 1624 1552 1417 1402 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 712 721 777 756 850 JMS.rc 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 5 5 6 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 65 3 3 4 15 ['/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc/GEOSldas_CAP.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc/GEOSldas_ExtData.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc/GEOSldas_HIST.rc', '/discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc/GEOSldas_LDAS.rc'] CAP.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run ExtData.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run HIST.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run PE90x540-CF LDAS.rc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/build/Linux/etc /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run creating gcm style batch Run scripts lenkf.j Experiment directory: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3 creating batch Run scripts > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run/ Check restart file: > cat cap_restart 19810101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-08 13:52 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-08 13:52 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst Check executable: > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50817987 2019-05-28 16:14 ../build/Linux/bin/GEOSldas.x* CAP.rc: Changed BEG_DATE: 19810101 000000 END_DATE: 19860101 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 11 to BEG_DATE: 19810101 000000 END_DATE: 19810701 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 MAPL_ENABLE_TIMERS: YES HISTORY.rc: tavg1_2D_lfs_Nx.resolution: 360 181, tavg1_2D_lnd_Nx.resolution: 360 181, Added to L236: 'CNCO2' , 'CATCHCN' , LDAS.rc: Made sure that FPAR scaling is OFF. Add to the end: DTCN: 10800 Did an interactive run: > interactive.py -A sp3 -n 672 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L278 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > ./lenkf.j Using parallel NetCDF for file: ../input/restart/catchcn_internal_rst forrtl: severe (174): SIGSEGV, segmentation fault occurred Image PC Routine Line Source GEOSldas.x 0000000001F2D0B4 Unknown Unknown Unknown libc-2.11.3.so 00002AAAAF322910 Unknown Unknown Unknown GEOSldas.x 0000000000AAD26E mapl_genericmod_m 1013 MAPL_Generic.F90 GEOSldas.x 00000000005511AE geos_ensgridcompm 1722 GEOS_EnsGridComp.F90 GEOSldas.x 0000000001433478 Unknown Unknown Unknown GEOSldas.x 0000000001435586 Unknown Unknown Unknown GEOSldas.x 00000000014456AB Unknown Unknown Unknown GEOSldas.x 0000000001433FEB Unknown Unknown Unknown GEOSldas.x 00000000014D375C Unknown Unknown Unknown > nedit lenkf.j & #SBATCH --ntasks=168 > nedit LDAS.rc & NY: 168 > ./lenkf.j g5_modules: Setting BASEDIR and modules for borgu105 total tiles 475330 land_distribute: 2742 3063 2816 3420 3633 2818 2867 2824 2807 2948 3090 3120 3149 3112 2635 3538 3735 3700 3825 3906 4089 4162 4318 4153 4205 3968 3801 3785 3466 3366 3056 3096 2881 1657 0 0 0 0 0 0 2509 2474 2517 2539 2669 2649 2678 2654 2461 2844 3264 2738 3353 3983 4805 2487 2497 2744 2680 2651 2820 2692 2779 2778 2787 2760 2870 2880 2846 2927 2873 2763 2686 2707 2494 2486 2532 2589 4930 4849 2648 2601 2660 2656 2684 2609 4996 5187 2895 3119 3060 3133 3137 3112 3183 3260 3149 3255 3168 3128 3096 4757 4294 4291 3710 2799 3058 2693 2819 3400 3468 2697 3622 3595 2628 2857 2927 3133 3031 3191 3109 2995 2777 2721 0 0 0 0 0 0 0 3200 3289 2846 2969 3333 2656 2582 3213 2844 1655 2639 2416 2193 2679 3196 3294 2468 2320 2586 2889 2232 2784 2432 2532 2944 3804 3892 4041 3890 3708 3535 3155 3176 2819 0 1937 1879 JMS.rc 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 3 3 3 2 2 2 2 2 2 3 3 3 4 4 4 3 3 3 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 4 4 3 3 3 2 2 3 4 62 29 4 3 3 3 3 3 3 3 4 3 3 2 2 2 2 2 2 2 2 2 2 2 2 2 70 20 In MAPL_Shmem: NumCores per Node = 28 NumNodes in use = 6 Total PEs = 168 In MAPL_InitializeShmem (NodeRootsComm): NumNodes in use = 6 Integer*4 Resource Parameter USE_SHMEM: 1 Integer*4 Resource Parameter HEARTBEAT_DT: 450 Integer*4 Resource Parameter NUM_DT: 0 Integer*4 Resource Parameter DEN_DT: 1 Character Resource Parameter CALENDAR: GREGORIAN NOT using buffer I/O for file: cap_restart Read CAP restart properly, Current Date = 1981/01/01 Current Time = 00:00:00 It's running! > nedit lenkf.j & #SBATCH --ntasks=336 > nedit LDAS.rc & NY: 336 > interactive.py -A sp3 -n 336 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j total tiles 475330 land_distribute: 1464 1966 1497 1724 1970 2184 2452 2417 1386 1432 1448 1419 1418 1406 1379 1428 1526 1422 1536 1554 1577 1543 1580 1569 1500 1612 1391 2485 2297 2267 1468 1746 1954 1907 1918 1904 2002 2007 2082 2073 2089 2228 2090 2141 2012 2136 2069 2019 1949 1943 1858 1978 1807 1752 1714 1682 1684 1549 1507 2244 1535 1364 1866 625 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1580 1604 1799 1701 1572 1479 1536 1437 1848 1603 1876 1782 1701 1632 1793 2185 2130 2738 1625 1728 1824 2159 2358 2447 2487 2497 2744 2680 2651 2820 2692 2779 2778 2787 2760 2870 2880 2846 2927 2873 2763 2686 2707 2494 2486 2532 2589 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 2529 2401 2409 2440 2648 2601 2660 2656 2684 2609 2475 2521 2500 2687 2895 3119 3060 3133 3137 3112 3183 3260 3149 3255 3168 3128 3096 2465 2292 2179 2115 2139 2152 2018 1692 1488 2444 1925 1617 1525 1696 1517 1704 1759 2562 1443 2396 2480 2378 2446 2860 2834 1489 1568 1565 1513 1518 1550 1641 1551 1558 1500 1495 1415 2689 1394 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1635 1565 1533 1756 1905 1932 1978 2097 1236 1265 1391 1375 1207 1175 2038 1636 1208 1133 522 0 1371 1268 1726 1352 1531 1680 2046 2149 2182 2076 1504 1489 1675 1742 1341 1548 1457 1642 1917 2432 1280 1252 1412 1532 1797 2007 1917 1975 2082 1959 1962 1928 1880 1828 1846 1689 1585 1570 1624 1552 1417 1402 0 0 0 0 0 0 0 0 0 1193 1220 1403 JMS.rc 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 0 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 20 34 2 1 1 1 1 1 1 1 1 1 1 1 1 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 2 67 5 18 In MAPL_Shmem: NumCores per Node = 28 NumNodes in use = 12 Total PEs = 336 It's running. Why is DTCN still 5400? This is because there is another line setting DTCN to 5400 earlier in LDAS.rc. > qsub lenkf.j Need to talk to Weiyuan. ========== 20190709: 1. GEOSldas: trying different DTCN and number of cores and see how long it takes to finish 6 months of simulation > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > nedit LDAS.rc & L28: DTCN: 10800 > interactive.py -A sp3 -n 336 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running, and DTCN printed out is 10800. > qsub lenkf.j In the GEOSldas_CF90_test3 run (with DTCN=5400) that I did on 20190708, the cap_restart and the files in input/restart were not updated at the end of the 6-month simulation. Therefore, when I changed DTCN to 10800 and run it again this morning, it started from 19810101. This could be due to the "exit" I added to L280 in lenkf.j. Same for now after this GEOSldas_CF90_test3 run with DTCN=10800 is done: /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > cat cap_restart 19810101 000000 /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ls -l ../input/restart/ total 0 lrwxrwxrwx 1 fzeng g0620 153 2019-07-08 13:52 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-08 13:52 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst The GEOSldas_CF90_test3 run I did on 20190708 and this morning both used 336 cores. Now do another run with 168 cores and compare the output to see if number of cores has an impact on model output (suggested by Weiyuan). /discover/nobackup/fzeng/Catchment/M2n5P > cp -pr GEOSldas_CF90_test3 GEOSldas_CF90_test3_old > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > nedit lenkf.j & L11: #SBATCH --ntasks=168 Weiyuan said no need to change the value of NY in LDAS.rc. > interactive.py -A sp3 -n 168 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running, using 168 cores and DTCN printed out is 10800. > qsub lenkf.j Will compare the output between GEOSldas_CF90_test3 and GEOSldas_CF90_test3_old. 2. Met with Eunjee. 3. Wrote /discover/nobackup/fzeng/Catchment/M2n5P/m0002/exp_description to document the offline experiments that I did for Eunjee's drought study. 4. Organized some files: Data in /discover/nobackup/fzeng/forcing/data/M2n5P are for the study that we tried to look at how removal of the temporal variabilities in temperature and precipitation would affect C flux. Since we no longer pursue this study and the data has been archived in /archive/u/fzeng/data/forcing/data/M2n5P, removed the data from my nobackup disk. /discover/nobackup/fzeng/forcing/data > rm -rf M2n5P Data in /discover/nobackup/fzeng/forcing/data/GEOS_S2S is a small part of data in /discover/nobackup/fzeng/hindcasts/GEOS_S2S and is redundant. Removed it. /discover/nobackup/fzeng/forcing/data > rm -rf GEOS_S2S Now there is nothing in /discover/nobackup/fzeng/forcing/data. Removed the "data" directory. /discover/nobackup/fzeng/forcing > rmdir data /discover/nobackup/fzeng/forcing > tarem code /discover/nobackup/fzeng/forcing > ls code/ tarem* /discover/nobackup/fzeng > rm -rf forcing /discover/nobackup/fzeng > rm M2flx M2tmp s2s_vars tmpsw.txt tile2grid.out CNgrid.list geos5_CNData.txt #modifications# blue.mu.driver.tar M2clim.mat Pclim.mat P_M2_diff.mat ========== 20190710: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. Met with Randy, Melanie and Eunjee: 11am-12pm, 2pm-3pm 3. GEOSldas_CF90_test3: DTCN=10800 and 168 cores. GEOSldas_CF90_test3_old: DTCN=10800 and 336 cores. There is no output in either GEOSldas_CF90_test3 or GEOSldas_CF90_test3_old for 198101 through 198106 though. Maybe because the "exit" in L280 ofs lenkf.j? Re-do the runs for just 2 months each: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run/ > cat cap_restart 19810101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-08 13:52 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-08 13:52 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst > nedit CAP.rc & BEG_DATE: 19810101 000000 END_DATE: 19810301 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 > mv lenkf.j.orig lenkf.j > nedit lenkf.j & #SBATCH --ntasks=168 > interactive.py -A sp3 -n 168 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j When done: > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas_CF90_test3 GEOSldas_CF90_test3_168cores > mv GEOSldas_CF90_test3_old GEOSldas_CF90_test3 > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run/ > cat cap_restart 19810101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-08 13:52 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-08 13:52 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst > nedit CAP.rc & BEG_DATE: 19810101 000000 END_DATE: 19810301 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 > mv lenkf.j.orig lenkf.j > nedit lenkf.j & #SBATCH --ntasks=336 > interactive.py -A sp3 -n 336 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j When done: > cd /discover/nobackup/fzeng/Catchment/M2n5P > mv GEOSldas_CF90_test3 GEOSldas_CF90_test3_336cores module load other/nco-4.6.8-gcc-5.3-sp3 [so that ncdiff works, see NCCS email on 20181116] > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1 > ncdiff jan01.monthly.198207.nc4 /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS_old/GEOS5/RAW_GEOS5.V2/Monthly/1982/jan01/ens1/geosgcm_vis2d/jan01.geosgcm_vis2d.monthly.198207.nc4 NEWmOLD > ncview NEWmOLD & All the fields in common "LWS", "PRECTOT", "PS", "SLRSF" and "SPEED" are identical between the new and old monthly output, i.e. all zeros in NEWmOLD. Great! > rm NEWmOLD ========== 20190711: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. Prepared meeting slides. 3. Met with Randy and Eunjee: 2pm-4pm ========== 20190712: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. Organized some data in my NOBACKUP. > cd /discover/nobackup/fzeng > mkdir data > mv noaaCO2 data > mv princeton data > mv FLUXNET data > mv hindcasts data > cd data > mkdir LandCover > cd LandCover Copied the land cover data I downloaded from local computer to Discover. ========== 20190715: Discover down for maintenance the whole day. ========== 20190716: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. For hindcast data bias correction: bias correct 2011 monthly hindcast data. Referred to notes on 20190607 and 20190610. Started from 3pm. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2011 FCST_EYR=2011 > PART2_BCSD-Calc.H.sh This took me about 10 minutes of my time. Search for the steps took me another 10 minutes. 3. GEOSldas: (1) Using 864 cores: > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/tranCO2_test2/Linux/bin > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/ > source $ESMADIR/src/g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.bat > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run/ Check restart file: > cat cap_restart 19810101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 11:22 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test3.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 11:22 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst Check executable: > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50817987 2019-05-28 16:14 ../build/Linux/bin/GEOSldas.x* CAP.rc: Changed BEG_DATE: 19810101 000000 END_DATE: 19860101 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 11 to BEG_DATE: 19810101 000000 END_DATE: 19860101 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 MAPL_ENABLE_TIMERS: YES HISTORY.rc: tavg1_2D_lfs_Nx.resolution: 360 181, tavg1_2D_lnd_Nx.resolution: 360 181, Added to L236: 'CNCO2' , 'CATCHCN' , LDAS.rc: Made sure that FPAR scaling is OFF. DTCN: 10800 NX: 6 > nedit lenkf.j & #SBATCH --ntasks=864 Commented out L85-97: #if( -e IMS.rc ) then # set oldtasks = `head -n 1 IMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif #if( -e JMS.rc ) then # set oldtasks = `head -n 1 JMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif Commented out L100-106: #if ( "$gridname" == "CF" ) then # set new_ny = `echo "NY: "$numprocs` # sed -i "/NY:/c\\$new_ny" LDAS.rc #else # set new_nx = `echo "NX: "$numprocs` # sed -i "/NX:/c\\$new_nx" LDAS.rc #endif > cp lenkf.j lenkf.j.orig > nedit lenkf.j & Add "exit" after L278 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" Did an interactive run: > interactive.py -A sp3 -n 864 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped after it finished a few days. > mv lenkf.j.orig lenkf.j > ./ldas_batchrun.j (2) Using 144 cores: > cd /discover/nobackup/fzeng/Catchment/M2n5P/run > nedit GEOSldas_CF90_test3.exec & EXP_ID : GEOSldas_CF90_test4 > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/tranCO2_test2/Linux/bin > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/ > source $ESMADIR/src/g5_modules > ./ldas_setup setup --runmodel /discover/nobackup/fzeng/Catchment/M2n5P/ /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.exec /discover/nobackup/fzeng/Catchment/M2n5P/run/GEOSldas_CF90_test3.bat > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run/ Check restart file: > cat cap_restart 19810101 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 12:02 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/Y1981/M01/GEOSldas_CF90_test4.catchcn_internal_rst.19810101_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 12:02 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test4.vegdyn_internal_rst Check executable: > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50817987 2019-05-28 16:14 ../build/Linux/bin/GEOSldas.x* CAP.rc: Changed BEG_DATE: 19810101 000000 END_DATE: 19860101 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 11 to BEG_DATE: 19810101 000000 END_DATE: 19860101 000000 JOB_SGMT: 00000600 000000 NUM_SGMT: 1 MAPL_ENABLE_TIMERS: YES HISTORY.rc: tavg1_2D_lfs_Nx.resolution: 360 181, tavg1_2D_lnd_Nx.resolution: 360 181, Added to L236: 'CNCO2' , 'CATCHCN' , LDAS.rc: Made sure that FPAR scaling is OFF. DTCN: 10800 > ./ldas_batchrun.j 4. Compare: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3_168cores/output/CF0090x6C/cat/ens0000/Y1981/M01 > module load other/nco-4.6.8-gcc-5.3-sp3 > ncdiff GEOSldas_CF90_test3.tavg1_2D_lnd_Nx.monthly.198101.nc4 /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3_336cores/output/CF0090x6C/cat/ens0000/Y1981/M01/GEOSldas_CF90_test3.tavg1_2D_lnd_Nx.monthly.198101.nc4 168m336 > ncview 168m336 & Found quite large differences. Created ~/python/clm4-to-clm4.5/compare_results_diffNoCores.py to double check the output and found the same thing. Did I do something wrong? Will compare the output from GEOSldas_CF90_test3 (using 864 cores) and GEOSldas_CF90_test4 (using 144 cores). > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/cat/ens0000/Y1981/M01 > module load other/nco-4.6.8-gcc-5.3-sp3 > ncdiff GEOSldas_CF90_test3.tavg1_2D_lnd_Nx.monthly.198101.nc4 /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/cat/ens0000/Y1981/M01/GEOSldas_CF90_test4.tavg1_2D_lnd_Nx.monthly.198101.nc4 test3mtest4 > ncview test3mtest4 & The difference is 0 for CNGPP, CNNEE and CNLAI that I checked, and I assume it is also 0 for the other fields. Also checked using ~/python/clm4-to-clm4.5/compare_results_diffNoCores.py and the difference is also 0. This is great! ========== 20190717: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. IDS meeting. 3. Met with Randy, Melanie and Eunjee. 4. For hindcast data bias correction: downscale 2011 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Referred to notes on 20190619. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2011 # SYR and EYR should be the same as FCST_EYR=2011 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh Took me <2 minutes. 5. GEOSldas: Tried different number of cores and see how long a 6-month simulation takes. (1) > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > nedit LDAS.rc & NX: 7 > nedit lenkf.j & #SBATCH --ntasks=1008 > nedit lenkf.j & Add "exit" after L278 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" Did an interactive run: > interactive.py -A sp3 -n 1008 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped after it finished a few days. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j NX=30,15,10,8: salloc: error: Job submit/allocate failed: Job violates accounting/QOS policy (job submit limit, user's size and/or time limits) Traceback (most recent call last): File "/home/fzeng/bin/interactive.py", line 163, in main() File "/home/fzeng/bin/interactive.py", line 129, in main sp.check_call(cmd.split()) File "/usr/local/other/SLES11/SIVO-PyD/1.10.0/lib/python2.7/subprocess.py", line 504, in check_call raise CalledProcessError(retcode, cmd) subprocess.CalledProcessError: Command '['/usr/bin/ssh', '-XYqt', '-p2255', 'discover15', 'salloc', '--ntasks=4320', '--constraint=sp3', '--account=g0620', '--mail-type=BEGIN', '--partition=compute', '--qos=debug', '--time=1:00:00']' returned non-zero exit status 1 NX=7 works. Weiyuan said: For debug qos, you cannot have that many cores. You can simply remove --qos=debug, but it may take you sometime to wait. You can submit that interactive request and wait. Make the time 8:00:00 , so it will keep open until you are off work. Killed the job with NX=7. It's taking much longer than NX=6 because 90 is not dividable by 7. > cat cap_restart 19810701 000000 Good! > ls -l ../input/restart/ total 0 lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 14:16 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M07/GEOSldas_CF90_test3.catchcn_internal_rst.19810701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 11:22 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst Good. (2) > nedit LDAS.rc & NX: 30 > nedit lenkf.j & #SBATCH --ntasks=4320 > nedit lenkf.j & Add "exit" after L278 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > interactive.py -A sp3 -n 4320 -a g0620 -t 8:00:00 > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's hanging at the point of reading some input file (e.g. tile file?) even after 2 hours. Stopped this run. (3) 19810701 to 19811231. > nedit LDAS.rc & NX: 10 > nedit lenkf.j & #SBATCH --ntasks=1440 > interactive.py -A sp3 -n 1440 -a g0620 -t 4:00:00 > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped it. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j Hanged there for >12 hours and stopped due to time limit! (4) 19820101 to 19820630: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > nedit LDAS.rc & NX: 15 > nedit lenkf.j & #SBATCH --ntasks=2160 Commented out L85-97: #if( -e IMS.rc ) then # set oldtasks = `head -n 1 IMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif #if( -e JMS.rc ) then # set oldtasks = `head -n 1 JMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif Commented out L100-106: #if ( "$gridname" == "CF" ) then # set new_ny = `echo "NY: "$numprocs` # sed -i "/NY:/c\\$new_ny" LDAS.rc #else # set new_nx = `echo "NX: "$numprocs` # sed -i "/NX:/c\\$new_nx" LDAS.rc #endif Add "exit" after L278 "mpirun -map-by core --mca btl ^vader -np $numprocs $GEOSBIN/GEOSldas.x" > interactive.py -A sp3 -n 2160 -a g0620 -t 1:00:00 > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > ./lenkf.j It's hanging there for quite some time. Need to leave. Stopped it. Assume that it will run. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j It took 1.75 hours to finish 6 months of simulation. 6. For comparison with 5 above: 19810701 to 19811231: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > ./ldas_batchrun.j ========== 20190718: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 2. For hindcast data bias correction: combine the individual fields into one file for each month. Referred to notes on 20190620. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2011 FCST_EYR=2011 > PART4_Combine.H.sh Took me ~5 minutes. When done, used "ncview" to check the final output. Took me ~5 minutes. 3. For IDS project: set up the 4 ensemble experiments for 2011. Referred to notes on 20190625. Took me ~1 hour (i.e. 50 minutes). /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ls e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2011/M01 e0004s_transientCO2.ens0000.catchcn_ldas_rst.20110101_0000z /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > ls e0004s_transientCO2_05/output/SMAP_EASEv2_M09_GLOBAL/rs/ens0000/Y2015/M01 e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150101_0000z We have restarts for both 20110101 and 20150101. Good. Take e0004s_transientCO2_BChindcast_2011Jan01_ens1 as an example. (1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2011Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2011-01-01-00-00-00 end_time = 2011-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2011_ens1 (2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2011Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-07-18 11:01 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4) Create year_co2.txt: For 2011 ones: > cd run > echo 2011 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2011 [Correct!] For 2015 ones: > cd run > echo 2015 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2015 [Correct!] (5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/fzeng/Catchment/BChindcast/2014Jan01_ens1/run/LDAS.rc. (7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2011Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2011/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20110101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2011/jan01/ens1/GEOS5.201101.nc4 Stopped it at date_time_new 20110101_120730z. (8) Submit the job: > qsub lenkf.0.j 4. For hindcast data bias correction: bias correct 2015 monthly hindcast data. Referred to notes on 20190607 and 20190610. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART2_BCSD-Calc.H.sh & FCST_SYR=2015 FCST_EYR=2015 > PART2_BCSD-Calc.H.sh Took me ~5 minutes. 5. Tried Weiyuan's new method to make GEOSldas run faster. > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src/GEOSldas_GridComp/GEOSmetforce_GridComp > cp -p LDAS_Forcing.F90 LDAS_Forcing.F90.orig > nedit LDAS_Forcing.F90 & Updated the two subroutines GEOS_openfile and LDAS_GetVar to be the same as those in Weiyuan's /gpfsm/dnb32/wjiang/develop_ldas/GEOSldas_UNSTABLE/src/GEOSldas_GridComp/GEOSmetforce_GridComp/LDAS_Forcing.F90 > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN > source g5_modules > make -j 8 install > cd .. > mkdir -p exec/tranCO2_test3 > /bin/cp -pr Linux exec/tranCO2_test3/. (1) > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3 > mv build build2 > ln -s /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/tranCO2_test3 build 19810701 to 19811231. > cd run > nedit LDAS.rc & NX: 10 > nedit lenkf.j & #SBATCH --ntasks=1440 > interactive.py -A sp3 -n 1440 -a g0620 -t 4:00:00 Hanging there. There may be something wrong with Discover since Weiyuan is having the same issue. These below have not been done: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped it. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j 6. For Huishent, thought about how to convert CF90 ndep from tile to grid. ========== 20190719: 1. For hindcast data bias correction: downscale 2015 bias corrected hindcast data from monthly to daily, and from daily to 6-hourly. Referred to notes on 20190619. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART3_TmpDisagg.H.sh & FCST_SYR=2015 # SYR and EYR should be the same as FCST_EYR=2015 # the monthly BCSD files with start and end years > PART3_TmpDisagg.H.sh Took me about 10 minutes to do this and check the log files from bias-correction of the monthly data. 2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. 3. For IDS project: process the four 2011 ensemble run output and check the output on GrADS. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2011Jan01_ens1 2011 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2011Jan01_ens2 2011 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2011Jan01_ens3 2011 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2011Jan01_ens4 2011 > mv e0004s_transientCO2_BChindcast_monthly_2014.ctl e0004s_transientCO2_BChindcast_monthly.ctl Include all the 4 years (2011, 2014, 2015, 2016) in this control file. Checked July GPP and NEE of the 4 ensembles on GrADS. Looks correct. Took me 20 minutes. 4. For hindcast data bias correction: combine the individual fields into one file for each month. Referred to notes on 20190620. > cd ~/geos5/FORECASTS_BCSD/Fanwei/BCSD/FAME_Dec_V2 > nedit PART4_Combine.H.sh & FCST_SYR=2015 FCST_EYR=2015 > PART4_Combine.H.sh Took me 2 minutes. 5. Tried Weiyuan's new method to make GEOSldas run faster. Continued from 5 above yesterday. (1) > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run 19810701 to 19811231. > nedit LDAS.rc & NX: 6 > nedit lenkf.j & #SBATCH --ntasks=864 Added "exit" to L280. > interactive.py -A sp3 -n 864 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped it. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j (2) > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4 > mv build build2 > ln -s /gpfsm/dnb31/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/exec/tranCO2_test3 build 19820701 to 19821231. > cd run > nedit LDAS.rc & NX: 15 > nedit lenkf.j & #SBATCH --ntasks=2160 Added "exit" to L280. > interactive.py -A sp3 -n 2160 -a g0620 -t 2:00:00 It was hanging there. Cancelled it. > nedit lenkf.j & Remove "exit" added to L280 above. > ./ldas_batchrun.j 6. For IDS project: set up the 4 ensemble experiments for 2015. Referred to notes on 20190625. Took me ~40 minutes. Take e0004s_transientCO2_BChindcast_2015Jan01_ens1 as an example. (1) Prepare files and run directory for experiment setup: > cd /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run > nedit M09_CN_e0004s_transientCO2_BChindcast.exe M09_e0004s_transientCO2_BChindcast.bat & M09_CN_e0004s_transientCO2_BChindcast.exe: exp_id = e0004s_transientCO2_BChindcast_2015Jan01_ens1 exp_domain = SMAP_EASEv2_M09_GLOBAL N_ens = 1 start_time = 2015-01-01-00-00-00 end_time = 2015-10-01-00-00-00 force_dtstep = 21600 restart_path = /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ restart_domain = SMAP_EASEv2_M09_GLOBAL restart_id = e0004s_transientCO2 met_tag = HindcastBC_netcdf met_path = /discover/nobackup/projects/gmao/geos_carb/fzeng/FORECASTS/GEOS5/CLIM/GEOS5v2/BCSD_Final/6-hourly/ M09_e0004s_transientCO2_BChindcast.bat: job-name = BChindcast_2015_ens1 (2) Run ldsetup: > cd /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/exec/e0004s_transientCO2_BChindcast/Linux/bin > source /discover/nobackup/fzeng/LDASsa_m3-16_0_p2_CatchCatchCN_for_MERRA3/src/g5_modules > ./ldsetup setup /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09 /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_CN_e0004s_transientCO2_BChindcast.exe /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/run/M09_e0004s_transientCO2_BChindcast.bat --runmodel --monthsperjob 10 --landmodel catchCN (3) Check the executable, restart file and create year_co2.txt: > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2015Jan01_ens1 > ls -l build/Linux/bin/LDASsaCN_mpi.x (to make sure the executable is the right one) -rwxr-xr-x 1 fzeng g0620 69619628 2019-06-26 11:25 build/Linux/bin/LDASsaCN_mpi.x* > ls -l input/restart/ (to make sure the restart file is the right one) lrwxrwxrwx 1 fzeng s1460 80 2019-07-19 13:34 output -> /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_05/output/ (4) Create year_co2.txt: For 2015 ones: > cd run > echo 2015 > year_co2.txt > cat year_co2.txt (to make sure the year of CO2 is correct) 2015 [Correct!] (5) Modify the restart_id in the first job script: > nedit lenkf.0.j & [No change made!] -restart_path ../input/restart/output \ -restart_domain SMAP_EASEv2_M09_GLOBAL \ -restart_id e0004s_transientCO2 \ (6) Create LDAS.rc: > nedit LDAS.rc & INIT_MON: jan01 ENS_NUM: 1 NOTE: The space is important, so just copy and paste from /discover/nobackup/fzeng/Catchment/BChindcast/2014Jan01_ens1/run/LDAS.rc. (7) I moved the noaaCO2 data to a different directory a few days ago. Created a symbolic link so the existing executable still works. /discover/nobackup/fzeng > ln -s data/noaaCO2 . (8) Did an interactive run to make sure it works: > interactive.py -A sp3 -n 140 -a g0620 -X --debug > cd /discover/nobackup/projects/gmao/geos_carb/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_transientCO2_BChindcast_2015Jan01_ens1/run > ./lenkf.0.j It's running, and using the right restart file and meteorology data. Reading restart file ../input/restart/output/SMAP_EASEv2_M09_GLOBAL//rs/ens0000//Y2015/M01/e0004s_transientCO2.ens0000.catchcn_ldas_rst.20150101_0000z get_forcing(): bias-corrected GEOS-5 hindcast forcing data set opening../input/met_forcing/6-hourly//2015/jan01/ens1/GEOS5.201501.nc4 Stopped it at date_time_new 20150101_060000z. (8) Submit the job: > qsub lenkf.0.j ========== 20190722: 1. For IDS project: process the four 2015 ensemble run output and check the output on GrADS. > cd ~/Catchment/SMAP_M09 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2015Jan01_ens1 2015 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2015Jan01_ens2 2015 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2015Jan01_ens3 2015 > tile2grid_ease_BChindcast_monthly e0004s_transientCO2_BChindcast_2015Jan01_ens4 2015 > mv e0004s_transientCO2_BChindcast_monthly_2014.ctl e0004s_transientCO2_BChindcast_monthly.ctl Include all the 4 years (2011, 2014, 2015, 2016) in this control file. Checked July GPP and NEE of the 4 ensembles on GrADS. Looks correct. Took me 10 minutes. 2. Met with Eunjee to discuss the results of the simulations I have done for the IDS project. 3. Email Kristi, Abheera and Shrad about the order of their names on Eunjee's AMS abstract. 4. The GEOSldas_CF90_test3 and GEOSldas_CF90_test4 got time out for some reason that I don't understand. GEOSldas_CF90_test3: LDAS ERROR (3000) from repair_forcing: Tair too low slurmstepd-borgr008: *** JOB 33483398 CANCELLED AT 2019-07-20T05:50:05 DUE TO TIME LIMIT on borgr008 *** GEOSldas_CF90_test4: Hanged there when it's reading the restart file. Just re-submit the jobs and see how they go. > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > cat cap_restart 19810701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 14:16 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M07/GEOSldas_CF90_test3.catchcn_internal_rst.19810701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 11:22 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst > ./ldas_batchrun.j > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > cat cap_restart 19820701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-17 22:39 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/Y1982/M07/GEOSldas_CF90_test4.catchcn_internal_rst.19820701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 12:02 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test4.vegdyn_internal_rst > ./ldas_batchrun.j ========== 20190723: 1. Responded to Lei's questions about the bias-corrected hindcast data. 2. Improved the python scripts about calculating lats and lons for meshgrid plotting. 3. Met with Lesley and Eunjee about the IDS project. 4. For the GEOSldas_CF90_test3 and GEOSldas_CF90_test4 runs: The run using 2160 cores again hung there when it was reading the restart file until time was out. The run using 864 cores crashed at “AGCM Date: 1981/07/01 Time: 03:00:00”. The good thing is it provides more information this time: AGCM Date: 1981/07/01 Time: 03:00:00 Time ----------------------------------- 1981-07-01T03:00:00 end Time ------------------------------- get_forcing(): assuming GEOS-5 forcing data set opening file: ../input/met_forcing/MERRA2_land_forcing/precip_corr_CPCUGPCP22cl im_MERRA2_BMTXS//MERRA2_100/diag/Y1981/M07/MERRA2_100.tavg1_2d_lfo_Nx_corr.1981 0701_0330z.nc4 forrtl: severe (66): output statement overflows record, unit -5, file Internal List-Directed Write Image PC Routine Line Source GEOSldas.x 0000000001F1FE03 Unknown Unknown Unknown GEOSldas.x 0000000001F7F487 Unknown Unknown Unknown GEOSldas.x 0000000001F7CFC5 Unknown Unknown Unknown GEOSldas.x 0000000000466000 ldas_forcemod_mp_ 5123 LDAS_Forcing.F90 GEOSldas.x 0000000000465D10 ldas_forcemod_mp_ 5034 LDAS_Forcing.F90 GEOSldas.x 0000000000464960 ldas_forcemod_mp_ 353 LDAS_Forcing.F90 GEOSldas.x 000000000045C007 geos_metforcegrid 912 GEOS_MetforceGridComp.F90 Weiyuan's suggestions: ------- I am not sure why the code breaks there. The thing I can think of is completely clean up the build by “make realclean” and then make again. You also may change character(10), private :: tmpstring10 to character(40), private :: tmpstring10 But I think even the change is working, something must have been wrong at that point. Re-make the build should solve the problem. ------- First, try re-compiling without making the change. > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN > source g5_modules > make realclean > make -j 8 install > cd .. > /bin/cp -pr Linux exec/tranCO2_test3/. Re-submit the jobs and see how they go. > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50814095 2019-07-23 13:04 ../build/Linux/bin/GEOSldas.x* > cat cap_restart 19810701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 14:16 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M07/GEOSldas_CF90_test3.catchcn_internal_rst.19810701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 11:22 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst > ./ldas_batchrun.j > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50814095 2019-07-23 13:04 ../build/Linux/bin/GEOSldas.x* > cat cap_restart 19820701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-17 22:39 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/Y1982/M07/GEOSldas_CF90_test4.catchcn_internal_rst.19820701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 12:02 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test4.vegdyn_internal_rst > ./ldas_batchrun.j 5. Prepared meeting slides. ========== 20190724: 1. Git training: 8:30 - 10:45 2. Read and comment on Eunjee's AMS abstract. 3. Prepared meeting slides. 4. Updated LDAS_Forcing.F90, GEOS_MetforceGridComp.F90 and Shared/LDAS_RepairForcing.F90 with Weiyuan's help. Re-compiled the code. > cd .. > /bin/cp -pr Linux exec/tranCO2_test3/. > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > nedit lenkf.j & #SBATCH --ntasks=120 Change #if( -e JMS.rc ) then # set oldtasks = `head -n 1 JMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif to if( -e JMS.rc ) then set oldtasks = `head -n 1 JMS.rc` if( $numprocs != $oldtasks) then $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing endif endif And added "exit" right after this block. > nedit LDAS.rc & NY 120 NX 1 > ./lenkf.j Copy JMS.rc from scratch to run. > nedit lenkf.j & #SBATCH --ntasks=720 Change if( -e JMS.rc ) then set oldtasks = `head -n 1 JMS.rc` if( $numprocs != $oldtasks) then $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing endif endif to #if( -e JMS.rc ) then # set oldtasks = `head -n 1 JMS.rc` # if( $numprocs != $oldtasks) then # $GEOSBIN/preprocess_ldas.x optimize ../input/tile.data $numprocs nothing nothing nothing # endif #endif Remove "exit" added above. > nedit LDAS.rc & NY 120 NX 6 > ./ldas_batchrun.j Do something similar to GEOSldas_CF90_test4. 5. Met with Randy and Eunjee. 6. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. ========== 20190729: 1. Deal with emails (took July 25-26 off). 2. Reviewed Melanie's AGU abstract and provided comments. 3. Tried Git. 4. GEOSldas: The GEOSldas_CF90_test3 run using 720 cores crashed again. This time I got a different error message: get_forcing(): assuming GEOS-5 forcing data set opening file: ../input/met_forcing/MERRA2_land_forcing/precip_corr_CPCUGPCP22cl im_MERRA2_BMTXS//MERRA2_100/diag/Y1981/M07/MERRA2_100.tavg1_2d_lfo_Nx_corr.1981 0701_1830z.nc4 AGCM Date: 1981/07/01 Time: 18:07:30 Time ----------------------------------- 1981-07-01T18:07:30 end Time ------------------------------- forrtl: error (73): floating divide by zero Image PC Routine Line Source GEOSldas.x 0000000001F2C9FF Unknown Unknown Unknown libc-2.11.3.so 00002AAAAF322910 Unknown Unknown Unknown GEOSldas.x 000000000080F7A2 compute_rc_mod_mp 646 compute_rc.F90 GEOSldas.x 000000000080E0A2 compute_rc_mod_mp 363 compute_rc.F90 GEOSldas.x 0000000000764041 geos_catchcngridc 6812 GEOS_CatchCNGridComp.F90 GEOSldas.x 0000000000735888 geos_catchcngridc 4544 GEOS_CatchCNGridComp.F90 The GEOSldas_CF90_test4 run which used 1800 cores hung there when it was reading the restart file. Following Weiyuan's suggestion: > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src/GEOSldas_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatchCN_GridComp > nedit GEOS_CatchCNGridComp.F90 & Changed the line 5584 of GEOS_CatchCNGridComp.F90 NTILES = size(PS) to NTILES = size(PS,1) > cd /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN/src > setenv ESMADIR /discover/nobackup/fzeng/offline_code/GEOSldas_m4-17_UCatchCN > source g5_modules > make -j 8 install > cd .. > /bin/cp -pr Linux exec/tranCO2_test3/. Re-submit the jobs and see how they go. (1) GEOSldas_CF90_test3: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50814095 2019-07-29 15:48 ../build/Linux/bin/GEOSldas.x* > cat cap_restart 19810701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-16 14:16 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/Y1981/M07/GEOSldas_CF90_test3.catchcn_internal_rst.19810701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 11:22 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test3.vegdyn_internal_rst > interactive.py -A sp3 -n 720 -a g0620 -X --debug > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test3/run > ./lenkf.j It's running. Stopped it at 1981-07-02T03:00:00 > ./ldas_batchrun.j (2) GEOSldas_CF90_test4: > cd /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/run > ls -l ../build/Linux/bin/GEOSldas.x -rwxr-xr-x 1 fzeng g0620 50814095 2019-07-29 15:48 ../build/Linux/bin/GEOSldas.x* > cat cap_restart 19820701 000000 > ls -l ../input/restart/ lrwxrwxrwx 1 fzeng g0620 153 2019-07-17 22:39 catchcn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/Y1982/M07/GEOSldas_CF90_test4.catchcn_internal_rst.19820701_0000 lrwxrwxrwx 1 fzeng g0620 128 2019-07-16 12:02 vegdyn_internal_rst -> /discover/nobackup/fzeng/Catchment/M2n5P/GEOSldas_CF90_test4/output/CF0090x6C/rs/ens0000/GEOSldas_CF90_test4.vegdyn_internal_rst > interactive.py -A sp3 -n 1800 -a g0620 -t 01:00:00 Couldn't get the cores. > ./ldas_batchrun.j 5. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. ========== 20190730: 1. Both GEOSldas_CF90_test3 and GEOSldas_CF90_test4 crashed. Talked to Weiyuan. He will think about this. 2. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019. ========== 20190731: 1. Worked on upgrading CLM4 to CLM4.5. See notes in /discover/nobackup/fzeng/clm4-to-clm4.5/notes/notes_daily_2019.