20150701: 0. Set up a run p0007s: see notes on 2015-06-30. 1. Greg showed me how he resovled the restart problem: /discover/nobackup/fzeng/Heracles-1_1/src > find . -iname '*moist*' (to locate the path for GEOS_MoistGridComp.F90) Then take a look at moist_internal_rst in GEOS_MoistGridComp.F90: There are 9 fields. Then take a look at moist_internal_rst in Greg's h0000 in nobackup: If there are 9 fields, the size of this file should be 144*91*72*9*4=33,970,752 bytes. However, the moist_internal_rst created by running regrid.pl is only 26,421,696 bytes. 26,421,696 / (144*91*72) / 4 = 7, suggesting there are only 7 fields. Did a xdiff between Heracles-1_1 and Heracles-1_0 (need to be checked up) in GEOS_MoistGridComp.F90. Where Greg found that Heracles-1_1 has 2 more fields compared to Heracles-1_0. So the regrid.pl is for Heracles-1_0 and has not been updated for Heracles-1_1 yet. Greg wrote a script fixit.f90 to add two fields to moist_internal_rst and give them 0. Got a new moist_internal_rst that is the right size (i.e. 33,970,752 bytes). Then Greg ran it. Got error message "Could not allocate entry list" because it's 4*12 (the memory/PE layout provided doesnot work). Greg saw this before when he was trying different combinations of NX and NY in AGCM.rc and gcm_run.j. Haswell node has 28 cores. 4*21 and 4*16 worked. Sandybridge node has 16 cores. If the PE layout is 4*16, it would run successfully on both Haswell and Sandybridge. Greg sent me an email below summarizing the above (see his email on 7/1/2015): "look at GEOS_MoistGridComp.F90 internal spec... there are 9 internal vars, but moist_internal_rst only has 7... NCPL & NCPI are new... fixit.f90 - add 2 fields (set them to 0) won't run on PE layout (48 cpu 4x12... try 4x16 on sandybridge- works try on hasw, with processors per core limited to 16: #SBATCH --ntasks-per-node=16 works! look at cn0045 for PE mapping that may work... #SBATCH --ntasks=84 #SBATCH --ntasks-per-node=21 works. this would force job to haswell cores (since more than 16 tasks) these were the steps to get Heracles-1_1 run h0000 working... the moist_internal_rst file that regrid.pl created only has 7 144x91x72 fields (which would work for Heracles-1_0), but Heracles-1_1 needs 9 fields. I added two fields filled with zeros (I took a chance). Also, a cap_restart is needed." 2. Greg guided me to generate restarts for Heracles-1_1: (see 'Create_restarts_for_Heracles-1_1' for more details) (1) Run regrid.pl to generate restarts: This should be done after compiling Heracles-1_1. /discover/nobackup/fzeng/Heracles-1_1/src > source g5_modules /discover/nobackup/fzeng/Heracles-1_1/Linux/bin > regrid.pl Do you want script to select MERRA input restarts for you (y/n)? [n] y Enter ymd (yyyymmdd): 20010228 Enter hour (hh): 21 Enter OUTPUT atmosphere grid: b Enter OUTPUT ocean grid: [c] ([] means default. Just hit enter for default. Same for below. When not sure, just choose default.) Type 'bcs' to see BCS tags or Enter GCM or DAS tag for outputs: [Ganymed-4_0] Regrid upperair restarts (y/n)? [y] Regrid surface restarts (y/n)? [y] Enter OUTPUT directory for new restarts: /discover/nobackup/fzeng/GEOS5_restarts Enter Experiment ID for OUTPUT restarts [bc_d5_merra_jan98] Write rst.lcv restart (y/n)? [y] Append tag/grid label to names of final restarts (y/n)? [n] (2) Run gcm_setup to create paths: dali17:/discover/nobackup/fzeng/Heracles-1_1/src/Applications/GEOSgcm_App > gcm_setup Enter the Experiment ID: h0000 Enter a 1-line Experiment Description: test Do you wish to CLONE an old experiment? (Default: NO or FALSE) Enter the Atmospheric Horizontal Resolution code: b Enter the Atmospheric Model Vertical Resolution: LM (Default: 72) Do you wish to run the COUPLED Ocean/Sea-Ice Model? (Default: NO or FALSE) Enter the Data_Ocean Horizontal Resolution code: o1 (1 -deg, 360x180 Reynolds) Default o2 (1/4-deg, 1440x720 MERRA-2) o3 (1/8-deg, 2880x1440 OSTIA) Do you wish to run GOCART? (Default: NO or FALSE) Enter the tag or directory (/filename) of the HISTORY.AGCM.rc.tmpl to use (To use HISTORY.AGCM.rc.tmpl from current build, Type: Current ) ------------------------------------------------------------------------- Hit ENTER to use Default Tag/Location: (Current) Enter Desired Location for the HOME Directory (to contain scripts and RC files) Hit ENTER to use Default Location: ---------------------------------- Default: /home/fzeng/geos5/h0000 Enter Desired Location for the EXPERIMENT Directory (to contain model output and restart files) Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/h0000 Enter Location for Build directory containing: src/ Linux/ etc... Hit ENTER to use Default Location: ---------------------------------- Default: /discover/nobackup/fzeng/Heracles-1_1 Current GROUPS: g0620 gmaoint m2val Enter your GROUP ID for Current EXP: (Default: g0620) (3) Copy the restarts to /discover/nobackup/fzeng/h0000: /discover/nobackup/fzeng/GEOS5_restarts > cp -p *_rst* ../h0000/. Run rsts to remove the "bc_d5_merra_jan98." and ".20010228_21z.bin" from the names of the restarts. We created rsts and made it executable: /discover/nobackup/fzeng/h0000 > nedit rsts & /discover/nobackup/fzeng/h0000 > chmod u+x rsts /discover/nobackup/fzeng/h0000 > rsts (4) Create cap_restart which include the end date of run: /discover/nobackup/fzeng/h0000 > nedit cap_restart & (5) Copy Greg's fixit.f90 and run it to add 2 fields to the moist_internal_rst: /discover/nobackup/fzeng/h0000 > cp -p /discover/nobackup/gkwalker/h0000/fixit.f90 . /discover/nobackup/fzeng/h0000 > ifort fixit.f90 /discover/nobackup/fzeng/h0000 > a.out /discover/nobackup/fzeng/h0000 > mv moist_internal_rst.new moist_internal_rst 3. Set up a test run job: (1) Move AGCM.rc CAP.rc* ExtData.rc* gcm_run.j* HISTORY.rc in ~/geos5/h0000 to h0000 in nobackup and create a symbolic link in ~/geos5/h0000 to save space in home directory: ~/geos5/h0000 > mv * /discover/nobackup/fzeng/h0000/. ~/geos5 > rmdir h0000 ~/geos5 > ln -s /discover/nobackup/fzeng/h0000 h0000 (NOTE: The order is important!!) (2) Modify AGCM.rc, CAP.rc and gcm_run.j: ~/geos5/h0000 > nedit AGCM.rc gcm_run.j CAP.rc & AGCM.rc: set NX to 4, NY to 21. CAP.rc: set END_DATE to 20010330 (to run one month) gcm_run.j: set walltime to 1 hour, ntasks to 84, add "#SBATCH --ntasks-per-node=21" and "#SBATCH --qos=debug" (3) Submit the job: ~/geos5/h0000 > sbatch gcm_run.j 4. Create restarts for Heracles-1_1-CN and set up a test run: (see "Create_restarts_for_Heracles-1_1-CN" for more details) (1) Run gcm_setup to create paths and the files listed below. This should be done after compiling Heracles-1_1-CN. gcm_run.j gcm_post.j gcm_archive.j gcm_regress.j gcm_convert.j gcm_plot.tmpl gcm_quickplot.csh gcm_moveplot.j gcm_stats.j gcm_forecast.tmpl gcm_forecast.setup CAP.rc.tmpl AGCM.rc.tmpl HISTORY.rc.tmpl ExtData.rc.tmpl dali17:/discover/nobackup/fzeng/Heracles-1_1-CN/src/Applications/GEOSgcm_App > gcm_setup (2) Move AGCM.rc CAP.rc* ExtData.rc* gcm_run.j* HISTORY.rc in ~/geos5/hcn0000 to hcn0000 in nobackup and create a symbolic link in ~/geos5/hcn0000 to save space in home directory: ~/geos5/hcn0000 > mv * /discover/nobackup/fzeng/hcn0000/. ~/geos5 > rmdir hcn0000 ~/geos5 > ln -s /discover/nobackup/fzeng/hcn0000 hcn0000 (3) Modify AGCM.rc, CAP.rc and gcm_run.j: ~/geos5/hcn0000 > nedit AGCM.rc CAP.rc gcm_run.j & AGCM.rc: set NX to 4, NY to 21. CAP.rc: set END_DATE to 20010430 (to run one month) gcm_run.j: set walltime to 1 hour, ntasks to 84, add "#SBATCH --ntasks-per-node=21" and "#SBATCH --qos=debug" (4) Modify HISTORY.rc (use Greg's) to just save the variables we want: ~/geos5/hcn0000 > cp -p HISTORY.rc HISTORY.rc.orig ~/geos5/hcn0000 > cp -p ~gkwalker/geos5/h0001/HISTORY.rc . And change the exp id in HISTORY.rc. (5) Add CN restart to the catch_internal_rst created by running regrid.pl above: Copy Greg's append_CN.f90: ~/Catchment > cp -p ~gkwalker/Catchment/append_CN.f90 . In L236, 272: change the user name and exp id to mine. Compile and run it to generate catch_internal_rst-CN: ~/Catchment > ifort append_CN.f90 ~/Catchment > a.out Replace the old catch_internal_rst with the catch_internal_rst-CN created above: /discover/nobackup/fzeng/hcn0000 > mv catch_internal_rst catch_internal_rst.orig /discover/nobackup/fzeng/hcn0000 > mv catch_internal_rst-CN catch_internal_rst (6) Copy cap_restart: /discover/nobackup/fzeng/h0000 > cp -p cap_restart *_rst /discover/nobackup/fzeng/hcn0000/. (7) Submit the job: /discover/nobackup/fzeng/hcn0000 > sbatch gcm_run.j 5. Take a look at the output just generated using GrADS: /discover/nobackup/fzeng/hcn0000/scratch > ls -lt | head /discover/nobackup/fzeng/hcn0000/scratch > grads ga-> sdfopen hcn0000.geosgcm_surf.20010331_0900z.nc4 ga-> set lon -180 180 ga-> set gxout grfill ga-> q file ga-> d cnlai ga-> cbarn ga-> c ga-> d cntotc ga-> cbarn ga-> quit Notes: In HISTORY.rc: Greg added the carbon model diasgnostic These are added to GEOS_SurfaceGridComp.F90 in MAPL?? In HISTORY.rc: 1st column: the short name in GEOS_SurfaceGridComp.F90 2nd column: "SURFACE' tells you they are from GEOS_SurfaceGridComp.F90 3rd column: the name given when it's written out When I start a new experiment, I will just copy the HISTORY.rc from a previous exp and change the exp id. Same for AGCM.rc. Compile Heracles-1_1-CN: Got error messages "out of memory" in several places. This suggests that there is some system problem in the discover node where the code compiles. The solution is to logout of discover and log in again and make sure this time is a different discover node. Then recompile the code. If this happens again, would need to contact NCCS support to let them know. Tips: To select column on Mac: press "control" key and use the curser to select the column. next: create a geos5 restart from offline restart read cn.txt!! ================== 20150702: 1. Greg and I met with Rolf: Checked in the offline land model with carbon physics. This includes checking in files that are modified, and adding the files Greg created. When adding new files, have to copy the files before checking them in. 2. Greg tried to get new catchment parameters & PFT/fracx working in GEOS5 on newer tiles (370129 tiles, located check the namelist in TestBed). The one currently used in GEOS5 has 64778 tiles. The new and old ones have differnt tile orders. 100 - land 0 - ocean ? - ice Created file in ... (in gcm_run.j) that are in the (originally larry's directory, in gcm_run.j). Greg wrote a script to create restart? for GEOS5 edit_catch_internal_hires.poor.f90 There is actually a symbolic link in that directory. The real data is in Sarith's directory. ================== 20150706: 1. Processed GMAO GEOS5 July soil moisture percentile forecast following 'cn.txt' (written by Greg). 2. Found nothing in nobackup/h0000/geosgcm_surf. This is because geosgcm_surf is saved every 3 hours (see HISTORY.rc). By specifying the end_date to Mar 31 in CAP.rc, the file for 22:30 is not saved because GEOS5 stops at 21z. Because of missing of this file, monthly mean was not computed. Solution: set end_date to Apr 1 in CAP.rc and submit the job, the monthly mean for March will be calculated. This did not happen to the unified version (hcn0000) because daily data was saved for geosgcm_surf there. 3. Greg figured out how to decide whether to use + or - in AGCM.rc: I found these comments in subroutine MAPL_ESMFStateReadFromFile (which may come from module MAPL_GenericMod): ! Implemented a suggestion by Arlindo to allow files beginning with "-" (dash) ! to be skipped if file does not exist and values defaulted ! Implemented a suggestion by Larry to allow files beginning with "+" (plus) ! to not fail if they do not contain enough variables or skipped alltogether ! if file does not exist. In both case values are defaulted So, it looks like using a "+" means the restart files exists, and to add (bootstrap) any missing variables (set to some default value, probably zero), and using "-" means to use default values for all variables if the file is not found. If the restart file exists and contains all the needed variables, using the "-" or "+" won't have any effect. 4. Checked out and compiled Heracles-4_0 following Greg's notes in 'cn.txt'. Next: create restarts and set up a test run (refer to 'cn.txt' and Greg's /discover/nobackup/gkwalker/h4-0000 (Heracles-4_0)) Once everything works, add CN to Heracles-4_0. In the next few weeks, Greg will transfer the SMAP project to me. e0004s: control run e0005s: uses soil moisture deried from SMAP e0006s: like e0005s, but uses 50% percentile always (median 'climatology' run) e0007s: like e0005s, but uses 0% percentile always e0008s: like e0005s, but uses 100% percentile always In these SMAP exp runs: Run SMAP.gs and RZMC.gs first to get the rzmc data, Greg will automate the procedure before he transfers it to me. Greg will find a way to create a catch restart with CN restart for Heracles-4_0. After all these: Add CN to offline m3-15. ================== 20150707: 1. The error message "Fatal compilation error: Out of memory" occurred again when compiling Heracles-4_0. Logged out of discover and logged in again, compiled the model code and this time the model compiled successfully. Emailed NCCS support about this. 2. Ran regrid.pl to create restarts (see "Create_restarts_for_Heracles-1_1" and "updated 7 July 2015 for Heracles-4_0" in "cn.txt"). Ran gcm_setup to create paths and files, and set up a run with end_date as 20010331 (see "updated 7 July 2015 for Heracles-4_0" in "cn.txt" and 20150701 in this file). Works! 3. Add CN model to Heracles-4_0 (a fresh cvs checkout) to obtain Heracles-4_0-CN: see "code modified for CN model, under Ganymed-4_1-CN/src" in "cn.txt". compare these files between Heracles-4_0 and Heracles-1_1, and between Heracles-1_1 and Heracles-1_1-CN. compare with Greg's Heracles-4_0-CN to double check. Disable the plotting job following Greg's instructions (see his email on 20150707): change "echo Submitting Plot Job: $FILE for ${ofile_season} qsub $FILE" to "echo NOT Submitting Plot Job: $FILE for ${ofile_season} #qsub $FILE" in src/GMAO_Shared/GEOS_Util/post/gcmpost.script Compiled Heracles-4_0-CN on discover05, got error message "Fatal compilation error: Out of memory". Logged out and relogged in, compiled on discover06, the error happened again. Logged out and relogged in, compiled on discover07. This time gmake[7]: *** [install] Error 2 gmake[7]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp' gmake[6]: *** [install] Error 2 gmake[6]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp' gmake[5]: *** [install] Error 2 gmake[5]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp' gmake[4]: *** [install] Error 2 gmake[4]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp' gmake[3]: *** [install] Error 2 gmake[3]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp gmake[2]: *** [install] Error 2 gmake[2]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp' gmake[1]: *** [install] Error 2 gmake[1]: Leaving directory `/gpfsm/dnb31/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp' Double checked and made sure that the modified code is the same as Greg's. Then realized that I forgot to copy the files for CN model to Heracles-4_0-CN!! Copy a list of files from Heracles-1_1-CN (DON'T copy from Ganymed-4_1-CN, because four files were modified since then, see my notes on 20150624). To do this, create a file "tester" under /discover/nobackup/fzeng/Heracles-1_1-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/: cp -p clmtype.F90 \ clm_time_manager.F90 \ shr_const_mod.F90 \ clm_varpar.F90 \ clm_varcon.F90 \ subgridAveMod.F90 \ CNAllocationMod.F90 \ CNAnnualUpdateMod.F90 \ CNCStateUpdate1Mod.F90 \ CNCStateUpdate2Mod.F90 \ CNCStateUpdate3Mod.F90 \ CNNStateUpdate1Mod.F90 \ CNNStateUpdate2Mod.F90 \ CNNStateUpdate3Mod.F90 \ CNDecompMod.F90 \ CNFireMod.F90 \ CNGapMortalityMod.F90 \ pftvarcon.F90 \ CNHarvestMod.F90 \ CNMRespMod.F90 \ CNGRespMod.F90 \ CNNDynamicsMod.F90 \ CNPhenologyMod.F90 \ CNPrecisionControlMod.F90 \ CNSetValueMod.F90 \ CNVegStructUpdateMod.F90 \ CNWoodProductsMod.F90 \ CNSummaryMod.F90 \ CNEcosystemDynMod.F90 \ nanMod.F90 \ clmtypeInitMod.F90 \ CNiniTimeVar.F90 \ CNBalanceCheckMod.F90 \ CN_DriverMod.F90 \ compute_rc.F90 \ unified_rc3f_matrix_calc.F90 \ catch_constants2.f90 \ /discover/nobackup/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp/. Then > chmod u+x tester (to make it executable) Then > tester (to run it) Remove tester when done. Compiled again. Still failed. Compared my unified_rc3f_matrix_calc.F90 with Greg's and found some difference. Copied Greg's. Compiled again. Failed again!! Checked and make sure that unified_rc3f_matrix_calc.F90, catch_constants.f90, catch_constants2.f90, GNUmakefile_openmp, GEOS_CatchGridComp.F90 are same as Greg's. Tomorrow: ####Compare what I have with Greg's to see where is different### ================== 20150708: 1. Compared what I have in the directory below with what Greg has and I have all the files needed. /discover/nobackup/fzeng/Heracles-4_0-CN/src/GEOSgcs_GridComp/GEOSgcm_GridComp/GEOSagcm_GridComp/GEOSphysics_GridComp/GEOSsurface_GridComp/GEOSland_GridComp/GEOScatch_GridComp Asked Greg's for help. Greg replied: "It looks like the compiled catch_constants is conflicting with catch_constants; it may be that the code changed (with an earlier date?) after an earlier compilation attempt. Try this: in the GEOScatch_GridComp directory, run "make clean", then try normal compilation again. I think this will clear out any code compiled in GEOScatch_GridComp, so you won't have to recompile everything that had already successfully compiled. The codes do look the same... if the above doesn't work, do a "make realclean" in the src directory and recompile everything." Tried running "make clean" in the GEOScatch_GridComp directory and then compiling the code normally. Works!!!! 2. Prepare for SMAP runs: (1) update my LDASsa_m2-14_2_p1_unified-SMAP_TestBed to use the latest FP forcing: cd /discover/nobackup/fzeng/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/src/Components/GEOSlana_GridComp /bin/cp -p /discover/nobackup/gkwalker/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/src/Components/GEOSlana_GridComp/clsm_ensdrv_force_routines.F90 . cp -p /discover/nobackup/gkwalker/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/src/Components/GEOSlana_GridComp/process_cat.F90.use_daily_wet2 . cd ../GEOScatch_GridComp cp -p /discover/nobackup/gkwalker/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/src/Components/GEOScatch_GridComp/catchment.F90.rzwater . then recompile (on discover-sp3): source g5_modules make clean cleanup (as is, with "set exp = 73q_e0004s") For the runs reading soil moisture data, copy Greg's executable (instead of compiling with process_cat.F90.use_daily_wet2 and catchment.F90.rzwater): cp -p /discover/nobackup/gkwalker/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/exec/LDASsa_mpi_73q_e0004s-use_wet2.x /discover/nobackup/fzeng/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/exec/. (2) The EASE 9km runs will save daily data, so we'll use a different namelist. Copy /home/gkwalker/Catchment/LDASsa_m2-14/SMAP_EASEv2_M09/LDASsa_inputs_driver-daily.nml to my directory (~/Catchment/LDASsa_m2-14/SMAP_EASEv2_M09). It will only save daily tile data. I can extract the SMAP and 0,50,10th percentile soil moistures using new scripts Greg wrote. Copy these files from ~gkwalker/Catchment to my ~/Catchment directory: process-smap SMAP.gs RZMC.gs SMAP.ctl wet2_percentile_SMAP.f90 wet2_percentile_RZMC.f90 Compile the fortran code: ifort -o wet2_percentile_SMAP wet2_percentile_SMAP.f90 ifort -o wet2_percentile_RZMC wet2_percentile_RZMC.f90 Run "process-smap". This will create daily soil moisture data for the offline to use, reading SMAP data and data from Greg's run e0004s. The data extraction will likely take several hours. Copy Greg's experiment setup files: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 > tar xvf /discover/nobackup/gkwalker/Catchment/SMAP_EASEv2_M09/SMAP_experiments.tar Greg modified wet2_percentile_SMAP.f90, wet2_percentile_RZMC.f90 and process-smap. Copy them to my directory again. Recompile wet2_percentile_SMAP.f90 and wet2_percentile_RZMC.f90 as above. Then run "process-smap", answer y to each experiment (this is only for the first time when we run the experiments). It will submit the job for each experiment. e0005s: ln -s /discover/nobackup/gkwalker/SMAP rzmc e0006s: ln -s /discover/nobackup/gkwalker/RZMC_0.50 rzmc e0007s: ln -s /discover/nobackup/gkwalker/RZMC_0.00 rzmc e0008s: ln -s /discover/nobackup/gkwalker/RZMC_1.00 rzmc ###Future:### Need to think about a way to automatically archive the data generated from the EASE 9km runs. Probably archive a file each month, maybe do this in lenkf.j: when the time acrosses the month boundary, archive the data. 3. Take a look at the output from p0007s: Found no difference between p0007s and p0006s when using plot_p000xs.gs to take a look at the burned area and LAI. Greg suggested to see whether the data is different: diff /discover/nobackup/fzeng/Catchment/princeton/p0007s_32/p0007_199007.gdat /discover/nobackup/fzeng/Catchment/princeton/p0006s_32/p0006_199007.gdat It says the files differ, so it's likely that the GrADS descriptor files do not point to the correct data. Found that p0007s.ctl pointed to p0006s data. After correcting the data path in p0007s.ctl, now the maps show the difference. 4. Set up a run for Heracles-4_0-CN: Ran gcm_setup to create paths and the files (gcm_run.j etc.) below. This should be done after compiling Heracles-4_0-CN. /discover/nobackup/fzeng/Heracles-4_0-CN/src/Applications/GEOSgcm_App > gcm_setup Copied restart files from Heracles-1_1-CN: /discover/nobackup/fzeng/hcn0000 > cp -p *_rst /discover/nobackup/fzeng/h4cn-0000/. Move AGCM.rc CAP.rc* ExtData.rc* gcm_run.j* HISTORY.rc in ~/geos5/h4cn-0000 to h4cn-0000 in nobackup and create a symbolic link in ~/geos5/h4cn-0000 to save space in home directory: ~/geos5/h4cn-0000 > mv * /discover/nobackup/fzeng/h4cn-0000 ~/geos5/h4cn-0000 > cd .. ~/geos5 > rmdir h4cn-0000 ~/geos5 > ln -s /discover/nobackup/fzeng/h4cn-0000 h4cn-0000 Modify AGCM.rc, CAP.rc and gcm_run.j, and compared them with Greg's in /discover/nobackup/gkwalker/h4-0001: (forgot to save original copies) ~/geos5/h4cn-0000 > nedit AGCM.rc CAP.rc gcm_run.j & AGCM.rc: set NX to 4, NY to 21, change "-" to "+" under "Boot-Strapable AGCM Model Restart Files". CAP.rc: for debug, set END_DATE to 20020530 (to run one month from the restart 20020430, see below), job_sgmt from 15 to 20 days, num_sgmt from 20 to 15, heartbeat_dt from 900 to 1800. (/discover/nobackup/fzeng/hcn0000 > cat cap_restart 20020430 210000) gcm_run.j: set walltime to 12 hour, ntasks to 84, add "#SBATCH --ntasks-per-node=21" Copy gcm_run.j to gcm_run_test.j and: set walltime to 1 hour, and add "#SBATCH --qos=debug" Save the original copy of HISTORY.rc: ~/geos5/h4cn-0000 > cp -p HISTORY.rc HISTORY.rc.orig Change HISTORY.rc: in "Collections:", move geosgcm_surf to top and comment out the rest; change geosgcm_surf.frequency from 3 hours to 24 hours; and in geosgcm_surf.fields, add the fields for the CN model. Compared it with Greg's in /discover/nobackup/gkwalker/h4-0001 Copy cap_restart from hcn0000: /discover/nobackup/fzeng/h4cn-0000 > cp -p /discover/nobackup/fzeng/hcn0000/cap_restart . Submit the job: /discover/nobackup/fzeng/h4cn-0000 > sbatch gcm_run_test.j 5. Deleted the data in e0004s_soilt to release some space: discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_soilt > /bin/rm -rf RUN e0004_*.gdat discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0004s_soilt > /bin/rm CN_* SMAP* ================== 20150709: 1. Set up a run of Heracles-4_0-CN to 20150701: (refer to /discover/nobackup/gkwalker/h4-0001) CAP.rc: change END_DATE to 20150701, num_sgmt to 20 Submit the job: /discover/nobackup/fzeng/h4cn-0000 > sbatch gcm_run.j 2. Summarize notes in "/discover/nobackup/fzeng/notes/Heracles-4_0-CN" 3. Compare the output from e0004s to e0008s to Greg's. e0005s differs, and my runs e0006s-e0008s didn't produce output. Greg suggested: In my /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09 directory: mv e0005s e0005s.bak mkdir e0005s cp -p e0005s.bak/setup e0005s/. cp -p e0005s.bak/lenkf.j e0005s/. cd e0005s nedit lenkf.j, change "#SBATCH --time=06:00:00" to "#SBATCH --time=00:15:00", add "#SBATCH --qos=debug", in the mpirun line, change "-end_month 07 -end_day 06" to "-end_month 04 -end_day 02", and on the tile2grid_ease_daily line, change "2015 07 06" to "2015 04 01" then sbatch lenkf.j. The e0005_20150401.gdat generated is identical to Greg's: /discover/nobackup/fzeng/Catchment/SMAP_EASEv2_M09/e0005s > diff e0005_20150401.gdat /discover/nobackup/gkwalker/Catchment/SMAP_EASEv2_M09/e0005s/e0005_20150401.gdat returns nothing. Found that the lenkf.j in e0005s to e0008s all have the same exp id e0005 and point to the same e0005s directory. Greg said "yes, that is certainly the problem!...e0005s-e0008s were all writing to the same directory (e0005s), which is why there is no output for e0006s-e0008s, and why e0005s looked bad (the output in it was probably the e0008s data). I'll fix the scripts (plus a couple other corrections) and set them up when we meet." "@dy--" in process-smap: subtract a day interpolate the forcing in clsm_ensdrv_drv_routine.F90 so even though the data for 2015-07-02 is available, can only run to 2015-07-01, because if we run to 2015-07-02, the 00:30 data of 2015-07-03 is required to do the interpolation but it doesn't exist. This will make the model fail on the last time step (e.g. 2015-07-01, 23:30) 4. Greg extracted daily GPP, NPP and NEE data for Lesley: see Greg's email on 20150709 for details. "Hi Lesley, I've extracted daily GPP, NPP, and NEE from our run and placed it on NCCS in /discover/nobackup/gkwalker/e0004s_carbon_flux The daily data is in yearly files, sequential unformatted binary, little endian. Each field is a 720x360 array (half degree). The (1,1) cell is centered at [179.75W,89.75S], and the (720,360) cell is centered at [179.75E,89.75N]. The units are grams of carbon per square meter per second. In the carbon model code, a comment for NEE says "positive for source". The offline land model was run on the EASE 9km grid using "cross_RPFPIT__precCPCUG5RPFPITv2" (MERRA-Land RPFP-IT with CPCU precipitation correction). The carbon state was initialized from an earlier offline run which used different tiles, forcing, and catchment parameters. This EASE 9km run was run a number of cycles 2001-2014, and is fairly close to a carbon equilibrium state. Along with the data, there is an IDL program, a GrADS descriptor file, and fortran code which will read the data. Let me know if you need further details or additional data. greg" If in the future she wants hourly or 3-hourly data of GPP, NPP and NEE, modify the name list LDASsa_input_driver.nml: set out_dtstep%xhourly to 3600 (hourly) or 10800 (3-hourly) and then follow what I did for e0004s_soilt (clsm_ensdrv_drv_routines.F90 and process_cat.F90) to only save these 3 fields instead of the default 78 fields (will fill up my nobackup pretty soon). 5. The restart files are binary in Heracles-1_1, but are netCDF in Heracles-4_0. How to know the format since can't tell from the filenames? See below: Run the commands below ((L387-388 in gcm_run.j in h4-0000)) on any of the restarts (e.g. catch_internal_rst) in the directory where the restart files are located, it will tell you if it's .hdf , binary or netcdf: set fvrst = `cat AGCM.rc | grep "^DYN_INTERNAL_RESTART_FILE" | cut -d ":" -f2` (L387 in gcm_run.j in h4-0000) set filetype = `/usr/bin/file -Lb --mime-type $fvrst` echo $filetype If it returns "appliation/x-hdf", it means that the file type is netCDF (hdf?). If it returns "application/octet-stream", it means that the file type is binary. ================== 20150710: 1. Met with Greg to update Higgins precipitation data through 20150708. Copy all the file in land01:/terra/gkwalker/ most recently Higgins daily precipitation product from http://www.esrl.noaa.gov/psd/data/gridded/data.unified.html source: ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/ Randy asked for 2015 data to be updated. The steps are: on land01:/terra/fzeng/higgins, wget -N ftp://ftp.cdc.noaa.gov/Datasets/cpc_us_precip/RT/precip.V1.0.2015.nc use GrADS to check the data is up to Jul 8, 2015 (for example, see (1)) modify (see (2)) GrADS script higgins.gs to output 2015 data through current date (NOAA web page says what date it's updated to), run grads script: grads -lbc "run higgins.gs" update day of year, compile, and run higgins.f90 to verify data let Randy know the data has been updated (he already knows the output format, see my email to Randy and Greg's email to me on 20150710). (1) ga> xdfopen higgins.xdf ga> q dims ga> q file ga> set time 8jul2015 ga> d precip ga> c ga> set time 9jul2015 ga> d precip (2) L55: change m<=6 to m<=7 L65-66: change to if (m=7) nd=8 2. Greg archived boundary condition data: /discover/nobackup/gkwalker/bcs > tar hcvf bcs_SMAP ("h" will archive the files the symbolic link points to) I should archive carbon model restart and land restart from the offline runs from time to time, so that in the future I can pick a restart date that is close to my experiment date. Question: How do I know the date of the CN restart file (the land restart filename tell the date, but CN? They are paired, so just put them in one tar file?) If I want to get the data (multiple files) from archive back: Follow the instructions in http://www.nccs.nasa.gov/primer/data.html 3. If I need to turn off CN model in GEOS5 and use prognostic LAI & SAI from offline runs in GEOS5 in the future, follow what Greg did in Ganymed-3_0_p1-CN and Ganymed-3_0_p1-SR in "cn.txt". SR-stomatal resistance. They are on ARCHIVE, but they are also in Greg's nobackup. Ganymed-3_0_p1-CN: regular GEOS5-CN Ganymed-3_0_p1-SR: CN model is turned off but compute_rc is called to calculate stomatal resistance. More realistic than Ganymed-3_0_p1-CN. Experiments include cn0030[a-d]. cn0042 is for another experiment (use CTDEF soil moisture fields from the offline run, again need to be formatted so that GEOS5 can read) GEOS_VegdynGridComp.F90 was modified for SR but no change in CN. Export the 12 fields. If I need to read in something, would be in this GEOS_VegdynGridComp.F90 and GEOS_LandGridComp.F90 Have to format the data from offline so that it can be read by GEOS5. GEOS_CatchGridComp.F90: import the 12 fields. These experiments used Ganymed-3_0_p1. 4. How to format the data from offline for GEOS5: (see "cn.txt") GEOS_CatchGridComp.F90.daily_LAI: original GEOS_CatchGridComp.F90 was modified to save daily LAI and SAI. lai_tile2grid_daily_geos5.f90: reads in the daily LAI and SAI written above in GEOS_CatchGridComp.F90.daily_LAI, calculates climatology and save the climatology in the format GEOS5 can read. L102 in lai_tile2grid_daily_geos5.f90: ###need to spend more time on this to understand### Since GEOS_CatchGridComp.F90.daily_LAI writes out data at 12z every day, the beginning of the 1st day is year 0, dec 31 (mon=12, day=31) and end is year 1, jan 1 (mon=1, day=1). How did Greg figure these out? Greg looked at the file: /discover/nobackup/ltackacs/bcs/Ganymed-1_0_m1/green_clim_144x91_DC.data tear apart this file to find out what the headers mean. MAPL_ReadForcingX in MAPL_Generic.F90 reads in the data generated by lai_tile2grid_daily_geos5.f90. 5. Data used in offline and GEOS5 are big endian as compile option. If not specified in open commend, the default is little endian. If can't read data or receive bad codes when reading data, it's probably that it needs to be converted to big endian. 6. Next week: Take a look at the e000xs output. Use Sarith's new tiles in GEOS5 Greg tried it: h0000x Will need catch_internal_rst from the offline run and the other rst files from previous GEOS5 run. set up an offline run to save daily soil hydrolic fields then we can use the code above to format it to be readable for GEOS5. Think about how to convert .mat MODIS burned area files to binary ================== 20150717: 1. Update the SMAP runs to the date data is available: run process-smap under ~/Catchment. 2. Take a look at the results from p000[5-7]s cycles and calculate monthly statistics. 3. Resume the runs for p000[5-7]s (there was problem with discover and the jobs got removed): "to continue the runs, you need to start from Jan 1, since the carbon model restart (CN_restart) is written out once per year. So in the case of p0005, you would not need to modify the script at all, but you would need to delete RUN/rs/ens0000/Y1963/M0[2-9] and M1?. I think it is OK to leave the "cat" data there, it will be overwritten. You can also delete RUN/rs/ens0000 Y1948 through Y1962, but it won't matter if you don't. The model will resume from 1 Jan 1963 (as long as you remove "rs" directories M02 through M12 first)." (Greg's email today) for p0005s: /discover/nobackup/fzeng/Catchment/princeton/p0005s/RUN/rs/ens0000/Y1963 > /bin/rm -rf M0[2-9] /discover/nobackup/fzeng/Catchment/princeton/p0005s/RUN/rs/ens0000/Y1963 > /bin/rm -rf M1? /discover/nobackup/fzeng/Catchment/princeton/p0005s > sbatch lenkf.j 4. compare fires between p0006s and p0007s: ~/Catchment > grads -lc "run plot_fire_diff_p.gs" ================== 20150721: 1. Update the SMAP runs to the date data is available: run process-smap under ~/Catchment. 2. Update Higgins precipitation data through 20150720 (see notes on 20150710). Higgins precipitation data needs to be updated about every 2 weeks. 3. Greg consolidated the offline and GEOS5 carbon model codes. Extract Greg's LDASsa_m3-15_2-CN.tar: /discover/nobackup/fzeng/ > tar -xvf /discover/nobackup/gkwalker/LDASsa_m3-15_2-CN.tar Extract Greg's Heracles-4_0-CN.tar: /discover/nobackup/fzeng/ > tar -xvf /discover/nobackup/gkwalker/Heracles-4_0-CN.tar Compile both. Use 'cleanup' to compile LDASsa_m3-15_2-CN. When compiling Heracles-4_0-CN, it's on discover06 again (compiling failed multiple times on this node). After ssh to discover-sp3, do this: > limit (to check the limitation) > limit datasize unlimited (see the email from NCCS on 20150708, https://software.intel.com/en-us/forums/topic/268149) > limit (to make sure the commend above works) then cd to the src directory, "source g5_modules" and then "make install". Not sure if I should have done "limit datasize unlimited" after I cd to the src directory or not. Hopefully it will compile successfully. Both compiled successfully. 4. GEOS5 post-processing didn't create monthly climatology for some months. Greg has had the same issue with GEOS5 post-processing... not sure why it doesn't create the monthly climatology when all monthly mean files are present. Here's how to recover them manually: setenv GEOSUTIL $NOBACKUP/Heracles-4_0-CN/src/GMAO_Shared/GEOS_Util $GEOSUTIL/post/gcmclim.script -source /discover/nobackup/fzeng/h4cn-0000 -collection geosgcm_surf -month 04 This should create h4cn-0000.geosgcm_surf.monthly.clim.M04.nc4 If the monthly mean file is missing, then you would run this: $GEOSUTIL/post/gcmpost.script -source /discover/nobackup/fzeng/EXP -recover YYYYMM where EXP is the experiment directory, and YYYYMM is the year and month ================== 20150722: 1. Work on the plot_e000xs.gx script to compare the SMAP runs with the control. 2. Every time the tile coordinate changes, need to create pft.dat, ndep.dat, and cli_t2m.dat for the new tile coordinate. Need to pull information from CLM_veg_typs_fracs. See Greg's email. When Sarith creates the boundary condition data sets, he would create CLM_veg_typs_fracs. See gkwalker/bcs/Gaymed 4v1 /../../CLM_veg_typs_fracs (Sarith creates this) Tile coordinate file: DC144x091… (Sarith has documentation on this) Greg's grid2tile_pft_ease.f90 (in Greg's local system): creates fpt.dat Greg needs to modify it. ###Take a look at the Ffafstetter ###Take a look at what changes Greg made to consolidate the offline and gcm codes. ###Take a look at the pft, ndep etc (see Greg's email) "Sarith can produce the tile coordinate file and catchment land parameters for any desired grid (such as the GEOS5 2x2.5 144x91, or the EASE 9km grid). It contains the tile coordinate file (ends with .til) and the CLM PFT and fractions derived from ESA (with help the CLM4 PFT data); CLM_veg_typs_fracs. For a description, see: land01:/l_data-new/model_parameters/geos5/bcs/Tag-SM_LandBCs2_v00/CF0090x6C_DE0360xPE0180/clsm/README For the offline, "pft.dat" could come from the CLM_veg_typs_fracs file (I will clean up a program that I have to do this; for now, see land01:/terra/gkwalker/clm4/pft/grid2tile_pft_ease.f90 (I used this to creat the pft.dat for the EASE 9km runs, filling in some missing land tiles in Sarith's CLM_veg_typs_fracs with data using my former method of generating PFT data from the original CLM PFT file surfdata_0.47x0.63_simyr2000_c091023.nc). The original PFT data came from https://svn-ccsm-inputdata.cgd.ucar.edu/trunk/inputdata/lnd/clm2/ (username:guestuser password:friendly). For GEOS5, PFT & fraction information would be in catch_internal_rst PFT & fractions: surfdata/surfdata_0.23x0.31_simyr2000_c100406.nc nitrogen deposition: ndepdata/ndep_clm_simyr2000_0.23x0.31_c091106.nc PFT physiology parameters: pftdata/pft-physiology.c100226 Routines to create nitrogen deposition, 2m temperature climatology, and PFT & fractions (obsolete), can be found on land01 in /terra/gkwalker/clm4/pft" Clsm_ensdrv_init_routines.F90: reads catch_internal_restart. Xdiff can look at the difference of up 3 files. Unfied_rc3f_matrix_calc: the only difference in the unified model code and the gcm code for the land model. The driver is different. 3. SMAP results: Take a look at lag Compare e0006-8s to e0004s ================== 20150723: 1. Work on the plot_e000xs.gx script to compare the SMAP runs with the control. 2. Read land01:/l_data-new/model_parameters/geos5/bcs/Tag-SM_LandBCs2_v00/CF0090x6C_DE0360xPE0180/clsm/README 3. Greg goes over CN restart structure with me. 4. Greg goes over the differences in the CNDecompMod.F90 in the offline and in the original CLM code: #if def (c directives, need to use ifort -cpp? to compile, cpp stands for c pre-processor) !dir$ (fortran directives) (in computer_rc.F90, it has to be there for intel 15 compiler) Dynamic vegetation (CNDV): If Randy wants to add CNDV, he probably wants to update the CLM part from 4 to 4.5. modify_offline_cn_restart_from_princeton_hires.f90 on land01 in Greg's home directory: uses openmp to do parallel calculation (everything else in the offline and GEOS5 uses mpi) To tell if it would use intel 15 or 14 compiler: take a look at the file g5_modules in /src The structure of the carbon model restart: Office: CN_restart: read and written using direct access Greg is going to show me: Use high-resolution tile parameters in GEOS5, but currently the modeling group does not have a tile coordinate file for the high-resolution. Greg put together a tile coordinate file Need to modify catch_internal_restart. grep bcsdir gcm_run.j (to search for the lines that have "bcsdir" in the file "gcm_run.j") ================== 20150724: 1. p0005s got kicked out of the queue. Resume it. See notes on 20150717. 2. process-smap to run the smap project to 20150721. 3. Greg made major changes to cn.txt. Read cn.txt. 4. Greg changed "UNGRIDDED_DIMS = (/NUM_ZON*NUM_VEG*VAR_PFT/)" to "UNGRIDDED_DIMS = (/NUM_ZON,NUM_VEG,VAR_PFT/)" in GEOS5's GEOS_CatchGridComp.F90 and tested it to make sure it works. I copy his updated GEOS_CatchGridComp.F90 and recompile GEOS5. After copy GEOS_CatchGridComp.F90 and ssh to discover-sp3: > limit datasize unlimit (to avoid errors when compiling, see notes on 20150721) src> source g5_modules src> make realclean src> make install Got the executable, but could not get GEOS5 to run. Greg changed "UNGRIDDED_DIMS = (/NUM_ZON,NUM_VEG,VAR_PFT/)" back to "UNGRIDDED_DIMS = (/NUM_ZON*NUM_VEG*VAR_PFT/)" in GEOS_CatchGridComp.F90 to make it work again. 5. Set up a GCM run with high resolution tiles set: Run regrid.pl to create the restart files for 20001231 for GEOS5, following the notes in "Create_restarts_for_Heracles-1_1" in /discover/nobackup/fzeng/notes. Run gcm_setup to create paths and files (including AGCM.rc, CAP.rc, gcm_run.j, etc.). See notes in notes in "Create_restarts_for_Heracles-1_1" in /discover/nobackup/fzeng/notes. Move all these files generated in home directory to nobackup and set up a symbolic link in home directory to point to nobackup. ~/geos5/h4cn-0001 > mv * /discover/nobackup/fzeng/h4cn-0001/. ~/geos5 > rmdir h4cn-0001 ~/geos5 > ln -s /discover/nobackup/fzeng/h4cn-0001 . Copy the restart files generated by regrid.pl to /discover/nobackup/fzeng/h4cn-0001. Copy Greg's listx and run it to change the names of the restart files. /discover/nobackup/fzeng/GEOS5_restarts > cp -p *_rst.20001231* /discover/nobackup/fzeng/h4cn-0001/. /discover/nobackup/fzeng/h4cn-0001 > cp -p /discover/nobackup/gkwalker/h0006y/listx . /discover/nobackup/fzeng/h4cn-0001 > listx Delete catch_internal_rst and keep the rest. We will use Greg's edit_catch_internal_hires_p0005s-CN_h0005.f90 to create catch_internal_rst from offline restarts. Copy Greg's edit_catch_internal_hires_p0005s-CN_h0005.f90: ~/geos5 > cp -p ~gkwalker/geos5/edit_catch_internal_hires_p0005s-CN_h0005.f90 . Change the path to my latest p0005s run (the one that is running). Compile edit_catch_internal_hires_p0005s-CN_h0005.f90 in the run directory and run the generated executable a.out to create catch_internal_rst from the CN and land restart files in my latest p0005s run: /discover/nobackup/fzeng/h4cn-0001 > ifort ~/geos5/edit_catch_internal_hires_p0005s-CN_h0005.f90 /discover/nobackup/fzeng/h4cn-0001 > a.out Copy cap_restart from a previous run and modify it to reflect the date of the restart (i.e. 20001231 in this case): /discover/nobackup/fzeng/h4cn-0001 > cp ../h4cn-0000/cap_restart . cap_restart: 20001231 210000 Make a back up copy of AGCM.rc, CAP.rc, gcm_run.j, and HISTORY.rc, then modify AGCM.rc, CAP.rc, and gcm_run.j (can use xdiff to take a look at what changes made). Copy Greg's HISTORY.rc. /discover/nobackup/fzeng/h4cn-0001 > cp -p /discover/nobackup/gkwalker/h0006x/HISTORY.rc . Make a back up copy of gcm_convert.j in /discover/nobackup/fzeng/h4cn-0001/convert and modify it (can use xdiff to take a look at what changes made). Check that fvcore_internal_rst is binary now (it will be converted to netcdf when the model runs): /discover/nobackup/fzeng/h4cn-0001 > file -Lb --mime-type fvcore_internal_rst Submit the job: /discover/nobackup/fzeng/h4cn-0001 > sbatch gcm_run.j Take a look at the results: /discover/nobackup/fzeng/h4cn-0001 > tail slurm-4745536.out /discover/nobackup/fzeng/h4cn-0001/convert > tail GEOSgcm.log SURFACEInitialize 2986 PHYSICSMAPL_GenericInitialize 1548 PHYSICS::Initialize 1277 AGCMMAPL_GenericInitialize 1548 AGCMInitialize 977 GCMMAPL_GenericInitialize 1548 GCMInitialize 895 GCSMAPL_GenericInitialize 1548 MAPL_Cap 641 application called MPI_Abort(MPI_COMM_WORLD, -21848) - process 29 This doesn't seem right. The model doesn't run properly when it reads catch_internal_rst. Remove everything in /discover/nobackup/fzeng/h4cn-0001/convert except gcm_convert.j and gcm_convert.j.bak. Check the catch_internal_rst created above with Greg's and they have the same size. Replace the catch_internal_rst created above with Greg's and submit the job. Fail again at the same place. Try using Greg's executable GEOSgcm.x but the run still failed. Greg will take a look at the change he made to GEOS_CatchGridComp.F90 (change * to , in ungridded_dim) and the script edit_catch_internal_hires_p0005s-CN_h0005.f90 to see why this happens. 6. ###Now (at least in m3-15) carbon model restart is done in process_cat.F90 (L1469-). process_cat.F90-CN_restart_May_1_each_year (in LDASsa_m2-09_p1_unified/src/Components/GEOSlana_GridComp/73n): save CN restart in a day different from 00z Jan 1 (which the model usually does) I can try modifying process_cat.F90 to save CN restart and land model restart at 21z on select day (e.g. Apr 30 every year). Copy Greg's process_cat.F90-CN_restart_May_1_each_year first.### ================== 20150727: 1. Deleted the Heracles-4_0-CN I have and copy Greg's again: cd $NOBACKUP /bin/rm -rf Heracles-4_0-CN tar xf /discover/nobackup/gkwalker/Heracles-4_0-CN.tar 2. Compile: ssh discover-sp3 limit datasize unlimited cd $NOBACKUP/Heracles-4_0-CN/src source g5_modules make install Saw some warning messages again, but got the executable. Greg said he also got those warning messages. Copy $NOBACKUP/Heracles-4_0-CN/src/Applications/GEOSgcm_App/GEOSgcm.x to $NOBACKUP/h4cn-0001 Go to that directory, restore catch_internal_rst.old to catch_internal_rst, clear out the things in the convert directory and leave only gcm_convert.j. Submit the debug gcm_run.j. Works! Delete the debug job, modify gcm_run.j and sbatch it. 3. Get burned area and carbon loss data from the offline carbon model for Peter Colarco and Melanie Cook (use results from p0007s_37) How much of carbon is lost (emission)? 1948-2012 monthly 2deg*2.5deg (ultimatly on tile level) closs and burned area (extract these two) vegetation types and fractions algorithm (e folding probability based on how close soil moisture is to wilting point) check if the burned area we saved is annual fraction or for each time step CNFireMod.F90 CN_veg_fracs.F90 Tile2grid_pft.f90 extract_burn.gs vegetation fractions sum to 1 regardless bare land (so you will see high 10 broadleaf deciduous shrub in the deserts) Greg created these files to extract the data: extract_burn.gs tile2grid_pft.f90 p0007s_burn.ctl pft_144x91.ctl read_burn.f90 read_pft.f90 I copy these files, use extract_burn.gs to create p0007s_burn.gdat, use tile2grid_pft.f90 to create pft_144x91.gdat. ================== 20150729: 1. Emailed Peter and Melanie: Hi Peter and Melanie, We have extracted the data from our offline run. The data file is p0007s_burn.gdat on /discover/nobackup/fzeng. It includes monthly data for fire carbon loss (gC/m2/s) and annual total fractional area burned for 1948-2012 (780 months). The data is unformatted binary, sequential access. The grid is 144x91, with (1,1) centered at (180W,90S) and (144,91) centered at (177.5E,90N). Undefined value is -9.99e8. The GrADS descriptor file is p0007s_burn.ctl in ~fzeng/Catchment. read_burn.f90 in ~fzeng/Catchment will read the data. pft_144x91.gdat on /discover/nobackup/fzeng has the fractions of each of the 15 vegetation types below. The GrADS descriptor file is pft_144x91.ctl on ~fzeng/Catchment. The data is unformatted binary, sequential access. The grid is 144x91, with (1,1) centered at (180W,90S) and (144,91) centered at (177.5E,90N). Undefined value is -999. read_pft.f90 in ~fzeng/Catchment will read the data. 1 needleleaf evergreen temperate tree 2 needleleaf evergreen boreal tree 3 needleleaf deciduous boreal tree 4 broadleaf evergreen tropical tree 5 broadleaf evergreen temperate tree 6 broadleaf deciduous tropical tree 7 broadleaf deciduous temperate tree 8 broadleaf deciduous boreal tree 9 broadleaf evergreen temperate shrub 10 broadleaf deciduous temperate shrub 11 broadleaf deciduous boreal shrub 12 arctic c3 grass 13 cool c3 grass 14 warm c4 grass 15 crop Please refer to CNFireMod.F90, CNSummaryMod.F90 and CN_driver.F90 in /discover/nobackup/fzeng/LDASsa_m2-14_2_p1_unified-SMAP_TestBed/src/Components/GEOSlana_GridComp on how annual total fractional area burned and carbon loss are calculated. Annual total fractional area burned (ann_farea_burned) is in subroutine CNFireArea in CNFireMod.F90. Carbon loss (closs) is in subroutine CNFireFluxes in CNFireMod.F90, L752-758 in CNSummaryMod.F90, and L794 in CN_driver.F90. Please refer to section 14.9 in “CLM4_Tech_Note.pdf” for more details on the fire routine in our offline model. The annual total fractional area burned (ann_farea_burned) we saved may not be what you really want. In the future we will save fractional area burned for each time step (farea_burned) which may be what you are interested in. Please let us know if you have any questions. Fanwei and Greg 2. diff -arq can take a look at the difference in two directories Greg made changes to the offline code: In process_cat.F90: not saving annual fractional area burned, but save area burned per second in each time step Surface turbulent Previous added viscous layer to make ** stable, now … (see Greg's cn.txt) Ga-> xdfopen filename(of a netcdf file) Greg is working on a routine that creates pft, nitrogen deposition and t2m_clim for new tile coordinate from old tile coordinate: grid2tile_pft_ndep_cli-t2m_alb.f90. ================== 20150730: 1. Copy Greg's Heracles-4_0-CN.tar and LDASsa_m3-15_2-CN.tar (both last modified on 20150728), and compile both. New about this version of the offline and GEOS5 carbon model: now the diagnostic area burn per second (variable name is burn) is saved. 2. Clean up (make realclean) and archive Heracles-1_1 and Heracles-1_1-CN. 3. Write a fortran script to compare the fire (area burned and fire carbon loss) between p0007s and MODIS and CASA-GFED3. ================== To Do: 1. Take a look at the files in 2 on 20150722 2. Take a look at the changes Greg made to consolidate the offline and GEOS5 carbon model codes 3. Take a look at the Ffafstetter 4. Take a look at what changes Greg made to consolidate the offline and gcm codes. 5. Take a look at the pft, ndep etc (see Greg's email)