Difference between revisions of "Data Challenges"

From CMB-S4 wiki
Jump to navigationJump to search
 
(22 intermediate revisions by 2 users not shown)
Line 1: Line 1:
At the SLAC meeting it was agreed to separate "experiment definitions" (bands, resolutions, sensitivity level) from "sky models" (LCDM+dust+sync+).
+
We separate "experiment definitions" (bands, resolutions, sensitivity level) from "sky models" (LCDM+dust+sync+).
Each of these will be designated by a number. The combination of a given experiment definition XX acting on a given sky model YY will be designated XX.YY.
+
Each of these is designated by a number. The combination of a given experiment definition XX acting on a given sky model YY is designated XX.YY.
  
Experiment models, sky models and the datasets obtained by applying for former to the latter will be stored at NERSC at <br />
+
Experiment models, sky models and the datasets obtained by applying for former to the latter are stored at NERSC at <br />
 
'''/project/projectdirs/cmbs4/{expt_xx, sky_yy, data_xx.yy}'''
 
'''/project/projectdirs/cmbs4/{expt_xx, sky_yy, data_xx.yy}'''
  
Line 15: Line 15:
 
== Sky Models ==
 
== Sky Models ==
  
=== Sky model 00 ===
+
'''[[Sky Models]]'''
This is simply LCDM + Gaussian dust + Gaussian Sync.
 
Since the dust and sync levels are set to the levels found in the BICEP/Keck field it wouldn't make any sense to use this model for a large sky fraction.
 
  
=== Sky Model 01 ===
 
This is [https://arxiv.org/abs/1608.02841 PySM] run in a1d1f1s1 mode - i.e. with the default settings for AME, dust, free-free and synchrotron.
 
  
=== Sky Model 02 ===
+
== Sim Data Sets ==
This is [https://arxiv.org/abs/1608.02841 PySM] run in a2d4f1s3 mode - [https://github.com/bthorne93/PySM GitHub] for some description of what this means.
 
  
=== Sky Model 03 ===
+
Many of the possible xx.yy combinations have been generated and are available on NERSC.
Versus 02 this switches the dust model to a Hensley/Draine model and is dubbed a2d7f1s3. (02 to 03 also marks the switch from pysm_1.0 to pysm_2.0 so the s2, f1, s3 components in principle change. Think the change is to do with the details of how the bandpass integration is being handled.)
 
  
=== Sky Model 04 ===
+
Most of 02.00 through 02.09 exist - these are the "Science Book config".
This model switches the dust model to a Tuhin Ghosh provided model which implements dust decorrelation. See [[HiDPol|HI-based dust polarization model for r forecasts]].
 
The AME, sync and free-free components remain the same as 03
 
  
=== Sky Model 05 ===
+
Most of 04.00 through 04.06 exist - these are the "CDT report config".
This is a toy model of dust which is highly decorrelated. See [[Toy highly decorrelated dust model]].
 
There is NO AME/sync/Free-free - only dust.
 
  
=== Sky Model 06 ===
+
For experiment config 06, only foreground models 00, 07, and 09 were generated - these are the "DSR report config"
This is a model from Raphael Flauger and Brandon Hensley based on MHD sims of galactic magnetic field populated with dust and relativistic electrons according to some recipe.
 
Some notes on its construction from Brandon are [sky_models.pdf here].
 
 
 
== Sim Data Sets ==
 
 
 
=== 01.00 ===
 
This is the 01 experiment definition acting on the 00 sky model (Gaussian dust and sync).
 
The full 1000 realizations have been generated and are available at '''/project/projectdirs/cmbs4/data_xx.yy/01.00''' in the NERSC system.
 
We can see these maps [[01.00 sim input maps | in this posting]].
 
The directory contains the fully combined LCDM+noise+dust+sync realizations, noise-only realizations, and signal-only realizations of lensed LCDM.
 
Some preliminary results of re-analyzing these maps can be found in
 
[[http://bicep.rc.fas.harvard.edu/CMB-S4/analysis_logbook/20170224_cmbs4_dc1_final Justin's posting]]
 
and [[http://users.physics.harvard.edu/~buza/20170317_s4dc1 Victor's Mar 17]] logbook posting.
 
 
 
=== 01.01 ===
 
This is the 01 experiment definition acting on the 01 sky model.
 
The full 1000 realizations have been generated and are available at '''/project/projectdirs/cmbs4/data_xx.yy/01.01''' in the NERSC system.
 
We can see these maps [[01.01 sim input maps - first try | in this posting]].
 
Note that these all contain the same PySM foreground plus randomly permuted LCDM and noise realizations drawn from the sets used for 01.00.
 
The directory contains only the fully combined signal+noise realizations - when analyzing these maps one should use the noise realizations from 01.00.
 
 
 
=== 01.02 ===
 
This is the 01 experiment definition acting on the 02 sky model.
 
The full 1000 realizations have been generated and are available at '''/project/projectdirs/cmbs4/data_xx.yy/01.02''' in the NERSC system.
 
Note that these all contain the same PySM foreground plus randomly permuted LCDM and noise realizations drawn from the sets used for 01.00.
 
The directory contains only the fully combined signal+noise realizations - when analyzing these maps one should use the noise realizations from 01.00.
 
  
 
== NERSC Info ==
 
== NERSC Info ==
  
Shared space is available on [http://www.nersc.gov NERSC], where everyone is welcome to [http://crd.lbl.gov/cmb sign-up for an account] under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files are located in the shared CMB-S4 file space (/project/projectdirs/cmbs4). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:
+
Shared space is available on [http://www.nersc.gov NERSC], where everyone is welcome to [http://crd.lbl.gov/cmb sign-up for an account] under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files are located in the shared CMB-S4 file space (<code>/global/cfs/cdirs/cmbs4</code>). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:
 
* log in to the cmbs4 project account following the instructions [[http://www.nersc.gov/users/accounts/user-accounts/production-project-accounts here]]  
 
* log in to the cmbs4 project account following the instructions [[http://www.nersc.gov/users/accounts/user-accounts/production-project-accounts here]]  
 
* sync the files to an appropriate subdirectory in the project space
 
* sync the files to an appropriate subdirectory in the project space
Line 73: Line 37:
  
 
For any NERSC issues, including access to the filegroup and/or project account, please contact [mailto:jdborrill@lbl.gov Julian Borrill].
 
For any NERSC issues, including access to the filegroup and/or project account, please contact [mailto:jdborrill@lbl.gov Julian Borrill].
 +
 +
=== HPSS Archive ===
 +
 +
Older data challenge maps are moved to the [https://docs.nersc.gov/filesystems/archive/ High Performance Storage System]. These are stored in the <code>low_ell_BB</code> directory in the <code>cmbs4</code> user home directory of HPSS. For each Data Challenge, maps are broken up in several different archives to achieve archive file sizes of ~few 100 GB.
 +
 +
List the data challenge sets that have been archived here:
 +
02.00,  02.01, 02.02, 02.03, 02.04, 02.05, 02.06, 02.09
 +
02b.00, 02b.03
 +
02c.00, 02c.03
 +
03.00,  03.03
 +
03b.00, 03b.03
 +
03c.00, 03c.03

Latest revision as of 11:39, 25 February 2020

We separate "experiment definitions" (bands, resolutions, sensitivity level) from "sky models" (LCDM+dust+sync+). Each of these is designated by a number. The combination of a given experiment definition XX acting on a given sky model YY is designated XX.YY.

Experiment models, sky models and the datasets obtained by applying for former to the latter are stored at NERSC at
/project/projectdirs/cmbs4/{expt_xx, sky_yy, data_xx.yy}

Experiment Definitions

Experiment Definitions

Input Sky Components

Sky Components

Sky Models

Sky Models


Sim Data Sets

Many of the possible xx.yy combinations have been generated and are available on NERSC.

Most of 02.00 through 02.09 exist - these are the "Science Book config".

Most of 04.00 through 04.06 exist - these are the "CDT report config".

For experiment config 06, only foreground models 00, 07, and 09 were generated - these are the "DSR report config"

NERSC Info

Shared space is available on NERSC, where everyone is welcome to sign-up for an account under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files are located in the shared CMB-S4 file space (/global/cfs/cdirs/cmbs4). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:

  • log in to the cmbs4 project account following the instructions [here]
  • sync the files to an appropriate subdirectory in the project space
  • ensure that the permissions are set appropriately (g+rX,o-rwx)

Remember to include a README and to post the details on this wiki page.

For any NERSC issues, including access to the filegroup and/or project account, please contact Julian Borrill.

HPSS Archive

Older data challenge maps are moved to the High Performance Storage System. These are stored in the low_ell_BB directory in the cmbs4 user home directory of HPSS. For each Data Challenge, maps are broken up in several different archives to achieve archive file sizes of ~few 100 GB.

List the data challenge sets that have been archived here:

02.00,  02.01, 02.02, 02.03, 02.04, 02.05, 02.06, 02.09
02b.00, 02b.03
02c.00, 02c.03
03.00,  03.03
03b.00, 03b.03
03c.00, 03c.03