Difference between revisions of "Data Challenges"

From CMB-S4 wiki
Jump to navigationJump to search
(→‎Data Challenge 1: Added external link to maps pager)
 
(79 intermediate revisions by 6 users not shown)
Line 1: Line 1:
At the Chicago meeting it was agreed that it would be useful to provide common inputs for all science forecasting, to include extra-galactic foreground, galactic foreground and noise maps at a range of frequencies. It was further agreed that, in order to support the widest range of uses, these would be provided as all-sky HEALPix maps at Nside-8192.
+
We separate "experiment definitions" (bands, resolutions, sensitivity level) from "sky models" (LCDM+dust+sync+).
 +
Each of these is designated by a number. The combination of a given experiment definition XX acting on a given sky model YY is designated XX.YY.
  
Possible sky-model inputs are:
+
Experiment models, sky models and the datasets obtained by applying for former to the latter are stored at NERSC at <br />
* Extra-galactice: Alvarez/Battaglia/Bond, PSM
+
'''/project/projectdirs/cmbs4/{expt_xx, sky_yy, data_xx.yy}'''
* Galactic: pySM, PSM
 
Please add to these lists as appropriate.
 
  
Inputs will be hosted at [http://www.nersc.gov NERSC], where everyone is welcome to [http://crd.lbl.gov/cmb sign-up for an account] under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files will then be located in the shared CMB-S4 file space (/project/projectdirs/cmbs4). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:
+
== Experiment Definitions ==
* log in to the cmbs4 project account following the instructions [[http://www.nersc.gov/users/accounts/user-accounts/production-project-accounts here]]  
+
 
* sync the files to an appropriate subdirectory in the project space
+
'''[[Experiment Definitions]]'''
* ensure that the permissions are set appropriately (g+rX,o-rwx)
+
 
Remember to include a README and to post the details on this wiki page.
+
== Input Sky Components ==
  
For any NERSC issues, including access to the filegroup and/or project account, please contact [mailto:jdborrill@lbl.gov Julian Borrill].
+
'''[[Sky Components]]'''
  
== Data Challenge 1 ==  
+
== Sky Models ==
Intended as a simple test to develop our practice, compare spectral- and map-based forecasting, contrast different map-based codes and methods, and coordinate r and non-r science forecasting.
 
  
'''[http://bicep.rc.fas.harvard.edu/cbischoff/20170111_data_challenge_1/ Maps pager for data challenge 1]'''
+
'''[[Sky Models]]'''
  
=== Instrument specification ===
 
{| class="wikitable" style="text-align: right; margin-left: 51px;"
 
! Frequency (GHz) !!  30 !!  40 !!  85 !!  95 !! 145 !! 155 !! 220 !! 270
 
|-
 
| Bandwidth (GHz) || 9.0 || 12.0 || 20.4 || 22.8 || 31.9 || 34.1 || 48.4 || 59.4
 
|-
 
| Beam FWHM (arcmin) || 76.6 || 57.5 || 27.0 || 24.2 || 15.9 || 14.8 || 10.7 || 8.5
 
|-
 
|}
 
  
* Tophat bandpasses [[Tophat bands for Data Challenge | described in this posting]], added to above table --[[User:Cbischoff|Cbischoff]] ([[User talk:Cbischoff|talk]]) 15:51, 8 November 2016 (UTC)
+
== Sim Data Sets ==
* Beam widths copied from [http://users.physics.harvard.edu/~buza/20160531_fisher/ Victor's posting here] --[[User:Cbischoff|Cbischoff]] ([[User talk:Cbischoff|talk]]) 19:20, 14 December 2016 (UTC)
 
  
Beam window functions for each frequency can be found on NERSC in:
+
Many of the possible xx.yy combinations have been generated and are available on NERSC.
  /project/projectdirs/cmbs4/data_challenges/phase1/sky/wfunc/
 
In addition to beam smoothing at high ell, these window functions drop sharply to zero below ell=30 (maps for the data challenge contain no information below ell=30).
 
  
=== Scalar, tensor & non-Gaussian CMB: Borrill ===
+
Most of 02.00 through 02.09 exist - these are the "Science Book config".
* Planck cosmology + interesting values of r, f_NL
 
  
=== Extragalactic foregrounds: Alvarez / Battaglia / Bond / Stein===
+
Most of 04.00 through 04.06 exist - these are the "CDT report config".
  
See [[Wish_List_and_Existing_Capability_for_Simulated_Maps#Peak_Patch_Extragalactic_Sky_Model|Peak Patch Extragalactic Sky Model]]
+
For experiment config 06, only foreground models 00, 07, and 09 were generated - these are the "DSR report config"
  
=== Galactic foregrounds: pySM / Ben ===
+
== NERSC Info ==
* Nside 512 maps with delta-function bandpasses delivered at 20, 25, 30, 35, 40, 45, 50, 60, 70, 80, 85, 90, 95, 100, 120, 140, 145, 150, 155, 165, 195, 215, 220, 235, 245, 270, 280, 340, 350, 400, 545, 860 GHz (CMB-S4 + S3, WMAP, Planck & LiteBIRD frequencies)
 
  
=== Noise: homogeneous isotropic matched to N_ell: Victor / Clem / Colin / John K ===
+
Shared space is available on [http://www.nersc.gov NERSC], where everyone is welcome to [http://crd.lbl.gov/cmb sign-up for an account] under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files are located in the shared CMB-S4 file space (<code>/global/cfs/cdirs/cmbs4</code>). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:
 +
* log in to the cmbs4 project account following the instructions [[http://www.nersc.gov/users/accounts/user-accounts/production-project-accounts here]]
 +
* sync the files to an appropriate subdirectory in the project space
 +
* ensure that the permissions are set appropriately (g+rX,o-rwx)
 +
Remember to include a README and to post the details on this wiki page.
  
* N_ell functional form and parameters listed in [http://users.physics.harvard.edu/~buza/20161220_chkS4/ Table 2 here] (Victor's 2016-12-20 posting)
+
For any NERSC issues, including access to the filegroup and/or project account, please contact [mailto:jdborrill@lbl.gov Julian Borrill].
  
=== Masks: Clem / John K ===
+
=== HPSS Archive ===
  
Using BICEP/Keck apodization mask (fsky ~ 1%) shown in [[First_steps_to_sim_input_maps|fourth figure here]]. File located on NERSC at:
+
Older data challenge maps are moved to the [https://docs.nersc.gov/filesystems/archive/ High Performance Storage System]. These are stored in the <code>low_ell_BB</code> directory in the <code>cmbs4</code> user home directory of HPSS. For each Data Challenge, maps are broken up in several different archives to achieve archive file sizes of ~few 100 GB.
  /project/projectdirs/cmbs4/data_challenges/phase1/sky/amask/map/bk14_fall_n512.fits
 
  
=== Schedule ===
+
List the data challenge sets that have been archived here:
* Dec 7: Beta version of Phase 1 maps
+
02.00,  02.01, 02.02, 02.03, 02.04, 02.05, 02.06, 02.09
* Dec 14: Telecon
+
02b.00, 02b.03
* Dec 17: Noise power (N_ell) prescriptions to Clem from Victor and Colin
+
02c.00, 02c.03
* Dec 21: A few realizations of Phase 1 maps delivered by Clem
+
03.00,  03.03
* Dec 21: Telecon (note phase shift)
+
03b.00, 03b.03
* Jan 4: Telecon
+
03c.00, 03c.03
* Jan 18: Telecon
 
* Feb 1: Telecon
 
* Feb 15: Telecon
 
* Feb 27, 28, Mar 1: S4 general meeting at SLAC
 
* Mar 2, 3: S4 CDT meeting at SLAC
 

Latest revision as of 11:39, 25 February 2020

We separate "experiment definitions" (bands, resolutions, sensitivity level) from "sky models" (LCDM+dust+sync+). Each of these is designated by a number. The combination of a given experiment definition XX acting on a given sky model YY is designated XX.YY.

Experiment models, sky models and the datasets obtained by applying for former to the latter are stored at NERSC at
/project/projectdirs/cmbs4/{expt_xx, sky_yy, data_xx.yy}

Experiment Definitions

Experiment Definitions

Input Sky Components

Sky Components

Sky Models

Sky Models


Sim Data Sets

Many of the possible xx.yy combinations have been generated and are available on NERSC.

Most of 02.00 through 02.09 exist - these are the "Science Book config".

Most of 04.00 through 04.06 exist - these are the "CDT report config".

For experiment config 06, only foreground models 00, 07, and 09 were generated - these are the "DSR report config"

NERSC Info

Shared space is available on NERSC, where everyone is welcome to sign-up for an account under the "Data Analysis for Post-Planck CMB Experiments" allocation (PI Borrill). The files are located in the shared CMB-S4 file space (/global/cfs/cdirs/cmbs4). In order to manage this space, all files stored there should be owned by the cmbs4 project account but accessible to the cmbs4 group. To do this:

  • log in to the cmbs4 project account following the instructions [here]
  • sync the files to an appropriate subdirectory in the project space
  • ensure that the permissions are set appropriately (g+rX,o-rwx)

Remember to include a README and to post the details on this wiki page.

For any NERSC issues, including access to the filegroup and/or project account, please contact Julian Borrill.

HPSS Archive

Older data challenge maps are moved to the High Performance Storage System. These are stored in the low_ell_BB directory in the cmbs4 user home directory of HPSS. For each Data Challenge, maps are broken up in several different archives to achieve archive file sizes of ~few 100 GB.

List the data challenge sets that have been archived here:

02.00,  02.01, 02.02, 02.03, 02.04, 02.05, 02.06, 02.09
02b.00, 02b.03
02c.00, 02c.03
03.00,  03.03
03b.00, 03b.03
03c.00, 03c.03