Technology Development Telecon - (DM) Data Management

From CMB-S4 wiki
Jump to navigationJump to search

Charge to CMB-S4 Technology Development Working Group

In the first edition of the Technology Book, the experimental CMB community summarized the current state of CMB technology and evaluated its current technical readiness with a 5-level Technology Status Level (TSL) and manufacturing readiness with a 5-level Production Status Level (PSL). For each technology, we identified Technology Development (TD) efforts necessary to advance it for possible use in CMB-S4. As a next step of the collaborative community wide effort, the CMB-S4 TD prioritization working group will evaluate TD topics based on impacts they have on cost, schedule, and science return. By the time of the Argonne meeting (March 2018) the working group will produce a prioritized list of the TD topics that the community should pursue to ensure timely maturity of technologies that will enable the successful advancement of the project.

We have grouped the relevant technologies into the following areas to tackle this immense task: Telescope and Site; Cryogenics, Cryostats and Optics; Detectors and Readout; and Data Management. Calibration of evaluation metrics across the subgroups is important for fair comparison of the TD topics. In addition, many TD topics are inter-dependent. To capture these ideas, the overall working group will communicate across all the subgroups in monthly combined group meetings.

The scope of each subgroup is as follows:

  • Telescope and site: Covers telescope, mount, site, power generation, etc...
  • Cryogenics, cryostats and optics: Covers cryogenics (4K and mK), cryostats, windows, filters, lenses, HWP etc...
  • Detector and readout: Covers detector (detector array and holder) and readout (warm/cold), etc...
  • Data management: Covers DAQ, data transfer, simulation, analysis, publication, etc...

Tab-Separated Table


  • October (combined call on Nov 2)
    • Define baseline to compare impact of R&Ds to
    • Start populating list of R&D item and do evaluation on some entries
    • Make modification/improvements to organization method if necessary
  • November (combined call on Nov 30)
    • Continue to populate and evaluate list of R&D items as sub-group
    • Draft list by end of month
    • evaluation may not be complete by this time
  • December (combined call on Dec 28)
    • Sub-group list with priority evaluation by end of month such that we can start combining lists in 2018
  • January (combined call on Feb 1, maybe more combined call)
    • Start normalization/ combine lists from different groups
    • Draft of combined list by end of month
  • February (combined call on March 1, maybe more combined call)
    • Modify/fine tune combined list
    • Discuss on what we’ll show at CMB-S4 workshop at ANL
  • March
    • CMB-S4 workshop at ANL: Present at CMB-S4 meeting

CMB-S4 Technology Development Telecon 2017-11-10 Subgroup Meeting: Data Management

  • Agenda
    • Go over the DM tab of the TD spreadsheet (drawn from the CDT report) identifying any gaps, determining which elements do not require technology development per se, and starting to flesh out the items which do.
      • Next Milestone:
        • November (combined call on Nov 30)
        • Continue to populate and evaluate list of R&D items as sub-group
        • Draft list by end of month -- evaluation may not be complete by this time
    • Set the telecon frequency/schedule
  • Notes
    • Present: Julian B, Joy D, Ken G, Nathan W, Graca R, Yuji C.
    • Computation Resources -- not a technological development. We will use available DOE/NSF resources, and probably buy ~$100k of computers / special hardware, but there is not money to build super computer for
    • DAQ: TD effort is technology dependant, for example for uMux there is stuff to do. That might be on the Readout group side though
    • Compression might need some RD -- currently using FLAC for SPT3G but it's limited, can't do more than 24 bit samples
    • Transmission: issue for South Pole. Real time can send optimally compressed / downsampled data. Question of how many computers you want there.
    • On site storage: very unlikely to have the resources to keep the entire dataset on spin
    • Time Domain:
      • Live Monitoring: huge data rate, probably current live monitors are not going to scale. Can look at LSST-type alerts. You have to plan all kind of hierarchal observations plan from daily to weekly to monthly. Develop something to catch issues early, some checks are computer intensive, some are not. Computer intensive one: have to figure out how to do that at South Pole given limited transmission bandwidth. Risk is 'small' (i.e. path forward is relatively clear), but lots of work to be done.
      • Pre-processing: framework needs to be compute and human efficiency (lots of data, framework common to a very large collaboration now). This needs a significant developmen effort. This will partly come from developing simulations to assess systematic errors as a precursor to systematic mitigation.
      • Map making: work to be done to scale to S4 data. Characterization with covariance matrices, monte carlo etc will be a big piece of it.
    • Science:
      • Foregrounds: dealing with the large dataset issue is already done by maps, foreground algorithms are going to progressively get better with S3, so changes in algorithm seem to be incremental (unless we need to deal with simulations right now)

  • Action Items
    • all: Meet next week Nov 17 11am PT
    • all: look through the spreadsheet and starting thinking about filling the "Impact" and "R&D" columns
    • Joy : check with Toki when columns need to be filled + what is the meaning of the 1-5 numbers

CMB-S4 Technology Development Telecon 2017-10-05 Subgroup Meeting: Data Management

  • Agenda
    • Charge to the group
    • CDT report
    • Schedule & milestones
  • Notes
    • Present: Colin B, Julian B, Yuji C, Joy D, Salman H
    • Apologies: Laura N
    • Charge:
      • DM is a bit different from the other areas, and couples to them (eg. in experiment modeling, design validation, systematics mitigation, ... )
    • CDT report:
      • Julian walked people through the current draft of the DM section of the CDT report, inviting comments and additions
        • Instrument Data (acquisition, transmission, storage)
        • Time Domain (live monitoring, pre-processing, map-making)
          • Time domain processing will be different for different telescopes (eg. with or without HWP), but should still all run within a common framework.
          • Live monitoring needs to combine data from multiple telescopes
        • Science Analysis (foregrounds, results, feedback)
          • Analysis complexity is increased by having multiple instruments of multiple types, as well as the hybrid scanning strategy.
          • Feedback can also inform changes in the instrument configuration and/or scanning strategy.
        • Simulations (experiment modeling, sky modeling, data generation)
        • Publication (data products, software tools, archiving)
          • Need to include internal data distribution; data and software standards.
        • Computational Resources (science data facility)
    • Schedule & milestones
      • Doodle poll still open and active; discussion about telecon frequency (tbd)
      • Milestones:
        • Develop a comprehensive list of DM requirements coupled to the project timeline.
        • Determine the TD required for these.
        • Prioritize this TD based on its impact on risk, cost, and science return.
        • Take this prioritized list to the full group for integration.
  • Action Items
    • Julian to update CDT report to include the comments here (done).
    • Circulate telecon time once doodle polling is complete; next telecon is week of October 16th.