# Difference between revisions of "LBNL-2020: Low-ell BB"

From CMB-S4 wiki

Jump to navigationJump to search (→Notes) |
|||

(3 intermediate revisions by 3 users not shown) | |||

Line 3: | Line 3: | ||

Parallel 1, session D. Tuesday March 31, 9:00 to 11:00 am pacific. | Parallel 1, session D. Tuesday March 31, 9:00 to 11:00 am pacific. | ||

− | https:// | + | https://fnal.zoom.us/j/5196960377 |

Monitor: Kimmy Wu | Monitor: Kimmy Wu | ||

Line 29: | Line 29: | ||

* Results from data challenge 06 [45 min] | * Results from data challenge 06 [45 min] | ||

** Parametrized likelihood + lensing template [Bischoff, 15 min] [https://docs.google.com/presentation/d/1bI90NApMEv4W_suqxp9a89wMH_KEdfu0qRKs9S1GmLY/edit#slide=id.g726e0d289c_0_65 slides] | ** Parametrized likelihood + lensing template [Bischoff, 15 min] [https://docs.google.com/presentation/d/1bI90NApMEv4W_suqxp9a89wMH_KEdfu0qRKs9S1GmLY/edit#slide=id.g726e0d289c_0_65 slides] | ||

− | ** Bayesian optimal analysis [Millea, 15 min] | + | ** Bayesian optimal analysis [Millea, 15 min] [https://docs.google.com/presentation/d/19ArC5olLsiz-Xl3Cm0w5LgyyzY7lECuDG4KSCqORZEk/edit?usp=sharing slides] |

− | ** SO XForecast pipeline [Errard, 15 min] | + | ** SO XForecast pipeline [Errard, 15 min] [https://www.dropbox.com/s/u3gq105bm86k2rp/CMBS4_lowellBB_LBNL_March2020.pdf?dl=0 slides] |

* Next steps for data challenges [45 min] | * Next steps for data challenges [45 min] | ||

** ILC maps to study impacts on foregrounds [Umiltà, 10 min][[:File: LBNL-Lowell_BB-ILC.pdf | slides]] | ** ILC maps to study impacts on foregrounds [Umiltà, 10 min][[:File: LBNL-Lowell_BB-ILC.pdf | slides]] | ||

** Exact curve-sky iterative lensing estimation [Carron, 10 min] [[:File: Cmbs4_iterativelensing.pdf | slides]] | ** Exact curve-sky iterative lensing estimation [Carron, 10 min] [[:File: Cmbs4_iterativelensing.pdf | slides]] | ||

** Low ell foregrounds [[https://docs.google.com/presentation/d/1zywbOcyp579rUUOi0BzsQB308Fab5weHx254FhDwqH8/edit?usp=sharing Hensley], 15 min] | ** Low ell foregrounds [[https://docs.google.com/presentation/d/1zywbOcyp579rUUOi0BzsQB308Fab5weHx254FhDwqH8/edit?usp=sharing Hensley], 15 min] | ||

− | |||

** What other questions do we need to address using data challenges? [all] | ** What other questions do we need to address using data challenges? [all] | ||

* Inputs for summer agency review [20 min] | * Inputs for summer agency review [20 min] | ||

Line 46: | Line 45: | ||

== Notes == | == Notes == | ||

+ | |||

+ | === [https://docs.google.com/presentation/d/1bI90NApMEv4W_suqxp9a89wMH_KEdfu0qRKs9S1GmLY/edit#slide=id.g726e0d289c_0_65 Data Challenge 06: parametric likelihood pipeline] (Bischoff) === | ||

+ | |||

+ | * Review of Data Challenge 06: | ||

+ | ** 8 SAT freq + 20 GHz on LAT; 6 LAT freq. Previous data challenge, approximate delensing by reducing the lensing signal by expected AL residuals. | ||

+ | ** 3 Galactic foreground models: 2 are non-Gaussian, 1 is Gaussian. | ||

+ | * Lensing template used is generated on curve sky with QE. | ||

+ | * Computed correlation between lensing template and map. | ||

+ | * Observed that there is no measured correlation between the 95GHz (foreground-ful) maps and the lensing template. This is on Gaussian foregrounds, so it’s not entirely surprising. | ||

+ | * Q: why is there a lower correlation below ell=50. A: the modes below ell~30 are artificially removed to approximate that we can’t measure modes from the reionization bump. | ||

+ | * Q: Is constructing the covariance matrix with Gaussian sims a good enough approximation? A: The likelihood assumes the bandpowers to be non-Gaussian (H-L likelihood), but still assumes the statistics of the maps are Gaussian. Something to check is at what point these assumptions break down. | ||

+ | |||

+ | === [https://docs.google.com/presentation/d/19ArC5olLsiz-Xl3Cm0w5LgyyzY7lECuDG4KSCqORZEk/edit#slide=id.p Bayesian optimal lensing] (Millea) === | ||

+ | |||

+ | * All versions of delensing (Bayesian, iterative QE, etc) can be derived from lensing posterior probability. | ||

+ | ** MC sampling of posterior is the best you can do. | ||

+ | ** Can find maximum likelihood at fixed cosmological parameters -- obtain joint maximum a posteriori (MAP) estimate of CMB and phi. | ||

+ | ** Can integrate over CMB fields to obtain marginal MAP estimate of phi -- this is the iterative QE. | ||

+ | ** For Gaussian distributions, joint MAP phi is the same as marginal MAP phi, but this is not the case for non-Gaussian lensing. | ||

+ | * Comparison of approaches to this problem: sampling vs joint MAP vs marginal MAP | ||

+ | ** Sampling is computationally hard, joint MAP is computationally easiest. | ||

+ | ** Samples give you unbiased cosmological parameter estimation, but other methods work pretty well to delens for r because of separation of scales. | ||

+ | ** Software packages: CMBLensing.jl, LensIt | ||

+ | * Lensing templates from joint or marginal MAP yield A_lens ~ 0.1 for DC 06b 155 GHz (with no masking). | ||

+ | ** When you add a mask, joint MAP picks up a huge mean field. Leads to large decorrelation with true phi, but almost no effect on lensing template! | ||

+ | * Used sampling to check S4 delensing forecast. Found good agreement. | ||

+ | * Applying sampling analysis to SPT data. It is working on 100 deg^2. Have checked that computation can be done on 1500 deg^2. | ||

+ | * Working to add curved sky support to CMBLensing.jl | ||

+ | ** Made flat sky joint MAP lensing templates that can be used with data challenge 06b. Think that flat sky leads to ~5% sub-optimality. | ||

+ | * Q: Does comparison of joint vs marginal MAP results change in presence of foregrounds? A: They should respond differently to foregrounds. Need to try it and see. | ||

+ | |||

+ | === [https://www.dropbox.com/s/u3gq105bm86k2rp/CMBS4_lowellBB_LBNL_March2020.pdf?dl=0 SO-BB map-based parametric pipeline] (Errard) === | ||

+ | |||

+ | * Assume that noisy frequency maps are made up of sky model multiplied by parametrized mixing matrix plus noise. Use noise covariance to estimate foreground spectral parameters, then estimate sky signals. Similar to COMMANDER. | ||

+ | * SO BB pipeline: | ||

+ | ** Simulate noisy frequency maps (can replace with DC06) | ||

+ | ** fgbuster module for component separation (can also delens here) | ||

+ | ** NaMaster module for pure-B power spectrum estimation | ||

+ | ** emcee module to sample r likelihood, with optional marginalizations | ||

+ | * Applying this pipeline to CMB-S4 instrument configuration. Some issues with bandpasses, noise covariance. Using PySM sky signal. | ||

+ | ** Obtain good constraints on foreground spectral indices, CMB/dust/sync maps. | ||

+ | ** Some jobs still in queue at NERSC. Get sigma(r) ~ 3e-3 for case without delensing. Looking forward to cases with lower A_lens. | ||

+ | * Comment: Note different hitmaps between the SAT patterns and LAT patterns. | ||

+ | * Needed in order to run on DC 06 | ||

+ | ** Definition of bandpasses -- Colin to discuss with Josquin | ||

+ | ** Noise simulator for pixel-based noise covariance -- should be able to use the existing noise simulations | ||

+ | ** Splitting the patch to deal with spectral index variations. | ||

+ | ** Delensing -- have incorporated Carron lensing templates | ||

+ | |||

+ | === [[Media:LBNL-Lowell_BB-ILC.pdf|ILC on DC06 maps]] (Umiltà) === | ||

+ | |||

+ | * Linear combination of maps with weights estimated from covariance matrix of data maps. Constrained weights to preserve CMB signal. ILC in spherical harmonics basis. | ||

+ | * Tested using three different sky masks for DC 06. | ||

+ | * Currently only have 10 realizations. These have already been masked. Beam is equivalent to 95 GHz LAT beam. | ||

+ | * Q: slide 6 - the ILC map-CMB input shows some effects of the cleaning of foreground, but it can also be ambiguous cleaning. In some cases, 95GHz-CMB has lower residuals than the ILC map-CMB. There is trickiness when masks are applied even though by definition the ILC weights minimize the variance of the output map. | ||

+ | * Q on slide 8: hit map vs smoothed map are two masks used as shown on slide 3. | ||

+ | * Comment: Could smooth weights in ell, if desired. | ||

+ | * Comment: For purposes of producing lensing template for r analysis, best mask might be the apodized mask from SATs. Julien’s lensing pipeline does take variance map -- thinks that should be able to obtain unbiased map by mask or have Caterina do this. | ||

+ | * Q: What about B modes? A: How close the ILC map is to input B modes depends on foregrounds, E->B mixing. | ||

+ | |||

+ | === [[Media:Cmbs4_iterativelensing.pdf|Iterative curved-sky lensing estimates for cmbs4 deep]] (Carron) === | ||

+ | |||

+ | * Iterative estimator working on curve sky on the Pole deep map of DC06. | ||

+ | * MAP phi map for given cosmology, marginalized over unlensed fields. Include mean field subtraction. | ||

+ | * Suspect that the computational cost is comparable to the joint MAP. | ||

+ | * One of the computational bottleneck was due to the non-Gaussianity of the posterior, one way around it is to do a L-dependent step length. | ||

+ | * Demonstrates that with no-foreground sims, the Alens residuals with a marginal MAP (on curve sky) is 7%. (The QE on slide 3 is an improved version than those supplied for r-analysis so far.) | ||

+ | * Technical details: ignoring changes in mean-field (just use the one from the first step) is ok. Ignoring the phi-induced MF is okay. Here use EB modes between 200 and 3000. | ||

+ | * The residual lensing B mode power from this run is A_L~0.1. (Different color lines on slide 5 correspond to the number of iteration. Blue is step1, and +1 for all the other colors. The purple MAP line is the final converged version, at ~step 8.) | ||

+ | |||

+ | === [https://docs.google.com/presentation/d/1zywbOcyp579rUUOi0BzsQB308Fab5weHx254FhDwqH8/edit#slide=id.p Low-ell Foregrounds] (Hensley) === | ||

+ | |||

+ | * Three types of foreground models: SED parametrization, spatial distribution of foreground amplitude, spatial distribution of SED parameters. | ||

+ | * SED parametrization -- we have a version of this in parametric likelihood pipeline | ||

+ | ** Synchrotron SED curvature? Multiple dust populations? Polarized AME? Frequency-dependent beta_dust? | ||

+ | ** Many of these are already available in, e.g., PySM | ||

+ | * Spatial variation in foreground amplitudes | ||

+ | ** Assuming power law in ell so far. Does this work at high / low ell? | ||

+ | ** What is realistic non-Gaussianity? This is a big modeling challenge. | ||

+ | ** How does dust-sync correlation vary with scale? Would like to incorporate these sorts of models into data challenges. | ||

+ | * Spatial distribution of spectral parameters | ||

+ | ** Currently being addressed with decorrelation parameters, but decorrelation not detected in current data. | ||

+ | ** What is scale dependence, frequency dependence? | ||

+ | ** How to make realistic models with decorrelation? | ||

+ | * Ancillary data | ||

+ | ** Expand frequency coverage, i.e. S-PASS, C-BASS. Faraday rotation is a problem at very low frequencies. Do we need to measure synchrotron at high ell? | ||

+ | ** Measure dust at high frequencies. Lots of signal, but might not be useful if it decorrelates with CMB frequencies. Hard to predict without measuring. CIB separation (in temperature) makes this difficult. | ||

+ | ** Other ancillary data might be helpful, but probably won’t correlate better than ~90%. Examples: HI, dust extinction, starlight polarization, molecular lines. | ||

+ | ** Can probe foregrounds and magnetic field in 3 dimensions. | ||

+ | * Can we extrapolate from foreground-bright to faint regions? | ||

+ | ** Could test foreground models at high signal-to-noise. | ||

+ | ** But bright regions are systematically different because you are averaging over many regions along line of sight. Pick up molecular gas at high column density. | ||

+ | * Path forward | ||

+ | ** Critical needs from data analysis: non-Gaussian foregrounds at high ell (how high?), physically realistic frequency decorrelation (what is the range of reasonable values?) | ||

+ | ** Iterate data analysis with foreground modelers | ||

+ | ** How could we incorporate ancillary data sets? | ||

+ | *** Should we be thinking about this for forecasting? | ||

+ | |||

+ | === [https://docs.google.com/presentation/d/1nvKwzamazJmfpInylkX1JIgCbG9AnlYi5nRjb8MdOV8/edit#slide=id.p SAT Error Analysis Discussion] (McMahon) === | ||

+ | |||

+ | * Flowdown is working on error analysis -- tool to justify many aspects of flowdown, evaluate trades. Need this for upcoming reviews. | ||

+ | * Example outputs for flowdown: detector requirements, optics requirements, justification of band choices, readout crosstalk, shielding. | ||

+ | * High level flowdown: noise → science. But noise needs to incorporate everything within reason, i.e. systematics, etc. | ||

+ | * Need to think about sensitivity and systematics on a unified footing. Certain systematics might be able to demonstrate as irrelevant, but not in all cases. | ||

+ | * What is needed to be able to simulate tricky problems like edge taper, crosstalk, etc? | ||

+ | * Need to get low-ell BB participation on flowdown calls. | ||

+ | * Q: Output of tool is sigma(r) or bias on r? A: Probably bias, but a lot of these questions will have to be specific studies, not general tool. This is going to be very difficult work. | ||

+ | * Comment: Seems unrealistic to get this to a point where it can be run by engineers. Lots of assumptions built in, so it needs to be run carefully by people who understand. A: Engineers should be writing documentation, not determining solutions. | ||

+ | |||

+ | === [https://docs.google.com/presentation/d/1FrartvnO9tZtD26hHonQqK1AeoLMzPXZOutqw3BezxE/edit Summary of design justification studies] (Wu) === | ||

+ | |||

+ | * Ran out of time before this presentation. Continue discussion in Low-ell BB telecon. |

## Latest revision as of 16:30, 31 March 2020

## Contents

- 1 Connection details
- 2 Charge
- 3 Agenda
- 4 Notes
- 4.1 Data Challenge 06: parametric likelihood pipeline (Bischoff)
- 4.2 Bayesian optimal lensing (Millea)
- 4.3 SO-BB map-based parametric pipeline (Errard)
- 4.4 ILC on DC06 maps (Umiltà)
- 4.5 Iterative curved-sky lensing estimates for cmbs4 deep (Carron)
- 4.6 Low-ell Foregrounds (Hensley)
- 4.7 SAT Error Analysis Discussion (McMahon)
- 4.8 Summary of design justification studies (Wu)

## Connection details

Parallel 1, session D. Tuesday March 31, 9:00 to 11:00 am pacific.

https://fnal.zoom.us/j/5196960377

Monitor: Kimmy Wu

Scribe: Colin Bischoff, google doc for notes (to be transferred to wiki)

## Charge

The primary focus of this meeting will be technical and scientific progress towards the baseline design in general and the summer agency review in particular.

- Present results from or statuses of data challenge 6 with real delensing incorporated
- Foreground models / sims
- Impact of foregrounds on delensing.
- Foregrounds at low-ell

- Sketch path(s) forward for data challenges:
- What questions need answering by the summer’s agency review that need results from data challenges?

- Outline trade studies that need to be done (or present results of trade studies) for the delensing LAT
- Sensitivity vs resolution
- Frequency allocation

- Outline trade studies that need to be done (or present results of trade studies) for SATs
- Do we need 20GHz?

## Agenda

- Results from data challenge 06 [45 min]
- Next steps for data challenges [45 min]
- Inputs for summer agency review [20 min]

## Notes

### Data Challenge 06: parametric likelihood pipeline (Bischoff)

- Review of Data Challenge 06:
- 8 SAT freq + 20 GHz on LAT; 6 LAT freq. Previous data challenge, approximate delensing by reducing the lensing signal by expected AL residuals.
- 3 Galactic foreground models: 2 are non-Gaussian, 1 is Gaussian.

- Lensing template used is generated on curve sky with QE.
- Computed correlation between lensing template and map.
- Observed that there is no measured correlation between the 95GHz (foreground-ful) maps and the lensing template. This is on Gaussian foregrounds, so it’s not entirely surprising.
- Q: why is there a lower correlation below ell=50. A: the modes below ell~30 are artificially removed to approximate that we can’t measure modes from the reionization bump.
- Q: Is constructing the covariance matrix with Gaussian sims a good enough approximation? A: The likelihood assumes the bandpowers to be non-Gaussian (H-L likelihood), but still assumes the statistics of the maps are Gaussian. Something to check is at what point these assumptions break down.

### Bayesian optimal lensing (Millea)

- All versions of delensing (Bayesian, iterative QE, etc) can be derived from lensing posterior probability.
- MC sampling of posterior is the best you can do.
- Can find maximum likelihood at fixed cosmological parameters -- obtain joint maximum a posteriori (MAP) estimate of CMB and phi.
- Can integrate over CMB fields to obtain marginal MAP estimate of phi -- this is the iterative QE.
- For Gaussian distributions, joint MAP phi is the same as marginal MAP phi, but this is not the case for non-Gaussian lensing.

- Comparison of approaches to this problem: sampling vs joint MAP vs marginal MAP
- Sampling is computationally hard, joint MAP is computationally easiest.
- Samples give you unbiased cosmological parameter estimation, but other methods work pretty well to delens for r because of separation of scales.
- Software packages: CMBLensing.jl, LensIt

- Lensing templates from joint or marginal MAP yield A_lens ~ 0.1 for DC 06b 155 GHz (with no masking).
- When you add a mask, joint MAP picks up a huge mean field. Leads to large decorrelation with true phi, but almost no effect on lensing template!

- Used sampling to check S4 delensing forecast. Found good agreement.
- Applying sampling analysis to SPT data. It is working on 100 deg^2. Have checked that computation can be done on 1500 deg^2.
- Working to add curved sky support to CMBLensing.jl
- Made flat sky joint MAP lensing templates that can be used with data challenge 06b. Think that flat sky leads to ~5% sub-optimality.

- Q: Does comparison of joint vs marginal MAP results change in presence of foregrounds? A: They should respond differently to foregrounds. Need to try it and see.

### SO-BB map-based parametric pipeline (Errard)

- Assume that noisy frequency maps are made up of sky model multiplied by parametrized mixing matrix plus noise. Use noise covariance to estimate foreground spectral parameters, then estimate sky signals. Similar to COMMANDER.
- SO BB pipeline:
- Simulate noisy frequency maps (can replace with DC06)
- fgbuster module for component separation (can also delens here)
- NaMaster module for pure-B power spectrum estimation
- emcee module to sample r likelihood, with optional marginalizations

- Applying this pipeline to CMB-S4 instrument configuration. Some issues with bandpasses, noise covariance. Using PySM sky signal.
- Obtain good constraints on foreground spectral indices, CMB/dust/sync maps.
- Some jobs still in queue at NERSC. Get sigma(r) ~ 3e-3 for case without delensing. Looking forward to cases with lower A_lens.

- Comment: Note different hitmaps between the SAT patterns and LAT patterns.
- Needed in order to run on DC 06
- Definition of bandpasses -- Colin to discuss with Josquin
- Noise simulator for pixel-based noise covariance -- should be able to use the existing noise simulations
- Splitting the patch to deal with spectral index variations.
- Delensing -- have incorporated Carron lensing templates

### ILC on DC06 maps (Umiltà)

- Linear combination of maps with weights estimated from covariance matrix of data maps. Constrained weights to preserve CMB signal. ILC in spherical harmonics basis.
- Tested using three different sky masks for DC 06.
- Currently only have 10 realizations. These have already been masked. Beam is equivalent to 95 GHz LAT beam.
- Q: slide 6 - the ILC map-CMB input shows some effects of the cleaning of foreground, but it can also be ambiguous cleaning. In some cases, 95GHz-CMB has lower residuals than the ILC map-CMB. There is trickiness when masks are applied even though by definition the ILC weights minimize the variance of the output map.
- Q on slide 8: hit map vs smoothed map are two masks used as shown on slide 3.
- Comment: Could smooth weights in ell, if desired.
- Comment: For purposes of producing lensing template for r analysis, best mask might be the apodized mask from SATs. Julien’s lensing pipeline does take variance map -- thinks that should be able to obtain unbiased map by mask or have Caterina do this.
- Q: What about B modes? A: How close the ILC map is to input B modes depends on foregrounds, E->B mixing.

### Iterative curved-sky lensing estimates for cmbs4 deep (Carron)

- Iterative estimator working on curve sky on the Pole deep map of DC06.
- MAP phi map for given cosmology, marginalized over unlensed fields. Include mean field subtraction.
- Suspect that the computational cost is comparable to the joint MAP.
- One of the computational bottleneck was due to the non-Gaussianity of the posterior, one way around it is to do a L-dependent step length.
- Demonstrates that with no-foreground sims, the Alens residuals with a marginal MAP (on curve sky) is 7%. (The QE on slide 3 is an improved version than those supplied for r-analysis so far.)
- Technical details: ignoring changes in mean-field (just use the one from the first step) is ok. Ignoring the phi-induced MF is okay. Here use EB modes between 200 and 3000.
- The residual lensing B mode power from this run is A_L~0.1. (Different color lines on slide 5 correspond to the number of iteration. Blue is step1, and +1 for all the other colors. The purple MAP line is the final converged version, at ~step 8.)

### Low-ell Foregrounds (Hensley)

- Three types of foreground models: SED parametrization, spatial distribution of foreground amplitude, spatial distribution of SED parameters.
- SED parametrization -- we have a version of this in parametric likelihood pipeline
- Synchrotron SED curvature? Multiple dust populations? Polarized AME? Frequency-dependent beta_dust?
- Many of these are already available in, e.g., PySM

- Spatial variation in foreground amplitudes
- Assuming power law in ell so far. Does this work at high / low ell?
- What is realistic non-Gaussianity? This is a big modeling challenge.
- How does dust-sync correlation vary with scale? Would like to incorporate these sorts of models into data challenges.

- Spatial distribution of spectral parameters
- Currently being addressed with decorrelation parameters, but decorrelation not detected in current data.
- What is scale dependence, frequency dependence?
- How to make realistic models with decorrelation?

- Ancillary data
- Expand frequency coverage, i.e. S-PASS, C-BASS. Faraday rotation is a problem at very low frequencies. Do we need to measure synchrotron at high ell?
- Measure dust at high frequencies. Lots of signal, but might not be useful if it decorrelates with CMB frequencies. Hard to predict without measuring. CIB separation (in temperature) makes this difficult.
- Other ancillary data might be helpful, but probably won’t correlate better than ~90%. Examples: HI, dust extinction, starlight polarization, molecular lines.
- Can probe foregrounds and magnetic field in 3 dimensions.

- Can we extrapolate from foreground-bright to faint regions?
- Could test foreground models at high signal-to-noise.
- But bright regions are systematically different because you are averaging over many regions along line of sight. Pick up molecular gas at high column density.

- Path forward
- Critical needs from data analysis: non-Gaussian foregrounds at high ell (how high?), physically realistic frequency decorrelation (what is the range of reasonable values?)
- Iterate data analysis with foreground modelers
- How could we incorporate ancillary data sets?
- Should we be thinking about this for forecasting?

### SAT Error Analysis Discussion (McMahon)

- Flowdown is working on error analysis -- tool to justify many aspects of flowdown, evaluate trades. Need this for upcoming reviews.
- Example outputs for flowdown: detector requirements, optics requirements, justification of band choices, readout crosstalk, shielding.
- High level flowdown: noise → science. But noise needs to incorporate everything within reason, i.e. systematics, etc.
- Need to think about sensitivity and systematics on a unified footing. Certain systematics might be able to demonstrate as irrelevant, but not in all cases.
- What is needed to be able to simulate tricky problems like edge taper, crosstalk, etc?
- Need to get low-ell BB participation on flowdown calls.
- Q: Output of tool is sigma(r) or bias on r? A: Probably bias, but a lot of these questions will have to be specific studies, not general tool. This is going to be very difficult work.
- Comment: Seems unrealistic to get this to a point where it can be run by engineers. Lots of assumptions built in, so it needs to be run carefully by people who understand. A: Engineers should be writing documentation, not determining solutions.

### Summary of design justification studies (Wu)

- Ran out of time before this presentation. Continue discussion in Low-ell BB telecon.