https://pfspipe.ipmu.jp/jira This file is an XML representation of an issue en-us 8.3.4 803005 13-09-2019 https://pfspipe.ipmu.jp/jira/browse/PIPE2D-309 DRP 2-D Pipeline <p>Distill a week's worth of brain dumps into a design for the 2D pipeline.</p> PIPE2D-309

Write DRP 2D pipeline design document

Story Normal Done Done price price Mon, 5 Nov 2018 21:28:14 +0000 Fri, 15 Feb 2019 18:27:00 +0000 Thu, 17 Jan 2019 15:56:51 +0000 0 3 <p>My raw notes:</p> <div class="code panel" style="border-width: 1px;"><div class="codeContent panelContent"> <pre class="code-java"> As per imaging pipelines: ingestImages constructBias constructDark ingestCalibs constructFiberFlat: * Input: dithered quartz exposures <span class="code-keyword">for</span> an arm * For each input: + Apply ISR + Mask CRs * Combine * Output: imaging flat calib constructFiberTrace * Input: raw quartz exposures at slitOffset=0 <span class="code-keyword">for</span> a spectrograph, detectorMap, pfiConfig * For each input: + Apply ISR + Mask CRs + Find and centroid traces; measure profiles - Alternatively: using detectorMap, re-centroid each trace * Merge traces * Identify traces from detectorMap * Extract spectra * Normalise each trace profile by spectrum divided by reference spectrum (unity, flat, 4500K black body???) + This needs to be done over arms within the same spectrograph * Output: trace calib Extract spectra: * Needs to work when the traces overlap + Simultaneous fit, (tri-diagonal) matrix inversion (can be inverted in linear time) + Alternative: extract just the peaks, normalise the spectra, subtract from the image, then put each spectrum in one by one and extract as normal constructDetectorMap * Input: arc exposures <span class="code-keyword">for</span> an arm, trace, detectorMap * For each input: + Apply ISR + Mask CRs + Extract spectra - Optionally subtract continuum and write calexp + For each fiber: - Identify lines - Get list of line, dlambda * For each fiber: + Fit residuals + Update detectorMap * Output: modified detectorMap calib constructPsf * Input: raw donut exposures, fiberTrace, detectorMap * For each exposure: + Apply ISR + Mask CRs * Fit donuts with Neven's magic * Output: PSF model parameters calib Want to avoid deleting and re-ingesting detectorMap (potential <span class="code-keyword">for</span> trouble). How about we either read the initial detectorMap from a fixed file in obs_pfs, or use a different dataset type (e.g., <span class="code-quote">"bootstrapDetectorMap"</span>)? reduceExposure: * Input: raw <span class="code-keyword">for</span> same arms <span class="code-keyword">for</span> entire FoV, fiberTrace, detectorMap, pfiConfig * For each arm: + Apply ISR + Mask CRs - Do we want to fit a NevenPsf here <span class="code-keyword">for</span> better CR finding performance? + Optionally tweak detectorMap based on sky lines * subtractSky2d * For each arm: + Extract spectra * For each arm: + Apply barycentric wavelength correction * Output: pfsArm, calexp mergeArms: * Input: pfsArm <span class="code-keyword">for</span> entire FoV, pfiConfig, fiberTrace * subtractSky1d (can <span class="code-keyword">do</span> after merging <span class="code-keyword">if</span> the dichroic causes trouble) * Merge arms * Output: pfsMerged, pfsPsf, pfsSky1d, pfsSky2d calculateReferenceFlux: * Input: pfiConfig, pfsMerged <span class="code-keyword">for</span> entire FoV, grid of F-star model spectra * For each F-star spectrum: + Get best-fit model spectrum + Optionally use broad-band photometry to scale model * Output: pfsReference fluxCalibrate: * Input: pfsMerged <span class="code-keyword">for</span> entire FoV, pfsReference, pfiConfig * With F-star spectra: + Model flux calibration as a function of RA,Dec + Weight by distance from nominal fiber position? * For each spectrum: + Apply flux calibration model + Optionally tweak the flux calibration to match broad-band photometry * Output: pfsFlux, pfsObject coaddSpectra: * Input: a large selection of pfsArm files * For each object: + Note: we need to use the original extracted spectra (i.e., un-resampled), which means we need to read the pfsArm and apply the various corrections (sky subtraction, barycentric, flux calibration) + Coadd spectra * Output: pfsCoadd subtractSky2d: * Input: pfsArm+calexp <span class="code-keyword">for</span> entire FoV, pfiConfig, fiberTrace * Remove sky continuum - Fit continuum to sky fibers - Model as a function of RA,Dec - Subtract from all fibers in 2D * Remove sky lines - Model sky line intensity as a function of RA,Dec - Generate PSF at sky line position from fiberId,wavelength; scale and subtract * Output: pfsSky2d, pfsSkySub2d subtractSky1d: * Input: pfsArm <span class="code-keyword">for</span> entire FoV, pfiConfig * With sky fibers: + Model spectrum as a function of ra,dec * Subtract sky model from all inputs * Output: pfsSky1d, pfsSkySub1d Note: the datamodel appears to combine the concepts of format (class) and filename template (datasetType), whereas I think these should be separate. For example, a PfsObject <span class="code-keyword">class </span>might be used to persist spectra before and after flux calibration. Dataset names (class, <span class="code-keyword">if</span> important) * pfsArm (PfsGroup): Extracted spectra from a single arm. Only a very basic calibration has been applied (relative flux calibration using the quartz, wavelength from whatever detectorMap we have), and no sky subtraction. * pfsMerged (PfsGroup): Merged (spectra from different arms combined), sky-subtracted spectra <span class="code-keyword">for</span> the full FoV. Barycentric wavelength correction has been applied, but not flux calibrated (only relative flux calibration using the quartz). * pfiConfig: Configuration of the Prime Focus Instrument: mapping of fiber to object. * fiberTrace: Position and profile of each fiber as a function of row. * detectorMap: Position and wavelength of each fiber as a function of row. * pfsPsf: PSF fit <span class="code-keyword">for</span> arm. * pfsSky1d: 1D sky fit <span class="code-keyword">for</span> arm. * pfsSky2d: 2D sky fit <span class="code-keyword">for</span> arm. * pfsSkySub2d: Result from subtracting sky in 2D (debugging output). * pfsSkySub1d: Result from subtracting sky in 1D (debugging output). * pfsReference: Reference flux spectra <span class="code-keyword">for</span> objects within a FoV. * pfsFlux: Flux calibration solution <span class="code-keyword">for</span> FoV. * pfsObject (PfsObject): Merged, flux calibrated spectrum from a single epoch. * pfsCoadd (PfsObject): Coadded flux calibrated spectrum from multiple epochs. Dataset spectrum classes: * PfsGroup (suggestions <span class="code-keyword">for</span> better name are welcome!): A group of extracted spectra, e.g., from an arm or a full FoV. * PfsObject: A single spectrum, <span class="code-keyword">for</span> a single object. <span class="code-keyword">class </span>PfsPsf(Psf): Image computeImage(std::size_t fiberId, <span class="code-object">float</span> wavelength) { <span class="code-keyword">return</span> nevenImpl(_detectorMap.wavelengthToPoint(wavelength, fiberId), fiberId); } Image computeImage(Point2D p) { <span class="code-keyword">return</span> nevenImpl(p, detectorMap.getFiberId(p)); } Image nevenImpl(Point2D p, std::size_t fiberId); <span class="code-keyword">private</span>: DetectorMap _detectorMap; } Kernel1D calculateLineSpreadFunction(Psf, FiberTrace, std::size_t fiberId, <span class="code-object">float</span> wavelength); </pre> </div></div> <p><a href="https://pfspipe.ipmu.jp/jira/secure/ViewProfile.jspa?name=hassan" class="user-hover" rel="hassan">hassan</a>, shall we merge this?</p> <p>Ok. Remember we also would still need to write or generate an API document for the individual python methods.</p> <p>That's much easier now that we have an end-to-end pipeline where some thought has been put into the interfaces.</p> <p>Merged to master.</p> <p>What was merged to master?  Can you provide a URL in this ticket.</p> <p><a href="https://github.com/Subaru-PFS/drp_doc/tree/master/design" class="external-link" rel="nofollow">https://github.com/Subaru-PFS/drp_doc/tree/master/design</a></p> <p>That link is to the whole directory. Please provide a link to the document.</p> <p>Attached the document in PDF form.</p> Development Rank 0|s001ns: Reviewers hassan