[DM-36523] AP pipeline infrastructure in S23B - Jira

XML

Word

Printable

Details

Type: Epic
Status: In Progress
Resolution: Unresolved
Fix Version/s: None
Component/s: None
Labels:
None

Epic Name:
s23b-ap-infrastructure
Story Points:
150
WBS:
02C.03.05
Team:
Alert Production
Cycle:
Spring 2023

Description

Infrastructure work to support Alert Production. This includes middleware, verification and testing frameworks, as well as performance tuning.

Attachments

Stories in Epic (Custom Issue Matrix)

Key	Summary	Story Points	Assignee	Status
DM-39414	Experiment with using Flux in AP Sasquatch dashboards	6	Unassigned	To Do
DM-39412	Add Jenkins build ID to ap_verify Sasquatch metadata	2	Krzysztof Findeisen	To Do
DM-39411	Add timing metrics for GetTemplateTask and TransformDiaSourceCatalogTask	3	Unassigned	To Do
DM-38671	Migrate or remove legacy analysis_ap cocde	4	John Parejko	To Do
DM-38646	Write technote describing withdrawal of bad records in AP	4	Eric Bellm	To Do
DM-38869	Write Processing Summary for MakeWarpTask docs	1	Unassigned	To Do
DM-39311	`size` is redundant in plotImageSubtractionCutouts	1	Unassigned	To Do
DM-38966	Create unit tests of apdb schema for ap_association	8	Unassigned	To Do
DM-38286	NaiveCentroid/SdssCentroid and CentroidChecker have mismatched names	1	John Parejko	To Do
DM-38247	Include timestamps in ap_verify logs	2	Unassigned	To Do
DM-38230	Update defaultNames in meas_astrom to match conventions	3	Erin Howard	To Do
DM-38215	Parallelize upload_hsc_rc2.py	4	Krzysztof Findeisen	To Do
DM-37081	numAssociatedSsObjects metric produced without solar system association	1	Unassigned	To Do
DM-37054	Rename TransformDiaSourceCatalogTask and update its description		Unassigned	To Do
DM-39071	Create new lsst repo for SolarSystem Tasks	4	Ari Heinze	To Do
DM-38991	Update ap_verify refcats	4	Unassigned	To Do
DM-38319	Introduce top-level packages in developer guide	2	Krzysztof Findeisen	To Do
DM-38096	Update BPS docs for USDF	4	Erin Howard	To Do
DM-38117	Explore renaming SAT->SATURATED and INTRP/INTERP->INTERPOLATED	2	Unassigned	To Do
DM-37299	USDF Prompt Processing system running very slowly	6	Unassigned	To Do
DM-37555	Update pipeline documentation for image differencing	2	Bruno Sanchez	To Do
DM-37420	Cutout failures: missing dataIds	2	Unassigned	To Do
DM-36511	Create LATISS templates for Embargo Repo		Unassigned	To Do
DM-36505	Propagate database schema/namespace into AP configurations	4	Unassigned	To Do
DM-36477	Remove ap_verify_hits2015 dataset	1	Krzysztof Findeisen	To Do
DM-38452	Update ap_verify DC2 templates	2	Erin Howard	To Do
DM-38450	Update ap_verify HSC templates	2	Erin Howard	To Do
DM-37911	Write dipoleNdata, dipoleLnl, dipoleCov fields to APDB	4	Brianna Smart	To Do
DM-36319	Fix Prompt Prototype crashes for repeat APDB operations	6	Krzysztof Findeisen	To Do
DM-27486	Investigate decam calibrate refcat mag limit configs	2	Meredith Rawls	To Do
DM-21382	Separate GenericMap's visitor API from its implementation	6	Krzysztof Findeisen	To Do
DM-21483	Make a new shared stack of the last n dailies on the SLAC dev nodes, once they exist	1	Nate Lust	To Do
DM-31836	Write gen3 formatter for Zooniverse data	6	Harshit Rai	To Do
DM-32138	Remove VisitInfo.exposureId	6	Krzysztof Findeisen	To Do
DM-21268	GenericMap can't distinguish int/float sizes in Python	6	Krzysztof Findeisen	To Do
DM-21216	GenericMap converts C-strings to bool	2	Krzysztof Findeisen	To Do
DM-35055	Assist with the creation of timing metrics for DRP	2	Krzysztof Findeisen	To Do
DM-33746	Rename ap_verify.pipeline_driver	1	Krzysztof Findeisen	To Do
DM-34925	Investigate memory use spike at end of DcrAssembleCoaddTask	2	Ian Sullivan	To Do
DM-34906	Systematic error in ephemerides in ap_verify datasets	6	Unassigned	To Do
DM-34769	Remove characterizeImage refObjLoader hooks	1	John Parejko	To Do
DM-34758	DiaCalculation plugins report fatal errors	2	Unassigned	To Do
DM-33453	Update ap_pipe pipelines to match drp_pipe naming	2	Erin Howard	To Do
DM-22960	Fix warnings in DcrAssembleCoadd	2	Ian Sullivan	To Do
DM-23088	ImagePsfMatchTask has no run method		Unassigned	To Do
DM-24326	Add a PSF object to the Astropy CCData representation	8	John Parejko	To Do
DM-25160	Record use of CCDData alert stamps in DPDD and DMTN-093	4	Eric Bellm	To Do
DM-24949	Clarify dev guide documentation on adding a new package	0.5	Unassigned	To Do
DM-36796	Rename config/pipeline directories in ap_pipe and ap_verify	3	Krzysztof Findeisen	To Do
DM-36744	Add real-bogus CI output to Chronograf	2	Harshit Rai	To Do
DM-35846	Investigate approaches for enforcing sdm_schema in the output of ap_association	2	Unassigned	To Do
DM-35828	Use MDC or equivalent in prompt_prototype logging	6	Krzysztof Findeisen	To Do
DM-36051	Emulate DMTN-143 image transfer in Prompt Processing Prototype	6	Unassigned	To Do
DM-35504	Unify ap_pipe configs	2	Unassigned	To Do
DM-35302	Source detection config metadata propagated to output catalogs	2	John Parejko	To Do
DM-35649	Create task docs for CreateRandomApFakesTask	1	Unassigned	To Do
DM-35646	Rewrite ap_pipe doc intro to match RFC-775 role	1	Unassigned	To Do
DM-16130	Move image rotation calculation to VisitInfo.cc	2	Ian Sullivan	To Do
DM-16119	Investigate additional pixels masked with "NO_DATA" after DCR shift	3	Ian Sullivan	To Do
DM-16078	convert coadd_utils to numpydoc		John Swinbank	To Do
DM-19469	Inconsistent pybind11 documentation	4	Unassigned	To Do
DM-19912	Rename fluxMin and fluxMax in various star/source selector configs		Unassigned	To Do
DM-17692	Examine ref and source flux error propagation in PhotoCalTask and colorterms		Unassigned	To Do
DM-16524	HtmIndexer.indexPoints should take coords (as Angles) for input		Unassigned	To Do
DM-17070	Clean up lsst.afw.multiband	2	Unassigned	To Do
DM-16861	Convert afw.image to numpydoc	6	Unassigned	To Do
DM-16773	ap_verify_hits2015 hard to manage with lsstsw		Unassigned	To Do
DM-16662	Require changes to Jenkins conda environment to be RFCed	1	John Swinbank	To Do
DM-17458	Complete docstrings in ip_diffim	2	Unassigned	To Do
DM-15738	Remove daf::base::Persistable	6	Unassigned	To Do
DM-13602	Set association DB location at run time, not config time	4	Unassigned	To Do
DM-13170	Make SkyWcs.getFitsMetadata fit a TAN-SIP WCS if necessary	4	Unassigned	To Do
DM-11995	Make grid computation for ImageMapReduce more robust	4	Unassigned	To Do
DM-15559	makeUpdatedDetector should update readoutCorner		Unassigned	To Do
DM-15538	pybind11 bindings for DateTime are incorrect/incomplete		John Parejko	To Do
DM-14984	The default value for isr.doLinearize should be False		Unassigned	To Do
DM-15038	Update TransformMap to use ast::FrameDict	2	Unassigned	To Do
DM-10083	Rename overscan/prescan accessors in AmpInfo	8	Unassigned	To Do
DM-10640	Remove use of cout for error reporting in GaussianProcess	2	Unassigned	To Do
DM-10629	Remove ints used as booleans from GaussianProcess	1	Unassigned	To Do
DM-10628	Remove duplicate code from GaussianProcess.*	3	Unassigned	To Do
DM-11248	SingularTransformException not wrapped in pybind11	2	Unassigned	To Do
DM-15762	surface brightness/fluence flags in ExposureInfo and PhotoCalib	6	John Parejko	To Do
DM-7565	Add bbox to ExposureInfo	10	Unassigned	To Do
DM-7102	Add centroid reliability flag or improve Centroid_flag behavior		Unassigned	To Do
DM-7101	Output centroid covariance		Unassigned	To Do
DM-7100	Add blended/isolated flags		Unassigned	To Do
DM-6070	Test that measurement plugins get aperture-corrected fluxes		Unassigned	To Do
DM-5680	makePsfCandidate needs to be easier and safer to use		Unassigned	To Do
DM-5658	Update tables of packages that depend on scipy	2	Unassigned	To Do
DM-15164	Augment crosstalk correction test case	4	Meredith Rawls	To Do
DM-14262	Cleanup fallout from removing interpolated from badFlags list	4	Unassigned	To Do
DM-14136	convert ip_isr documentation to numpydoc	6	Unassigned	To Do
DM-37253	Make Prompt Processing service configurable	6	Krzysztof Findeisen	In Progress
DM-36608	Create metric for Nsources per DiaObject as a function of magnitude/seeing	4	Brianna Smart	In Progress
DM-38449	Update ap_verify DECam templates	2	Erin Howard	In Progress
DM-32696	Improve ds9 wrapper	1	Nima Sedaghat	In Progress
DM-35309	Improve efficiency of trailed source measurement	1	Brianna Smart	In Progress
DM-38190	Write up discussion about failure modes in Prompt Processing	9	Krzysztof Findeisen	In Review
DM-23592	Raise descriptive error for broken display install (GitHub afw #516)	1	Ian Sullivan	In Review
~~DM-39395~~	Use day_obs in run names	1	Krzysztof Findeisen	Done
~~DM-39378~~	Rename "spuriousness" to "reliability" in the DiaSource table	2	Harshit Rai	Done
~~DM-38605~~	Rename analysis_ap's zooniverseCutouts task, script, and docs	1	John Parejko	Done
~~DM-38752~~	Output products can go to the wrong collection	2	Krzysztof Findeisen	Done
~~DM-38743~~	Get familiarized with running ap_verify from the command line.	2	Harshit Rai	Done
~~DM-38891~~	Add setup_module() to the test suits(s) of meas_transiNet package	1	Nima Sedaghat	Done
~~DM-38873~~	Remove outdated doc build instructions	1	Brianna Smart	Done
~~DM-38853~~	Change ap_verify CI packages to use Butler.transfer_from	2	Krzysztof Findeisen	Done
~~DM-39290~~	DM-39286 seems to cause problems with ci_imsim	1	Ian Sullivan	Done
~~DM-38602~~	Include failing VisitInfo serialization version in error message	1	John Parejko	Done
~~DM-38300~~	Stringification of an afwDetection.Threshold with stdev raises exception	0.5	John Parejko	Done
~~DM-38225~~	Intermittent init-output collision in Prompt Processing	2	Krzysztof Findeisen	Done
~~DM-38191~~	IPython import in nb_utils fails flake8 on Jenkins	1	John Parejko	Done
~~DM-37196~~	Modernize APDB schema to reflect desired usage	18	John Parejko	Done
~~DM-39231~~	DM-38846 broke documenteer (pipelines.lsst.io)	1	Ian Sullivan	Done
~~DM-39127~~	DM-36743 broke ap_verify	1	Nima Sedaghat	Done
~~DM-39124~~	DM-37147 breaks ci_imsim	1	Eric Bellm	Done
~~DM-39086~~	Out-of-date docs in butler prune-datasets	1	Krzysztof Findeisen	Done
~~DM-39072~~	Move precursor solar system validation datasets to the USDF	6	Ari Heinze	Done
~~DM-38981~~	Update the ap_verify tutorial according to recent changes: fetch model packages		Harshit Rai	Done
~~DM-37421~~	Cutout failures: diaSource outside bbox	4	Brianna Smart	Done
~~DM-36696~~	Make pex.config example docs be full sentences	2	Brianna Smart	Done
~~DM-38481~~	Add meas_transiNet to ap_pipe	2	Nima Sedaghat	Done
~~DM-38421~~	Add secure login directions to DMTN-214	1	Brianna Smart	Done
~~DM-37801~~	Move diffim and meas_algorithms task docs from python files to ReST	7	Brianna Smart	Done
~~DM-37673~~	MakeWarpTask needs a task topic doc page	4	Brianna Smart	Done
~~DM-36162~~	Upgrade from SimplePipelineExecutor in prompt prototype	15	Krzysztof Findeisen	Done
~~DM-21525~~	Compiler warning about catching polymorphic exceptions by value	0.5	Matthias Wittgen	Done
~~DM-36743~~	Add the real-bogus PipelineTask to ap_verify for the DC2 CI dataset	6	Nima Sedaghat	Done
~~DM-36742~~	Add real-bogus trained model to DC2 CI repository	8	Harshit Rai	Done
~~DM-35962~~	Remove examples/ from all packages	4	John Parejko	Done
~~DM-38916~~	Link to bind documentation from query methods	2	Krzysztof Findeisen	Done
~~DM-32265~~	ap_verify_hits2015 detectors 2 and 61 broken in gen3		Unassigned	Won't Fix
~~DM-32771~~	Persist analysis_ap plots with the butler	4	Meredith Rawls	Won't Fix
~~DM-28524~~	Add versioning to Persistable	8	Unassigned	Won't Fix

Activity

There are no comments yet on this issue.

People

Assignee:: Ian Sullivan

Reporter:: Ian Sullivan

Watchers:: Ian Sullivan

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 07/Oct/22 10:41 PM

Updated:: 25/May/23 7:22 PM

Jenkins

No builds found.

Data Management