[DM-36525] AP pipeline infrastructure in F23B - Jira

XML

Word

Printable

Details

Type: Epic
Status: In Progress
Resolution: Unresolved
Fix Version/s: None
Component/s: None
Labels:
None

Epic Name:
f23b-ap-infrastructure
Story Points:
102
WBS:
02C.03.05
Team:
Alert Production
Cycle:
Fall 2023

Description

Infrastructure work to support Alert Production. This includes middleware, verification and testing frameworks, as well as performance tuning.

Attachments

Stories in Epic (Custom Issue Matrix)

Key	Summary	Story Points	Assignee	Status
DM-40434	Visits get processed twice in Prompt Processing	4	Krzysztof Findeisen	To Do
DM-40400	Create total AP pipeline timing metric	6	Krzysztof Findeisen	To Do
DM-40397	Add ap_verify pipeline with CalibrateImageTask to CI	2	John Parejko	To Do
DM-40388	Switch AP pipelines to use new CalibrateImageTask	1	John Parejko	To Do
DM-40313	Migrate ap_verify to SeparablePipelineExecutor	4	Krzysztof Findeisen	To Do
DM-40245	Intelligently identify datasets to preload	8	Unassigned	To Do
DM-40193	Disk space leak in prompt processing (aka DNS failures)		Unassigned	To Do
DM-39961	Investigate what other flags we want to propagate to APDB	2	Unassigned	To Do
DM-39935	Remove MaskedImage.getArrays()	1	John Parejko	To Do
DM-40820	Simulate one-visit-in-advance notification in uploader(s)	6	Krzysztof Findeisen	To Do
DM-40796	Remove "FAKE" mask plane check	1	Bruno Sanchez	To Do
DM-41713	Many calibs (bias/flat/dark) preloaded in LATISS testing	2	Krzysztof Findeisen	To Do
DM-41670	Account for non-atomic APDB writes in error-handling	4	Krzysztof Findeisen	To Do
DM-41667	Remove services still using the prompt-proto name	1	Krzysztof Findeisen	To Do
DM-41666	Migrate Prompt Processing fan-out to use renamed service	2	Krzysztof Findeisen	To Do
DM-38117	Explore renaming SAT->SATURATED and INTRP/INTERP->INTERPOLATED	2	Unassigned	To Do
DM-38319	Introduce top-level packages in developer guide	2	Krzysztof Findeisen	To Do
DM-38286	SdssCentroid and CentroidChecker have mismatched names	1	John Parejko	To Do
DM-38230	Update defaultNames in meas_astrom to match conventions	3	Erin Howard	To Do
DM-41163	Mention dataset maintenance scripts in PR template	1	Krzysztof Findeisen	To Do
DM-41147	Fix instrument overrides of ApVerify pipelines		Unassigned	To Do
DM-41596	Expand quantum graph error detection	4	Unassigned	To Do
DM-41533	Add individual flags to alert packets	4	Unassigned	To Do
DM-41532	Remove bit-packed flags field from APDB and alert packets	4	Unassigned	To Do
DM-41531	Output new DiaSource flag fields	4	Unassigned	To Do
DM-41530	Add individual flag fields to APDB schema	4	Unassigned	To Do
DM-41332	Rewrite ap_pipe tutorials using standard repos	4	Unassigned	To Do
DM-33746	Rename ap_verify.pipeline_driver	1	Krzysztof Findeisen	To Do
DM-36319	Fix Prompt Prototype crashes for repeat APDB operations	6	Krzysztof Findeisen	To Do
DM-24326	Add a PSF object to the Astropy CCData representation	8	John Parejko	To Do
DM-25160	Record use of CCDData alert stamps in DPDD and DMTN-093	4	Eric Bellm	To Do
DM-31836	Write gen3 formatter for Zooniverse data	6	Harshit Rai	To Do
DM-38671	Migrate or remove legacy analysis_ap cocde	4	John Parejko	To Do
DM-38646	Write technote describing withdrawal of bad records in AP	4	Eric Bellm	To Do
DM-38869	Write Processing Summary for MakeWarpTask docs	1	Unassigned	To Do
DM-39607	Create tools for interactive ("hover") display of cutouts	12	Anastasios Tzanidakis	To Do
DM-39502	Use new Instrument table in ApdbQuery	1	Unassigned	To Do
DM-39501	Use new DetectorVisitProcessingSummary table	1	Unassigned	To Do
DM-39500	Populate new Instrument table when an APDB instance is created	2	John Parejko	To Do
DM-39498	New APDB table to hold the flag definitions	4	Unassigned	To Do
DM-39497	Write DetectorVisitProcessingSummary to APDB	6	Unassigned	To Do
DM-39472	Investigate interfaces for automated download of data from Zooniverse	4	Eric Bellm	To Do
DM-38966	Create unit tests of apdb schema for ap_association	8	Unassigned	To Do
DM-39311	`size` is redundant in plotImageSubtractionCutouts	1	Unassigned	To Do
DM-39414	Experiment with using Flux in AP Sasquatch dashboards	6	Unassigned	To Do
DM-39880	Clean up test runs in apdb and the central repo	2	Erin Howard	To Do
DM-41346	Change coaddName template default to goodSeeing	4	Unassigned	To Do
DM-36744	Add real-bogus CI output to Chronograf	2	Harshit Rai	To Do
DM-37420	Cutout failures: missing dataIds	2	Unassigned	To Do
DM-35504	Unify ap_pipe configs	2	Unassigned	To Do
DM-35649	Create task docs for CreateRandomApFakesTask	1	Unassigned	To Do
DM-35646	Rewrite ap_pipe doc intro to match RFC-775 role	1	Unassigned	To Do
DM-37054	Rename TransformDiaSourceCatalogTask and update its description		Unassigned	To Do
DM-37299	USDF Prompt Processing system running very slowly	6	Unassigned	To Do
DM-37081	numAssociatedSsObjects metric produced without solar system association	1	Unassigned	To Do
DM-34758	DiaCalculation plugins report fatal errors	2	Unassigned	To Do
DM-35302	Source detection config metadata propagated to output catalogs	2	John Parejko	To Do
DM-35055	Assist with the creation of timing metrics for DRP	2	Krzysztof Findeisen	To Do
DM-34925	Investigate memory use spike at end of DcrAssembleCoaddTask	2	Ian Sullivan	To Do
DM-34906	Systematic error in ephemerides in ap_verify datasets	6	Unassigned	To Do
DM-36505	Propagate database schema/namespace into AP configurations	4	Unassigned	To Do
DM-35846	Investigate approaches for enforcing sdm_schema in the output of ap_association	2	Unassigned	To Do
DM-17070	Clean up lsst.afw.multiband	2	Unassigned	To Do
DM-16861	Convert afw.image to numpydoc	6	Unassigned	To Do
DM-17458	Complete docstrings in ip_diffim	2	Unassigned	To Do
DM-19912	Rename fluxMin and fluxMax in various star/source selector configs		Unassigned	To Do
DM-21216	GenericMap converts C-strings to bool	2	Krzysztof Findeisen	To Do
DM-19469	Inconsistent pybind11 documentation	4	Unassigned	To Do
DM-21382	Separate GenericMap's visitor API from its implementation	6	Krzysztof Findeisen	To Do
DM-17692	Examine ref and source flux error propagation in PhotoCalTask and colorterms		Unassigned	To Do
DM-23088	ImagePsfMatchTask has no run method		Unassigned	To Do
DM-22960	Fix warnings in DcrAssembleCoadd	2	Ian Sullivan	To Do
DM-21483	Make a new shared stack of the last n dailies on the SLAC dev nodes, once they exist	1	Nate Lust	To Do
DM-21268	GenericMap can't distinguish int/float sizes in Python	6	Krzysztof Findeisen	To Do
DM-27486	Investigate decam calibrate refcat mag limit configs	2	Meredith Rawls	To Do
DM-15164	Augment crosstalk correction test case	4	Meredith Rawls	To Do
DM-15038	Update TransformMap to use ast::FrameDict	2	Unassigned	To Do
DM-14984	The default value for isr.doLinearize should be False		Unassigned	To Do
DM-16773	ap_verify_hits2015 hard to manage with lsstsw		Unassigned	To Do
DM-16662	Require changes to Jenkins conda environment to be RFCed	1	John Swinbank	To Do
DM-13170	Make SkyWcs.getFitsMetadata fit a TAN-SIP WCS if necessary	4	Unassigned	To Do
DM-11248	SingularTransformException not wrapped in pybind11	2	Unassigned	To Do
DM-7565	Add bbox to ExposureInfo	10	Unassigned	To Do
DM-16524	HtmIndexer.indexPoints should take coords (as Angles) for input		Unassigned	To Do
DM-15559	makeUpdatedDetector should update readoutCorner		Unassigned	To Do
DM-15538	pybind11 bindings for DateTime are incorrect/incomplete		John Parejko	To Do
DM-10640	Remove use of cout for error reporting in GaussianProcess	2	Unassigned	To Do
DM-10629	Remove ints used as booleans from GaussianProcess	1	Unassigned	To Do
DM-10628	Remove duplicate code from GaussianProcess.*	3	Unassigned	To Do
DM-7102	Add centroid reliability flag or improve Centroid_flag behavior		Unassigned	To Do
DM-7101	Output centroid covariance		Unassigned	To Do
DM-7100	Add blended/isolated flags		Unassigned	To Do
DM-16130	Move image rotation calculation to VisitInfo.cc	2	Ian Sullivan	To Do
DM-16119	Investigate additional pixels masked with "NO_DATA" after DCR shift	3	Ian Sullivan	To Do
DM-16078	convert coadd_utils to numpydoc		John Swinbank	To Do
DM-15738	Remove daf::base::Persistable	6	Unassigned	To Do
DM-15762	surface brightness/fluence flags in ExposureInfo and PhotoCalib	6	John Parejko	To Do
DM-10083	Rename overscan/prescan accessors in AmpInfo	8	Unassigned	To Do
DM-11995	Make grid computation for ImageMapReduce more robust	4	Unassigned	To Do
DM-14262	Cleanup fallout from removing interpolated from badFlags list	4	Unassigned	To Do
DM-14136	convert ip_isr documentation to numpydoc	6	Unassigned	To Do
DM-6070	Test that measurement plugins get aperture-corrected fluxes		Unassigned	To Do
DM-13602	Set association DB location at run time, not config time	4	Unassigned	To Do
DM-5680	makePsfCandidate needs to be easier and safer to use		Unassigned	To Do
DM-5658	Update tables of packages that depend on scipy	2	Unassigned	To Do
DM-40690	Some logs still get shredded in dev dashboard	4	Krzysztof Findeisen	To Do
DM-41665	Create new Phalanx/Argo deployment of Prompt Processing	4	Krzysztof Findeisen	To Do
DM-41047	Keep all flag columns in transformed DiaSource output	6	Erin Howard	In Progress
DM-32696	Improve ds9 wrapper	1	Nima Sedaghat	In Progress
DM-39833	Create user guide for Zooniverse real-bogus classification	4	Harshit Rai	In Progress
DM-37555	Update pipeline documentation for image differencing	2	Bruno Sanchez	In Progress
DM-36608	Create metric for Nsources per DiaObject as a function of magnitude/seeing	4	Brianna Smart	In Progress
DM-33453	Update ap_pipe pipelines to match drp_pipe naming	2	Erin Howard	In Progress
DM-38096	Update BPS docs for USDF	4	Erin Howard	In Review
DM-36958	Deprecate deblend configs in characterizeImage	0.5	John Parejko	In Review
~~DM-40545~~	Fix LRU bug in image subtraction cutouts maker	2	John Parejko	Done
~~DM-40264~~	Reorganize the Zooniverse-related repositories	4	Harshit Rai	Done
~~DM-40163~~	Rework dataset transfer code in MiddlewareInterface	3	Krzysztof Findeisen	Done
~~DM-40981~~	Fix bug in compare_sources	1	Ian Sullivan	Done
~~DM-41729~~	Exposure numbers don't get parsed in Grafana	1	Krzysztof Findeisen	Done
~~DM-41664~~	Rename prompt_prototype package and containers	3	Krzysztof Findeisen	Done
~~DM-41836~~	Prompt Processing may wait too long for raws	1	Krzysztof Findeisen	Done
~~DM-41808~~	Fix refObjLoader breakage in atmospec	1	John Parejko	Done
~~DM-38190~~	Write up discussion about failure modes in Prompt Processing	9	Krzysztof Findeisen	Done
~~DM-38247~~	Include timestamps in ap_verify logs	1	John Parejko	Done
~~DM-41639~~	Allow Apdb queries for visit completion	2	Krzysztof Findeisen	Done
~~DM-41576~~	Assist with AP debugging	2	Meredith Rawls	Done
~~DM-41874~~	Check ap_verify change related to DM-40942	1	John Parejko	Done
~~DM-41126~~	Use PS1 for astrometry in cosmos_pdr2	1	John Parejko	Done
~~DM-41345~~	DM-39895 broke analysis_ap tests	2	John Parejko	Done
~~DM-39822~~	Assess requirements for training RB classifier with a larger dataset	4	Nima Sedaghat	Done
~~DM-39796~~	Update task config defaults to LSST values	16	John Parejko	Done
~~DM-39662~~	Create ApPipeWith Fakes for LsstCamImSim	2	Bruno Sanchez	Done
~~DM-39593~~	Implement automatic fetching of latest labels from Zooniverse for updating our central database	8	Harshit Rai	Done
~~DM-39503~~	Update analysis_ap to reflect APDB schema change	4	John Parejko	Done
~~DM-39071~~	Create new lsst repo for SolarSystem Tasks	4	Mario Juric	Done
~~DM-36772~~	Migrate PostgreSQL credentials to mounted file	7	Krzysztof Findeisen	Done
~~DM-34769~~	Remove characterizeImage refObjLoader	1	John Parejko	Done
~~DM-36638~~	Add snap combination as a subtask at the beginning of CalibrateImageTask	4	John Parejko	Done
~~DM-41730~~	Exception logs don't have metadata	3	Krzysztof Findeisen	Done
DM-38452	Update ap_verify DC2 templates	2	Erin Howard	Reviewed
DM-40839	Remove obsolete environment variables from prompt-proto-service	1	Krzysztof Findeisen	Reviewed
~~DM-36051~~	Emulate DMTN-143 image transfer in Prompt Processing Prototype	6	Unassigned	Won't Fix

Activity

There are no comments yet on this issue.

People

Assignee:: Ian Sullivan

Reporter:: Ian Sullivan

Watchers:: Ian Sullivan

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 07/Oct/22 10:42 PM

Updated:: 4 days ago 6:50 PM

Jenkins

No builds found.

Data Management