Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36519

AP Algorithm development in S23B

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      s23b-ap-algorithms&analysis
    • Story Points:
      160
    • WBS:
      02C.03.04
    • Team:
      Alert Production
    • Cycle:
      Spring 2023

      Description

      Regular analysis of processing of precursor data through AP pipelines, including identifying issues, tracking them down, and resolving them. Includes testing algorithmic changes on large datsets as well as developing and testing timeseries features.
      Develop the tools required for machine learning classification (real-bogus) of image difference cutouts

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
        DM-39371

        Create a centralized database for the upcoming stream of real-bogus labels

        		 Nima Sedaghat To Do
        DM-38862

        Investigate timeseries features computed in DP0.2

        		 20 Anastasios Tzanidakis To Do
        DM-39275

        Compare image differencing results using each mode

        		 4 Ian Sullivan To Do
        DM-38248

        Write concept docs for meas_transiNet

        		 4 Nima Sedaghat To Do
        DM-38968

        Lower number of spurious warning during normal AuxTel processing

        		 Unassigned To Do
        DM-37386

        Investigate adding CenterAll flag during detect and measure

        		 8 John Parejko To Do
        DM-38528

        Investigate properties of diaSources in crowded fields

        		 6 Erin Howard To Do
        DM-38502

        Assist with tests of generalizing real-bogus classifier

        		 4 Harshit Rai To Do
        DM-38500

        Define metrics to summarize spuriousness scores for a visit

        		 6 Harshit Rai To Do
        DM-38451

        Generate new AP DC2 templates

        		 2 Erin Howard To Do
        DM-37699

        Process the full HSC Cosmos dataset with preconvolution

        		 6 Erin Howard To Do
        DM-37968

        Use source flags instead of mask plane in image differencing

        		 2 Bruno Sanchez To Do
        DM-22273

        Create mini-refcat for ap_verify HiTS datasets from Gaia DR2

        		 2 Unassigned To Do
        DM-34112

        Investigate refcat shards used in ci_hits2015

        		 2 John Parejko To Do
        DM-34102

        Further investigation of low fakes completeness

        		 8 Unassigned To Do
        DM-21217

        Add option to select image to convolve based on seeing

        		 4 Ian Sullivan To Do
        DM-21017

        Consider changing the automated algorithmic path in ip_diffim to disable automated deconvolution

        		 Unassigned To Do
        DM-32769

        Implement streak detection and masking in AP

        		 6 Meredith Rawls To Do
        DM-33912

        Fix poorly constrained background model in crowded fields

        		 6 Erin Howard To Do
        DM-33798

        Improve bitmask propagation in ip_diffim

        		 10 Unassigned To Do
        DM-28628

        Actually adopt new calibs in ap_verify HiTS datasets

        		 4 Erin Howard To Do
        DM-22825

        DCR coadds missing some artifact rejection

        		 4 Ian Sullivan To Do
        DM-23072

        Confirm appropriate pixel scale for DECam coadds

        		 6 Meredith Rawls To Do
        DM-23854

        Add spatially varying PSF solution to fixed decorrelation afterburner correction

        		 5 Bruno Sanchez To Do
        DM-22762

        Refactor AL subtraction preparatory code steps in ImageDifferenceTask into separate methods to improve code structure

        		 Unassigned To Do
        DM-22432

        Workaround to keep ScaleZeroPointTask while having calibrated templates

        		 Unassigned To Do
        DM-25051

        Implement proper variance plane calculation in the decorrelation afterburner

        		 4 Bruno Sanchez To Do
        DM-24863

        Make new HSC coadd templates with smaller skymaps

        		 4 Meredith Rawls To Do
        DM-24431

        Incorporate distortion model into obs_decam

        		 4 John Parejko To Do
        DM-35694

        Write unit tests for doApplyFinalizedPsf option

        		 2 Unassigned To Do
        DM-35669

        Inject/ingest DC2 truth data into the repo and stream them down to rbClassify

        		 Nima Sedaghat To Do
        DM-18688

        Develop a metric for spatial non-uniformity of DIASources

        		 10 Meredith Rawls To Do
        DM-20458

        Enlarge regions masked via defects on DECam ccds

        		 Unassigned To Do
        DM-17923

        Inconsistent source detection workflow with difference imaging

        		 Unassigned To Do
        DM-15929

        Define flags for DIAObjects

        		 4 Eric Bellm To Do
        DM-17018

        deprecate and disable ScaleZeroPointTask

        		 4 John Parejko To Do
        DM-8607

        Confusing warning running dipoleMeasTask.py example in ip_diffim

        		 2 Gabor Kovacs [X] (Inactive) To Do
        DM-7684

        Clean up ip_diffim tests

        		 6 Gabor Kovacs [X] (Inactive) To Do
        DM-15351

        Add "real commands" examples to ap_pipe documentation

        		 2 Unassigned To Do
        DM-15340

        Expand results of DMTN-037 using Decam data

        		 6 Ian Sullivan To Do
        DM-38454

        Add a butler-hybrid mode to the ModelPackage storage modes

        		 Nima Sedaghat In Progress
        DM-37697

        Process the full HSC Cosmos dataset with auto convolution mode

        		 4 Erin Howard In Progress
        DM-21306

        Evaluate DCR model performance on sources of extreme color

        		 8 Ian Sullivan In Progress
        DM-34910

        Test DCR correction with new image differencing

        		 4 Ian Sullivan In Progress
        DM-23853

        Create test cases for the decorrelation afterburner and image differencing

        		 8 Gabor Kovacs [X] (Inactive) In Progress
        DM-36760

        Reprocess the Saha galactic bulge DECam dataset

        		 20 Erin Howard In Progress
        DM-36055

        Conduct tests of generalization for the traditional real-bogus classifier

        		 4 Nima Sedaghat In Progress
        DM-17526

        Investigate known quasars in HiTS

        		 4 Ian Sullivan In Progress
        DM-37696

        Process the full DECam HiTS dataset with auto convolution mode

        		 4 Erin Howard In Review
        DM-36402

        Draft technote on AP period-finding

        		 20 Anastasios Tzanidakis In Review
         
        		DM-38651

        Reprocess DC2 data

        		 6 Meredith Rawls Done
         
        		DM-38897

        Participate in DiaSource labelling sprint for ap_verify in May

        		 6 Nima Sedaghat Done
         
        		DM-38896

        Participate in DiaSource labelling sprint for ap_verify in May

        		 6 Harshit Rai Done
         
        		DM-38852

        Run a simple pipeline on USDF machines, with DC2 data

        		 2 Harshit Rai Done
         
        		DM-38846

        Remove deprecated image differencing tasks

        		 4 Ian Sullivan Done
         
        		DM-39286

        Fix preconvolution bug

        		 1 Ian Sullivan Done
         
        		DM-39278

        Set maxDistToPeak to 5

        		 1 Brianna Smart Done
         
        		DM-38590

        Rerun HSC COSMOS without fake source injection

        		 2 Erin Howard Done
         
        		DM-38285

        Run AP_verify with 0.5, 5, 10 as maxDistToPeak

        		 6 Brianna Smart Done
         
        		DM-38243

        Generate new AP DECam templates

        		 4 Erin Howard Done
         
        		DM-38242

        Generate new AP HSC templates

        		 4 Erin Howard Done
         
        		DM-38186

        Process the HSC diffim dataset with convolution mode auto

        		 2 Bruno Sanchez Done
         
        		DM-37147

        Propagate spuriousness scores to APDB

        		 4 Eric Bellm Done
         
        		DM-39170

        Add spuriousness min/max arguments to plotImageSubtractionCutouts

        		 2 Harshit Rai Done
         
        		DM-39143

        Implement script for uploading free metrics to Sasquatch

        		 6 Krzysztof Findeisen Done
         
        		DM-37478

        Design and implement "model-packages" and update existing code accordingly

        		 30 Nima Sedaghat Done
         
        		DM-38527

        Investigate HSC calibration failure

        		 2 Erin Howard Done
         
        		DM-38504

        Reorganize TransiNet-related files and tasks to distinguish between real-bogus and end2end

        		 1 Nima Sedaghat Done
         
        		DM-37698

        Process the full DECam HiTS dataset with preconvolution

        		 6 Erin Howard Done
         
        		DM-37695

        Process the full HSC Cosmos dataset

        		 4 Erin Howard Done
         
        		DM-37694

        Process the full DECam HiTS dataset

        		 4 Erin Howard Done
         
        		DM-37959

        Process the HSC diffim sprint dataset using preconvolution

        		 4 Bruno Sanchez Done
         
        		DM-32197

        Use psFlux chi2 values to investigate image subtraction quality

        		 4 Brianna Smart Done
         
        		DM-32888

        Investigate negative totFlux values

        		 6 Erin Howard Done
         
        		DM-28957

        Provide estimate of range of diaSources expected for APDB test

        		 2 Eric Bellm Done
         
        		DM-36056

        Investigate domain adaptation algorithm for real-bogus classifier

        		 6 Nima Sedaghat Done
        DM-38189

        Analyze the results of diffim with preconvolution mode

        		 2 Bruno Sanchez Reviewed
        DM-38188

        Analyze the results of diffim with auto convolution mode

        		 2 Bruno Sanchez Reviewed
         
        		DM-22038

        Make CI-scale HSC dataset for diffim testing

        		 6 Meredith Rawls Won't Fix
         
        		DM-32963

        Investigate increase in HiTS Characterize runtime due to config changes in obs_decam

        		 4 Meredith Rawls Won't Fix
         
        		DM-32962

        Investigate increase in HiTS DIASource counts due to config changes in obs_decam

        		 6 Meredith Rawls Won't Fix
         
        		DM-32770

        Process the DECAT DDF dataset

        		 6 Unassigned Won't Fix
         
        		DM-33506

        Can't inspect ap_verify CI files in Jenkins scratch space

        		 2 Unassigned Won't Fix
         
        		DM-25989

        sqlite3 encounters database locked when APDB is large

        		 8 Unassigned Won't Fix
         
        		DM-24726

        obs_decam does not test inter-chip crosstalk due to non-CT detectors in testdata_decam

        		 Unassigned Won't Fix
         
        		DM-38851

        Set default for maxDistToPeak in AP

        		 4 Brianna Smart Invalid
         
        		DM-38910

        Process HSC COSMOS using DRP pipeline

        		 4 Erin Howard Invalid
         
        		DM-30462

        Properly scale the flux normalization of the ZOGY score image

        		 4 Gabor Kovacs [X] (Inactive) Invalid
         
        		DM-30097

        Investigate spurious noise sources in HSC ap_pipe run

        		 2 Meredith Rawls Invalid
         
        		DM-32800

        Test image differencing quality under different seeing conditions

        		 6 Ian Sullivan Invalid
         
        		DM-33861

        Process two tracts of DC2 with refactored Image Differencing

        		 6 Meredith Rawls Invalid
         
        		DM-30991

        Investigate odd per-visit DIA Source plots in DC2

        		 2 Meredith Rawls Invalid
         
        		DM-33691

        Following RA/Dec trimming, HSC COSMOS AP run still has low completeness

        		 Meredith Rawls Invalid
         
        		DM-23857

        Add support for spatially varying correction to the decorrelation afterburner

        		 10 Gabor Kovacs [X] (Inactive) Invalid
         
        		DM-24338

        Audit status of calibrations in obs_decam

        		 4 Meredith Rawls Invalid
         
        		DM-25108

        Execute AP pipeline on New Horizons test dataset

        		 10 Ian Sullivan Invalid
         
        		DM-25107

        Make templates from New Horizons AP test dataset

        		 6 Ian Sullivan Invalid
         
        		DM-25106

        Single frame processing of New Horizons HSC data for AP testing

        		 5 Ian Sullivan Invalid

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            sullivan Ian Sullivan
            Reporter:
            sullivan Ian Sullivan
            Watchers:
            Ian Sullivan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Jenkins

                No builds found.