Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36521

AP Algorithm development in F23B

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      f23b-ap-algorithms&analysis
    • Story Points:
      141
    • WBS:
      02C.03.04
    • Team:
      Alert Production
    • Cycle:
      Fall 2023

      Description

      Regular analysis of processing of precursor data through AP pipelines, including identifying issues, tracking them down, and resolving them. Includes testing algorithmic changes on large datsets as well as developing and testing timeseries features.
      Develop the tools required for machine learning classification (real-bogus) of image difference cutouts

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
        DM-40457

        Write new technote about crowded field processing

        		 8 Erin Howard To Do
        DM-40431

        Compare astropy multiband Lomb-Scargle implementation to gatspy

        		 8 Anastasios Tzanidakis To Do
        DM-40398

        Assist with analysis of new CalibrateImageTask

        		 4 Unassigned To Do
        DM-40833

        Investigate the failures in SFM with the Saha Bulge dataset

        		 4 Erin Howard To Do
        DM-40831

        Add astropy Lomb-Scargle to AP timeseries feature computation

        		 20 Anastasios Tzanidakis To Do
        DM-40945

        Modify the zooniverse-tools tests so they use a real csv fetched from Zooniverse

        		 Harshit Rai To Do
        DM-40507

        Switch ip_diffim dipole measurements to use PsfDipoleFlux

        		 4 John Parejko To Do
        DM-40487

        Add uncertainty to the output of DipoleFitTask

        		 Unassigned To Do
        DM-40996

        Investigate subtractImages failures from Saha Bulge

        		 4 Unassigned To Do
        DM-40995

        Investigate characterizeImage failures from Saha Bulge

        		 4 Unassigned To Do
        DM-40994

        Investigate calibration failures from Saha Bulge

        		 4 Unassigned To Do
        DM-41391

        Refine selection of kernel candidates for image differencing

        		 4 Ian Sullivan To Do
        DM-38248

        Write concept docs for meas_transiNet

        		 4 Nima Sedaghat To Do
        DM-37968

        Use source flags instead of mask plane in image differencing

        		 2 Bruno Sanchez To Do
        DM-41640

        Propagate calib_psf_used information to stars catalog in CalibrateImageTask

        		 2 John Parejko To Do
        DM-41497

        Enable source measurement in EDGE regions of difference images

        		 6 Ian Sullivan To Do
        DM-41496

        Enable source detection in EDGE regions of difference images

        		 12 Ian Sullivan To Do
        DM-33912

        Fix poorly constrained background model in crowded fields

        		 6 Erin Howard To Do
        DM-33798

        Improve bitmask propagation in ip_diffim

        		 10 Unassigned To Do
        DM-34112

        Investigate refcat shards used in ci_hits2015

        		 2 John Parejko To Do
        DM-34102

        Further investigation of low fakes completeness

        		 8 Unassigned To Do
        DM-24431

        Incorporate distortion model into obs_decam

        		 4 John Parejko To Do
        DM-23854

        Add spatially varying PSF solution to fixed decorrelation afterburner correction

        		 5 Bruno Sanchez To Do
        DM-25051

        Implement proper variance plane calculation in the decorrelation afterburner

        		 4 Bruno Sanchez To Do
        DM-24863

        Make new HSC coadd templates with smaller skymaps

        		 4 Meredith Rawls To Do
        DM-38968

        Lower number of spurious warning during normal AuxTel processing

        		 Unassigned To Do
        DM-35694

        Write unit tests for doApplyFinalizedPsf option

        		 2 Unassigned To Do
        DM-35669

        Inject/ingest DC2 truth data into the repo and stream them down to rbClassify

        		 Nima Sedaghat To Do
        DM-17018

        deprecate and disable ScaleZeroPointTask

        		 4 John Parejko To Do
        DM-22273

        Create mini-refcat for ap_verify HiTS datasets from Gaia DR2

        		 2 Unassigned To Do
        DM-21217

        Add option to select image to convolve based on seeing

        		 4 Ian Sullivan To Do
        DM-22762

        Refactor AL subtraction preparatory code steps in ImageDifferenceTask into separate methods to improve code structure

        		 Unassigned To Do
        DM-17923

        Inconsistent source detection workflow with difference imaging

        		 Unassigned To Do
        DM-21017

        Consider changing the automated algorithmic path in ip_diffim to disable automated deconvolution

        		 Unassigned To Do
        DM-22825

        DCR coadds missing some artifact rejection

        		 4 Ian Sullivan To Do
        DM-23072

        Confirm appropriate pixel scale for DECam coadds

        		 6 Meredith Rawls To Do
        DM-22432

        Workaround to keep ScaleZeroPointTask while having calibrated templates

        		 Unassigned To Do
        DM-20458

        Enlarge regions masked via defects on DECam ccds

        		 Unassigned To Do
        DM-18688

        Develop a metric for spatial non-uniformity of DIASources

        		 10 Meredith Rawls To Do
        DM-15351

        Add "real commands" examples to ap_pipe documentation

        		 2 Unassigned To Do
        DM-15340

        Expand results of DMTN-037 using Decam data

        		 6 Ian Sullivan To Do
        DM-15929

        Define flags for DIAObjects

        		 4 Eric Bellm To Do
        DM-7684

        Clean up ip_diffim tests

        		 6 Gabor Kovacs [X] (Inactive) To Do
        DM-8607

        Confusing warning running dipoleMeasTask.py example in ip_diffim

        		 2 Gabor Kovacs [X] (Inactive) To Do
        DM-40405

        Assess performance of real-bogus classifier on DC2

        		 8 Nima Sedaghat In Progress
        DM-40886

        Ingest DECam images with known satellite streaks

        		 4 Meredith Rawls In Progress
        DM-40869

        Confirm definition of truth values for Zooniverse scoring

        		 2 Nima Sedaghat In Progress
        DM-40859

        Reconcile DC2 truth values

        		 6 Nima Sedaghat In Progress
        DM-40798

        Exclude sky sources from RB classification

        		 6 Harshit Rai In Progress
        DM-41541

        Prepare a presentation on dipoles from real-bogus perspective

        		 4 Harshit Rai In Progress
        DM-41519

        Process the full DECam HiTS dataset with fakes

        		 2 Erin Howard In Progress
        DM-37699

        Process the full HSC Cosmos dataset with preconvolution

        		 6 Erin Howard In Progress
        DM-28628

        Actually adopt new calibs in ap_verify HiTS datasets

        		 12 Erin Howard In Progress
        DM-23853

        Create test cases for the decorrelation afterburner and image differencing

        		 8 Gabor Kovacs [X] (Inactive) In Progress
        DM-39835

        Run the Zooniverse HSC classification project and collate results

        		 4 Harshit Rai In Progress
        DM-38528

        Investigate properties of diaSources in crowded fields

        		 6 Erin Howard In Progress
        DM-39416

        Run test of active learning

        		 8 Nima Sedaghat In Progress
        DM-34910

        Test DCR correction with new image differencing

        		 4 Ian Sullivan In Progress
        DM-17526

        Investigate known quasars in HiTS

        		 4 Ian Sullivan In Progress
        DM-21306

        Evaluate DCR model performance on sources of extreme color

        		 8 Ian Sullivan In Progress
        DM-41518

        Investigate duplicate diaSourceIds in DM-37699

        		 16 Erin Howard In Review
        DM-36402

        Draft technote on AP period-finding

        		 20 Anastasios Tzanidakis In Review
        DM-38500

        Define metrics to summarize spuriousness scores for a visit

        		 8 Harshit Rai In Review
        DM-38454

        Add a butler mode to the ModelPackage storage modes

        		 8 Nima Sedaghat In Review
        DM-38862

        Investigate timeseries features computed in DP0.2

        		 20 Anastasios Tzanidakis In Review
        DM-41285

        Add sky sources to CalibrateImageTask

        		 2 John Parejko In Review
        DM-41955

        Remove known bad sources from detectAndMeasure output

        		 2 Ian Sullivan In Review
         
        		DM-40409

        Assist with making a DC2 repository on google cloud

        		 6 Harshit Rai Done
         
        		DM-40387

        Validate new CalibrateImage with ap_verify_ci datasets

        		 8 John Parejko Done
         
        		DM-40832

        Finish reprocessing the DECam Saha Galactic bulge dataset

        		 8 Erin Howard Done
         
        		DM-40907

        Prepare a list of dipoles in DC2 diff images, along with cutout triplets

        		 4 Harshit Rai Done
         
        		DM-40949

        Investigate ap_verify changes from DM-39796

        		 2 Ian Sullivan Done
         
        		DM-41213

        Clarify error message for missing rbClassifier_data

        		 1 Krzysztof Findeisen Done
         
        		DM-41896

        Re-run HSC Cosmos with preconvolution

        		 2 Erin Howard Done
         
        		DM-41072

        Run RB-Classifier on DL 0.1.2

        		 4 Nima Sedaghat Done
         
        		DM-37697

        Process the full HSC Cosmos dataset with auto convolution mode

        		 6 Erin Howard Done
         
        		DM-32769

        Implement streak detection and masking in SFM

        		 12 Meredith Rawls Done
         
        		DM-38451

        Generate new AP DC2 templates

        		 6 Erin Howard Done
         
        		DM-39473

        Write converter to ingest Zooniverse project results

        		 4 Nima Sedaghat Done
         
        		DM-39275

        Compare image differencing results using each mode

        		 4 Ian Sullivan Done
         
        		DM-37386

        Add a CenterAll flag during detect and measure

        		 3 John Parejko Done
         
        		DM-41956

        Exclude off-chip diaObjects from association

        		 2 Ian Sullivan Done
         
        		DM-40942

        Switch AstrometryTask to use ScienceSourceSelector

        		 2 John Parejko Done
         
        		DM-39371

        Create a centralized database for the upcoming stream of real-bogus labels

        		 1 Nima Sedaghat Invalid

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            sullivan Ian Sullivan
            Reporter:
            sullivan Ian Sullivan
            Watchers:
            Ian Sullivan
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Jenkins

                No builds found.