Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36008

AP pipeline infrastructure in F22B

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Epic Name:
      f22b-ap-infrastructure
    • Story Points:
      29
    • WBS:
      02C.03.05
    • Team:
      Alert Production
    • Cycle:
      Fall 2022

      Description

      Infrastructure work to support Alert Production. This includes middleware, verification and testing frameworks, as well as performance tuning.
       

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
        DM-36789

        Clean up apdb schema

        4 Kenneth Herner To Do
        DM-36787

        Shepherd metric naming proposal

        2 John Parejko To Do
        DM-36744

        Add real-bogus CI output to Chronograf

        2 Nima Sedaghat To Do
        DM-36742

        Add real-bogus trained model to DC2 CI repository

        4 Nima Sedaghat To Do
        DM-36505

        Propagate database schema/namespace into AP configurations

        4 Unassigned To Do
        DM-35846

        Investigate approaches for enforcing sdm_schema in the output of ap_association

        2 Unassigned To Do
        DM-35828

        Use MDC or equivalent in prompt_prototype logging

        6 Unassigned To Do
        DM-35962

        Remove examples/ from all packages

        2 John Parejko To Do
        DM-35649

        Create task docs for CreateRandomApFakesTask

        1 Unassigned To Do
        DM-35646

        Rewrite ap_pipe doc intro to match RFC-775 role

        1 Unassigned To Do
        DM-36477

        Remove ap_verify_hits2015 dataset

        1 Krzysztof Findeisen To Do
        DM-36051

        Emulate DMTN-143 image transfer in Prompt Processing Prototype

        6 Unassigned To Do
        DM-35302

        Source detection config metadata propagated to output catalogs

        2 John Parejko To Do
        DM-36970

        Design interface for notebook-level queries to the APDB

        4 John Parejko To Do
        DM-34769

        Remove characterizeImage refObjLoader hooks

        1 John Parejko To Do
        DM-34758

        DiaCalculation plugins report fatal errors

        2 Unassigned To Do
        DM-32138

        Remove VisitInfo.exposureId

        6 Unassigned To Do
        DM-35055

        Assist with the creation of timing metrics for DRP

        2 Krzysztof Findeisen To Do
        DM-32771

        Persist analysis_ap plots with the butler

        4 Meredith Rawls To Do
        DM-31836

        Write gen3 formatter for Zooniverse data

        6 John Parejko To Do
        DM-31866

        Clean up requests usage in EphemerisQueryTask

        6 Krzysztof Findeisen To Do
        DM-34975

        Detailed performance characterization of the AP pipeline

        8 John Parejko To Do
        DM-28524

        Add versioning to Persistable

        8 Unassigned To Do
        DM-27547

        Geometrically check that PsfCandidates are sufficiently far from edge

        4 John Parejko To Do
        DM-24326

        Add a PSF object to the Astropy CCData representation

        8 Unassigned To Do
        DM-25160

        Record use of CCDData alert stamps in DPDD and DMTN-093

        4 Eric Bellm To Do
        DM-21268

        GenericMap can't distinguish int/float sizes in Python

        6 Krzysztof Findeisen To Do
        DM-21382

        Separate GenericMap's visitor API from its implementation

        6 Krzysztof Findeisen To Do
        DM-21216

        GenericMap converts C-strings to bool

        2 Krzysztof Findeisen To Do
        DM-17070

        Clean up lsst.afw.multiband

        2 Unassigned To Do
        DM-16861

        Convert afw.image to numpydoc

        6 Unassigned To Do
        DM-17458

        Complete docstrings in ip_diffim

        2 Unassigned To Do
        DM-16773

        ap_verify_hits2015 hard to manage with lsstsw

        Unassigned To Do
        DM-16078

        convert coadd_utils to numpydoc

        John Swinbank To Do
        DM-34906

        Systematic error in ephemerides in ap_verify datasets

        6 Unassigned To Do
        DM-34925

        Investigate memory use spike at end of DcrAssembleCoaddTask

        2 Ian Sullivan To Do
        DM-14136

        convert ip_isr documentation to numpydoc

        6 Unassigned To Do
        DM-36425

        Discuss purpose of tables produced in postprocess.py

        2 John Parejko To Do
        DM-33039

        Re-examine how to handle dataset management scripts

        6 Krzysztof Findeisen To Do
        DM-36319

        Fix Prompt Prototype crashes for repeat APDB operations

        6 Krzysztof Findeisen To Do
        DM-36162

        Upgrade from SimplePipelineExecutor in prompt prototype

        6 Unassigned To Do
        DM-36511

        Create LATISS templates for Embargo Repo

        Unassigned To Do
        DM-35645

        Brainstorm circuit-breaking failures for AP processing

        1 John Parejko In Progress
        DM-36608

        Create metric for Nsources per DiaObject as a function of magnitude/seeing

        4 Brianna Smart In Progress
        DM-16824

        Convert meas_algorithms to numpydoc format

        John Parejko In Progress
        DM-36057

        Write up GPU requirements for real-bogus model training

        2 Nima Sedaghat In Review
        DM-36743

        Add the real-bogus PipelineTask to ap_verify for the DC2 CI dataset

        4 Nima Sedaghat In Review
         
        DM-36792

        Design discussion about AP failure modes

        3 Krzysztof Findeisen Done
         
        DM-36712

        Migrate activator.py to use Kafka/Knative

        4 Krzysztof Findeisen Done
         
        DM-36509

        Set up test Prompt Processing central repository at USDF

        3 Krzysztof Findeisen Done
         
        DM-36248

        Use name in loadRegion

        0.5 John Parejko Done
         
        DM-36246

        Create Analysis_Tools metrics for median flux metric and ratio of psf to apterure flux plot

        4 Brianna Smart Done
         
        DM-35871

        Add refcat name arg to ReferenceObjectLoader init

        4 John Parejko Done
         
        DM-35941

        Remove use of implicit dimensions in Prompt Prototype

        5 Krzysztof Findeisen Done
         
        DM-35939

        Convert pipe_tasks to numpydoc and task topics

        6 Erin Howard Done
         
        DM-35976

        Investigate using analysis tools package

        6 Brianna Smart Done
         
        DM-35975

        Investigate using analysis tools package

        6 Erin Howard Done
         
        DM-36602

        Prepare pool of "original" raws at USDF

        1 Hsin-Fang Chiang Done
         
        DM-36409

        Update stack-os-matrix docs to reflect current system

        1 Krzysztof Findeisen Done
         
        DM-36385

        Deprecate ap_verify_hits2015 dataset

        1 Krzysztof Findeisen Done
         
        DM-36080

        Separate GCP-specific code in Prompt Processing prototype

        6 Krzysztof Findeisen Done
         
        DM-35326

        Write Community post describing print_metricvalues script

        4 John Parejko Done
         
        DM-36999

        Complete integration test at USDF

        2 Krzysztof Findeisen Done
         
        DM-36673

        Create barPlot.py for analysis_tools

        6 Brianna Smart Done
         
        DM-36238

        Create analysis_tools metrics for numDipoles and numDiaSrcs

        4 Erin Howard Done
         
        DM-36234

        Create AP Number of Associated Solar System Objects metric in analysis_tools

        4 Eric Bellm Done
         
        DM-36198

        Add parquet transform tasks to ap_verify

        2 Krzysztof Findeisen Done
         
        DM-36157

        Add notes about dataset contents ap_verify_ci_cosmos_pdr2

        0.5 John Parejko Done
         
        DM-35060

        Only do minimal necessary processing in characterizeImage by default

        8 John Parejko Done
         
        DM-35054

        Assist with creating DC2 CI dataset

        2 Krzysztof Findeisen Done
         
        DM-35053

        Implement syncing of output products for Prompt Processing

        14 Krzysztof Findeisen Done
         
        DM-31698

        Move ConvertReferenceCatalog classes out of ingestIndex file

        2 John Parejko Done
         
        DM-28329

        Standardize on jyear and remove "EPOCH" header keys

        4 John Parejko Done
         
        DM-25161

        Develop proposal for PSF representation in astropy CCDData

        15 John Parejko Done
         
        DM-30869

        Modernize MetricTask for better Gen 3 workflow

        4 Krzysztof Findeisen Done
         
        DM-13711

        Add type of observation information to visitInfo

        4 John Parejko Done
         
        DM-36791

        Use unique IDs in upload.py output

        3 Krzysztof Findeisen Done
        DM-36437

        Create NumDiaSourcesMetricAll and CountUniqueAction for analysis_tools

        2 Erin Howard Reviewed
        DM-36263

        Create Analysis_Tools metric and plots for unassociatedDiasources and min/max/median seeing

        3 Brianna Smart Reviewed
         
        DM-36510

        Import LATISS calibs to OGA Repo

        1 Krzysztof Findeisen Invalid
         
        DM-35754

        Assist with creating DC2 testdata repository for CI

        4 Nima Sedaghat Invalid
         
        DM-33095

        Streamline updating of ap_verify CI datasets

        4 Krzysztof Findeisen Invalid

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            sullivan Ian Sullivan
            Reporter:
            horvat Nikolina Horvat
            Watchers:
            Nikolina Horvat
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:

                Jenkins Builds

                No builds found.