Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-11118

Build stubbed out verify_ap

    Details

    • Templates:
    • Story Points:
      6
    • Sprint:
      Alert Production F17 - 7
    • Team:
      Alert Production

      Description

      We have a start at a design for the verify_ap. It will require changing the API for the prototype ap pipeline, but should contain place holders for anything we know we need but do not have yet.

      This will give us a better idea what the processing flow looks like and also give us a place to add metrics calculations and a place to stick things like association.

      We wrote down the following pseudocode as a starting point:

      main():
          find input data
          "--dataset"
          load_dataset_config --> yaml file with stuff like "HiTS" : "hits_data"
          (for starters, self-contained datasets that we know will work)
          input_repo_dir = eups.getPackageDir(datasetname)"mapper" : "obs_package"
          ?? eups setup obs_package ?? (supposedly this can be done within python?)
          "--output"
          copy stub input repo to output
          "--dataIdString" (note: often just "--Id" as a command line string in other places)
          ap_pipe.ingest_raws(data_location, repo_location)
          ap_pipe.ingest_calibs(data_location, repo_location)
          ap_pipe.process(repo_location, rerun, parallelization_level, dataIdString)
          --> how to get templates? in same repo with input data?
          ap_pipe.diff(repo_location, rerun, parallelization_level, dataIdString)
          ap_pipe.assoc(repo_location, rerun)
          ap_pipe.afterburner?? --> and/or, the afterburner could live in the same package as main, and be called as verify_ap_afterburner
      

        Attachments

          Container Issues

            Issue Links

              Activity

                People

                • Assignee:
                  krzys Krzysztof Findeisen
                  Reporter:
                  krughoff Simon Krughoff
                  Reviewers:
                  David Reiss, Meredith Rawls
                  Watchers:
                  David Reiss, Krzysztof Findeisen, Meredith Rawls, Simon Krughoff
                • Votes:
                  0 Vote for this issue
                  Watchers:
                  4 Start watching this issue

                  Dates

                  • Created:
                    Updated:
                    Resolved:

                    Summary Panel