Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-27857

Update ap_verify dataset conversion scripts

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      AP S21-1 (December)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Testing add_gen3_repo.py on DM-26326 revealed that the scripts have not been updated in response to changes in ConvertRepoTask, including at least the removal of ConvertRepoConfig.doWriteCuratedCalibrations. Update the scripts so that they work again, but do not update the repositories until the default collections stabilize.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Running the scripts creates datasets called skyMap, not deepCoadd_skyMap (per DM-27147). Since this is a breaking change to the Gen 3 repos, the scripts are not usable unless the repositories and ap_verify pipeline config are updated to match.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Running the scripts creates datasets called skyMap , not deepCoadd_skyMap (per DM-27147 ). Since this is a breaking change to the Gen 3 repos, the scripts are not usable unless the repositories and ap_verify pipeline config are updated to match.
            Hide
            jbosch Jim Bosch added a comment - - edited

            Looks good, didn't even make any comments on PRs.

            As an aside, it does look like we really need an intermediate-level interface for Gen2->Gen3 conversion; the command-line script and the pure-Python function that maps to it directly just aren't flexible enough, while constructing and using a ConvertRepoTask involves a bit too much boilerplate.  Unfortunately, I think that will require real work in terms of looking at what different invocations actually have in common vs. where they're more likely to differ.  I've just created DM-27968 for that, in case you have any thoughts on how best to approach this problem.

            Show
            jbosch Jim Bosch added a comment - - edited Looks good, didn't even make any comments on PRs. As an aside, it does look like we really need an intermediate-level interface for Gen2->Gen3 conversion; the command-line script and the pure-Python function that maps to it directly just aren't flexible enough, while constructing and using a ConvertRepoTask involves a bit too much boilerplate.  Unfortunately, I think that will require real work in terms of looking at what different invocations actually have in common vs. where they're more likely to differ.  I've just created DM-27968 for that, in case you have any thoughts on how best to approach this problem.

              People

              Assignee:
              krzys Krzysztof Findeisen
              Reporter:
              krzys Krzysztof Findeisen
              Reviewers:
              Jim Bosch
              Watchers:
              Jim Bosch, Krzysztof Findeisen
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.