Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-24436

Use obs and camera overrides in make_apdb.py

    Details

    • Type: Improvement
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_pipe, ap_verify
    • Labels:
      None

      Description

      make_apdb.py takes an ApPipeConfig and uses it to configure the database. At present, it loads config files requested by the user from the command-line, but does not try to load observatory or camera-specific overrides for the data going into the APDB. This caused ap_verify to run afoul of DM-24435, and may also lead to inconsistent behavior if we ever want to configure the APDB in a per-camera manner (column names for special filters, for example?).

      Modify make_apdb.py to search for and load ap_pipe override files. This requires that make_apdb.py know what camera it is working with. I see two options:

      • provide an input Butler repository, and infer the obs package and camera from the mapper. This has the advantage of common syntax with ap_pipe.py.
      • explicitly name the obs-package and camera on the command line, e.g. "obs_subaru/hsc". This has the advantage of being less confusing for the user, who might otherwise expect that the location of the repository is somehow relevant.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            On second thought, an input Butler repository would be a bad idea, since it will make it harder to use make_apdb.py with Gen 3 before DM-22663. Explicit camera string it is.

            Show
            krzys Krzysztof Findeisen added a comment - On second thought, an input Butler repository would be a bad idea, since it will make it harder to use make_apdb.py with Gen 3 before DM-22663 . Explicit camera string it is.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            It turns out that the way ConfigFileAction and ConfigValueAction work is very hard to reconcile with also providing the camera through command-line arguments (CmdLineTask does it by manually hacking the input command-line argument before passing it to the "real" parser).

            There are a couple of ways around it in make_apdb, but all would require a lot of rewriting of Gen 2 code. Given the so far hypothetical benefits, I think it's best to put this capability off until DM-22663, and in the meantime work around DM-24435 by simply not freezing the config.

            Show
            krzys Krzysztof Findeisen added a comment - - edited It turns out that the way ConfigFileAction and ConfigValueAction work is very hard to reconcile with also providing the camera through command-line arguments ( CmdLineTask does it by manually hacking the input command-line argument before passing it to the "real" parser). There are a couple of ways around it in make_apdb , but all would require a lot of rewriting of Gen 2 code. Given the so far hypothetical benefits, I think it's best to put this capability off until DM-22663 , and in the meantime work around DM-24435 by simply not freezing the config.

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Watchers:
                Eric Bellm, Krzysztof Findeisen, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: