Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-12315

Generalize ap_pipe to non-HiTS data

    Details

    • Story Points:
      8
    • Epic Link:
    • Sprint:
      AP F18-1, AP F18-2
    • Team:
      Alert Production

      Description

      Currently, ap_pipe has some hardcoded assumptions that it is working on DECam data, and the ingestion step is specific to HiTS. Both of these assumptions should be lifted (likely delegating much of the work to a standards-compliant obs_* package) to allow ap_pipe to be run on arbitrary datasets.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment -

            Hi Meredith Rawls, can you take a look at this reworking of the ApPipeTask configs? Thanks!

            If you have any workflows on data other than ap_verify_hits2015 or the HSC datasets I mentioned in my last comment, I'd appreciate if you could try out the new system and tell me if the burden on the command-line user is too much.

            Show
            krzys Krzysztof Findeisen added a comment - Hi Meredith Rawls , can you take a look at this reworking of the ApPipeTask configs? Thanks! If you have any workflows on data other than ap_verify_hits2015 or the HSC datasets I mentioned in my last comment, I'd appreciate if you could try out the new system and tell me if the burden on the command-line user is too much.
            Hide
            mrawls Meredith Rawls added a comment -

            Thank you for the clear use cases! I'm impressed you've gotten ap_pipe to work with HSC data.

            It looks like you ran ap_pipe on lsst-dev, but I'm not sure what working directory you were in, and I'd like to take a look at the actual processed images if possible. Can you please point me to somewhere I can see them?

            I may be missing something, but it looks like you moved some critical config info from ap_pipe into the dataset repo (i.e., ap_verify_hits2015) because it was DECam-specific. Did you generate a similar but HSC-style config file for refcats, filterMaps, etc. for your three use case test runs? I am concerned about requiring users to have a bunch of custom configs set in order to run ap_pipe at all, and I wonder if it would make sense for the obs_packages to have a set of default refcat/filterMap configs that advanced users may choose to override.

            Show
            mrawls Meredith Rawls added a comment - Thank you for the clear use cases! I'm impressed you've gotten ap_pipe to work with HSC data. It looks like you ran ap_pipe on lsst-dev, but I'm not sure what working directory you were in, and I'd like to take a look at the actual processed images if possible. Can you please point me to somewhere I can see them? I may be missing something, but it looks like you moved some critical config info from ap_pipe into the dataset repo (i.e., ap_verify_hits2015) because it was DECam-specific. Did you generate a similar but HSC-style config file for refcats, filterMaps, etc. for your three use case test runs? I am concerned about requiring users to have a bunch of custom configs set in order to run ap_pipe at all, and I wonder if it would make sense for the obs_packages to have a set of default refcat/filterMap configs that advanced users may choose to override.
            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Everything was run from lsst-dev01:/scratch/krzys001/. I'll have to look at the PR comments about "critical config info", but the default refcats for ProcessCcdTask (DECam, HSC) work fine on lsst-dev.

            I had to create a custom config for the third run (the one that used validation_data_hsc), but that's because Astrometry.net refcats are very incompatible with the LSST-style refcats we've been assuming.

            Show
            krzys Krzysztof Findeisen added a comment - - edited Everything was run from lsst-dev01:/scratch/krzys001/ . I'll have to look at the PR comments about "critical config info", but the default refcats for ProcessCcdTask ( DECam , HSC ) work fine on lsst-dev . I had to create a custom config for the third run (the one that used validation_data_hsc ), but that's because Astrometry.net refcats are very incompatible with the LSST-style refcats we've been assuming.
            Hide
            mrawls Meredith Rawls added a comment -

            OK, that makes sense. I am unhappy with the answer being "run ap_pipe on lsst-dev" but you're correct that the processCcd configs cover my concern on lsst-dev. I suppose it's impossible to know a priori what refcats are available in other places so I don't have a better suggestion for now.

            I'd like to see the commands you used to run ap_pipe here documented in some way, so that when people ask "how can I run ap_pipe on my favorite HSC data?" they don't have to wade over to this ticket and/or your lsst-dev scratch directory. Now that we support both DECam and HSC, I think it would be helpful (e.g., to folks who might want to use ap_pipe with yet another camera) to have examples of each in the docs.

            It's fine to close and merge this ticket, and please create a new one to add more examples to the docs.

            Show
            mrawls Meredith Rawls added a comment - OK, that makes sense. I am unhappy with the answer being "run ap_pipe on lsst-dev" but you're correct that the processCcd configs cover my concern on lsst-dev. I suppose it's impossible to know a priori what refcats are available in other places so I don't have a better suggestion for now. I'd like to see the commands you used to run ap_pipe here documented in some way, so that when people ask "how can I run ap_pipe on my favorite HSC data?" they don't have to wade over to this ticket and/or your lsst-dev scratch directory. Now that we support both DECam and HSC, I think it would be helpful (e.g., to folks who might want to use ap_pipe with yet another camera) to have examples of each in the docs. It's fine to close and merge this ticket, and please create a new one to add more examples to the docs.
            Hide
            krzys Krzysztof Findeisen added a comment -

            I think it's less "you must run on lsst-dev" and more "if you use your own refcats, you are responsible for supporting them". Hopefully not a big issue in practice, since anybody who's run ProcessCcdTask has had to have solved the problem already.

            I've created DM-15351 to deal with the examples question.

            Thanks for the review!

            Show
            krzys Krzysztof Findeisen added a comment - I think it's less "you must run on lsst-dev " and more "if you use your own refcats, you are responsible for supporting them". Hopefully not a big issue in practice, since anybody who's run ProcessCcdTask has had to have solved the problem already. I've created DM-15351 to deal with the examples question. Thanks for the review!

              People

              • Assignee:
                krzys Krzysztof Findeisen
                Reporter:
                krzys Krzysztof Findeisen
                Reviewers:
                Meredith Rawls
                Watchers:
                Eric Bellm, Jim Bosch, John Swinbank, Krzysztof Findeisen, Meredith Rawls
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: