Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16205

Document ap_pipe using new Task doc framework

    Details

      Description

      Now that we have a Task documentation system, we should modify the existing documentation for ApPipeTask to use it. In addition to helping standardize ap_pipe's documentation, redocumenting ApPipeTask will serve as a useful test case for the new system.

        Attachments

          Activity

          Hide
          krzys Krzysztof Findeisen added a comment -

          [I'm not sure what epic to put this in; it's not emergent, but it seems out of scope for DM-14431. John Swinbank?]

          Show
          krzys Krzysztof Findeisen added a comment - [I'm not sure what epic to put this in; it's not emergent, but it seems out of scope for DM-14431 . John Swinbank ?]
          Hide
          swinbank John Swinbank added a comment -

          I think this does count as “emergent” for our purposes, as in it's worth that we didn't include in the plan for the F18 cycle.

          Show
          swinbank John Swinbank added a comment - I think this does count as “emergent” for our purposes, as in it's worth that we didn't include in the plan for the F18 cycle.
          Hide
          mrawls Meredith Rawls added a comment -

          This is somewhat duplicated by DM-18017.

          Another thing we should clarify in the ap_pipe docs is how to make sure the ref_cat dataset names are self-consistent. ap_verify handles this behind the scenes, but an ap_pipe user who just grabs the ref_cats directory from an ap_verify-style dataset needs to rename the pan_starrs subdirectory to ps1_pv3_thingWithNumbers to avoid a "No locations for get: ref_cat_config" error. I'm sure Gen3 will make this even more fun.

          Show
          mrawls Meredith Rawls added a comment - This is somewhat duplicated by DM-18017 . Another thing we should clarify in the ap_pipe docs is how to make sure the ref_cat dataset names are self-consistent. ap_verify handles this behind the scenes, but an ap_pipe user who just grabs the ref_cats directory from an ap_verify-style dataset needs to rename the pan_starrs subdirectory to ps1_pv3_thingWithNumbers to avoid a "No locations for get: ref_cat_config" error. I'm sure Gen3 will make this even more fun.
          Hide
          krzys Krzysztof Findeisen added a comment - - edited

          I'm not sure I like the thought of that being in the ap_pipe docs, for two reasons:

          • I'm worried it will encourage the use of ap_verify datasets for more "human-readable" work, which is not something they can be easily adapted to
          • configuring refcats is not an issue specific to ap_pipe; we only care because of ProcessCcdTask. Can a page on refcat handling be put into an appropriate package (maybe meas_algorithms, since that's where LoadIndexedReferenceObjectsTask is from) and linked from the ap_pipe docs?
          Show
          krzys Krzysztof Findeisen added a comment - - edited I'm not sure I like the thought of that being in the ap_pipe docs, for two reasons: I'm worried it will encourage the use of ap_verify datasets for more "human-readable" work, which is not something they can be easily adapted to configuring refcats is not an issue specific to ap_pipe ; we only care because of ProcessCcdTask . Can a page on refcat handling be put into an appropriate package (maybe meas_algorithms , since that's where LoadIndexedReferenceObjectsTask is from) and linked from the ap_pipe docs?
          Hide
          danielsf Scott Daniel added a comment -

          I'm not sure if this is related, but I just tried running ap_pipe.py by cloning ap_verify_ci_hits2015 and running ingest_datasets.py on it (from ap_verify).  ingest_datasets.py dumped everything (including the ref_cats and the registry) into my_repository/ingested/ rather than into my_repository, which made it very confusing to run ap_pipe.py on the output.  I don't know if this behavior is by design or is some kind of bug (obviously, it doesn't disrupt ap_verify).  Any new documentation for ap_pipe.py should probably make users aware of this.  I ultimately got ap_pipe.py to run by creating symlinks from my_repo/ingested/ to my_repo/ for ref_cats/, the registry, and the directories actually containing the raw images, per John Parejko's advice.

          Show
          danielsf Scott Daniel added a comment - I'm not sure if this is related, but I just tried running ap_pipe.py by cloning ap_verify_ci_hits2015 and running ingest_datasets.py on it (from ap_verify ).  ingest_datasets.py dumped everything (including the ref_cats and the registry) into my_repository/ingested/ rather than into my_repository , which made it very confusing to run ap_pipe.py on the output.  I don't know if this behavior is by design or is some kind of bug (obviously, it doesn't disrupt ap_verify ).  Any new documentation for ap_pipe.py should probably make users aware of this.  I ultimately got ap_pipe.py to run by creating symlinks from my_repo/ingested/ to my_repo/ for ref_cats/ , the registry, and the directories actually containing the raw images, per John Parejko's advice.
          Hide
          krzys Krzysztof Findeisen added a comment - - edited

          Oops, sorry for overlooking that in our previous discussion. The output directory used by ingest_datasets.py is not a repository, but a regular directory that contains multiple repositories (data, calib, and, when used with ap_verify, output).

          I think putting the refcats in the data repository is correct, but it would be good to check with a refcat expert.

          Show
          krzys Krzysztof Findeisen added a comment - - edited Oops, sorry for overlooking that in our previous discussion. The output directory used by ingest_datasets.py is not a repository, but a regular directory that contains multiple repositories (data, calib, and, when used with ap_verify , output). I think putting the refcats in the data repository is correct, but it would be good to check with a refcat expert.

            People

            • Assignee:
              Unassigned
              Reporter:
              krzys Krzysztof Findeisen
              Watchers:
              John Swinbank, Krzysztof Findeisen, Meredith Rawls, Scott Daniel
            • Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

              • Created:
                Updated: