Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-35060

Only do minimal necessary processing in characterizeImage by default

    XMLWordPrintable

    Details

      Description

      This ticket is part of the implementation of RFC-857, reducing the amount of extra processing that is happening in a default configuration of characterizeImage.

      1. Restrict plugins used in characterize to only a minimal list that is either required or is our recommended algorithm (e.g. shape HSM over SdssShape).
      2. Default measurePsf to psfex instead of piff, and ensure piff is used where it is needed (possibly just FinalizeCharacterizeTask, which already defaults to piff).
      3. Default to psfIterations=1.
      4. Cleanup ap_pipe's characterizeImage plugin list/configs to reflect that the default plugins are now appropriate for AP (this should remove the additions that were made on DM-34826).

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment - - edited

            Definitely not (3).

            I think for DRP purposes we should switch any analysis of PSF properties to the finalize measurements, but I can imagine ways in which that's nontrivial and best punted. So let's do (2), especially since I think my past self may be wrong in thinking that AP won't want plots/metrics based on that slot, and it should be negligible in terms of computational performance.

            Show
            jbosch Jim Bosch added a comment - - edited Definitely not (3). I think for DRP purposes we should switch any analysis of PSF properties to the finalize measurements, but I can imagine ways in which that's nontrivial and best punted. So let's do (2), especially since I think my past self may be wrong in thinking that AP won't want plots/metrics based on that slot, and it should be negligible in terms of computational performance.
            Hide
            Parejkoj John Parejko added a comment -

            Jim Bosch: this should be a short review, but I'm not sure I've considered all the DRP implications.

            pex_config is failing the GitHub action checks, but I can't see why, and drp_pipe is failing on lines I didn't change; ideas on what's going on with those?

            PRs:

            Show
            Parejkoj John Parejko added a comment - Jim Bosch : this should be a short review, but I'm not sure I've considered all the DRP implications. pex_config is failing the GitHub action checks, but I can't see why, and drp_pipe is failing on lines I didn't change; ideas on what's going on with those? PRs: https://github.com/lsst/pipe_tasks/pull/701 https://github.com/lsst/drp_pipe/pull/32 https://github.com/lsst/pex_config/pull/84 https://github.com/lsst/ap_pipe/pull/118
            Hide
            jbosch Jim Bosch added a comment -

            Looks good!

            I've commented about both GitHub Actions issues on the PRs.

            Were the DM-35538 problems the only exceptions you saw in the new logging in pipe_tasks when running ci_hsc or ci_imsim? If so, great; if there were other exceptions that looked like they should be caught and silently squashed, we should be a little more nuanced on the change here before merging it.

            Show
            jbosch Jim Bosch added a comment - Looks good! I've commented about both GitHub Actions issues on the PRs. Were the DM-35538 problems the only exceptions you saw in the new logging in pipe_tasks when running ci_hsc or ci_imsim? If so, great; if there were other exceptions that looked like they should be caught and silently squashed, we should be a little more nuanced on the change here before merging it.
            Show
            Parejkoj John Parejko added a comment - Post-rebase Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/36965/pipeline
            Hide
            Parejkoj John Parejko added a comment -

            Re: DM-35538, there's a bunch of errors related to what Yusra AlSayyad told me were a known problem in `transformForcedSourceOnDiaObjectTable` (apparently the DIA sources or objects have a bunch of known failures right now and someone is planning to fix it?). I don't know if there's a ticket for that, and I don't recall any of the details. That's the only other thing I saw though (although it did send me down a bit of rabbit hole while debugging my own problem); it's going to be hard to filter that out, since I think it is an actual error (it's a KeyError on most of the fields).

            Show
            Parejkoj John Parejko added a comment - Re: DM-35538 , there's a bunch of errors related to what Yusra AlSayyad told me were a known problem in `transformForcedSourceOnDiaObjectTable` (apparently the DIA sources or objects have a bunch of known failures right now and someone is planning to fix it?). I don't know if there's a ticket for that, and I don't recall any of the details. That's the only other thing I saw though (although it did send me down a bit of rabbit hole while debugging my own problem); it's going to be hard to filter that out, since I think it is an actual error (it's a KeyError on most of the fields).

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              Parejkoj John Parejko
              Reviewers:
              Jim Bosch
              Watchers:
              Eli Rykoff, Eric Bellm, Jim Bosch, John Parejko, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.