Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-3380

Port HSC hooks for simulated source injection

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_tasks
    • Labels:
      None

      Description

      Port HSC hooks injecting simulated sources into real images to test processing.

      This includes the code in fakes.py in pipe_tasks and its callers. The pipeline does not include code for actually adding the fake sources; it just provides a callback interface that is implemented by third-party plugins such as https://github.com/clackner2007/fake-sources.

        Attachments

          Issue Links

            Activity

            Hide
            rowen Russell Owen added a comment -

            Jim Bosch your suggestion would work, but I confess I lean towards going the other route:

            • skip the ABC because the API is so simple
            • Have exactly one task, e.g. BaseFakeSourcesTask, that adds the necessary fields when constructed and logs a warning that no sources will be added when it is run.
            • Add a config flag to run the task, and if it is false then don't construct it (and hence don't add the fields)

            The advantages I see are:

            • It is simple, adding just one nearly trivial task
            • It can be called, e.g. for testing purposes

            The main disadvantages I see are:

            • It is more different from the HSC code, though still not very different
            • It requires every caller to add a config parameter that enables the task. But this is such a common pattern that I see it as only a minor disadvantage. And if one omitted that flag the worst that happens is a few unused fields get added to the source catalog.
            Show
            rowen Russell Owen added a comment - Jim Bosch your suggestion would work, but I confess I lean towards going the other route: skip the ABC because the API is so simple Have exactly one task, e.g. BaseFakeSourcesTask , that adds the necessary fields when constructed and logs a warning that no sources will be added when it is run. Add a config flag to run the task, and if it is false then don't construct it (and hence don't add the fields) The advantages I see are: It is simple, adding just one nearly trivial task It can be called, e.g. for testing purposes The main disadvantages I see are: It is more different from the HSC code, though still not very different It requires every caller to add a config parameter that enables the task. But this is such a common pattern that I see it as only a minor disadvantage. And if one omitted that flag the worst that happens is a few unused fields get added to the source catalog.
            Hide
            jbosch Jim Bosch added a comment -

            Ah, that's a good point. I think Nate Lust's implementation issue with the ABC came from the fact that it was failing when constructed, but we shouldn't be constructing it when doFakes=False anyway. Hopefully he can confirm. So if we have a doFakes option, we can indeed use ABC.

            So I think your original proposal works, and I'll just let you two go ahead with that (sorry about all the mixed signals, Nate Lust). I believe with the ABC approach there's no need to have a warning in BaseFakeSourcesTask.run, since we can guarantee it'll never be called.

            Show
            jbosch Jim Bosch added a comment - Ah, that's a good point. I think Nate Lust 's implementation issue with the ABC came from the fact that it was failing when constructed, but we shouldn't be constructing it when doFakes=False anyway. Hopefully he can confirm. So if we have a doFakes option, we can indeed use ABC. So I think your original proposal works, and I'll just let you two go ahead with that (sorry about all the mixed signals, Nate Lust ). I believe with the ABC approach there's no need to have a warning in BaseFakeSourcesTask.run , since we can guarantee it'll never be called.
            Hide
            jbosch Jim Bosch added a comment -

            Two small comments on the code:

            • I think the __metaclass__ line needs to go after the class docstring if you want the docstring to function properly.
            • Commit message should probably be changed to reflect the fact that there's no longer a dummy task, and I think the sense of "users can not" is the opposite of what was intended (maybe "not" -> "now"?).
            Show
            jbosch Jim Bosch added a comment - Two small comments on the code: I think the __metaclass__ line needs to go after the class docstring if you want the docstring to function properly. Commit message should probably be changed to reflect the fact that there's no longer a dummy task, and I think the sense of "users can not" is the opposite of what was intended (maybe "not" -> "now"?).
            Hide
            nlust Nate Lust added a comment -

            I changed FakeSoucesTask to BaseFakeSourcesTask which is an abc. I added a configuration option to not run the fake sources by default. I eliminated the dummy class all together. This task will then not run if 1 the doFakes option is not set and 2. if the doFakes option is set, and the task is not retargeted, the task will fail as it is an abc. A user must set doFakes, and retarget the task to a subclass of BaseFakeSourcesTask

            Show
            nlust Nate Lust added a comment - I changed FakeSoucesTask to BaseFakeSourcesTask which is an abc. I added a configuration option to not run the fake sources by default. I eliminated the dummy class all together. This task will then not run if 1 the doFakes option is not set and 2. if the doFakes option is set, and the task is not retargeted, the task will fail as it is an abc. A user must set doFakes, and retarget the task to a subclass of BaseFakeSourcesTask
            Hide
            swinbank John Swinbank added a comment -

            I note (for my own benefit) that this is already mentioned in the release notes. Many thanks!

            Show
            swinbank John Swinbank added a comment - I note (for my own benefit) that this is already mentioned in the release notes. Many thanks!

              People

              Assignee:
              nlust Nate Lust
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              Russell Owen
              Watchers:
              Jim Bosch, John Swinbank, Nate Lust, Russell Owen
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.