Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8255

Error if CmdLineTask is given an empty rerun folder

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: pipe_base
    • Labels:
      None

      Description

      CmdLineTask gets confused when it's given a rerun folder that already exists but is empty, and resulting in this error message:

      Error: input directory specified by --rerun must have the same mapper as INPUT
      

      This happens in use cases similar to what's wanted in RFC-249: a user creates a symlink for a rerun before running the task.

        Attachments

          Issue Links

            Activity

            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Add a test for this scenario, and using the current master it fails with the error as in the descriptions.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Add a test for this scenario, and using the current master it fails with the error as in the descriptions.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            This happens because the "rerun" logic in ArgumentParser takes the existence of the rerun folder as that folder being a butler repo linked to a parent repo (and then fails in the following step of the mapper class check). I'm adding a one-line fix for that.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - This happens because the "rerun" logic in ArgumentParser takes the existence of the rerun folder as that folder being a butler repo linked to a parent repo (and then fails in the following step of the mapper class check). I'm adding a one-line fix for that.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Paul Price do you mind reviewing this one-line change plus tests? This will allow running tasks with a pre-created empty rerun folder (or a manually set symlink to somewhere else).

            Jenkins passed.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Paul Price do you mind reviewing this one-line change plus tests? This will allow running tasks with a pre-created empty rerun folder (or a manually set symlink to somewhere else). Jenkins passed .
            Hide
            price Paul Price added a comment -

            Please think about whether it's important that the rerun directory and _parent link are no longer being created together.

            I would have thought that the only code in Jenkins that really exercises the butler and creates a rerun was ci_hsc, but you didn't run that.

            Show
            price Paul Price added a comment - Please think about whether it's important that the rerun directory and _parent link are no longer being created together. I would have thought that the only code in Jenkins that really exercises the butler and creates a rerun was ci_hsc, but you didn't run that.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Thank you Paul.

            I think this makes --rerun foo behave more similarly to --output. When using --output, the output directory can pre-exist without a _parent link. The input/output roots, derived from either --rerun or --output arguments, are passed to CameraMapper. The output directory is created if is doesn't exist yet. Then the _parent link is created if it doesn't exist yet, or the pre-existing link is checked (errors if they don't match).

            Does the --output logic make this a bit more convincing to you?

            This seems to only became important for the use case in RFC-249, so I'll mark this triggered by RFC-249 and wait for RFC-249.

            Also thank you for reminding me to run ci_hsc.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Thank you Paul. I think this makes --rerun foo behave more similarly to --output . When using --output , the output directory can pre-exist without a _parent link. The input/output roots, derived from either --rerun or --output arguments, are passed to CameraMapper. The output directory is created if is doesn't exist yet. Then the _parent link is created if it doesn't exist yet, or the pre-existing link is checked (errors if they don't match). Does the --output logic make this a bit more convincing to you? This seems to only became important for the use case in RFC-249 , so I'll mark this triggered by RFC-249 and wait for RFC-249 . Also thank you for reminding me to run ci_hsc .
            Hide
            price Paul Price added a comment -

            OK, I'll buy it. Thanks for bearing with me.

            Show
            price Paul Price added a comment - OK, I'll buy it. Thanks for bearing with me.
            Hide
            hchiang2 Hsin-Fang Chiang added a comment -

            Merged now that RFC-249 is adopted.

            Show
            hchiang2 Hsin-Fang Chiang added a comment - Merged now that RFC-249 is adopted.

              People

              • Assignee:
                hchiang2 Hsin-Fang Chiang
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                Paul Price
                Watchers:
                Hsin-Fang Chiang, Paul Price
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel