Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-16482

Add limited support for ExposureRange to Pre-flight

    Details

      Description

      DM-16467 cannot be completed until we have ExposureRange some support in pre-flight. Minimum support that we need for that is to be able to specify ExposureRange in DatasetTypes, support in user filter query can be implemented later.

        Attachments

          Issue Links

            Activity

            Hide
            salnikov Andy Salnikov added a comment -

            Jim Bosch, I have implemented minimal changes to support ExposureRange, though this is just a temporary hack to unblock isrTask migration (but unit test should be solid). I do agree that we need better schema to reflect the nature of calibrations, but I think it has to be something different from regular DataUnits. In preflight I have to treat ExposureRange differently from other DataUnits, and reason for it is that there can only be a single regular DataUnit per Quantum but there maybe multiple different ExposureRanges in the same Quantum. That distinction will remain even if you rename it to CalibIdentifier, so something in the schema should reflect this difference.

            Show
            salnikov Andy Salnikov added a comment - Jim Bosch , I have implemented minimal changes to support ExposureRange, though this is just a temporary hack to unblock isrTask migration (but unit test should be solid). I do agree that we need better schema to reflect the nature of calibrations, but I think it has to be something different from regular DataUnits. In preflight I have to treat ExposureRange differently from other DataUnits, and reason for it is that there can only be a single regular DataUnit per Quantum but there maybe multiple different ExposureRanges in the same Quantum. That distinction will remain even if you rename it to CalibIdentifier, so something in the schema should reflect this difference.
            Hide
            jbosch Jim Bosch added a comment -

            there can only be a single regular DataUnit per Quantum but there maybe multiple different ExposureRanges in the same Quantum

            How is this different from any other gather-style step?  In the classic coaddition example, there are multiple Visits that contribute (via Visit+Patch inputs) to a Quantum (Patch+Filter).

            (That question aside, I'll start looking at the code changes now.)

            Show
            jbosch Jim Bosch added a comment - there can only be a single regular DataUnit per Quantum but there maybe multiple different ExposureRanges in the same Quantum How is this different from any other gather-style step?  In the classic coaddition example, there are multiple Visits that contribute (via Visit+Patch inputs) to a Quantum (Patch+Filter). (That question aside, I'll start looking at the code changes now.)
            Hide
            salnikov Andy Salnikov added a comment -

            The difference between ExposureRanges and other DataUnits is that for normal units all relationships are defined at the DataUnit level and do not depend on Datasets. If there is many-to-one or many-to-many relationship between units it is determined by the structure/topology of the "unit universe" (e.g. relation between Visits and Patches is determined by corresponding DataUnits/Joins). That topology may also be trimmed by the pre-existing input Datasets but in general that unit topology is the same for any Dataset. If for example we build a graph to produce two DatasetType with Patch unit then input to those DatasetTypes will be identical set of Visits (latter can also come from different DatasetTypes, and may be trimmed by existing inputs).

            For ExposureRanges the whole thing is different, we do not have DataUnit-level relationship between them and other units, this is by its nature (or by its size) can only be defined at Dataset level and it is Dataset-specific. So in the same Quantum we end up with one set of ExposureRange units for one calibration and completely unrelated set of ExposureRange units for other calibration. It is of course possible to say that whole universe of ExposureRange units exists with all possible combinations of begin/end validity times and we include all of them (with some time constraints) in quanta but that is not really helpful (even at second-level precision this is a lot), especially for defining outputs with those units.

            Show
            salnikov Andy Salnikov added a comment - The difference between ExposureRanges and other DataUnits is that for normal units all relationships are defined at the DataUnit level and do not depend on Datasets. If there is many-to-one or many-to-many relationship between units it is determined by the structure/topology of the "unit universe" (e.g. relation between Visits and Patches is determined by corresponding DataUnits/Joins). That topology may also be trimmed by the pre-existing input Datasets but in general that unit topology is the same for any Dataset. If for example we build a graph to produce two DatasetType with Patch unit then input to those DatasetTypes will be identical set of Visits (latter can also come from different DatasetTypes, and may be trimmed by existing inputs). For ExposureRanges the whole thing is different, we do not have DataUnit-level relationship between them and other units, this is by its nature (or by its size) can only be defined at Dataset level and it is Dataset-specific. So in the same Quantum we end up with one set of ExposureRange units for one calibration and completely unrelated set of ExposureRange units for other calibration. It is of course possible to say that whole universe of ExposureRange units exists with all possible combinations of begin/end validity times and we include all of them (with some time constraints) in quanta but that is not really helpful (even at second-level precision this is a lot), especially for defining outputs with those units.
            Hide
            jbosch Jim Bosch added a comment -

            Ah, ok, I understand now.  I am thinking that when we add the extra level of indirection with CalibIdentifier, then we will need to have a separate CalibIdentifier DataUnit table that contains all of the ExposureRange information.  That would make it like a regular (pre-defined) DataUnit rather than something strictly pulled out of the Dataset table.

            Anyhow, code all looks good for now, and while I hope to make the schema changes that would allow us to clean it up soon, I'm not yet sure when I'll get to them.

            Show
            jbosch Jim Bosch added a comment - Ah, ok, I understand now.  I am thinking that when we add the extra level of indirection with CalibIdentifier , then we will need to have a separate CalibIdentifier DataUnit table that contains all of the ExposureRange information.  That would make it like a regular (pre-defined) DataUnit rather than something strictly pulled out of the Dataset table. Anyhow, code all looks good for now, and while I hope to make the schema changes that would allow us to clean it up soon, I'm not yet sure when I'll get to them.
            Hide
            salnikov Andy Salnikov added a comment -

            Thanks for review! Merged to master.

            New CalibIdentifier would help to make it uniform, though maintaining CalibIdentifier <-> Exposure join table may become an issue.

            Show
            salnikov Andy Salnikov added a comment - Thanks for review! Merged to master. New CalibIdentifier would help to make it uniform, though maintaining CalibIdentifier <-> Exposure join table may become an issue.

              People

              • Assignee:
                salnikov Andy Salnikov
                Reporter:
                salnikov Andy Salnikov
                Reviewers:
                Jim Bosch
                Watchers:
                Andy Salnikov, Jim Bosch, Vaikunth Thukral
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel