Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-6255

Improve detail for for DRP imchar/jointcal in LDM-151

    XMLWordPrintable

    Details

    • Story Points:
      6
    • Epic Link:
    • Sprint:
      LDM-151 Week 2
    • Team:
      Data Release Production

      Description

      Write more detailed descriptions and possibly draw a rough diagram for the single-frame processing and simultaneous calibration components of Data Release Production.

      Does not necessarily involve turning this section into prose.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            Decreasing story point to reflect change in scope.

            Show
            jbosch Jim Bosch added a comment - Decreasing story point to reflect change in scope.
            Hide
            jbosch Jim Bosch added a comment -

            John Swinbank, could you take a quick look at this to see how the level of detail feels vis a vis planning needs (I'll make the GitHub PR shortly)?

            I strongly suspect the BootstrapImChar section is more detail than you need; I was worried about unknown unknowns in that area and wanted to go into extra detail to make sure we wouldn't have any big surprises there. I'm hoping the later sections are more at the level we'll need for planning.

            Show
            jbosch Jim Bosch added a comment - John Swinbank , could you take a quick look at this to see how the level of detail feels vis a vis planning needs (I'll make the GitHub PR shortly)? I strongly suspect the BootstrapImChar section is more detail than you need; I was worried about unknown unknowns in that area and wanted to go into extra detail to make sure we wouldn't have any big surprises there. I'm hoping the later sections are more at the level we'll need for planning.
            Hide
            jbosch Jim Bosch added a comment -

            Colin Slater: some notes for you on diagrams:

            • I haven't done any draft diagrams for this section; I found it easier to express the tricky stuff as a pseudocode block. Diagrams might be easier for users to interpret, if you want to take a look at what I've written and cook one up (though I think we may not want to capture the level of detail in the pseudocode in a diagram).
            • I've determined that the content of the RefineJointCal and BootstrapJointCal pipelines is identical, so I've merged them in the text with the new name StandardJointCal. Diagrams should be updated accordingly (I've not yet updated the draft overview diagram I had already put in the document, as I know you're already working on its replacement).
            Show
            jbosch Jim Bosch added a comment - Colin Slater : some notes for you on diagrams: I haven't done any draft diagrams for this section; I found it easier to express the tricky stuff as a pseudocode block. Diagrams might be easier for users to interpret, if you want to take a look at what I've written and cook one up (though I think we may not want to capture the level of detail in the pseudocode in a diagram). I've determined that the content of the RefineJointCal and BootstrapJointCal pipelines is identical, so I've merged them in the text with the new name StandardJointCal. Diagrams should be updated accordingly (I've not yet updated the draft overview diagram I had already put in the document, as I know you're already working on its replacement).
            Hide
            swinbank John Swinbank added a comment -

            Thanks for the ping; on my list for this afternoon.

            Show
            swinbank John Swinbank added a comment - Thanks for the ping; on my list for this afternoon.
            Hide
            swinbank John Swinbank added a comment -

            First comment – the travis build is failing, so this work is not showing up at https://ldm-151.lsst.io/v/DM-6255/LDM-151.pdf.

            ! LaTeX Error: File `xcolor.sty' not found.
            Type X to quit or <RETURN> to proceed,
            or enter new name. (Default extension: sty)
            Enter file name: 
            No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself.
            The build has been terminated
            

            Show
            swinbank John Swinbank added a comment - First comment – the travis build is failing, so this work is not showing up at https://ldm-151.lsst.io/v/DM-6255/LDM-151.pdf . ! LaTeX Error: File `xcolor.sty' not found. Type X to quit or <RETURN> to proceed, or enter new name. (Default extension: sty) Enter file name: No output has been received in the last 10m0s, this potentially indicates a stalled build or something wrong with the build itself. The build has been terminated
            Hide
            jbosch Jim Bosch added a comment -

            Bleh. I thought that was a pretty standard latex package. I'll find another package that's more common.

            Show
            jbosch Jim Bosch added a comment - Bleh. I thought that was a pretty standard latex package. I'll find another package that's more common.
            Hide
            swinbank John Swinbank added a comment -

            I pushed a fix, and Travis is now happy.

            I dunno if we'd want to find something more common in the long term, but suggest that this is fine for now.

            Show
            swinbank John Swinbank added a comment - I pushed a fix, and Travis is now happy. I dunno if we'd want to find something more common in the long term, but suggest that this is fine for now.
            Hide
            jbosch Jim Bosch added a comment -

            Great. I had a fix to avoid xcolor by defining a color manually, but I like yours better.

            Show
            jbosch Jim Bosch added a comment - Great. I had a fix to avoid xcolor by defining a color manually, but I like yours better.
            Hide
            jbosch Jim Bosch added a comment -

            Robert Lupton this is probably ready for you to start taking a look at; see the GitHub PR linked on the side.

            Show
            jbosch Jim Bosch added a comment - Robert Lupton this is probably ready for you to start taking a look at; see the GitHub PR linked on the side.
            Hide
            swinbank John Swinbank added a comment -

            Jim Bosch – thanks for this document. I have a few questions about the details of the procedures you're proposing (and will probably want to walk through them with you when we're doing detailed planning in a week or two) but I think this presentation is exceptionally clear and useful.

            For this section of the document, I think the level of detail you're capturing is quite appropriate for planning, especially bearing in mind that a lot (not all, of course) of the hard work (where we'll really need to invest time) will be in figuring out the details of the algorithmic components that make these high-level pipelines possible.

            A few points which are worth bearing in mind:

            • At several places, you refer to things which are "TBD". In some cases, I'm not sure whether they will be determined by you (or Robert, etc) based on some hard thinking as part of putting this document together, or whether they will be the result of research or experimentation during construction. Where it's the latter, we should call them out.
            • Related to the above, where there are decisions to be made during construction, it would be helpful to indicate how that decision should be made: is it driven by external factors (e.g. availability of Gaia catalogues), by research time that needs to be scheduled, etc?
            • How robust is this design against algorithmic components that don't come up to scratch? Would failure to deliver some component operating at some level of performance cause major changes to plan?
            • Related to the above, we should attempt to capture performance & fidelity requirements for algorithms wherever we can. I understand – and we've discussed before – that this is not necessarily possible in general (often, it's a matter of assembling the system an then hammering on the tall poles), but the more definitive we can be about this in advance, the better. (Plus, we must have a justification for believing that the system as designed will satisfy the science requirements.)
            Show
            swinbank John Swinbank added a comment - Jim Bosch – thanks for this document. I have a few questions about the details of the procedures you're proposing (and will probably want to walk through them with you when we're doing detailed planning in a week or two) but I think this presentation is exceptionally clear and useful. For this section of the document, I think the level of detail you're capturing is quite appropriate for planning, especially bearing in mind that a lot (not all, of course) of the hard work (where we'll really need to invest time) will be in figuring out the details of the algorithmic components that make these high-level pipelines possible. A few points which are worth bearing in mind: At several places, you refer to things which are "TBD". In some cases, I'm not sure whether they will be determined by you (or Robert, etc) based on some hard thinking as part of putting this document together, or whether they will be the result of research or experimentation during construction. Where it's the latter, we should call them out. Related to the above, where there are decisions to be made during construction, it would be helpful to indicate how that decision should be made: is it driven by external factors (e.g. availability of Gaia catalogues), by research time that needs to be scheduled, etc? How robust is this design against algorithmic components that don't come up to scratch? Would failure to deliver some component operating at some level of performance cause major changes to plan? Related to the above, we should attempt to capture performance & fidelity requirements for algorithms wherever we can. I understand – and we've discussed before – that this is not necessarily possible in general (often, it's a matter of assembling the system an then hammering on the tall poles), but the more definitive we can be about this in advance, the better. (Plus, we must have a justification for believing that the system as designed will satisfy the science requirements.)
            Hide
            jbosch Jim Bosch added a comment -

            At several places, you refer to things which are "TBD". In some cases, I'm not sure whether they will be determined by you (or Robert, etc) based on some hard thinking as part of putting this document together, or whether they will be the result of research or experimentation during construction. Where it's the latter, we should call them out.

            I believe all of these are research projects (or at least "thinking projects" that we'll need to plan for, not complete prior to planning). I'll update the text to clarify.

            Related to the above, where there are decisions to be made during construction, it would be helpful to indicate how that decision should be made: is it driven by external factors (e.g. availability of Gaia catalogues), by research time that needs to be scheduled, etc?

            I'll try to add annotations to this effect.

            How robust is this design against algorithmic components that don't come up to scratch? Would failure to deliver some component operating at some level of performance cause major changes to plan?

            I'm afraid it's not robust at all, but I think it generally gets more robust the deeper we get into the production. Essentially, we have a lot of iterative procedures that re hard to bootstrap, and that tends to put a much bigger burden on whichever of the components we choose to start with. If one of those components doesn't work out, the best response may be to restructure the pipelines to put some other component first.

            Related to the above, we should attempt to capture performance & fidelity requirements for algorithms wherever we can.

            I agree this is important (and I'm sorry I keep forgetting about it), but I'm hoping we can mostly defer figuring out success criteria until the designs themselves are better described (just because this would be enough extra work for me that I don't think I can get it all done by Zeljko Ivezic's deadline).

            Show
            jbosch Jim Bosch added a comment - At several places, you refer to things which are "TBD". In some cases, I'm not sure whether they will be determined by you (or Robert, etc) based on some hard thinking as part of putting this document together, or whether they will be the result of research or experimentation during construction. Where it's the latter, we should call them out. I believe all of these are research projects (or at least "thinking projects" that we'll need to plan for, not complete prior to planning). I'll update the text to clarify. Related to the above, where there are decisions to be made during construction, it would be helpful to indicate how that decision should be made: is it driven by external factors (e.g. availability of Gaia catalogues), by research time that needs to be scheduled, etc? I'll try to add annotations to this effect. How robust is this design against algorithmic components that don't come up to scratch? Would failure to deliver some component operating at some level of performance cause major changes to plan? I'm afraid it's not robust at all, but I think it generally gets more robust the deeper we get into the production. Essentially, we have a lot of iterative procedures that re hard to bootstrap, and that tends to put a much bigger burden on whichever of the components we choose to start with. If one of those components doesn't work out, the best response may be to restructure the pipelines to put some other component first. Related to the above, we should attempt to capture performance & fidelity requirements for algorithms wherever we can. I agree this is important (and I'm sorry I keep forgetting about it), but I'm hoping we can mostly defer figuring out success criteria until the designs themselves are better described (just because this would be enough extra work for me that I don't think I can get it all done by Zeljko Ivezic 's deadline).
            Hide
            swinbank John Swinbank added a comment -

            By the way, in the spirit of Zeljko's wide-then-deep plan, I don't think going back to add annotations as above should necessarily be your top priority: I think from my point of view of planning it's useful to sketch out the later stages of the system then return to refine this later.

            (Similarly, I'll add some more minor comments/request for clarification to the PR, but I don't think they need to be addressed urgently.)

            Show
            swinbank John Swinbank added a comment - By the way, in the spirit of Zeljko's wide-then-deep plan, I don't think going back to add annotations as above should necessarily be your top priority: I think from my point of view of planning it's useful to sketch out the later stages of the system then return to refine this later. (Similarly, I'll add some more minor comments/request for clarification to the PR, but I don't think they need to be addressed urgently.)
            Hide
            jbosch Jim Bosch added a comment -

            Sounds good. I've just clarified all the places I actually wrote "TBD" in the text, but I'll save the other issues for later (including looking for places where I implied TBD without actually writing it).

            Show
            jbosch Jim Bosch added a comment - Sounds good. I've just clarified all the places I actually wrote "TBD" in the text, but I'll save the other issues for later (including looking for places where I implied TBD without actually writing it).
            Hide
            jbosch Jim Bosch added a comment -

            I've merged the ticket branch to the draft branch, since I'm not planning any more work here soon. I'm leaving the ticket open for now so I don't forget to return to John Swinbank's comments at some point, but I'd be happy to take another approach to capturing those if he has a recommendation.

            Show
            jbosch Jim Bosch added a comment - I've merged the ticket branch to the draft branch, since I'm not planning any more work here soon. I'm leaving the ticket open for now so I don't forget to return to John Swinbank 's comments at some point, but I'd be happy to take another approach to capturing those if he has a recommendation.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Reviewers:
              John Swinbank
              Watchers:
              Jim Bosch, John Swinbank
              Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins Builds

                  No builds found.