Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-9962

Write proposal for storing coadd HeavyFootprints

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      DRP S17-5
    • Team:
      Data Release Production

      Description

      Deblending results in the form of HeavyFootprints should be available to users, which means they must be stored.

      This proposal should include an estimate of how much storage this will require for a given area of sky, which can probably be derived from the average HeavyFootprint size in HSC data. This will be a function of depth, so it will not be the same at the beginning of the survey as it will be at the end (but we should have HSC data that spans those depths).

      This will be a conservative estimate - it is possible that deblender outputs may take a form that allows them to be highly compressed and quickly reconstituted given coadd data.

      We will not consider the possibility that it will be necessary to store per-epoch HeavyFootprints for ForcedSources.

        Attachments

          Issue Links

            Activity

            Hide
            jbosch Jim Bosch added a comment -

            HSC does store HeavyFootprints, but we don't make them terribly easy to access and I don't think anyone has used them. AFAIK, SDSS is the only other survey with a deblender sophisticated enough to produce them, and Robert Lupton certainly has at least anecdotal reports of people using them; he may be the only one who can make that statement quantitative (or the argument in general stronger).

            Show
            jbosch Jim Bosch added a comment - HSC does store HeavyFootprints, but we don't make them terribly easy to access and I don't think anyone has used them. AFAIK, SDSS is the only other survey with a deblender sophisticated enough to produce them, and Robert Lupton certainly has at least anecdotal reports of people using them; he may be the only one who can make that statement quantitative (or the argument in general stronger).
            Hide
            rhl Robert Lupton added a comment -

            The SDSS deblender was already sophisticated enough that you couldn't just look at a parent and visualise its children, and the tables in the DB are the results of measurements on those children.

            Whenever anyone asked me about problems in photometry in SDSS the first thing I did was to look at the children; if this had been more readily available I'd have had fewer questions. The LSST deblender will be more complicated; it'll also be better, but it's solving a much harder problem. There is no question that we will need easy access to the HeavyFootprints. What I do not know is how often this will be needed, and what would be the latency to recalculate them.

            Show
            rhl Robert Lupton added a comment - The SDSS deblender was already sophisticated enough that you couldn't just look at a parent and visualise its children, and the tables in the DB are the results of measurements on those children. Whenever anyone asked me about problems in photometry in SDSS the first thing I did was to look at the children; if this had been more readily available I'd have had fewer questions. The LSST deblender will be more complicated; it'll also be better, but it's solving a much harder problem. There is no question that we will need easy access to the HeavyFootprints . What I do not know is how often this will be needed, and what would be the latency to recalculate them.
            Hide
            tjenness Tim Jenness added a comment -

            An RFC sorting out exactly what is going to be included in the change request seems like a good approach to me. I don't want to throw this at the CCB without explicit sign off in DM first and without knowing exactly what the proposal is (including the costing).

            Show
            tjenness Tim Jenness added a comment - An RFC sorting out exactly what is going to be included in the change request seems like a good approach to me. I don't want to throw this at the CCB without explicit sign off in DM first and without knowing exactly what the proposal is (including the costing).
            Hide
            jbosch Jim Bosch added a comment -

            I think all reviewers have made comments now and I've addressed them to the best of my ability, so I'm marking this ticket as Done. If the next step is an RFC, I probably shouldn't be the person to do it - this effort is now moving outside my area of expertise (i.e. from measuring footprints to knowing what the CCB wants).

            Show
            jbosch Jim Bosch added a comment - I think all reviewers have made comments now and I've addressed them to the best of my ability, so I'm marking this ticket as Done. If the next step is an RFC, I probably shouldn't be the person to do it - this effort is now moving outside my area of expertise (i.e. from measuring footprints to knowing what the CCB wants).
            Hide
            tjenness Tim Jenness added a comment -

            RFC-329 filed.

            Show
            tjenness Tim Jenness added a comment - RFC-329 filed.

              People

              • Assignee:
                jbosch Jim Bosch
                Reporter:
                jbosch Jim Bosch
                Reviewers:
                Mario Juric, Robert Lupton, Tim Jenness
                Watchers:
                Gregory Dubois-Felsmann, Jim Bosch, John Swinbank, Kian-Tat Lim, Mario Juric, Robert Lupton, Tim Jenness
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel