Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13375

DPDD does not mention DRP-generated DIAObject/DIASource

    Details

    • Team:
      DM Science

      Description

      The DPDD currently lists schemas for Object, Source, and ForcedSource under §4.3 - "The Level 2 Catalogs". This omits the DIASource and DIAObject tables, which LDM-151 says will also be generated during data release production. 

      A discussion of these tables should be added to the DRP section of the DPDD so that their rationale and usage can be explained, particularly in relation to other characterizations of variable objects in the Object and prompt-processing-generated DIAObject tables. Alternatively, if either of these tables are meant to be for internal usage only (e.g. solely as inputs to Object), LDM-151 should make that clear.

      Assigning to Mario Juric with the intention that it will be reassigned to the next subsystem scientist.

        Attachments

          Issue Links

            Activity

            Hide
            ctslater Colin Slater added a comment -

            Sorry, to be more precise, the question is: Do AP force photometry records mostly "look like" full DIASource records, or do they look more like (as an example) ForceSource records with only one or two fluxes. Similarly, should queries for DIASources also potentially include force photometry (filterable with a flag if the user doesn't want it), or should the user need to run a separate query for force photometry? 

            Show
            ctslater Colin Slater added a comment - Sorry, to be more precise, the question is: Do AP force photometry records mostly "look like" full DIASource records, or do they look more like (as an example) ForceSource records with only one or two fluxes. Similarly, should queries for DIASources also potentially include force photometry (filterable with a flag if the user doesn't want it), or should the user need to run a separate query for force photometry? 
            Hide
            ebellm Eric Bellm added a comment -

            I think the question of where to put AP forced photometry in the PPDB depends in part on whether past forced photometry records are included in alerts. It's clear that precovery and forced photometry measurements on the current visit are not. But if forced sources are just rows in the DIASource table with forced=True, it would be trivial to include any that exist in the past 12 months of lightcurve history. This would be scientifically far better than only reporting crude upper limits (see https://jira.lsstcorp.org/browse/RFC-348, which I think Leanne Guy is also shepherding to the CCB...), and since we'd be replacing an upper limit with a measurement the packet size would not be strongly affected. If folks are amenable I'll RFC this.

            I agree with John Swinbank that whether the DIAForcedSource table exists separately is likely an implementation detail. Since we are only forcing PSF fluxes the table will be narrower.

            Show
            ebellm Eric Bellm added a comment - I think the question of where to put AP forced photometry in the PPDB depends in part on whether past forced photometry records are included in alerts. It's clear that precovery and forced photometry measurements on the current visit are not. But if forced sources are just rows in the DIASource table with forced=True, it would be trivial to include any that exist in the past 12 months of lightcurve history. This would be scientifically far better than only reporting crude upper limits (see https://jira.lsstcorp.org/browse/RFC-348 , which I think Leanne Guy is also shepherding to the CCB...), and since we'd be replacing an upper limit with a measurement the packet size would not be strongly affected. If folks are amenable I'll RFC this. I agree with John Swinbank that whether the DIAForcedSource table exists separately is likely an implementation detail. Since we are only forcing PSF fluxes the table will be narrower.
            Hide
            ebellm Eric Bellm added a comment -

            My answer to Colin Slater's questions would be:

            DIAForcedSource records don't look like DIASource records (they don't include dipole or trailed source fits/fluxes, spuriousness, etc.)

            It would be convenient for users to be able to retrieve both DIASource and forced photometry at once, but I don't think it's essential.

            Show
            ebellm Eric Bellm added a comment - My answer to Colin Slater 's questions would be: DIAForcedSource records don't look like DIASource records (they don't include dipole or trailed source fits/fluxes, spuriousness, etc.) It would be convenient for users to be able to retrieve both DIASource and forced photometry at once, but I don't think it's essential.
            Hide
            swinbank John Swinbank added a comment -

            ... whether past forced photometry records are included in alerts ...

            Given that the DPDD as currently written:

            • Includes forced photometry in the DIASource table;
            • States that alert packets include the “previous 12 months of DIASource records”, without qualification;

            I'd suggest that the naïve answer to that is that they should be, and I'm not even sure it needs an RFC (although I support making one, for clarity).

            Show
            swinbank John Swinbank added a comment - ... whether past forced photometry records are included in alerts ... Given that the DPDD as currently written: Includes forced photometry in the DIASource table; States that alert packets include the “previous 12 months of DIASource records”, without qualification; I'd suggest that the naïve answer to that is that they should be, and I'm not even sure it needs an RFC (although I support making one, for clarity).
            Hide
            lguy Leanne Guy added a comment - - edited

            This topic was discussed at length at the dm-sst meeting 2018-08-24. A separate RFC will be created to address the PPDB tables DIASource/DIAForcedSource. 

            Show
            lguy Leanne Guy added a comment - - edited This topic was discussed at length at the dm-sst meeting 2018-08-24 . A separate RFC will be created to address the PPDB tables DIASource/DIAForcedSource. 

              People

              • Assignee:
                lguy Leanne Guy
                Reporter:
                ctslater Colin Slater
                Watchers:
                Colin Slater, Eric Bellm, Gregory Dubois-Felsmann, Jim Bosch, John Swinbank, Leanne Guy, Zeljko Ivezic
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Due:
                  Created:
                  Updated:

                  Summary Panel