Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30342

Propagate flags to Object tables

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: DM
    • Labels:
      None

      Description

      Many flags and relevant columns are not currently being propagated to the Object tables. Update the relevant YAML definition (e.g., https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L318-L320) to include the desired columns.

      Columns to add:

      detect_isIsolated,
      detect_isDeblendedModelSource,
      detect_isDeblendedSource,
      detect_fromBlend
      merge_peak_sky

      (more to be added in comments?)

        Attachments

          Activity

          No builds found.
          jcarlin Jeffrey Carlin created issue -
          jcarlin Jeffrey Carlin made changes -
          Field Original Value New Value
          Assignee Fred Moolekamp [ fred3m ]
          Hide
          lauren Lauren MacArthur added a comment - - edited

          I'm not sure what the full scope is for these tables, but it may also be useful to have the merge_measurement_band flags as well. These tell you which band was used for the "forced" measurements.

          While someone is in there, I also noticed that calib_psf_used appears in two places (I can't say for sure if that was oversight or deliberate...if it comes from ref I think it'll only be appropriate for the "reference" band, but it looks like all bands get an entry in the objectTable_tract tables...paging Yusra AlSayyad ):
          https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L321
          https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L354

          Show
          lauren Lauren MacArthur added a comment - - edited I'm not sure what the full scope is for these tables, but it may also be useful to have the merge_measurement_band flags as well. These tell you which band was used for the "forced" measurements. While someone is in there, I also noticed that  calib_psf_used  appears in two places (I can't say for sure if that was oversight or deliberate...if it comes from ref I think it'll only be appropriate for the "reference" band, but it looks like all bands get an entry in the objectTable_tract tables...paging Yusra AlSayyad ): https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L321 https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L354
          Hide
          yusra Yusra AlSayyad added a comment -

          Fred, changes are required in obs_lsst, sdm_schemas too.
          And must pass a ci_hsc Jenkins run to merge.

          Show
          yusra Yusra AlSayyad added a comment - Fred, changes are required in obs_lsst, sdm_schemas too. And must pass a ci_hsc Jenkins run to merge.
          fred3m Fred Moolekamp made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          fred3m Fred Moolekamp added a comment -

          Changes have been made. I tested ci_hsc locally, so now I'm waiting on the Jenkins build.

          Show
          fred3m Fred Moolekamp added a comment - Changes have been made. I tested ci_hsc locally, so now I'm waiting on the Jenkins build.
          Hide
          fred3m Fred Moolekamp added a comment -

          Hi Jeff,
          Would you mind reviewing this ticket to propagate the flags to Object tables?

          Thanks,
          -Fred

          Show
          fred3m Fred Moolekamp added a comment - Hi Jeff, Would you mind reviewing this ticket to propagate the flags to Object tables? Thanks, -Fred
          fred3m Fred Moolekamp made changes -
          Reviewers Jeffrey Carlin [ jcarlin ]
          Status In Progress [ 3 ] In Review [ 10004 ]
          Hide
          fred3m Fred Moolekamp added a comment -

          Lauren MacArthur I can't find `merge_measurement_band` in the lsst github search. Which task generates it? I need to know the datatype.

          Show
          fred3m Fred Moolekamp added a comment - Lauren MacArthur I can't find `merge_measurement_band` in the lsst github search. Which task generates it? I need to know the datatype.
          Hide
          jcarlin Jeffrey Carlin added a comment -

          I don't have an Object table at hand, but I think that this should be `merge_measurement`, and the band is appended during table creation?

          Show
          jcarlin Jeffrey Carlin added a comment - I don't have an Object table at hand, but I think that this should be `merge_measurement`, and the band is appended during table creation?
          Show
          lauren Lauren MacArthur added a comment - They get set here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py#L247-L252
          Hide
          jcarlin Jeffrey Carlin added a comment -

          Question for Lauren MacArthur or Lee Kelvin – is `merge_peak_sky` the recommended way to select sky sources/objects? I recall something more explicit ("sky_source"?) in the past.

          Show
          jcarlin Jeffrey Carlin added a comment - Question for Lauren MacArthur or Lee Kelvin – is `merge_peak_sky` the recommended way to select sky sources/objects? I recall something more explicit ("sky_source"?) in the past.
          Hide
          fred3m Fred Moolekamp added a comment - - edited

          I think that `sky_source` is for single visits and when they are merged then `merge_peak_sky` is used on the coadd catalogs. If this creates a conflict then we should probably create a new `isSky` flag and set that for both single visits and coadds.

          Show
          fred3m Fred Moolekamp added a comment - - edited I think that `sky_source` is for single visits and when they are merged then `merge_peak_sky` is used on the coadd catalogs. If this creates a conflict then we should probably create a new `isSky` flag and set that for both single visits and coadds.
          Hide
          jcarlin Jeffrey Carlin added a comment -

          Ah, thanks. I feel like `isSky` would be more intuitive, but it's not absolutely necessary...

          Show
          jcarlin Jeffrey Carlin added a comment - Ah, thanks. I feel like `isSky` would be more intuitive, but it's not absolutely necessary...
          Hide
          jcarlin Jeffrey Carlin added a comment -

          PRs have been reviewed. We decided that `merge_measurement` flags will be left for a later ticket, since it's unclear how to handle them.

          The PR for `sdm_schemas` may need clarification from Yusra AlSayyad.

          Show
          jcarlin Jeffrey Carlin added a comment - PRs have been reviewed. We decided that `merge_measurement` flags will be left for a later ticket, since it's unclear how to handle them. The PR for `sdm_schemas` may need clarification from Yusra AlSayyad .
          jcarlin Jeffrey Carlin made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          Hide
          lskelvin Lee Kelvin added a comment -

          Jeffrey Carlin, yes, Fred Moolekamp's reply above is correct: sky_source for visit level sky source identification, and merge_peak_sky for coadd/object level sky object identification. I'd certainly be in favor of a unified isSky flag however.

          Show
          lskelvin Lee Kelvin added a comment - Jeffrey Carlin , yes, Fred Moolekamp 's reply above is correct: sky_source for visit level sky source identification, and merge_peak_sky for coadd/object level sky object identification. I'd certainly be in favor of a unified isSky flag however.
          Hide
          fred3m Fred Moolekamp added a comment -

          Created DM-30353 and DM-20354 for creating the `isSky` column and adding `merge_measurement` column respectively.

          Show
          fred3m Fred Moolekamp added a comment - Created DM-30353 and DM-20354 for creating the `isSky` column and adding `merge_measurement` column respectively.
          Hide
          fred3m Fred Moolekamp added a comment -

          Jenkins build is successful on centos-7 but I'm still waiting on mac-os (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34306/pipeline/56/). Jeffrey Carlin is it ok to merge when the Jenkins build passes? If not, what still needs to be done?

          Show
          fred3m Fred Moolekamp added a comment - Jenkins build is successful on centos-7 but I'm still waiting on mac-os ( https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34306/pipeline/56/ ). Jeffrey Carlin is it ok to merge when the Jenkins build passes? If not, what still needs to be done?
          Hide
          fred3m Fred Moolekamp added a comment -

          Jenkins build passed. Waiting on reviewer approval to merge.

          Show
          fred3m Fred Moolekamp added a comment - Jenkins build passed. Waiting on reviewer approval to merge.
          Hide
          yusra Yusra AlSayyad added a comment -

          If you merge this, I'm just going to change Merge_peak_sky back to merge_peak_sky behind you. Users aren't going to remember that there's one column that starts with a capital.

          Show
          yusra Yusra AlSayyad added a comment - If you merge this, I'm just going to change Merge_peak_sky back to merge_peak_sky behind you. Users aren't going to remember that there's one column that starts with a capital.
          Hide
          yusra Yusra AlSayyad added a comment -

          Since I'm on the west coast, I'm happy to fix it on this ticket, re-Jenkins and merge it before midnight.

          Show
          yusra Yusra AlSayyad added a comment - Since I'm on the west coast, I'm happy to fix it on this ticket, re-Jenkins and merge it before midnight.
          yusra Yusra AlSayyad made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          yusra Yusra AlSayyad made changes -
          Epic Link DM-29152 [ 458509 ]
          Story Points 2

            People

            Assignee:
            fred3m Fred Moolekamp
            Reporter:
            jcarlin Jeffrey Carlin
            Reviewers:
            Jeffrey Carlin
            Watchers:
            Fred Moolekamp, Jeffrey Carlin, Lauren MacArthur, Lee Kelvin, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.