Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30342

Propagate flags to Object tables

    XMLWordPrintable

Details

    • Bug
    • Status: Done
    • Resolution: Done
    • None
    • DM
    • None

    Description

      Many flags and relevant columns are not currently being propagated to the Object tables. Update the relevant YAML definition (e.g., https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L318-L320) to include the desired columns.

      Columns to add:

      detect_isIsolated,
      detect_isDeblendedModelSource,
      detect_isDeblendedSource,
      detect_fromBlend
      merge_peak_sky

      (more to be added in comments?)

      Attachments

        Activity

          lauren Lauren MacArthur added a comment - - edited

          I'm not sure what the full scope is for these tables, but it may also be useful to have the merge_measurement_band flags as well. These tell you which band was used for the "forced" measurements.

          While someone is in there, I also noticed that calib_psf_used appears in two places (I can't say for sure if that was oversight or deliberate...if it comes from ref I think it'll only be appropriate for the "reference" band, but it looks like all bands get an entry in the objectTable_tract tables...paging yusra ):
          https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L321
          https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L354

          lauren Lauren MacArthur added a comment - - edited I'm not sure what the full scope is for these tables, but it may also be useful to have the merge_measurement_band flags as well. These tell you which band was used for the "forced" measurements. While someone is in there, I also noticed that  calib_psf_used  appears in two places (I can't say for sure if that was oversight or deliberate...if it comes from ref I think it'll only be appropriate for the "reference" band, but it looks like all bands get an entry in the objectTable_tract tables...paging yusra ): https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L321 https://github.com/lsst/obs_subaru/blob/master/policy/Object.yaml#L354

          Fred, changes are required in obs_lsst, sdm_schemas too.
          And must pass a ci_hsc Jenkins run to merge.

          yusra Yusra AlSayyad added a comment - Fred, changes are required in obs_lsst, sdm_schemas too. And must pass a ci_hsc Jenkins run to merge.

          Changes have been made. I tested ci_hsc locally, so now I'm waiting on the Jenkins build.

          fred3m Fred Moolekamp added a comment - Changes have been made. I tested ci_hsc locally, so now I'm waiting on the Jenkins build.

          Hi Jeff,
          Would you mind reviewing this ticket to propagate the flags to Object tables?

          Thanks,
          -Fred

          fred3m Fred Moolekamp added a comment - Hi Jeff, Would you mind reviewing this ticket to propagate the flags to Object tables? Thanks, -Fred

          lauren I can't find `merge_measurement_band` in the lsst github search. Which task generates it? I need to know the datatype.

          fred3m Fred Moolekamp added a comment - lauren I can't find `merge_measurement_band` in the lsst github search. Which task generates it? I need to know the datatype.

          I don't have an Object table at hand, but I think that this should be `merge_measurement`, and the band is appended during table creation?

          jcarlin Jeffrey Carlin added a comment - I don't have an Object table at hand, but I think that this should be `merge_measurement`, and the band is appended during table creation?
          lauren Lauren MacArthur added a comment - They get set here: https://github.com/lsst/pipe_tasks/blob/master/python/lsst/pipe/tasks/mergeMeasurements.py#L247-L252

          Question for lauren or lskelvin – is `merge_peak_sky` the recommended way to select sky sources/objects? I recall something more explicit ("sky_source"?) in the past.

          jcarlin Jeffrey Carlin added a comment - Question for lauren or lskelvin – is `merge_peak_sky` the recommended way to select sky sources/objects? I recall something more explicit ("sky_source"?) in the past.
          fred3m Fred Moolekamp added a comment - - edited

          I think that `sky_source` is for single visits and when they are merged then `merge_peak_sky` is used on the coadd catalogs. If this creates a conflict then we should probably create a new `isSky` flag and set that for both single visits and coadds.

          fred3m Fred Moolekamp added a comment - - edited I think that `sky_source` is for single visits and when they are merged then `merge_peak_sky` is used on the coadd catalogs. If this creates a conflict then we should probably create a new `isSky` flag and set that for both single visits and coadds.

          Ah, thanks. I feel like `isSky` would be more intuitive, but it's not absolutely necessary...

          jcarlin Jeffrey Carlin added a comment - Ah, thanks. I feel like `isSky` would be more intuitive, but it's not absolutely necessary...

          PRs have been reviewed. We decided that `merge_measurement` flags will be left for a later ticket, since it's unclear how to handle them.

          The PR for `sdm_schemas` may need clarification from yusra.

          jcarlin Jeffrey Carlin added a comment - PRs have been reviewed. We decided that `merge_measurement` flags will be left for a later ticket, since it's unclear how to handle them. The PR for `sdm_schemas` may need clarification from yusra .
          lskelvin Lee Kelvin added a comment -

          jcarlin, yes, fred3m's reply above is correct: sky_source for visit level sky source identification, and merge_peak_sky for coadd/object level sky object identification. I'd certainly be in favor of a unified isSky flag however.

          lskelvin Lee Kelvin added a comment - jcarlin , yes, fred3m 's reply above is correct: sky_source for visit level sky source identification, and merge_peak_sky for coadd/object level sky object identification. I'd certainly be in favor of a unified isSky flag however.

          Created DM-30353 and DM-20354 for creating the `isSky` column and adding `merge_measurement` column respectively.

          fred3m Fred Moolekamp added a comment - Created DM-30353 and DM-20354 for creating the `isSky` column and adding `merge_measurement` column respectively.

          Jenkins build is successful on centos-7 but I'm still waiting on mac-os (https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34306/pipeline/56/). jcarlin is it ok to merge when the Jenkins build passes? If not, what still needs to be done?

          fred3m Fred Moolekamp added a comment - Jenkins build is successful on centos-7 but I'm still waiting on mac-os ( https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/34306/pipeline/56/ ). jcarlin is it ok to merge when the Jenkins build passes? If not, what still needs to be done?

          Jenkins build passed. Waiting on reviewer approval to merge.

          fred3m Fred Moolekamp added a comment - Jenkins build passed. Waiting on reviewer approval to merge.

          If you merge this, I'm just going to change Merge_peak_sky back to merge_peak_sky behind you. Users aren't going to remember that there's one column that starts with a capital.

          yusra Yusra AlSayyad added a comment - If you merge this, I'm just going to change Merge_peak_sky back to merge_peak_sky behind you. Users aren't going to remember that there's one column that starts with a capital.

          Since I'm on the west coast, I'm happy to fix it on this ticket, re-Jenkins and merge it before midnight.

          yusra Yusra AlSayyad added a comment - Since I'm on the west coast, I'm happy to fix it on this ticket, re-Jenkins and merge it before midnight.

          People

            fred3m Fred Moolekamp
            jcarlin Jeffrey Carlin
            Jeffrey Carlin
            Fred Moolekamp, Jeffrey Carlin, Lauren MacArthur, Lee Kelvin, Yusra AlSayyad
            Votes:
            0 Vote for this issue
            Watchers:
            5 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.