Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10826

fix jointcal collectRefStar handling of position errors

    Details

    • Story Points:
      4
    • Sprint:
      AP S19-5, AP S19-6
    • Team:
      Alert Production

      Description

      Jointcal does some odd things when building its RefStar catalog (Associations::collectRefStars), including creating fake errors (maybe because the original reference catalogs didn't have useful errors?), making a magnitude from the flux (is that really necessary? Why can't we work in flux units here?), and making BaseStars followed by a cast, when it should just be able to make RefStars.

        Attachments

          Issue Links

            Activity

            Hide
            yusra Yusra AlSayyad added a comment -

            Pasted from github and private slack communication:

            Running with default configs should do something sensible. Our current default reference catalogs have '0' for their ra/dec errors.

            See two logs in /datasets/hsc/repo/rerun/private/yusra/RC2/w_2019_06_DM-10826/logs

            • HSC-G-9615-attempt1.log run with defaults:

              [yusra@lsst-dev01 logs]$ tail HSC-G-9615-attempt1.log
              jointcal.Associations INFO: Unmatched objects: 12
              jointcal.Associations INFO: Matched 146 objects in 26094_103
              jointcal.Associations INFO: Unmatched objects: 14
              jointcal.astrometryReferenceSelector INFO: Selected 1281797/1281797 references
              jointcal INFO: Applying color terms for filterName='g' reference catalog=ps1_pv3_3pi_20170110
              jointcal FATAL: Failed processing tract 9615, TypeError: 
                File "src/table/Schema.cc", line 211, in lsst::afw::table::SchemaItem<T> lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = double; std::string = std::basic_string<char>]
                  Field 'coord_ra_err' does not have the given type. {0}
              lsst::pex::exceptions::TypeError: 'Field 'coord_ra_err' does not have the given type.'
              

            • HSC-G-9615-attempt2.log was run with --config astrometryReferenceErr=10 and it completes.

            How to reproduce (note replacement of username):
            jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun RC/w_2019_06/DM-17400-sfm:private/parejkoj/RC2/w_2019_06_DM-10826 --id ccd=0..8^10..103 visit=26024^26028^26032^26036^26044^26046^26048^26050^26058^26060^26062^26070^26072^26074^26080^26084^26094 filter=HSC-G tract=9615 --no-versions --clobber-config &> HSC-G-9615.log

            Show
            yusra Yusra AlSayyad added a comment - Pasted from github and private slack communication: Running with default configs should do something sensible. Our current default reference catalogs have '0' for their ra/dec errors. See two logs in /datasets/hsc/repo/rerun/private/yusra/RC2/w_2019_06_ DM-10826 /logs HSC-G-9615-attempt1.log run with defaults: [yusra@lsst-dev01 logs]$ tail HSC-G-9615-attempt1.log jointcal.Associations INFO: Unmatched objects: 12 jointcal.Associations INFO: Matched 146 objects in 26094_103 jointcal.Associations INFO: Unmatched objects: 14 jointcal.astrometryReferenceSelector INFO: Selected 1281797/1281797 references jointcal INFO: Applying color terms for filterName='g' reference catalog=ps1_pv3_3pi_20170110 jointcal FATAL: Failed processing tract 9615, TypeError: File "src/table/Schema.cc", line 211, in lsst::afw::table::SchemaItem<T> lsst::afw::table::detail::SchemaImpl::find(const string&) const [with T = double; std::string = std::basic_string<char>] Field 'coord_ra_err' does not have the given type. {0} lsst::pex::exceptions::TypeError: 'Field 'coord_ra_err' does not have the given type.' HSC-G-9615-attempt2.log was run with --config astrometryReferenceErr=10 and it completes. How to reproduce (note replacement of username): jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun RC/w_2019_06/ DM-17400 -sfm:private/parejkoj/RC2/w_2019_06_ DM-10826 --id ccd=0..8^10..103 visit=26024^26028^26032^26036^26044^26046^26048^26050^26058^26060^26062^26070^26072^26074^26080^26084^26094 filter=HSC-G tract=9615 --no-versions --clobber-config &> HSC-G-9615.log
            Hide
            Parejkoj John Parejko added a comment - - edited

            I've fixed the above issue and rebased/squashed. Are you happy with this, Yusra AlSayyad? Should we have another ticket to set the subaru/lsst refcat coord errors?

            New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29790/pipeline

            Show
            Parejkoj John Parejko added a comment - - edited I've fixed the above issue and rebased/squashed. Are you happy with this, Yusra AlSayyad ? Should we have another ticket to set the subaru/lsst refcat coord errors? New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29790/pipeline
            Hide
            yusra Yusra AlSayyad added a comment -

            Please add the config for obs_subaru/decam (10mas) and obs_lsst (1mas) on this ticket. jointcal needs to do the thing (without commandline overrides).

            Show
            yusra Yusra AlSayyad added a comment - Please add the config for obs_subaru/decam (10mas) and obs_lsst (1mas) on this ticket. jointcal needs to do the thing (without commandline overrides).
            Hide
            Parejkoj John Parejko added a comment -

            I added config overrides for lsst, decam, and subaru (and a ticket to remove them once the Gaia DR2 catalog is in place), and did a trial run with your command above, which seems to have worked. Do you want to take a quick look at the config PRs?

            Show
            Parejkoj John Parejko added a comment - I added config overrides for lsst, decam, and subaru (and a ticket to remove them once the Gaia DR2 catalog is in place), and did a trial run with your command above, which seems to have worked. Do you want to take a quick look at the config PRs?
            Hide
            Parejkoj John Parejko added a comment -

            Thank you for the review comments, Yusra AlSayyad and Andy Salnikov! I filed DM-19640 to remind us to remove these overrides once Gaia DR2 is in place.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thank you for the review comments, Yusra AlSayyad and Andy Salnikov ! I filed DM-19640 to remind us to remove these overrides once Gaia DR2 is in place. Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Andy Salnikov, Dominique Boutigny, Hisanori Furusawa, John Parejko, John Swinbank, Pierre Astier, Simon Krughoff, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                8 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel