Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-13053

allow refcat selection improve fitting

    Details

    • Story Points:
      4
    • Sprint:
      AP S18-6, AP F18-1, AP F18-2, AP F18-3, AP S19-2, AP S19-3
    • Team:
      Alert Production

      Description

      Dominique Boutigny's work on selecting reference sources by S/N and magnitude might help improve the refcat that jointcal uses. But Paul Price made a ReferenceSourceSelectorTask that we can configure for this purpose, instead of having code internal to jointcal.

      This ticket is to convert Dominique Boutigny's work in the u/fix_outliers branch (specifically this commit: https://github.com/lsst/jointcal/commit/e8393ac2e21117c15458156423445b19f75b503a ) to use ReferenceSourceSelectorTask, add a unittest to demonstrate that it does affect the fit, and update the other tests to reflect any changes in the fit metrics.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Yes, just one sourceSelector for now. I have a ticket for changing that (DM-12990), but it's rather difficult, because source selection happens while the data is being loaded and before the ccdImages are created. I'm putting it off until it turns out that we actually need it, and/or we've sorted out some of the memory pressure questions of working with tracts that have a lot of visits. Jointcal gets a pretty marked speedup because it only has to read the catalogs and metadata once for both astrometry and photometry. The refcat loading is a tiny fraction of that time, so splitting that was easy.

            Show
            Parejkoj John Parejko added a comment - Yes, just one sourceSelector for now. I have a ticket for changing that ( DM-12990 ), but it's rather difficult, because source selection happens while the data is being loaded and before the ccdImages are created. I'm putting it off until it turns out that we actually need it, and/or we've sorted out some of the memory pressure questions of working with tracts that have a lot of visits. Jointcal gets a pretty marked speedup because it only has to read the catalogs and metadata once for both astrometry and photometry. The refcat loading is a tiny fraction of that time, so splitting that was easy.
            Hide
            yusra Yusra AlSayyad added a comment -

            Yes, I would like two separate referenceSelection subtasks so we can apply two different cuts for photometry and astrometry. 

            Show
            yusra Yusra AlSayyad added a comment - Yes, I would like two separate referenceSelection subtasks so we can apply two different cuts for photometry and astrometry. 
            Hide
            Parejkoj John Parejko added a comment -

            Yusra AlSayyad: can you review the update that splits it into two selectors? Two new commits: one to cleanup numpydoc strings, the other to do the splitting.

            New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29470/pipeline

            Show
            Parejkoj John Parejko added a comment - Yusra AlSayyad : can you review the update that splits it into two selectors? Two new commits: one to cleanup numpydoc strings, the other to do the splitting. New Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29470/pipeline
            Hide
            yusra Yusra AlSayyad added a comment -

            Looks good. 

            Show
            yusra Yusra AlSayyad added a comment - Looks good. 
            Hide
            Parejkoj John Parejko added a comment -

            Thanks for the reviews. I'll leave it to others to see how much this helps the DC2 fitting.

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thanks for the reviews. I'll leave it to others to see how much this helps the DC2 fitting. Merged and done.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Reviewers:
                Yusra AlSayyad
                Watchers:
                Dominique Boutigny, Eli Rykoff, Jim Bosch, John Parejko, John Swinbank, Pierre Astier, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel