Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-19761

Test run gaia dr2 refcat with jointcal and/or processCcd

    Details

    • Story Points:
      2
    • Epic Link:
    • Sprint:
      AP S19-6
    • Team:
      Alert Production

      Description

      I have created a "symlink repository" for HSC at /scratch/parejkoj/hsc (should be group-writeable by lsst_users), which we can use to do a trial run of processing data with the Gaia DR2 catalog in jointcal and/or processCcd. This is easier than moving data or symlinking into the read-only /datasets/ref_cats.

      We now have a /datasets/refCats/htm/v1/gaia_DR2 symlink to my /scratch gaia refcat, which we can use to do a trial processing run with this new catalog in jointcal.

      Hsin-Fang Chiang: can you please help me start an appropriate HSC run (PDR1, RC2 or something else small-ish I don't really care) using this? I'm not sure what we need to do to test it other than just demonstrating that it runs to completion: it has more sources than Gaia DR1, but not more than PS1, and it's still only usable for astrometry.

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Hsin-Fang Chiang suggested picking a tract, filter, and visit list from this RC2 table:

            https://jira.lsstcorp.org/browse/DM-11345?focusedCommentId=90372&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-90372

            For lack of any obvious choice, lets try HSC-I 9813 (33 visits).

            Show
            Parejkoj John Parejko added a comment - Hsin-Fang Chiang suggested picking a tract, filter, and visit list from this RC2 table: https://jira.lsstcorp.org/browse/DM-11345?focusedCommentId=90372&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-90372 For lack of any obvious choice, lets try HSC-I 9813 (33 visits).
            Hide
            Parejkoj John Parejko added a comment -

            I've attached the jointcal config I used for my trial run.

            Unfortunately, it looks like jointcal was not happy with this refcat: there were fewer matches than expected, and the fitter oscillated and then failed to converge. I've noticed some oddities in the refcat source ids, and I'm going to dig into that: it may be that my multiprocessing locks aren't doing what I wanted.

            The log is here: /scratch/parejkoj/gaia/jointcal-9813-I-gaia.log

            Show
            Parejkoj John Parejko added a comment - I've attached the jointcal config I used for my trial run. Unfortunately, it looks like jointcal was not happy with this refcat: there were fewer matches than expected, and the fitter oscillated and then failed to converge. I've noticed some oddities in the refcat source ids, and I'm going to dig into that: it may be that my multiprocessing locks aren't doing what I wanted. The log is here: /scratch/parejkoj/gaia/jointcal-9813-I-gaia.log
            Hide
            Parejkoj John Parejko added a comment -

            Jointcal is somewhat happier with the Gaia DR2 refcat I generated using the latest version of DM-19616. Since the current refcat lives on my desktop, I just ran it with validation_data_hsc. It found plenty of reference matches, but still did not converge. I suspect we will have to modify the ReferenceSourceSelector options to better match Gaia instead of PS1, but we can do that on another ticket. For now, the fact that jointcal found a reasonable number of reference sources per sensor is enough to convince me that the refcat I generated is ok enough.

            Show
            Parejkoj John Parejko added a comment - Jointcal is somewhat happier with the Gaia DR2 refcat I generated using the latest version of DM-19616 . Since the current refcat lives on my desktop, I just ran it with validation_data_hsc. It found plenty of reference matches, but still did not converge. I suspect we will have to modify the ReferenceSourceSelector options to better match Gaia instead of PS1, but we can do that on another ticket. For now, the fact that jointcal found a reasonable number of reference sources per sensor is enough to convince me that the refcat I generated is ok enough.

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                Parejkoj John Parejko
                Watchers:
                Hsin-Fang Chiang, Jim Bosch, John Parejko, John Swinbank, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                5 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel