Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30055

There is still a tiny Gen2 vs Gen3 AP COSMOS CI difference

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:

      Description

      Turning on doScaleDiffimVariance across the board fixed HiTS. Making sure doBrighterFatter is the same for Gen2 and Gen3 fixed most - but not quite all - of HSC/COSMOS. Both of those things happened in DM-29344. However, chronograf shows the ratio of DIA to direct sources is still ever-so-slightly different.

      The ratio is 4.81 for Gen2 and 4.75 for Gen3 (visit 59160, detector 51)

      The ratio is 7.29 for Gen2 and 7.20 for Gen3 (visit 59150, detector 50)

      Whyyy.

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-29344 Investigate the CI differences between Gen 2 and 3 in COSMOS field

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            Can you point me to the new gen2 & gen3 HSC/COSMOS runs?

            Show
            lauren Lauren MacArthur added a comment - Can you point me to the new gen2 & gen3 HSC/COSMOS runs?
            Hide
            Permalink
            mrawls Meredith Rawls added a comment -

            I'm not sure where the CI output repos live. Krzysztof Findeisen may know? So I quickly kicked off my own ap_verify runs for you, using the latest weekly, which should have dropped after I merged DM-29344. They will be in /project/mrawls/ap_verify_runs/DM-30055-gen2 and DM-30055-gen3 in the same directory. There are logs in the same directory too, in DM-30055-gen2.out and DM-30055-gen3.out. Both runs should finish shortly.

            Show
            mrawls Meredith Rawls added a comment - I'm not sure where the CI output repos live. Krzysztof Findeisen may know? So I quickly kicked off my own ap_verify runs for you, using the latest weekly, which should have dropped after I merged DM-29344 . They will be in /project/mrawls/ap_verify_runs/ DM-30055 -gen2 and DM-30055 -gen3 in the same directory. There are logs in the same directory too, in DM-30055 -gen2.out and DM-30055 -gen3.out . Both runs should finish shortly.
            Hide
            Permalink
            krzys Krzysztof Findeisen added a comment -

            Jenkins does have scratch space in /project/jenkins/prod/, but it's hard to match up directories to runs, and I, at least, don't have read permission for the actual repos. But if we can't reproduce by running ap_verify anywhere, that would be scary for a different reason...

            Show
            krzys Krzysztof Findeisen added a comment - Jenkins does have scratch space in /project/jenkins/prod/ , but it's hard to match up directories to runs, and I, at least, don't have read permission for the actual repos. But if we can't reproduce by running ap_verify anywhere, that would be scary for a different reason...
            Hide
            Permalink
            lauren Lauren MacArthur added a comment -

            I just had a look at the results from SFM that Meredith pointed to above.  Things are looking pretty identical at that stage for both calexps of visit-ccd: 59150-050 59160-051 (those are the only two I see in /project/mrawls/ap_verify_runs/DM-30055-gen2/output/01527/HSC-G/corr/) having looked at the usual "all the things" (and the "pretty" qualifier is again just the offset in deblend_peakId when non-zero). Is it possible some "downstream" configs are out of sync?

            Show
            lauren Lauren MacArthur added a comment - I just had a look at the results from SFM that Meredith pointed to above.  Things are looking pretty identical at that stage for both calexps of visit-ccd: 59150-050 59160-051 (those are the only two I see in /project/mrawls/ap_verify_runs/ DM-30055 -gen2/output/01527/HSC-G/corr/ ) having looked at the usual "all the things" (and the "pretty" qualifier is again just the offset in deblend_peakId when non-zero). Is it possible some "downstream" configs are out of sync?
            Hide
            Permalink
            mrawls Meredith Rawls added a comment -

            Thanks Lauren. Those are indeed the only two visit+detector combos in the CI dataset, and what you found jives with what I expected. What do you think Eric Bellm - is this "close enough to identical" or in need of further examination?

            Show
            mrawls Meredith Rawls added a comment - Thanks Lauren. Those are indeed the only two visit+detector combos in the CI dataset, and what you found jives with what I expected. What do you think Eric Bellm - is this "close enough to identical" or in need of further examination?
            Hide
            Permalink
            ebellm Eric Bellm added a comment -

            Chris Morrison [X] indicated he saw floating-point differences when reading the raws--if he can isolate that let's spawn a new middleware ticket and close this one.

            Show
            ebellm Eric Bellm added a comment - Chris Morrison [X] indicated he saw floating-point differences when reading the raws--if he can isolate that let's spawn a new middleware ticket and close this one.
            Hide
            Permalink
            sullivan Ian Sullivan added a comment -

            Could this be due to the issue resolved in DM-30030?

            Show
            sullivan Ian Sullivan added a comment - Could this be due to the issue resolved in DM-30030 ?
            Hide
            Permalink
            lauren Lauren MacArthur added a comment - - edited

            When I did the comparison noted above, it included:

            • calexp image arrays (image, variance, mask planes)
            • calexp photoCalib objects
            • calexp PSFs
            • calexp WCSs
            • every column in the source tables

            which, modulo the deblend_peakId caveat, were all identical.  As such, I don’t think  either of the previous conjectures could be at play.

            Show
            lauren Lauren MacArthur added a comment - - edited When I did the comparison noted above, it included: calexp image arrays (image, variance, mask planes) calexp photoCalib objects calexp PSFs calexp WCSs every column in the source tables which, modulo the  deblend_peakId caveat, were all identical.  As such, I don’t  think   either of the previous conjectures could be at play.
            Hide
            Permalink
            sullivan Ian Sullivan added a comment -

            Gen 2 CI for AP has been turned off, so we are no longer planning to track down this difference.

            Show
            sullivan Ian Sullivan added a comment - Gen 2 CI for AP has been turned off, so we are no longer planning to track down this difference.

              People

              Assignee:
              Unassigned Unassigned
              Reporter:
              mrawls Meredith Rawls
              Watchers:
              Eric Bellm, Ian Sullivan, Krzysztof Findeisen, Lauren MacArthur, Meredith Rawls
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.