# Jacobian lost in meas_mosaic photometric solution

XMLWordPrintable

#### Details

• Type: Story
• Status: Done
• Resolution: Done
• Fix Version/s: None
• Component/s:
• Labels:
• Story Points:
1
• Sprint:
AP S19-5
• Team:

#### Description

It looks like we lost the Jacobian from the meas_mosaic photometric solution with this deletion from DM-10156:
I noticed this in comparing meas_mosaic-calibrated magnitudes between the w_2019_10 and w_2019_14 RC2 reprocessings, which look like:

#### Activity

No builds found.
Lauren MacArthur created issue -
Field Original Value New Value
Link This issue relates to DM-10156 [ DM-10156 ]
Hide
John Parejko added a comment -

I wonder if this is coupled with DM-19015? We weren't actually using the meas_mosaic persisted PhotoCalib anywhere, except that bug made us use it as a stop-gap. So, changes to the persisted PhotoCalib wouldn't have shown up anywhere except for that.

Show
John Parejko added a comment - I wonder if this is coupled with DM-19015 ? We weren't actually using the meas_mosaic persisted PhotoCalib anywhere, except that bug made us use it as a stop-gap. So, changes to the persisted PhotoCalib wouldn't have shown up anywhere except for that.
 Link This issue relates to DM-19015 [ DM-19015 ]
 Sprint AP S19-5 [ 833 ] Story Points 1 Assignee John Parejko [ parejkoj ] Labels SciencePipelines Priority Undefined [ 10000 ] Critical [ 2 ]
 Status To Do [ 10001 ] In Progress [ 3 ]
Hide
Lauren MacArthur added a comment -

In DM-13202 I confirmed that the results using the meas_mosaic persisted PhotoCalib vs. meas_mosaic-special calibrations (à la fcr files) to a catalog were (for all intents and purposes) identical, so this is a new bug.

Show
Lauren MacArthur added a comment - In DM-13202 I confirmed that the results using the meas_mosaic persisted PhotoCalib vs. meas_mosaic -special calibrations (à la fcr files) to a catalog were (for all intents and purposes) identical, so this is a new bug.
Hide
John Parejko added a comment -

Yes, I know. But we would never have noticed if not for the bug in DM-19015, since we aren't making use of the meas_mosaic PhotoCalibs.

Show
John Parejko added a comment - Yes, I know. But we would never have noticed if not for the bug in DM-19015 , since we aren't making use of the meas_mosaic PhotoCalibs.
Hide
John Parejko added a comment -

I believe this PR fixes this bug. None of the tests here can confirm it though.

https://github.com/lsst/meas_mosaic/pull/51

Show
John Parejko added a comment - I believe this PR fixes this bug. None of the tests here can confirm it though. https://github.com/lsst/meas_mosaic/pull/51 Jenkins run: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29673/pipeline
 Reviewers Lauren MacArthur [ lauren ] Status In Progress [ 3 ] In Review [ 10004 ]
Hide
Lauren MacArthur added a comment - - edited

Ok, I will run mosaic.py with the ticket branch on a tract and check the results...on Monday

Show
Lauren MacArthur added a comment - - edited Ok, I will run  mosaic.py with the ticket branch on a tract and check the results...on Monday
 Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_DM-19265.png [ 37647 ] Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_jointcal_w14_vs_w10.png [ 37648 ] Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_mm_w10_vs_w06.png [ 37649 ]
 Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_jointcal_w14_vs_w10.png [ 37648 ]
 Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_jointcal_w10_vs_w06.png [ 37650 ]
Hide
Lauren MacArthur added a comment - - edited

Ok, using your ticket branch, I ran:

 mosaic.py /datasets/hsc/repo --rerun RC/w_2019_14/DM-18300-sfm:private/lauren/DM-19265 --numCoresForRead=12 --diagnostics --diagDir=/scratch/lauren/DM-19265/mosaicDiag/ --id tract=9813 ccd=0..8^10..103 visit=1228..1232:2^1238..1248:2^19658..19662:2^19680..19684:2^19694..19698:2^19708..19712:2^30482..30504:2 filter=HSC-I 

The following is the comparison of this output to that of the w_2019_10 RC2 run for visit 1228 (meas_mosaic calibrations being applied here):

So the good news is that the Jacobian is back Puzzling, though, is that there is still some difference between this branch and the w_2019_10 RC2 run. In fact, this difference also exists (in spirit) for the w14 vs w10 jointcal calibrations:

Comparing w10 to w06 meas_mosaic looks like:

which again show similar – but not identical – differences. Here is the same comparison but between the w10 vs w06 jointcal calibrations:

So it seems there have been some subtle changes from weekly06-to-weekly10-to-weekly14 that slightly affect the uberCal results. I had a glance at the weekly changelogs and nothing in particular jumped out at me. Any ideas? If this is expected, then no further digging is required...otherwise we may need to figure out the source of these differences (however small?).

Show
Lauren MacArthur added a comment - - edited Ok, using your ticket branch, I ran: mosaic.py / datasets / hsc / repo - - rerun RC / w_2019_14 / DM - 18300 - sfm:private / lauren / DM - 19265 - - numCoresForRead = 12 - - diagnostics - - diagDir = / scratch / lauren / DM - 19265 / mosaicDiag / - - id tract = 9813 ccd = 0. . 8 ^ 10. . 103 visit = 1228. . 1232 : 2 ^ 1238. . 1248 : 2 ^ 19658. . 19662 : 2 ^ 19680. . 19684 : 2 ^ 19694. . 19698 : 2 ^ 19708. . 19712 : 2 ^ 30482. . 30504 : 2 filter = HSC - I The following is the comparison of this output to that of the w_2019_10 RC2 run for visit 1228 ( meas_mosaic calibrations being applied here): So the good news is that the Jacobian is back Puzzling, though, is that there is still some difference between this branch and the  w_2019_10 RC2 run. In fact, this difference also exists (in spirit) for the w14 vs w10 jointcal calibrations: Comparing w10 to w06 meas_mosaic looks like: which again show similar – but not identical – differences. Here is the same comparison but between the w10 vs w06 jointcal calibrations: So it seems there have been some subtle changes from weekly06-to-weekly10-to-weekly14 that slightly affect the uberCal results. I had a glance at the weekly changelogs and nothing in particular jumped out at me. Any ideas? If this is expected, then no further digging is required...otherwise we may need to figure out the source of these differences (however small?).
 Attachment compareVisit-v1228-diff_base_GaussianFlux-sky-gals_jointcal_w14_vs_w10.png [ 37651 ]
 Link This issue blocks DM-17743 [ DM-17743 ]
Hide
Robert Lupton added a comment -

Are the raw Gaussian mags identical?

Show
Robert Lupton added a comment - Are the raw Gaussian mags identical?
Hide
Lauren MacArthur added a comment - - edited

Yes:

Show
Lauren MacArthur added a comment - - edited Yes:
Hide
Lauren MacArthur added a comment -

Seeing as these residual differences are likely unrelated to this issue, I think you’re good to merge this fix.  I’ll file a ticket to look into the cause of said differences.

Show
Lauren MacArthur added a comment - Seeing as these residual differences are likely unrelated to this issue, I think you’re good to merge this fix.  I’ll file a ticket to look into the cause of said differences.
 Status In Review [ 10004 ] Reviewed [ 10101 ]
Hide
John Parejko added a comment -

Thanks: yes, lets deal with each bit separately.

Merged and done.

Show
John Parejko added a comment - Thanks: yes, lets deal with each bit separately. Merged and done.
 Resolution Done [ 10000 ] Status Reviewed [ 10101 ] Done [ 10002 ]
 Team Data Release Production [ 10301 ] Alert Production [ 10300 ]
 Epic Link DM-16742 [ 235428 ]

#### People

Assignee:
John Parejko
Reporter:
Lauren MacArthur
Reviewers:
Lauren MacArthur
Watchers:
Jim Bosch, John Parejko, Lauren MacArthur, Robert Lupton, Yusra AlSayyad