Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30922

Jointcal should be able to handle None for input skyWcs and photoCalib

    XMLWordPrintable

    Details

    • Urgent?:
      No

      Description

      As per our discussion at the Jun 28th Metrics Meeting, we decided the easiest way for unrecoverable known types of SFM processing failures in getting the wcs or photocalib would be to set the calexp values to None. This will be handled in warping/coaddition with DM-30921. The hope was that jointcal could recover from these failures (it has been known to fix bad wcs's), but it's unclear if jointcal can recover if the input skyWcs is None (for example). If this is not a simple patch, it might be possible to write a separate simple SFM task that will infer the SFM astrometric solution of any uncalibrated detectors using the camera geometry + other detectors in the same visit.

        Attachments

          Activity

          Hide
          Parejkoj John Parejko added a comment -

          If the WCS failed, jointcal can't crossmatch the sources: it uses the catalog ra/dec positions for the crossmatch (which happens before the model is constructed). So we'd have to either put "faked" coordinates into the source tables that are reasonably close (using the DM-20188 approach, perhaps?) or rework jointcal to try to compute those object positions. I think using the DM-20188 to fill in the missing coordinates is probably the best one.

          jointcal also uses the input WCS to compute the common tangent point for each detector, so we'd need some kind of rough guess WCS anyway.

          If we were using the Affine Astrometry fitter as implemented in DM-20188 (and undergoing testing in DM-29514) for our single frame fitting, might we have fewer such failures in general? What are the classes of WCS failures? If that visit+detector is just bad (e.g. totally saturated, no sources), then there's nothing for jointcal to do and it should just fit that visit without that detector. I believe that approach works as-is: the existing tests include multiple detector+visit combinations that don't all have the same detectors per visit.

          Show
          Parejkoj John Parejko added a comment - If the WCS failed, jointcal can't crossmatch the sources: it uses the catalog ra/dec positions for the crossmatch (which happens before the model is constructed). So we'd have to either put "faked" coordinates into the source tables that are reasonably close (using the DM-20188 approach, perhaps?) or rework jointcal to try to compute those object positions. I think using the DM-20188 to fill in the missing coordinates is probably the best one. jointcal also uses the input WCS to compute the common tangent point for each detector, so we'd need some kind of rough guess WCS anyway. If we were using the Affine Astrometry fitter as implemented in DM-20188 (and undergoing testing in DM-29514 ) for our single frame fitting, might we have fewer such failures in general? What are the classes of WCS failures? If that visit+detector is just bad (e.g. totally saturated, no sources), then there's nothing for jointcal to do and it should just fit that visit without that detector. I believe that approach works as-is: the existing tests include multiple detector+visit combinations that don't all have the same detectors per visit.
          Hide
          erykoff Eli Rykoff added a comment -

          John Parejko Thanks for this detail! So I assumed this was the case that jointcal would use the initial WCS (or the coordinates) for the matching, but Lauren MacArthur had said that she's seen jointcal recover valid WCSs from totally wackadoo initial fits, but maybe they weren't so crazy as to be 10 arcseconds off. Or something else is going on.

          There are indeed a few of classes of WCS failures that we've seen with DC2 data. One of them is the type of failure that we'd like to be able to recover from (if we were running jointcal!) when the astrometry fitter just misses the mark due to either shallow (u-band) images with few stars; mismatches with the refcat; related issues typically at the edges of the FOV. I expect that these can be improved with knowledge of the camera model, or using full focal plane info in some capacity, etc. (But these are not going to be implemented on the DP0.2 timescale, I imagine). Another class of failure that we don't have in DC2 but happens in real data is that a detector is completely blown out by scattered light, airplanes, Sirius, etc. And these we don't expect jointcal to recover from if there are no sources to match!

          In both types of failure, though, our current plan to keep the workflow happy is to have the calibrate task write None for the wcs (and the initial photoCalib as well). And so I think the action item on this ticket is to have jointcal simply check for None for each of these and drop them on the floor. This will prevent jointcal from recovering the first class of recoverable failures, but to recover these would require a working initial wcs anyway which is not the job of jointcal.

          I'm happy to take this on (as was the original plan), provided you agree that it can be done entirely at the python layer.

          Show
          erykoff Eli Rykoff added a comment - John Parejko Thanks for this detail! So I assumed this was the case that jointcal would use the initial WCS (or the coordinates) for the matching, but Lauren MacArthur had said that she's seen jointcal recover valid WCSs from totally wackadoo initial fits, but maybe they weren't so crazy as to be 10 arcseconds off. Or something else is going on. There are indeed a few of classes of WCS failures that we've seen with DC2 data. One of them is the type of failure that we'd like to be able to recover from (if we were running jointcal!) when the astrometry fitter just misses the mark due to either shallow (u-band) images with few stars; mismatches with the refcat; related issues typically at the edges of the FOV. I expect that these can be improved with knowledge of the camera model, or using full focal plane info in some capacity, etc. (But these are not going to be implemented on the DP0.2 timescale, I imagine). Another class of failure that we don't have in DC2 but happens in real data is that a detector is completely blown out by scattered light, airplanes, Sirius, etc. And these we don't expect jointcal to recover from if there are no sources to match! In both types of failure, though, our current plan to keep the workflow happy is to have the calibrate task write None for the wcs (and the initial photoCalib as well). And so I think the action item on this ticket is to have jointcal simply check for None for each of these and drop them on the floor. This will prevent jointcal from recovering the first class of recoverable failures, but to recover these would require a working initial wcs anyway which is not the job of jointcal. I'm happy to take this on (as was the original plan), provided you agree that it can be done entirely at the python layer.
          Hide
          Parejkoj John Parejko added a comment -

          I'd like to see some more details of those recoveries, Lauren MacArthur. I think I recall that too, but I don't know any of the details. Jointcal needs the coordinates for the cross matches (initial crossmatch default is 1 arcsec), and I can't really come up with a situation in which a completely incorrect single frame WCS is replaced.

          I really think the DM-20188 affine astrometry fitter should be used more broadly: it's fitting a much simpler model, and if we have a decent distortion model in the CameraGeom, it should result in fewer failed fits, I hope. It's trivial to switch over to it in AstrometryTask.

          Implementing this approach should be trivial (in gen2, if there was no WCS there'd be no SourceCatalog, so that detector+visit would just not exist), I think the only question is whether there is anything in the SourceTableVisit parquet file or VisitSummary ExposureTable. I'd put a check in _make_one_input_data() for visitRecord.getWcs() is None and return None, then check for that and continue in the inner loop of _load_data(). If such data is not in VisitSummary and SourceTableVisit, then it should already not be included in the fit in gen3.

          I can maybe help you cook up a test case for this for gen3. We might have to write some modified summary tables though.

          Show
          Parejkoj John Parejko added a comment - I'd like to see some more details of those recoveries, Lauren MacArthur . I think I recall that too, but I don't know any of the details. Jointcal needs the coordinates for the cross matches (initial crossmatch default is 1 arcsec), and I can't really come up with a situation in which a completely incorrect single frame WCS is replaced. I really think the DM-20188 affine astrometry fitter should be used more broadly: it's fitting a much simpler model, and if we have a decent distortion model in the CameraGeom, it should result in fewer failed fits, I hope. It's trivial to switch over to it in AstrometryTask. Implementing this approach should be trivial (in gen2, if there was no WCS there'd be no SourceCatalog, so that detector+visit would just not exist), I think the only question is whether there is anything in the SourceTableVisit parquet file or VisitSummary ExposureTable. I'd put a check in _make_one_input_data() for visitRecord.getWcs() is None and return None , then check for that and continue in the inner loop of _load_data() . If such data is not in VisitSummary and SourceTableVisit , then it should already not be included in the fit in gen3. I can maybe help you cook up a test case for this for gen3. We might have to write some modified summary tables though.
          Hide
          lauren Lauren MacArthur added a comment - - edited

          Here are a couple of examples.

          Noting that the pink outlines are the raw WCSs, dashed teal outlines are the SFM WCSs and the dashed gray are jointcal WCSs:

          Look at CCD 16, in particular (and see DM-27868 for a few more details on this one):

          Look at CCD 100, in particular:

          Show
          lauren Lauren MacArthur added a comment - - edited Here are a couple of examples. Noting that the pink outlines are the raw WCSs, dashed teal outlines are the SFM WCSs and the dashed gray are jointcal WCSs: Look at CCD 16, in particular (and see DM-27868 for a few more details on this one): Look at CCD 100, in particular:

            People

            Assignee:
            Unassigned Unassigned
            Reporter:
            erykoff Eli Rykoff
            Watchers:
            Eli Rykoff, Jim Bosch, John Parejko, Lauren MacArthur
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:

                CI Builds

                No builds found.