Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8811

validate_drp should do something reasonable when presented with unknown filtername

    Details

    • Type: Story
    • Status: Won't Fix
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: Validation
    • Labels:

      Description

      After the validate_base shift, validateDrp.py now requires a filter to be listed in the etc/metrics.yaml in order for it to run. If no match is found, it bails out.

      1. validateDrp.py should instead do "something reasonable". Likely this means picking 'r' with a clear message that it is doing so.
      2. validateDrp.py should successfully calculate all metrics that it can. Failing on one metric or with the ocnfiguration of same should not lead to crashing:

      [wmwv@lsst-dev01 tmp]$ validateDrp.py DATA/rerun/20161216
      CameraMapper WARN: Calibration root directory not found: DATA/rerun/20161216/CALIB
      CameraMapper WARN: Calibration root directory not found: DATA/rerun/20161216/CALIB
      1077 sources in ccd 16  visit 903986
      953 sources in ccd 24  visit 903988
      1174 sources in ccd 18  visit 903990
      1049 sources in ccd 23  visit 903986
      1140 sources in ccd 25  visit 903990
      932 sources in ccd 23  visit 903988
      916 sources in ccd 17  visit 903988
      778 sources in ccd 22  visit 903986
      683 sources in ccd 100  visit 903986
      938 sources in ccd 16  visit 903988
      Photometric scatter (median) - SNR > 100.0 : 6.9 mmag
      /nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/photerrmodel.py:74: RuntimeWarning: invalid value encountered in sqrt
        return np.sqrt(sigmaSq)
      Median value of the astrometric scatter - all magnitudes: 48.763 marcsec
      Astrometric scatter (median) - snr > 100.0 : 9.6 marcsec
      No stars found that are 199.0 arcmin--201.0 arcmin apart.
      Traceback (most recent call last):
        File "/nfs/home/wmwv/local/lsst/validate_drp/bin/validateDrp.py", line 95, in <module>
          validate.run(args.repo, **kwargs)
        File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/validate.py", line 104, in run
          **kwargs)
        File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/validate.py", line 242, in runOneFilter
          job=job, linkedBlobs=linkedBlobs)
        File "/nfs/home/wmwv/local/lsst/validate_drp/python/lsst/validate/drp/calcsrd/pf1.py", line 72, in __init__
          PA2.get_spec(spec_name, filter_name=self.filter_name)
        File "/nfs/home/wmwv/local/lsst/validate_base/python/lsst/validate/base/metric.py", line 346, in get_spec
          name, filter_name, self.name))
      RuntimeError: No PA2 spec found for name=stretch filter_name=HSC-I
      

      The spirit of validate_drp is that it should just run on a data repo and produce reasonable input. An additional level is providing specific comparison against LSST SRD-level performance requirements or specific other performance levels.

        Attachments

          Issue Links

            Activity

            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            This was specifically triggered by looking at HSC data, where filters are named things like 'HSC-R' and 'HSC-I'. While specific rules could be written for those filters, the general point remains. Reasonable defaults should exist and be used.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - This was specifically triggered by looking at HSC data, where filters are named things like 'HSC-R' and 'HSC-I'. While specific rules could be written for those filters, the general point remains. Reasonable defaults should exist and be used.
            Hide
            jsick Jonathan Sick added a comment -

            So we're forced to have filters for specifications because validate_drp needs to be able check metrics as specified in the SRD, which of course have filter dependence. An additional complication with PA2 is that it needs the (filter-dependent) specification level of PF1 in order to calculate it (no way around this; it's how PA2 in the SRD is written).

            What we can do is have a way of marking a specification level in a metrics.yaml as the default. (Perhaps by having a specification with no filter dependence; this would be used as a fall-back if the filter names do not match).

            In the meantime we should probably add HSC-* explicitly to the filter lists for certain specifications in metrics.yaml, given the importance of HSC as a DM-supported camera. That's much better than trying to be implicit and automagically mapping HSC-R to, say, r, behind the scenes.

            And you're of course right that the exception should be handled so that validate_drp keeps going even if there's a failure. We can probably add an exception handler wrapper around each measurement attempt.

            Show
            jsick Jonathan Sick added a comment - So we're forced to have filters for specifications because validate_drp needs to be able check metrics as specified in the SRD, which of course have filter dependence. An additional complication with PA2 is that it needs the (filter-dependent) specification level of PF1 in order to calculate it (no way around this; it's how PA2 in the SRD is written). What we can do is have a way of marking a specification level in a metrics.yaml as the default. (Perhaps by having a specification with no filter dependence; this would be used as a fall-back if the filter names do not match). In the meantime we should probably add HSC-* explicitly to the filter lists for certain specifications in metrics.yaml , given the importance of HSC as a DM-supported camera. That's much better than trying to be implicit and automagically mapping HSC-R to, say, r , behind the scenes. And you're of course right that the exception should be handled so that validate_drp keeps going even if there's a failure. We can probably add an exception handler wrapper around each measurement attempt.
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Yeah, I'm adding the HSC-* filters as part of DM-6328.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Yeah, I'm adding the HSC-* filters as part of DM-6328 .
            Hide
            wmwood-vasey Michael Wood-Vasey added a comment -

            Calculating pass/fail statistics of course requires specific thresholds, which are filter dependent.

            But calculating the simple metrics such as PA1, AM1 can be done, even if there isn't a level to compare them to.

            Dependent metrics such as PA2 would benefit from a reasonable default. I like the idea of a specification with no filter dependence as the default, with per-filter specifications providing overrides. I will leave it to you, Jonathan Sick, to design the format for this.

            Show
            wmwood-vasey Michael Wood-Vasey added a comment - Calculating pass/fail statistics of course requires specific thresholds, which are filter dependent. But calculating the simple metrics such as PA1, AM1 can be done, even if there isn't a level to compare them to. Dependent metrics such as PA2 would benefit from a reasonable default. I like the idea of a specification with no filter dependence as the default, with per-filter specifications providing overrides. I will leave it to you, Jonathan Sick , to design the format for this.
            Hide
            swinbank John Swinbank added a comment -

            Closing this as “Won't Fix”, on the basis that validate_drp is being replaced.

            Show
            swinbank John Swinbank added a comment - Closing this as “Won't Fix”, on the basis that validate_drp is being replaced.

              People

              • Assignee:
                Unassigned
                Reporter:
                wmwood-vasey Michael Wood-Vasey
                Watchers:
                John Swinbank, Jonathan Sick, Michael Wood-Vasey
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved: