Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-15857

jointcal of w_2018_38 fails to run

    Details

    • Story Points:
      2
    • Sprint:
      AP F18-4, AP F18-5
    • Team:
      Alert Production

      Description

      Running w_2018_38 jointcal with HSC-RC2 data failed, as follows:

      INFO  2018-09-25T01:02:15.015 jointcal.Associations ()(src/Associations.cc:141)- Unmatched objects: 95
      FATAL 2018-09-25T01:02:29.029 jointcal ()(jointcal.py:103)- Failed processing tract 9615, TypeError: _basicget(): incompatible function arguments. The following argument types are supported:
          1. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyB) -> ndarray::Array<unsigned char, 1, 0>
          2. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyU) -> ndarray::Array<unsigned short, 1, 0>
          3. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyI) -> ndarray::Array<int, 1, 0>
          4. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyL) -> ndarray::Array<long, 1, 0>
          5. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyF) -> ndarray::Array<float, 1, 0>
          6. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyD) -> ndarray::Array<double, 1, 0>
          7. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyFlag) -> ndarray::Array<bool const, 1, 1>
          8. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyArrayB) -> ndarray::Array<unsigned char, 2, 1>
          9. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyArrayU) -> ndarray::Array<unsigned short, 2, 1>
          10. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyArrayI) -> ndarray::Array<int, 2, 1>
          11. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyArrayF) -> ndarray::Array<float, 2, 1>
          12. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyArrayD) -> ndarray::Array<double, 2, 1>
          13. (self: lsst.afw.table.baseColumnView.baseColumnView._BaseColumnViewBase, arg0: lsst.afw.table.schema.schema.KeyAngle) -> ndarray::Array<double, 1, 0>
       
      Invoked with: <lsst.afw.table.simple.simple.SimpleColumnView object at 0x7f41807f77d8>, None
      

      The command I ran to get the above error was:

      jointcal.py /datasets/hsc/repo --calib /datasets/hsc/repo/CALIB --rerun RC/w_2018_38/DM-15690:private/your-user-name/w38jointcal --id ccd=0..8^10..103 visit=380^384^388^404^408^424^426^436^440 filter=HSC-Y tract=9615 -C=/home/hchiang2/weeklies/w38/w38-jointcal-config.py 
      

      The config override file has:

      config.astrometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
      config.photometryRefObjLoader.ref_dataset_name='ps1_pv3_3pi_20170110'
      config.astrometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
      config.photometryRefObjLoader.filterMap={'B': 'g', 'r2': 'r', 'N1010': 'z', 'N816': 'i', 'I': 'i', 'N387': 'g', 'i2': 'i', 'R': 'r', 'N921': 'z', 'N515': 'g', 'V': 'r'}
      

        Attachments

          Issue Links

            Activity

            Hide
            Parejkoj John Parejko added a comment -

            Thanks Jim Bosch. Jenkins passed: I hope we got everything this time!

            Merged and done.

            Show
            Parejkoj John Parejko added a comment - Thanks Jim Bosch . Jenkins passed: I hope we got everything this time! Merged and done.
            Hide
            lauren Lauren MacArthur added a comment -

            John Parejko and I are now in the process of testing a very promising new idea: only add the {{instFlux}}aliases when the old field has units of "count" (or "dn", or "adu", just in case).  Happily we seem to be quite good about doing that where we should.

            I fear I may have just uncovered an important exception to the above. No units for:

            (Field['D'](name="modelfit_CModel_flux", doc="flux from the final cmodel fit"), Key<D>(offset=984, nElements=1))
            (Field['D'](name="modelfit_CModel_fluxErr", doc="flux uncertainty from the final cmodel fit"), Key<D>(offset=992, nElements=1))
            

            So I'm getting

            Field or subfield with name 'modelfit_CModel_instFlux' not found with type 'D'. {0}
            lsst::pex::exceptions::NotFoundError: 'Field or subfield with name 'modelfit_CModel_instFlux' not found with type 'D'.'
            

            with a simple

            butler.get("deepCoadd_forced_src", dataIdI, immediate=True)
            

            Note that this is for w_2018_40 (I'm using Eli's stack at /project/erykoff/stack/)

            Show
            lauren Lauren MacArthur added a comment - John Parejko and I are now in the process of testing a very promising new idea: only add the {{instFlux}}aliases when the old field has units of "count" (or "dn", or "adu", just in case).  Happily we seem to be quite good about doing that where we should. I fear I may have just uncovered an important exception to the above. No units for: (Field[ 'D' ](name= "modelfit_CModel_flux" , doc= "flux from the final cmodel fit" ), Key<D>(offset= 984 , nElements= 1 )) (Field[ 'D' ](name= "modelfit_CModel_fluxErr" , doc= "flux uncertainty from the final cmodel fit" ), Key<D>(offset= 992 , nElements= 1 )) So I'm getting Field or subfield with name 'modelfit_CModel_instFlux' not found with type 'D' . { 0 } lsst::pex::exceptions::NotFoundError: 'Field or subfield with name ' modelfit_CModel_instFlux ' not found with type ' D '.' with a simple butler.get( "deepCoadd_forced_src" , dataIdI, immediate=True) Note that this is for w_2018_40 (I'm using Eli's stack at /project/erykoff/stack/ )
            Hide
            lauren Lauren MacArthur added a comment -

            A few more cases (but not a thorough search):

            (Field['D'](name="modelfit_CModel_initial_flux_inner", doc="flux within the fit region, with no extrapolation"), Key<D>(offset=928, nElements=1))
            (Field['D'](name="modelfit_CModel_dev_flux_inner", doc="flux within the fit region, with no extrapolation"), Key<D>(offset=976, nElements=1))
            (Field['D'](name="modelfit_CModel_flux_inner", doc="flux within the fit region, with no extrapolation"), Key<D>(offset=1000, nElements=1))
            

             

            Show
            lauren Lauren MacArthur added a comment - A few more cases (but not a thorough search): (Field[ 'D' ](name= "modelfit_CModel_initial_flux_inner" , doc= "flux within the fit region, with no extrapolation" ), Key<D>(offset= 928 , nElements= 1 )) (Field[ 'D' ](name= "modelfit_CModel_dev_flux_inner" , doc= "flux within the fit region, with no extrapolation" ), Key<D>(offset= 976 , nElements= 1 )) (Field[ 'D' ](name= "modelfit_CModel_flux_inner" , doc= "flux within the fit region, with no extrapolation" ), Key<D>(offset= 1000 , nElements= 1 ))  
            Hide
            Parejkoj John Parejko added a comment -

            Please file a new ticket for those. I'm not sure how to handle that, but it does strongly suggest that we need to fix modelfit to output units!

            Show
            Parejkoj John Parejko added a comment - Please file a new ticket for those. I'm not sure how to handle that, but it does strongly suggest that we need to fix modelfit to output units!
            Hide
            lauren Lauren MacArthur added a comment -

            See DM-16068.

            Show
            lauren Lauren MacArthur added a comment - See DM-16068 .

              People

              • Assignee:
                Parejkoj John Parejko
                Reporter:
                hchiang2 Hsin-Fang Chiang
                Reviewers:
                John Parejko
                Watchers:
                Eli Rykoff, Hsin-Fang Chiang, Jim Bosch, John Parejko, Lauren MacArthur, Russell Owen, Yusra AlSayyad
              • Votes:
                0 Vote for this issue
                Watchers:
                7 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel