Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22230

Different PipelineTask config values depending upon how run

    Details

    • Templates:
    • Story Points:
      2
    • Sprint:
      DB_F19_10
    • Team:
      Data Access and Database

      Description

      With w45 (and even a more current ticket branch of ctrl_mpexec: tickets/DM-22166), running ci_hsc_gen3 with -t/-C fails vs running successfully with pipeline yaml.

      The full output with full traceback is in the attached run_cmdline.out (generated by executing run_cmdline.sh after bootstrapping a ci_hsc gen3 repo). The error message is: RuntimeError: Unable to uniquely identify index for seeing 8.000000: [3, 4].

      With help from Andy Salnikov, the make*.sh scripts output config values for charImage, and they are different (ignoring line ordering). The output of the make*.sh scripts are in attached corresponding .out files. Also attached the QuantumGraph pickle files in case helpful.

      $ diff <(sort yaml.out) <(sort cmdline.out)
      3600c3600
      < config.catalogCalculation.plugins.names=['base_ClassificationExtendedness', 'base_FootprintArea']
      ---
      > config.catalogCalculation.plugins.names=['base_FootprintArea', 'base_ClassificationExtendedness']
      3688c3688
      < config.measureApCorr.allowFailure=['ext_convolved_ConvolvedFlux_0_3_3', 'ext_convolved_ConvolvedFlux_0_4_5', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_1_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_3_3_3', 'ext_convolved_ConvolvedFlux_3_4_5', 'ext_convolved_ConvolvedFlux_3_6_0', 'ext_convolved_ConvolvedFlux_0_kron', 'ext_convolved_ConvolvedFlux_1_kron', 'ext_convolved_ConvolvedFlux_2_kron', 'ext_convolved_ConvolvedFlux_3_kron']
      ---
      > config.measureApCorr.allowFailure=['ext_convolved_ConvolvedFlux_0_3_3', 'ext_convolved_ConvolvedFlux_0_4_5', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_1_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_3_3_3', 'ext_convolved_ConvolvedFlux_3_4_5', 'ext_convolved_ConvolvedFlux_3_6_0', 'ext_convolved_ConvolvedFlux_0_kron', 'ext_convolved_ConvolvedFlux_1_kron', 'ext_convolved_ConvolvedFlux_2_kron', 'ext_convolved_ConvolvedFlux_3_kron', 'ext_convolved_ConvolvedFlux_0_3_3', 'ext_convolved_ConvolvedFlux_0_4_5', 'ext_convolved_ConvolvedFlux_0_6_0', 'ext_convolved_ConvolvedFlux_1_3_3', 'ext_convolved_ConvolvedFlux_1_4_5', 'ext_convolved_ConvolvedFlux_1_6_0', 'ext_convolved_ConvolvedFlux_2_3_3', 'ext_convolved_ConvolvedFlux_2_4_5', 'ext_convolved_ConvolvedFlux_2_6_0', 'ext_convolved_ConvolvedFlux_3_3_3', 'ext_convolved_ConvolvedFlux_3_4_5', 'ext_convolved_ConvolvedFlux_3_6_0', 'ext_convolved_ConvolvedFlux_4_3_3', 'ext_convolved_ConvolvedFlux_4_4_5', 'ext_convolved_ConvolvedFlux_4_6_0', 'ext_convolved_ConvolvedFlux_0_kron', 'ext_convolved_ConvolvedFlux_1_kron', 'ext_convolved_ConvolvedFlux_2_kron', 'ext_convolved_ConvolvedFlux_3_kron', 'ext_convolved_ConvolvedFlux_4_kron']
      3830c3830
      < config.measurement.plugins['ext_convolved_ConvolvedFlux'].seeing=[3.5, 5.0, 6.5, 8.0]
      ---
      > config.measurement.plugins['ext_convolved_ConvolvedFlux'].seeing=[3.5, 5.0, 6.5, 8.0, 8.0]
      4207c4207
      < config.measurement.plugins.names=['base_CircularApertureFlux', 'ext_shapeHSM_HsmPsfMoments', 'base_PsfFlux', 'modelfit_CModel', 'modelfit_DoubleShapeletPsfApprox', 'base_GaussianFlux', 'base_PixelFlags', 'ext_photometryKron_KronFlux', 'ext_shapeHSM_HsmShapeRegauss', 'ext_convolved_ConvolvedFlux', 'ext_shapeHSM_HsmSourceMomentsRound', 'base_SdssShape', 'base_Jacobian', 'ext_shapeHSM_HsmSourceMoments', 'base_FPPosition', 'base_SdssCentroid']
      ---
      > =['ext_shapeHSM_HsmSourceMomentsRound', 'base_PsfFlux', 'modelfit_DoubleShapeletPsfApprox', 'base_FPPosition', 'ext_shapeHSM_HsmShapeRegauss', 'base_PixelFlags', 'base_CircularApertureFlux', 'ext_convolved_ConvolvedFlux', 'modelfit_CModel', 'ext_photometryKron_KronFlux', 'base_SdssShape', 'base_GaussianFlux', 'ext_shapeHSM_HsmPsfMoments', 'base_SdssCentroid', 'ext_shapeHSM_HsmSourceMoments', 'base_Jacobian']
      

      I think config.measurement.plugins.names is the same (just different order), but the other "array" outputs listed in diff have more values in the cmdline -t-C version.

        Attachments

        1. cmdline_qgraph.pickle
          1.45 MB
        2. cmdline.out
          364 kB
        3. make_cmdline.sh
          2 kB
        4. make_yaml.sh
          0.3 kB
        5. run_cmdline.out
          4 kB
        6. run_cmdline.sh
          2 kB
        7. yaml_qgraph.pickle
          1.45 MB
        8. yaml.out
          364 kB

          Issue Links

            Activity

              People

              • Assignee:
                salnikov Andy Salnikov
                Reporter:
                mgower Michelle Gower
                Watchers:
                Andy Salnikov, Jim Bosch, Michelle Gower
              • Votes:
                0 Vote for this issue
                Watchers:
                3 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel