Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-30674

Gen3 DC2 reprocessing with bps and w_2021_24 stack

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Logs:

      /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z

        Attachments

          Issue Links

          relates to

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30730 DC2 Reprocessing with w_2021_24 (gen2)

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30747 Confirm brighter fatter correction is now being applied in DC2/imSim processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-30812 Compare the data products of the gen2 vs. gen3 w_2021_24 DC2 runs up to Single Frame Processing

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-31070 Compare the data products of the gen2 vs. gen3 w_2021_28 DC2 runs up to Single Frame Processing

          • Won't Fix

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            madamow Monika Adamow added a comment -

             

            root INFO: Creating quantum graph took 1167.18 seconds
            		 
            root INFO: Reading quantum graph (/scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z/2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z.qgraph)
            		 
            root INFO: Reading quantum graph with 119243 nodes took 203.51 seconds
            		 
            root INFO: Run QuantumGraph file /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z/2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z.qgraph
            		 
            root INFO: Pre-transform steps took 1412.72 seconds
            		 
            root INFO: Creating Generic Workflow
            		 
             root INFO: Creating Generic Workflow took 2903.64 seconds
            		 
            root INFO: Generic Workflow name 2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z
            		 
            root INFO: Creating specific implementation of workflow
            		 
            root INFO: Creating specific implementation of workflow took 91.76 seconds
            		 
            root INFO: Total submission creation time = 4408.14
            		 
            Submit dir: /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z

             

            Show
            madamow Monika Adamow added a comment -   root INFO: Creating quantum graph took 1167.18 seconds root INFO: Reading quantum graph (/scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z/2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z.qgraph) root INFO: Reading quantum graph with 119243 nodes took 203.51 seconds root INFO: Run QuantumGraph file /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z/2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z.qgraph root INFO: Pre-transform steps took 1412.72 seconds root INFO: Creating Generic Workflow  root INFO: Creating Generic Workflow took 2903.64 seconds root INFO: Generic Workflow name 2.2i_runs_test-med-1_w_2021_24_DM-30674_20210610T143649Z root INFO: Creating specific implementation of workflow root INFO: Creating specific implementation of workflow took 91.76 seconds root INFO: Total submission creation time = 4408.14 Submit dir: /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T143649Z  
            Hide
            Permalink
            madamow Monika Adamow added a comment - - edited

            All isr quanta end with:

            RuntimeError: Must supply a kernel if config.doBrighterFatter=True.

            Fixed. Jim Bosch's comment from slack:

            For users of /repo/dc2: I've just redefined the 2.2i/calib collection to point to the new 2.2i/calib/DM-30694 curated-calibration collection instead of the old 2.2i/calib/PREOPS-301 collection, because the new one now has the new form of brighter-fatter correction. That also means that we've got two copies of all other curated calibrations (since those haven't changed), which is unfortunate but not something we have a solution for (maybe someday we'll unstore the datasets in the older collection, or even delete them).

            https://lsstc.slack.com/archives/C01FBUGM2CV/p1623349274498700

            Show
            madamow Monika Adamow added a comment - - edited All isr quanta end with: RuntimeError: Must supply a kernel if config.doBrighterFatter=True. Fixed. Jim Bosch's comment from slack: For users of  /repo/dc2 : I've just redefined the  2.2i/calib  collection to point to the new  2.2i/calib/ DM-30694  curated-calibration collection instead of the old  2.2i/calib/PREOPS-301  collection, because the new one now has the new form of brighter-fatter correction. That also means that we've got two copies of all other curated calibrations (since those haven't changed), which is unfortunate but not something we have a solution for (maybe someday we'll  unstore  the datasets in the older collection, or even delete them). https://lsstc.slack.com/archives/C01FBUGM2CV/p1623349274498700
            Hide
            Permalink
            madamow Monika Adamow added a comment -

            1347 total fails for imageDifference:

            • 983 "Templates constructed from multiple Tracts not yet supported"
            • 360  "Expected Failure: Insufficient Template Coverage"
            •     4 "Cannot find any objects suitable for KernelCandidacy"
            Show
            madamow Monika Adamow added a comment - 1347 total fails for imageDifference: 983 "Templates constructed from multiple Tracts not yet supported" 360  "Expected Failure: Insufficient Template Coverage"     4 "Cannot find any objects suitable for KernelCandidacy"
            Hide
            Permalink
            madamow Monika Adamow added a comment -

            1 calibrate job failed with

            (scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/jobs/calibrate/23920_calibrate_.2529898.err):

             

            INFO  2021-06-10T17:44:08.261-0500 calibrate.measurement (calibrate:{instrument: 'LSSTCam-imSim', detector: 51, visit: 306188, ...})(sfm.py:290)- Measuring 863 sources (772 parents, 91 children) 
            		 
            python: /software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/include/boost/gil/image_view.hpp:316: boost::gil::image_view<L>::x_iterator boost::gil::image_view<L>::x_at(boost::gil::image_view<L>::x_coord_t, boost::gil::image_view<L>::y_coord_t) const [with Loc = boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<int, boost::gil::layout<boost::mp11::mp_list<boost::gil::gray_color_t> > >*> >; boost::gil::image_view<L>::x_iterator = boost::gil::pixel<int, boost::gil::layout<boost::mp11::mp_list<boost::gil::gray_color_t> > >*; boost::gil::image_view<L>::x_coord_t = long int; boost::gil::image_view<L>::y_coord_t = long int]: Assertion `0 <= y && y < height()' failed.
            		 
            Caught signal 6, backtrace follows:
            		 
            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-6-g0bf7090+18535a8d22/lib/libutils.so(+0x26389) [0x2ac84b8eb389]
            		 
            /lib64/libc.so.6(+0x36450) [0x2ac8445fc450]
            		 
            /lib64/libc.so.6(gsignal+0x37) [0x2ac8445fc3d7]
            		 
            /lib64/libc.so.6(abort+0x148) [0x2ac8445fdac8]
            		 
            /lib64/libc.so.6(+0x2f1a6) [0x2ac8445f51a6]
            		 
            /lib64/libc.so.6(+0x2f252) [0x2ac8445f5252]
            		 
            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-4-gee9d251+bafe306c76/lib/libmeas_extensions_shapeHSM.so(void lsst::meas::extensions::shapeHSM::HsmMomentsAlgorithm::calculate<double>(lsst::afw::table::SourceRecord&, std::shared_ptr<lsst::afw::image::Image<double> > const&, std::shared_ptr<lsst::afw::image::Mask<int> > const&, lsst::geom::Box2I const&, lsst::geom::Point<double, 2> const&, int, float, bool, bool, bool) const+0x1390) [0x2ac8693e19c0]
            		 
            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-4-gee9d251+bafe306c76/lib/libmeas_extensions_shapeHSM.so(lsst::meas::extensions::shapeHSM::HsmPsfMomentsAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0x1c5) [0x2ac8693e1e45]
            		 
            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-13-g64487bb+2343269ff4/python/lsst/meas/base/algorithm.so(+0x17b22) [0x2ac86453eb22]
            		 
            /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-13-g64487bb+2343269ff4/python/lsst/meas/base/algorithm.so(+0x18b9b) [0x2ac86453fb9b]
            		 
            python(PyCFunction_Call+0x54) [0x556b884a7df4]
            		 
            python((null)+0x31e) [0x556b884b6f2e]
            		 
            python(_PyObject_MakeTpCall+0x31e) [0x556b884b6f2e]
            		 
            python(+0x1b269e) [0x556b8852e69e]
            		 
            python((null)+0x4ca3) [0x556b885508c3]
            		 
            python(_PyEval_EvalFrameDefault+0x4ca3) [0x556b885508c3]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python(+0x1b1f91) [0x556b8852df91]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(+0x1b1f91) [0x556b8852df91]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python(+0x1b2007) [0x556b8852e007]
            		 
            python((null)+0x1782) [0x556b8854d3a2]
            		 
            python(_PyEval_EvalFrameDefault+0x1782) [0x556b8854d3a2]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python(+0x1b2007) [0x556b8852e007]
            		 
            python((null)+0x1782) [0x556b8854d3a2]
            		 
            python(_PyEval_EvalFrameDefault+0x1782) [0x556b8854d3a2]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(+0x1b1f91) [0x556b8852df91]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(+0x1b254c) [0x556b8852e54c]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python((null)+0xa4b) [0x556b8854c66b]
            		 
            python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x378) [0x556b8852d8d8]
            		 
            python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8]
            		 
            python(+0x1b1f91) [0x556b8852df91]
            		 
            python(PyObject_Call+0x5e) [0x556b884a10be]
            		 
            python((null)+0x21c1) [0x556b8854dde1]
            		 
            python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python((null)+0x2c1) [0x556b8849bdf1]
            		 
            python(_PyObject_FastCallDict+0x2c1) [0x556b8849bdf1]
            		 
            python((null)+0x63) [0x556b884a6983]
            		 
            python(_PyObject_Call_Prepend+0x63) [0x556b884a6983]
            		 
            python(+0x181b99) [0x556b884fdb99]
            		 
            python((null)+0x31e) [0x556b884b6f2e]
            		 
            python(_PyObject_MakeTpCall+0x31e) [0x556b884b6f2e]
            		 
            python((null)+0x4f2e) [0x556b88550b4e]
            		 
            python(_PyEval_EvalFrameDefault+0x4f2e) [0x556b88550b4e]
            		 
            python((null)+0x1a6) [0x556b8852d706]
            		 
            python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706]
            		 
            python((null)+0x92f) [0x556b8854c54f]
            		 
            python(_PyEval_EvalFrameDefault+0x92f) [0x556b8854c54f]
            		 
            python((null)+0x2c3) [0x556b8852c503]
            		 
            python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503]
            		 
            python(PyEval_EvalCodeEx+0x39) [0x556b8852d559]
            		 
            python(PyEval_EvalCode+0x1b) [0x556b885d09ab]
            		 
            python(+0x254a43) [0x556b885d0a43]
            		 
            python(+0x26e6b3) [0x556b885ea6b3]
            		 
            python(+0x2735b2) [0x556b885ef5b2]
            		 
            python(PyRun_SimpleFileExFlags+0x1b2) [0x556b885ef792]
            		 
            python(Py_RunMain+0x36d) [0x556b885efd0d]
            		 
            python(Py_BytesMain+0x39) [0x556b885efec9]
            		 
            /lib64/libc.so.6((null)+0xf5) [0x2ac8445e8555]
            		 
            /lib64/libc.so.6(__libc_start_main+0xf5) [0x2ac8445e8555]
            		 
            python(+0x1e9369) [0x556b88565369]

             

            Show
            madamow Monika Adamow added a comment - 1 calibrate job failed with (scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/ DM-30674 /20210610T194537Z/jobs/calibrate/23920_calibrate_.2529898.err):   INFO  2021-06-10T17:44:08.261-0500 calibrate.measurement (calibrate:{instrument: 'LSSTCam-imSim', detector: 51, visit: 306188, ...})(sfm.py:290)- Measuring 863 sources (772 parents, 91 children)  python: /software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/include/boost/gil/image_view.hpp:316: boost::gil::image_view<L>::x_iterator boost::gil::image_view<L>::x_at(boost::gil::image_view<L>::x_coord_t, boost::gil::image_view<L>::y_coord_t) const [with Loc = boost::gil::memory_based_2d_locator<boost::gil::memory_based_step_iterator<boost::gil::pixel<int, boost::gil::layout<boost::mp11::mp_list<boost::gil::gray_color_t> > >*> >; boost::gil::image_view<L>::x_iterator = boost::gil::pixel<int, boost::gil::layout<boost::mp11::mp_list<boost::gil::gray_color_t> > >*; boost::gil::image_view<L>::x_coord_t = long int; boost::gil::image_view<L>::y_coord_t = long int]: Assertion `0 <= y && y < height()' failed. Caught signal 6, backtrace follows: /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/utils/21.0.0-6-g0bf7090+18535a8d22/lib/libutils.so(+0x26389) [0x2ac84b8eb389] /lib64/libc.so.6(+0x36450) [0x2ac8445fc450] /lib64/libc.so.6(gsignal+0x37) [0x2ac8445fc3d7] /lib64/libc.so.6(abort+0x148) [0x2ac8445fdac8] /lib64/libc.so.6(+0x2f1a6) [0x2ac8445f51a6] /lib64/libc.so.6(+0x2f252) [0x2ac8445f5252] /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-4-gee9d251+bafe306c76/lib/libmeas_extensions_shapeHSM.so(void lsst::meas::extensions::shapeHSM::HsmMomentsAlgorithm::calculate<double>(lsst::afw::table::SourceRecord&, std::shared_ptr<lsst::afw::image::Image<double> > const&, std::shared_ptr<lsst::afw::image::Mask<int> > const&, lsst::geom::Box2I const&, lsst::geom::Point<double, 2> const&, int, float, bool, bool, bool) const+0x1390) [0x2ac8693e19c0] /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_extensions_shapeHSM/21.0.0-4-gee9d251+bafe306c76/lib/libmeas_extensions_shapeHSM.so(lsst::meas::extensions::shapeHSM::HsmPsfMomentsAlgorithm::measure(lsst::afw::table::SourceRecord&, lsst::afw::image::Exposure<float, int, float> const&) const+0x1c5) [0x2ac8693e1e45] /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-13-g64487bb+2343269ff4/python/lsst/meas/base/algorithm.so(+0x17b22) [0x2ac86453eb22] /software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/meas_base/21.0.0-13-g64487bb+2343269ff4/python/lsst/meas/base/algorithm.so(+0x18b9b) [0x2ac86453fb9b] python(PyCFunction_Call+0x54) [0x556b884a7df4] python((null)+0x31e) [0x556b884b6f2e] python(_PyObject_MakeTpCall+0x31e) [0x556b884b6f2e] python(+0x1b269e) [0x556b8852e69e] python((null)+0x4ca3) [0x556b885508c3] python(_PyEval_EvalFrameDefault+0x4ca3) [0x556b885508c3] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python(+0x1b1f91) [0x556b8852df91] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(+0x1b1f91) [0x556b8852df91] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python(+0x1b2007) [0x556b8852e007] python((null)+0x1782) [0x556b8854d3a2] python(_PyEval_EvalFrameDefault+0x1782) [0x556b8854d3a2] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python(+0x1b2007) [0x556b8852e007] python((null)+0x1782) [0x556b8854d3a2] python(_PyEval_EvalFrameDefault+0x1782) [0x556b8854d3a2] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(+0x1b1f91) [0x556b8852df91] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(+0x1b254c) [0x556b8852e54c] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python((null)+0xa4b) [0x556b8854c66b] python(_PyEval_EvalFrameDefault+0xa4b) [0x556b8854c66b] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x378) [0x556b8852d8d8] python(_PyFunction_Vectorcall+0x378) [0x556b8852d8d8] python(+0x1b1f91) [0x556b8852df91] python(PyObject_Call+0x5e) [0x556b884a10be] python((null)+0x21c1) [0x556b8854dde1] python(_PyEval_EvalFrameDefault+0x21c1) [0x556b8854dde1] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python((null)+0x2c1) [0x556b8849bdf1] python(_PyObject_FastCallDict+0x2c1) [0x556b8849bdf1] python((null)+0x63) [0x556b884a6983] python(_PyObject_Call_Prepend+0x63) [0x556b884a6983] python(+0x181b99) [0x556b884fdb99] python((null)+0x31e) [0x556b884b6f2e] python(_PyObject_MakeTpCall+0x31e) [0x556b884b6f2e] python((null)+0x4f2e) [0x556b88550b4e] python(_PyEval_EvalFrameDefault+0x4f2e) [0x556b88550b4e] python((null)+0x1a6) [0x556b8852d706] python(_PyFunction_Vectorcall+0x1a6) [0x556b8852d706] python((null)+0x92f) [0x556b8854c54f] python(_PyEval_EvalFrameDefault+0x92f) [0x556b8854c54f] python((null)+0x2c3) [0x556b8852c503] python(_PyEval_EvalCodeWithName+0x2c3) [0x556b8852c503] python(PyEval_EvalCodeEx+0x39) [0x556b8852d559] python(PyEval_EvalCode+0x1b) [0x556b885d09ab] python(+0x254a43) [0x556b885d0a43] python(+0x26e6b3) [0x556b885ea6b3] python(+0x2735b2) [0x556b885ef5b2] python(PyRun_SimpleFileExFlags+0x1b2) [0x556b885ef792] python(Py_RunMain+0x36d) [0x556b885efd0d] python(Py_BytesMain+0x39) [0x556b885efec9] /lib64/libc.so.6((null)+0xf5) [0x2ac8445e8555] /lib64/libc.so.6(__libc_start_main+0xf5) [0x2ac8445e8555] python(+0x1e9369) [0x556b88565369]  
            Hide
            Permalink
            madamow Monika Adamow added a comment -

            2 writeSourceTable jobs

            /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/jobs/writeSourceTable/38098_writeSourceTable_.2543502.err
            		 
            /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/jobs/writeSourceTable/43034_writeSourceTable_.2543497.err

             

            failed with:

             

            INFO  2021-06-10T18:19:20.776-0500 writeSourceTable (writeSourceTable:{instrument: 'LSSTCam-imSim', detector: 40, visit: 242505, ...})(postprocess.py:330)- Generating parquet table from src catalog 242505040
            		 
            Traceback (most recent call last):
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1007, in _write_in_memory_to_artifact
            		 
                formatter.write(inMemoryDataset)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/formatters/parquet.py", line 197, in write
            		 
                _writeParquet(location.path, inMemoryDataset)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/formatters/parquet.py", line 170, in _writeParquet
            		 
                pq.write_table(table, path, compression="none")
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/pyarrow/parquet.py", line 1800, in write_table
            		 
                with ParquetWriter(
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/pyarrow/parquet.py", line 633, in __init__
            		 
                sink = self.file_handle = filesystem.open_output_stream(
            		 
              File "pyarrow/_fs.pyx", line 660, in pyarrow._fs.FileSystem.open_output_stream
            		 
              File "pyarrow/error.pxi", line 141, in pyarrow.lib.pyarrow_internal_check_status
            		 
              File "pyarrow/error.pxi", line 110, in pyarrow.lib.check_status
            		 
            OSError: [Errno 28] Failed to open local file '/repo/dc2/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/source/20221130/r/r_sim_1.4/242505/source_LSSTCam-imSim_r_r_sim_1_4_242505_R11_S11_2_2i_runs_test-med-1_w_2021_24_DM-30674_20210610T194537Z.parq'. Detail: [errno 28] No space left on device
            		 
             
            		 
            The above exception was the direct cause of the following exception:
            		 
             
            		 
            Traceback (most recent call last):
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/bin/pipetask", line 29, in <module>
            		 
                sys.exit(main())
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main
            		 
                return cli()
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 829, in __call__
            		 
                return self.main(*args, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main
            		 
                rv = self.invoke(ctx)
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke
            		 
                return _process_result(sub_ctx.command.invoke(sub_ctx))
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke
            		 
                return ctx.invoke(self.callback, **ctx.params)
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke
            		 
                return callback(*args, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func
            		 
                return f(get_current_context(), *args, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run
            		 
                script.run(qgraphObj=qgraph, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run
            		 
                f.runPipeline(qgraphObj, taskFactory, args)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 634, in runPipeline
            		 
                executor.execute(graph, butler)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 300, in execute
            		 
                self._executeQuantaInProcess(graph, butler)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 350, in _executeQuantaInProcess
            		 
                self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute
            		 
                self.runQuantum(task, quantum, taskDef, butler)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 276, in runQuantum
            		 
                task.runQuantum(butlerQC, inputRefs, outputRefs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_tasks/21.0.0-100-g362f0c00+bcfc530e34/python/lsst/pipe/tasks/postprocess.py", line 312, in runQuantum
            		 
                butlerQC.put(outputs, outputRefs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-25-gc280871+534e165131/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put
            		 
                self._put(valuesAttribute, refs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-25-gc280871+534e165131/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put
            		 
                butler.put(value, ref)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/core/utils.py", line 265, in inner
            		 
                return func(self, *args, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/_butler.py", line 893, in put
            		 
                self.datastore.put(obj, ref)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/core/utils.py", line 265, in inner
            		 
                return func(self, *args, **kwargs)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1614, in put
            		 
                storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)
            		 
              File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1009, in _write_in_memory_to_artifact
            		 
                raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} "
            		 
            RuntimeError: Failed to serialize dataset source@{instrument: 'LSSTCam-imSim', detector: 40, visit: 242505, ...}, sc=DataFrame] (id=6fdcc436-b746-42d6-b134-34c5d1bc45d1) of type <class 'pandas.core.frame.DataFrame'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/source/20221130/r/r_sim_1.4/242505/source_LSSTCam-imSim_r_r_sim_1_4_242505_R11_S11_2_2i_runs_test-med-1_w_2021_24_DM-30674_20210610T194537Z.parq

             

            Show
            madamow Monika Adamow added a comment - 2 writeSourceTable jobs /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/jobs/writeSourceTable/38098_writeSourceTable_.2543502.err /scratch/madamow/DC2/submit/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/jobs/writeSourceTable/43034_writeSourceTable_.2543497.err   failed with:   INFO  2021-06-10T18:19:20.776-0500 writeSourceTable (writeSourceTable:{instrument: 'LSSTCam-imSim', detector: 40, visit: 242505, ...})(postprocess.py:330)- Generating parquet table from src catalog 242505040 Traceback (most recent call last):   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1007, in _write_in_memory_to_artifact     formatter.write(inMemoryDataset)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/formatters/parquet.py", line 197, in write     _writeParquet(location.path, inMemoryDataset)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/formatters/parquet.py", line 170, in _writeParquet     pq.write_table(table, path, compression="none")   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/pyarrow/parquet.py", line 1800, in write_table     with ParquetWriter(   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/pyarrow/parquet.py", line 633, in __init__     sink = self.file_handle = filesystem.open_output_stream(   File "pyarrow/_fs.pyx", line 660, in pyarrow._fs.FileSystem.open_output_stream   File "pyarrow/error.pxi", line 141, in pyarrow.lib.pyarrow_internal_check_status   File "pyarrow/error.pxi", line 110, in pyarrow.lib.check_status OSError: [Errno 28] Failed to open local file '/repo/dc2/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/source/20221130/r/r_sim_1.4/242505/source_LSSTCam-imSim_r_r_sim_1_4_242505_R11_S11_2_2i_runs_test-med-1_w_2021_24_DM-30674_20210610T194537Z.parq'. Detail: [errno 28] No space left on device   The above exception was the direct cause of the following exception:   Traceback (most recent call last):   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/bin/pipetask", line 29, in <module>     sys.exit(main())   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/pipetask.py", line 43, in main     return cli()   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 829, in __call__     return self.main(*args, **kwargs)   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 782, in main     rv = self.invoke(ctx)   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1259, in invoke     return _process_result(sub_ctx.command.invoke(sub_ctx))   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 1066, in invoke     return ctx.invoke(self.callback, **ctx.params)   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/core.py", line 610, in invoke     return callback(*args, **kwargs)   File "/software/lsstsw/stack_20210520/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/click/decorators.py", line 21, in new_func     return f(get_current_context(), *args, **kwargs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 103, in run     script.run(qgraphObj=qgraph, **kwargs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cli/script/run.py", line 167, in run     f.runPipeline(qgraphObj, taskFactory, args)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 634, in runPipeline     executor.execute(graph, butler)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 300, in execute     self._executeQuantaInProcess(graph, butler)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/mpGraphExecutor.py", line 350, in _executeQuantaInProcess     self.quantumExecutor.execute(qnode.taskDef, qnode.quantum, butler)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 103, in execute     self.runQuantum(task, quantum, taskDef, butler)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/ctrl_mpexec/21.0.0-30-g82f2559+4e26309f57/python/lsst/ctrl/mpexec/singleQuantumExecutor.py", line 276, in runQuantum     task.runQuantum(butlerQC, inputRefs, outputRefs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_tasks/21.0.0-100-g362f0c00+bcfc530e34/python/lsst/pipe/tasks/postprocess.py", line 312, in runQuantum     butlerQC.put(outputs, outputRefs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-25-gc280871+534e165131/python/lsst/pipe/base/butlerQuantumContext.py", line 198, in put     self._put(valuesAttribute, refs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/pipe_base/21.0.0-25-gc280871+534e165131/python/lsst/pipe/base/butlerQuantumContext.py", line 98, in _put     butler.put(value, ref)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/core/utils.py", line 265, in inner     return func(self, *args, **kwargs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/_butler.py", line 893, in put     self.datastore.put(obj, ref)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/core/utils.py", line 265, in inner     return func(self, *args, **kwargs)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1614, in put     storedInfo = self._write_in_memory_to_artifact(inMemoryDataset, ref)   File "/software/lsstsw/stack_20210520/stack/miniconda3-py38_4.9.2-0.6.0/Linux64/daf_butler/21.0.0-96-g57da90f2+7b9a448d34/python/lsst/daf/butler/datastores/fileDatastore.py", line 1009, in _write_in_memory_to_artifact     raise RuntimeError(f"Failed to serialize dataset {ref} of type {type(inMemoryDataset)} " RuntimeError: Failed to serialize dataset source@{instrument: 'LSSTCam-imSim', detector: 40, visit: 242505, ...}, sc=DataFrame] (id=6fdcc436-b746-42d6-b134-34c5d1bc45d1) of type <class 'pandas.core.frame.DataFrame'> to location file:///repo/dc2/2.2i/runs/test-med-1/w_2021_24/DM-30674/20210610T194537Z/source/20221130/r/r_sim_1.4/242505/source_LSSTCam-imSim_r_r_sim_1_4_242505_R11_S11_2_2i_runs_test-med-1_w_2021_24_DM-30674_20210610T194537Z.parq  
            Hide
            Permalink
            madamow Monika Adamow added a comment -

            After resubmitting dag file, the two writeSourceTable jobs ended correctly

            Show
            madamow Monika Adamow added a comment - After resubmitting dag file, the two writeSourceTable jobs ended correctly

              People

              Assignee:
              madamow Monika Adamow
              Reporter:
              madamow Monika Adamow
              Watchers:
              Lauren MacArthur, Monika Adamow, Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.