Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-34375

w_2022_15 breaks (some) pipelines

    XMLWordPrintable

    Details

    • Story Points:
      0.1
    • Epic Link:
    • Sprint:
      DB_S22_12
    • Team:
      Data Access and Database
    • Urgent?:
      No

      Description

      The following runs fine on w_14 but fails as shown on w_15

      [mfl@lsst-devl03]$ pipetask run -b /repo/main -o LATISS/runs/quickLook -d "exposure.day_obs>20220307 AND instrument='LATISS'" -p $RAPID_ANALYSIS_DIR/pipelines/quickLook.yaml --skip-existing-in LATISS/runs/quickLook --clobber-outputs --no-versions -j 12
      		 
      lsst.ctrl.mpexec.cmdLineFwk INFO: QuantumGraph contains 87 quanta for 1 tasks, graph ID: '1649371392.469013-2253483'
      		 
      lsst.daf.butler.cli.utils ERROR: Caught an exception, details are in traceback:
      		 
      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      		 
          self.dialect.do_execute(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      		 
          cursor.execute(statement, parameters)
      		 
      psycopg2.errors.NotNullViolation: null value in column "component" violates not-null constraint
      		 
      DETAIL:  Failing row contains (LATISS/runs/quickLook/20220407T224315Z/quickLookTask_config/quic..., lsst.pipe.base.formatters.pexConfig.PexConfigFormatter, Config, null, null, 2433, d8c8e5e4-28e5-4cf9-9c28-e4327263a1ab).
      		 
      The above exception was the direct cause of the following exception:
      		 
      Traceback (most recent call last):
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g109a262d75+f8e7eff54a/python/lsst/ctrl/mpexec/cli/cmd/commands.py", line 130, in run
      		 
          script.run(qgraphObj=qgraph, **kwargs)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g109a262d75+f8e7eff54a/python/lsst/ctrl/mpexec/cli/script/run.py", line 187, in run
      		 
          f.runPipeline(qgraphObj, taskFactory, args)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g109a262d75+f8e7eff54a/python/lsst/ctrl/mpexec/cmdLineFwk.py", line 707, in runPipeline
      		 
          preExecInit.initialize(
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g109a262d75+f8e7eff54a/python/lsst/ctrl/mpexec/preExecInit.py", line 102, in initialize
      		 
          self.saveConfigs(graph)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/ctrl_mpexec/g109a262d75+f8e7eff54a/python/lsst/ctrl/mpexec/preExecInit.py", line 328, in saveConfigs
      		 
          self.butler.put(taskDef.config, configName, {})
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/core/utils.py", line 56, in inner
      		 
          return func(self, *args, **kwargs)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/_butler.py", line 1167, in put
      		 
          self.datastore.put(obj, ref)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/core/utils.py", line 56, in inner
      		 
          return func(self, *args, **kwargs)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/datastores/fileDatastore.py", line 2098, in put
      		 
          self._register_datasets(artifacts)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/datastores/genericDatastore.py", line 123, in _register_datasets
      		 
          self.addStoredItemInfo(expandedRefs, expandedItemInfos)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/datastores/fileDatastore.py", line 369, in addStoredItemInfo
      		 
          self._table.insert(*records)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/registry/opaque.py", line 73, in insert
      		 
          self._db.insert(self._table, *data)
      		 
        File "/software/lsstsw/stack_20220215/stack/miniconda3-py38_4.9.2-2.0.0/Linux64/daf_butler/gda25978692+f83c363f3e/python/lsst/daf/butler/registry/interfaces/_database.py", line 1421, in insert
      		 
          connection.execute(table.insert(), rows)
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1289, in execute
      		 
          return meth(self, multiparams, params, _EMPTY_EXECUTION_OPTS)
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/sql/elements.py", line 325, in _execute_on_connection
      		 
          return connection._execute_clauseelement(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1481, in _execute_clauseelement
      		 
          ret = self._execute_context(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1845, in _execute_context
      		 
          self._handle_dbapi_exception(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 2026, in _handle_dbapi_exception
      		 
          util.raise_(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/util/compat.py", line 207, in raise_
      		 
          raise exception
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/base.py", line 1802, in _execute_context
      		 
          self.dialect.do_execute(
      		 
        File "/software/lsstsw/stack_20220215/conda/miniconda3-py38_4.9.2/envs/lsst-scipipe/lib/python3.8/site-packages/sqlalchemy/engine/default.py", line 732, in do_execute
      		 
          cursor.execute(statement, parameters)
      		 
      sqlalchemy.exc.IntegrityError: (psycopg2.errors.NotNullViolation) null value in column "component" violates not-null constraint
      		 
      DETAIL:  Failing row contains (LATISS/runs/quickLook/20220407T224315Z/quickLookTask_config/quic..., lsst.pipe.base.formatters.pexConfig.PexConfigFormatter, Config, null, null, 2433, d8c8e5e4-28e5-4cf9-9c28-e4327263a1ab).
      		 
      [SQL: INSERT INTO main_20210215.file_datastore_records (dataset_id, path, formatter, storage_class, component, checksum, file_size) VALUES (%(dataset_id)s, %(path)s, %(formatter)s, %(storage_class)s, %(component)s, %(checksum)s, %(file_size)s)]
      		 
      [parameters: {'dataset_id': UUID('d8c8e5e4-28e5-4cf9-9c28-e4327263a1ab'), 'path': 'LATISS/runs/quickLook/20220407T224315Z/quickLookTask_config/quickLookTask_config_LATISS_runs_quickLook_20220407T224315Z.py', 'formatter': 'lsst.pipe.base.formatters.pexConfig.PexConfigFormatter', 'storage_class': 'Config', 'component': None, 'checksum': None, 'file_size': 2433}]
      		 
      (Background on this error at: https://sqlalche.me/e/14/gkpj)

        Attachments

          Issue Links

          is triggered by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-33489 Add unit tests for QuantumBackedButler

          • Done
          is triggering

          Improvement - An improvement or enhancement to an existing feature or task. DM-34377 Add support for postgresql testing for datastore

          • Done

            Activity

            Ascending order - Click to sort in descending order
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Datastore stores the dataset_it and the component in its file_datastore_records table and requires that those are the primary key. DM-33489 mistakenly removed the code that turned component=None into an equivalent placeholder string (since primary keys can't use NULL values).

            Show
            tjenness Tim Jenness added a comment - Datastore stores the dataset_it and the component in its file_datastore_records table and requires that those are the primary key. DM-33489 mistakenly removed the code that turned component=None into an equivalent placeholder string (since primary keys can't use NULL values).
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            I tested the patch with ci_hsc_gen3 and Postgres, it does fix the error.

            Show
            salnikov Andy Salnikov added a comment - I tested the patch with ci_hsc_gen3 and Postgres, it does fix the error.
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Thanks for quick fix.

            Show
            tjenness Tim Jenness added a comment - Thanks for quick fix.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Datastore stores the dataset_it and the component in its file_datastore_records table and requires that those are the primary key. DM-33489 mistakenly removed the code that turned component=None into an equivalent placeholder string (since primary keys can't use NULL values).

            I certainly knew all that, I guess I was just too happy to do a little code cleanup Should have asked you for a review too.

            Show
            salnikov Andy Salnikov added a comment - Datastore stores the dataset_it and the component in its file_datastore_records table and requires that those are the primary key. DM-33489 mistakenly removed the code that turned component=None into an equivalent placeholder string (since primary keys can't use NULL values). I certainly knew all that, I guess I was just too happy to do a little code cleanup Should have asked you for a review too.
            Hide
            Permalink
            mfisherlevine Merlin Fisher-Levine added a comment -

            I'm off to bed now for once, but if this really breaks all pipelines might we consider cutting a new weekly? Is that even possible? (I don't really know if you can reuse tags, for example). Anyway, just a thought/something to consider.

            Show
            mfisherlevine Merlin Fisher-Levine added a comment - I'm off to bed now for once, but if this really breaks all pipelines might we consider cutting a new weekly? Is that even possible? (I don't really know if you can reuse tags, for example). Anyway, just a thought/something to consider.
            Hide
            Permalink
            salnikov Andy Salnikov added a comment -

            Merged. Sorry for this terrible mess.

            Show
            salnikov Andy Salnikov added a comment - Merged. Sorry for this terrible mess.

              People

              Assignee:
              salnikov Andy Salnikov
              Reporter:
              mfisherlevine Merlin Fisher-Levine
              Reviewers:
              Tim Jenness
              Watchers:
              Andy Salnikov, Merlin Fisher-Levine, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.