Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17055

Change Sigmas to Err in dax_ppdb schemas

    Details

      Description

      Some of the columns names in dax_ppdb still has several columns that were not converted by myself from Sigma to Err. This ticket will convert the column names to the expected Err name.

        Attachments

          Activity

          Show
          cmorrison Chris Morrison added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29258/pipeline
          Hide
          salnikov Andy Salnikov added a comment - - edited

          Jenkins build failed in unit test (https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log):

          [2019-01-16T01:04:24.849210Z]     def test_getAfwSchemaWithExtras(self):
          [2019-01-16T01:04:24.849214Z]         """Test for getAfwSchema method using extra afw schemas.
          [2019-01-16T01:04:24.849218Z]     
          [2019-01-16T01:04:24.849221Z]             Same as above but use non-default afw schemas, this adds few extra
          [2019-01-16T01:04:24.849226Z]             columns to the table schema
          [2019-01-16T01:04:24.849230Z]             """
          [2019-01-16T01:04:24.849233Z]         engine = create_engine('sqlite://')
          [2019-01-16T01:04:24.849236Z]     
          [2019-01-16T01:04:24.849240Z]         # create standard (baseline) schema, but use afw column map
          [2019-01-16T01:04:24.849243Z]         afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(),
          [2019-01-16T01:04:24.849246Z]                            DiaSource=make_minimal_dia_source_schema())
          [2019-01-16T01:04:24.849250Z]         schema = PpdbSchema(engine=engine,
          [2019-01-16T01:04:24.849253Z]                             dia_object_index="baseline",
          [2019-01-16T01:04:24.849256Z]                             dia_object_nightly=False,
          [2019-01-16T01:04:24.849260Z]                             schema_file=_data_file_name("ppdb-schema.yaml"),
          [2019-01-16T01:04:24.849263Z]                             column_map=_data_file_name("ppdb-afw-map.yaml"),
          [2019-01-16T01:04:24.849266Z]                             afw_schemas=afw_schemas)
          [2019-01-16T01:04:24.849270Z]         schema.makeSchema()
          [2019-01-16T01:04:24.849274Z]     
          [2019-01-16T01:04:24.849278Z]         afw_schema, col_map = schema.getAfwSchema("DiaObject")
          [2019-01-16T01:04:24.849281Z]         self.assertEqual(len(col_map), 94)
          [2019-01-16T01:04:24.849284Z]         self.assertIsInstance(afw_schema, afwTable.Schema)
          [2019-01-16T01:04:24.849288Z]         # no BLOBs in afwTable, so count is lower
          [2019-01-16T01:04:24.849291Z]         self.assertEqual(afw_schema.getFieldCount(), 82)
          [2019-01-16T01:04:24.849295Z]     
          [2019-01-16T01:04:24.849298Z]         afw_schema, col_map = schema.getAfwSchema("DiaSource")
          [2019-01-16T01:04:24.849301Z] >       self.assertEqual(len(col_map), 109)
          [2019-01-16T01:04:24.849305Z] E       AssertionError: 108 != 109
          

          I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr. Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?

          Show
          salnikov Andy Salnikov added a comment - - edited Jenkins build failed in unit test ( https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log): [2019-01-16T01:04:24.849210Z] def test_getAfwSchemaWithExtras(self): [2019-01-16T01:04:24.849214Z] """Test for getAfwSchema method using extra afw schemas. [2019-01-16T01:04:24.849218Z] [2019-01-16T01:04:24.849221Z] Same as above but use non-default afw schemas, this adds few extra [2019-01-16T01:04:24.849226Z] columns to the table schema [2019-01-16T01:04:24.849230Z] """ [2019-01-16T01:04:24.849233Z] engine = create_engine('sqlite://') [2019-01-16T01:04:24.849236Z] [2019-01-16T01:04:24.849240Z] # create standard (baseline) schema, but use afw column map [2019-01-16T01:04:24.849243Z] afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(), [2019-01-16T01:04:24.849246Z] DiaSource=make_minimal_dia_source_schema()) [2019-01-16T01:04:24.849250Z] schema = PpdbSchema(engine=engine, [2019-01-16T01:04:24.849253Z] dia_object_index="baseline", [2019-01-16T01:04:24.849256Z] dia_object_nightly=False, [2019-01-16T01:04:24.849260Z] schema_file=_data_file_name("ppdb-schema.yaml"), [2019-01-16T01:04:24.849263Z] column_map=_data_file_name("ppdb-afw-map.yaml"), [2019-01-16T01:04:24.849266Z] afw_schemas=afw_schemas) [2019-01-16T01:04:24.849270Z] schema.makeSchema() [2019-01-16T01:04:24.849274Z] [2019-01-16T01:04:24.849278Z] afw_schema, col_map = schema.getAfwSchema("DiaObject") [2019-01-16T01:04:24.849281Z] self.assertEqual(len(col_map), 94) [2019-01-16T01:04:24.849284Z] self.assertIsInstance(afw_schema, afwTable.Schema) [2019-01-16T01:04:24.849288Z] # no BLOBs in afwTable, so count is lower [2019-01-16T01:04:24.849291Z] self.assertEqual(afw_schema.getFieldCount(), 82) [2019-01-16T01:04:24.849295Z] [2019-01-16T01:04:24.849298Z] afw_schema, col_map = schema.getAfwSchema("DiaSource") [2019-01-16T01:04:24.849301Z] > self.assertEqual(len(col_map), 109) [2019-01-16T01:04:24.849305Z] E AssertionError: 108 != 109 I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr . Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?
          Hide
          salnikov Andy Salnikov added a comment - - edited

          Looks good after unit test fix.

          Show
          salnikov Andy Salnikov added a comment - - edited Looks good after unit test fix.
          Hide
          cmorrison Chris Morrison added a comment -

          Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.

          Show
          cmorrison Chris Morrison added a comment - Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.
          Show
          cmorrison Chris Morrison added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29259/pipeline
          Hide
          salnikov Andy Salnikov added a comment -

          Looks OK too (I guess it's sufficiently tested now )

          Show
          salnikov Andy Salnikov added a comment - Looks OK too (I guess it's sufficiently tested now )
          Hide
          cmorrison Chris Morrison added a comment -

          Linux Jenkins has pass. Will now merge and close ticket.

          Show
          cmorrison Chris Morrison added a comment - Linux Jenkins has pass. Will now merge and close ticket.

            People

            • Assignee:
              cmorrison Chris Morrison
              Reporter:
              cmorrison Chris Morrison
              Reviewers:
              Andy Salnikov
              Watchers:
              Andy Salnikov, Chris Morrison, John Swinbank
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel