Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17055

Change Sigmas to Err in dax_ppdb schemas

    XMLWordPrintable

    Details

      Description

      Some of the columns names in dax_ppdb still has several columns that were not converted by myself from Sigma to Err. This ticket will convert the column names to the expected Err name.

        Attachments

          Activity

          Ascending order - Click to sort in descending order
          Hide
          Permalink
          cmorrison Chris Morrison [X] (Inactive) added a comment -

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29258/pipeline

          Show
          cmorrison Chris Morrison [X] (Inactive) added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29258/pipeline
          Hide
          Permalink
          salnikov Andy Salnikov added a comment - - edited

          Jenkins build failed in unit test (https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log):

          [2019-01-16T01:04:24.849210Z]     def test_getAfwSchemaWithExtras(self):
          		 
          [2019-01-16T01:04:24.849214Z]         """Test for getAfwSchema method using extra afw schemas.
          		 
          [2019-01-16T01:04:24.849218Z]     
          [2019-01-16T01:04:24.849221Z]             Same as above but use non-default afw schemas, this adds few extra
          		 
          [2019-01-16T01:04:24.849226Z]             columns to the table schema
          		 
          [2019-01-16T01:04:24.849230Z]             """
          		 
          [2019-01-16T01:04:24.849233Z]         engine = create_engine('sqlite://')
          		 
          [2019-01-16T01:04:24.849236Z]     
          [2019-01-16T01:04:24.849240Z]         # create standard (baseline) schema, but use afw column map
          		 
          [2019-01-16T01:04:24.849243Z]         afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(),
          		 
          [2019-01-16T01:04:24.849246Z]                            DiaSource=make_minimal_dia_source_schema())
          		 
          [2019-01-16T01:04:24.849250Z]         schema = PpdbSchema(engine=engine,
          		 
          [2019-01-16T01:04:24.849253Z]                             dia_object_index="baseline",
          		 
          [2019-01-16T01:04:24.849256Z]                             dia_object_nightly=False,
          		 
          [2019-01-16T01:04:24.849260Z]                             schema_file=_data_file_name("ppdb-schema.yaml"),
          		 
          [2019-01-16T01:04:24.849263Z]                             column_map=_data_file_name("ppdb-afw-map.yaml"),
          		 
          [2019-01-16T01:04:24.849266Z]                             afw_schemas=afw_schemas)
          		 
          [2019-01-16T01:04:24.849270Z]         schema.makeSchema()
          		 
          [2019-01-16T01:04:24.849274Z]     
          [2019-01-16T01:04:24.849278Z]         afw_schema, col_map = schema.getAfwSchema("DiaObject")
          		 
          [2019-01-16T01:04:24.849281Z]         self.assertEqual(len(col_map), 94)
          		 
          [2019-01-16T01:04:24.849284Z]         self.assertIsInstance(afw_schema, afwTable.Schema)
          		 
          [2019-01-16T01:04:24.849288Z]         # no BLOBs in afwTable, so count is lower
          		 
          [2019-01-16T01:04:24.849291Z]         self.assertEqual(afw_schema.getFieldCount(), 82)
          		 
          [2019-01-16T01:04:24.849295Z]     
          [2019-01-16T01:04:24.849298Z]         afw_schema, col_map = schema.getAfwSchema("DiaSource")
          		 
          [2019-01-16T01:04:24.849301Z] >       self.assertEqual(len(col_map), 109)
          		 
          [2019-01-16T01:04:24.849305Z] E       AssertionError: 108 != 109

          I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr. Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?

          Show
          salnikov Andy Salnikov added a comment - - edited Jenkins build failed in unit test ( https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log): [2019-01-16T01:04:24.849210Z] def test_getAfwSchemaWithExtras(self): [2019-01-16T01:04:24.849214Z] """Test for getAfwSchema method using extra afw schemas. [2019-01-16T01:04:24.849218Z] [2019-01-16T01:04:24.849221Z] Same as above but use non-default afw schemas, this adds few extra [2019-01-16T01:04:24.849226Z] columns to the table schema [2019-01-16T01:04:24.849230Z] """ [2019-01-16T01:04:24.849233Z] engine = create_engine('sqlite://') [2019-01-16T01:04:24.849236Z] [2019-01-16T01:04:24.849240Z] # create standard (baseline) schema, but use afw column map [2019-01-16T01:04:24.849243Z] afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(), [2019-01-16T01:04:24.849246Z] DiaSource=make_minimal_dia_source_schema()) [2019-01-16T01:04:24.849250Z] schema = PpdbSchema(engine=engine, [2019-01-16T01:04:24.849253Z] dia_object_index="baseline", [2019-01-16T01:04:24.849256Z] dia_object_nightly=False, [2019-01-16T01:04:24.849260Z] schema_file=_data_file_name("ppdb-schema.yaml"), [2019-01-16T01:04:24.849263Z] column_map=_data_file_name("ppdb-afw-map.yaml"), [2019-01-16T01:04:24.849266Z] afw_schemas=afw_schemas) [2019-01-16T01:04:24.849270Z] schema.makeSchema() [2019-01-16T01:04:24.849274Z] [2019-01-16T01:04:24.849278Z] afw_schema, col_map = schema.getAfwSchema("DiaObject") [2019-01-16T01:04:24.849281Z] self.assertEqual(len(col_map), 94) [2019-01-16T01:04:24.849284Z] self.assertIsInstance(afw_schema, afwTable.Schema) [2019-01-16T01:04:24.849288Z] # no BLOBs in afwTable, so count is lower [2019-01-16T01:04:24.849291Z] self.assertEqual(afw_schema.getFieldCount(), 82) [2019-01-16T01:04:24.849295Z] [2019-01-16T01:04:24.849298Z] afw_schema, col_map = schema.getAfwSchema("DiaSource") [2019-01-16T01:04:24.849301Z] > self.assertEqual(len(col_map), 109) [2019-01-16T01:04:24.849305Z] E AssertionError: 108 != 109 I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr . Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?
          Hide
          Permalink
          salnikov Andy Salnikov added a comment - - edited

          Looks good after unit test fix.

          Show
          salnikov Andy Salnikov added a comment - - edited Looks good after unit test fix.
          Hide
          Permalink
          cmorrison Chris Morrison [X] (Inactive) added a comment -

          Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.

          Show
          cmorrison Chris Morrison [X] (Inactive) added a comment - Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.
          Hide
          Permalink
          cmorrison Chris Morrison [X] (Inactive) added a comment -

          Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29259/pipeline

          Show
          cmorrison Chris Morrison [X] (Inactive) added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29259/pipeline
          Hide
          Permalink
          salnikov Andy Salnikov added a comment -

          Looks OK too (I guess it's sufficiently tested now )

          Show
          salnikov Andy Salnikov added a comment - Looks OK too (I guess it's sufficiently tested now )
          Hide
          Permalink
          cmorrison Chris Morrison [X] (Inactive) added a comment -

          Linux Jenkins has pass. Will now merge and close ticket.

          Show
          cmorrison Chris Morrison [X] (Inactive) added a comment - Linux Jenkins has pass. Will now merge and close ticket.

            People

            Assignee:
            cmorrison Chris Morrison [X] (Inactive)
            Reporter:
            cmorrison Chris Morrison [X] (Inactive)
            Reviewers:
            Andy Salnikov
            Watchers:
            Andy Salnikov, Chris Morrison [X] (Inactive), John Swinbank
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:

                Jenkins

                No builds found.