Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17055

Change Sigmas to Err in dax_ppdb schemas

    XMLWordPrintable

Details

    Description

      Some of the columns names in dax_ppdb still has several columns that were not converted by myself from Sigma to Err. This ticket will convert the column names to the expected Err name.

      Attachments

        Activity

          cmorrison Chris Morrison [X] (Inactive) added a comment - Jenkins:  https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29258/pipeline
          salnikov Andy Salnikov added a comment - - edited

          Jenkins build failed in unit test (https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log):

          [2019-01-16T01:04:24.849210Z]     def test_getAfwSchemaWithExtras(self):
          [2019-01-16T01:04:24.849214Z]         """Test for getAfwSchema method using extra afw schemas.
          [2019-01-16T01:04:24.849218Z]     
          [2019-01-16T01:04:24.849221Z]             Same as above but use non-default afw schemas, this adds few extra
          [2019-01-16T01:04:24.849226Z]             columns to the table schema
          [2019-01-16T01:04:24.849230Z]             """
          [2019-01-16T01:04:24.849233Z]         engine = create_engine('sqlite://')
          [2019-01-16T01:04:24.849236Z]     
          [2019-01-16T01:04:24.849240Z]         # create standard (baseline) schema, but use afw column map
          [2019-01-16T01:04:24.849243Z]         afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(),
          [2019-01-16T01:04:24.849246Z]                            DiaSource=make_minimal_dia_source_schema())
          [2019-01-16T01:04:24.849250Z]         schema = PpdbSchema(engine=engine,
          [2019-01-16T01:04:24.849253Z]                             dia_object_index="baseline",
          [2019-01-16T01:04:24.849256Z]                             dia_object_nightly=False,
          [2019-01-16T01:04:24.849260Z]                             schema_file=_data_file_name("ppdb-schema.yaml"),
          [2019-01-16T01:04:24.849263Z]                             column_map=_data_file_name("ppdb-afw-map.yaml"),
          [2019-01-16T01:04:24.849266Z]                             afw_schemas=afw_schemas)
          [2019-01-16T01:04:24.849270Z]         schema.makeSchema()
          [2019-01-16T01:04:24.849274Z]     
          [2019-01-16T01:04:24.849278Z]         afw_schema, col_map = schema.getAfwSchema("DiaObject")
          [2019-01-16T01:04:24.849281Z]         self.assertEqual(len(col_map), 94)
          [2019-01-16T01:04:24.849284Z]         self.assertIsInstance(afw_schema, afwTable.Schema)
          [2019-01-16T01:04:24.849288Z]         # no BLOBs in afwTable, so count is lower
          [2019-01-16T01:04:24.849291Z]         self.assertEqual(afw_schema.getFieldCount(), 82)
          [2019-01-16T01:04:24.849295Z]     
          [2019-01-16T01:04:24.849298Z]         afw_schema, col_map = schema.getAfwSchema("DiaSource")
          [2019-01-16T01:04:24.849301Z] >       self.assertEqual(len(col_map), 109)
          [2019-01-16T01:04:24.849305Z] E       AssertionError: 108 != 109
          

          I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr. Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?

          salnikov Andy Salnikov added a comment - - edited Jenkins build failed in unit test ( https://ci.lsst.codes/job/stack-os-matrix/29258/artifact/centos-6.devtoolset-6.py3/lsstsw/build/dax_ppdb/_build.log): [2019-01-16T01:04:24.849210Z] def test_getAfwSchemaWithExtras(self): [2019-01-16T01:04:24.849214Z] """Test for getAfwSchema method using extra afw schemas. [2019-01-16T01:04:24.849218Z] [2019-01-16T01:04:24.849221Z] Same as above but use non-default afw schemas, this adds few extra [2019-01-16T01:04:24.849226Z] columns to the table schema [2019-01-16T01:04:24.849230Z] """ [2019-01-16T01:04:24.849233Z] engine = create_engine('sqlite://') [2019-01-16T01:04:24.849236Z] [2019-01-16T01:04:24.849240Z] # create standard (baseline) schema, but use afw column map [2019-01-16T01:04:24.849243Z] afw_schemas = dict(DiaObject=make_minimal_dia_object_schema(), [2019-01-16T01:04:24.849246Z] DiaSource=make_minimal_dia_source_schema()) [2019-01-16T01:04:24.849250Z] schema = PpdbSchema(engine=engine, [2019-01-16T01:04:24.849253Z] dia_object_index="baseline", [2019-01-16T01:04:24.849256Z] dia_object_nightly=False, [2019-01-16T01:04:24.849260Z] schema_file=_data_file_name("ppdb-schema.yaml"), [2019-01-16T01:04:24.849263Z] column_map=_data_file_name("ppdb-afw-map.yaml"), [2019-01-16T01:04:24.849266Z] afw_schemas=afw_schemas) [2019-01-16T01:04:24.849270Z] schema.makeSchema() [2019-01-16T01:04:24.849274Z] [2019-01-16T01:04:24.849278Z] afw_schema, col_map = schema.getAfwSchema("DiaObject") [2019-01-16T01:04:24.849281Z] self.assertEqual(len(col_map), 94) [2019-01-16T01:04:24.849284Z] self.assertIsInstance(afw_schema, afwTable.Schema) [2019-01-16T01:04:24.849288Z] # no BLOBs in afwTable, so count is lower [2019-01-16T01:04:24.849291Z] self.assertEqual(afw_schema.getFieldCount(), 82) [2019-01-16T01:04:24.849295Z] [2019-01-16T01:04:24.849298Z] afw_schema, col_map = schema.getAfwSchema("DiaSource") [2019-01-16T01:04:24.849301Z] > self.assertEqual(len(col_map), 109) [2019-01-16T01:04:24.849305Z] E AssertionError: 108 != 109 I think this happens because make_minimal_dia_source_schema() makes column with the name psFluxErr . Before the renaming YAML schema defined psFluxSigma, so old schema contained both psFluxErr and psFluxSigma. After renaming there is just psFluxErr, so total number of columns reduced by one. Unit test constant (109) needs an update, can you do that?
          salnikov Andy Salnikov added a comment - - edited

          Looks good after unit test fix.

          salnikov Andy Salnikov added a comment - - edited Looks good after unit test fix.

          Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.

          cmorrison Chris Morrison [X] (Inactive) added a comment - Ah, thanks for catching that. I've fixed that issue but found another one where `lsst.geom.Angle` values that were NaN were not mapped to NULL properly and were producing errors. Take a look at the latest commit and let me know what you think. I'll fire off another Jenkins job as well.
          cmorrison Chris Morrison [X] (Inactive) added a comment - Jenkins: https://ci.lsst.codes/blue/organizations/jenkins/stack-os-matrix/detail/stack-os-matrix/29259/pipeline

          Looks OK too (I guess it's sufficiently tested now )

          salnikov Andy Salnikov added a comment - Looks OK too (I guess it's sufficiently tested now )

          Linux Jenkins has pass. Will now merge and close ticket.

          cmorrison Chris Morrison [X] (Inactive) added a comment - Linux Jenkins has pass. Will now merge and close ticket.

          People

            cmorrison Chris Morrison [X] (Inactive)
            cmorrison Chris Morrison [X] (Inactive)
            Andy Salnikov
            Andy Salnikov, Chris Morrison [X] (Inactive), John Swinbank
            Votes:
            0 Vote for this issue
            Watchers:
            3 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.