Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-17276

XML - Update MTMount XMLs to conform to Naming Conventions

    XMLWordPrintable

    Details

    • Story Points:
      5
    • Sprint:
      TSSW Sprint - Jan 21 - Feb 01, TSSW Sprint - Mar 30 - Apr 13
    • Team:
      Telescope and Site

      Description

      Once the software is officially delivered to LSST, it will be necessary to update the design and code to conform to SAL constraints and conventions, as defined in https://confluence.lsstcorp.org/display/SYSENG/SAL+constraints+and+recommendations

      This includes:

      • Naming conventions
      • Enumeration definition
      • Implementation of all standard commands and events.

        Attachments

          Activity

          No builds found.
          rbovill Rob Bovill created issue -
          rbovill Rob Bovill made changes -
          Field Original Value New Value
          Risk Score 0
          Hide
          rbovill Rob Bovill added a comment -

          Note to QA: Once this task is complete, update robotframework_ts_xml/scripts/Attribute_Naming.py and robotframework_ts_xml/scripts/XML_Generic.py to remove the special handling for this CSC and regenerate the test suites.

          Show
          rbovill Rob Bovill added a comment - Note to QA: Once this task is complete, update robotframework_ts_xml/scripts/Attribute_Naming.py and robotframework_ts_xml/scripts/XML_Generic.py to remove the special handling for this CSC and regenerate the test suites.
          rbovill Rob Bovill made changes -
          Link This issue is cloned by TSS-2985 [ TSS-2985 ]
          rbovill Rob Bovill made changes -
          Link This issue is cloned by TSS-2985 [ TSS-2985 ]
          rbovill Rob Bovill made changes -
          Assignee Dave Mills [ dmills ]
          dmills Dave Mills made changes -
          Project Telescope and Site Software [ 10403 ] Data Management [ 10501 ]
          Key TSS-2915 DM-17276
          Workflow Telescope and Site Software workflow [ 173039 ] Software Simplified Workflow for Project DM [ 251687 ]
          Status Unplanned [ 1 ] To Do [ 10001 ]
          Business Value 1
          Issue Type Task [ 3 ] Story [ 10001 ]
          Epic SP Progress value
          Team Telescope and Site [ 13500 ]
          Component/s ts_main_telescope [ 16710 ]
          Component/s WBS Mount Control [ 14232 ]
          Epic Story Progress value
          jbuffill James Buffill [X] (Inactive) made changes -
          Labels MainTelescope Mount
          jbuffill James Buffill [X] (Inactive) made changes -
          Start date 03/Aug/18
          End date 18/Jan/19
          dmills Dave Mills made changes -
          Epic Link DM-20191 [ 326477 ]
          dmills Dave Mills made changes -
          End date 18/Jan/19 18/Jan/19
          dmills Dave Mills made changes -
          Story Points 5
          aclements Andy Clements made changes -
          Assignee Dave Mills [ dmills ] Andrew Heyer [ aheyer ]
          dmills Dave Mills made changes -
          Labels MainTelescope Mount MainTelescope Mount SAL-TODO
          dmills Dave Mills made changes -
          Resolution Done [ 10000 ]
          Status To Do [ 10001 ] Invalid [ 11005 ]
          aclements Andy Clements made changes -
          Resolution Done [ 10000 ]
          Status Invalid [ 11005 ] To Do [ 10001 ]
          aclements Andy Clements made changes -
          Epic Link DM-20191 [ 326477 ] DM-22023 [ 425560 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Epic Link DM-22023 [ 425560 ] DM-20191 [ 326477 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Oct 28 - Nov 10 [ 966 ]
          aclements Andy Clements made changes -
          Epic Link DM-20191 [ 326477 ] DM-22023 [ 425560 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Oct 28 - Nov 10 [ 966 ] TSSW Sprint - Nov 11 - Nov 22 [ 969 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Rank Ranked lower
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Nov 11 - Nov 22 [ 969 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Rank Ranked higher
          dmills Dave Mills made changes -
          Labels MainTelescope Mount SAL-TODO MainTelescope Mount xml
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Jan 21 - Feb 01 [ 993 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Rank Ranked lower
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment -

          Fixed the blank spaces in MTMount and added a test to check for blank spaces. However in doing so other subsystems fail for blank spaces.

          Do we want to allow blank spaces in units? The blank units I saw were used to a create a unit like "deg / s" however when I changed it to "deg/s" it still seems to be recognized as an astropy unit. 

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - Fixed the blank spaces in MTMount and added a test to check for blank spaces. However in doing so other subsystems fail for blank spaces. Do we want to allow blank spaces in units? The blank units I saw were used to a create a unit like "deg / s" however when I changed it to "deg/s" it still seems to be recognized as an astropy unit. 
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status In Progress [ 3 ] In Review [ 10004 ]
          Reviewers Rob Bovill [ rbovill ]
          Hide
          rbovill Rob Bovill added a comment -

          Since Astropy accepts units in the format of `m / s` and `m/s` then our tests must as well. I made a comment in the PR with a suggested update to the test that will properly handle the formatting and still catch fields of only blank spaces.
          https://github.com/lsst-ts/ts_xml/pull/224#discussion_r373596193

          Show
          rbovill Rob Bovill added a comment - Since Astropy accepts units in the format of `m / s` and `m/s` then our tests must as well. I made a comment in the PR with a suggested update to the test that will properly handle the formatting and still catch fields of only blank spaces. https://github.com/lsst-ts/ts_xml/pull/224#discussion_r373596193
          Hide
          rbovill Rob Bovill added a comment -

          MTMount XML changes approved. Tests now passing. Updates to Unit tests are also good. See https://github.com/lsst-ts/ts_xml/pull/224 for details.

          Show
          rbovill Rob Bovill added a comment - MTMount XML changes approved. Tests now passing. Updates to Unit tests are also good. See https://github.com/lsst-ts/ts_xml/pull/224 for details.
          rbovill Rob Bovill made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          aclements Andy Clements made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]
          Hide
          aclements Andy Clements added a comment -

          So maybe someone can correct me, but this ticket does not seem to be complete.  When I comment out the exception in the ts_xml tests for this ticket, I get the following error:

           

          ________________________________________________ test_attribute_naming[xmlfile102-MTMount-Telemetry] _________________________________________________
          xmlfile = PosixPath('/opt/CODE/ts_xml/sal_interfaces/MTMount/MTMount_Telemetry.xml'), csc = 'MTMount', topic = 'Telemetry'
          @pytest.mark.parametrize("xmlfile,csc,topic", ts_xml.get_xmlfile_csc_topic())
           def test_attribute_naming(xmlfile, csc, topic):
           """Test that the <EFDB_Name> field for topic attributes is properly formed,
           i.e. it begins with a lowercase letter and contains only alphanumeric
           and underscore characters.
           
           Parameters
           ----------
           xmlfile : `pathlib.Path`
           Full filepath to the Commands or Events XML file for the CSC.
           csc : `testutils.subsystems`
           Name of the CSC
           topic : `xmlfile.stem`
           One of ['Commands','Events','Telemetry']
           """
           saltype = "SAL" + topic.rstrip('s')
           # Check for known issues.
           jira = check_for_issues(csc, topic)
           if jira:
           pytest.skip(jira + ": " + str(xmlfile.name) + " <EFDB_Name> is not properly formed.")
           # Test the attribute <Description> fields.
           with open(str(xmlfile), "r", encoding="utf-8") as f:
           tree = et.parse(f)
           root = tree.getroot()
           for name in root.findall(f"./{saltype}/item/EFDB_Name"):
           # re.match() returns None if the string does not match the regex.
          > assert re.match(r"^[a-z]([a-zA-Z0-9_]*$)", name.text) is not None, \
           name.text + ' in ' + str(xmlfile.name) + ' does not begin with a lowercase '\
           'letter and/or contains non-alphanumeric characters.'
          E AssertionError: CCW_Status in MTMount_Telemetry.xml does not begin with a lowercase letter and/or contains non-alphanumeric characters.
          E assert None is not None
          E + where None = <function match at 0x7fce629aa488>('^[a-z]([a-zA-Z0-9_]*$)', 'CCW_Status')
          E + where <function match at 0x7fce629aa488> = re.match
          E + and 'CCW_Status' = <Element 'EFDB_Name' at 0x7fce50a647c8>.text
          tests/test_AttributeNaming.py:45: AssertionError
          

           

          I remember we are still waiting for Tekniker to update their code for this?

          let me know....

          Show
          aclements Andy Clements added a comment - So maybe someone can correct me, but this ticket does not seem to be complete.  When I comment out the exception in the ts_xml tests for this ticket, I get the following error:   ________________________________________________ test_attribute_naming[xmlfile102-MTMount-Telemetry] _________________________________________________ xmlfile = PosixPath( '/opt/CODE/ts_xml/sal_interfaces/MTMount/MTMount_Telemetry.xml' ), csc = 'MTMount' , topic = 'Telemetry' @pytest .mark.parametrize( "xmlfile,csc,topic" , ts_xml.get_xmlfile_csc_topic()) def test_attribute_naming(xmlfile, csc, topic): "" "Test that the <EFDB_Name> field for topic attributes is properly formed, i.e. it begins with a lowercase letter and contains only alphanumeric and underscore characters. Parameters ---------- xmlfile : `pathlib.Path` Full filepath to the Commands or Events XML file for the CSC. csc : `testutils.subsystems` Name of the CSC topic : `xmlfile.stem` One of [ 'Commands' , 'Events' , 'Telemetry' ] "" " saltype = "SAL" + topic.rstrip( 's' ) # Check for known issues. jira = check_for_issues(csc, topic) if jira: pytest.skip(jira + ": " + str(xmlfile.name) + " <EFDB_Name> is not properly formed." ) # Test the attribute <Description> fields. with open(str(xmlfile), "r" , encoding= "utf-8" ) as f: tree = et.parse(f) root = tree.getroot() for name in root.findall(f "./{saltype}/item/EFDB_Name" ): # re.match() returns None if the string does not match the regex. > assert re.match(r "^[a-z]([a-zA-Z0-9_]*$)" , name.text) is not None, \ name.text + ' in ' + str(xmlfile.name) + ' does not begin with a lowercase ' \ 'letter and/or contains non-alphanumeric characters.' E AssertionError: CCW_Status in MTMount_Telemetry.xml does not begin with a lowercase letter and/or contains non-alphanumeric characters. E assert None is not None E + where None = <function match at 0x7fce629aa488 >( '^[a-z]([a-zA-Z0-9_]*$)' , 'CCW_Status' ) E + where <function match at 0x7fce629aa488 > = re.match E + and 'CCW_Status' = <Element 'EFDB_Name' at 0x7fce50a647c8 >.text tests/test_AttributeNaming.py: 45 : AssertionError   I remember we are still waiting for Tekniker to update their code for this? let me know....
          aclements Andy Clements made changes -
          Resolution Done [ 10000 ]
          Status Done [ 10002 ] To Do [ 10001 ]
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment -

          Can you tell me what lines you comment out?

          When pull request https://github.com/lsst-ts/ts_xml/pull/224 was merged we passed all the tests and added another test to catch an issue that Rob Discovered was not being caught. I hope I didn't accidentally comment out any tests...

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - Can you tell me what lines you comment out? When pull request https://github.com/lsst-ts/ts_xml/pull/224  was merged we passed all the tests and added another test to catch an issue that Rob Discovered was not being caught. I hope I didn't accidentally comment out any tests...
          Hide
          aclements Andy Clements added a comment -

          SKIPPED [1] /opt/CODE/ts_xml/tests/test_AttributeNaming.py:38: DM-17276: MTMount_Telemetry.xml <EFDB_Name> is not properly formed.

          SKIPPED [1] /opt/CODE/ts_xml/tests/test_TopicNaming.py:119: DM-17276: MTMount_Telemetry.xml <EFDB_Topic> is not properly formed.

          are the files where the tests are being skipped.  you can comment out the skip at the top of the tests.

           

          Show
          aclements Andy Clements added a comment - SKIPPED [1] /opt/CODE/ts_xml/tests/test_AttributeNaming.py:38: DM-17276 : MTMount_Telemetry.xml <EFDB_Name> is not properly formed. SKIPPED [1] /opt/CODE/ts_xml/tests/test_TopicNaming.py:119: DM-17276 : MTMount_Telemetry.xml <EFDB_Topic> is not properly formed. are the files where the tests are being skipped.  you can comment out the skip at the top of the tests.  
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited

          What branch are you using to run the tests? I wish to duplicate your results to resolve the issue. 

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited What branch are you using to run the tests? I wish to duplicate your results to resolve the issue. 
          Hide
          aclements Andy Clements added a comment - - edited

          tickets/DM-23820

          Show
          aclements Andy Clements added a comment - - edited tickets/ DM-23820
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment -

          the ticket DM-23820 is marked Done, does this mean this is still an issue. I also don't see 23280 as a branch on https://github.com/lsst-ts/ts_xml/branches/all. Perhaps we can talk after standup and you can walk me through what you are doing?

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - the ticket DM-23820 is marked Done, does this mean this is still an issue. I also don't see 23280 as a branch on  https://github.com/lsst-ts/ts_xml/branches/all . Perhaps we can talk after standup and you can walk me through what you are doing?
          Hide
          aclements Andy Clements added a comment -

          23280 - is my typo.  DM-23820 is marked done, but the code that skips this ticket is still in the repo - when commented out - the test fails.

          The code is in tests/test_AttributeNaming.py:

           

          def check_for_issues(csc, topic):
           if csc == "MTMount" and (topic == "Telemetry"):
           jira = "DM-17276"
           
          

          you can see that this test is skipping the MTMount telemetry file.

           

           

          Show
          aclements Andy Clements added a comment - 23280 - is my typo.  DM-23820 is marked done, but the code that skips this ticket is still in the repo - when commented out - the test fails. The code is in tests/test_AttributeNaming.py:   def check_for_issues(csc, topic): if csc == "MTMount" and (topic == "Telemetry" ): jira = "DM-17276"   you can see that this test is skipping the MTMount telemetry file.    
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Jan 21 - Feb 01 [ 993 ] TSSW Sprint - Jan 21 - Feb 01, TSSW Sprint - Mar 16 - Mar 30 [ 993, 1006 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Rank Ranked higher
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited

          I replicated the issue, 

          To resolve this the first step I will make is getting in contact with Tekniker. 

          I could simply fix the topics however this wouldn't really resolve anything since the software using the topics won't be updated. Therefor for actual resolution of the telemetry file the following steps will be taken. 

          1) Meeting with Alberto and Russell discuss the updates of the Telemetry topics
          2) There are a few that Russell and myself already discussed that we prefer become events
          3) The Topics that we decide to keep as Telemetry update them in the labview software and update in the XML
          4) Test the software with the TMA simulator to exercise the publishing of the telemetry topics. 

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited I replicated the issue,  To resolve this the first step I will make is getting in contact with Tekniker.  I could simply fix the topics however this wouldn't really resolve anything since the software using the topics won't be updated. Therefor for actual resolution of the telemetry file the following steps will be taken.  1) Meeting with Alberto and Russell discuss the updates of the Telemetry topics 2) There are a few that Russell and myself already discussed that we prefer become events 3) The Topics that we decide to keep as Telemetry update them in the labview software and update in the XML 4) Test the software with the TMA simulator to exercise the publishing of the telemetry topics. 
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Sprint TSSW Sprint - Jan 21 - Feb 01, TSSW Sprint - Mar 16 - Mar 30 [ 993, 1006 ] TSSW Sprint - Jan 21 - Feb 01, TSSW Sprint - Mar 30 - Apr 13 [ 993, 1009 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Rank Ranked lower
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment -

          1) Completed, we have discussed and Russell myself and alberto can work on the XML with no issue because Russell works on "NewMTMount" subsystem while the work Alberto and myself will do is on "MTMount" and therefore not effecting each others work. 

          2) Russell and Alberto have discussed the telemetry topics which are to become events. 
          3) same as 2

          4) In order to do this testing we need ethernet cards which I have made a requisition for. 

          Also meeting on this Friday April 6 to complete any issues with the MTMount xml. 

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - 1) Completed, we have discussed and Russell myself and alberto can work on the XML with no issue because Russell works on "NewMTMount" subsystem while the work Alberto and myself will do is on "MTMount" and therefore not effecting each others work.  2) Russell and Alberto have discussed the telemetry topics which are to become events.  3) same as 2 4) In order to do this testing we need ethernet cards which I have made a requisition for.  Also meeting on this Friday April 6 to complete any issues with the MTMount xml. 
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status In Progress [ 3 ] To Do [ 10001 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status To Do [ 10001 ] In Progress [ 3 ]
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited

          If the XML tests pass we are all good. Do not merge into develop please. (edit) the test which now passes that did not previously is test_AttributeNaming.py. This test should be passing. I will need to make other tasks in the future to fix the others. 

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited If the XML tests pass we are all good. Do not merge into develop please. (edit) the test which now passes that did not previously is test_AttributeNaming.py. This test should be passing. I will need to make other tasks in the future to fix the others. 
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Status In Progress [ 3 ] In Review [ 10004 ]
          Reviewers Rob Bovill [ rbovill ] Eric Coughlin [ ecoughlin ]
          Hide
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited

          I will need to revisit this to resolve tests which are currently being skipped. 

          • test_TopicNaming.py
          • test_NoReservedWords.py

          For now I need to continue onto other work. 

          Tasks tracked here DM-24358

          Show
          aheyer Andrew Heyer [X] (Inactive) added a comment - - edited I will need to revisit this to resolve tests which are currently being skipped.  test_TopicNaming.py test_NoReservedWords.py For now I need to continue onto other work.  Tasks tracked here  DM-24358
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Reviewers Eric Coughlin [ ecoughlin ] Rob Bovill [ rbovill ]
          Hide
          rbovill Rob Bovill added a comment -

          The issue in the MTMount XMLs in regards to the test_AttributeNaming.py unit test have been resolved.  The work for the remaining issues will be tracked using the linked ticket.  Then all fixes will be merged to develop.

          Show
          rbovill Rob Bovill added a comment - The issue in the MTMount XMLs in regards to the test_AttributeNaming.py unit test have been resolved.  The work for the remaining issues will be tracked using the linked ticket.  Then all fixes will be merged to develop.
          rbovill Rob Bovill made changes -
          Status In Review [ 10004 ] Reviewed [ 10101 ]
          aheyer Andrew Heyer [X] (Inactive) made changes -
          Resolution Done [ 10000 ]
          Status Reviewed [ 10101 ] Done [ 10002 ]

            People

            Assignee:
            aheyer Andrew Heyer [X] (Inactive)
            Reporter:
            rbovill Rob Bovill
            Reviewers:
            Rob Bovill
            Watchers:
            Andrew Heyer [X] (Inactive), Andy Clements, Eric Coughlin, Rob Bovill
            Votes:
            0 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated:
              Resolved:
              Start date:
              End date:

                Jenkins

                No builds found.