Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7309

Port obs_cfht to Python 3

    XMLWordPrintable

    Details

      Description

      This ticket covers the work required to get obs_cfht working with Python 3.

        Attachments

          Issue Links

          blocks

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-7328 Port validate_drp to Python 3

          • Done
          is blocked by

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-7069 Port daf_persistence to Python 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-7247 Adapt daf_butlerUtils to py3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-7289 Port meas_algorithms to python 3

          • Done

          Story - Created by Jira Software - do not edit or delete. Issue type for a user story. DM-7292 Port pipe_tasks to Python 3

          • Done
          relates to

          Bug - A problem which impairs or prevents the functions of the product. DM-6163 Some validate_drp measurements are not reproducible

          • Won't Fix

          Epic - Created by Jira Software - do not edit or delete. Issue type for a big user story that needs to be broken down. DM-6179 Support Python 3 migration

          • Done
          mentioned in

          Page Loading...

            Activity

            Ascending order - Click to sort in descending order
            No builds found.
            tjenness Tim Jenness created issue -
            tjenness Tim Jenness made changes -
            Field Original Value New Value
            Link This issue relates to DM-6179 [ DM-6179 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-7069 [ DM-7069 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-7289 [ DM-7289 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-7292 [ DM-7292 ]
            tjenness Tim Jenness made changes -
            Link This issue is blocked by DM-7247 [ DM-7247 ]
            Hide
            Permalink
            tjenness Tim Jenness added a comment -

            Running futurize quickly on obs_cfht suggests only minor fixes are required:

            • print function,
            • checking whether divisions should be integer divisions in a couple of places,
            • deciding whether long should be imported from past.builtins.
            Show
            tjenness Tim Jenness added a comment - Running futurize quickly on obs_cfht suggests only minor fixes are required: print function, checking whether divisions should be integer divisions in a couple of places, deciding whether long should be imported from past.builtins .
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Link This issue blocks DM-7328 [ DM-7328 ]
            nlust Nate Lust made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            fred3m Fred Moolekamp made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            fred3m Fred Moolekamp made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            fred3m Fred Moolekamp made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            fred3m Fred Moolekamp made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            fred3m Fred Moolekamp made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            Parejkoj John Parejko made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            Parejkoj John Parejko made changes -
            Assignee Dominique Boutigny [ boutigny ] John Parejko [ parejkoj ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            Parejkoj John Parejko made changes -
            Sprint Alert Production F16 - 9 [ 247 ]
            Parejkoj John Parejko made changes -
            Story Points 1
            Parejkoj John Parejko made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            spietrowicz Steve Pietrowicz made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            Parejkoj John Parejko made changes -
            Sprint Alert Production F16 - 9 [ 247 ] Alert Production F16 - 9, Alert Production F16 - 11c [ 247, 296 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Doing this to finish py3 porting.

            Show
            Parejkoj John Parejko added a comment - Doing this to finish py3 porting.
            Parejkoj John Parejko made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            So, I accidentally pushed these three commits to master. They pass Jenkins, and validate_drp shows a small change but still within the spec (see below). Fred: could you please look at the three latest commits on obs_cfht and check them? Unless I'm told otherwise, I'm of a mind to just leave them there.

            https://github.com/lsst/obs_cfht/commits/tickets/DM-7309

            validate_drp results below:

            pre-py3:

            minimum: PF1=15% of diffs more than PA2 = 15.00 mmag (target is < 20 %)
            		 
            design : PF1=15% of diffs more than PA2 = 15.00 mmag (target is < 10 %)
            		 
            stretch: PF1=21% of diffs more than PA2 = 10.00 mmag (target is <  5 %)
            		 
            ...
            		 
            PF1 : 15.59 %    < 10.00 %    == False

            post-py3:

            minimum: PF1=16% of diffs more than PA2 = 15.00 mmag (target is < 20 %)
            		 
            design : PF1=16% of diffs more than PA2 = 15.00 mmag (target is < 10 %)
            		 
            stretch: PF1=24% of diffs more than PA2 = 10.00 mmag (target is <  5 %)
            		 
            ...
            		 
            PF1 : 16.67 %    < 10.00 %    == False

            Show
            Parejkoj John Parejko added a comment - So, I accidentally pushed these three commits to master. They pass Jenkins, and validate_drp shows a small change but still within the spec (see below). Fred: could you please look at the three latest commits on obs_cfht and check them? Unless I'm told otherwise, I'm of a mind to just leave them there. https://github.com/lsst/obs_cfht/commits/tickets/DM-7309 validate_drp results below: pre-py3: minimum: PF1=15% of diffs more than PA2 = 15.00 mmag (target is < 20 %) design : PF1=15% of diffs more than PA2 = 15.00 mmag (target is < 10 %) stretch: PF1=21% of diffs more than PA2 = 10.00 mmag (target is < 5 %) ... PF1 : 15.59 % < 10.00 % == False post-py3: minimum: PF1=16% of diffs more than PA2 = 15.00 mmag (target is < 20 %) design : PF1=16% of diffs more than PA2 = 15.00 mmag (target is < 10 %) stretch: PF1=24% of diffs more than PA2 = 10.00 mmag (target is < 5 %) ... PF1 : 16.67 % < 10.00 % == False
            Parejkoj John Parejko made changes -
            Reviewers Fred Moolekamp [ fred3m ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            And, actually, re-running it, that difference is apparently within the run-to-run noise (some random seed isn't getting set?).

            Show
            Parejkoj John Parejko added a comment - And, actually, re-running it, that difference is apparently within the run-to-run noise (some random seed isn't getting set?).
            Hide
            Permalink
            fred3m Fred Moolekamp added a comment -

            The coding changes look good to me. It might be worth opening a ticket to look into variances in the validate_drp results to understand where they are coming from.

            Show
            fred3m Fred Moolekamp added a comment - The coding changes look good to me. It might be worth opening a ticket to look into variances in the validate_drp results to understand where they are coming from.
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            It appears that the difference I'm seeing above is the problem specified in DM-6163. Looks like we're in the clear here.

            In the interest of transparency, these are the relevant commits:

            https://github.com/lsst/obs_cfht/commit/706a1e83e023d3607a16d9b9d2eaa9cb2989177a
            https://github.com/lsst/obs_cfht/commit/ed003298d1a53686c19293d42b24b01828891c35
            https://github.com/lsst/obs_cfht/commit/6c39fbd916ab01d73a85ac8eb6d6882038f88711

            Show
            Parejkoj John Parejko added a comment - It appears that the difference I'm seeing above is the problem specified in DM-6163 . Looks like we're in the clear here. In the interest of transparency, these are the relevant commits: https://github.com/lsst/obs_cfht/commit/706a1e83e023d3607a16d9b9d2eaa9cb2989177a https://github.com/lsst/obs_cfht/commit/ed003298d1a53686c19293d42b24b01828891c35 https://github.com/lsst/obs_cfht/commit/6c39fbd916ab01d73a85ac8eb6d6882038f88711
            tjenness Tim Jenness made changes -
            Link This issue relates to DM-6163 [ DM-6163 ]
            Hide
            Permalink
            Parejkoj John Parejko added a comment -

            Final jenkins run passed, merged lsst_py3 addition. We're in the home stretch!

            Show
            Parejkoj John Parejko added a comment - Final jenkins run passed, merged lsst_py3 addition. We're in the home stretch!
            Parejkoj John Parejko made changes -
            Resolution Done [ 10000 ]
            Status In Review [ 10004 ] Done [ 10002 ]
            krughoff Simon Krughoff made changes -
            Team Alert Production [ 10300 ]
            tjenness Tim Jenness made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14236 ] This issue links to "Page (Confluence)" [ 14236 ]
            krughoff Simon Krughoff made changes -
            Epic Link DM-7362 [ 26448 ]

              People

              Assignee:
              Parejkoj John Parejko
              Reporter:
              tjenness Tim Jenness
              Reviewers:
              Fred Moolekamp
              Watchers:
              Fabio Hernandez, Fred Moolekamp, John Parejko, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.