Uploaded image for project: 'Request For Comments'
  1. Request For Comments
  2. RFC-698

Drop rule suggesting .py extensions for Python code files

    XMLWordPrintable

    Details

    • Type: RFC
    • Status: Implemented
    • Resolution: Done
    • Component/s: DM
    • Labels:
      None

      Description

      Ending Python module files with .py is sufficiently established in the broader Python community that it's just noise to repeat it in our own developer guide (unlike C++, there are not multiple competing extensions).

      Ending Python script files with .py is not as well-established, but I believe it's actually a minor anti-pattern, as it exposes an implementation detail (the language the script is written in) to a caller who shouldn't care. And while it's unlikely that most of our scripts will ever be in a language other than Python, we should not make that more difficult for no reason.

      At the same time, we already have many scripts that already end in .py, and establishing a rule that says they shouldn't have a suffix could result in exactly the sort of churn we want to avoid: DM has a well-established history of not being able to avoid churning existing code to meet new standards when the standards change.

      I therefore propose that we drop mention of suffixes for Python files from the developer guide entirely. The new butler and pipetask commands do not have a .py suffix (will will continue to if this RFC is accepted), and are already slated to replace all of the .py-ending CmdLineTasks; I think that will make the future consistent enough.

        Attachments

          Issue Links

            Activity

            Hide
            pkubanek Petr Kubanek added a comment -

            I will rather vote to mention that while ending scripts in .py is not prohibited, it's discouraged. To make clear what's our preference. And rather keep MUST have .py ending for modules.

            Show
            pkubanek Petr Kubanek added a comment - I will rather vote to mention that while ending scripts in .py is not prohibited, it's discouraged. To make clear what's our preference. And rather keep MUST have .py ending for modules.
            Hide
            rowen Russell Owen added a comment - - edited

            I think the main advantage to having the .py extension is that the editor knows the language of the file and flake8 can run on it. So I would strongly suggest if we adopt this that we require that there be minimal implementation in such files. With such a requirement I am mildly in favor of this propose. Without it, I am strongly against.

            I hope we can all agree that complex python scripts with no .py extension is a bad idea.

            Show
            rowen Russell Owen added a comment - - edited I think the main advantage to having the .py extension is that the editor knows the language of the file and flake8 can run on it. So I would strongly suggest if we adopt this that we require that there be minimal implementation in such files. With such a requirement I am mildly in favor of this propose. Without it, I am strongly against. I hope we can all agree that complex python scripts with no .py extension is a bad idea.
            Hide
            jbosch Jim Bosch added a comment -

            I'm absolutely in favor of keeping script files small, and with no real logic (one-off scripts that never get committed to git are another matter, in both that and whether a .py extension makes sense, and even whether our coding conventions apply).

            From a more meta perspective, I'm not really in favor of using the dev guide to re-state best practices that are well established and well stated outside our own documentation ecosystem. Maybe it's just me, but I feel like our conventions are too big and hard to internalize already (this RFC exists because I wasn't aware of the current rule, after all), and I would much prefer for them to more tightly focus on policy and style choices that are about standardization rather than education.

            I also philosophically dislike constraining any aspect of interface to editor integration concerns - that's a level of implementation detail even deeper than language choice. I don't deny using suffixes for editors is convenient, and it's why I will continue to write one-off scripts with .py extensions - but if we need to work a little harder (or even clutter up our repos or file headers with a bit more stuff to make everyone's editors happy) to keep editor behavior from affecting interface decisions, so be it.

            Show
            jbosch Jim Bosch added a comment - I'm absolutely in favor of keeping script files small, and with no real logic (one-off scripts that never get committed to git are another matter, in both that and whether a .py extension makes sense, and even whether our coding conventions apply). From a more meta perspective, I'm not really in favor of using the dev guide to re-state best practices that are well established and well stated outside our own documentation ecosystem. Maybe it's just me, but I feel like our conventions are too big and hard to internalize already (this RFC exists because I wasn't aware of the current rule, after all), and I would much prefer for them to more tightly focus on policy and style choices that are about standardization rather than education. I also philosophically dislike constraining any aspect of interface to editor integration concerns - that's a level of implementation detail even deeper than language choice. I don't deny using suffixes for editors is convenient, and it's why I will continue to write one-off scripts with .py extensions - but if we need to work a little harder (or even clutter up our repos or file headers with a bit more stuff to make everyone's editors happy) to keep editor behavior from affecting interface decisions, so be it.
            Hide
            swinbank John Swinbank added a comment - - edited

            I agree with Jim's comments above, but would add — I suspect that most editors can be convinced to detect file types based on the contents of the hashbang line in addition to (or instead of) the file extension, so the flake8 issue may be a red-herring.

            Also, we already have reasonably strong conventions that scripts should be small. I suggest we use implementation of RFC as a reason to push on DM-24281 and turn those conventions into documented standards.

            Show
            swinbank John Swinbank added a comment - - edited I agree with Jim's comments above, but would add — I suspect that most editors can be convinced to detect file types based on the contents of the hashbang line in addition to (or instead of) the file extension, so the flake8 issue may be a red-herring. Also, we already have reasonably strong conventions that scripts should be small. I suggest we use implementation of RFC as a reason to push on DM-24281 and turn those conventions into documented standards.
            Hide
            ktl Kian-Tat Lim added a comment - - edited

            I'm not really in favor of using the dev guide to re-state best practices that are well established and well stated outside our own documentation ecosystem.

            I'm not sure about this, as not every developer will necessarily be fully plugged into the external ecosystem. At a minimum, we should have a reference to the best statements of best practices for educational purposes. But other than that I'm fine with not explicitly duplicating such conventions in our style guides.

            As to the particular case mentioned here, I had not really interpreted the rule to apply to non-module scripts, but given that the text can be interpreted differently, I welcome clarification or removal.

            Show
            ktl Kian-Tat Lim added a comment - - edited I'm not really in favor of using the dev guide to re-state best practices that are well established and well stated outside our own documentation ecosystem. I'm not sure about this, as not every developer will necessarily be fully plugged into the external ecosystem. At a minimum, we should have a reference to the best statements of best practices for educational purposes. But other than that I'm fine with not explicitly duplicating such conventions in our style guides. As to the particular case mentioned here, I had not really interpreted the rule to apply to non-module scripts, but given that the text can be interpreted differently, I welcome clarification or removal.
            Hide
            jbosch Jim Bosch added a comment -

            This is only ever so slightly contentious around the edges; there seems to be broad agreement on the core proposal.  I'm adopting this with the intent to keep it minimal: clarify that the rule on Python module naming does not extend to script files, with a brief aside that such files should not contain much code.

            I'd be happy for someone else to use this as motivation to work on DM-24281, but I don't want to make either ticket a blocker on the other; I already have an embarrassingly long list of adopted-not-implemented RFCs to follow up on, and don't want to make implementing this one any harder than it has to be.

            Show
            jbosch Jim Bosch added a comment - This is only ever so slightly contentious around the edges; there seems to be broad agreement on the core proposal.  I'm adopting this with the intent to keep it minimal: clarify that the rule on Python module naming does not extend to script files, with a brief aside that such files should not contain much code. I'd be happy for someone else to use this as motivation to work on DM-24281 , but I don't want to make either ticket a blocker on the other; I already have an embarrassingly long list of adopted-not-implemented RFCs to follow up on, and don't want to make implementing this one any harder than it has to be.
            Hide
            ktl Kian-Tat Lim added a comment - - edited

            One tricky thing about implementing this RFC: it may be difficult to flake8-check the (minimal) script file (outside an editor, unlike the comments above). https://github.com/tholo/pytest-flake8/issues/27

            Show
            ktl Kian-Tat Lim added a comment - - edited One tricky thing about implementing this RFC: it may be difficult to flake8-check the (minimal) script file (outside an editor, unlike the comments above). https://github.com/tholo/pytest-flake8/issues/27
            Hide
            tjenness Tim Jenness added a comment -

            I considered this during the RFC discussion. I'm inclined to say that this policy only applies to minimal script files and not scripts that include business logic. There is only one line of python in the minimalist files so living without the flake8 check seems like a low risk to code maintainability.

            In theory the linting job that runs on github could be changed to include a flake8 bin.src/* since only python files should be in bin.src. Then we would catch any problems even if pytest missed them.

            Show
            tjenness Tim Jenness added a comment - I considered this during the RFC discussion. I'm inclined to say that this policy only applies to minimal script files and not scripts that include business logic. There is only one line of python in the minimalist files so living without the flake8 check seems like a low risk to code maintainability. In theory the linting job that runs on github could be changed to include a flake8 bin.src/* since only python files should be in bin.src. Then we would catch any problems even if pytest missed them.

              People

              Assignee:
              jbosch Jim Bosch
              Reporter:
              jbosch Jim Bosch
              Watchers:
              Jim Bosch, John Swinbank, Kian-Tat Lim, Petr Kubanek, Russell Owen, Tim Jenness
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:
                Planned End:

                  Jenkins

                  No builds found.