Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-7363

convert stack packages from paf policy to yaml policy

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      2
    • Sprint:
      DB_F16_8, DB_F16_9, DB_F16_10, DB_S17_12, DB_S17_01, DB_S17_2, DB_S17_4, DB_S17_5
    • Team:
      Data Access and Database

      Description

      go through the stack & find all the paf Butler Policies. load them as daf.persistence.Policy and write them out to yaml. in git, remove the paf file, add the yaml file.

        Attachments

          Issue Links

            Activity

            Hide
            npease Nate Pease added a comment -

            Perry, can you please take a look at these changes? I thought you would be most appropriate because of the work you've done recently with the paf files, you're familiar with the content.

            Show
            npease Nate Pease added a comment - Perry, can you please take a look at these changes? I thought you would be most appropriate because of the work you've done recently with the paf files, you're familiar with the content.
            Hide
            pgee Perry Gee added a comment -

            Nate Pease

            There is a problem with obs_test which is really my fault, but I think you are in a better position to address it.

            When I made my modifications for the RFCs, I modified testMapper.paf, not testMapper.yaml, and I was treating testMapper.paf as the reference for my ultimate comparisons of before and after. (I do this by creating a Mapper of each type and looking at its internal datastructures).

            Now that testMapper.paf is gone, testMapper.yaml has taken over, and many of the datasets in the two are different (and most of my cleanup and deletions from DM-8128 are gone.

            All of the other Mapper.yaml files appear to create the same exposures and datasets, so they are all functioning as equivalents to the Mapper.paf files (good)
            ------------------------------------
            One other issue I am concerned about is that between the two of us, we have pretty much removed all of the original comments from the paf files. Plus, since your Mapper.yaml files are alphabetized, there is really no possibility of keeping the datasets and exposures in sections according to their use in the code. Those two things will make maintenance of the yaml files difficult (for example, if I remove a Task which makes use of 5 different datasets, I will have to hunt around to find them and delete them.

            Show
            pgee Perry Gee added a comment - Nate Pease There is a problem with obs_test which is really my fault, but I think you are in a better position to address it. When I made my modifications for the RFCs, I modified testMapper.paf, not testMapper.yaml, and I was treating testMapper.paf as the reference for my ultimate comparisons of before and after. (I do this by creating a Mapper of each type and looking at its internal datastructures). Now that testMapper.paf is gone, testMapper.yaml has taken over, and many of the datasets in the two are different (and most of my cleanup and deletions from DM-8128 are gone. All of the other Mapper.yaml files appear to create the same exposures and datasets, so they are all functioning as equivalents to the Mapper.paf files (good) ------------------------------------ One other issue I am concerned about is that between the two of us, we have pretty much removed all of the original comments from the paf files. Plus, since your Mapper.yaml files are alphabetized, there is really no possibility of keeping the datasets and exposures in sections according to their use in the code. Those two things will make maintenance of the yaml files difficult (for example, if I remove a Task which makes use of 5 different datasets, I will have to hunt around to find them and delete them.
            Hide
            pgee Perry Gee added a comment -

            Send this back to me when you get the ordering issue resolved, and I will try to insert the comments from before.

            Show
            pgee Perry Gee added a comment - Send this back to me when you get the ordering issue resolved, and I will try to insert the comments from before.
            Hide
            pgee Perry Gee added a comment -

            Nate Pease
            I was expecting a response to my comments above before I finished this review. If you are waiting on me, please respond.

            Show
            pgee Perry Gee added a comment - Nate Pease I was expecting a response to my comments above before I finished this review. If you are waiting on me, please respond.
            Hide
            npease Nate Pease added a comment -

            hey Perry Gee, thanks for pinging me. I did leave a note for you on Slack, but here is ok too.

            I think I have addressed all your comments, and the changes are visible in the PR's:
            testMapper.yaml was entirely overwritten with the contents of testMapper.paf, so there should be no missing items.
            the order of datasets in the yaml files now matches the order in the paf files.
            I also migrated to yaml any comments in the paf files that I had missed previously.

            Show
            npease Nate Pease added a comment - hey Perry Gee , thanks for pinging me. I did leave a note for you on Slack, but here is ok too. I think I have addressed all your comments, and the changes are visible in the PR's: testMapper.yaml was entirely overwritten with the contents of testMapper.paf, so there should be no missing items. the order of datasets in the yaml files now matches the order in the paf files. I also migrated to yaml any comments in the paf files that I had missed previously.
            Hide
            pgee Perry Gee added a comment -

            OK, thanks. I saw the checkins come through to git, but I assume that you would reply here when you were done. My fault.

            The ordering will make it easier for people who were used to the order in the paf file, so that's good.

            All the Mappers now have 0 differences from the once with the paf files, also good.

            I think that the problems with the comments were mostly my mistakes. I will open a new issue to move the useful comment to obs_base exposures.yaml and datasets.yaml

            Show
            pgee Perry Gee added a comment - OK, thanks. I saw the checkins come through to git, but I assume that you would reply here when you were done. My fault. The ordering will make it easier for people who were used to the order in the paf file, so that's good. All the Mappers now have 0 differences from the once with the paf files, also good. I think that the problems with the comments were mostly my mistakes. I will open a new issue to move the useful comment to obs_base exposures.yaml and datasets.yaml
            Hide
            pgee Perry Gee added a comment -

            I'm done. I see a couple of end of line space in the test policy paf file.

            Otherwise, everything seems fine.

            Show
            pgee Perry Gee added a comment - I'm done. I see a couple of end of line space in the test policy paf file. Otherwise, everything seems fine.

              People

              • Assignee:
                npease Nate Pease
                Reporter:
                npease Nate Pease
                Reviewers:
                Perry Gee
                Watchers:
                Nate Pease, Perry Gee
              • Votes:
                0 Vote for this issue
                Watchers:
                2 Start watching this issue

                Dates

                • Created:
                  Updated:
                  Resolved:

                  Summary Panel