Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-32870

Use MatchProbabilistic PipelineTask for DC2 plots

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None
    • 2
    • Data Release Production
    • No

    Description

      Update the notebook from DM-31781 to use outputs from the PipelineTask created in DM-32034 (currently, the parquet tables are manually written to/read from disk).

      Attachments

        Activity

          dtaranu Dan Taranu added a comment -

          fred3m, can you see if this works for you without having to setup/install gauss2d/(meas_extensions_)multiprofit? I tested by including only modelling_research in my PYTHONPATH and it seemed to work.

          If you need to compare, I have w44 and w40 notebooks (the latter in an old subdirectory since it won't work as-is anymore).

          dtaranu Dan Taranu added a comment - fred3m , can you see if this works for you without having to setup/install gauss2d/(meas_extensions_)multiprofit? I tested by including only modelling_research in my PYTHONPATH and it seemed to work. If you need to compare, I have w44 and w40 notebooks (the latter in an old subdirectory since it won't work as-is anymore).

          Sure. Sorry, I forgot about this today but I'll test it out tomorrow.

          fred3m Fred Moolekamp added a comment - Sure. Sorry, I forgot about this today but I'll test it out tomorrow.
          dtaranu Dan Taranu added a comment - - edited

          For reference, the command to run the matcher:

          pipetask --long-log run -b /repo/dc2 --input "2.2i/runs/test-med-1/w_2021_44/DM-32391,2.2i/truth_summary" --output "u/dtaranu/DM-32034/w_2021_44_match_0.5asec" -d "instrument = 'LSSTCam-imSim' and tract=3829 and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --register-dataset-types

          dtaranu Dan Taranu added a comment - - edited For reference, the command to run the matcher: pipetask --long-log run -b /repo/dc2 --input "2.2i/runs/test-med-1/w_2021_44/ DM-32391 ,2.2i/truth_summary" --output "u/dtaranu/ DM-32034 /w_2021_44_match_0.5asec" -d "instrument = 'LSSTCam-imSim' and tract=3829 and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --register-dataset-types

          Something isn't right. When I run the notebook before using this task I get

          for the 2nd and 3rd plots in the notebook, but if I use the notebook contained in this ticket I get

          when run on the same data.

          The task command that I used to merge the catalogs was

          pipetask --long-log run -b /repo/dc2 --output-run "u/fred3m/DM-32436/DC2_44/20211210T214832Z" --extend-run -d "instrument = 'LSSTCam-imSim' and tract in (3828,3829) and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --input 2.2i/runs/test-med-1/w_2021_44/DM-32391,2.2i/truth_summary --no-versions

          fred3m Fred Moolekamp added a comment - Something isn't right. When I run the notebook before using this task I get for the 2nd and 3rd plots in the notebook, but if I use the notebook contained in this ticket I get when run on the same data. The task command that I used to merge the catalogs was pipetask --long-log run -b /repo/dc2 --output-run "u/fred3m/ DM-32436 /DC2_44/20211210T214832Z" --extend-run -d "instrument = 'LSSTCam-imSim' and tract in (3828,3829) and skymap='DC2'" -p /project/dtaranu/dc2/match_catalog_pipe/match_tract_catalog_probabilistic.yaml --input 2.2i/runs/test-med-1/w_2021_44/ DM-32391 ,2.2i/truth_summary --no-versions

          Ok, thanks to assistance from dtaranu in slack we were able to get the notebook working.

          fred3m Fred Moolekamp added a comment - Ok, thanks to assistance from dtaranu in slack we were able to get the notebook working.

          The changes work to recreate the plots from the previous version of the notebook.

          fred3m Fred Moolekamp added a comment - The changes work to recreate the plots from the previous version of the notebook.

          People

            dtaranu Dan Taranu
            dtaranu Dan Taranu
            Fred Moolekamp
            Dan Taranu, Fred Moolekamp
            Votes:
            0 Vote for this issue
            Watchers:
            2 Start watching this issue

            Dates

              Created:
              Updated:
              Resolved:

              Jenkins

                No builds found.