Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-14289

Update qa_explorer visualization code to use the new ParquetTable datasets

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None
    • Story Points:
      16
    • Epic Link:
    • Sprint:
      DRP F18-5, DRP F18-6
    • Team:
      Data Release Production

      Description

      Originally, the functions of performing QA calculations and visualizing the results within qa_explorer were both contained within the QADataset object; however, with the completing of DM-13876 and DM-13877, I have externalized the computations to their own task, with results persisted in parquet format and accessible via the butler. So now, I need to make sure that the visualization part of the code works with this new setup, and to update the demos appropriately.

        Attachments

          Issue Links

            Activity

            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            I believe that this has been completed enough to merge. Not all the functionality of the original qa_explorer is here yet (notably the visit-level exploration), but the visualizations are now up-to-date with the functionality of the new deepCoadd_obj ParquetTable dataset, and I think it's high time to get this all into master.

            Also notably, the updates on this branch contain the generic PostprocessTask and MultibandPostprocessTask that enable defining functors in a YAML file and running the calculation of those functors on the deepCoadd_obj tables.

            Yusra AlSayyad let me know if you are able to review this. I was also thinking it might be good for Lauren MacArthur and/or Hsin-Fang Chiang to take a look also, since I think this is the biggest step so far toward replacing a large amount of the functionality currently in pipe_analysis.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - I believe that this has been completed enough to merge. Not all the functionality of the original qa_explorer is here yet (notably the visit-level exploration), but the visualizations are now up-to-date with the functionality of the new deepCoadd_obj ParquetTable dataset, and I think it's high time to get this all into master. Also notably, the updates on this branch contain the generic PostprocessTask and MultibandPostprocessTask that enable defining functors in a YAML file and running the calculation of those functors on the deepCoadd_obj tables. Yusra AlSayyad let me know if you are able to review this. I was also thinking it might be good for Lauren MacArthur and/or Hsin-Fang Chiang to take a look also, since I think this is the biggest step so far toward replacing a large amount of the functionality currently in pipe_analysis .
            Hide
            tmorton Tim Morton [X] (Inactive) added a comment -

            BTW, the new relevant notebook demo has been moved to notebooks/coadd_demo.ipynb, and the old demo notebook have been sequestered into notebooks/old.

            Show
            tmorton Tim Morton [X] (Inactive) added a comment - BTW, the new relevant notebook demo has been moved to notebooks/coadd_demo.ipynb , and the old demo notebook have been sequestered into notebooks/old .
            Hide
            yusra Yusra AlSayyad added a comment -

            I'm putting most of the comments in github but its easier to track a global wish list here:

            • Already talked about this, but I wish there was a template to repeat a functor on multiple filters, so we don't have to specify them 6X in the yaml.
            • We'll want to add functors outside the qa_explorer/functors module. What do you think of a registry?
            • The flux/magnitude calculations need to be right and we need FluxErrs and to use fluxMag0 error. I usually just copy: https://github.com/smonkewitz/scisql/blob/master/src/photometry.h
            • Need to adapt to the flux-->_instFlux change.
            Show
            yusra Yusra AlSayyad added a comment - I'm putting most of the comments in github but its easier to track a global wish list here: Already talked about this, but I wish there was a template to repeat a functor on multiple filters, so we don't have to specify them 6X in the yaml. We'll want to add functors outside the qa_explorer/functors module. What do you think of a registry? The flux/magnitude calculations need to be right and we need FluxErrs and to use fluxMag0 error. I usually just copy: https://github.com/smonkewitz/scisql/blob/master/src/photometry.h Need to adapt to the flux --> _instFlux change.

              People

              Assignee:
              tmorton Tim Morton [X] (Inactive)
              Reporter:
              tmorton Tim Morton [X] (Inactive)
              Reviewers:
              Yusra AlSayyad
              Watchers:
              Colin Slater, Hsin-Fang Chiang, Tim Morton [X] (Inactive), Yusra AlSayyad
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.