Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38743

Get familiarized with running ap_verify from the command line.

    XMLWordPrintable

    Details

    • Type: Story
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: None
    • Labels:
      None

      Description

      Harshit Rai is getting himself familiarized with running ap_verify from the command line – not the standard "jenkins run".

        Attachments

          Issue Links

            Activity

            Hide
            sullivan Ian Sullivan added a comment -

            The notes below will be helpful to get started running ap_verify from the command line: https://pipelines.lsst.io/modules/lsst.ap.verify/running.html

            Note that you can run these commands either on your laptop or at the USDF, though in either case you will have to download and setup the appropriate data repository.

            Show
            sullivan Ian Sullivan added a comment - The notes below will be helpful to get started running ap_verify from the command line: https://pipelines.lsst.io/modules/lsst.ap.verify/running.html Note that you can run these commands either on your laptop or at the USDF, though in either case you will have to download and setup the appropriate data repository.
            Hide
            harshit Harshit Rai added a comment -

            Ian Sullivan  Nima Sedaghat  I managed to run ap_verify.py package from command line on my personal machine as well as on USDF. I ran the package on all three datasets- hits_2015, cosmos_pdr2, and dc2- without any issues. I can now start looking into the package code if it will be helpful in my next tasks. 

            Show
            harshit Harshit Rai added a comment - Ian Sullivan   Nima Sedaghat   I managed to run ap_verify.py package from command line on my personal machine as well as on USDF. I ran the package on all three datasets- hits_2015, cosmos_pdr2, and dc2- without any issues. I can now start looking into the package code if it will be helpful in my next tasks. 
            Hide
            harshit Harshit Rai added a comment -

            To run ap_verify.py on the USDF
            1. Run the following command to load the environment-
               source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_15/loadLSST-ext.bash

            2. Run the setup command to enter the setup mode-
               setup lsst_distrib -t w_2023_15     (should say setup)

            3. Download the dataset on which you wish to run ap_verify.py
               git clone <git-repo-of-the-dataset>

            4. Once the dataset is downloaded, run the setup using the following command-
               setup -r /path/to/my/downloaded/dataset

            5, Run ap_verify.py on USDF using the following command-
               ap_verify.py --dataset <name-of-the-dataset> -j<number-of-processes> --output /path/for/butler/repo

             

            Show
            harshit Harshit Rai added a comment - To run ap_verify.py on the USDF 1. Run the following command to load the environment-     source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_15/loadLSST-ext.bash 2. Run the setup command to enter the setup mode-     setup lsst_distrib -t w_2023_15     (should say setup) 3. Download the dataset on which you wish to run ap_verify.py     git clone <git-repo-of-the-dataset> 4. Once the dataset is downloaded, run the setup using the following command-     setup -r /path/to/my/downloaded/dataset 5, Run ap_verify.py on USDF using the following command-     ap_verify.py --dataset <name-of-the-dataset> -j<number-of-processes> --output /path/for/butler/repo  
            Hide
            harshit Harshit Rai added a comment - - edited

            Nima Sedaghat and I got on a call to discuss this ticket in more detail. Here are some points-

            1. The previous method of sourcing loadLSST-ext.bash uses a read-only version of the installed stack- which wont be very useful in our case. So, getting access to the shared stack made more sense.
            2. I used the following command to source from the shared stack-
              source "/sdf/group/rubin/sw/d_latest/loadLSST.bash"
            3. After the above step, I used the following command to enter the setup mode-
              setup lsst_distrib
            4. To confirm that packages from the shared stack are being used, I used the following command-
              which ap_verify.py
              It generated the output- /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/ap_verify/g471cda119c+5db0c99945/bin/ap_verify.py 
            5. After this, I cloned the dc2 dataset and registered it in the environment using the commands mentioned in my previous comment.
            6. Then I ran the ap_verify.py on this dataset. The pipeline run completed successfully.
            7. We wanted to run the ap_verify.py from the tickets/DM-36743 branch. So I cloned the repo, switched to that branch, and entered the setup mode using the command setup -kr .
            8. Re-running the following commands confirmed that ap_verify from the downloaded repo was being used-
              which ap_verify.py => /sdf/data/rubin/user/harshit/ap_verify/bin/ap_verify.py
              eups list | grep ap_verify.py => ap_verify LOCAL:/sdf/data/rubin/user/harshit/ap_verify setup
            9. I reran the ap_verify.py on the dc2 data, but this time, the pipeline run failed with the following error-
              ImportError: Could not get attribute 'transiNet' from 'lsst.meas' when importing 'lsst.meas.transiNet.TransiNetTask' (No module named 'lsst.meas.transiNet') 
              This behavior is in line with our expectation.
            Show
            harshit Harshit Rai added a comment - - edited Nima Sedaghat  and I got on a call to discuss this ticket in more detail. Here are some points- The previous method of sourcing loadLSST-ext.bash uses a read-only version of the installed stack- which wont be very useful in our case. So, getting access to the shared stack made more sense. I used the following command to source from the shared stack- source "/sdf/group/rubin/sw/d_latest/loadLSST.bash" After the above step, I used the following command to enter the setup mode- setup lsst_distrib To confirm that packages from the shared stack are being used, I used the following command- which ap_verify.py It generated the output-  /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/ap_verify/g471cda119c+5db0c99945/bin/ap_verify.py   After this, I cloned the dc2 dataset and registered it in the environment using the commands mentioned in my previous comment. Then I ran the ap_verify.py on this dataset. The pipeline run completed successfully. We wanted to run the ap_verify.py from the tickets/ DM-36743 branch . So I cloned the repo, switched to that branch, and entered the setup mode using the command setup -kr . Re-running the following commands confirmed that ap_verify from the downloaded repo was being used- which ap_verify.py =>  /sdf/data/rubin/user/harshit/ap_verify/bin/ap_verify.py eups list | grep ap_verify.py => ap_verify LOCAL:/sdf/data/rubin/user/harshit/ap_verify setup I reran the ap_verify.py on the dc2 data, but this time, the pipeline run failed with the following error- ImportError: Could not get attribute 'transiNet' from 'lsst.meas' when importing 'lsst.meas.transiNet.TransiNetTask' (No module named 'lsst.meas.transiNet')   This behavior is in line with our expectation.

              People

              Assignee:
              harshit Harshit Rai
              Reporter:
              nima Nima Sedaghat
              Reviewers:
              Nima Sedaghat
              Watchers:
              Eric Bellm, Harshit Rai, Ian Sullivan, Nima Sedaghat
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.