Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-38743

Get familiarized with running ap_verify from the command line.

    XMLWordPrintable

Details

    • Story
    • Status: Done
    • Resolution: Done
    • None
    • None
    • None

    Description

      harshit is getting himself familiarized with running ap_verify from the command line – not the standard "jenkins run".

      Attachments

        Issue Links

          Activity

            No builds found.
            nima Nima Sedaghat created issue -
            nima Nima Sedaghat made changes -
            Field Original Value New Value
            Link This issue relates to DM-37472 [ DM-37472 ]
            nima Nima Sedaghat made changes -
            Description Harshit is getting himself familiarized with running ap_verify from the command line – not the standard "jenkins run". [~harshit] is getting himself familiarized with running ap_verify from the command line – not the standard "jenkins run".
            nima Nima Sedaghat made changes -
            Link This issue relates to DM-37472 [ DM-37472 ]
            nima Nima Sedaghat made changes -
            Link This issue relates to DM-36742 [ DM-36742 ]
            sullivan Ian Sullivan added a comment -

            The notes below will be helpful to get started running ap_verify from the command line: https://pipelines.lsst.io/modules/lsst.ap.verify/running.html

            Note that you can run these commands either on your laptop or at the USDF, though in either case you will have to download and setup the appropriate data repository.

            sullivan Ian Sullivan added a comment - The notes below will be helpful to get started running ap_verify from the command line: https://pipelines.lsst.io/modules/lsst.ap.verify/running.html Note that you can run these commands either on your laptop or at the USDF, though in either case you will have to download and setup the appropriate data repository.
            sullivan Ian Sullivan made changes -
            Epic Link DM-36523 [ 2254240 ]
            Sprint AP S23-5 (April) [ 1227 ]
            Story Points 2
            krzys Krzysztof Findeisen made changes -
            Link This issue relates to DM-36477 [ DM-36477 ]
            harshit Harshit Rai added a comment -

            sullivan  nima  I managed to run ap_verify.py package from command line on my personal machine as well as on USDF. I ran the package on all three datasets- hits_2015, cosmos_pdr2, and dc2- without any issues. I can now start looking into the package code if it will be helpful in my next tasks. 

            harshit Harshit Rai added a comment - sullivan   nima   I managed to run ap_verify.py package from command line on my personal machine as well as on USDF. I ran the package on all three datasets- hits_2015, cosmos_pdr2, and dc2- without any issues. I can now start looking into the package code if it will be helpful in my next tasks. 
            harshit Harshit Rai made changes -
            Status To Do [ 10001 ] In Progress [ 3 ]
            harshit Harshit Rai made changes -
            Reviewers Nima Sedaghat [ nima ]
            Status In Progress [ 3 ] In Review [ 10004 ]
            harshit Harshit Rai made changes -
            Comment [ Ran the ap_verify.py package on local machine as well as on USDF cluster. Processes ran fine without any errors/issues. ]
            harshit Harshit Rai made changes -
            Attachment image-2023-04-18-14-50-00-632.png [ 67549 ]
            harshit Harshit Rai added a comment -

            To run ap_verify.py on the USDF
            1. Run the following command to load the environment-
               source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_15/loadLSST-ext.bash

            2. Run the setup command to enter the setup mode-
               setup lsst_distrib -t w_2023_15     (should say setup)

            3. Download the dataset on which you wish to run ap_verify.py
               git clone <git-repo-of-the-dataset>

            4. Once the dataset is downloaded, run the setup using the following command-
               setup -r /path/to/my/downloaded/dataset

            5, Run ap_verify.py on USDF using the following command-
               ap_verify.py --dataset <name-of-the-dataset> -j<number-of-processes> --output /path/for/butler/repo

             

            harshit Harshit Rai added a comment - To run ap_verify.py on the USDF 1. Run the following command to load the environment-     source /cvmfs/sw.lsst.eu/linux-x86_64/lsst_distrib/w_2023_15/loadLSST-ext.bash 2. Run the setup command to enter the setup mode-     setup lsst_distrib -t w_2023_15     (should say setup) 3. Download the dataset on which you wish to run ap_verify.py     git clone <git-repo-of-the-dataset> 4. Once the dataset is downloaded, run the setup using the following command-     setup -r /path/to/my/downloaded/dataset 5, Run ap_verify.py on USDF using the following command-     ap_verify.py --dataset <name-of-the-dataset> -j<number-of-processes> --output /path/for/butler/repo  
            harshit Harshit Rai added a comment - - edited

            nima and I got on a call to discuss this ticket in more detail. Here are some points-

            1. The previous method of sourcing loadLSST-ext.bash uses a read-only version of the installed stack- which wont be very useful in our case. So, getting access to the shared stack made more sense.
            2. I used the following command to source from the shared stack-
              source "/sdf/group/rubin/sw/d_latest/loadLSST.bash"
            3. After the above step, I used the following command to enter the setup mode-
              setup lsst_distrib
            4. To confirm that packages from the shared stack are being used, I used the following command-
              which ap_verify.py
              It generated the output- /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/ap_verify/g471cda119c+5db0c99945/bin/ap_verify.py 
            5. After this, I cloned the dc2 dataset and registered it in the environment using the commands mentioned in my previous comment.
            6. Then I ran the ap_verify.py on this dataset. The pipeline run completed successfully.
            7. We wanted to run the ap_verify.py from the tickets/DM-36743 branch. So I cloned the repo, switched to that branch, and entered the setup mode using the command setup -kr .
            8. Re-running the following commands confirmed that ap_verify from the downloaded repo was being used-
              which ap_verify.py => /sdf/data/rubin/user/harshit/ap_verify/bin/ap_verify.py
              eups list | grep ap_verify.py => ap_verify LOCAL:/sdf/data/rubin/user/harshit/ap_verify setup
            9. I reran the ap_verify.py on the dc2 data, but this time, the pipeline run failed with the following error-
              ImportError: Could not get attribute 'transiNet' from 'lsst.meas' when importing 'lsst.meas.transiNet.TransiNetTask' (No module named 'lsst.meas.transiNet') 
              This behavior is in line with our expectation.
            harshit Harshit Rai added a comment - - edited nima  and I got on a call to discuss this ticket in more detail. Here are some points- The previous method of sourcing loadLSST-ext.bash uses a read-only version of the installed stack- which wont be very useful in our case. So, getting access to the shared stack made more sense. I used the following command to source from the shared stack- source "/sdf/group/rubin/sw/d_latest/loadLSST.bash" After the above step, I used the following command to enter the setup mode- setup lsst_distrib To confirm that packages from the shared stack are being used, I used the following command- which ap_verify.py It generated the output-  /sdf/group/rubin/sw/conda/envs/lsst-scipipe-5.1.0/share/eups/Linux64/ap_verify/g471cda119c+5db0c99945/bin/ap_verify.py   After this, I cloned the dc2 dataset and registered it in the environment using the commands mentioned in my previous comment. Then I ran the ap_verify.py on this dataset. The pipeline run completed successfully. We wanted to run the ap_verify.py from the tickets/ DM-36743 branch . So I cloned the repo, switched to that branch, and entered the setup mode using the command setup -kr . Re-running the following commands confirmed that ap_verify from the downloaded repo was being used- which ap_verify.py =>  /sdf/data/rubin/user/harshit/ap_verify/bin/ap_verify.py eups list | grep ap_verify.py => ap_verify LOCAL:/sdf/data/rubin/user/harshit/ap_verify setup I reran the ap_verify.py on the dc2 data, but this time, the pipeline run failed with the following error- ImportError: Could not get attribute 'transiNet' from 'lsst.meas' when importing 'lsst.meas.transiNet.TransiNetTask' (No module named 'lsst.meas.transiNet')   This behavior is in line with our expectation.
            nima Nima Sedaghat made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            harshit Harshit Rai made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            sullivan Ian Sullivan made changes -
            Rank Ranked higher

            People

              harshit Harshit Rai
              nima Nima Sedaghat
              Nima Sedaghat
              Eric Bellm, Harshit Rai, Ian Sullivan, Nima Sedaghat
              Votes:
              0 Vote for this issue
              Watchers:
              4 Start watching this issue

              Dates

                Created:
                Updated:
                Resolved:

                Jenkins

                  No builds found.