Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-40654

Multi-site CM MSTEST campaign submission with BPS and USDF PanDA

    XMLWordPrintable

Details

    • Story
    • Status: In Progress
    • Resolution: Unresolved
    • None
    • cm_prod, cm_tools, DM, PanDA
    • None
    • Campaign Management
    • No

    Description

      Use the CM tools and USDF based PanDA installation to process a multi-site test campaign (subset of HSC PDR2).

      Attachments

        Issue Links

          Activity

            yanny Brian Yanny added a comment -

            Butlers have been set up and populated at FRDF and USDF with repo names: hsc_pdr2_multisite.  At USDF, the /repo/main will continue to be used for this multisite exercise.  The contents of the repos for this multisite test and details on how they were populated are described in PREOPS-3833.  

            We are now proceeding to process the data at these three sites.  First the visits (about 2000 at each FRDF, UKDF  and 1000 at USDF) of 103 detectors (i.e. 200K 'quanta') will be put through steps 1, 2a and 2b at each local site.

            We will use a combination of direct bps submits and cm_tools (version 1) to launch and tract the processing.   Following the experience from DP0.2 and PDR2 campaigns, we will divide the visits up into 4-5 groups of about 400 visits (40K quanta) each for step 1 processing group.  We will use 5-pipetask clustering wherein the five pipetasks associated with step1 are grouped together into single PanDa jobs per detector, each running about 2-3 minutes.   If too many visits are submitted in one group, then the request memory must be increased on the pipetaskInit and finalJob pipetasks and quantum graph generation can take prohibitively long.  We aim for a couple of hours for qG generation (build) and finaljob (merging outputs back to local butlers).

            We'll start with stack w_2023_48, but can adapt if needed to include changes to the pipelines.   We understand some science code changes are still in progress.

            We will use the USDF PanDA for managing the workflows.  Currently running jobs are listed here:

            https://usdf-panda-bigmon.slac.stanford.edu:8443/idds/wfprogress/

            (use IAM authentication to gain access the first time).

            We will initially have three campaigns running at the three sites (FRDF, UKDF, USDF) with these names:

            HSC/runs/PDR2-VVDS-FR/w_2023_48_DM-40654

            HSC/runs/PDR2-VVDS-UK/w_2023_48_DM-40654

            HSC/runs/PDR2-VVDS/w_2023_48_DM-40654

            However, we may need to adjust these names as we learn how to bring outputs from FRDF and UKDF to USDF and merge them.

            We will use pipeline yaml,  requestMemory, and clustering directives in the DRP_PIPE product when possible, however these may need to be overridden in some circumstances.

            Jobs will be launched from the USDF nodes, at least initially.

            We note that the panda submissions with a bps submit <file.yaml> --compute-site <REMOTE_SITE> directive currently require the <file.yaml> to be sitting in the directory that the bps submit is sent from, i.e. it can't be given with a /path/file.yaml specification, otherwise the PanDA pilot job doesn't know where to find it at the remote sites. (We understand that this is being worked on by the PanDA team).

            We also used cm_tools code from an unmerged branch DM-39660 which allows for feeding of explicit lists of visits and/or tracts to a cm tools generation.  This is useful in generating groups of visits to process remotely when one doesn't have direct access to the butlers at the remote sites.  

            Other small adjustments to the cm tools scripting may be needed to handle the multisite submissions, and we will try and start a branch which has them or at least try and document these as we go.

             

            Following running of step1,2a,2b, the star catalogs from the output of the astrometrically registered visits from all three sites will be brought back to the USDF where step2c 'global photometric calibration' will be run on all catalogs simultaneously to get a global photometric solution consistent across the footprint.

             

            yanny Brian Yanny added a comment - Butlers have been set up and populated at FRDF and USDF with repo names: hsc_pdr2_multisite.  At USDF, the /repo/main will continue to be used for this multisite exercise.  The contents of the repos for this multisite test and details on how they were populated are described in PREOPS-3833.   We are now proceeding to process the data at these three sites.  First the visits (about 2000 at each FRDF, UKDF  and 1000 at USDF) of 103 detectors (i.e. 200K 'quanta') will be put through steps 1, 2a and 2b at each local site. We will use a combination of direct bps submits and cm_tools (version 1) to launch and tract the processing.   Following the experience from DP0.2 and PDR2 campaigns, we will divide the visits up into 4-5 groups of about 400 visits (40K quanta) each for step 1 processing group.  We will use 5-pipetask clustering wherein the five pipetasks associated with step1 are grouped together into single PanDa jobs per detector, each running about 2-3 minutes.   If too many visits are submitted in one group, then the request memory must be increased on the pipetaskInit and finalJob pipetasks and quantum graph generation can take prohibitively long.  We aim for a couple of hours for qG generation (build) and finaljob (merging outputs back to local butlers). We'll start with stack w_2023_48, but can adapt if needed to include changes to the pipelines.   We understand some science code changes are still in progress. We will use the USDF PanDA for managing the workflows.  Currently running jobs are listed here: https://usdf-panda-bigmon.slac.stanford.edu:8443/idds/wfprogress/ (use IAM authentication to gain access the first time). We will initially have three campaigns running at the three sites (FRDF, UKDF, USDF) with these names: HSC/runs/PDR2-VVDS-FR/w_2023_48_ DM-40654 HSC/runs/PDR2-VVDS-UK/w_2023_48_ DM-40654 HSC/runs/PDR2-VVDS/w_2023_48_ DM-40654 However, we may need to adjust these names as we learn how to bring outputs from FRDF and UKDF to USDF and merge them. We will use pipeline yaml,  requestMemory, and clustering directives in the DRP_PIPE product when possible, however these may need to be overridden in some circumstances. Jobs will be launched from the USDF nodes, at least initially. We note that the panda submissions with a bps submit <file.yaml> --compute-site <REMOTE_SITE> directive currently require the <file.yaml> to be sitting in the directory that the bps submit is sent from, i.e. it can't be given with a /path/file.yaml specification, otherwise the PanDA pilot job doesn't know where to find it at the remote sites. (We understand that this is being worked on by the PanDA team). We also used cm_tools code from an unmerged branch DM-39660 which allows for feeding of explicit lists of visits and/or tracts to a cm tools generation.  This is useful in generating groups of visits to process remotely when one doesn't have direct access to the butlers at the remote sites.   Other small adjustments to the cm tools scripting may be needed to handle the multisite submissions, and we will try and start a branch which has them or at least try and document these as we go.   Following running of step1,2a,2b, the star catalogs from the output of the astrometrically registered visits from all three sites will be brought back to the USDF where step2c 'global photometric calibration' will be run on all catalogs simultaneously to get a global photometric solution consistent across the footprint.  
            yanny Brian Yanny added a comment - - edited

            After the long break we are again attempting to send jobs to the three multi-processing sites:

            FRDF at CC-IN2P3

            UKDF at LANCS and

            USDF at SLAC.

            For the FRDF and UKDF sites, jobs can be submitted and run at the remote sites, except when it gets to the finalJob step we are now getting errors like this:

            ' Failed to execute payload:Transferring populated_by records like visit_definition requires a full Butler.'

             

            This was not occurring in december, so we're not yet sure of what's going on.  We will put more error info here shortly.

             

            Also, to help debug, we'd like to get login permission for the pilots (or lsstsvc1 account users at SLAC to be able to login directly to the equivalent accounts at FRDF and UKDF.  We can follow up with the reps at the remote data sites for this.

             

             More error messages (including 'connection pool full' can be found here:

            https://storage.googleapis.com/drp-us-central1-logging/logs/CC-IN2P3_Rubin_Merge/PandaJob_9237638/payload.stderr

             

            https://storage.googleapis.com/drp-us-central1-logging/logs/CC-IN2P3_Rubin_Merge/PandaJob_9237638/pilotlog.txt

             

            and here:

            https://usdf-panda-bigmon.slac.stanford.edu:8443/condor_logs/harvester-2/24-01-20_00/grid.1093021.0.err

            and here

            https://usdf-panda-bigmon.slac.stanford.edu:8443/condor_logs/harvester-2/24-01-20_00/grid.1093021.0.out

             

             

            yanny Brian Yanny added a comment - - edited After the long break we are again attempting to send jobs to the three multi-processing sites: FRDF at CC-IN2P3 UKDF at LANCS and USDF at SLAC. For the FRDF and UKDF sites, jobs can be submitted and run at the remote sites, except when it gets to the finalJob step we are now getting errors like this: ' Failed to execute payload:Transferring populated_by records like visit_definition requires a full Butler.'   This was not occurring in december, so we're not yet sure of what's going on.  We will put more error info here shortly.   Also, to help debug, we'd like to get login permission for the pilots (or lsstsvc1 account users at SLAC to be able to login directly to the equivalent accounts at FRDF and UKDF.  We can follow up with the reps at the remote data sites for this.    More error messages (including 'connection pool full' can be found here: https://storage.googleapis.com/drp-us-central1-logging/logs/CC-IN2P3_Rubin_Merge/PandaJob_9237638/payload.stderr   https://storage.googleapis.com/drp-us-central1-logging/logs/CC-IN2P3_Rubin_Merge/PandaJob_9237638/pilotlog.txt   and here: https://usdf-panda-bigmon.slac.stanford.edu:8443/condor_logs/harvester-2/24-01-20_00/grid.1093021.0.err and here https://usdf-panda-bigmon.slac.stanford.edu:8443/condor_logs/harvester-2/24-01-20_00/grid.1093021.0.out    
            yanny Brian Yanny added a comment - - edited

            ok, with Michelle Gower's help we used ticket https://jira.lsstcorp.org/browse/DM-42206 

            to get past the 'full butler' problem.  We're not sure if the panda v1.0.8 bps is in sync with this yet (or if we should not be loading this one or if there's an update). Anyway we now can successfully run to FRDF (and now working on UKDF) again. And will proceed with all of step1, 2a, 2b with w_2024_02 stack and v1.0.8 panda.  We need to explore the 'prun' command to see if we can launch a command to run remotely at a remote site for chain-collection and other operations like that, as using a special login account won't work for 'scripting' production.

             

            yanny Brian Yanny added a comment - - edited ok, with Michelle Gower's help we used ticket  https://jira.lsstcorp.org/browse/DM-42206   to get past the 'full butler' problem.  We're not sure if the panda v1.0.8 bps is in sync with this yet (or if we should not be loading this one or if there's an update). Anyway we now can successfully run to FRDF (and now working on UKDF) again. And will proceed with all of step1, 2a, 2b with w_2024_02 stack and v1.0.8 panda.  We need to explore the 'prun' command to see if we can launch a command to run remotely at a remote site for chain-collection and other operations like that, as using a special login account won't work for 'scripting' production.  

            We began testing 2.0 of the testing of multisite at USDF, FR and UK.

            We are using cm_prod/cm_tools and editing the config files so we can use cm v1 to make all of our bps files and use the database to keep track of what has/hasn't run and do error checking.

             

            To run, login to rubin-devl

            . venv311/bin/activate

            cd PDR2/

            source setup_env_usdf_2024_02_remote.sh

            cd cm_prod/

            This next step changes depending on which of the 3 sites you want to run at, each site has it's own setup file

            these are the lines that are different between the 3 different sites:

             

            UK:

            config="pdr2-vvds-uk.yaml"

            config_name="pdr2-vvds-uk"

            p_name="PDR2-VVDS-UK"

             

            FR: 

            config="pdr2-vvds-fr.yaml"

            config_name="pdr2-vvds-fr"

            p_name="PDR2-VVDS-FR"

             

            US:

            config="pdr2-vvds-us.yaml"

            config_name="pdr2-vvds-us"

            p_name="VVDS"

             

            to setup a specific site you would do:

            When we verify that things are working we will turn in our templates to the cm_prod ticket/DM-40654 for review.. which dosn't exist yet we need to make that branch still

            source examples/pdr2-vvds-fr_setup.sh

            Step1: ran fine in FR the errors are the "normal" errors that we would expect about too few stars for psf so we are ready to try experimenting with 2a in france

             

            In the UK: we had issues with things running really slowly, see:

            https://lsstc.slack.com/archives/C01J0QS3X70/p1706261294967269

            for details

            More debugging will happen next week so we are leaving the UK alone for the weekend.

             

            We realized we needed to re-run the USDF files with the updated stack today, so that is currently running.  If there are issues we will update the ticket.

             

            in UK we are having some issues with things running slowly, 

             

             

            Step1:

            The things we needed to modify were: 

            jena Jen Adelman-Mccarthy added a comment - We began testing 2.0 of the testing of multisite at USDF, FR and UK. We are using cm_prod/cm_tools and editing the config files so we can use cm v1 to make all of our bps files and use the database to keep track of what has/hasn't run and do error checking.   To run, login to rubin-devl . venv311/bin/activate cd PDR2/ source setup_env_usdf_2024_02_remote.sh cd cm_prod/ This next step changes depending on which of the 3 sites you want to run at, each site has it's own setup file these are the lines that are different between the 3 different sites:   UK: config="pdr2-vvds-uk.yaml" config_name="pdr2-vvds-uk" p_name="PDR2-VVDS-UK"   FR:  config="pdr2-vvds-fr.yaml" config_name="pdr2-vvds-fr" p_name="PDR2-VVDS-FR"   US: config="pdr2-vvds-us.yaml" config_name="pdr2-vvds-us" p_name="VVDS"   to setup a specific site you would do: When we verify that things are working we will turn in our templates to the cm_prod ticket/ DM-40654 for review.. which dosn't exist yet we need to make that branch still source examples/pdr2-vvds-fr_setup.sh Step1: ran fine in FR the errors are the "normal" errors that we would expect about too few stars for psf so we are ready to try experimenting with 2a in france   In the UK: we had issues with things running really slowly, see: https://lsstc.slack.com/archives/C01J0QS3X70/p1706261294967269 for details More debugging will happen next week so we are leaving the UK alone for the weekend.   We realized we needed to re-run the USDF files with the updated stack today, so that is currently running.  If there are issues we will update the ticket.   in UK we are having some issues with things running slowly,      Step1: The things we needed to modify were: 

            Testing 2a at the CC-IN2P3

            Because we can not yet run a collect step over panda at remote sites, we are essentially going to run step2a's for each step1 group using the step1/groupX outputs as inputs for step2a/groupX 

            this means that our groups for step1 and 2a are the same exactly there is no combining, and will allow us to at least start running 2a while we think about how we can run collect steps.

             

            For each site we can define the beginning of the campaign like this and run the same step/group chunks of data for steps 1 and 2a

            Step1

               |

            Step2a

             

             

            then we need to do the chaining of the outputs before we can start Step2b

            There may be a work around - rather chaining the outputs of 2a out we could just feed a comma separated list of outputs from the previous step into the next step.  This is viable for smaller datasets but becomes impractical at some point and we really do need to learn to properly chain collections across sites.

             

            after 2b we will need to make a formal chain collection and bring it back to the usdf using rucio

            To chain the collection make a small shell script that does the following:

            1) setup stack

            2)  chains the collection at the remote site, and uses the chained collection as the input to the next step

             

            we need to discuss still how these collections are best brought back to the usdf using rucio, and how do we track them before we can start 2c

             

            jena Jen Adelman-Mccarthy added a comment - Testing 2a at the CC-IN2P3 Because we can not yet run a collect step over panda at remote sites, we are essentially going to run step2a's for each step1 group using the step1/groupX outputs as inputs for step2a/groupX  this means that our groups for step1 and 2a are the same exactly there is no combining, and will allow us to at least start running 2a while we think about how we can run collect steps.   For each site we can define the beginning of the campaign like this and run the same step/group chunks of data for steps 1 and 2a Step1    | Step2a     then we need to do the chaining of the outputs before we can start Step2b There may be a work around - rather chaining the outputs of 2a out we could just feed a comma separated list of outputs from the previous step into the next step.  This is viable for smaller datasets but becomes impractical at some point and we really do need to learn to properly chain collections across sites.   after 2b we will need to make a formal chain collection and bring it back to the usdf using rucio To chain the collection make a small shell script that does the following: 1) setup stack 2)  chains the collection at the remote site, and uses the chained collection as the input to the next step   we need to discuss still how these collections are best brought back to the usdf using rucio, and how do we track them before we can start 2c  
            yanny Brian Yanny added a comment -

            Here is an update on where the multisite test stands as of the JTM on Feb 7, 2024:

            We have processed about 1500 expnums thru step1, step2a (visitSummary) and about 70 tracts thru step 2b (isolated_star_cat, isolated_star_sources) at  FRDF and about 800 expnums and 50 tracts thru step1, 2a and 2b at USDF.

            For UKDF, we are hitting a temporary issue with timeouts that is preventing large groups from getting through the finalJob step1.  We have bits and pieces of step2a thru at UKDF and may need a parameter setting adjustment (see: https://jira.lsstcorp.org/browse/DM-42709).

            We can get through small groups of step1,2a,2b at UKDF if necessary.  

            We have in place outputs from step2a and step2b which are needed for running step2c back at USDF, once the step2a (visitSummary) and step2b (isolated_star_cat, isolated_star_sources) datasetType outputs are transferred via Rucio back to USDF and registered in the VVDS run area in the butler.

            The name of the collections at FRDF with the outputs to transfer back via Rucio are:

            The butler repo at FRDF is named 'hsc_pdr2_multisite', and also at UKDF.

            The full path to this repo is also available if needed.

            For step2a outputs, all data of datasetType 'visitSummary' from these collections should be transferred back to USDF:

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2a/group0/w00_001

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2a/group1/w00_000

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2a/group2/w00_000

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2a/group3/w00_001

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2a/group4/w00_000

            For step2b outputs, all data of datasetTypes 'isolated_star_cat' and 'isolated_star_sources' from these collections should be transferred back to USDF:

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group0/w00_002

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group1/w00_001

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group2/w00_001

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group3/w00_000

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group4/w00_000

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group5/w00_000

            HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_DM-40654/step2b/group6/w00_000

             

            Please note that 'normally', these separate collections would be combined using collection-chain to make just one collection to search for outputs to transfer for each Rucio transfer operation.  We are still working to implement that 'collection-chain' remotely at FRDF and UKDF.

            At USDF, it's 'third' of the step1,step2a,step2b outputs are in these collections in /repo/main butler

            HSC/runs/PDR2/VVDS/w_2024_02_DM-40654/step1 (chained collection of 3 groups)

            HSC/runs/PDR2/VVDS/w_2024_02_DM-40654/step2a (chained collection of 3 groups)

            HSC/runs/PDR2/VVDS/w_2024_02_DM-40654/step2b (chained collection of 7 groups)

             

            The plan is to transfer and register objects with datasetTypes visitSummary, isolated_star_cat and isolated_star_sources from FRDF and UKDF (if available) from FRDF to USDF. The collection  names from FRDF can be probably 'kept the same' when these collection elements are registered in USDF.

             

            Once these are in place we are ready to run step2c (fgcm global photometric calibration) at USDF.

            Then the outputs of that calibration can be transferred back to FRDF, UKDF for applying in step2d, step2e step3, ... and processing can continue there.

            We are working to integrate 'collection chaining' and 'remote butler query setup for grouping expnum/tract sets at remote sites' within the CM tools/ cm service functionality to reduce the number of collections that need to be searched/transferred via rucio.

             

             

            yanny Brian Yanny added a comment - Here is an update on where the multisite test stands as of the JTM on Feb 7, 2024: We have processed about 1500 expnums thru step1, step2a (visitSummary) and about 70 tracts thru step 2b (isolated_star_cat, isolated_star_sources) at  FRDF and about 800 expnums and 50 tracts thru step1, 2a and 2b at USDF. For UKDF, we are hitting a temporary issue with timeouts that is preventing large groups from getting through the finalJob step1.  We have bits and pieces of step2a thru at UKDF and may need a parameter setting adjustment (see: https://jira.lsstcorp.org/browse/DM-42709 ). We can get through small groups of step1,2a,2b at UKDF if necessary.   We have in place outputs from step2a and step2b which are needed for running step2c back at USDF, once the step2a (visitSummary) and step2b (isolated_star_cat, isolated_star_sources) datasetType outputs are transferred via Rucio back to USDF and registered in the VVDS run area in the butler. The name of the collections at FRDF with the outputs to transfer back via Rucio are: The butler repo at FRDF is named 'hsc_pdr2_multisite', and also at UKDF. The full path to this repo is also available if needed. For step2a outputs, all data of datasetType 'visitSummary' from these collections should be transferred back to USDF: HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2a/group0/w00_001 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2a/group1/w00_000 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2a/group2/w00_000 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2a/group3/w00_001 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2a/group4/w00_000 For step2b outputs, all data of datasetTypes 'isolated_star_cat' and 'isolated_star_sources' from these collections should be transferred back to USDF: HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group0/w00_002 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group1/w00_001 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group2/w00_001 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group3/w00_000 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group4/w00_000 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group5/w00_000 HSC/runs/PDR2/PDR2-VVDS-FR/w_2024_02_ DM-40654 /step2b/group6/w00_000   Please note that 'normally', these separate collections would be combined using collection-chain to make just one collection to search for outputs to transfer for each Rucio transfer operation.  We are still working to implement that 'collection-chain' remotely at FRDF and UKDF. At USDF, it's 'third' of the step1,step2a,step2b outputs are in these collections in /repo/main butler HSC/runs/PDR2/VVDS/w_2024_02_ DM-40654 /step1 (chained collection of 3 groups) HSC/runs/PDR2/VVDS/w_2024_02_ DM-40654 /step2a (chained collection of 3 groups) HSC/runs/PDR2/VVDS/w_2024_02_ DM-40654 /step2b (chained collection of 7 groups)   The plan is to transfer and register objects with datasetTypes visitSummary, isolated_star_cat and isolated_star_sources from FRDF and UKDF (if available) from FRDF to USDF. The collection  names from FRDF can be probably 'kept the same' when these collection elements are registered in USDF.   Once these are in place we are ready to run step2c (fgcm global photometric calibration) at USDF. Then the outputs of that calibration can be transferred back to FRDF, UKDF for applying in step2d, step2e step3, ... and processing can continue there. We are working to integrate 'collection chaining' and 'remote butler query setup for grouping expnum/tract sets at remote sites' within the CM tools/ cm service functionality to reduce the number of collections that need to be searched/transferred via rucio.    
            yanny Brian Yanny added a comment - - edited

            The step1,2a,2b outputs are now also available from the UKDF.

            The butler name is: /cephfs/grid/lsst/repos/hsc_pdr2_multisite

            The step2a run collections (with the visitSummary datasetType datasets) are called:

              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group0/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group1/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group2/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group3/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group4/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2a/group5/w00_000 

            The step2b run collections (with the isolated_star_cat and isolated_star_sources datasetTypes datasets) are called:

              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2b/group0/w00_001 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2b/group1/w00_000 
              HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_DM-40654/step2b/group2/w00_001 

             

            The size of visitSummary is about 1.3MB/visit (3 GB/data facility in 1K-2K files)

            The size of isolated_star_cat and isolated_star_sources are about 0.1-3MB/tract (<1GB/data facility in 100-200 files)

             

            yanny Brian Yanny added a comment - - edited The step1,2a,2b outputs are now also available from the UKDF. The butler name is: /cephfs/grid/lsst/repos/hsc_pdr2_multisite The step2a run collections (with the visitSummary datasetType datasets) are called:   HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group0/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group1/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group2/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group3/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group4/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2a/group5/w00_000  The step2b run collections (with the isolated_star_cat and isolated_star_sources datasetTypes datasets) are called:   HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2b/group0/w00_001    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2b/group1/w00_000    HSC/runs/PDR2/PDR2-UK-VVDS/w_2024_02_ DM-40654 /step2b/group2/w00_001    The size of visitSummary is about 1.3MB/visit (3 GB/data facility in 1K-2K files) The size of isolated_star_cat and isolated_star_sources are about 0.1-3MB/tract (<1GB/data facility in 100-200 files)  

            People

              yanny Brian Yanny
              yanny Brian Yanny
              Brian Yanny, Eric Charles, Fabio Hernandez, George Beckett, Jen Adelman-Mccarthy, Kian-Tat Lim, Michelle Gower, Orion Eiger, Peter Love, Quentin Le Boulc'h, Sierra Villarreal, Steve Pietrowicz, Tim Jenness, Wen Guan, Yusra AlSayyad, Zhaoyu Yang
              Votes:
              0 Vote for this issue
              Watchers:
              16 Start watching this issue

              Dates

                Created:
                Updated:

                Jenkins

                  No builds found.