Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-22990

LOE: Gen3 Activities

    XMLWordPrintable

    Details

    • Type: Epic
    • Status: In Progress
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ctrl_bps
    • Labels:
      None
    • Epic Name:
      LOE: Gen3 Activities
    • Story Points:
      44
    • WBS:
      02C.07.08
    • Team:
      Data Facility

      Description

      Facilitate transition from Gen2 to Gen3 Middleware. Bucket epic to deal with emergent supporting issues.

      This ticket had been created automatically
      Primavera ID: A7550

        Attachments

        Stories in Epic (Custom Issue Matrix)

        Key Summary Story Points Assignee Status
        DM-33550

        Support different HTCondor walltime syntax

        Unassigned To Do
        DM-28932

        bps documentation improvements from initial users

        4 Unassigned To Do
        DM-37138

        Allow the user to override BPS configuration settings on the command line

        Mikolaj Kowalik To Do
        DM-37131

        Allow ctrl_bps to pass plugin specific options

        Mikolaj Kowalik To Do
        DM-35145

        Add ping method to HTCondor plugin

        Unassigned To Do
        DM-37020

        Replace "requirements" with "constraint" in HTCondor Python function calls in ctrl_bps_htcondor

        Mikolaj Kowalik To Do
        DM-32439

        Make bps report support filtering jobs based on a run pattern

        Unassigned To Do
        DM-30659

        Modify bps Pegasus plugin to provide info needed for updated reporting of failures

        Unassigned To Do
        DM-32871

        Change ctrl_bps pathlib and os.path to lsst.resources.ResourcePath

        Unassigned To Do
        DM-31246

        bps report should show job info separately from Quanta info

        Unassigned To Do
        DM-31245

        rewrite BpsConfig

        Unassigned To Do
        DM-31237

        Update Pegasus plugin to handle mergeExecutionButler final job.

        Unassigned To Do
        DM-28682

        Basic BPS plugin that could be run in ci tests

        20 Unassigned To Do
        DM-29634

        bps HTCondor needs to excape quotes in run command lines

        Unassigned To Do
        DM-29851

        Show total counts in run summary

        4 Unassigned To Do
        DM-29845

        Throttle jobs after many failures

        Unassigned To Do
        DM-31879

        Reduce number of files created by bps HTCondor plugin

        6 Unassigned To Do
        DM-16085

        Write BPS design document

        9 Michelle Gower To Do
        DM-31511

        Prototype bps loading pipetask json log files to elasticsearch

        Michelle Gower In Progress
        DM-29559

        Add bps report option to list Quanta failures.

        4 Michelle Gower In Progress
         
        DM-22994

        Run ci_hsc_gen3 with weekly stack + Oracle

        Michelle Gower Done
         
        DM-22993

        Run ci_hsc_gen3 with weekly stack + BPS

        Michelle Gower Done
         
        DM-33331

        Monitor memory usage at select steps of BPS submission

        Mikolaj Kowalik Done
         
        DM-33267

        Enable black/isort on ctrl_bps

        Mikolaj Kowalik Done
         
        DM-23497

        Run RC2 bootstrap with Oracle

        Michelle Gower Done
         
        DM-28825

        Create new main Gen3 repo schema

        Christopher Stephens [X] (Inactive) Done
         
        DM-28801

        Add documentation to LSST developer guide for using BPS on NCSA resources

        0.5 Mikolaj Kowalik Done
         
        DM-29918

        Investigate bad cgroup/HTCondor memory analysis in BPS vs. fgcmcal

        Michelle Gower Done
         
        DM-29893

        un-hardcode bps report output

        Mikolaj Kowalik Done
         
        DM-29007

        Lazy command line creation

        Michelle Gower Done
         
        DM-28960

        Add CHAINED collection creation to BPS

        Michelle Gower Done
         
        DM-28929

        bps incorrectly assumes htcondor attribute is a string

        1 Michelle Gower Done
         
        DM-26844

        Modify BPS configuration class to allow initialization from a dictionary

        Mikolaj Kowalik Done
         
        DM-26829

        RC2 gen3 + bps run with w_2020_37 stack

        Monika Adamow Done
         
        DM-26734

        Ensure that logs from PipelineTasks are stored and available to BPS users

        Michelle Gower Done
         
        DM-30104

        Make pickling the clustered quantum graph optional

        Mikolaj Kowalik Done
         
        DM-29756

        BPS - Automatic retry with larger memory

        Mikolaj Kowalik Done
         
        DM-28560

        DC2 w_2020_48 conversion to gen3

        Monika Adamow Done
         
        DM-28248

        Convert RC2 w_2020_50 with w_2021_02 stack

        Monika Adamow Done
         
        DM-26249

        Update ctrl_bps to meet LSST coding standards

        Michelle Gower Done
         
        DM-26238

        Convert RC2 w_2020_30 to gen3 repo

        Monika Adamow Done
         
        DM-25297

        Convert RC2 Gen2 run to Gen3 as test of process and code

        Michelle Gower Done
         
        DM-35066

        Print number of clusters in clustered graph

        Michelle Gower Done
         
        DM-34959

        Replace read_gpickle and write_gpickle in GenericWorkflow

        Mikolaj Kowalik Done
         
        DM-32594

        Remove dependency on inflection in bps_config

        Michelle Gower Done
         
        DM-31691

        pipetask doesn't find dataset in execution butler from previous run.

        Michelle Gower Done
         
        DM-31043

        Update ctrl_bps setup.cfg with flake8's max-doc-length=79

        2 Mikolaj Kowalik Done
         
        DM-30643

        Gen3 UUID database migration

        1 Christopher Stephens [X] (Inactive) Done
         
        DM-29305

        NCSA cluster plus pip installed htcondor

        4 Mikolaj Kowalik Done
         
        DM-27676

        Add ability to remove submitted runs

        6 Michelle Gower Done
         
        DM-27675

        Add ability to pass single large QuantumGraph to jobs.

        10 Michelle Gower Done
         
        DM-27648

        Make bps' CLI --log-level behavior consistent

        2 Mikolaj Kowalik Done
         
        DM-27517

        Convert RC2 w_2020_42 with w_2020_46 stack

        Monika Adamow Done
         
        DM-27498

        Convert RC2 w_2020_42 with w_2020_45 stack

        Monika Adamow Done
         
        DM-27434

        Add ctrl_bps to lsst_distrib package

        Michelle Gower Done
         
        DM-27888

        Filename passed to QuantumGraphs' save() should have extension .qgraph

        Mikolaj Kowalik Done
         
        DM-27782

        Convert RC2 v21_0_0_rc1 to gen3 with w_2020_48 stack

        Monika Adamow Done
         
        DM-27571

        Gen3 RC2 run with w_2020_46 stack

        Monika Adamow Done
         
        DM-27570

        bps report id not showing running jobs in Total line

        6 Michelle Gower Done
         
        DM-29953

        bps submit TypeError about SerializedDataCoordinate

        Michelle Gower Done
         
        DM-29575

        bps restart command

        Mikolaj Kowalik Done
         
        DM-28653

        Modify BPS to support execution butler

        10 Michelle Gower Done
         
        DM-28345

        RC2 w_2021_02 gen3 conversion with w_2021_03 stack

        Monika Adamow Done
         
        DM-27379

        RC2 w_2020_42 gen3 conversion with w_2020_44 stack

        Monika Adamow Done
         
        DM-25821

        Convert RC2 w_2020_26 to gen3 repo

        Michelle Gower Done
         
        DM-29665

        DB error messages in DC2 run

        Christopher Stephens [X] (Inactive) Done
         
        DM-29614

        Have bps report show info from multiple submit nodes

        Mikolaj Kowalik Done
         
        DM-27113

        Convert RC2 w_2020_38 to gen3 with w_2020_42 stack

        Monika Adamow Done
         
        DM-27049

        bpsub printing many warning messages about Missing search section

        Michelle Gower Done
         
        DM-26016

        Update BPS to take pipeline yaml as input

        Michelle Gower Done
         
        DM-30056

        requestCpus not ending up in HTCondor submit files

        1 Michelle Gower Done
         
        DM-27039

        Make changes to bps_core in preparation for more subcommands.

        Michelle Gower Done
         
        DM-26494

        Add QuantumGraph generation from pipeline yaml command line to BPS config

        Mikolaj Kowalik Done
         
        DM-26458

        Update ctrl_bps to work with new QuantumGraph implementation

        Michelle Gower Done
         
        DM-26408

        Implement initial set of unit tests for ctrl_bps

        Mikolaj Kowalik Done
         
        DM-26402

        Change ctrl_bps commands to use click

        Michelle Gower Done
         
        DM-26401

        Investigate delays before job starts in the RC2 single tract run

        Greg Daues Done
         
        DM-26397

        Add explicit DAG export from ctrl_bps

        2 Mikolaj Kowalik Done
         
        DM-26385

        Combine --init-only into single compute job

        Michelle Gower Done
         
        DM-26271

        Run RC2 one tract in Gen3 w_2020_31

        Monika Adamow Done
         
        DM-36849

        Create BPS poster for ADASS 2022

        4 Michelle Gower Done
         
        DM-32241

        bps submit can take multiple hours

        Michelle Gower Done
         
        DM-32220

        HTCondor plugin copying execution butler back from job

        1 Michelle Gower Done
         
        DM-32217

        BPS displays cumulative timings instead of time intervals during a submission

        Mikolaj Kowalik Done
         
        DM-32201

        HTCondor plugin bug if output collection contains period

        Michelle Gower Done
         
        DM-32199

        If submit or prepare, have bps check early that WMS is in environment.

        4 Michelle Gower Done
         
        DM-31970

        Fix submission AttributeError in htcondor plugin when not using execution butler

        Michelle Gower Done
         
        DM-31944

        concurrency_limit split in htcondor submit file

        Michelle Gower Done
         
        DM-32047

        Add a requestMemoryMax config in ctrl_bps

        Mikolaj Kowalik Done
         
        DM-32027

        Rename outCollection to outputRun

        1 Michelle Gower Done
         
        DM-32074

        Modify how HTCondor plugin handles transfer of execution butler

        Michelle Gower Done
         
        DM-32066

        BPS jobs with memory autoscaling enabled remain idle after the first run attempt

        Mikolaj Kowalik Done
         
        DM-31541

        Change pipetaskInit job to not read full QuantumGraph

        4 Mikolaj Kowalik Done
         
        DM-27666

        Command line has full path for QuantumGraph filename

        2 Michelle Gower Done
         
        DM-25310

        Run ci_hsc_gen3 with weekly stack + PostgreSQL

        Monika Adamow Done
         
        DM-27575

        requestMemory not ending up in submit files.

        Michelle Gower Done
         
        DM-36848

        Write BPS paper for ADASS 2022

        Michelle Gower Done
         
        DM-25005

        Problems using HTCondor API + LSST stack based upon import order

        Michelle Gower Won't Fix
         
        DM-30827

        Allow user to specify number of retries in bps yaml.

        2 Unassigned Invalid
         
        DM-32754

        Catch execution butler creation error

        Unassigned Invalid
         
        DM-31199

        bps pipetaskInit sometimes needs lots of memory

        Unassigned Invalid
         
        DM-29574

        Save bps yaml

        Unassigned Invalid
         
        DM-29560

        Update bps log directory structure

        Unassigned Invalid

          Activity

          There are no comments yet on this issue.

            People

            Assignee:
            mgower Michelle Gower
            Reporter:
            mgower Michelle Gower
            Watchers:
            Vladislav Ekimtov [X] (Inactive)
            Votes:
            0 Vote for this issue
            Watchers:
            1 Start watching this issue

              Dates

              Created:
              Updated:
              Start date:
              End date:

                Jenkins

                No builds found.