[DM-22990] LOE: Gen3 Activities - Jira

XML

Word

Printable

Details

Type: Epic
Status: Done
Resolution: Done
Fix Version/s: None
Component/s: ctrl_bps
Labels:
None

Epic Name:
LOE: Gen3 Activities
Story Points:
44
WBS:
02C.07.08
Team:
Data Facility

Description

Facilitate transition from Gen2 to Gen3 Middleware. Bucket epic to deal with emergent supporting issues.

This ticket had been created automatically
Primavera ID: A7550

Attachments

Stories in Epic (Custom Issue Matrix)

Key	Summary	Story Points	Assignee	Status
~~DM-33331~~	Monitor memory usage at select steps of BPS submission		Mikolaj Kowalik	Done
~~DM-33267~~	Enable black/isort on ctrl_bps		Mikolaj Kowalik	Done
~~DM-35066~~	Print number of clusters in clustered graph		Michelle Gower	Done
~~DM-34959~~	Replace read_gpickle and write_gpickle in GenericWorkflow		Mikolaj Kowalik	Done
~~DM-30104~~	Make pickling the clustered quantum graph optional		Mikolaj Kowalik	Done
~~DM-30056~~	requestCpus not ending up in HTCondor submit files	1	Michelle Gower	Done
~~DM-29953~~	bps submit TypeError about SerializedDataCoordinate		Michelle Gower	Done
~~DM-29918~~	Investigate bad cgroup/HTCondor memory analysis in BPS vs. fgcmcal		Michelle Gower	Done
~~DM-29893~~	un-hardcode bps report output		Mikolaj Kowalik	Done
~~DM-30643~~	Gen3 UUID database migration	1	Christopher Stephens [X] (Inactive)	Done
~~DM-28801~~	Add documentation to LSST developer guide for using BPS on NCSA resources	0.5	Mikolaj Kowalik	Done
~~DM-28825~~	Create new main Gen3 repo schema		Christopher Stephens [X] (Inactive)	Done
~~DM-27782~~	Convert RC2 v21_0_0_rc1 to gen3 with w_2020_48 stack		Monika Adamow	Done
~~DM-27676~~	Add ability to remove submitted runs	6	Michelle Gower	Done
~~DM-27675~~	Add ability to pass single large QuantumGraph to jobs.	10	Michelle Gower	Done
~~DM-27666~~	Command line has full path for QuantumGraph filename	2	Michelle Gower	Done
~~DM-27648~~	Make bps' CLI --log-level behavior consistent	2	Mikolaj Kowalik	Done
~~DM-32241~~	bps submit can take multiple hours		Michelle Gower	Done
~~DM-29007~~	Lazy command line creation		Michelle Gower	Done
~~DM-28960~~	Add CHAINED collection creation to BPS		Michelle Gower	Done
~~DM-29305~~	NCSA cluster plus pip installed htcondor	4	Mikolaj Kowalik	Done
~~DM-28929~~	bps incorrectly assumes htcondor attribute is a string	1	Michelle Gower	Done
~~DM-27888~~	Filename passed to QuantumGraphs' save() should have extension .qgraph		Mikolaj Kowalik	Done
~~DM-29756~~	BPS - Automatic retry with larger memory		Mikolaj Kowalik	Done
~~DM-29575~~	bps restart command		Mikolaj Kowalik	Done
~~DM-29614~~	Have bps report show info from multiple submit nodes		Mikolaj Kowalik	Done
~~DM-29665~~	DB error messages in DC2 run		Christopher Stephens [X] (Inactive)	Done
~~DM-28345~~	RC2 w_2021_02 gen3 conversion with w_2021_03 stack		Monika Adamow	Done
~~DM-28248~~	Convert RC2 w_2020_50 with w_2021_02 stack		Monika Adamow	Done
~~DM-32594~~	Remove dependency on inflection in bps_config		Michelle Gower	Done
~~DM-31970~~	Fix submission AttributeError in htcondor plugin when not using execution butler		Michelle Gower	Done
~~DM-31944~~	concurrency_limit split in htcondor submit file		Michelle Gower	Done
~~DM-32027~~	Rename outCollection to outputRun	1	Michelle Gower	Done
~~DM-31691~~	pipetask doesn't find dataset in execution butler from previous run.		Michelle Gower	Done
~~DM-32217~~	BPS displays cumulative timings instead of time intervals during a submission		Mikolaj Kowalik	Done
~~DM-32201~~	HTCondor plugin bug if output collection contains period		Michelle Gower	Done
~~DM-32199~~	If submit or prepare, have bps check early that WMS is in environment.	4	Michelle Gower	Done
~~DM-28560~~	DC2 w_2020_48 conversion to gen3		Monika Adamow	Done
~~DM-28653~~	Modify BPS to support execution butler	10	Michelle Gower	Done
~~DM-27517~~	Convert RC2 w_2020_42 with w_2020_46 stack		Monika Adamow	Done
~~DM-27498~~	Convert RC2 w_2020_42 with w_2020_45 stack		Monika Adamow	Done
~~DM-27434~~	Add ctrl_bps to lsst_distrib package		Michelle Gower	Done
~~DM-27379~~	RC2 w_2020_42 gen3 conversion with w_2020_44 stack		Monika Adamow	Done
~~DM-36849~~	Create BPS poster for ADASS 2022	4	Michelle Gower	Done
~~DM-36848~~	Write BPS paper for ADASS 2022		Michelle Gower	Done
~~DM-26016~~	Update BPS to take pipeline yaml as input		Michelle Gower	Done
~~DM-26734~~	Ensure that logs from PipelineTasks are stored and available to BPS users		Michelle Gower	Done
~~DM-25821~~	Convert RC2 w_2020_26 to gen3 repo		Michelle Gower	Done
~~DM-25310~~	Run ci_hsc_gen3 with weekly stack + PostgreSQL		Monika Adamow	Done
~~DM-25297~~	Convert RC2 Gen2 run to Gen3 as test of process and code		Michelle Gower	Done
~~DM-27113~~	Convert RC2 w_2020_38 to gen3 with w_2020_42 stack		Monika Adamow	Done
~~DM-27575~~	requestMemory not ending up in submit files.		Michelle Gower	Done
~~DM-27571~~	Gen3 RC2 run with w_2020_46 stack		Monika Adamow	Done
~~DM-27570~~	bps report id not showing running jobs in Total line	6	Michelle Gower	Done
~~DM-22994~~	Run ci_hsc_gen3 with weekly stack + Oracle		Michelle Gower	Done
~~DM-22993~~	Run ci_hsc_gen3 with weekly stack + BPS		Michelle Gower	Done
~~DM-26271~~	Run RC2 one tract in Gen3 w_2020_31		Monika Adamow	Done
~~DM-26249~~	Update ctrl_bps to meet LSST coding standards		Michelle Gower	Done
~~DM-26238~~	Convert RC2 w_2020_30 to gen3 repo		Monika Adamow	Done
~~DM-23497~~	Run RC2 bootstrap with Oracle		Michelle Gower	Done
~~DM-31541~~	Change pipetaskInit job to not read full QuantumGraph	4	Mikolaj Kowalik	Done
~~DM-32066~~	BPS jobs with memory autoscaling enabled remain idle after the first run attempt		Mikolaj Kowalik	Done
~~DM-32074~~	Modify how HTCondor plugin handles transfer of execution butler		Michelle Gower	Done
~~DM-32047~~	Add a requestMemoryMax config in ctrl_bps		Mikolaj Kowalik	Done
~~DM-32220~~	HTCondor plugin copying execution butler back from job	1	Michelle Gower	Done
~~DM-31043~~	Update ctrl_bps setup.cfg with flake8's max-doc-length=79	2	Mikolaj Kowalik	Done
~~DM-27049~~	bpsub printing many warning messages about Missing search section		Michelle Gower	Done
~~DM-27039~~	Make changes to bps_core in preparation for more subcommands.		Michelle Gower	Done
~~DM-26494~~	Add QuantumGraph generation from pipeline yaml command line to BPS config		Mikolaj Kowalik	Done
~~DM-26458~~	Update ctrl_bps to work with new QuantumGraph implementation		Michelle Gower	Done
~~DM-26408~~	Implement initial set of unit tests for ctrl_bps		Mikolaj Kowalik	Done
~~DM-26402~~	Change ctrl_bps commands to use click		Michelle Gower	Done
~~DM-26401~~	Investigate delays before job starts in the RC2 single tract run		Greg Daues	Done
~~DM-26397~~	Add explicit DAG export from ctrl_bps	2	Mikolaj Kowalik	Done
~~DM-26385~~	Combine --init-only into single compute job		Michelle Gower	Done
~~DM-26844~~	Modify BPS configuration class to allow initialization from a dictionary		Mikolaj Kowalik	Done
~~DM-26829~~	RC2 gen3 + bps run with w_2020_37 stack		Monika Adamow	Done
~~DM-37020~~	Replace "requirements" with "constraint" in HTCondor Python function calls in ctrl_bps_htcondor		Mikolaj Kowalik	Done
~~DM-25005~~	Problems using HTCondor API + LSST stack based upon import order		Michelle Gower	Won't Fix
~~DM-30827~~	Allow user to specify number of retries in bps yaml.	2	Unassigned	Invalid
~~DM-31199~~	bps pipetaskInit sometimes needs lots of memory		Unassigned	Invalid
~~DM-29574~~	Save bps yaml		Unassigned	Invalid
~~DM-29560~~	Update bps log directory structure		Unassigned	Invalid
~~DM-32754~~	Catch execution butler creation error		Unassigned	Invalid

Activity

There are no comments yet on this issue.

People

Assignee:: Michelle Gower

Reporter:: Michelle Gower

Watchers:: Vladislav Ekimtov [X] (Inactive)

Votes:: 0 Vote for this issue

Watchers:: 1 Start watching this issue

Dates

Created:: 09/Jan/20 11:14 PM

Updated:: 26/Jul/23 7:12 PM

Resolved:: 26/Jul/23 7:12 PM

Start date:: 02/Jan/20

End date:: 30/Sep/20

Jenkins

No builds found.

Data Management