Hsin-Fang Chiang added a comment - 02/Dec/19 8:41 PM 20191127T192022+0000 20191127T192345+0000 Attempted again running two graphs simultaneously. The master was a m5.2xlarge on-demand instance. The CPU utilization was ~50-70% on the master instance while it managed the sfm jobs. There was a surge of database transactions shortly after the sfm jobs started, but soon it evened out to be ~50-70 average active sessions during sfm processing. For the workers, I got 150 m5.xlarge Spot instances for the single frame processing part and 150 r4.2xlarge Spot instances later. From the RDS performance insight, xact_commit.avg (transaction commits per second) was typically ~10 and db load (average active sessions) ~1. Workflow wall time : 8 hrs, 44 mins Cumulative job wall time : 62 days, 16 hrs Cumulative job wall time as seen from submit side : 62 days, 23 hrs Cumulative job badput wall time : 0.0 secs Cumulative job badput wall time as seen from submit side : 0.0 secs   Workflow wall time : 9 hrs, 59 mins Cumulative job wall time : 62 days, 23 hrs Cumulative job wall time as seen from submit side : 63 days, 6 hrs Cumulative job badput wall time : 0.0 secs Cumulative job badput wall time as seen from submit side : 0.0 secs # All (All) Transformation Count Succeeded Failed Min Max Mean Total dagman::post 29788 29788 0 0.0 9.0 1.052 31343.0 pegasus::dirmanager 1 1 0 3.0 3.0 3.0 3.0 pegasus::transfer 2712 2712 0 2.176 6.768 4.21 11417.243 pipetask 27075 27075 0 17.861 6243.819 199.636 5405141.464   # All (All) Transformation Count Succeeded Failed Min Max Mean Total dagman::post 29788 29788 0 0.0 7.0 1.012 30148.0 pegasus::dirmanager 1 1 0 2.0 2.0 2.0 2.0 pegasus::transfer 2712 2712 0 2.167 9.002 4.255 11540.378 pipetask 27075 27075 0 19.297 6300.657 200.601 5431273.315

Hsin-Fang Chiang added a comment - 03/Dec/19 8:40 PM 20191203T195830+0000 20191203T195926+0000 20191203T200145+0000 20191203T200315+0000 20191203T200433+0000 I attempted to run 5 graphs at the same time. In the initialization jobs (first blocking job of the graph), two died with   Traceback (most recent call last): File "/home/centos/w_2019_38/stack/miniconda3-4.5.12-1172c30/Linux64/sqlalchemy/1.2.16+2/lib/python/SQLAlchemy-1.2.16-py3.7-linux-x86_64.egg/sqlalchemy/engine/base.py", line 1230, in _execute_context cursor, statement, parameters, context File "/home/centos/w_2019_38/stack/miniconda3-4.5.12-1172c30/Linux64/sqlalchemy/1.2.16+2/lib/python/SQLAlchemy-1.2.16-py3.7-linux-x86_64.egg/sqlalchemy/engine/default.py", line 536, in do_execute cursor.execute(statement, parameters) psycopg2.OperationalError: out of shared memory HINT: You might need to increase max_locks_per_transaction. One finished its init job and continued on with single frame processing. The other two are still trying but I'll just kill them.

Hsin-Fang Chiang added a comment - 17/Dec/19 6:13 PM 20191216T201334+0000 20191216T201429+0000 20191216T201752+0000 20191216T201957+0000 20191216T202127+0000 After Dino Bektesevic increased the cache sizes of the RDS instance to ~1GB, I tried the same 5 graphs again. After two of the 5 workflows finished their initialization jobs, many ISR (but not all) jobs failed with conn = _connect(dsn, connection_factory=connection_factory, **kwasync) sqlalchemy.exc.OperationalError: (psycopg2.OperationalError) FATAL: remaining connection slots are reserved for non-replication superuser connections At the time there were 200 m5.xlarge workers. Dino then increased max_connections from 350 to 1000. Re-submitting new workflows with the same 200 workers, the connection slot error was no longer seen.

Hsin-Fang Chiang added a comment - 20/Dec/19 11:11 PM In the next attempt I (stupidly) requested 300 m5.xlarge workers; as each instance ran 4 jobs that meant 1200 connections. So I hit the same error again. We then increased max_connections to 2500.

Hsin-Fang Chiang added a comment - 22/Jan/20 7:29 PM We are wrapping up this effort for now or at least until the new LDF / DOE lab decision is made. Closing this ticket.  The report is being written in https://dmtn-137.lsst.io/ ( DM-21820 )