Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-8021

Deal with large pickles

    XMLWordPrintable

    Details

    • Type: Bug
    • Status: Done
    • Resolution: Done
    • Fix Version/s: None
    • Component/s: ctrl_pool, pipe_drivers
    • Labels:
      None

      Description

      Lauren MacArthur is running:

      coaddDriver.py /tigress/HSC/HSC --rerun lauren/LSST/DM-6816/cosmos --job DM-6816-cosmos-y-coaddDriver --time 100 --cores 96 --batch-type=slurm --mpiexec='-bind-to socket' --id tract=0 filter=HSC-Y --selectId ccd=0..103 filter=HSC-Y visit=274..302:2^306..334:2^342..370:2^1858..1862:2^1868..1882:2^11718..11742:2^22602..22608:2^22626..22632:2^22642..22648:2^22658..22664:2 --batch-submit '--mem-per-cpu 8000'
      

      and it is producing:

      OverflowError on tiger-r8c1n12:19889 in map: integer 2155421250 does not fit in 'int'
      Traceback (most recent call last):
        File "/tigress/HSC/LSST/stack_20160915/Linux64/ctrl_pool/12.1+5/python/lsst/ctrl/pool/pool.py", line 99, in wrapper
          return func(*args, **kwargs)
        File "/tigress/HSC/LSST/stack_20160915/Linux64/ctrl_pool/12.1+5/python/lsst/ctrl/pool/pool.py", line 218, in wrapper
          return func(*args, **kwargs)
        File "/tigress/HSC/LSST/stack_20160915/Linux64/ctrl_pool/12.1+5/python/lsst/ctrl/pool/pool.py", line 554, in map
          self.comm.scatter(initial, root=self.rank)
        File "MPI/Comm.pyx", line 1286, in mpi4py.MPI.Comm.scatter (src/mpi4py.MPI.c:109079)
        File "MPI/msgpickle.pxi", line 707, in mpi4py.MPI.PyMPI_scatter (src/mpi4py.MPI.c:48114)
        File "MPI/msgpickle.pxi", line 168, in mpi4py.MPI.Pickle.dumpv (src/mpi4py.MPI.c:41672)
        File "MPI/msgbuffer.pxi", line 35, in mpi4py.MPI.downcast (src/mpi4py.MPI.c:29070)
      OverflowError: integer 2155421250 does not fit in 'int'
      application called MPI_Abort(MPI_COMM_WORLD, 1) - process 0```
      

      We need to fix or work around this problem.

        Attachments

          Issue Links

            Activity

            price Paul Price created issue -
            swinbank John Swinbank made changes -
            Field Original Value New Value
            Watchers John Swinbank, Lauren MacArthur, Paul Price [ John Swinbank, Lauren MacArthur, Paul Price ] Fritz Mueller, John Swinbank, Lauren MacArthur, Paul Price [ Fritz Mueller, John Swinbank, Lauren MacArthur, Paul Price ]
            price Paul Price made changes -
            Reviewers Fritz Mueller [ fritzm ]
            Status To Do [ 10001 ] In Review [ 10004 ]
            price Paul Price made changes -
            Story Points 5 2
            price Paul Price made changes -
            Reviewers Fritz Mueller [ fritzm ] Nate Pease [ npease ]
            Status In Review [ 10004 ] In Review [ 10004 ]
            npease Nate Pease [X] (Inactive) made changes -
            Status In Review [ 10004 ] Reviewed [ 10101 ]
            price Paul Price made changes -
            Resolution Done [ 10000 ]
            Status Reviewed [ 10101 ] Done [ 10002 ]
            swinbank John Swinbank made changes -
            Epic Link DM-6172 [ 24685 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ]
            jbosch Jim Bosch made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            rowen Russell Owen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            krzys Krzysztof Findeisen made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]
            swinbank John Swinbank made changes -
            Remote Link This issue links to "Page (Confluence)" [ 14355 ] This issue links to "Page (Confluence)" [ 14355 ]

              People

              Assignee:
              price Paul Price
              Reporter:
              price Paul Price
              Reviewers:
              Nate Pease [X] (Inactive)
              Watchers:
              Fritz Mueller, John Swinbank, Lauren MacArthur, Nate Pease [X] (Inactive), Paul Price
              Votes:
              0 Vote for this issue
              Watchers:
              5 Start watching this issue

                Dates

                Created:
                Updated:
                Resolved:

                  Jenkins

                  No builds found.