Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-10981

Schedule and implement a conversation with DAX about L1 DB

    Details

      Description

      We need some sort of mocked up interface to the level 1 database. Specifically, this requires the ability to retrieve DIAObjects based on spatial position and the ability to put potentially updated versions of those DIAObjects back from multiple threads.

      Ideas were to either use a sqlite database and interact directly through SQL, but it might be informative to have a shimmed butler interface to do this interaction.

      In either case, the schema are defined and reside in the cat package.

        Attachments

          Activity

          Hide
          ebellm Eric Bellm added a comment - - edited

          Here are my notes from a conversation with Andy Salnikov, Simon Krughoff, and Chris Morrison:

          Andy has implemented an SQLAlchemy-based prototype system for testing the L1 database system: https://github.com/lsst-dm/l1dbproto He's been testing it on MySQL and Postgres at IN2P3. The ap_proto script (https://github.com/lsst-dm/l1dbproto/blob/master/bin/ap_proto) simulates the L1 workflow.

          We'll need to have a database set up on whatever system we want the MVS pipelines to run on. Andy recommends Postres on SSDs for best performance, but that's not possible yet at NCSA. We will investigate just using lsst-db at NCSA for now, and possibly also an AWS system for continuous integration.

          We discussed the question of whether we expect the science pipelines to contain SQL directly or if those functions might be dealt with by the Butler in the future. Andy thinks that for the L1 database performance will likely be impacted enough that we will want to work with direct database access. For L1 pipeline minimum viable system we (UW) will write a small abstraction layer so we don't bake our assumptions in too deeply. It's possible the l1dbproto package has much of this functionality already.

          Andy Salnikov will send us the l1db.cfg file and point us to the most appropriate ticket branch of l1dbproto.

          Show
          ebellm Eric Bellm added a comment - - edited Here are my notes from a conversation with Andy Salnikov , Simon Krughoff , and Chris Morrison : Andy has implemented an SQLAlchemy-based prototype system for testing the L1 database system: https://github.com/lsst-dm/l1dbproto He's been testing it on MySQL and Postgres at IN2P3. The ap_proto script ( https://github.com/lsst-dm/l1dbproto/blob/master/bin/ap_proto ) simulates the L1 workflow. We'll need to have a database set up on whatever system we want the MVS pipelines to run on. Andy recommends Postres on SSDs for best performance, but that's not possible yet at NCSA. We will investigate just using lsst-db at NCSA for now, and possibly also an AWS system for continuous integration. We discussed the question of whether we expect the science pipelines to contain SQL directly or if those functions might be dealt with by the Butler in the future. Andy thinks that for the L1 database performance will likely be impacted enough that we will want to work with direct database access. For L1 pipeline minimum viable system we (UW) will write a small abstraction layer so we don't bake our assumptions in too deeply. It's possible the l1dbproto package has much of this functionality already. Andy Salnikov will send us the l1db.cfg file and point us to the most appropriate ticket branch of l1dbproto.
          Hide
          cmorrison Chris Morrison added a comment -

          Hey Eric Bellm your paragraph above seems to cut off at the last sentence. Is there more you want to write?

          Show
          cmorrison Chris Morrison added a comment - Hey Eric Bellm your paragraph above seems to cut off at the last sentence. Is there more you want to write?
          Hide
          ebellm Eric Bellm added a comment -

          update: the config files are now in the /cfg directory of l1dbproto, and master is the best branch to work from.

          Show
          ebellm Eric Bellm added a comment - update: the config files are now in the /cfg directory of l1dbproto, and master is the best branch to work from.
          Hide
          ebellm Eric Bellm added a comment -

          Chris Morrison Left off one word, now fixed.

          Show
          ebellm Eric Bellm added a comment - Chris Morrison Left off one word, now fixed.
          Hide
          cmorrison Chris Morrison added a comment -

          Perhaps create tickets for the DB investigation at NCSA and link them here. Other than that looks good.

          Show
          cmorrison Chris Morrison added a comment - Perhaps create tickets for the DB investigation at NCSA and link them here. Other than that looks good.
          Hide
          ebellm Eric Bellm added a comment -
          Show
          ebellm Eric Bellm added a comment - Tickets now created in https://jira.lsstcorp.org/browse/DM-10768

            People

            • Assignee:
              ebellm Eric Bellm
              Reporter:
              krughoff Simon Krughoff
              Reviewers:
              Chris Morrison
              Watchers:
              Chris Morrison, Eric Bellm, Simon Krughoff
            • Votes:
              0 Vote for this issue
              Watchers:
              3 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved:

                Summary Panel