Uploaded image for project: 'Data Management'
  1. Data Management
  2. DM-36742

Add real-bogus trained model to DC2 CI repository

    XMLWordPrintable

    Details

    • Type: Story
    • Status: In Review
    • Resolution: Unresolved
    • Fix Version/s: None
    • Component/s: ap_verify
    • Labels:
      None
    • Story Points:
      4
    • Sprint:
      AP S23-2 (January), AP S23-3 (February), AP S23-4 (March)
    • Team:
      Alert Production
    • Urgent?:
      No

      Description

      Add the DC2 real bogus trained model to the new CI dataset (DM-34845). In addition to adding the data to the git-lfs repo, add the trained model to the ingest step of ap_verify. This will require defining the trained model for the butler, and it is suggested that we treat the model the same way as we treat our "curated calibration" datasets.

        Attachments

          Issue Links

            Activity

            Hide
            krzys Krzysztof Findeisen added a comment - - edited

            Whether an environment variable in a config field will get properly expanded depends a lot on how that path is actually evaluated – it's certainly not guaranteed. I'd recommend testing it with your existing setup first.

            Much later EDIT: a meaningless objection, since it's your code that decides how the config field is evaluated. Apologies for the noise!

            Show
            krzys Krzysztof Findeisen added a comment - - edited Whether an environment variable in a config field will get properly expanded depends a lot on how that path is actually evaluated – it's certainly not guaranteed. I'd recommend testing it with your existing setup first. Much later EDIT: a meaningless objection, since it's your code that decides how the config field is evaluated. Apologies for the noise!
            Hide
            nima Nima Sedaghat added a comment -

            I actually missed the last part of the meeting. That's why I'm wondering about the discussed storage options for the model packages.

            I can think of these options – and indeed I don't think we need to limit ourselves to only one:
            1- local storage: model-packages that are local to the meas_TransiNet repository and are always fetched with it (e.g. the dummy model). We can probably access them just using a relative path :think:

            2- neighbor git data-repository: I suppose these can also be accessed using a relative path, assuming the repository is always cloned into its default location.

            3- butler repository: to be discussed.

             

            I guess we are now waiting for John Parejko to get a pre-confirmation re. feasibility of these implementations.

             

            Show
            nima Nima Sedaghat added a comment - I actually missed the last part of the meeting. That's why I'm wondering about the discussed storage options for the model packages. I can think of these options – and indeed I don't think we need to limit ourselves to only one: 1- local storage: model-packages that are local to the meas_TransiNet repository and are always fetched with it (e.g. the dummy model). We can probably access them just using a relative path :think: 2- neighbor git data-repository: I suppose these can also be accessed using a relative path, assuming the repository is always cloned into its default location. 3- butler repository: to be discussed.   I guess we are now waiting for John Parejko to get a pre-confirmation re. feasibility of these implementations.  
            Hide
            ktl Kian-Tat Lim added a comment -

            I don't have the full context here, but I read Ian's summary as saying that local storage is out (I fully agree), neighbor git repository is in (accessed via eups environment variable, not relative path), and Butler repository seems unlikely (since this is more like code than data).

            Show
            ktl Kian-Tat Lim added a comment - I don't have the full context here, but I read Ian's summary as saying that local storage is out (I fully agree), neighbor git repository is in (accessed via eups environment variable, not relative path), and Butler repository seems unlikely (since this is more like code than data).
            Hide
            nima Nima Sedaghat added a comment -

            The relevant model package is now being stored in the neighboring package (rbClassify_data). Here's the relevant PR:
            https://github.com/lsst-dm/rbClassifier_data/pull/1

             

            I think this addresses the purpose of this ticket. Please let me know if something is missing.

            Show
            nima Nima Sedaghat added a comment - The relevant model package is now being stored in the neighboring package (rbClassify_data). Here's the relevant PR: https://github.com/lsst-dm/rbClassifier_data/pull/1   I think this addresses the purpose of this ticket. Please let me know if something is missing.
            Hide
            krzys Krzysztof Findeisen added a comment -

            Am I correct in assuming that finishing DM-37478 will make this issue obsolete? If not, what is the new scope?

            Show
            krzys Krzysztof Findeisen added a comment - Am I correct in assuming that finishing DM-37478 will make this issue obsolete? If not, what is the new scope?

              People

              Assignee:
              nima Nima Sedaghat
              Reporter:
              sullivan Ian Sullivan
              Reviewers:
              Ian Sullivan
              Watchers:
              Eric Bellm, Ian Sullivan, John Parejko, Kian-Tat Lim, Krzysztof Findeisen, Nima Sedaghat
              Votes:
              0 Vote for this issue
              Watchers:
              6 Start watching this issue

                Dates

                Created:
                Updated:

                  Jenkins

                  No builds found.