Fix Version/s: None
We now have an initial/preliminary obscore configuration for embargo repo, so we can go ahead with adding obscore table based on that. Actual job is done using migration scripts in daf_butler_migrate to create the table and butler command in dax_obscore to populate that with pre-existing data.
I want to test this on a copy of embargo repo to make sure (once more) that migration scripts work OK. I also think that migration scripts were written for namespace=oga, and new config uses namespace=embargo (I think embargo is better choice), so likely I'll need to update migration scripts as well.
I think everything should be ready for migrating embargo at USDF. Here is the plan:
- Agree on time slot, 10-15 minutes of a relatively quiet period, to avoid any potential issues.
- I need to ask someone (Kian-Tat Lim?) to update butler.yaml with a new manager line.
- Run all above commands at USDF.
I can test basic things after that but to make sure everything works and obscore is updated correctly, we need to ingest some new data and check obscore.
I ran migration on USDF repo, notes are here: https://confluence.lsstcorp.org/display/~salnikov/2023-02-03+Adding+obscore+to+USDF+embargo+repo
There was something unexpected during the migration - when a script tried to create a new table it got stuck on a lock (likely metadata tables). There was a transaction with INSERT query that was "running" for some time and did not want to finish, and it was locking something. It was in a "idle in transaction" state and ClientRead event, so it looked like client did not want to read data. It is likely that this query was from a job that Merlin killed shortly before I started my process. My guess this is probably due to a proxy which did not destroy the connection (and did not kill transaction) and server was still thinking that client is not dead yet. Not sure if this is going to be an issue in general.
And I see that new records are being added to obscore table, looks like it's doing something right.
A small change was needed to a migration script in daf_butler_migrate to change namespace from "oga" to "embargo". Self-reviewed and merged.
Do you have a suggestion for how to poke at the data that are accumulating? I could probably figure out how to do something with SQLAlchemy from a USDF-RSP notebook, if that connection is allowed.
A direct SQL query is probably the only way until ObsTAP is there. sqlalchemy should work OK from USDF, I think authentication should be already configured for you (if not then you need to create .pgpass file). Embargo repo is in oga schema, other than that it should be easy to write simple select(). Let me know if you need an example.
And another approach, which I frequently to use personally, is some other database tool, e.g. DBeaver, and connect to the server directly and run SQL queries. You may need to set SSH tunnel for that if you are outside SLAC.
Closing this as obscore is created, and it is regularly updated. We'll need to extend ingest process (needs to update raw regions after ingesting them), but it will need a separate ticket.
Testing obscore migration on a copy of embargo repo on my test server.
Starting from a clean state:
So need to apply migration for both obscore and obscore-config managers. Start with obscore:
Add it to alembic:
And migrate (this only updates butler_attributes):
Edit butler.yaml and add above line.
Add butler-config to alembic:
And migrate (add obscore table), this needs an option for config file location:
Now populate it with the existing data: