Hi Spencer Nelson,

Since the sample client application is intended to give users code to build from, I'm concerned that we've got a lot of code in streamsim that would might reside more naturally elsewhere. In particular it seems like basically all of serialization.py could live in lsst/alert_packet, and similarly the _KafkaClient abstraction could be in a separate package as well.

By doing that users could build client applications against a more permanent Rubin interface (acknowledging that we're a looong ways from having that be stable). Leaning harder on alert_packet would also let us avoid hard-coding schema versions as is done currently.

Might be worth talking about this live so I can understand better where you're coming from. I would welcome John Swinbank's thoughts here also.

Eric Bellm, I pretty much agree, but we decided to do it this way so that we don't have to think about providing lsst/alert_packet as an easy-to-install Python package. It doesn't have a setup.py or anything, so it's not clear to me how we'd throw it on PyPI for redistribution outside of the stack.

If you think we can distribute parts of the stack as PyPI/Conda packages publicly, I'm all for moving the code over. It wouldn't be terribly hard with lsst/alert_packet, I think, since it doesn't have any dependencies on other LSST stack packages.

Hi both —

We discussed this at the DM-CCB today.

At the moment, we do not have a policy regarding distributing Rubin-written code through PyPI (/Conda/etc). In principle, the DM-CCB is supportive of the notion: I think it is broadly fair to characterize their attitude of anything which can go on PyPI (because it doesn't have extensive dependencies on the rest of our stack) should. However, they also worried about the logistics of making this happen: are individual developers empowered to simply make releases of code at their whim, rather than going through the (heavyweight) Pipelines release process? Do packages simply get posted using individual developer accounts? If so, what happens when that developer is unavailable for some reason?

For now, we suggest:

• If you both think it would be useful to have alert_packet on PyPI, I'm inclined to agree with you. We can proceed following the template established by astro_metadata_translator . In particular, this includes both setup.py and SConstruct files, and is managed as a “third party” package as far as our release process is concerned (ie, it's owned by the “DM Externals” team on GitHub). Tim Jenness set this up and makes releases to PyPI — I'm sure he'd be willing to advise on what hoops to jump through.
• The DM-CCB will start drafting some policy text to make clearer what's allowed and how it ought to be managed in future. It may be that we'll need to tweak whatever we come up with now to comply with that at some point, but there's no requirement to stop and wait for policy to be available before making forward progress.

Does that help?

That really helps, yes. I like that plan a lot. I'll make a ticket to distribute lsst/alert_stream on PyPI and Conda, and then to merge alert-stream-simulator's streamsim into alert_stream. I think we should delay moving the Kafka client in until we sort out whether to use confluent_kafka or aiokafka (DM-24679).

Note that 20.0 release is imminent so if you don't want a 20.0 tag turning up on the repo (which would seemingly force you to start at v20.0 on pypi or else be very confused) you should do the team tweak ASAP.

Tim Jenness what's the "team tweak?"

See above comment:

it's owned by the “DM Externals” team on GitHub

It may be that John or I need to make that change for you.

All of the above sounds good to me. Spencer Nelson, I would think that uncertainty about what kafka library to build on would push us towards getting the client interface behind an abstraction layer sooner--I guess that depends if we think we can get to DM-24679 before circulating the simulator more broadly.

The way we use teams on GitHub is (too briefly) described at https://developer.lsst.io/stack/adding-a-new-package.html#adding-a-new-package-to-the-build.

I'm not completely sure I understand what permissions are required here (see https://lsstc.slack.com/archives/C2JP8GGVC/p1590617175372300), but I have given the “DM Externals” team ”write” access (to match astro_metadata_translator).

(I note that a_m_t also has the “DMLT” team with “admin” access, but that isn't specified in the Developer Guide so I didn't set that.)

Okay, Avro serialization and the schema are now pulled out of alert-stream-simulator and its streamsim package. It now references those things from lsst.alert.packet, which is marked as a dependency, and which it can get from PyPI.

The streamsim package still has more in it that we could take out, but this other stuff is less obviously redundant. Specifically:

• streamsim.serialization is now mostly to do with serializing timestamps (in a way that really only matters internally to alert-stream-simulator) and with serializing to Confluent Wire Format (which only matters when you write to Kafka). The timestamp stuff should definitely stay in alert-stream-simulator; the wire format stuff could conceivably belong in some new lsst.alert.stream.kafka package.
• streamsim._kafka is a large pile of stuff for interacting with Kafka sanely. That could all go elsewhere, although the API was written with the simulator in mind.

Personally, I think code reuse can be overrated, and we shouldn't fret too much about throwing this stuff out and doing it over when we do the actual Kafka client used in the production pipeline. That thing will have different needs anyway. So, I lean towards calling this good.

But I want to raise the question - should we pull more out of alert-stream-simulator and its streamsim package, or are we good?

Agreed that the timestamps code clearly stays in the simulator.

The Confluent Wire Format and _kafka code are both pieces that brokers are going to need in order to take the stream output and couple it to their systems: either they will use our code or will need to create equivalents themselves. As I said above if we package them up someplace reusable now that (I think) makes it easier to manage the interface going forward. But if your instincts from e.g. SciMMA are different I'm happy to hear the counterargument.

Aha, right, you expect the code to be useful to brokers, not necessarily to ap_association or whatever. That's a good argument. I'll make a new distributable package for them.

Spencer Nelson and I discussed this by video as some additional complications had arisen. Much of the _kafka infrastructure currently in the simulator is code that is more relevant for the DM side (topic creation and modification, etc.) rather than the client library, and we expect that that code should firmly live in Stack land--which makes it challenging to distribute to third parties. On the other hand we didn't feel especially ready to stand up a full-on Kafka client library.

We decided to take a bit of an usual path forward: to distribute two example client libraries. The first is the existing one, using the _kafka code currently in this package. The second is the third-party adc_streaming code (https://github.com/astronomy-commons/adc-streaming). We thought this would provide broker teams a few options for connecting with the simulator while emphasizing that these interfaces are not final.

Looks good!