At the discussion, we talked about replacing the current multicast based discovery mechanism with something else.  Ideas include etcd, redis, and consul.  Upon review, I think the most straight-forward approach with the least learning curve is redis.  The purpose of this external service is to store the information of active firefly instances.  Nothing fancy.

We will deploy one redis container as a known service for which multiple firefly containers will register to.
This requires adding redis to our kubernetes deployment, which is simple.
The bulk of the work is to implement the new discovery mechanism and testing it.  Because this involves integrating with third-party interfaces, there may be unforeseeable obstacles.

My best guess is around 2 weeks.

Developer has done preliminary study and proposed a preferred solution.

Loi Ly Could you provide two conceptual diagrams for both redis and consul? The LSP lead would like to understand the design before we start implementing the solution. 

Attached is the diagrams requested.  There's not much there, so feel free to ask questions if you have any.  I'll do my best to answer them.
Firefly cache auto-discovery conceptual design - Loi Ly - Confluence.pdf.pdf

The conceptual design diagrams are attached for consul and redis. Redis is recommended for its simplicity and light weight container.