From Laptop to Lambda: Outsourcing Everyday Jobs to Thousands of Containers [pdf]
This is really great! I wrote and operate a pipeline framework for analyzing data from multiple astronomical observatories in low-latency [1], and I'm currently adding this sort of burst-performance scaling for things like background simulations. Things like Kubernetes spin-up time or packaging huge libraries for AWS Lambda are, indeed, challenges. Getting those startup times down and doing autoscaling in a relatively platform-agnostic way with low boilerplate overhead would be really game-changing for these sorts of analyses (among other applications).If you can run an analysis in low-latency and detect a joint gravitational wave/EM source, for example, you can quickly follow it up with other telescopes, like we did with the first direct kilonova observation [2]. Though the gamma-ray detection localization wasn't good enough to really aid counterpart searches for that event (GW170817), there are other types of candidates (like GW+neutrinos, my project) for which this rapid localization would be a huge improvement. And if you can do really hard things like estimate the GW source parameters before merger (e.g. whether you're looking at a binary neutron star merger, which is likely to emit detectable light if it's close enough), you can try to get fast-slewing telescopes pointing on source at merger time. Not to mention that burst processing would let you do more clever things with your statistics (getting better sensitivity) in low-latency as well. So stuff that makes this easier is really exciting!Maybe if the API is amenable to my own solution I'll be able to implement this as a backend :)[1] http://multimessenger.science
[2] https://journals.aps.org/prl/abstract/10.1103/PhysRevLett.11...