I am rebuilding a data pipeline that processes billions of records. An overview of what I built is as follows:collect data from (n*k) sources-> derive new data -> generate a unified/merged collection of data (n) data.The current solution is all hand crafted code.I know this is a 10,000 foot view of the problem, but are there any guides or books on how to better design and implement this type of solution?