Show HN: Shipyard – Serverless Workflow Platform to Simplify DataOps
Hey Hacker News! I’m Blake, one of the co-founders at Shipyard.<p>In my past line of work, I was running the data teams for a digital advertising agency, handling high-throughput information for Fortune 500 brands. From my time in the trenches, I noticed that:<p>- Cloud services are needlessly complex for the average user, but low-code platforms can’t handle complex business problems or scale<p>- Data Teams are often being tasked with navigating and solving technical problems outside their wheelhouse (DevOps, app hosting, database management, queueing systems, etc.)<p>- Organizations struggle to make use of their data because solutions take weeks to build and are difficult to get into the hands of others (typically residing on someone’s local machine)<p>Over the past year, I’ve tried to synthesize what I’ve learned to solve some of these issues. My goal is to make it easier for people to quickly action on their data in an automated fashion.<p>With Shipyard, our team has built a high-end workflow product that allows data teams to quickly build, automate, and share pipelines without the hassle of managing infrastructure. Rather than highlighting features, I thought the community might enjoy understanding our core product tenets.<p>- Your code should run the same locally as it does on our platform. Code shouldn’t be written to accommodate the limitations and configuration of the tools you’re using.<p>- Templates should be a first-class citizen. When you build something to do a specific task internally, you should be able to share it for use by anyone in your org, regardless of their technical skill. You should also have the ability to easily see template usage and make wide-scale updates.<p>- Workflows should be simple, flexible, and modular. Any shape of pathing or parallelism should be supported. Data generated upstream should be readily available to all tasks downstream. Components of a workflow should small in scope and reusable anywhere.<p>- Tooling should serve data teams in a way that lets them spend more time focusing on what they’re good at - solving problems with data.<p>We’re releasing the product to the public today. Out of the gate, we have over 50 open-source blueprints built out to help teams quickly integrate their cloud services together, without needing to write any code. However, you can run any custom code of your own (Python, Bash) using your favorite packages (DBT, Great Expectations, Pandas, etc.) and link it all together to create any powerful data solution. We’ve seen teams use the product to build data quality assurance alerts, 3rd party vendor integrations, run live marketing updates via API calls, self-service data refreshes, and build out your typical ETL pipelines.<p>We’ll be around today to answer any questions the community may have. Looking forward to the discussion!