Show HN: Yobulk – Open-source CSV importer powered by GPT3
wait… if importing malformed csvs gets automated that’s like half of a data professional’s job gone in a poof of smoke /s. jk -- great use case<p>so often w/ pandas I’d:
1. “yeet” the csv into a dataframe
2. use dataframe methods to massage the data to a “clean” state
3. push as much of the df methods into pd.read_csv() parameter options<p>it’s be great to iterate more quickly on the above loop. better yet — what if it would could auto-generate a letter to send to the folks from whom you got this data on how they could better output to csv to make ingestion simpler and easier for downstream users…. but maybe that letter would just be "don't use CSV!"<p>related to flat data formats, it obviously makes sense to start with CSV, but what about the future? If this tool became ubiquitous, how might a SWE or data professional's job change? What opportunities be created? As in:<p>1. CSV is ubiquitous but has no singularly well-adopted standard.
2. software and data engineers struggle with CSVs as a result of #1.
3. tool is created to reduce pain and friction.
4. profit? a new market? a new standard?<p>Last, but most personally interestingly, how much do you know about the Apache Arrow ecosystem and how it's mission might overlap with YoBulk's