Apache Arrow and the “Things I Hate About Pandas”
I started with Python because of (or rather, thanks to) pandas, it was my gateway drug. Over the past ~5 years I've done all sorts of things with it, including converting the whole company I worked at. At one of my employer's, I sampled our big data platform, because it was tedious and slow to work with and used pandas instead.All that being said, I'd stress pretty clearly that I never let a single line of pandas into production. There are a few reasons that I've long wanted to summarise, but just real quick: 1) It's a heavy dependency and things can go wrong, 2) It can act in unexpected ways - throw in an empty value in a list of integers and you suddenly get floats (I know why, but still), or increase the number of rows beyond a certain threshold and type inference works differently. 3) It can be very slow, especially if your workflow is write heavy (at the same time it's blazing fast for reads and joins in most cases, thanks to its columnar data structure). 4) The API evolves and breaking changes are not infrequent - that's a great thing for exploratory work, but not when you want to update libs on your production.pandas is an amazing library, the best at exploratory work, bar none. But I would not let it power some unsupervised service.