Clustering Data with Time and ~10 million records I have a dataset with features like product categories, their dimensions, price, units sold on a given day.
I want to create clusters out of this dataset (~12-15 million records) and I am using data for 1 year. However, some products might be seasonal and may sell more in end of year vs start of year. The end goal is to classify the individual products as high selling or low selling.
I have the following questions -

*

*Since the dataset is huge, I am still using K-Means++ on a fairly large cluster and running the K-Means is fine. However, Is the performance of K-Means degrade for such large datasets? If yes, what should be the turnaround for this? I tried BIRCH but the results were similar.


*The product category have around 100 distinct categories (with over 8-9 million distinct individual products). Is doing one-hot encoding the right way or is there any other way? As per my understanding, one hot encoding may result in sparse dataset and might not add much value to the model. I tried using PCA to first create 2 components and then do clustering however, PCA1 itself was explaining 99% of variance in data with 99% feature importance of 'units sold' so I think I am missing something here. I also used robustscalar to scale the data before doing anything.


*Since I have product data on a daily level for the entire year, how to factor in the time series component for clustering? Is Panel Data Clustering Possible? How to go about it?
Right now, the clustering exercise I was performing was after aggregating product sales on a Quarter level. However, I need to factor in the seasonality part. It would be great if someone can provide a detailed response and let me know if more information is needed.
Thanks!

Simplest thing would be to 'featurize' the history. Represent the time-series part with time series features like from the tsfresh package. A lot of the 'seasonal' features though will require more than 1 year if the products exhibit a yearly seasonal pulse but you could instead do something simple like percentage of total sales that occurred in week 1, 2, 3 ... 52 as a column. Could do other simple measures like a simple mean, number/percentage of 0s etc. Cluster on those along with the product attribute information and drop the time element (will also shrink your dataset significantly).

The end goal is to classify the individual products as high selling or
low selling.

Then I would just sum the sales and put a simple threshold on that to split it up, no need to cluster. ABC analysis is used very frequently for demand segmentation.
