How to know if I have enough controllable info for ML modelling? I have a problem statement where I would like to predict the likelihood that broker will fulfill his order commitment or not.
For ex: Let's say there are 3 parties involved in this transaction and they are company A, broker A, end_customer A. Works like below
Broker A finds out that end_customer A needs 500 units of product (ex:sensors). So, broker A places/registers an order with company A for 500 units. (so he gets an early bird discount and also this prevents other brokers from placing the same order for same customer...Something Like design registration/protection). Based on this company A starts building products in stages and be happy that they will earn revenue from 500 units (Yay! big order). But what happens most of the time is, broker A only buys 100 units. This is the pain point. As company A, we would like to know why does this happen? how can we predict such behavior in advance, so we can save our production resources and also plan our revenues accordingly. For ex: If another broker B, comes in and places an order with us (company A), we would like to predict the likelihood of this transaction being fulfilled or not. By fulfill, I mean if the broker manages to order 50% (250 out of 500 above) of qty requested, we consider that as fulfilled. Again, this 50% pc is just a random number. It can be 60 pc or 70 pc as well based on business needs. IRL, 100 pc exact fulfillment has not happened in our case (90% of the time)
My dataset contains characteristics of the product such as product segment, product family etc. They also contain characteristics of the broker such as broker profile (big/small in terms of business to company A, location of broker A, market segment that broker A operates in etc., end_customer detail-location only
So, my questions are as follows
a) I have historical data of order requested and order placed. I create a rule-based labelling based on % difference between order requested and order placed. If diff > 50%, it is label 0(not fulfilled), otherwise it is label 1 (fulfilled). In future even if a very new broker comes in, I would like to predict the likelihood that whether he will meet his order requested target or not.
b) Can this problem even be modelled using ML? Do you think there are lot of external factors to this which we cannot control? So, this might impact our prediction problem. Am not interested in causal modelling. Just prediction. Do you think this problem is not amenable for ML? I ask because mostly we will be predicting outcomes for existing brokers (seen in historical data) but still do you think it is logical to predict an outcome for a brand new broker?
c) If it is not solvable using ML, what kind of additional info do you think will be good to have to solve this problem using ML?
d) If we decide to solve using ML, Am I right to understand that traditional algo like logistic, decision trees etc can be used for this classification?
e) Any other suggestions ?

I have my doubts that you will be able to find anything of value from this data, because it involves market data, which is very chaotic and involves... people, which all have unknown motivations. However you should still try, you never know.
To summarize your question and comments:

*

*end companies will need some amount of a certain product

*brokers predict how much of a product end companies will need before they know it and place an order on their "behalf" (because they get a discount) to companies which produce said product

*the brokers inevitably make a mistake with their predictions and only order a portion of the initial order (because they can, since it is not a binding contract, or rather you do not want to enforce it)

*the company which produces the aforementioned product suffers because of this

*you would like to predict the actual amount that the broker will eventually buy, while taking into account the fact that:

*

*the broker might be making bad predictions about how much of a product an end company will need

*the broker might place a higher than necessary order just to get a bigger discount (or maybe doesn't like the producing company)

*the end company might not actually need said amount of product

*the end company might not like this particular broker and obtain the product elsewhere

*who knows what other shenanigans are happening behind closed doors



Whatever the case, in it's simplest form you can try to figure out if there is any pattern in the brokers or end companies behavior. You could try a classification model, by turning your dependent variable into a binary variable (1 for more than half the order, 0 else), or even better you could try modelling the actual numbers (or ratios).
If you have your data into a standard format for regression such as




Actually Sold
Initial Order
Broker
End Company




100
500
A
A


200
400
A
B


100
600
B
A


...







you could try your luck with any of the standard models, such as linear regression, to try and figure out if maybe there is an issue with the brokers, end companies, or both. In this case you would try to predict "Actual" based on the other variables. Of course if you have any other variables which might help in figuring this out you can definitely include them.
Alternatively you could try your luck with classification (predicting 1 or 0), if your data is in the format




Outcome
Initial Order
Broker
End Company




1
500
A
A


0
400
A
B


1
600
B
A


...







and try to predict the probability of 1 or 0 occurring (the probability of a successful transaction or not), for ex. using logistic regression.
