Can statistical models be used for this kind of problem? Sorry for the long post:
Here is the situation (it's a silly example just to illustrate the point):
A computer repair company is primarily focused on repairing computers. However, the company has recently become interested in estimating the profits they will earn by repairing computers (e.g. let's assume that the company has no idea how much money they will actually have to spend repairing a computer until they are finished - thus, they want to predict the profit in advance before negotiating with the customer). They decide that for a given computer, the profits can be estimated by measuring three variables associated with each computer: X1, X2, X3. Thus, the general form of the model can be expressed as:
Profits = f(X1, X2, X3)
For this example, let's say that the company has 3 labs. The first lab is made of junior employees, the second lab is made of intermediate employees and the third lab is made of senior employees. Let's assume that employees from the third lab are on average "better at measuring X1, X2, X3" than employees from the second lab, and employees from the second lab are better than employees from the first lab. The first lab is the biggest lab, the second lab is smaller, and the third lab is the smallest.
So for the same computer : an employee from the first lab might measure a value of X1, but an employee from the second lab might measure a different value of X1 for the same computer. Theoretically, it is also possible that an employee from the third lab might measure the same value of X1 as the employee from the second lab.
When the company receives computers from their customers, they are generally first sent to the first lab. From here, the computers have their values of X1, X2, X3 measured, then repaired, and then sent back to customer - OR - the computers are sent to the second lab (e.g. if the junior employees don't know how to fix it), where their values of X1, X2, X3 are re-measured, repaired and sent back to the customer - OR - sent to the third lab, X1, X2, X3 remeasured, repaired and sent back to the customer.
The company is interested in getting the computers back to the customers - they are not as interested in recording the more accurate measurements for X1, X2, X3 for each computer (e.g. if the junior employees from the first lab can fix the computer, the company will not waste time sending the computer to the third lab just to record a better set of measurements). For each repaired computer, the company also tracks the profits they made.
Question: Now, let's assume these two scenarios for modelling
Scenario 1: Suppose for any given computer, the company only keeps the last set of measurements for X1, X2, X3, because they believe that these are the most accurate set of measurements. But the company wants to make their predictions on incoming computers as early on as possible, i.e. based on measurements taken from the first lab**. This is creating a "mismatch"** : a statistical model created using the available data contains measurements of different quality. When the first lab will make measurements for a computer, it's likely that they might be somewhat incorrect and likely to change. Are there any statistical models that can be used in this scenario? Is this an example of "measurement error" models?
Scenario 2: Suppose now the company decides to keep all measurements made for each computer. This means that some computers might have 3 measurements (lab 1 : X1. X2, X3), some computers might have 6 measurements (lab 1 : X1, X2, X3 AND lab 2: X1, X2, X3) and computers might have 9 measurements (lab 1: X1, X2, X3 AND lab 2: X1, X2, X3 AND lab 3: X1, X2, X3). However, the company always wants to make predictions for new predictions based on the measurement taken at the first lab. This now looking like a longitudinal/repeated measures problem - but I am still not sure.
I am not sure how to make models for scenario 1 and scenario 2. I know that in this problem, it would be reasonable to believe that more complicated computers get sent to lab 3 - thus, try to cluster the data and make separate models based on how complex the computers are. But imagine if we had no way of knowing which computers will receive 3 measurements or which will receive 9 measurements.
At this point, I am thinking of just giving up and creating a model that uses the data from the most recent lab - and just hoping that the differences in measurements between the first lab, second lab and third lab are negligible (and make models using the most recent data) ..... Or try and make layered hypothesis tests : take computers that reached lab 2 and compare the updates in measurements between lab 1 THEN take computers that reached lab 3 and compare the updates in measurements between lab 2 and lab 1, etc.). But in the real world, there could be thousands of "points" (i.e. "labs") in which the observed data can get updated before it stabilizes).
Has anyone worked on such kinds of problems?
Sorry for the long post
Thanks

One answer might be to create a generative model that assumes some distribution over the properties of computers to be repaired, something about the measurement process at each lab, the process for deciding whether to repair it send to another lab, and the same processes there. You may or may not struggle with this due to a lack of sufficient data identifying all aspects of the model.
Or you leave out the bit where you model the distribution of items to be repaired and chain a bunch of non-generative models (i.e. specific input distributions sort of taken as given) after each other (or a single multi-output neural network or something like that) and model just the process in the company. Maybe that would be better informed (but also more fragile to changes in the system).
Clearly, either approach would work better on the second scenario you described, but even better if you collected even more data that's not routinely created (e.g. send some items to be measured in all 3 labs no matter what and possibly even randomize where these items get repaired).
