  With high over-subscription rates and significant operational costs,
observatories must ensure that their operations are efficient and effective. A
number of key performance indicators are generally used to evaluate the
observatory's performance among which are the numbers of publications and
citations of refereed journal articles to measure the overall scientific
impact. Those measures, however, are broad and can not assess whether the
observatory was successful on a project-by-project basis to deliver data to the
PIs enabling them to carry out their science and to publish their results. In
particular the reasons that prevented PIs from publishing remain hidden.
Understanding and acting upon those reasons, however, have the potential to
substantially improve the observatory's operational model. Of course not every
approved project even should lead to a publication. Indeed, the risk of not
finding the expected (or any unexpected) science in the data the PI receives is
an inherent and indispensable part of the scientific process. But even here,
measuring the fraction of such projects can lead to valuable insights which
might then be used to instruct future proposal review committees. To fully
close the loop on the end-to-end data-flow, ALMA has started in March 2015 to
send survey questions to PIs where two years after the end of the proprietary
period no publication making use of the delivered data could be identified. We
describe our method as well as the type of conclusions we hope to be able to
draw once a statistically relevant sample of answers has been received.

Instrumentation and Methods for Astrophysics