Does there exist any causal explainability tool for NNs? Say I have a neural network which has 1,000,000 parameters built using 100 features and I wish to understand the underlying data-generating process for how the model arrived at each prediction. Simply looking at the value of the node will not provide any information. I am looking to implement a solution that will help to explain the model either by means of a structural causal model (which shows me the causal relationship between endogenous variables) or via an explanation that is comparable to human reasoning. I am opposed to using SHAP values or feature importance tools, not least because it doesn't measure causality, but because values of the importance of features can be skewed by the effect of multi-collinearity. Removing collinearity (for example using PCA) to rectify the skewness leads to reduced interpretability of my variables (contrary to my original task).

I am afraid that the question you are asking doesn't have an answer that would be satisfactory for you. Causal inference us used for finding causal relations in noisy data. A neural network is a mathematical function. It is deterministic, so it directly causes the result to be what it is. Everybmuktiplication, by every weight, addition of every bias, every other mathematical transformation has some, deterministic, impact on the result, so it “causes” it. Surely some of the parameters have a greater impact than others, but causality is not about the strength of the relationship.
What you could do is you could build a simpler model that approximates the neural network and using it will allow you to understand better how it works and this is exactly what expandability tools like LIME do.
