Hi All,
We are trying to use tensorflow over spark for numerical computation and further visualize in tensorboard for calculation transperancy.
For TF1.0 - found this cool library - GitHub - databricks/tensorframes: [DEPRECATED] Tensorflow wrapper for DataFrames on Apache Spark 1
which actually allows us to define a spark dataframe column as a tensor and apply operations on top of that.
However, it’s deprecated now. The Github page suggests to use Pandas UDF which does not work with @tf.function decorator and hence we cannot visualize stuff with tensorboard.
In summary, we are interested in the below:

Tensorflow with apache spark.
Manipulate Spark dataframe as Tensors.
Visualize the operations in tensorboard.

Appreciate any leads on this.
Thanks.