Google Cloud Dataflow - Java SDK vs Python SDK
I would go ahead with java SDK as the features and external connectors are more in java. But python SDK is also catching up.
As far as performance considerations are concerned when we submit a beam job to dataflow, the job steps will be sent in an API call to the google cloud dataflow. Hence, I think there's no significant difference in performance as far as dataflow is concerned
I'm starting to use Google Cloud Dataflow for batch and streaming processing. The jobs being developed are mostly for ingesting data from different sources (MySQL, Kafka, and file systems), cleansing them, do some streaming and bath aggregation, and writing back to Google Cloud Storage.
Given these tasks, are there any recommendations for using the Java SDK or Python SDK for writing the jobs? Any noticeable differences in terms of performance and features between them?
For example, I noticed that for the Java SDK, the built-in I/O PTransform JdbcIO is available. This PTransform reads and writes data on JDBC, and this is not available in the Python SDK (so far). Is it possible to use the Java SDK only to create a pipeline that reads from a MySQL database and writes to Google Cloud Storage, while for the other pipelines use a different SDK (e.g. Python)?
Thanks in advance for your time!