title: How to aggregate columns dynamically in pyspark
body: <p>I want to calculate percentage of non-missing value <code>pct_&lt;original_name&gt;_valid</code> for each of the input columns. In this example there are only 2 columns, so it's easy to manually script the code below. But when there are 30+ columns, I don't want to do this manually. Is it even possible to do this dynamically? (for instance, taking a list of column names as an input)</p>
  
  <pre><code>import pyspark.sql.functions as F
  
  d = [{'name': 'Alice', 'age': 1}, {'name': 'Bae', 'age': None}]
  df = spark.createDataFrame(d)
  
  df.withColumn('name_valid', F.when(col("name").isNotNull(),1).otherwise(0))\
  .withColumn('age_valid', F.when(col("age").isNotNull(),1).otherwise(0))\
  .agg(
      (100.0*F.sum(col("name_valid"))/F.count(F.lit(1))).alias("pct_name_valid"),
      (100.0*F.sum(col("age_valid"))/F.count(F.lit(1))).alias("pct_age_valid")
  )\
  .show()
  </code></pre>
  
  <p>Here is the result:</p>
  
  <pre><code>+--------------+-------------+
  |pct_name_valid|pct_age_valid|
  +--------------+-------------+
  |         100.0|         50.0|
  +--------------+-------------+
  </code></pre>
  
  <p>As mentioned earlier, I don't want to manually do this for all 30+ columns. Is there any way I can do like:</p>
  
  <pre><code>my_output = calculate_non_missing_percentage(df, my_columns = ["name", "age", "gender", "school", "color"])
  </code></pre>