Question: I am using Azure data factory to copy source data into landing zone (adls gen2) and then using auto-loader to load into bronze delta tables. everything works perfectly except I am not able to derive pipeline\_name, runid and trigger\_time as derived columns in parquet file along with input source table.

schema here is structType built using actual source sql server table schema, it does not cover additional derived columns in ADF.

```
sourceFilePath = 'wasbs://landing-zone@dlslandingzonedev.blob.core.windows.net/' \
   + domain_name + '/' + database_name + '/' \
   + schema_table_name.replace(database_name+'.','') + '/'
df = (spark
     .readStream
     .format("cloudFiles")
     .options(**cloudFile)
     .schema(schema)
     .option("rescueDataColumn", "_rescued_data")
     .load(sourceFilePath)
     )
# Traceability columns
# from pyspark.sql.functions import *
df = (
  df.withColumn("audit_fileName", input_file_name()) 
    .withColumn("audit_createdTimestamp", current_timestamp())
)

```

here is writestream DF

```
streamQuery = (df
           .writeStream
           .format("delta")
           .outputMode("append")
           .trigger(once=True)
           .queryName(queryName)
           .option("checkpointLocation",checkpointLocation)
           .option("mergeSchema", "true")
           .start(tablePath)
          )

```

with mergeSchema True - I was expecting stream to detect 3 additional columns from data factory while it writes into delta format. Is this limitation of parquet? do I have read data as csv / json instead? or I have to add derived columns schema definition.

Answer: 
Like you have read from the [documentation](https://cloud.google.com/sql/docs/postgres/users#superuser_restrictions), these commands require `superuser` privileges, which are not available to users in CloudSQL yet.

> 
> Because Cloud SQL for PostgreSQL is a managed service, it restricts access to certain system procedures and tables that require advanced privileges.
> 
> 
> 

There is actually an open [feature request](https://issuetracker.google.com/196607815) focusing on this feature. It does not seem to have an ETA yet as per the available info. Regardless, it appears that there are replication services which do not require `superuser`, as this [thread](https://stackoverflow.com/questions/53108828/) says. Otherwise, GCP also offers the [Data Migration Service](https://cloud.google.com/database-migration/docs/postgres/faq), which, according to the documentation, supports migrating foreign key constraints for PostgreSQL instances.
Actually you don't need superuser to issue

```
SET session_replication_role = replica;
SET session_replication_role = default;

```

you just need REPLICATION ( I just tested it )

```
ALTER ROLE <your user> REPLICATION;

```