[System]
You are a helpful assistant that answers strictly in JSON. Here's the JSON schema you must adhere to:
<schema>
{"$schema":"http://json-schema.org/draft-04/schema#","$comment":"https://probot.github.io/apps/stale/","$ref":"#/definitions/configuration","definitions":{"configuration":{"properties":{"daysUntilStale":{"title":"Days Until Stale","description":"Number of days of inactivity before an Issue or Pull Request becomes stale.","type":"number","default":60},"daysUntilClose":{"title":"Days Until Close","description":"Number of days of inactivity before an Issue or Pull Request with the stale label is closed.","type":["integer","boolean"],"default":7},"onlyLabels":{"title":"Only Labels","description":"Only issues or pull requests with all of these labels are check if stale.","type":"array","items":{"title":"Label","type":"string"},"default":[]},"exemptLabels":{"title":"Exempt Labels","description":"Issues or Pull Requests with these labels will never be considered stale.","type":"array","items":{"title":"Label","type":"string"},"default":[]},"exemptProjects":{"title":"Exempt Projects","description":"Set to true to ignore issues in a milestone.","type":"boolean","default":false},"exemptAssignees":{"title":"Exempt Assignees","description":"Set to true to ignore issues with an assignee.","type":"boolean","default":false},"staleLabel":{"title":"Stale Label","description":"Label to use when marking as stale.","type":"string","default":"wontfix"},"markComment":{"title":"Mark Comment","description":"Comment to post when marking as stale.","type":["string","boolean"],"default":"This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions."},"unmarkComment":{"title":"Unmark Comment","description":"Comment to post when removing the stale label.","type":["string","boolean"]},"closeComment":{"title":"Closed Comment","description":"Comment to post when closing a stale issue or pull request.","type":["string","boolean"]},"limitPerRun":{"title":"Limit Per Run","description":"Limit the number of actions per hour.","type":"number","minimum":1,"maximum":30,"default":30},"only":{"title":"Only","description":"Limit to only issues or pulls requests.","enum":["issues","pulls"]}}}},"description":"A GitHub app that closes abandoned issues and pull requests","id":"https://json.schemastore.org/stale.json","properties":{"pulls":{"$ref":"#/definitions/configuration","title":"Pulls","description":"Specify configuration settings that are specific to pull requests."},"issues":{"$ref":"#/definitions/configuration","title":"Issues","description":"Specify configuration settings that are specific to issues."}},"type":"object"}
</schema>

[User] Provide a JSON object that satisfies the schema "stale". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.

Days Until Stale

Number of days of inactivity before an Issue or Pull Request becomes stale.

Days Until Close

Number of days of inactivity before an Issue or Pull Request with the stale label is closed.

Only Labels

Only issues or pull requests with all of these labels are check if stale.

Label

Exempt Labels

Issues or Pull Requests with these labels will never be considered stale.

Label

Exempt Projects

Set to true to ignore issues in a milestone.

Exempt Assignees

Set to true to ignore issues with an assignee.

Stale Label

Label to use when marking as stale.

Mark Comment

Comment to post when marking as stale.

This issue has been automatically marked as stale because it has not had recent activity. It will be closed if no further activity occurs. Thank you for your contributions.

Unmark Comment

Comment to post when removing the stale label.

Closed Comment

Comment to post when closing a stale issue or pull request.

Limit Per Run

Limit the number of actions per hour.

Only

Limit to only issues or pulls requests.

A GitHub app that closes abandoned issues and pull requests

Pulls

Specify configuration settings that are specific to pull requests.

Issues

Specify configuration settings that are specific to issues.

Provide a JSON object that satisfies the schema "stale". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.


[System]
You are a helpful assistant that answers strictly in JSON. Here's the JSON schema you must adhere to:
<schema>
{"$schema":"http://json-schema.org/draft-07/schema#","$id":"https://json.schemastore.org/starlake.json","title":"Starlake Data Pipeline","description":"JSON Schema for Starlake Data Pipeline","type":"object","properties":{"version":{"type":"integer","enum":[1]}},"allOf":[{"$ref":"#/definitions/StarlakeV1Base"}],"required":["version"],"definitions":{"ConvertibleToString":{"anyOf":[{"type":"string"},{"type":"boolean"},{"type":"number"},{"type":"integer"},{"type":"null"}]},"MergeOnV1":{"oneOf":[{"const":"TARGET","description":"TODO"},{"const":"SOURCE_AND_TARGET","description":"TODO"}]},"PrimitiveTypeV1":{"$ref":"#/definitions/ConvertibleToString","description":"Define the value type","oneOf":[{"const":"string","description":"Any string that match the '.*' regex"},{"const":"long","description":"Any whole number that match the '[-|+|0-9][0-9]*' regex.\nints are mapped to as 'int' in some database whereas 'longs are mapped as 'bigint' and shorts as 'smallint'"},{"const":"int","description":"Any whole number that match the '[-|+|0-9][0-9]*' regex.\nints are mapped to as 'int' in some database whereas 'longs are mapped as 'bigint' and shorts as 'smallint'"},{"const":"short","description":"Any whole number that match the '[-|+|0-9][0-9]*' regex.\nints are mapped to as 'int' in some database whereas 'longs are mapped as 'bigint' and shorts as 'smallint'"},{"const":"double","description":"Any decimal number that match the '[-+]?\\d*\\.?\\d+[Ee]?[-+]?\\d*' regex"},{"const":"boolean","description":"Any string that match the '(?i)true|yes|[y1]<-TF->(?i)false|no|[n0]' regex,\nwhere the value on the left of '<-T' represent true and values on the right of 'F->' represent the false"},{"const":"byte","description":"Any single char"},{"const":"date","description":"Any date that match the 'yyyy-MM-dd' regex (2023-12-31)"},{"const":"timestamp","description":"date/time that match the 'yyyy-MM-dd HH:mm:ss' regex s (2019-12-31 23:59:02).\nFor epoch timestamp, set pattern attribute to 'epoch_second' or 'epoch_milli'"},{"const":"decimal","description":"Any floating value that match the '-?\\d*\\.{0,1}\\d+' regex"},{"const":"variant","description":"Semi structured data type eq. JSON / XML"},{"const":"struct","description":"Any attribute that has children. Set the array to true if this attribute is made of a list of attributes"}]},"TrimV1":{"$ref":"#/definitions/ConvertibleToString","description":"How to trim the input string","oneOf":[{"const":"LEFT","description":"Remove all leading space chars from the input"},{"const":"RIGHT","description":"Remove all trailing spaces from the input"},{"const":"BOTH","description":"Remove all leading and trailing spaces from the input"},{"const":"NONE","description":"Do not remove leading or trailing spaces from the input"}]},"TableDdlV1":{"description":"DDL used to create a table","type":"object","properties":{"createSql":{"$ref":"#/definitions/ConvertibleToString","description":"SQL CREATE DDL statement"},"pingSql":{"$ref":"#/definitions/ConvertibleToString","description":"How to test if the table exist.\nUse the following statement by default: 'select count(*) from tableName where 1=0'"},"selectSql":{"$ref":"#/definitions/ConvertibleToString","description":"Override the default select defined by Starlake"}},"required":["createSql"]},"Materialization":{"$ref":"#/definitions/ConvertibleToString","description":"Table types supported by the Sink option","oneOf":[{"const":"TABLE","description":"SQL Table"},{"const":"VIEW","description":"SQL View"},{"const":"MATERIALIZED_VIEW","description":"SQL Materialized View"}]},"TableTypeBase":{"$ref":"#/definitions/ConvertibleToString","description":"Table types supported by the Extract module","oneOf":[{"const":"TABLE","description":"SQl Table"},{"const":"VIEW","description":"SQl View"},{"const":"SYSTEM TABLE","description":"Database specific system table"},{"const":"MATERIALIZED VIEW","description":"SQL Materialized View"},{"const":"GLOBAL TEMPORARY","description":""},{"const":"LOCAL TEMPORARY","description":""},{"const":"ALIAS","description":"Table alias"},{"const":"SYNONYM","description":"Table synonym"}]},"TableTypeV1":{"$ref":"#/definitions/ConvertibleToString","description":"Table types supported by the Extract module","oneOf":[{"$ref":"#/definitions/TableTypeBase"},{"$ref":"#/definitions/ConvertibleToString","description":"Any valid table types","not":{"$ref":"#/definitions/TableTypeBase"}}]},"TypeV1":{"type":"object","description":"Custom type definition. Custom types are defined in the types/types.sl.yml file","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"unique id for this type"},"pattern":{"$ref":"#/definitions/ConvertibleToString","description":"Regex used to validate the input field"},"primitiveType":{"$ref":"#/definitions/PrimitiveTypeV1","description":"To what primitive type should this type be mapped.\n This is the memory representation of the type, When saving, this primitive type is mapped to the database specific type. Default: string"},"zone":{"$ref":"#/definitions/ConvertibleToString","description":"useful when parsing specific string:\n - double: To parse a french decimal (comma as decimal separator) set it to fr_FR locale.\n- decimal: to set the precision and scale of this number, '38,9' by default.\n- "},"sample":{"$ref":"#/definitions/ConvertibleToString","description":"This field makes sure that the pattern matches the value you want to match. This will be checked on startup"},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Describes this type"},"ddlMapping":{"$ref":"#/definitions/MapString","description":"Configure here the type mapping for each datawarehouse.\\nWill be used when inferring DDL from schema."}},"required":["name","pattern"]},"PositionV1":{"description":"First and last char positions of an attribute in a fixed length record","type":"object","properties":{"first":{"type":"number","description":"Zero based position of the first character for this attribute"},"last":{"type":"number","description":"Zero based position of the last character to include in this attribute"}},"required":["first","last"]},"ConnectionV1":{"description":"Connection properties to a datawarehouse.","type":"object","properties":{"type":{"$ref":"#/definitions/ConvertibleToString","description":"aka jdbc, bigquery, snowflake, redshift ..."},"sparkFormat":{"$ref":"#/definitions/ConvertibleToString","description":"Set only if you want to use the Spark engine"},"loader":{"$ref":"#/definitions/ConvertibleToString","description":"Loader we should use with this connection. Superseded by the loader defined in the file metadata"},"quote":{"$ref":"#/definitions/ConvertibleToString","description":"Set only if you want to use the Spark engine"},"separator":{"$ref":"#/definitions/ConvertibleToString","description":"Set only if you want to use the Spark engine"},"options":{"$ref":"#/definitions/MapString","description":"Connection options"}},"required":["type"]},"DagGenerationConfigV1":{"description":"Dag configuration.","type":"object","properties":{"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Dag config description"},"template":{"$ref":"#/definitions/ConvertibleToString","description":"Dag template to use for this config. Usually a .py.j2 file"},"filename":{"$ref":"#/definitions/ConvertibleToString","description":"{schedule}, {domain}, {table} in the file name are used for DAG generation purposes"},"options":{"$ref":"#/definitions/MapString","description":"DAG generation options"}},"required":["template","filename"]},"RowLevelSecurityV1":{"description":"Row level security policy to apply to the output data.","type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"This Row Level Security unique name"},"predicate":{"$ref":"#/definitions/ConvertibleToString","description":"The condition that goes to the WHERE clause and limit the visible rows."},"grants":{"description":"user / groups / service accounts to which this security level is applied.\nex : user:me@mycompany.com,group:group@mycompany.com,serviceAccount:mysa@google-accounts.com","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"description":{"$ref":"#/definitions/ConvertibleToString","description":"Description for this access policy"}},"required":["name","grants"]},"AccessControlEntryV1":{"description":"Column level security policy to apply to the attribute.","type":"object","properties":{"role":{"$ref":"#/definitions/ConvertibleToString","description":"This role to give to the granted users"},"grants":{"description":"user / groups / service accounts to which this security level is applied.\nex : user:me@mycompany.com,group:group@mycompany.com,serviceAccount:mysa@google-accounts.com","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"name":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"}},"required":["role","grants"]},"FormatV1":{"$ref":"#/definitions/ConvertibleToString","description":"DSV by default. Supported file formats are :\\n- DSV : Delimiter-separated values file. Delimiter value is specified in the \"separator\" field.\\n- POSITION : FIXED format file where values are located at an exact position in each line.\\n- JSON_FLAT : For optimisation purpose, we differentiate JSON with top level values from JSON\\n  with deep level fields. JSON_FLAT are JSON files with top level fields only.\\n- JSON :  Deep JSON file. Use only when your json documents contain sub-documents, otherwise prefer to\\n  use JSON_FLAT since it is much faster.\\n- XML : XML files","oneOf":[{"const":"DATAFRAME","description":"Loader is developer responsibility. loader attribute references a python file that returns a dataframe"},{"const":"DSV","description":"any single or multiple character delimited file. Separator is specified in the separator field"},{"const":"POSITION","description":"any fixed position file. Positions are specified in the position field"},{"const":"JSON","description":"any deep json file.\nTo improve performance, prefer the JSON_FLAT format if your json documents are flat"},{"const":"JSON_ARRAY","description":"any json file containing an array of json objects."},{"const":"JSON_FLAT","description":"any flat json file.\nTo improve performance, prefer this format if your json documents are flat"},{"const":"XML","description":"any xml file. Use the metadata.xml.rowTag field to specify the root tag of your xml file"},{"const":"TEXT_XML","description":"TODO"},{"const":"KAFKA","description":"TODO"},{"const":"KAFKASTREAM","description":"TODO"},{"const":"GENERIC","description":"TODO"},{"const":"PARQUET","description":"TODO"}]},"MapString":{"type":"object","description":"Map of string","additionalProperties":{"$ref":"#/definitions/ConvertibleToString"}},"MapConnectionV1":{"type":"object","description":"Map of jdbc engines","additionalProperties":{"$ref":"#/definitions/ConnectionV1"}},"MapJdbcEngineV1":{"type":"object","description":"Map of jdbc engines","additionalProperties":{"$ref":"#/definitions/JdbcEngineV1"}},"MapTableDdlV1":{"type":"object","description":"Map of table ddl","additionalProperties":{"$ref":"#/definitions/TableDdlV1"}},"JdbcEngineV1":{"type":"object","description":"Jdbc engine","properties":{"tables":{"$ref":"#/definitions/MapTableDdlV1","description":"List of all SQL create statements used to create audit tables for this JDBC engine.\nTables are created only if the execution of the pingSQL statement fails"},"quote":{"type":"string","description":"TODO"},"viewPrefix":{"type":"string","description":"TODO"},"preActions":{"type":"string","description":"TODO"},"partitionBy":{"type":"string","description":"TODO"},"clusterBy":{"type":"string","description":"TODO"},"strategyBuilder":{"type":"string","description":"TODO"},"columnRemarks":{"type":"string","description":"TODO"},"tableRemarks":{"type":"string","description":"TODO"}},"required":["tables","quote","strategyBuilder"]},"PrivacyV1":{"type":"object","properties":{"options":{"$ref":"#/definitions/MapString","description":"Privacy strategies. The following default strategies are defined by default:\n- none: Leave the data as is\n- hide: replace the data with an empty string\n- hideX(\"s\", n): replace the string with n occurrences of the string 's'\n- md5: Redact the data using the MD5 algorithm\n- sha1: Redact the data using the SHA1 algorithm\n- sha256: Redact the data using the SHA256 algorithm\n - sha512: Redact the data using the SHA512 algorithm\n- initials: keep only the first char of each word in the data"}}},"InternalV1":{"type":"object","description":"configure Spark internal options","properties":{"cacheStorageLevel":{"$ref":"#/definitions/ConvertibleToString","description":"How the RDD are cached. Default is MEMORY_AND_DISK_SER.\nAvailable options are (https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/storage/StorageLevel.html):\n- MEMORY_ONLY\n- MEMORY_AND_DISK\n- MEMORY_ONLY_SER\n- MEMORY_AND_DISK_SER\n- DISK_ONLY\n- OFF_HEAP"},"intermediateBigqueryFormat":{"$ref":"#/definitions/ConvertibleToString","description":"May be parquet or ORC. Default is parquet. Used for BigQuery intermediate storage. Use ORC for for JSON files to keep the original data structure.\nhttps://stackoverflow.com/questions/53674838/spark-writing-parquet-arraystring-converts-to-a-different-datatype-when-loadin"},"temporaryGcsBucket":{"$ref":"#/definitions/ConvertibleToString","description":"The GCS bucket that temporarily holds the data before it is loaded to BigQuery."},"substituteVars":{"description":"Internal use. Do not modify.","type":"boolean"},"bqAuditSaveInBatchMode":{"description":"TODO","type":"boolean"}}},"AccessPoliciesV1":{"type":"object","properties":{"apply":{"description":"Should access policies be enforced ?","type":"boolean"},"location":{"$ref":"#/definitions/ConvertibleToString","description":"GCP project location. Required if apply is true."},"database":{"$ref":"#/definitions/ConvertibleToString","description":"GCP Project id. Required if apply is true."},"taxonomy":{"$ref":"#/definitions/ConvertibleToString","description":"Taxonomy name. Required if apply is true."}}},"SparkSchedulingV1":{"type":"object","properties":{"maxJobs":{"description":"Max number of Spark jobs to run in parallel, default is 1","type":"integer"},"poolName":{"$ref":"#/definitions/ConvertibleToString","description":"Pool name to use for Spark jobs, default is 'default'"},"mode":{"$ref":"#/definitions/ConvertibleToString","description":"This can be FIFO or FAIR, to control whether jobs within the pool queue up behind each other (the default) or share the pool\u2019s resources fairly."},"file":{"$ref":"#/definitions/ConvertibleToString","description":"Scheduler filename in the metadata folder. If not set, defaults to fairscheduler.xml."}}},"ExpectationsConfigV1":{"type":"object","properties":{"path":{"$ref":"#/definitions/ConvertibleToString","description":"When using filesystem storage, the path to the expectations file"},"active":{"description":"should expectations be executed ?","type":"boolean"},"failOnError":{"description":"should load / transform fail on expectation error ?","type":"boolean"}}},"ExpectationItemV1":{"type":"object","properties":{"expect":{"$ref":"#/definitions/ConvertibleToString","description":"When using filesystem storage, the path to the expectations file"},"failOnError":{"description":"should load / transform fail on expectation error ?","type":"boolean"}}},"MetricsV1":{"type":"object","properties":{"path":{"$ref":"#/definitions/ConvertibleToString","description":"When using filesystem storage, the path to the metrics file"},"discreteMaxCardinality":{"description":"Max number of unique values accepted for a discrete column. Default is 10","type":"integer"},"active":{"description":"Should metrics be computed ?","type":"boolean"}}},"AllSinksV1":{"type":"object","properties":{"connectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"JDBC: Connection String"},"clustering":{"description":"FS or BQ: List of attributes to use for clustering","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"days":{"type":"number","description":"BQ: Number of days before this table is set as expired and deleted. Never by default."},"requirePartitionFilter":{"type":"boolean","description":"BQ: Should be require a partition filter on every request ? No by default."},"materializedView":{"$ref":"#/definitions/Materialization","description":"Should we materialize as a table or as a view when saving the results ? TABLE by default."},"enableRefresh":{"type":"boolean","description":"BQ: Enable automatic refresh of materialized view ? false by default."},"refreshIntervalMs":{"type":"number","description":"BQ: Refresh interval in milliseconds. Default to BigQuery default value"},"id":{"$ref":"#/definitions/ConvertibleToString","description":"ES: Attribute to use as id of the document. Generated by Elasticsearch if not specified."},"format":{"$ref":"#/definitions/ConvertibleToString","description":"FS: File format"},"extension":{"$ref":"#/definitions/ConvertibleToString","description":"FS: File extension"},"sharding":{"description":"columns to use for sharding. table will be named table_{sharding(0)}_{sharding(1)}","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"partition":{"type":"array","items":{"type":"string"},"description":"FS or BQ: List of partition attributes"},"coalesce":{"type":"boolean","description":"When outputting files, should we coalesce it to a single file. Useful when CSV is the output format."},"path":{"type":"string","description":"Optional path attribute if you want to save the file outside of the default location (datasets folder)"},"options":{"$ref":"#/definitions/MapString","description":"spark  options to use"}}},"WriteStrategyTypeBase":{"$ref":"#/definitions/ConvertibleToString","description":"TODO","oneOf":[{"const":"OVERWRITE","description":"TODO"},{"const":"APPEND","description":"TODO"},{"const":"UPSERT_BY_KEY","description":"TODO"},{"const":"UPSERT_BY_KEY_AND_TIMESTAMP","description":"TODO"},{"const":"DELETE_THEN_INSERT","description":"TODO"},{"const":"SCD2","description":"TODO"},{"const":"OVERWRITE_BY_PARTITION","description":"TODO"}]},"WriteStrategyTypeV1":{"$ref":"#/definitions/ConvertibleToString","description":"Table types supported by the Extract module","oneOf":[{"$ref":"#/definitions/WriteStrategyTypeBase"}]},"OpenWriteStrategyTypeV1":{"$ref":"#/definitions/ConvertibleToString","description":"Table types supported by the Extract module","oneOf":[{"$ref":"#/definitions/WriteStrategyTypeBase"},{"$ref":"#/definitions/ConvertibleToString","description":"Any valid table types","not":{"$ref":"#/definitions/WriteStrategyTypeBase"}}]},"WriteStrategyV1":{"type":"object","properties":{"type":{"$ref":"#/definitions/OpenWriteStrategyTypeV1","description":"TODO"},"types":{"type":"object","description":"TODO","additionalProperties":{"$ref":"#/definitions/OpenWriteStrategyTypeV1"}},"key":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"TODO"},"timestamp":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"queryFilter":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"on":{"$ref":"#/definitions/MergeOnV1","description":"TODO"},"startTs":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"endTs":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"}}},"MetadataV1":{"type":"object","properties":{"format":{"$ref":"#/definitions/FormatV1"},"encoding":{"$ref":"#/definitions/ConvertibleToString","description":"UTF-8 if not specified."},"multiline":{"type":"boolean","description":"are json objects on a single line or multiple line ? Single by default.  false means single. false also means faster"},"array":{"type":"boolean","description":"Is the json stored as a single object array ? false by default. This means that by default we have on json document per line."},"withHeader":{"type":"boolean","description":"does the dataset has a header ? true by default"},"separator":{"$ref":"#/definitions/ConvertibleToString","description":"the values delimiter,  ';' by default value may be a multichar string starting from Spark3"},"quote":{"$ref":"#/definitions/ConvertibleToString","description":"The String quote char, '\"' by default"},"escape":{"$ref":"#/definitions/ConvertibleToString","description":"escaping char '\\' by default"},"sink":{"$ref":"#/definitions/AllSinksV1"},"ignore":{"$ref":"#/definitions/ConvertibleToString","description":"Pattern to ignore or UDF to apply to ignore some lines"},"directory":{"$ref":"#/definitions/ConvertibleToString","description":"Folder on the local filesystem where incoming files are stored.\n Typically, this folder will be scanned periodically to move the dataset to the cluster for ingestion.\n                     Files located in this folder are moved to the stage folder for ingestion by the \"import\" command."},"extensions":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"recognized filename extensions. json, csv, dsv, psv are recognized by default.\nOnly files with these extensions will be moved to the stage folder."},"ack":{"$ref":"#/definitions/ConvertibleToString","description":"Ack extension used for each file. \".ack\" if not specified.\nFiles are moved to the stage folder only once a file with the same name as the source file and with this extension is present.\nTo move a file without requiring an ack file to be present, set explicitly this property to the empty string value \"\"."},"options":{"$ref":"#/definitions/MapString","description":"Options to add to the spark reader"},"loader":{"$ref":"#/definitions/ConvertibleToString","description":"Loader to use, 'spark' or 'native'. Default to 'spark' of SL_LOADER env variable is set to 'native'"},"emptyIsNull":{"description":"Treat empty columns as null in DSV files. Default to false","type":"boolean"},"dagRef":{"$ref":"#/definitions/ConvertibleToString","description":"Cron expression to use for this domain/table"},"freshness":{"$ref":"#/definitions/FreshnessV1","description":"Configure freshness checks on this dataset"},"nullValue":{"$ref":"#/definitions/ConvertibleToString","description":"Treat a specific input string as a null value indicator"},"schedule":{"$ref":"#/definitions/ConvertibleToString","description":"Cron expression to use for this domain/table"},"writeStrategy":{"$ref":"#/definitions/WriteStrategyV1","description":"TODO"}}},"AreaV1":{"type":"object","properties":{"incoming":{"$ref":"#/definitions/ConvertibleToString","description":"Files are read from  this folder for ingestion by the \"import\" command."},"stage":{"$ref":"#/definitions/ConvertibleToString","description":"Files recognized by the extensions property are moved to this folder for ingestion by the \"import\" command."},"unresolved":{"$ref":"#/definitions/ConvertibleToString","description":"Files that cannot be ingested (do not match by any table pattern) are moved to this folder."},"archive":{"$ref":"#/definitions/ConvertibleToString","description":"Files that have been ingested are moved to this folder if SL_ARCHIVE is set to true."},"ingesting":{"$ref":"#/definitions/ConvertibleToString","description":"Files that are being ingested are moved to this folder."},"replay":{"$ref":"#/definitions/ConvertibleToString","description":"Invalid records are stored in this folder in source format when SL_SINK_REPLAY_TO_FILE is set to true."},"hiveDatabase":{"$ref":"#/definitions/ConvertibleToString","description":""}}},"FreshnessV1":{"type":"object","properties":{"warn":{"$ref":"#/definitions/ConvertibleToString","description":"How old may be the data before a warning is raised. Use syntax like '3 day' or '2 hour' or '30 minute'"},"error":{"$ref":"#/definitions/ConvertibleToString","description":"How old may be the data before an error is raised. Use syntax like '3 day' or '2 hour' or '30 minute'"}}},"TableV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Schema name, must be unique among all the schemas belonging to the same domain.\n  *                     Will become the hive table name On Premise or BigQuery Table name on GCP."},"pattern":{"$ref":"#/definitions/ConvertibleToString","description":"filename pattern to which this schema must be applied.\n  *                     This instructs the framework to use this schema to parse any file with a filename that match this pattern."},"attributes":{"description":"Attributes parsing rules.","type":"array","items":{"$ref":"#/definitions/AttributeV1"}},"metadata":{"$ref":"#/definitions/MetadataV1","description":"Dataset metadata"},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"free text"},"streams":{"type":"array","description":"attach streams to table (Snowflake only)","items":{"$ref":"#/definitions/ConvertibleToString"}},"presql":{"type":"array","description":"Reserved for future use.","items":{"$ref":"#/definitions/ConvertibleToString"}},"postsql":{"type":"array","description":"Reserved for future use.","items":{"$ref":"#/definitions/ConvertibleToString"}},"tags":{"description":"Set of string to attach to this Schema","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"rls":{"description":" Row level security on this schema.","type":"array","items":{"$ref":"#/definitions/RowLevelSecurityV1"}},"expectations":{"description":"Expectations to check after Load / Transform has succeeded","type":"array","items":{"$ref":"#/definitions/ExpectationItemV1"}},"primaryKey":{"description":"List of columns that make up the primary key","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"acl":{"description":"Map of rolename -> List[Users].","type":"array","items":{"$ref":"#/definitions/AccessControlEntryV1"}},"rename":{"$ref":"#/definitions/ConvertibleToString","description":"If present, the table is renamed with this name. Useful when use in conjunction with the 'extract' module"},"sample":{"$ref":"#/definitions/ConvertibleToString","description":"Store here a couple of records illustrating the table data."},"filter":{"$ref":"#/definitions/ConvertibleToString","description":"remove all records that do not match this condition"},"patternSample":{"$ref":"#/definitions/ConvertibleToString","description":"Sample of filename matching this schema"}},"required":["name","pattern","attributes"]},"MetricTypeV1":{"$ref":"#/definitions/ConvertibleToString","description":"TODO","oneOf":[{"const":"DISCRETE","description":"TODO"},{"const":"CONTINUOUS","description":"TODO"},{"const":"TEXT","description":"TODO"},{"const":"NONE","description":"TODO"}]},"AttributeV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Attribute name as defined in the source dataset and as received in the file"},"type":{"$ref":"#/definitions/ConvertibleToString","description":"semantic type of the attribute. Default to string"},"array":{"type":"boolean","description":"Is it an array ?"},"required":{"type":"boolean","description":"Should this attribute always be present in the source. Default to true."},"privacy":{"$ref":"#/definitions/ConvertibleToString","description":"Should this attribute be applied a privacy transformation at ingestion time"},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"free text for attribute description"},"rename":{"$ref":"#/definitions/ConvertibleToString","description":"If present, the attribute is renamed with this name"},"sample":{"$ref":"#/definitions/ConvertibleToString","description":"Sample data for this attribute"},"metricType":{"$ref":"#/definitions/MetricTypeV1","description":"If present, what kind of stat should be computed for this field"},"attributes":{"type":"array","description":"List of sub-attributes (valid for JSON and XML files only)","items":{"$ref":"#/definitions/AttributeV1"}},"position":{"$ref":"#/definitions/PositionV1"},"default":{"$ref":"#/definitions/ConvertibleToString","description":"Default value for this attribute when it is not present."},"tags":{"type":"array","description":"Tags associated with this attribute","items":{"$ref":"#/definitions/ConvertibleToString"}},"trim":{"$ref":"#/definitions/TrimV1"},"script":{"$ref":"#/definitions/ConvertibleToString","description":"Scripted field : SQL request on renamed column"},"foreignKey":{"$ref":"#/definitions/ConvertibleToString","description":"If this attribute is a foreign key, reference to [domain.]table[.attribute]"},"ignore":{"type":"boolean","description":"Should this attribute be ignored on ingestion. Default to false"},"accessPolicy":{"$ref":"#/definitions/ConvertibleToString","description":"Policy tag to assign to this attribute. Used for column level security"}},"required":["name"]},"AutoTaskDescV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Main SQL request to execute (do not forget to prefix table names with the database name to avoid conflicts)"},"sql":{"$ref":"#/definitions/ConvertibleToString","description":"Main SQL request to execute (do not forget to prefix table names with the database name to avoid conflicts)"},"streams":{"type":"array","description":"attach streams to task (Snowflake only)","items":{"$ref":"#/definitions/ConvertibleToString"}},"database":{"$ref":"#/definitions/ConvertibleToString","description":"Output Database (refer to a project id in BigQuery). Default to SL_DATABASE env var if set."},"domain":{"$ref":"#/definitions/ConvertibleToString","description":"Output domain in output Area (Will be the Database name in Hive or Dataset in BigQuery)"},"table":{"$ref":"#/definitions/ConvertibleToString","description":"Dataset Name in output Area (Will be the Table name in Hive & BigQuery)"},"partition":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"List of columns used for partitioning the output."},"presql":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"List of SQL requests to executed before the main SQL request is run"},"postsql":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"List of SQL requests to executed after the main SQL request is run"},"sink":{"$ref":"#/definitions/AllSinksV1"},"rls":{"type":"array","items":{"$ref":"#/definitions/RowLevelSecurityV1"}},"expectations":{"description":"Expectations to check after Load / Transform has succeeded","type":"array","items":{"$ref":"#/definitions/ExpectationItemV1"}},"acl":{"description":"Map of rolename -> List[Users].","type":"array","items":{"$ref":"#/definitions/AccessControlEntryV1"}},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Output table description"},"freshness":{"$ref":"#/definitions/FreshnessV1","description":"Configure freshness checks on the output table"},"attributes":{"description":"Attributes comments and access policies","type":"array","items":{"$ref":"#/definitions/AttributeDescV1"}},"python":{"$ref":"#/definitions/ConvertibleToString","description":"Python script URI to execute instead of the SQL request"},"tags":{"description":"Set of string to attach to the output table","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"writeStrategy":{"$ref":"#/definitions/WriteStrategyV1","description":"TODO"},"schedule":{"$ref":"#/definitions/ConvertibleToString","description":"Cron expression to use for this task"},"dagRef":{"$ref":"#/definitions/ConvertibleToString","description":"Cron expression to use for this domain/table"},"taskTimeoutMs":{"type":"integer","description":"Number of milliseconds before a communication timeout."},"parseSQL":{"type":"boolean","description":"Should we parse this SQL make it update the table according to write strategy or just execute it ?"},"connectionRef":{"type":"string","description":"Used when the default connection ref present in the application.sl.yml file is not the one to use to run the SQL request for this task."}},"required":[]},"LockV1":{"type":"object","properties":{"path":{"$ref":"#/definitions/ConvertibleToString","description":"Name of the lock"},"timeout":{"type":"integer","description":"TODO"},"pollTime":{"type":"integer","description":"TODO. Default 5 seconds"},"refreshTime":{"type":"integer","description":"TODO. Default 5 seconds"}}},"AuditV1":{"type":"object","properties":{"path":{"$ref":"#/definitions/ConvertibleToString","description":"Main SQL request to execute (do not forget to prefix table names with the database name to avoid conflicts)"},"sink":{"$ref":"#/definitions/AllSinksV1","description":"Output Database (refer to a project id in BigQuery). Default to SL_DATABASE env var if set."},"maxErrors":{"$ref":"#/definitions/ConvertibleToString","description":"Output domain in output Area (Will be the Database name in Hive or Dataset in BigQuery)"},"database":{"$ref":"#/definitions/ConvertibleToString","description":"Dataset Name in output Area (Will be the Table name in Hive & BigQuery)"},"domain":{"$ref":"#/definitions/ConvertibleToString"},"domainExpectation":{"$ref":"#/definitions/ConvertibleToString"},"domainRejected":{"$ref":"#/definitions/ConvertibleToString"},"active":{"type":"boolean","description":"Output table description"},"sql":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"}},"required":[]},"AttributeDescV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Column name"},"type":{"$ref":"#/definitions/ConvertibleToString","description":"Column Type"},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Column description"},"accessPolicy":{"$ref":"#/definitions/ConvertibleToString","description":"Access policy to apply to this column"}},"required":["name"]},"DomainV1":{"type":"object","description":"A schema in JDBC database or a folder in HDFS or a dataset in BigQuery.","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Domain name. Make sure you use a name that may be used as a folder name on the target storage.\n                   - When using HDFS or Cloud Storage,  files once ingested are stored in a sub-directory named after the domain name.\n                   - When used with BigQuery, files are ingested and sorted in tables under a dataset named after the domain name."},"metadata":{"$ref":"#/definitions/MetadataV1"},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Domain Description (free text)"},"tags":{"description":"Set of string to attach to this domain","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}},"rename":{"$ref":"#/definitions/ConvertibleToString","description":"If present, the attribute is renamed with this name"},"database":{"$ref":"#/definitions/ConvertibleToString","description":"Output Database (refer to a project id in BigQuery). Default to SL_DATABASE env var if set."}}},"AutoJobDescV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Optional name. If not specified, the name of the file without the extension is used."},"tasks":{"type":"array","items":{"$ref":"#/definitions/AutoTaskDescV1","description":"List of transform tasks to execute"}},"comment":{"$ref":"#/definitions/ConvertibleToString","description":"Optional description."},"default":{"$ref":"#/definitions/AutoTaskDescV1","description":"Default task properties to apply to all tasks defined in tasks section and in included files"}}},"JDBCTableV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Table name. Set to '*' to extract all tables. Scope: Schema and Data extraction."},"sql":{"$ref":"#/definitions/ConvertibleToString","description":"SQL used to extract schema and data. Ignore columns attribute if set."},"columns":{"description":"List of columns to extract. All columns by default.","type":"array","minItems":1,"items":{"oneOf":[{"$ref":"#/definitions/ConvertibleToString","description":"Column name to extract. Scope: Schema and Data extraction."},{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Column name to extract. Scope: Schema and Data extraction."},"rename":{"$ref":"#/definitions/ConvertibleToString","description":"Rename database column name. Scope: Schema and Data extraction."}},"required":["name"]}]}},"partitionColumn":{"$ref":"#/definitions/ConvertibleToString"},"numPartitions":{"type":"integer"},"connectionOptions":{"$ref":"#/definitions/MapString"},"fetchSize":{"type":"integer"},"fullExport":{"type":"boolean"},"filter":{"$ref":"#/definitions/ConvertibleToString","description":"Filter applied along data extraction. Scope: Data extraction"},"stringPartitionFunc":{"$ref":"#/definitions/ConvertibleToString","description":"SQL template used on partition column's of type String. Some implementations are already defined, see ai.starlake.extract.JdbcDbUtils.getStringPartitionFunc. Mandatory variables: col, nb_partitions. Scope: Data extraction."}}},"OutputV1":{"type":"object","properties":{"encoding":{"$ref":"#/definitions/ConvertibleToString","description":"Specifies encoding (charset) of saved CSV files."},"withHeader":{"type":"boolean","description":"If true, writes the names of columns as the first line."},"separator":{"$ref":"#/definitions/ConvertibleToString","description":"Character used as a separator for each field and value. TODO: check if we are expecting character."},"quote":{"$ref":"#/definitions/ConvertibleToString","description":"Character used for escaping quoted values where the separator can be part of the value. TODO: check if we are expecting character."},"escape":{"$ref":"#/definitions/ConvertibleToString","description":"Character used for escaping quotes inside an already quoted value. TODO: check if we are expecting character."},"nullValue":{"$ref":"#/definitions/ConvertibleToString","description":"String representation of a null value."},"datePattern":{"$ref":"#/definitions/ConvertibleToString","description":"Java date pattern to apply on date object. Have a look at https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html"},"timestampPattern":{"$ref":"#/definitions/ConvertibleToString","description":"Java timestamp pattern to apply on timestamp object. Have a look at https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html"}}},"JDBCSchemaBase":{"type":"object","properties":{"catalog":{"$ref":"#/definitions/ConvertibleToString","description":"Optional catalog name in the source database. Scope: Schema and Data extraction."},"schema":{"$ref":"#/definitions/ConvertibleToString","description":"Database schema where source tables are located. For mysql use this instead of catalog. Scope: Schema and Data extraction."},"tableRemarks":{"$ref":"#/definitions/ConvertibleToString","description":"Query template used to retrieve table remark. Available variables: catalog, schema, table. Scope: Schema extraction."},"columnRemarks":{"$ref":"#/definitions/ConvertibleToString","description":"Query template used to retrieve all columns' remark of a table. Available variables: catalog, schema, table. Scope: Schema extraction."},"tableTypes":{"description":"One or many of the predefined table types. Scope: Schema and Data extraction.","type":"array","items":{"$ref":"#/definitions/TableTypeV1"}},"template":{"$ref":"#/definitions/ConvertibleToString","description":"Template used during schema extraction in order to generate load files (domain and tables). Scope: Schema extraction."},"pattern":{"$ref":"#/definitions/ConvertibleToString","description":"Pattern template used to define load tables' file pattern. Available variables: catalog, schema, table. Scope: Schema extraction."},"numericTrim":{"$ref":"#/definitions/TrimV1","description":"Trim strategies applied to numeric fields set on load table's definition. Scope: Schema extraction."},"partitionColumn":{"$ref":"#/definitions/ConvertibleToString","description":"Column to use in order to parallelize data extraction. Scope: Data extraction."},"numPartitions":{"type":"integer","description":"Number of data partitions to create. Scope: Data extraction."},"connectionOptions":{"$ref":"#/definitions/MapString","description":"Options to set on data connection. Scope: Data extraction."},"fetchSize":{"type":"integer","description":"Number of rows to be fetched from the database when additional rows are needed. By default, most JDBC drivers use a fetch size of 10, so if you are reading 1000 objects, increasing the fetch size to 256 can significantly reduce the time required to fetch the query's results. The optimal fetch size is not always obvious. Scope: Data extraction."},"stringPartitionFunc":{"$ref":"#/definitions/ConvertibleToString","description":"SQL template used on partition column's of type String. Some implementations are already defined, see ai.starlake.extract.JdbcDbUtils.getStringPartitionFunc. Mandatory variables: col, nb_partitions. Scope: Data extraction."},"fullExport":{"type":"boolean","description":"Define if we should fetch the entire table's or not. If not, maximum value of partitionColumn seen during last extraction is used in order to fetch incremental data. Scope: Data extraction."},"sanitizeName":{"type":"boolean","description":"Sanitize domain's name by keeping alpha numeric characters only. Scope: Schema and Data extraction."}}},"DefaultJDBCSchemaV1":{"$ref":"#/definitions/JDBCSchemaBase"},"JDBCSchemaV1":{"type":"object","allOf":[{"$ref":"#/definitions/JDBCSchemaBase"},{"properties":{"tables":{"type":"array","description":"List of tables to extract. Scope: Schema and Data extraction.","items":{"$ref":"#/definitions/JDBCTableV1"}},"exclude":{"type":"array","description":"List of tables to exclude. Applied on tables list. Scope: Schema and Data extraction.","items":{"$ref":"#/definitions/ConvertibleToString"}}}}]},"JDBCSchemasV1":{"type":"object","allOf":[{"$ref":"#/definitions/ExtractV1Base"},{"properties":{"jdbcSchemas":{"description":"Describe what to fetch from data connection. Scope: Schema and Data extraction.","type":"array","items":{"$ref":"#/definitions/JDBCSchemaV1"}},"auditConnectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"Connection used to read/store audit from it. If not defined, fallbacks to connectionRef. Expected connection name as defined in the connections section of the application.conf file. Scope: Data extraction."},"output":{"$ref":"#/definitions/OutputV1","description":"Override the output format of data extraction. Scope: Data extraction."},"default":{"$ref":"#/definitions/DefaultJDBCSchemaV1","description":"Configuration merged into each jdbcSchemas. Scope: Schema and Data extraction."}},"not":{"required":["openAPI"]}}]},"OpenAPIObjectSchemasV1":{"type":"object","properties":{"include":{"description":"List of regex used to include open api schemas (#/components/schemas). Defaults to ['.*']. 'Includes' is evaluated before 'excludes'","type":"array","minItems":1,"items":{"$ref":"#/definitions/ConvertibleToString"}},"exclude":{"description":"List of regex used to exclude open api schemas (#/components/schemas). Defaults to [].","type":"array","items":{"$ref":"#/definitions/ConvertibleToString"}}}},"OpenAPIRouteObjectExplosionV1":{"type":"object","properties":{"on":{"description":"Explode route's object to more object definition. Use object's path with route path as final name. Defaults to ALL","type":"string","anyOf":[{"const":"ALL","title":"Keep properties of type object or array."},{"const":"OBJECT","title":"Keep properties of type object. Don't dive on array type."},{"const":"ARRAY","title":"Keep properties of type array. If encounters an object, dive deeper."}]},"exclude":{"type":"array","description":"filter out on field path. Each field is separated by _. Default to []","items":{"$ref":"#/definitions/ConvertibleToString"}},"rename":{"type":"object","description":"Regex applied on object path. If matches, use the given name otherwise fallback to route_path + object path as final name","additionalProperties":{"$ref":"#/definitions/ConvertibleToString"}}}},"OpenAPIRoutesV1":{"type":"object","properties":{"paths":{"description":"List of regex used to include open api path '.*'","type":"array","minItems":1,"items":{"$ref":"#/definitions/ConvertibleToString"}},"as":{"$ref":"#/definitions/ConvertibleToString","description":"Force all routes matching the pattern to be saved as the given name if they don't conflict"},"operations":{"description":"List of operations to retrieve schema from. Defaults to ['GET']. Supported values are GET and POST.","type":"array","minItems":1,"items":{"anyOf":[{"const":"GET","description":"Retain operation of type GET in openAPI"},{"const":"POST","title":"Retain operation of type POST in openAPI"}]}},"exclude":{"description":"List of regex used to excludes api path []","type":"array","minItems":1,"items":{"$ref":"#/definitions/ConvertibleToString"}},"excludeFields":{"description":"List of regex used to excludes fields. Fields and their subfields are separated by _.","type":"array","minItems":1,"items":{"$ref":"#/definitions/ConvertibleToString"}},"explode":{"$ref":"#/definitions/OpenAPIRouteObjectExplosionV1","description":"Explodes on route's object and split the schema."}}},"OpenAPIDomainV1":{"type":"object","properties":{"name":{"$ref":"#/definitions/ConvertibleToString","description":"Domain name used to group tables extracted from openAPI spec"},"basePath":{"$ref":"#/definitions/ConvertibleToString","description":"Common base path used to remove from path in order to generate final table name."},"schemas":{"$ref":"#/definitions/OpenAPIObjectSchemasV1","description":"Describe what to fetch from data connection. Scope: Schema and Data extraction."},"routes":{"description":"Describe what to fetch from data connection. Scope: Schema and Data extraction.","type":"array","minItems":1,"items":{"$ref":"#/definitions/OpenAPIRoutesV1"}}},"required":["name"]},"OpenAPIV1":{"type":"object","properties":{"basePath":{"$ref":"#/definitions/ConvertibleToString","description":"Common base path used to remove from path in order to generate final table name."},"formatTypeMapping":{"$ref":"#/definitions/MapString","description":"mapping a format used for string and the starlake attribute type"},"domains":{"description":"Describe what to fetch from data connection. Scope: Schema and Data extraction.","type":"array","minItems":1,"items":{"$ref":"#/definitions/OpenAPIDomainV1"}}}},"OpenAPIsV1":{"type":"object","allOf":[{"$ref":"#/definitions/ExtractV1Base"},{"properties":{"openAPI":{"$ref":"#/definitions/OpenAPIV1","description":"Describe how to extract domains and tables from OpenAPI spec"}},"required":["openAPI"]}]},"ExtractV1Base":{"type":"object","properties":{"sanitizeAttributeName":{"type":"string","anyOf":[{"const":"ON_EXTRACT","description":"attribute name is sanitized and stored as field name. Default."},{"const":"ON_LOAD","title":"attribute name is sanitized and stored as rename property when attribute's name differs from sanitized name"}]},"connectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"With OpenAPI: connection used to reference OpenAPI spec. Supports file (local, storage) or public http url. With others, connection used to retrieve schema and then data. If not defined, fallback to application settings."}}},"InputRefV1":{"description":"Input for ref object","type":"object","properties":{"database":{"$ref":"#/definitions/ConvertibleToString","description":"Database pattern to match, none if any database"},"domain":{"$ref":"#/definitions/ConvertibleToString","description":"Domain pattern to match, none if any domain match"},"table":{"$ref":"#/definitions/ConvertibleToString","description":"Table pattern to match"}},"required":["table"]},"OutputRefV1":{"description":"Output for ref object","type":"object","properties":{"database":{"$ref":"#/definitions/ConvertibleToString","description":""},"domain":{"$ref":"#/definitions/ConvertibleToString","description":""},"table":{"$ref":"#/definitions/ConvertibleToString","description":""}},"required":["table","domain","database"]},"RefV1":{"description":"Describe how to resolve a reference in a transform task","type":"object","properties":{"input":{"$ref":"#/definitions/InputRefV1","description":"The input table to resolve"},"output":{"$ref":"#/definitions/OutputRefV1","description":"The output table resolved with the domain and database"}},"required":["input","output"]},"KafkaTopicConfigV1":{"properties":{"topicName":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"maxRead":{"type":"integer","description":"TODO"},"fields":{"type":"array","items":{"$ref":"#/definitions/ConvertibleToString"},"description":"TODO"},"partitions":{"type":"integer","description":"TODO"},"replicationFactor":{"type":"integer","description":"TODO"},"createOptions":{"$ref":"#/definitions/MapString","description":"TODO"},"accessOptions":{"$ref":"#/definitions/MapString","description":"TODO"},"headers":{"type":"object","additionalProperties":{"$ref":"#/definitions/MapString"},"description":"TODO"}}},"KafkaConfigV1":{"type":"object","properties":{"serverOptions":{"$ref":"#/definitions/MapString","description":"TODO"},"topics":{"type":"object","description":"Map of topic configs","additionalProperties":{"$ref":"#/definitions/KafkaTopicConfigV1"}},"cometOffsetsMode":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"customDeserializers":{"$ref":"#/definitions/MapString","description":"TODO"}}},"DagRefV1":{"type":"object","properties":{"load":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"transform":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"}}},"HttpV1":{"type":"object","properties":{"interface":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"port":{"type":"integer","description":"TODO"}}},"AppConfigV1":{"type":"object","properties":{"env":{"$ref":"#/definitions/ConvertibleToString","description":"Default environment to use. May be also set using the SL_ENV environment variable"},"datasets":{"$ref":"#/definitions/ConvertibleToString","description":"When using filesystem storage, default path to store the datasets"},"incoming":{"$ref":"#/definitions/ConvertibleToString","description":"Incoming folder to use during autoload"},"dags":{"$ref":"#/definitions/ConvertibleToString","description":"DAG generation config folder. metadata/dags by default"},"tests":{"$ref":"#/definitions/ConvertibleToString","description":"Path to tests folder. Default is ${metadata}/tests"},"writeStrategies":{"$ref":"#/definitions/ConvertibleToString","description":"Location where are located user defined write strategies; Default is ${metadata}/write-strategies"},"metadata":{"$ref":"#/definitions/ConvertibleToString","description":"default metadata folder name. May be also set using the SL_METADATA environment variable"},"metrics":{"$ref":"#/definitions/MetricsV1"},"validateOnLoad":{"type":"boolean","description":"Validate the YAML file when loading it. If set to true fails on any error"},"audit":{"$ref":"#/definitions/AuditV1"},"archive":{"type":"boolean","description":"Should ingested files be archived after ingestion ?"},"sinkReplayToFile":{"type":"boolean","description":"Should invalid records be stored in a replay file ?"},"lock":{"$ref":"#/definitions/LockV1"},"defaultWriteFormat":{"$ref":"#/definitions/ConvertibleToString","description":"Default write format in Spark. parquet is the default"},"defaultRejectedWriteFormat":{"$ref":"#/definitions/ConvertibleToString","description":"Default write format in Spark for rejected records. parquet is the default"},"defaultAuditWriteFormat":{"$ref":"#/definitions/ConvertibleToString","description":"Default write format in Spark for audit records. parquet is the default"},"csvOutput":{"type":"boolean","description":"output files in CSV format ? Default is false"},"csvOutputExt":{"$ref":"#/definitions/ConvertibleToString","description":"CSV file extension when csvOutput is true. Default is .csv"},"privacyOnly":{"type":"boolean","description":"Only generate privacy tasks. Reserved for internal use"},"emptyIsNull":{"type":"boolean","description":"Should empty strings be considered as null values ?"},"loader":{"$ref":"#/definitions/ConvertibleToString","description":"Default loader to use when none is specified in the schema. Valid values are 'spark' or 'native'. Default is 'spark'"},"rowValidatorClass":{"$ref":"#/definitions/ConvertibleToString","description":""},"treeValidatorClass":{"$ref":"#/definitions/ConvertibleToString","description":""},"loadStrategyClass":{"$ref":"#/definitions/ConvertibleToString","description":"In what order should the files for a same table be loaded ? By time (default) or by or name ?\n","anyOf":[{"const":"ai.starlake.job.load.IngestionNameStrategy","description":"Order pending files by name"},{"const":"ai.starlake.job.load.IngestionTimeStrategy","title":"Order pending files by creation date time"}]},"grouped":{"type":"boolean","description":"Should we load of the files to be stored in the same table in a single task or one by one ?"},"groupedMax":{"type":"integer","description":"Maximum number of files to be stored in the same table in a single task"},"scd2StartTimestamp":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"scd2EndTimestamp":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"area":{"$ref":"#/definitions/AreaV1","description":"stage, ingesting ... areas configuration"},"hadoop":{"$ref":"#/definitions/MapString","description":"Hadoop configuration if applicable"},"connections":{"$ref":"#/definitions/MapConnectionV1","description":"Connections configurations"},"jdbcEngines":{"$ref":"#/definitions/MapJdbcEngineV1","description":"JDBC engine configurations"},"privacy":{"$ref":"#/definitions/PrivacyV1","description":"Privacy algorithms"},"root":{"$ref":"#/definitions/ConvertibleToString","description":"Root folder for the application. May be also set using the SL_ROOT environment variable"},"internal":{"$ref":"#/definitions/InternalV1","description":"Internal configuration"},"accessPolicies":{"$ref":"#/definitions/AccessPoliciesV1","description":"Access policies configuration"},"sparkScheduling":{"$ref":"#/definitions/SparkSchedulingV1","description":"Spark Job scheduling configuration"},"udfs":{"$ref":"#/definitions/ConvertibleToString","description":"Coma separated list of UDF to register in Spark jobs. May be also set using the SL_UDFS environment variable"},"expectations":{"$ref":"#/definitions/ExpectationsConfigV1","description":"Expectations configuration"},"sqlParameterPattern":{"$ref":"#/definitions/ConvertibleToString","description":"Pattern to use to replace parameters in SQL queries in addition to the jinja syntax {{param}}. Default is ${param}"},"rejectAllOnError":{"$ref":"#/definitions/ConvertibleToString","description":"Should we reject all records when an error occurs ? Default is false"},"rejectMaxRecords":{"type":"integer","description":"Maximum number of records to reject when an error occurs. Default is 100"},"maxParCopy":{"type":"integer","description":""},"kafka":{"$ref":"#/definitions/KafkaConfigV1","description":"TODO"},"dsvOptions":{"$ref":"#/definitions/MapString","description":"DSV ingestion extra options"},"forceViewPattern":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"forceDomainPattern":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"forceTablePattern":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"forceJobPattern":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"forceTaskPattern":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"useLocalFileSystem":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"sessionDurationServe":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"database":{"$ref":"#/definitions/ConvertibleToString","description":"Default target database (projectId in GCP). May be also set using the SL_DATABASE environment variable"},"tenant":{"$ref":"#/definitions/ConvertibleToString","description":"reserved"},"connectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"Default connection to use when loading / transforming data"},"loadConnectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"Default connection to use when loading / transforming data"},"transformConnectionRef":{"$ref":"#/definitions/ConvertibleToString","description":"Default connection to use when loading / transforming data"},"schedulePresets":{"$ref":"#/definitions/MapString","description":"Default connection to use when loading / transforming data"},"maxParTask":{"type":"integer","description":"How many job to run simultaneously in dev mode (experimental)"},"refs":{"type":"array","description":"TODO","items":{"$ref":"#/definitions/RefV1"}},"dagRef":{"$ref":"#/definitions/DagRefV1","description":"Default connection to use when loading / transforming data"},"forceHalt":{"type":"boolean","description":"Force application to stop even when there is some pending thread."},"jobIdEnvName":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"archiveTablePattern":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"archiveTable":{"type":"boolean","description":"TODO"},"version":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"autoExportSchema":{"type":"boolean","description":"TODO"},"longJobTimeoutMs":{"type":"integer","description":"TODO"},"shortJobTimeoutMs":{"type":"integer","description":"TODO"},"createSchemaIfNotExists":{"type":"boolean","description":"TODO"},"http":{"$ref":"#/definitions/HttpV1","description":"TODO"},"timezone":{"$ref":"#/definitions/ConvertibleToString","description":"TODO"},"maxInteractiveRecords":{"type":"integer","description":"TODO"},"duckdbMode":{"type":"boolean","description":"is duckdb mode active"},"duckdbExtensions":{"type":"string","description":"Comma separated list of duckdb extensions to load. Default is spatial, json, httpfs"},"duckdbPath":{"$ref":"#/definitions/ConvertibleToString","description":"Where to store duckdb files if not using default"},"testCsvNullString":{"$ref":"#/definitions/ConvertibleToString","description":"null string value in tests"},"hiveInTest":{"$ref":"#/definitions/ConvertibleToString","description":"Internal use only"},"spark":{"type":"object","description":"Map of string","additionalProperties":true},"extra":{"type":"object","description":"Map of string","additionalProperties":true},"duckDbEnableExternalAccess":{"description":"Allow DuckDB to load / Save data from / to external sources. Default to true","type":"boolean"}}},"StarlakeV1Base":{"type":"object","properties":{"types":{"type":"array","items":{"$ref":"#/definitions/TypeV1"}},"dag":{"$ref":"#/definitions/DagGenerationConfigV1"},"extract":{"oneOf":[{"$ref":"#/definitions/JDBCSchemasV1"},{"$ref":"#/definitions/OpenAPIsV1","description":"Defines OpenAPI schemas extraction"}]},"load":{"$ref":"#/definitions/DomainV1"},"transform":{"$ref":"#/definitions/AutoJobDescV1"},"task":{"$ref":"#/definitions/AutoTaskDescV1"},"env":{"$ref":"#/definitions/MapString"},"table":{"$ref":"#/definitions/TableV1"},"refs":{"type":"array","items":{"$ref":"#/definitions/RefV1"}},"application":{"$ref":"#/definitions/AppConfigV1"}},"oneOf":[{"required":["extract"]},{"required":["load"]},{"required":["transform"]},{"required":["env"]},{"required":["types"]},{"required":["tables"]},{"required":["table"]},{"required":["task"]},{"required":["application"]},{"required":["refs"]},{"required":["dag"]}]}}}
</schema>

[User] Provide a JSON object that satisfies the schema "Starlake Data Pipeline". Here are the required top‑level fields: version. Return only the valid JSON object.

Starlake Data Pipeline

JSON Schema for Starlake Data Pipeline

TODO

TODO

Define the value type

Any string that match the '.*' regex

Any whole number that match the '[-|+|0-9][0-9]*' regex.
ints are mapped to as 'int' in some database whereas 'longs are mapped as 'bigint' and shorts as 'smallint'

Any decimal number that match the '[-+]?\d*\.?\d+[Ee]?[-+]?\d*' regex

Any string that match the '(?i)true|yes|[y1]<-TF->(?i)false|no|[n0]' regex,
where the value on the left of '<-T' represent true and values on the right of 'F->' represent the false

Any single char

Any date that match the 'yyyy-MM-dd' regex (2023-12-31)

date/time that match the 'yyyy-MM-dd HH:mm:ss' regex s (2019-12-31 23:59:02).
For epoch timestamp, set pattern attribute to 'epoch_second' or 'epoch_milli'

Any floating value that match the '-?\d*\.{0,1}\d+' regex

Semi structured data type eq. JSON / XML

Any attribute that has children. Set the array to true if this attribute is made of a list of attributes

How to trim the input string

Remove all leading space chars from the input

Remove all trailing spaces from the input

Remove all leading and trailing spaces from the input

Do not remove leading or trailing spaces from the input

DDL used to create a table

SQL CREATE DDL statement

How to test if the table exist.
Use the following statement by default: 'select count(*) from tableName where 1=0'

Override the default select defined by Starlake

Table types supported by the Sink option

SQL Table

SQL View

SQL Materialized View

Table types supported by the Extract module

SQl Table

SQl View

Database specific system table

SQL Materialized View

Table alias

Table synonym

Any valid table types

Custom type definition. Custom types are defined in the types/types.sl.yml file

unique id for this type

Regex used to validate the input field

To what primitive type should this type be mapped.
 This is the memory representation of the type, When saving, this primitive type is mapped to the database specific type. Default: string

useful when parsing specific string:
 - double: To parse a french decimal (comma as decimal separator) set it to fr_FR locale.
- decimal: to set the precision and scale of this number, '38,9' by default.
-

This field makes sure that the pattern matches the value you want to match. This will be checked on startup

Describes this type

Configure here the type mapping for each datawarehouse.\nWill be used when inferring DDL from schema.

First and last char positions of an attribute in a fixed length record

Zero based position of the first character for this attribute

Zero based position of the last character to include in this attribute

Connection properties to a datawarehouse.

aka jdbc, bigquery, snowflake, redshift ...

Set only if you want to use the Spark engine

Loader we should use with this connection. Superseded by the loader defined in the file metadata

Connection options

Dag configuration.

Dag config description

Dag template to use for this config. Usually a .py.j2 file

{schedule}, {domain}, {table} in the file name are used for DAG generation purposes

DAG generation options

Row level security policy to apply to the output data.

This Row Level Security unique name

The condition that goes to the WHERE clause and limit the visible rows.

user / groups / service accounts to which this security level is applied.
ex : user:me@mycompany.com,group:group@mycompany.com,serviceAccount:mysa@google-accounts.com

Description for this access policy

Column level security policy to apply to the attribute.

This role to give to the granted users

TODO

DSV by default. Supported file formats are :\n- DSV : Delimiter-separated values file. Delimiter value is specified in the "separator" field.\n- POSITION : FIXED format file where values are located at an exact position in each line.\n- JSON_FLAT : For optimisation purpose, we differentiate JSON with top level values from JSON\n  with deep level fields. JSON_FLAT are JSON files with top level fields only.\n- JSON :  Deep JSON file. Use only when your json documents contain sub-documents, otherwise prefer to\n  use JSON_FLAT since it is much faster.\n- XML : XML files

Loader is developer responsibility. loader attribute references a python file that returns a dataframe

any single or multiple character delimited file. Separator is specified in the separator field

any fixed position file. Positions are specified in the position field

any deep json file.
To improve performance, prefer the JSON_FLAT format if your json documents are flat

any json file containing an array of json objects.

any flat json file.
To improve performance, prefer this format if your json documents are flat

any xml file. Use the metadata.xml.rowTag field to specify the root tag of your xml file

TODO

TODO

TODO

TODO

TODO

Map of string

Map of jdbc engines

Map of jdbc engines

Map of table ddl

Jdbc engine

List of all SQL create statements used to create audit tables for this JDBC engine.
Tables are created only if the execution of the pingSQL statement fails

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

Privacy strategies. The following default strategies are defined by default:
- none: Leave the data as is
- hide: replace the data with an empty string
- hideX("s", n): replace the string with n occurrences of the string 's'
- md5: Redact the data using the MD5 algorithm
- sha1: Redact the data using the SHA1 algorithm
- sha256: Redact the data using the SHA256 algorithm
 - sha512: Redact the data using the SHA512 algorithm
- initials: keep only the first char of each word in the data

configure Spark internal options

How the RDD are cached. Default is MEMORY_AND_DISK_SER.
Available options are (https://spark.apache.org/docs/latest/api/java/index.html?org/apache/spark/storage/StorageLevel.html):
- MEMORY_ONLY
- MEMORY_AND_DISK
- MEMORY_ONLY_SER
- MEMORY_AND_DISK_SER
- DISK_ONLY
- OFF_HEAP

May be parquet or ORC. Default is parquet. Used for BigQuery intermediate storage. Use ORC for for JSON files to keep the original data structure.
https://stackoverflow.com/questions/53674838/spark-writing-parquet-arraystring-converts-to-a-different-datatype-when-loadin

The GCS bucket that temporarily holds the data before it is loaded to BigQuery.

Internal use. Do not modify.

TODO

Should access policies be enforced ?

GCP project location. Required if apply is true.

GCP Project id. Required if apply is true.

Taxonomy name. Required if apply is true.

Max number of Spark jobs to run in parallel, default is 1

Pool name to use for Spark jobs, default is 'default'

This can be FIFO or FAIR, to control whether jobs within the pool queue up behind each other (the default) or share the pool’s resources fairly.

Scheduler filename in the metadata folder. If not set, defaults to fairscheduler.xml.

When using filesystem storage, the path to the expectations file

should expectations be executed ?

should load / transform fail on expectation error ?

When using filesystem storage, the path to the metrics file

Max number of unique values accepted for a discrete column. Default is 10

Should metrics be computed ?

JDBC: Connection String

FS or BQ: List of attributes to use for clustering

BQ: Number of days before this table is set as expired and deleted. Never by default.

BQ: Should be require a partition filter on every request ? No by default.

Should we materialize as a table or as a view when saving the results ? TABLE by default.

BQ: Enable automatic refresh of materialized view ? false by default.

BQ: Refresh interval in milliseconds. Default to BigQuery default value

ES: Attribute to use as id of the document. Generated by Elasticsearch if not specified.

FS: File format

FS: File extension

columns to use for sharding. table will be named table_{sharding(0)}_{sharding(1)}

FS or BQ: List of partition attributes

When outputting files, should we coalesce it to a single file. Useful when CSV is the output format.

Optional path attribute if you want to save the file outside of the default location (datasets folder)

spark  options to use

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

Table types supported by the Extract module

Any valid table types

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

UTF-8 if not specified.

are json objects on a single line or multiple line ? Single by default.  false means single. false also means faster

Is the json stored as a single object array ? false by default. This means that by default we have on json document per line.

does the dataset has a header ? true by default

the values delimiter,  ';' by default value may be a multichar string starting from Spark3

The String quote char, '"' by default

escaping char '\' by default

Pattern to ignore or UDF to apply to ignore some lines

Folder on the local filesystem where incoming files are stored.
 Typically, this folder will be scanned periodically to move the dataset to the cluster for ingestion.
                     Files located in this folder are moved to the stage folder for ingestion by the "import" command.

recognized filename extensions. json, csv, dsv, psv are recognized by default.
Only files with these extensions will be moved to the stage folder.

Ack extension used for each file. ".ack" if not specified.
Files are moved to the stage folder only once a file with the same name as the source file and with this extension is present.
To move a file without requiring an ack file to be present, set explicitly this property to the empty string value "".

Options to add to the spark reader

Loader to use, 'spark' or 'native'. Default to 'spark' of SL_LOADER env variable is set to 'native'

Treat empty columns as null in DSV files. Default to false

Cron expression to use for this domain/table

Configure freshness checks on this dataset

Treat a specific input string as a null value indicator

TODO

Files are read from  this folder for ingestion by the "import" command.

Files recognized by the extensions property are moved to this folder for ingestion by the "import" command.

Files that cannot be ingested (do not match by any table pattern) are moved to this folder.

Files that have been ingested are moved to this folder if SL_ARCHIVE is set to true.

Files that are being ingested are moved to this folder.

Invalid records are stored in this folder in source format when SL_SINK_REPLAY_TO_FILE is set to true.

How old may be the data before a warning is raised. Use syntax like '3 day' or '2 hour' or '30 minute'

How old may be the data before an error is raised. Use syntax like '3 day' or '2 hour' or '30 minute'

Schema name, must be unique among all the schemas belonging to the same domain.
  *                     Will become the hive table name On Premise or BigQuery Table name on GCP.

filename pattern to which this schema must be applied.
  *                     This instructs the framework to use this schema to parse any file with a filename that match this pattern.

Attributes parsing rules.

Dataset metadata

free text

attach streams to table (Snowflake only)

Reserved for future use.

Reserved for future use.

Set of string to attach to this Schema

Row level security on this schema.

Expectations to check after Load / Transform has succeeded

List of columns that make up the primary key

Map of rolename -> List[Users].

If present, the table is renamed with this name. Useful when use in conjunction with the 'extract' module

Store here a couple of records illustrating the table data.

remove all records that do not match this condition

Sample of filename matching this schema

TODO

TODO

TODO

TODO

TODO

Attribute name as defined in the source dataset and as received in the file

semantic type of the attribute. Default to string

Is it an array ?

Should this attribute always be present in the source. Default to true.

Should this attribute be applied a privacy transformation at ingestion time

free text for attribute description

If present, the attribute is renamed with this name

Sample data for this attribute

If present, what kind of stat should be computed for this field

List of sub-attributes (valid for JSON and XML files only)

Default value for this attribute when it is not present.

Tags associated with this attribute

Scripted field : SQL request on renamed column

If this attribute is a foreign key, reference to [domain.]table[.attribute]

Should this attribute be ignored on ingestion. Default to false

Policy tag to assign to this attribute. Used for column level security

Main SQL request to execute (do not forget to prefix table names with the database name to avoid conflicts)

attach streams to task (Snowflake only)

Output Database (refer to a project id in BigQuery). Default to SL_DATABASE env var if set.

Output domain in output Area (Will be the Database name in Hive or Dataset in BigQuery)

Dataset Name in output Area (Will be the Table name in Hive & BigQuery)

List of columns used for partitioning the output.

List of SQL requests to executed before the main SQL request is run

List of SQL requests to executed after the main SQL request is run

Expectations to check after Load / Transform has succeeded

Map of rolename -> List[Users].

Output table description

Configure freshness checks on the output table

Attributes comments and access policies

Python script URI to execute instead of the SQL request

Set of string to attach to the output table

TODO

Cron expression to use for this task

Cron expression to use for this domain/table

Number of milliseconds before a communication timeout.

Should we parse this SQL make it update the table according to write strategy or just execute it ?

Used when the default connection ref present in the application.sl.yml file is not the one to use to run the SQL request for this task.

Name of the lock

TODO

TODO. Default 5 seconds

TODO. Default 5 seconds

Output table description

TODO

Column name

Column Type

Column description

Access policy to apply to this column

A schema in JDBC database or a folder in HDFS or a dataset in BigQuery.

Domain name. Make sure you use a name that may be used as a folder name on the target storage.
                   - When using HDFS or Cloud Storage,  files once ingested are stored in a sub-directory named after the domain name.
                   - When used with BigQuery, files are ingested and sorted in tables under a dataset named after the domain name.

Domain Description (free text)

Set of string to attach to this domain

If present, the attribute is renamed with this name

Output Database (refer to a project id in BigQuery). Default to SL_DATABASE env var if set.

Optional name. If not specified, the name of the file without the extension is used.

List of transform tasks to execute

Optional description.

Default task properties to apply to all tasks defined in tasks section and in included files

Table name. Set to '*' to extract all tables. Scope: Schema and Data extraction.

SQL used to extract schema and data. Ignore columns attribute if set.

List of columns to extract. All columns by default.

Column name to extract. Scope: Schema and Data extraction.

Rename database column name. Scope: Schema and Data extraction.

Filter applied along data extraction. Scope: Data extraction

SQL template used on partition column's of type String. Some implementations are already defined, see ai.starlake.extract.JdbcDbUtils.getStringPartitionFunc. Mandatory variables: col, nb_partitions. Scope: Data extraction.

Specifies encoding (charset) of saved CSV files.

If true, writes the names of columns as the first line.

Character used as a separator for each field and value. TODO: check if we are expecting character.

Character used for escaping quoted values where the separator can be part of the value. TODO: check if we are expecting character.

Character used for escaping quotes inside an already quoted value. TODO: check if we are expecting character.

String representation of a null value.

Java date pattern to apply on date object. Have a look at https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

Java timestamp pattern to apply on timestamp object. Have a look at https://docs.oracle.com/javase/8/docs/api/java/time/format/DateTimeFormatter.html

Optional catalog name in the source database. Scope: Schema and Data extraction.

Database schema where source tables are located. For mysql use this instead of catalog. Scope: Schema and Data extraction.

Query template used to retrieve table remark. Available variables: catalog, schema, table. Scope: Schema extraction.

Query template used to retrieve all columns' remark of a table. Available variables: catalog, schema, table. Scope: Schema extraction.

One or many of the predefined table types. Scope: Schema and Data extraction.

Template used during schema extraction in order to generate load files (domain and tables). Scope: Schema extraction.

Pattern template used to define load tables' file pattern. Available variables: catalog, schema, table. Scope: Schema extraction.

Trim strategies applied to numeric fields set on load table's definition. Scope: Schema extraction.

Column to use in order to parallelize data extraction. Scope: Data extraction.

Number of data partitions to create. Scope: Data extraction.

Options to set on data connection. Scope: Data extraction.

Number of rows to be fetched from the database when additional rows are needed. By default, most JDBC drivers use a fetch size of 10, so if you are reading 1000 objects, increasing the fetch size to 256 can significantly reduce the time required to fetch the query's results. The optimal fetch size is not always obvious. Scope: Data extraction.

Define if we should fetch the entire table's or not. If not, maximum value of partitionColumn seen during last extraction is used in order to fetch incremental data. Scope: Data extraction.

Sanitize domain's name by keeping alpha numeric characters only. Scope: Schema and Data extraction.

List of tables to extract. Scope: Schema and Data extraction.

List of tables to exclude. Applied on tables list. Scope: Schema and Data extraction.

Describe what to fetch from data connection. Scope: Schema and Data extraction.

Connection used to read/store audit from it. If not defined, fallbacks to connectionRef. Expected connection name as defined in the connections section of the application.conf file. Scope: Data extraction.

Override the output format of data extraction. Scope: Data extraction.

Configuration merged into each jdbcSchemas. Scope: Schema and Data extraction.

List of regex used to include open api schemas (#/components/schemas). Defaults to ['.*']. 'Includes' is evaluated before 'excludes'

List of regex used to exclude open api schemas (#/components/schemas). Defaults to [].

Explode route's object to more object definition. Use object's path with route path as final name. Defaults to ALL

Keep properties of type object or array.

Keep properties of type object. Don't dive on array type.

Keep properties of type array. If encounters an object, dive deeper.

filter out on field path. Each field is separated by _. Default to []

Regex applied on object path. If matches, use the given name otherwise fallback to route_path + object path as final name

List of regex used to include open api path '.*'

Force all routes matching the pattern to be saved as the given name if they don't conflict

List of operations to retrieve schema from. Defaults to ['GET']. Supported values are GET and POST.

Retain operation of type GET in openAPI

Retain operation of type POST in openAPI

List of regex used to excludes api path []

List of regex used to excludes fields. Fields and their subfields are separated by _.

Explodes on route's object and split the schema.

Domain name used to group tables extracted from openAPI spec

Common base path used to remove from path in order to generate final table name.

Describe what to fetch from data connection. Scope: Schema and Data extraction.

mapping a format used for string and the starlake attribute type

Describe how to extract domains and tables from OpenAPI spec

attribute name is sanitized and stored as field name. Default.

attribute name is sanitized and stored as rename property when attribute's name differs from sanitized name

With OpenAPI: connection used to reference OpenAPI spec. Supports file (local, storage) or public http url. With others, connection used to retrieve schema and then data. If not defined, fallback to application settings.

Input for ref object

Database pattern to match, none if any database

Domain pattern to match, none if any domain match

Table pattern to match

Output for ref object

Describe how to resolve a reference in a transform task

The input table to resolve

The output table resolved with the domain and database

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

Map of topic configs

TODO

TODO

TODO

TODO

TODO

TODO

Default environment to use. May be also set using the SL_ENV environment variable

When using filesystem storage, default path to store the datasets

Incoming folder to use during autoload

DAG generation config folder. metadata/dags by default

Path to tests folder. Default is ${metadata}/tests

Location where are located user defined write strategies; Default is ${metadata}/write-strategies

default metadata folder name. May be also set using the SL_METADATA environment variable

Validate the YAML file when loading it. If set to true fails on any error

Should ingested files be archived after ingestion ?

Should invalid records be stored in a replay file ?

Default write format in Spark. parquet is the default

Default write format in Spark for rejected records. parquet is the default

Default write format in Spark for audit records. parquet is the default

output files in CSV format ? Default is false

CSV file extension when csvOutput is true. Default is .csv

Only generate privacy tasks. Reserved for internal use

Should empty strings be considered as null values ?

Default loader to use when none is specified in the schema. Valid values are 'spark' or 'native'. Default is 'spark'

In what order should the files for a same table be loaded ? By time (default) or by or name ?

Order pending files by name

Order pending files by creation date time

Should we load of the files to be stored in the same table in a single task or one by one ?

Maximum number of files to be stored in the same table in a single task

TODO

TODO

stage, ingesting ... areas configuration

Hadoop configuration if applicable

Connections configurations

JDBC engine configurations

Privacy algorithms

Root folder for the application. May be also set using the SL_ROOT environment variable

Internal configuration

Access policies configuration

Spark Job scheduling configuration

Coma separated list of UDF to register in Spark jobs. May be also set using the SL_UDFS environment variable

Expectations configuration

Pattern to use to replace parameters in SQL queries in addition to the jinja syntax {{param}}. Default is ${param}

Should we reject all records when an error occurs ? Default is false

Maximum number of records to reject when an error occurs. Default is 100

TODO

DSV ingestion extra options

reserved

reserved

reserved

reserved

reserved

reserved

reserved

Default target database (projectId in GCP). May be also set using the SL_DATABASE environment variable

reserved

Default connection to use when loading / transforming data

How many job to run simultaneously in dev mode (experimental)

TODO

Force application to stop even when there is some pending thread.

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

TODO

is duckdb mode active

Comma separated list of duckdb extensions to load. Default is spatial, json, httpfs

Where to store duckdb files if not using default

null string value in tests

Internal use only

Map of string

Map of string

Allow DuckDB to load / Save data from / to external sources. Default to true

Defines OpenAPI schemas extraction

Provide a JSON object that satisfies the schema "Starlake Data Pipeline". Here are the required top‑level fields: version. Return only the valid JSON object.