[System]
You are a helpful assistant that answers strictly in JSON. Here's the JSON schema you must adhere to:
<schema>
{"$schema":"https://json-schema.org/draft/2019-09/schema","title":"Catalog","description":"Each catalog source defines a portion of a Flow Catalog, by defining collections, derivations, tests, and materializations of the Catalog. Catalog sources may reference and import other sources, in order to collections and other entities that source defines.","type":"object","properties":{"$schema":{"title":"JSON-Schema against which the Catalog is validated.","type":"string"},"captures":{"title":"Captures of this Catalog.","examples":[{"acmeCo/capture":{"autoDiscover":{"addNewBindings":true,"evolveIncompatibleCollections":true},"bindings":[{"resource":{"stream":"a_stream"},"target":"target/collection"}],"endpoint":{"connector":{"config":"connector-config.yaml","image":"connector/image:tag"}}}}],"type":"object","patternProperties":{"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$":{"$ref":"#/definitions/CaptureDef"}},"additionalProperties":false},"collections":{"title":"Collections of this Catalog.","examples":[{"acmeCo/collection":{"key":["/json/ptr"],"schema":{"properties":{"bar":{"const":42},"foo":{"type":"integer"}},"type":"object"}}}],"type":"object","patternProperties":{"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$":{"$ref":"#/definitions/CollectionDef"}},"additionalProperties":false},"import":{"title":"Import other Flow catalog sources.","description":"By importing another Flow catalog source, its collections, schemas, and derivations are bundled into the publication context of this specification. Imports are relative or absolute URLs, relative to this specification's location.","type":"array","items":{"$ref":"#/definitions/RelativeUrl"}},"materializations":{"title":"Materializations of this Catalog.","examples":[{"acmeCo/materialization":{"bindings":[{"fields":{"recommended":true},"resource":{"table":"a_table"},"source":"source/collection"}],"endpoint":{"connector":{"config":"connector-config.yaml","image":"connector/image:tag"}}}}],"type":"object","patternProperties":{"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$":{"$ref":"#/definitions/MaterializationDef"}},"additionalProperties":false},"tests":{"title":"Tests of this Catalog.","examples":[{"acmeCo/conversions/test":[{"ingest":{"collection":"acmeCo/collection","description":"Description of the ingestion.","documents":[{"a":"document"},{"another":"document"}]}},{"verify":{"collection":"acmeCo/collection","description":"Description of the verification.","documents":[{"a":"document"},{"another":"document"}]}}]}],"type":"object","patternProperties":{"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$":{"$ref":"#/definitions/TestDef"}},"additionalProperties":false}},"additionalProperties":false,"definitions":{"AutoDiscover":{"description":"Settings to determine how Flow should stay abreast of ongoing changes to collections and schemas.","type":"object","properties":{"addNewBindings":{"description":"Automatically add new bindings discovered from the source.","default":false,"type":"boolean"},"evolveIncompatibleCollections":{"description":"Whether to automatically evolve collections and/or materialization bindings to handle changes to collections that would otherwise be incompatible with the existing catalog.","default":false,"type":"boolean"}},"additionalProperties":false},"Capture":{"description":"Capture names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.","examples":["acmeCo/capture"],"type":"string","pattern":"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$"},"CaptureBinding":{"examples":[{"resource":{"stream":"a_stream"},"target":"target/collection"}],"type":"object","required":["resource","target"],"properties":{"backfill":{"title":"Backfill counter for this binding.","description":"Every increment of this counter will result in a new backfill of this binding from the captured endpoint. For example when capturing from a SQL table, incrementing this counter will cause the table to be re-captured in its entirety from the source database.\n\nNote that a backfill does *not* truncate the target collection, and documents published by a backfilled binding will coexist with (and be ordered after) any documents which were published as part of a preceding backfill.","type":"integer","format":"uint32","minimum":0.0},"disable":{"title":"Whether to disable the binding","description":"Disabled bindings are inactive, and not validated. They can be used to represent discovered resources that are intentionally not being captured.","type":"boolean"},"resource":{"title":"Endpoint resource to capture from."},"target":{"title":"Name of the collection to capture into.","$ref":"#/definitions/Collection"}},"additionalProperties":false},"CaptureDef":{"description":"A Capture binds an external system and target (e.x., a SQL table or cloud storage bucket) from which data should be continuously captured, with a Flow collection into that captured data is ingested. Multiple Captures may be bound to a single collection, but only one capture may exist for a given endpoint and target.","type":"object","required":["bindings","endpoint"],"properties":{"autoDiscover":{"title":"Continuously keep the collection spec and schema up-to-date","$ref":"#/definitions/AutoDiscover"},"bindings":{"title":"Bound collections to capture from the endpoint.","type":"array","items":{"$ref":"#/definitions/CaptureBinding"}},"delete":{"title":"Delete this capture within the control plane.","description":"When true, a publication will delete this capture.","type":"boolean"},"endpoint":{"title":"Endpoint to capture from.","$ref":"#/definitions/CaptureEndpoint"},"expectPubId":{"title":"Expected publication ID of this capture within the control plane.","description":"When present, a publication of the capture will fail if the last publication ID in the control plane doesn't match this value.","$ref":"#/definitions/Id"},"interval":{"title":"Interval of time between invocations of the capture.","description":"Configured intervals are applicable only to connectors which are unable to continuously tail their source, and which instead produce a current quantity of output and then exit. Flow will start the connector again after the given interval of time has passed.\n\nIntervals are relative to the start of an invocation and not its completion. For example, if the interval is five minutes, and an invocation of the capture finishes after two minutes, then the next invocation will be started after three additional minutes.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"},"shards":{"title":"Template for shards of this capture task.","$ref":"#/definitions/ShardTemplate"}},"additionalProperties":false},"CaptureEndpoint":{"description":"An endpoint from which Flow will capture.","oneOf":[{"title":"A Connector.","type":"object","required":["connector"],"properties":{"connector":{"$ref":"#/definitions/ConnectorConfig"}},"additionalProperties":false},{"title":"A local command (development only).","type":"object","required":["local"],"properties":{"local":{"$ref":"#/definitions/LocalConfig"}},"additionalProperties":false}]},"Collection":{"description":"Collection names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.","examples":["acmeCo/collection"],"type":"string","pattern":"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$"},"CollectionDef":{"description":"Collection describes a set of related documents, where each adheres to a common schema and grouping key. Collections are append-only: once a document is added to a collection, it is never removed. However, it may be replaced or updated (either in whole, or in part) by a future document sharing its key. Each new document of a given key is \"reduced\" into existing documents of the key. By default, this reduction is achieved by completely replacing the previous document, but much richer reduction behaviors can be specified through the use of annotated reduction strategies of the collection schema.","examples":[{"key":["/json/ptr"],"schema":{"properties":{"bar":{"const":42},"foo":{"type":"integer"}},"type":"object"}}],"type":"object","oneOf":[{"required":["schema"],"properties":{"readSchema":false,"writeSchema":false}},{"required":["readSchema","writeSchema"],"properties":{"schema":false}}],"required":["key"],"properties":{"delete":{"title":"Delete this collection within the control plane.","description":"When true, a publication will delete this collection.","type":"boolean"},"derive":{"$ref":"#/definitions/Derivation"},"expectPubId":{"title":"Expected publication ID of this collection within the control plane.","description":"When present, a publication of the collection will fail if the last publication ID in the control plane doesn't match this value.","$ref":"#/definitions/Id"},"journals":{"title":"Template for journals of this collection.","$ref":"#/definitions/JournalTemplate"},"key":{"title":"Composite key of this collection.","$ref":"#/definitions/CompositeKey"},"projections":{"title":"Projections and logical partitions of this collection.","examples":[{"a_field":"/json/ptr","a_partition":{"location":"/json/ptr","partition":true}}],"type":"object","patternProperties":{"^([^/~]|(~[01]))+(/([^/~]|(~[01]))+)*$":{"$ref":"#/definitions/Projection"}},"additionalProperties":false},"readSchema":{"title":"Schema against which collection documents are validated and reduced on read.","$ref":"#/definitions/Schema"},"reset":{"title":"Reset this collection within the control plane.","description":"When true, a publication will reset this collection. Resetting a collection is equivalent to deleting and then re-creating the collection, but is applied as a single publication.","type":"boolean"},"schema":{"title":"Schema against which collection documents are validated and reduced on write and read.","examples":["../path/to/schema#/$defs/subPath"],"$ref":"#/definitions/Schema"},"writeSchema":{"title":"Schema against which collection documents are validated and reduced on write.","$ref":"#/definitions/Schema"}},"additionalProperties":false},"CompositeKey":{"description":"Ordered JSON-Pointers which define how a composite key may be extracted from a collection document.","examples":[["/json/ptr"]],"type":"array","items":{"$ref":"#/definitions/JsonPointer"}},"CompressionCodec":{"description":"A CompressionCodec may be applied to compress journal fragments before they're persisted to cloud stoage. The compression applied to a journal fragment is included in its filename, such as \".gz\" for GZIP. A collection's compression may be changed at any time, and will affect newly-written journal fragments.","examples":["GZIP_OFFLOAD_DECOMPRESSION"],"type":"string","enum":["NONE","GZIP","ZSTANDARD","SNAPPY","GZIP_OFFLOAD_DECOMPRESSION"]},"ConnectorConfig":{"description":"Connector image and configuration specification.","type":"object","required":["config","image"],"properties":{"config":{"title":"Configuration of the connector."},"image":{"title":"Image of the connector.","type":"string"}}},"DekafConfig":{"description":"Dekaf service configuration","type":"object","required":["config","variant"],"properties":{"config":{"title":"Dekaf endpoint config."},"variant":{"title":"Dekaf variant type.","description":"Since we support integrating with a bunch of different providers via Dekaf, this allows us to store which of those connector variants this particular Dekaf connector was created as, in order to e.g link to the correct docs URL, show the correct name and logo, etc.","type":"string"}}},"Derivation":{"description":"Derive specifies how a collection is derived from other collections.","type":"object","required":["transforms","using"],"properties":{"shards":{"title":"Template for shards of this derivation task.","$ref":"#/definitions/ShardTemplate"},"shuffleKeyTypes":{"title":"Key component types of the shuffle keys used by derivation lambdas.","description":"Typically you omit this and Flow infers it from your transform shuffle keys. In some circumstances, Flow may require that you explicitly tell it of your shuffled key types.","type":"array","items":{"$ref":"#/definitions/ShuffleType"}},"transforms":{"title":"Transforms which make up this derivation.","type":"array","items":{"$ref":"#/definitions/TransformDef"}},"using":{"title":"The selected runtime for this derivation.","$ref":"#/definitions/DeriveUsing"}},"additionalProperties":false},"DeriveUsing":{"description":"A derivation runtime implementation.","oneOf":[{"title":"A Connector.","type":"object","required":["connector"],"properties":{"connector":{"$ref":"#/definitions/ConnectorConfig"}},"additionalProperties":false},{"title":"A SQLite derivation.","type":"object","required":["sqlite"],"properties":{"sqlite":{"$ref":"#/definitions/DeriveUsingSqlite"}},"additionalProperties":false},{"title":"A TypeScript derivation.","type":"object","required":["typescript"],"properties":{"typescript":{"$ref":"#/definitions/DeriveUsingTypescript"}},"additionalProperties":false},{"title":"A local command (development only).","type":"object","required":["local"],"properties":{"local":{"$ref":"#/definitions/LocalConfig"}},"additionalProperties":false}]},"DeriveUsingSqlite":{"type":"object","properties":{"migrations":{"title":"Ordered migrations which are used to initialize the database.","description":"Migrations may be provided as an inline string, or as a relative URL to a file containing the migration SQL.","type":"array","items":{"oneOf":[{"description":"A URL identifying a resource, which may be a relative local path with respect to the current resource (i.e, ../path/to/flow.yaml), or may be an external absolute URL (i.e., http://example/flow.yaml).","examples":["https://example/resource"],"type":"string","pattern":"^[^ ]+$"},{"type":"string","contentMediaType":"text/x.sql"}]}}},"additionalProperties":false},"DeriveUsingTypescript":{"type":"object","required":["module"],"properties":{"module":{"title":"TypeScript module implementing this derivation.","description":"Module is either a relative URL of a TypeScript module file, or is an inline representation of a Typescript module. The module must have a exported Derivation variable which is an instance implementing the corresponding Derivation interface.","oneOf":[{"description":"A URL identifying a resource, which may be a relative local path with respect to the current resource (i.e, ../path/to/flow.yaml), or may be an external absolute URL (i.e., http://example/flow.yaml).","examples":["https://example/resource"],"type":"string","pattern":"^[^ ]+$"},{"type":"string","contentMediaType":"text/x.typescript"}]}},"additionalProperties":false},"Field":{"description":"Field names a projection of a document location. They may include '/', but cannot begin or end with one. Many Fields are automatically inferred by Flow from a collection JSON Schema, and are the JSON Pointer of the document location with the leading '/' removed. User-provided Fields which act as a logical partitions are restricted to Unicode letters, numbers, '-', '_', or '.'","examples":["my_field"],"type":"string","pattern":"^([^/~]|(~[01]))+(/([^/~]|(~[01]))+)*$"},"FragmentTemplate":{"description":"A FragmentTemplate configures how journal fragment files are produced as part of a collection.","examples":[{"compressionCodec":"ZSTANDARD","flushInterval":"1h"}],"type":"object","properties":{"compressionCodec":{"title":"Codec used to compress Journal Fragments.","$ref":"#/definitions/CompressionCodec"},"flushInterval":{"title":"Maximum flush delay before in-progress fragments are closed and persisted","description":"into cloud storage. Intervals are converted into uniform time segments: 24h will \"roll\" all fragments at midnight UTC every day, 1h at the top of every hour, 15m a :00, :15, :30, :45 past the hour, and so on. If not set, then fragments are not flushed on time-based intervals.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"},"length":{"title":"Desired content length of each fragment, in megabytes before compression.","description":"When a collection journal fragment reaches this threshold, it will be closed off and pushed to cloud storage. If not set, a default of 512MB is used.","type":"integer","format":"uint32","maximum":4096.0,"minimum":32.0},"retention":{"title":"Duration for which historical fragments of a collection should be kept.","description":"If not set, then fragments are retained indefinitely.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"}},"additionalProperties":false},"FullSource":{"description":"A source collection and details of how it's read.","examples":[{"name":"source/collection"}],"type":"object","required":["name"],"properties":{"name":{"title":"Name of the collection to be read.","$ref":"#/definitions/Collection"},"notAfter":{"title":"Upper bound date-time for documents which should be processed.","description":"Source collection documents published after this date-time are filtered. `notAfter` is *only* a filter. Updating its value will not cause Flow to re-process documents that have already been read. Optional. Default is to process all documents.","type":["string","null"],"format":"date-time"},"notBefore":{"title":"Lower bound date-time for documents which should be processed.","description":"Source collection documents published before this date-time are filtered. `notBefore` is *only* a filter. Updating its value will not cause Flow to re-process documents that have already been read. Optional. Default is to process all documents.","type":["string","null"],"format":"date-time"},"partitions":{"title":"Selector over partition of the source collection to read.","examples":[{"exclude":{"other_partition":[32,64]},"include":{"a_partition":["A","B"]}}],"$ref":"#/definitions/PartitionSelector"}},"additionalProperties":false},"Id":{"type":"string"},"JournalTemplate":{"description":"A JournalTemplate configures the journals which make up the physical partitions of a collection.","examples":[{"fragments":{"compressionCodec":"ZSTANDARD","flushInterval":"1h"}}],"type":"object","required":["fragments"],"properties":{"fragments":{"title":"Fragment configuration of collection journals.","$ref":"#/definitions/FragmentTemplate"}},"additionalProperties":false},"JsonPointer":{"description":"JSON Pointer which identifies a location in a document.","examples":["/json/ptr"],"type":"string","pattern":"^(/([^/~]|(~[01]))+)*$"},"LocalConfig":{"description":"Local command and its configuration.","type":"object","required":["command","config"],"properties":{"command":{"title":"Command to execute","type":"array","items":{"type":"string"}},"config":{"title":"Configuration of the command."},"env":{"title":"Environment variables","type":"object","additionalProperties":{"type":"string"}},"protobuf":{"title":"Use protobuf codec instead of JSON.","type":"boolean"}}},"Materialization":{"description":"Materialization names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.","examples":["acmeCo/materialization"],"type":"string","pattern":"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$"},"MaterializationBinding":{"examples":[{"fields":{"recommended":true},"resource":{"table":"a_table"},"source":"source/collection"}],"type":"object","required":["resource","source"],"properties":{"backfill":{"title":"Backfill counter for this binding.","description":"Every increment of this counter will result in a new backfill of this binding from its source collection to its materialized resource. For example when materializing to a SQL table, incrementing this counter causes the table to be dropped and then rebuilt by re-reading the source collection.","type":"integer","format":"uint32","minimum":0.0},"disable":{"title":"Whether to disable the binding","description":"Disabled bindings are inactive, and not validated.","type":"boolean"},"fields":{"title":"Selected projections for this materialization.","default":{"recommended":true},"$ref":"#/definitions/MaterializationFields"},"onIncompatibleSchemaChange":{"title":"Action to take when a schema change is rejected due to incompatibility.","description":"This setting is used to determine the action to take when a schema change is rejected due to incompatibility with the target resource. By default, the binding will have its `backfill` counter incremented, causing it to be re-materialized from the source collection.","$ref":"#/definitions/OnIncompatibleSchemaChange"},"priority":{"title":"Priority applied to documents processed by this binding.","description":"When all bindings are of equal priority, Flow processes documents according to their associated publishing time, as encoded in the document UUID.\n\nHowever, when one binding has a higher priority than others, then *all* ready documents are processed through the binding before *any* documents of other bindings are processed.","type":"integer","format":"uint32","minimum":0.0},"resource":{"title":"Endpoint resource to materialize into."},"source":{"title":"The collection to be materialized.","$ref":"#/definitions/Source"}},"additionalProperties":false},"MaterializationDef":{"description":"A Materialization binds a Flow collection with an external system & target (e.x, a SQL table) into which the collection is to be continuously materialized.","type":"object","required":["bindings","endpoint"],"properties":{"bindings":{"title":"Bound collections to materialize into the endpoint.","type":"array","items":{"$ref":"#/definitions/MaterializationBinding"}},"delete":{"title":"Delete this materialization within the control plane.","description":"When true, a publication will delete this materialization.","type":"boolean"},"endpoint":{"title":"Endpoint to materialize into.","$ref":"#/definitions/MaterializationEndpoint"},"expectPubId":{"title":"Expected publication ID of this materialization within the control plane.","description":"When present, a publication of the materialization will fail if the last publication ID in the control plane doesn't match this value.","$ref":"#/definitions/Id"},"onIncompatibleSchemaChange":{"title":"Default handling of schema changes that are incompatible with the target resource.","description":"This can be overridden on a per-binding basis.","$ref":"#/definitions/OnIncompatibleSchemaChange"},"shards":{"title":"Template for shards of this materialization task.","$ref":"#/definitions/ShardTemplate"},"source":{"title":"Automatically materialize new bindings from a named capture","$ref":"#/definitions/SourceType"}},"additionalProperties":false},"MaterializationEndpoint":{"description":"An Endpoint connector used for Flow materializations.","oneOf":[{"title":"A Connector.","type":"object","required":["connector"],"properties":{"connector":{"$ref":"#/definitions/ConnectorConfig"}},"additionalProperties":false},{"title":"A local command (development only).","type":"object","required":["local"],"properties":{"local":{"$ref":"#/definitions/LocalConfig"}},"additionalProperties":false},{"title":"A Dekaf connection","type":"object","required":["dekaf"],"properties":{"dekaf":{"$ref":"#/definitions/DekafConfig"}},"additionalProperties":false}]},"MaterializationFields":{"description":"MaterializationFields defines a selection of projections to materialize, as well as optional per-projection, driver-specific configuration.","examples":[{"exclude":["removed"],"recommended":true,"require":{"added":{}}}],"type":"object","required":["recommended"],"properties":{"exclude":{"title":"Fields to exclude.","description":"This removes from recommended projections, where enabled.","type":"array","items":{"$ref":"#/definitions/Field"}},"groupBy":{"title":"Fields to use as the grouping key of this materialization.","description":"If not specified, the key of the source collection is used. Materialization bindings may select an ordered subset of scalar fields which will be grouped over, resulting in a natural index over the chosen group-by key. Fields may or may not be part of the collection key.","type":"array","items":{"$ref":"#/definitions/Field"}},"recommended":{"title":"Desired depth of fields which should be automatically selected.","description":"0 = No fields, 1 = Top-level fields only, 2 = Second level fields, and so on.","$ref":"#/definitions/RecommendedDepth"},"require":{"title":"Fields to require.","description":"This supplements any selected fields, where enabled. Values are passed to and interpreted by the connector, which may use it to customize DDL generation or other behaviors with respect to the field. Consult connector documentation to see what it supports.\n\nNote that this field has been renamed from `include`, which will continue to be accepted as an alias.","type":"object","additionalProperties":true}},"additionalProperties":false},"OnIncompatibleSchemaChange":{"description":"Determines how to handle incompatible schema changes for a given binding.","examples":["backfill"],"oneOf":[{"description":"Fail the publication of the incompatible schema change. This prevents any schema change from being applied if it is incompatible with the existing schema, as determined by the connector.","type":"string","enum":["abort"]},{"description":"Increment the backfill counter of the binding, causing it to start over from the beginning.","type":"string","enum":["backfill"]},{"description":"Disable the binding, which will be effectively excluded from the task until it is re-enabled.","type":"string","enum":["disableBinding"]},{"description":"Disable the entire task, preventing it from running until it is re-enabled.","type":"string","enum":["disableTask"]}]},"PartitionSelector":{"description":"Partition selectors identify a desired subset of the available logical partitions of a collection.","examples":[{"exclude":{"other_partition":[32,64]},"include":{"a_partition":["A","B"]}}],"type":"object","properties":{"exclude":{"description":"Partition field names and values which are excluded from the source collection. Any documents matching *any one* of the partition values will be excluded.","default":{},"type":"object","additionalProperties":{"type":"array","items":true}},"include":{"description":"Partition field names and corresponding values which must be matched from the Source collection. Only documents having one of the specified values across all specified partition names will be matched. For example, source: [App, Web] region: [APAC] would mean only documents of 'App' or 'Web' source and also occurring in the 'APAC' region will be processed.","default":{},"type":"object","additionalProperties":{"type":"array","items":true}}},"additionalProperties":false},"Projection":{"description":"Projections are named locations within a collection document which may be used for logical partitioning or directly exposed to databases into which collections are materialized.","anyOf":[{"$ref":"#/definitions/JsonPointer"},{"type":"object","required":["location"],"properties":{"location":{"title":"Location of this projection.","$ref":"#/definitions/JsonPointer"},"partition":{"title":"Is this projection a logical partition?","default":false,"type":"boolean"}},"additionalProperties":false}]},"RecommendedDepth":{"anyOf":[{"type":"boolean"},{"type":"integer","format":"uint","minimum":0.0}]},"RelativeUrl":{"description":"A URL identifying a resource, which may be a relative local path with respect to the current resource (i.e, ../path/to/flow.yaml), or may be an external absolute URL (i.e., http://example/flow.yaml).","examples":["https://example/resource"],"type":"string","pattern":"^[^ ]+$"},"Schema":{"description":"A schema is a draft 2020-12 JSON Schema which validates Flow documents. Schemas also provide annotations at document locations, such as reduction strategies for combining one document into another.\n\nSchemas may be defined inline to the catalog, or given as a relative or absolute URI. URIs may optionally include a JSON fragment pointer that locates a specific sub-schema therein.\n\nFor example, \"schemas/marketing.yaml#/$defs/campaign\" would reference the schema at location {\"$defs\": {\"campaign\": ...}} within ./schemas/marketing.yaml.","examples":["http://example/schema#/$defs/subPath","../path/to/schema#/$defs/subPath",{"properties":{"bar":{"const":42},"foo":{"type":"integer"}},"type":"object"}]},"ShardTemplate":{"description":"A ShardTemplate configures how shards process a catalog task.","examples":[{"hotStandbys":1,"maxTxnDuration":"30s"}],"type":"object","properties":{"disable":{"title":"Disable processing of the task's shards.","type":"boolean"},"hotStandbys":{"title":"Number of hot standbys to keep for each task shard.","description":"Hot standbys of a shard actively replicate the shard's state to another machine, and are able to be quickly promoted to take over processing for the shard should its current primary fail. If not set, then no hot standbys are maintained. EXPERIMENTAL: this field MAY be removed.","type":"integer","format":"uint32","minimum":0.0},"logLevel":{"title":"Log level of this tasks's shards.","description":"Log levels may currently be \"error\", \"warn\", \"info\", \"debug\", or \"trace\". If not set, the effective log level is \"info\".","type":"string"},"maxTxnDuration":{"title":"Maximum duration of task transactions.","description":"This duration upper-bounds the amount of time during which a transaction may process documents before it must flush and commit. It may run for less time if there aren't additional ready documents for it to process. If not set, the maximum duration defaults to five minutes for materializations, and one second for captures and derivations. EXPERIMENTAL: this field MAY be removed.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"},"minTxnDuration":{"title":"Minimum duration of task transactions.","description":"This duration lower-bounds the amount of time during which a transaction must process documents before it must flush and commit. It may run for more time if additional documents are available. The default value is zero seconds. Larger values may result in more data reduction, at the cost of more latency. EXPERIMENTAL: this field MAY be removed.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"},"readChannelSize":{"title":"Size of the reader channel used for decoded documents.","description":"Larger values are recommended for tasks having more than one shard split and long, bursty transaction durations. If not set, a reasonable default (currently 4,096) is used. EXPERIMENTAL: this field is LIKELY to be removed.","type":"integer","format":"uint32","minimum":0.0},"ringBufferSize":{"title":"Size of the ring buffer used to sequence documents for exactly-once semantics.","description":"The ring buffer is a performance optimization only: catalog tasks will replay portions of journals as needed when messages aren't available in the buffer. It can remain small if upstream task transactions are small, but larger transactions will achieve better performance with a larger ring. If not set, a reasonable default (currently 65,536) is used. EXPERIMENTAL: this field is LIKELY to be removed.","type":"integer","format":"uint32","minimum":0.0}},"additionalProperties":false},"Shuffle":{"description":"A Shuffle specifies how a shuffling key is to be extracted from collection documents.","examples":[{"key":["/json/ptr"]}],"oneOf":[{"title":"A Document may be shuffled to any task shard.","description":"Use 'any' if your transformation does not rely on internal task state, or if your derivation is not intended to scale beyond a single shard.","type":"string","enum":["any"]},{"title":"Key which identifies fields of sourced documents to extract and shuffle upon.","type":"object","required":["key"],"properties":{"key":{"$ref":"#/definitions/CompositeKey"}},"additionalProperties":false},{"title":"Lambda which extracts a shuffle key from the sourced documents of this transform.","description":"Lambdas may be provided inline, or as a relative URL to a file containing the lambda.","type":"object","required":["lambda"],"properties":{"lambda":true},"additionalProperties":false}]},"ShuffleType":{"description":"Type of a shuffled key component.","type":"string","enum":["boolean","integer","string"]},"Source":{"description":"A source collection and details of how it's read.","examples":["source/collection"],"anyOf":[{"$ref":"#/definitions/FullSource"},{"$ref":"#/definitions/Collection"}]},"SourceDef":{"description":"Specifies configuration for source captures, and defaults for new bindings that are added to the materialization. Changing these defaults has no effect on existing bindings.","type":"object","properties":{"capture":{"description":"Capture name","$ref":"#/definitions/Capture"},"deltaUpdates":{"description":"When adding new bindings from a source capture to a materialization, should the new bindings be marked as delta updates","type":"boolean"},"targetNaming":{"description":"When adding new bindings from a source capture to a materialization, how should the schema of the materialization binding be set","$ref":"#/definitions/TargetNaming"}},"additionalProperties":false},"SourceType":{"anyOf":[{"$ref":"#/definitions/Capture"},{"$ref":"#/definitions/SourceDef"}]},"TargetNaming":{"description":"How to name target resources (database tables, for example) for materializing a given Collection.","oneOf":[{"description":"Leave the materialization binding's schema field empty, therefore falling back to the default schema of the materialization. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `myTable`, without specifying the schema.\n\nThis used to be called `leaveEmpty`, and that value is still accepted, but specs will always be written with `noSchema` instead.","type":"string","enum":["noSchema"]},{"description":"Use the 2nd-to-last component of the collection name as the schema of the materialization binding. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `myTable` in the schema `mySchema`.\n\nThis used to be called `fromSourceName`, and that value is still accepted, but specs will always be written with `withSchema` instead.","type":"string","enum":["withSchema"]},{"description":"Use the 2nd-to-last component of the collection name to prefix the destination resource name, leaving the schema unspecified. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `mySchema_myTable`.","type":"string","enum":["prefixSchema"]},{"description":"Like `prefixSchema`, except that it will omit the prefix for the following common default schema names: - public - dbo","type":"string","enum":["prefixNonDefaultSchema"]}]},"Test":{"description":"Test names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.","examples":["acmeCo/conversions/test"],"type":"string","pattern":"^[\\p{Letter}\\p{Number}\\-_\\.]+(/[\\p{Letter}\\p{Number}\\-_\\.]+)*$"},"TestDef":{"description":"Test the behavior of reductions and derivations, through a sequence of test steps.","examples":[{"description":"An example test","steps":[{"ingest":{"collection":"acmeCo/collection","description":"Description of the ingestion.","documents":[{"a":"document"},{"another":"document"}]}},{"verify":{"collection":"acmeCo/collection","description":"Description of the verification.","documents":[{"a":"document"},{"another":"document"}]}}]}],"type":"object","required":["steps"],"properties":{"delete":{"title":"Delete this test within the control plane.","description":"When true, a publication will delete this test.","type":"boolean"},"description":{"title":"Description of this test.","type":"string"},"expectPubId":{"title":"Expected publication ID of this test within the control plane.","description":"When present, a publication of the test will fail if the last publication ID in the control plane doesn't match this value.","$ref":"#/definitions/Id"},"steps":{"title":"Sequential steps of this test.","type":"array","items":{"$ref":"#/definitions/TestStep"}}},"additionalProperties":false},"TestDocuments":{"description":"A test step describes either an \"ingest\" of document fixtures into a collection, or a \"verify\" of expected document fixtures from a collection.","examples":["../path/to/test-documents.json",[{"a":"document"},{"another":"document"}]]},"TestStep":{"description":"A test step describes either an \"ingest\" of document fixtures into a collection, or a \"verify\" of expected document fixtures from a collection.","examples":[{"ingest":{"collection":"acmeCo/collection","description":"Description of the ingestion.","documents":[{"a":"document"},{"another":"document"}]}},{"verify":{"collection":"acmeCo/collection","description":"Description of the verification.","documents":[{"a":"document"},{"another":"document"}]}}],"oneOf":[{"description":"Ingest document fixtures into a collection.","type":"object","required":["ingest"],"properties":{"ingest":{"$ref":"#/definitions/TestStepIngest"}},"additionalProperties":false},{"description":"Verify the contents of a collection match a set of document fixtures.","type":"object","required":["verify"],"properties":{"verify":{"$ref":"#/definitions/TestStepVerify"}},"additionalProperties":false}]},"TestStepIngest":{"description":"An ingestion test step ingests document fixtures into the named collection.","examples":[{"collection":"acmeCo/collection","description":"Description of the ingestion.","documents":[{"a":"document"},{"another":"document"}]}],"type":"object","required":["collection","documents"],"properties":{"collection":{"title":"Name of the collection into which the test will ingest.","$ref":"#/definitions/Collection"},"description":{"title":"Description of this test ingestion.","type":"string"},"documents":{"title":"Documents to ingest.","description":"Each document must conform to the collection's schema.","$ref":"#/definitions/TestDocuments"}},"additionalProperties":false},"TestStepVerify":{"description":"A verification test step verifies that the contents of the named collection match the expected fixtures, after fully processing all preceding ingestion test steps.","examples":[{"collection":"acmeCo/collection","description":"Description of the verification.","documents":[{"a":"document"},{"another":"document"}]}],"type":"object","required":["collection","documents"],"properties":{"collection":{"title":"Collection from which the test will verify.","$ref":"#/definitions/Source"},"description":{"title":"Description of this test verification.","type":"string"},"documents":{"title":"Documents to verify.","description":"Each document may contain only a portion of the matched document's properties, and any properties present in the actual document but not in this document fixture are ignored. All other values must match or the test will fail.","$ref":"#/definitions/TestDocuments"}},"additionalProperties":false},"Transform":{"description":"Transform names are Unicode letters, numbers, '-', '_', or '.'.","examples":["myTransform"],"type":"string","pattern":"^[\\p{Letter}\\p{Number}\\-_\\.]+$"},"TransformDef":{"description":"A Transform reads and shuffles documents of a source collection, and processes each document through either one or both of a register \"update\" lambda and a derived document \"publish\" lambda.","examples":[{"name":"my-transform","shuffle":"any","source":"some/source/collection"}],"type":"object","required":["name","shuffle","source"],"properties":{"backfill":{"title":"Backfill counter for this transform.","description":"Every increment of this counter will result in a new backfill of this transform. Specifically, the transform's lambda will be re-invoked for every applicable document of its source collection.\n\nNote that a backfill does *not* truncate the derived collection, and documents published by a backfilled transform will coexist with (and be ordered after) any documents which were published as part of a preceding backfill.","type":"integer","format":"uint32","minimum":0.0},"disable":{"title":"Whether to disable this transform.","description":"Disabled transforms are completely ignored at runtime and are not validated.","type":"boolean"},"lambda":{"title":"Lambda applied to the sourced documents of this transform.","description":"Lambdas may be provided inline, or as a relative URL to a file containing the lambda."},"name":{"title":"Name of this transformation.","description":"The names of transforms within a derivation must be unique and stable.","$ref":"#/definitions/Transform"},"priority":{"title":"Priority applied to documents processed by this transform.","description":"When all transforms are of equal priority, Flow processes documents according to their associated publishing time, as encoded in the document UUID.\n\nHowever, when one transform has a higher priority than others, then *all* ready documents are processed through the transform before *any* documents of other transforms are processed.","type":"integer","format":"uint32","minimum":0.0},"readDelay":{"title":"Delay applied to documents processed by this transform.","description":"Delays are applied as an adjustment to the UUID clock encoded within each document, which is then used to impose a relative ordering of all documents read by this derivation. This means that read delays are applied in a consistent way, even when back-filling over historical documents. When caught up and tailing the source collection, delays also \"gate\" documents such that they aren't processed until the current wall-time reflects the delay.","type":["string","null"],"pattern":"^\\d+(s|m|h|d)$"},"shuffle":{"title":"Shuffle by which source documents are mapped to processing shards.","$ref":"#/definitions/Shuffle"},"source":{"title":"Source collection read by this transform.","$ref":"#/definitions/Source"}},"additionalProperties":false}}}
</schema>

[User] Provide a JSON object that satisfies the schema "Catalog". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.

Catalog

Each catalog source defines a portion of a Flow Catalog, by defining collections, derivations, tests, and materializations of the Catalog. Catalog sources may reference and import other sources, in order to collections and other entities that source defines.

JSON-Schema against which the Catalog is validated.

Captures of this Catalog.

Collections of this Catalog.

Import other Flow catalog sources.

By importing another Flow catalog source, its collections, schemas, and derivations are bundled into the publication context of this specification. Imports are relative or absolute URLs, relative to this specification's location.

Materializations of this Catalog.

Tests of this Catalog.

Description of the ingestion.

Description of the verification.

Settings to determine how Flow should stay abreast of ongoing changes to collections and schemas.

Automatically add new bindings discovered from the source.

Whether to automatically evolve collections and/or materialization bindings to handle changes to collections that would otherwise be incompatible with the existing catalog.

Capture names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.

Backfill counter for this binding.

Every increment of this counter will result in a new backfill of this binding from the captured endpoint. For example when capturing from a SQL table, incrementing this counter will cause the table to be re-captured in its entirety from the source database.

Note that a backfill does *not* truncate the target collection, and documents published by a backfilled binding will coexist with (and be ordered after) any documents which were published as part of a preceding backfill.

Whether to disable the binding

Disabled bindings are inactive, and not validated. They can be used to represent discovered resources that are intentionally not being captured.

Endpoint resource to capture from.

Name of the collection to capture into.

A Capture binds an external system and target (e.x., a SQL table or cloud storage bucket) from which data should be continuously captured, with a Flow collection into that captured data is ingested. Multiple Captures may be bound to a single collection, but only one capture may exist for a given endpoint and target.

Continuously keep the collection spec and schema up-to-date

Bound collections to capture from the endpoint.

Delete this capture within the control plane.

When true, a publication will delete this capture.

Endpoint to capture from.

Expected publication ID of this capture within the control plane.

When present, a publication of the capture will fail if the last publication ID in the control plane doesn't match this value.

Interval of time between invocations of the capture.

Configured intervals are applicable only to connectors which are unable to continuously tail their source, and which instead produce a current quantity of output and then exit. Flow will start the connector again after the given interval of time has passed.

Intervals are relative to the start of an invocation and not its completion. For example, if the interval is five minutes, and an invocation of the capture finishes after two minutes, then the next invocation will be started after three additional minutes.

Template for shards of this capture task.

An endpoint from which Flow will capture.

A Connector.

A local command (development only).

Collection names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.

Collection describes a set of related documents, where each adheres to a common schema and grouping key. Collections are append-only: once a document is added to a collection, it is never removed. However, it may be replaced or updated (either in whole, or in part) by a future document sharing its key. Each new document of a given key is "reduced" into existing documents of the key. By default, this reduction is achieved by completely replacing the previous document, but much richer reduction behaviors can be specified through the use of annotated reduction strategies of the collection schema.

Delete this collection within the control plane.

When true, a publication will delete this collection.

Expected publication ID of this collection within the control plane.

When present, a publication of the collection will fail if the last publication ID in the control plane doesn't match this value.

Template for journals of this collection.

Composite key of this collection.

Projections and logical partitions of this collection.

Schema against which collection documents are validated and reduced on read.

Reset this collection within the control plane.

When true, a publication will reset this collection. Resetting a collection is equivalent to deleting and then re-creating the collection, but is applied as a single publication.

Schema against which collection documents are validated and reduced on write and read.

Schema against which collection documents are validated and reduced on write.

Ordered JSON-Pointers which define how a composite key may be extracted from a collection document.

A CompressionCodec may be applied to compress journal fragments before they're persisted to cloud stoage. The compression applied to a journal fragment is included in its filename, such as ".gz" for GZIP. A collection's compression may be changed at any time, and will affect newly-written journal fragments.

NONE GZIP ZSTANDARD SNAPPY GZIP_OFFLOAD_DECOMPRESSION

Connector image and configuration specification.

Configuration of the connector.

Image of the connector.

Dekaf service configuration

Dekaf endpoint config.

Dekaf variant type.

Since we support integrating with a bunch of different providers via Dekaf, this allows us to store which of those connector variants this particular Dekaf connector was created as, in order to e.g link to the correct docs URL, show the correct name and logo, etc.

Derive specifies how a collection is derived from other collections.

Template for shards of this derivation task.

Key component types of the shuffle keys used by derivation lambdas.

Typically you omit this and Flow infers it from your transform shuffle keys. In some circumstances, Flow may require that you explicitly tell it of your shuffled key types.

Transforms which make up this derivation.

The selected runtime for this derivation.

A derivation runtime implementation.

A Connector.

A SQLite derivation.

A TypeScript derivation.

A local command (development only).

Ordered migrations which are used to initialize the database.

Migrations may be provided as an inline string, or as a relative URL to a file containing the migration SQL.

A URL identifying a resource, which may be a relative local path with respect to the current resource (i.e, ../path/to/flow.yaml), or may be an external absolute URL (i.e., http://example/flow.yaml).

TypeScript module implementing this derivation.

Module is either a relative URL of a TypeScript module file, or is an inline representation of a Typescript module. The module must have a exported Derivation variable which is an instance implementing the corresponding Derivation interface.

Field names a projection of a document location. They may include '/', but cannot begin or end with one. Many Fields are automatically inferred by Flow from a collection JSON Schema, and are the JSON Pointer of the document location with the leading '/' removed. User-provided Fields which act as a logical partitions are restricted to Unicode letters, numbers, '-', '_', or '.'

A FragmentTemplate configures how journal fragment files are produced as part of a collection.

Codec used to compress Journal Fragments.

Maximum flush delay before in-progress fragments are closed and persisted

into cloud storage. Intervals are converted into uniform time segments: 24h will "roll" all fragments at midnight UTC every day, 1h at the top of every hour, 15m a :00, :15, :30, :45 past the hour, and so on. If not set, then fragments are not flushed on time-based intervals.

Desired content length of each fragment, in megabytes before compression.

When a collection journal fragment reaches this threshold, it will be closed off and pushed to cloud storage. If not set, a default of 512MB is used.

Duration for which historical fragments of a collection should be kept.

If not set, then fragments are retained indefinitely.

A source collection and details of how it's read.

Name of the collection to be read.

Upper bound date-time for documents which should be processed.

Source collection documents published after this date-time are filtered. `notAfter` is *only* a filter. Updating its value will not cause Flow to re-process documents that have already been read. Optional. Default is to process all documents.

Lower bound date-time for documents which should be processed.

Source collection documents published before this date-time are filtered. `notBefore` is *only* a filter. Updating its value will not cause Flow to re-process documents that have already been read. Optional. Default is to process all documents.

Selector over partition of the source collection to read.

A JournalTemplate configures the journals which make up the physical partitions of a collection.

Fragment configuration of collection journals.

JSON Pointer which identifies a location in a document.

Local command and its configuration.

Command to execute

Configuration of the command.

Environment variables

Use protobuf codec instead of JSON.

Materialization names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.

Backfill counter for this binding.

Every increment of this counter will result in a new backfill of this binding from its source collection to its materialized resource. For example when materializing to a SQL table, incrementing this counter causes the table to be dropped and then rebuilt by re-reading the source collection.

Whether to disable the binding

Disabled bindings are inactive, and not validated.

Selected projections for this materialization.

Action to take when a schema change is rejected due to incompatibility.

This setting is used to determine the action to take when a schema change is rejected due to incompatibility with the target resource. By default, the binding will have its `backfill` counter incremented, causing it to be re-materialized from the source collection.

Priority applied to documents processed by this binding.

When all bindings are of equal priority, Flow processes documents according to their associated publishing time, as encoded in the document UUID.

However, when one binding has a higher priority than others, then *all* ready documents are processed through the binding before *any* documents of other bindings are processed.

Endpoint resource to materialize into.

The collection to be materialized.

A Materialization binds a Flow collection with an external system & target (e.x, a SQL table) into which the collection is to be continuously materialized.

Bound collections to materialize into the endpoint.

Delete this materialization within the control plane.

When true, a publication will delete this materialization.

Endpoint to materialize into.

Expected publication ID of this materialization within the control plane.

When present, a publication of the materialization will fail if the last publication ID in the control plane doesn't match this value.

Default handling of schema changes that are incompatible with the target resource.

This can be overridden on a per-binding basis.

Template for shards of this materialization task.

Automatically materialize new bindings from a named capture

An Endpoint connector used for Flow materializations.

A Connector.

A local command (development only).

A Dekaf connection

MaterializationFields defines a selection of projections to materialize, as well as optional per-projection, driver-specific configuration.

Fields to exclude.

This removes from recommended projections, where enabled.

Fields to use as the grouping key of this materialization.

If not specified, the key of the source collection is used. Materialization bindings may select an ordered subset of scalar fields which will be grouped over, resulting in a natural index over the chosen group-by key. Fields may or may not be part of the collection key.

Desired depth of fields which should be automatically selected.

0 = No fields, 1 = Top-level fields only, 2 = Second level fields, and so on.

Fields to require.

This supplements any selected fields, where enabled. Values are passed to and interpreted by the connector, which may use it to customize DDL generation or other behaviors with respect to the field. Consult connector documentation to see what it supports.

Note that this field has been renamed from `include`, which will continue to be accepted as an alias.

Determines how to handle incompatible schema changes for a given binding.

Fail the publication of the incompatible schema change. This prevents any schema change from being applied if it is incompatible with the existing schema, as determined by the connector.

Increment the backfill counter of the binding, causing it to start over from the beginning.

Disable the binding, which will be effectively excluded from the task until it is re-enabled.

Disable the entire task, preventing it from running until it is re-enabled.

Partition selectors identify a desired subset of the available logical partitions of a collection.

Partition field names and values which are excluded from the source collection. Any documents matching *any one* of the partition values will be excluded.

Partition field names and corresponding values which must be matched from the Source collection. Only documents having one of the specified values across all specified partition names will be matched. For example, source: [App, Web] region: [APAC] would mean only documents of 'App' or 'Web' source and also occurring in the 'APAC' region will be processed.

Projections are named locations within a collection document which may be used for logical partitioning or directly exposed to databases into which collections are materialized.

Location of this projection.

Is this projection a logical partition?

A schema is a draft 2020-12 JSON Schema which validates Flow documents. Schemas also provide annotations at document locations, such as reduction strategies for combining one document into another.

Schemas may be defined inline to the catalog, or given as a relative or absolute URI. URIs may optionally include a JSON fragment pointer that locates a specific sub-schema therein.

For example, "schemas/marketing.yaml#/$defs/campaign" would reference the schema at location {"$defs": {"campaign": ...}} within ./schemas/marketing.yaml.

A ShardTemplate configures how shards process a catalog task.

Disable processing of the task's shards.

Number of hot standbys to keep for each task shard.

Hot standbys of a shard actively replicate the shard's state to another machine, and are able to be quickly promoted to take over processing for the shard should its current primary fail. If not set, then no hot standbys are maintained. EXPERIMENTAL: this field MAY be removed.

Log level of this tasks's shards.

Log levels may currently be "error", "warn", "info", "debug", or "trace". If not set, the effective log level is "info".

Maximum duration of task transactions.

This duration upper-bounds the amount of time during which a transaction may process documents before it must flush and commit. It may run for less time if there aren't additional ready documents for it to process. If not set, the maximum duration defaults to five minutes for materializations, and one second for captures and derivations. EXPERIMENTAL: this field MAY be removed.

Minimum duration of task transactions.

This duration lower-bounds the amount of time during which a transaction must process documents before it must flush and commit. It may run for more time if additional documents are available. The default value is zero seconds. Larger values may result in more data reduction, at the cost of more latency. EXPERIMENTAL: this field MAY be removed.

Size of the reader channel used for decoded documents.

Larger values are recommended for tasks having more than one shard split and long, bursty transaction durations. If not set, a reasonable default (currently 4,096) is used. EXPERIMENTAL: this field is LIKELY to be removed.

Size of the ring buffer used to sequence documents for exactly-once semantics.

The ring buffer is a performance optimization only: catalog tasks will replay portions of journals as needed when messages aren't available in the buffer. It can remain small if upstream task transactions are small, but larger transactions will achieve better performance with a larger ring. If not set, a reasonable default (currently 65,536) is used. EXPERIMENTAL: this field is LIKELY to be removed.

A Shuffle specifies how a shuffling key is to be extracted from collection documents.

A Document may be shuffled to any task shard.

Use 'any' if your transformation does not rely on internal task state, or if your derivation is not intended to scale beyond a single shard.

Key which identifies fields of sourced documents to extract and shuffle upon.

Lambda which extracts a shuffle key from the sourced documents of this transform.

Lambdas may be provided inline, or as a relative URL to a file containing the lambda.

Type of a shuffled key component.

A source collection and details of how it's read.

Specifies configuration for source captures, and defaults for new bindings that are added to the materialization. Changing these defaults has no effect on existing bindings.

Capture name

When adding new bindings from a source capture to a materialization, should the new bindings be marked as delta updates

When adding new bindings from a source capture to a materialization, how should the schema of the materialization binding be set

How to name target resources (database tables, for example) for materializing a given Collection.

Leave the materialization binding's schema field empty, therefore falling back to the default schema of the materialization. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `myTable`, without specifying the schema.

This used to be called `leaveEmpty`, and that value is still accepted, but specs will always be written with `noSchema` instead.

Use the 2nd-to-last component of the collection name as the schema of the materialization binding. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `myTable` in the schema `mySchema`.

This used to be called `fromSourceName`, and that value is still accepted, but specs will always be written with `withSchema` instead.

Use the 2nd-to-last component of the collection name to prefix the destination resource name, leaving the schema unspecified. For example, materialize the collection `acmeCo/mySchema/myTable` to a table called `mySchema_myTable`.

Like `prefixSchema`, except that it will omit the prefix for the following common default schema names: - public - dbo

Test names are paths of Unicode letters, numbers, '-', '_', or '.'. Each path component is separated by a slash '/', and a name may not begin or end in a '/'.

Test the behavior of reductions and derivations, through a sequence of test steps.

An example test

Description of the ingestion.

Description of the verification.

Delete this test within the control plane.

When true, a publication will delete this test.

Description of this test.

Expected publication ID of this test within the control plane.

When present, a publication of the test will fail if the last publication ID in the control plane doesn't match this value.

Sequential steps of this test.

A test step describes either an "ingest" of document fixtures into a collection, or a "verify" of expected document fixtures from a collection.

Description of the ingestion.

Description of the verification.

Ingest document fixtures into a collection.

Verify the contents of a collection match a set of document fixtures.

An ingestion test step ingests document fixtures into the named collection.

Description of the ingestion.

Name of the collection into which the test will ingest.

Description of this test ingestion.

Documents to ingest.

Each document must conform to the collection's schema.

A verification test step verifies that the contents of the named collection match the expected fixtures, after fully processing all preceding ingestion test steps.

Description of the verification.

Collection from which the test will verify.

Description of this test verification.

Documents to verify.

Each document may contain only a portion of the matched document's properties, and any properties present in the actual document but not in this document fixture are ignored. All other values must match or the test will fail.

Transform names are Unicode letters, numbers, '-', '_', or '.'.

A Transform reads and shuffles documents of a source collection, and processes each document through either one or both of a register "update" lambda and a derived document "publish" lambda.

Backfill counter for this transform.

Every increment of this counter will result in a new backfill of this transform. Specifically, the transform's lambda will be re-invoked for every applicable document of its source collection.

Note that a backfill does *not* truncate the derived collection, and documents published by a backfilled transform will coexist with (and be ordered after) any documents which were published as part of a preceding backfill.

Whether to disable this transform.

Disabled transforms are completely ignored at runtime and are not validated.

Lambda applied to the sourced documents of this transform.

Lambdas may be provided inline, or as a relative URL to a file containing the lambda.

Name of this transformation.

The names of transforms within a derivation must be unique and stable.

Priority applied to documents processed by this transform.

When all transforms are of equal priority, Flow processes documents according to their associated publishing time, as encoded in the document UUID.

However, when one transform has a higher priority than others, then *all* ready documents are processed through the transform before *any* documents of other transforms are processed.

Delay applied to documents processed by this transform.

Delays are applied as an adjustment to the UUID clock encoded within each document, which is then used to impose a relative ordering of all documents read by this derivation. This means that read delays are applied in a consistent way, even when back-filling over historical documents. When caught up and tailing the source collection, delays also "gate" documents such that they aren't processed until the current wall-time reflects the delay.

Shuffle by which source documents are mapped to processing shards.

Source collection read by this transform.

Provide a JSON object that satisfies the schema "Catalog". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.


[System]
You are a helpful assistant that answers strictly in JSON. Here's the JSON schema you must adhere to:
<schema>
{"$schema":"http://json-schema.org/draft-07/schema#","$id":"https://json.schemastore.org/fly.json","definitions":{"statics":{"type":"object","required":["guest_path","url_prefix"],"additionalProperties":false,"properties":{"guest_path":{"description":"The path inside your container where the assets to serve are located.","type":"string"},"url_prefix":{"description":"The URL prefix that should serve the static assets.","type":"string"}}},"services":{"description":"Configure the mapping of ports from the public Fly proxy to your application.\n\nYou can have:\n* **No services section**: The application has no mappings to the external internet - typically apps like databases or background job workers that talk over 6PN private networking to other apps.\n* **One services section**: One internal port mapped to one external port on the internet.\n* **Multiple services sections**: Map multiple internal ports to multiple external ports.","type":"object","properties":{"script_checks":{"deprecated":true,"description":"Health checks that run as one-off commands directly on the VM.\n\nThis type of check is **deprecated**. See `tcp_checks` or `http_checks` for alternatives."},"protocol":{"description":"The protocol used to communicate with your application. Can be: `tcp` or `udp`.","type":"string","enum":["tcp","udp"]},"internal_port":{"description":"The port this application listens on to communicate with clients. The default is 8080. We recommend applications use the default.","type":"integer","default":8080},"auto_stop_machines":{"description":"The action, if any, that Fly Proxy should take when the app is idle for several minutes. Options are \"off\", \"stop\", or \"suspend\".","type":"string","enum":["off","stop","suspend"],"default":"off"},"auto_start_machines":{"description":"Whether to automatically start an application\u2019s Machines when a new request is made to the application and there\u2019s no excess capacity, per region. If there\u2019s only one Machine in a region, then it\u2019s started whenever a request is made to the application. The default if not set is true.","type":"boolean","default":true},"min_machines_running":{"description":"The number of Machines to keep running, in the primary region only, when auto_stop_machines = true.","type":"integer","default":0},"concurrency":{"type":"object","description":"Control autoscaling metrics (connections or requests) and limits (hard and soft).","properties":{"type":{"type":"string","default":"connections","x-taplo":{"docs":{"enumValues":["Autoscale based on number of concurrent connections","Autoscale based on number of concurrent requests"]}},"enum":["connections","requests"]},"hard_limit":{"default":25,"type":"integer","description":"When an application instance is __at__ or __over__ this number, the system will bring up another instance."},"soft_limit":{"default":20,"type":"integer","description":"When an application instance is __at__ or __over__ this number, the system is likely to bring up another instance."}}},"ports":{"description":"For each external port you want to accept connections on, add a `ports` section.","type":"array","items":{"type":"object","properties":{"handlers":{"x-taplo":{"links":{"key":"https://fly.io/docs/reference/services/#connection-handlers"}},"description":"An array of strings that select handlers to terminate the connection at the edge.\n\nValid options: http, tls, proxy_proto, pg_tls, edge_http.","type":"array","items":{"type":"string","minLength":1,"enum":["http","tls","proxy_proto","pg_tls","edge_http"]}},"port":{"default":8080,"type":"integer","description":"The port to accept traffic on. Valid ports: 1-65535"},"start_port":{"type":"integer"},"end_port":{"type":"integer"},"force_https":{"type":"boolean","description":"Force HTTP connections to HTTPS. `force_https` requires the `http` handler in the `handlers` section."},"http_options":{"type":"object","properties":{"response":{"type":"object","properties":{"pristine":{"type":"boolean"},"headers":{"type":"object","additionalProperties":{"oneOf":[{"type":"boolean"},{"type":"string"},{"type":"array","items":{"type":"string"}}]}}}},"h2_backend":{"type":"boolean"}}},"proxy_proto_options":{"type":"object","properties":{"version":{"type":"string","default":"v1"}}},"tls_options":{"type":"object","properties":{"alpn":{"type":"array","items":{"type":"string"}},"versions":{"type":"array","items":{"type":"string"}},"default_self_signed":{"type":"boolean","default":false}}}}}},"machine_checks":{"type":"array","items":{"type":"object","properties":{"command":{"type":"array","items":{"type":"string"}},"entrypoint":{"type":"array","items":{"type":"string"}},"image":{"type":"string"},"kill_signal":{"type":"string"},"kill_timeout":{"type":"string","pattern":"^\\d+s$"}},"required":["command","entrypoint","image","kill_signal","kill_timeout"]}},"tcp_checks":{"description":"Basic TCP connection health checks. This is the default check that runs against the configured `internal_port`.","type":"array","x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#services-tcp_checks"}},"items":{"type":"object","properties":{"grace_period":{"description":"The time to wait after a VM starts before checking its health. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]},"interval":{"description":"Length of the pause between connectivity checks. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]},"restart_limit":{"default":0,"description":"The number of consecutive TCP check failures to allow before attempting to restart the VM. The default is `0`, which disables restarts based on failed TCP health checks.","type":"integer"},"timeout":{"description":"The maximum time a connection can take before being reported as failing its healthcheck. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]}}}},"http_checks":{"$ref":"#/definitions/http_checks"}}},"mounts":{"type":"object","x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#the-mounts-section"}},"description":"Mount [persistent storage volumes](https://fly.io/docs/reference/volumes) previously setup via `flyctl`. Both settings are required. Example:\n\n```toml\n[mounts]\n  source = \"myapp_data\"\n  destination = \"/data\"\n```","required":["source","destination"],"additionalProperties":false,"properties":{"source":{"description":"The name of the volume to mount as shown in `fly volumes list`.\n\nA volume of this name *must exist* in each of the app regions. If there's more than one volume in the target region with the same one, one will be picked randomly.","type":"string"},"snapshot_retention":{"description":"The number of snapshots to retain for the volume. Snapshots are taken daily.","type":"integer"},"destination":{"description":"The path at which the `source` volume should be mounted in the running app VM.","type":"string"},"processes":{"description":"The name of the process(es) to which this mount should be applied. See [multiple processes](https://community.fly.io/t/preview-multi-process-apps-get-your-workers-here/2316).","type":"array","items":{"type":"string","minLength":1}},"initial_size":{"anyOf":[{"type":"string"},{"type":"integer"}],"description":"The size of the volume to be created on first deploy."},"auto_extend_size_threshold":{"type":"integer","description":"The usage percentage threshold that triggers volume extension."},"auto_extend_size_increment":{"type":"string","description":"The size increment by which to extend the volume."},"auto_extend_size_limit":{"type":"string","description":"The total amount to extend a volume."}}},"metrics":{"type":"object","properties":{"port":{"type":"integer","description":"Port on which the application exposes metrics data."},"path":{"type":"string","description":"Path under which the metrics are exposed, typically '/metrics'."},"processes":{"type":"array","items":{"type":"string"},"description":"List of process names that this metrics configuration applies to."}}},"http_checks":{"description":"HTTP-based health checks run against the `internal_port`. These checks will pass when receiving a 2xx response. Any other response is considered a failure.","type":"array","x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#services-http_checks"}},"items":{"type":"object","properties":{"grace_period":{"description":"The time to wait after a VM starts before checking its health. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]},"interval":{"description":"Length of the pause between connectivity checks. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]},"restart_limit":{"default":0,"description":"The number of consecutive check failures to allow before attempting to restart the VM. The default is `0`, which disables restarts based on failed health checks.","type":"integer"},"timeout":{"description":"The maximum time a connection can take before being reported as failing its healthcheck. Units are milliseconds unless you specify them like `10s` or `1m`.","oneOf":[{"type":"integer"},{"type":"string"}]},"method":{"description":"The HTTP method to be used for the check.","type":"string"},"path":{"description":"The path of the URL to be requested.","type":"string"},"protocol":{"description":"The protocol to be used (`http` or `https`)","type":"string","enum":["http","https"]},"tls_skip_verify":{"type":"boolean","default":false,"description":"When set to `true` (and `protocol` is set to `https`), skip verifying the certificates sent by the server."},"headers":{"type":"object","description":"Set key/value pairs of HTTP headers to pass along with the check request."}}}}},"description":"https://fly.io/docs/reference/configuration","properties":{"files":{"type":"array","items":{"type":"object","properties":{"guest_path":{"type":"string","description":"The path in the guest where the file will be written."},"raw_value":{"type":"string","description":"Base64 encoded content to be written to the file."},"secret_name":{"type":"string","description":"The name of the secret whose value will be written to the file."},"local_path":{"type":"string","description":"Local path of the file to be written to the guest."},"processes":{"type":"array","items":{"type":"string"},"description":"List of process names that this file configuration applies to."}},"required":["guest_path"]}},"http_service":{"type":"object","properties":{"internal_port":{"type":"integer","description":"The port this service (and application) will use to communicate with clients. Default is 8080.","default":8080},"force_https":{"type":"boolean","description":"A Boolean which determines whether to enforce HTTP to HTTPS redirects."},"auto_stop_machines":{"description":"Whether to automatically stop or suspend an application\u2019s Machines when there\u2019s excess capacity, per region. Options are \"off\", \"stop\", and \"suspend\". If there\u2019s only one Machine in a region, then the Machine is stopped or suspended if it has no traffic. The Fly Proxy runs a process to automatically stop Machines every few minutes. The default if not set is \"off\".","type":"string","enum":["off","stop","suspend"],"default":"off"},"auto_start_machines":{"type":"boolean","description":"Whether to automatically start an application\u2019s Machines when a new request is made to the application and there\u2019s no excess capacity, per region."},"min_machines_running":{"type":"integer","description":"The number of Machines to keep running, in the primary region only, when `auto_stop_machines` is true."},"concurrency":{"type":"object","properties":{"type":{"type":"string","description":"Specifies what metric is used to determine when to auto start or stop, or when a given Machine should receive more or less traffic (load balancing).","enum":["connections","requests"]},"soft_limit":{"type":"integer","description":"When a Fly Machine is at or over this number of concurrent connections or requests, the system will deprioritize sending new traffic to that Machine."},"hard_limit":{"type":"integer","description":"When a Fly Machine is at or over this number of concurrent connections or requests, the system will stop sending new traffic to that Machine."}}},"http_options":{"type":"object","properties":{"response":{"type":"object","properties":{"pristine":{"type":"boolean","description":"Configures Fly Proxy to not add any Fly headers to HTTP responses."},"headers":{"type":"object","additionalProperties":{"oneOf":[{"type":"boolean"},{"type":"string"},{"type":"array","items":{"type":"string"}}]},"description":"Add or remove HTTP response headers."}}},"h2_backend":{"type":"boolean","description":"Indicates whether the app supports HTTP/2 cleartext (H2C) with prior knowledge or not."}}},"tls_options":{"type":"object","properties":{"alpn":{"type":"array","items":{"type":"string"}},"versions":{"type":"array","items":{"type":"string"}},"default_self_signed":{"type":"boolean","description":"When `true`, serve a self-signed certificate if no certificate exists. Default is `false`.","default":false}}},"checks":{"$ref":"#/definitions/http_checks"}}},"primary_region":{"type":"string","description":"The primary region where the application will be deployed. This setting also sets the PRIMARY_REGION environment variable."},"console_command":{"type":"string","description":"Command to run when you execute 'fly console'. This opens your framework's console in a new, dedicated Machine."},"swap_size_mb":{"type":"integer","description":"Size in megabytes for the swap file on the VM. Helps in handling memory spikes."},"metrics":{"oneOf":[{"type":"array","items":{"$ref":"#/definitions/metrics"}},{"$ref":"#/definitions/metrics"}]},"http_checks":{"$ref":"#/definitions/http_checks"},"processes":{"type":"object","additionalProperties":{"type":"string"}},"app":{"description":"Fly.io application name","type":"string"},"host_dedication_id":{"type":"string"},"vm":{"type":"array","x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#the-vm-section"}},"items":{"type":"object","properties":{"size":{"type":"string","enum":["shared-cpu-1x","shared-cpu-2x","shared-cpu-4x","shared-cpu-8x","performance-1x","performance-2x","performance-4x","performance-8x","performance-16x"]},"memory":{"oneOf":[{"type":"string"},{"type":"integer"}]},"cpus":{"type":"integer","enum":[1,2,4,8,16]},"cpu_kind":{"type":"string","enum":["shared","performance"]},"gpus":{"type":"integer","enum":[1,2,4,8]},"gpu_kind":{"type":"string","enum":["a10","a100-40gb","a100-80gb","l40s"]},"kernel_args":{"oneOf":[{"type":"array","items":{"type":"string"}},{"type":"string"}]},"host_dedication_id":{"type":"string"},"processes":{"type":"array","items":{"type":"string"}}}}},"kill_timeout":{"description":"Seconds to wait before forcing a VM process to exit. Default is 5 seconds.","oneOf":[{"type":"integer"},{"type":"string"}]},"kill_signal":{"description":"Signal to send to a process to shut it down gracefully. Default is SIGINT.","type":"string","enum":["SIGINT","SIGTERM","SIGQUIT","SIGUSR1","SIGUSR2","SIGKILL","SIGSTOP"]},"statics":{"description":"The `statics` sections expose static assets built into your application's container to Fly's Anycast network. You can serve HTML files, Javascript, and images without needing to run a web server inside your container.","x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#the-statics-sections"}},"oneOf":[{"$ref":"#/definitions/statics"},{"type":"array","items":{"$ref":"#/definitions/statics"}}]},"services":{"oneOf":[{"$ref":"#/definitions/services"},{"type":"array","items":{"$ref":"#/definitions/services"}}]},"deploy":{"type":"object","additionalProperties":false,"properties":{"release_command":{"x-taplo":{"links":{"key":"https://fly.io/docs/reference/configuration/#release_command"},"initKeys":["importantKey"]},"description":"Command to run after a build, with access to the production environment, but before deployment. Non-zero exit status will abort the deployment.\n\n```toml\n[deploy]\n  release_command =\"bundle exec rails db:migrate\"\n```","type":"string"},"release_command_vm":{"description":"VM configuration to use when running the release command.","type":"object","properties":{"size":{"type":"string","enum":["shared-cpu-1x","shared-cpu-2x","shared-cpu-4x","shared-cpu-8x","performance-1x","performance-2x","performance-4x","performance-8x","performance-16x"]},"memory":{"oneOf":[{"type":"string"},{"type":"integer"}]},"cpus":{"type":"integer","enum":[1,2,4,8,16]},"cpu_kind":{"type":"string","enum":["shared","performance"]}}},"max_unavailable":{"description":"For rolling deploys, you can use max_unavailable to control how many Machines can be down at a time.","type":"number"},"strategy":{"description":"Strategy for replacing VMs during a deployment.","type":"string","default":"canary","enum":["canary","rolling","bluegreen","immediate"],"x-taplo":{"docs":{"main":"Strategy for replacing VMs during a deployment.","enumValues":["This default strategy - for apps without volumes - will boot a single, new VM with the new release, verify its health, then proceed with a rolling restart strategy.","One by one, each running VM is taken down and replaced by a new release VM. This is the default strategy for apps with volumes.","For every running VM, a new one is booted alongside it. All new VMs must pass health checks to complete deployment, when traffic gets migrated to new VMs. If your app has multiple VMs, this strategy may reduce deploy time and downtime, assuming your app is scaled to 2 or more VMs.","Replace all VMs with new releases immediately without waiting for health checks to pass. This is useful in emergency situations where you're confident a release will be healthy."],"defaultValue":"Default is 'canary': boot a single, new VM with the new release, verify its health, then proceed with a rolling restart strategy."},"links":{"key":"https://fly.io/docs/reference/configuration/#strategy"}}},"wait_timeout":{"description":"Timeout for waiting for the Machine to be in a started state during a deploy.","type":"string","default":"5m","x-taplo":{"docs":{"main":"Timeout for waiting for the Machine to be in a started state during a deploy.","defaultValue":"Default is 5 minutes"},"links":{"key":"https://fly.io/docs/reference/configuration/#dealing-with-timeout-errors-while-deploying-a-new-version"}}}}},"mounts":{"oneOf":[{"$ref":"#/definitions/mounts"},{"type":"array","items":{"$ref":"#/definitions/mounts"}}]},"experimental":{"description":"Flags and features that are subject to change, deprecation or promotion to the main configuration.","type":"object","properties":{"cmd":{"description":"Override the server command (CMD) set by the Dockerfile. Specify as an array of strings:\n\n```toml\ncmd = [\"path/to/command\", \"arg1\", \"arg2\"]\n```","anyOf":[{"type":"string"},{"type":"array","items":{"type":"string"}}]},"entrypoint":{"description":"Override the ENTRYPOINT set by the Dockerfile. Specify as an array of strings:\n\n```toml\nentrypoint = [\"path/to/command\", \"arg1\", \"arg2\"]\n```","anyOf":[{"type":"string"},{"type":"array","items":{"type":"string"}}]},"auto_rollback":{"description":"Failed deployments should roll back automatically to the previous successfully deployed release. Defaults to `true`","type":"boolean"},"private_network":{"description":"Enables private network access to the Fly organization. Defaults to `true`.","default":true,"type":"boolean"}}},"env":{"description":"Set non-sensitive information as environment variables in the application's [runtime environment](https://fly.io/docs/reference/runtime-environment/).\nFor sensitive information, such as credentials or passwords, use the [secrets command](https://fly.io/docs/reference/secrets). For anything else though, the `env` section provides a simple way to set environment variables. Here's an example:\n```toml\n[env]\n  LOG_LEVEL = \"debug\"\n  S3_BUCKET = \"my-bucket\"\n```","type":"object","additionalProperties":{"type":"string"}},"build":{"description":"Build configuration options. See docs at https://fly.io/docs/reference/builders.","type":"object","properties":{"builder":{"description":"Builder Docker image to be used with the 'buildpacks' option","type":"string"},"buildpacks":{"description":"Buildpacks to be run by the 'builder' Docker image","type":"array","items":{"type":"string","minLength":1},"uniqueItems":true,"minItems":1},"args":{"description":"Build arguments passed to both Dockerfile and Buildpack builds. These arguments are **not available** on VMs at runtime.\n```toml\n[build.args]\n  USER = \"julieta\"\n  MODE = \"production\"\n```","type":"object","additionalProperties":{"type":"string"}},"build-target":{"description":"Specify the target stage for [multistage Dockerfile builds](https://docs.docker.com/develop/develop-images/multistage-build/).","type":"string"},"image":{"description":"Docker image to be deployed (skips the build process)","type":"string"},"dockerfile":{"description":"Dockerfile used for builds. Defaults to './Dockerfile'","type":"string"},"ignorefile":{"description":"A relative path to a .dockerignore file. Defaults to './.dockerignore'","type":"string"}},"additionalProperties":false}},"additionalProperties":true,"title":"Fly.io config schema (fly.toml)","type":"object","x-taplo-info":{"authors":["Joshua Sierles (https://github.com/jsierles)"],"patterns":["\\.*fly(.*)?\\.toml?$"]}}
</schema>

[User] Provide a JSON object that satisfies the schema "Fly.io config schema (fly.toml)". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.

The path inside your container where the assets to serve are located.

The URL prefix that should serve the static assets.

Configure the mapping of ports from the public Fly proxy to your application.

You can have:
* **No services section**: The application has no mappings to the external internet - typically apps like databases or background job workers that talk over 6PN private networking to other apps.
* **One services section**: One internal port mapped to one external port on the internet.
* **Multiple services sections**: Map multiple internal ports to multiple external ports.

Health checks that run as one-off commands directly on the VM.

This type of check is **deprecated**. See `tcp_checks` or `http_checks` for alternatives.

The protocol used to communicate with your application. Can be: `tcp` or `udp`.

The port this application listens on to communicate with clients. The default is 8080. We recommend applications use the default.

The action, if any, that Fly Proxy should take when the app is idle for several minutes. Options are "off", "stop", or "suspend".

Whether to automatically start an application’s Machines when a new request is made to the application and there’s no excess capacity, per region. If there’s only one Machine in a region, then it’s started whenever a request is made to the application. The default if not set is true.

The number of Machines to keep running, in the primary region only, when auto_stop_machines = true.

Control autoscaling metrics (connections or requests) and limits (hard and soft).

Autoscale based on number of concurrent connections

Autoscale based on number of concurrent requests

When an application instance is __at__ or __over__ this number, the system will bring up another instance.

When an application instance is __at__ or __over__ this number, the system is likely to bring up another instance.

For each external port you want to accept connections on, add a `ports` section.

An array of strings that select handlers to terminate the connection at the edge.

Valid options: http, tls, proxy_proto, pg_tls, edge_http.

The port to accept traffic on. Valid ports: 1-65535

Force HTTP connections to HTTPS. `force_https` requires the `http` handler in the `handlers` section.

Basic TCP connection health checks. This is the default check that runs against the configured `internal_port`.

The time to wait after a VM starts before checking its health. Units are milliseconds unless you specify them like `10s` or `1m`.

Length of the pause between connectivity checks. Units are milliseconds unless you specify them like `10s` or `1m`.

The number of consecutive TCP check failures to allow before attempting to restart the VM. The default is `0`, which disables restarts based on failed TCP health checks.

The maximum time a connection can take before being reported as failing its healthcheck. Units are milliseconds unless you specify them like `10s` or `1m`.

Mount [persistent storage volumes](https://fly.io/docs/reference/volumes) previously setup via `flyctl`. Both settings are required. Example:

```toml
[mounts]
  source = "myapp_data"
  destination = "/data"
```

The name of the volume to mount as shown in `fly volumes list`.

A volume of this name *must exist* in each of the app regions. If there's more than one volume in the target region with the same one, one will be picked randomly.

The number of snapshots to retain for the volume. Snapshots are taken daily.

The path at which the `source` volume should be mounted in the running app VM.

The name of the process(es) to which this mount should be applied. See [multiple processes](https://community.fly.io/t/preview-multi-process-apps-get-your-workers-here/2316).

The size of the volume to be created on first deploy.

The usage percentage threshold that triggers volume extension.

The size increment by which to extend the volume.

The total amount to extend a volume.

Port on which the application exposes metrics data.

Path under which the metrics are exposed, typically '/metrics'.

List of process names that this metrics configuration applies to.

HTTP-based health checks run against the `internal_port`. These checks will pass when receiving a 2xx response. Any other response is considered a failure.

The number of consecutive check failures to allow before attempting to restart the VM. The default is `0`, which disables restarts based on failed health checks.

The HTTP method to be used for the check.

The path of the URL to be requested.

The protocol to be used (`http` or `https`)

When set to `true` (and `protocol` is set to `https`), skip verifying the certificates sent by the server.

Set key/value pairs of HTTP headers to pass along with the check request.

The path in the guest where the file will be written.

Base64 encoded content to be written to the file.

The name of the secret whose value will be written to the file.

Local path of the file to be written to the guest.

List of process names that this file configuration applies to.

The port this service (and application) will use to communicate with clients. Default is 8080.

A Boolean which determines whether to enforce HTTP to HTTPS redirects.

Whether to automatically stop or suspend an application’s Machines when there’s excess capacity, per region. Options are "off", "stop", and "suspend". If there’s only one Machine in a region, then the Machine is stopped or suspended if it has no traffic. The Fly Proxy runs a process to automatically stop Machines every few minutes. The default if not set is "off".

The number of Machines to keep running, in the primary region only, when `auto_stop_machines` is true.

Specifies what metric is used to determine when to auto start or stop, or when a given Machine should receive more or less traffic (load balancing).

When a Fly Machine is at or over this number of concurrent connections or requests, the system will deprioritize sending new traffic to that Machine.

When a Fly Machine is at or over this number of concurrent connections or requests, the system will stop sending new traffic to that Machine.

Configures Fly Proxy to not add any Fly headers to HTTP responses.

Add or remove HTTP response headers.

Indicates whether the app supports HTTP/2 cleartext (H2C) with prior knowledge or not.

When `true`, serve a self-signed certificate if no certificate exists. Default is `false`.

The primary region where the application will be deployed. This setting also sets the PRIMARY_REGION environment variable.

Command to run when you execute 'fly console'. This opens your framework's console in a new, dedicated Machine.

Size in megabytes for the swap file on the VM. Helps in handling memory spikes.

Fly.io application name

Seconds to wait before forcing a VM process to exit. Default is 5 seconds.

Signal to send to a process to shut it down gracefully. Default is SIGINT.

SIGINT SIGTERM SIGQUIT SIGUSR1 SIGUSR2 SIGKILL SIGSTOP

The `statics` sections expose static assets built into your application's container to Fly's Anycast network. You can serve HTML files, Javascript, and images without needing to run a web server inside your container.

Command to run after a build, with access to the production environment, but before deployment. Non-zero exit status will abort the deployment.

```toml
[deploy]
  release_command ="bundle exec rails db:migrate"
```

VM configuration to use when running the release command.

For rolling deploys, you can use max_unavailable to control how many Machines can be down at a time.

Strategy for replacing VMs during a deployment.

Strategy for replacing VMs during a deployment.

This default strategy - for apps without volumes - will boot a single, new VM with the new release, verify its health, then proceed with a rolling restart strategy.

One by one, each running VM is taken down and replaced by a new release VM. This is the default strategy for apps with volumes.

For every running VM, a new one is booted alongside it. All new VMs must pass health checks to complete deployment, when traffic gets migrated to new VMs. If your app has multiple VMs, this strategy may reduce deploy time and downtime, assuming your app is scaled to 2 or more VMs.

Replace all VMs with new releases immediately without waiting for health checks to pass. This is useful in emergency situations where you're confident a release will be healthy.

Default is 'canary': boot a single, new VM with the new release, verify its health, then proceed with a rolling restart strategy.

Timeout for waiting for the Machine to be in a started state during a deploy.

Flags and features that are subject to change, deprecation or promotion to the main configuration.

Override the server command (CMD) set by the Dockerfile. Specify as an array of strings:

```toml
cmd = ["path/to/command", "arg1", "arg2"]
```

Override the ENTRYPOINT set by the Dockerfile. Specify as an array of strings:

```toml
entrypoint = ["path/to/command", "arg1", "arg2"]
```

Failed deployments should roll back automatically to the previous successfully deployed release. Defaults to `true`

Enables private network access to the Fly organization. Defaults to `true`.

Set non-sensitive information as environment variables in the application's [runtime environment](https://fly.io/docs/reference/runtime-environment/).
For sensitive information, such as credentials or passwords, use the [secrets command](https://fly.io/docs/reference/secrets). For anything else though, the `env` section provides a simple way to set environment variables. Here's an example:
```toml
[env]
  LOG_LEVEL = "debug"
  S3_BUCKET = "my-bucket"
```

Build configuration options. See docs at https://fly.io/docs/reference/builders.

Builder Docker image to be used with the 'buildpacks' option

Buildpacks to be run by the 'builder' Docker image

Build arguments passed to both Dockerfile and Buildpack builds. These arguments are **not available** on VMs at runtime.
```toml
[build.args]
  USER = "julieta"
  MODE = "production"
```

Specify the target stage for [multistage Dockerfile builds](https://docs.docker.com/develop/develop-images/multistage-build/).

Docker image to be deployed (skips the build process)

Dockerfile used for builds. Defaults to './Dockerfile'

A relative path to a .dockerignore file. Defaults to './.dockerignore'

Fly.io config schema (fly.toml)

Provide a JSON object that satisfies the schema "Fly.io config schema (fly.toml)". Here are the required top‑level fields: none (all optional). Return only the valid JSON object.