DataWeave Reference Documentation
The DataWeave Language is a powerful template engine that allows you to transform data to and from any kind of format (XML, CSV, JSON, Pojos, Maps, etc).
Let us start off with some examples to demonstrate the prowess of Dataweave as a data transformation tool.
String manipulation
This example shows how easy it is to work with strings in DataWeave.
Group by …
This example shows how easy it is to group by a given criteria and then transform that result to match the expected output.
Document Structure
DataWeave files are divided into two main sections:
The Header, which defines directives (optional)
The Body, which describes the output structure
The two sections are delimited by a separator, which is not required if no header is present. The separator consists of three dashes: "---"
Below is a taste of what a
.dwl file looks like. This code describes a conversion:
Header
The DataWeave header contains the directives, which define high level information about your transformation. The structure of the Header is a sequence of lines, each with its own Directives. The Header is terminated with '---'.
Through directives you can define:
DataWeave version
Input types and sources
Output type
Namespaces to import into your transform
Constants that can be referenced throughout the body
Functions that can be called throughout the body
All directives are declared on the header section of your DataWeave document and act upon the entire scope of it. Directives are a mechanism to declare variables and constants and namespace aliases which need to be referenced in the Document. They are also needed to declare the type of the output of your transform. In Anypoint Studio, you can optionally use them to declare additional inputs. You rarely need them for this as any data arriving in the incoming Mule Message is already implicitly recognized as an input.
Version Directive
Through this directive, you specify the version of the DataWeave syntax that is used to interpret the transformation.
%dw 1.0
Namespace Directive
This directive associates an alias with its subsequent URI. The directive is relevant only when either the input or the output is of type XML.
Input Directive
Inputs are declared by assigning a name and a content type. You may define as many input directives as you want. You can then refer to them (or their child elements) in any part of the DataWeave body through the names defined in the directive.
%input payload application/xml
Valid types are:
application/json
application/xml
application/java
application/csv
application/dw
text/json
text/xml
text/csv
CSV Input Parsing
When defining an input of type CSV, there are a few optional parameters you can add to the input directive to customize how the data is parsed. These are not defined in the DataWeave script but on the Mule XML code, in the Transform Message XML element.
In Anypoint Studio there are two ways to achieve this. You can either manually add the attributes to the project’s XML, or do it through the graphical interface, by selecting the element from the tree view in the input section and clicking the gear icon. See Using DataWeave in Studio for more details.
Output Directive
The Output Directive specifies what the output type is in a transformation, which is specified using content/type. Only one output can be specified, the structure of this output is then defined in the DataWeave body.
%output application/xml
Valid types are:
application/json
application/xml
application/java
application/csv
application/dw
text/json
text/xml
text/csv
Skip Null On
Whenever the output is of XML or JSON type and has null values in its elements or attributes, you can specify whether this generates an outbound message that contains fields with "null" values, or if these fields are ignored entirely. This can be set through an attribute in the output directive named skipNullOn, which can be set to three different values: elements, attributes, or everywhere.
%output application/xml skipNullOn="everywhere"
When set to:
elements: A key:value pair with a null value is ignored.
attributes: An XML attribute with a null value is skipped.
everywhere: Apply this rule to both elements and attributes.
Define Constant Directive
You can define a constant in the header, you can then reference it (or its child elements, if any exist) in the DataWeave
When you write your DataWeave file, you define an expression that generates one of the data types listed above.
Simple Literal Types
Literals can be of the following types:
String : Double quoted ("Hello") or Single quoted ('Hello')
Boolean : Literals true or false
Number : Decimal and Integer values are supported (ex: 2.0)
Dates : IS0-8601 enclosed by "|" (ex:|2003-10-01T23:57:59Z|)
Regex : Regex expression enclosed by "/" (ex:/(\d+)-(\d+)/)
This is a String literal expression
Streaming
DataWeave supports streaming a large payload. No configuration is necessary in the DataWeave code itself, but other components need to be set up for this to work. See DataWeave Streaming.
Example Transformation
Suppose you want to transform an XML document to JSON, appending extra content to the output.
DataWeave Canonical Model
As covered above, DataWeave uses three basic data types: Objects, Arrays, and Simple Types, the execution of a DataWeave transformation always produces one of these three types of data. In essence, the body of every DataWeave transformation is a single expression that defines the structure and contents of one such element (which can be an Object, Array, or Simple Literal).
This expression can be built using any of the following elements:
Objects
Arrays
Simple literals
Variable and Constant references
A DataWeave transformation can be as simple as the definition of a single element from the list above. Even a simple literal 'Hello world' is a valid DataWeave transformation. Simple literal.
If you declare input directives in your DataWeave’s header, regardless of its type (XML, JSON, Java), any execution that references these inputs produces, as stated before, an Object, an Array, or a Simple literal. When you understand the structure of these data types, expressed in the syntax of DataWeave expressions, you effectively understand DataWeave.
In Anypoint Studio, if you ever need to visualize the canonical DataWeave model of your data to get a better reference, set the output type of your transform to
application/dw. Your transform then outputs your data as a DataWeave expression, which resembles a JSON object. See the example below.
Literal Expressions
These correspond to the three different data-types: Simple, Object, and Array.
Simple Literal
Object Literal
Array Literal
Variables
Input Variables
Input directives allow you to make any number of input sources available in global variables, which can then be referenced in any part of the Transform’s body. To reference one of these, you can just call it by the name you defined in the directive. Remember that the Transform is itself an expression, so the entire body of the transform could be written as a simple variable reference to the input document. Consider the example below, which transforms an incoming JSON document into XML, and where the output XML structure mimics the input JSON structure. literal
Scoped to Array literal
Scoped to Object literal
Invalid Reference outside of Scope
Selectors
Value Selector Expressions
The complex structure of Objects and Arrays can be navigated using Selector Expressions. Each selector expression returns either an object, an array, or a simple type. A selector always operates within a given context, which can be a reference to a variable, an object literal, an array literal, or the invocation of a function. As DataWeave processes a selector, a new context is set for further selectors, so you can navigate through the complex structures of arrays and objects by using chains of selectors, who’s depth is limited only by the depth of the current context.
There are 4 types of selector expression:
Single Value selector .<key-name>
Multi Value selector .*<key-name>
Descendants Selector ..<key-name>
Indexed Selector [<index>]
Applying the Single level Explicit Selector, the Descendants Selector, or the Indexed Selector returns the value of the key:value pair that matches the expression.
Note: Each of these selector expressions supports having a '?' appended at the end of the chain. This changes the expression into a query that checks upon the existence of the key. The return type in this case is a boolean true or false.
Single Value selector
This selector returns the first value whose key matches the expression, that is, payload.name, which returns the value whose key matches name.
Multi Value selector
This selector returns an array with all the values whose key matches the expression.
Indexed Selector
This selector can be applied to String literals, Arrays and Objects. In the case of Objects, the value of the key:value pair found at the index is returned. The index is zero based.
If the index is bigger or equal to 0, it starts counting from the beginning.
If the index is negative, it starts counting from the end where -1 is the last element.
When using the Index Selector with a String, the string is broken down into an array, where each character is an index.
Range selector
Range selectors limit the output to only the elements specified by the range on that specific order. This selector allows you to slice an array or even invert it.
Attribute Selector Expressions
In order to query for the attributes on an Object, the syntax .@<key-name> is used. If you just use .@ (without <key-name>) it returns an object containing each key:value pair in it.
Applying Selectors to Arrays
When the context for selection is an Array, the result is always an Array. Each element on the context Array is queried for matching key:value pairs. In each case, only the value of the key:value pair is returned.
Descendants Selector
This selector is applied to the context using the form ..<field-name> and retrieves the values of all matching key:value pairs in the sub-tree under the current context. Regardless of the hierarchical structure these fields are organized in, they are all placed at the same level in the output.
Selectors modifiers
There are two selectors modifiers: ? and !. The question mark returns true or false whether the keys are present on the structures. The exclamation mark evaluates the selection and fails if any key is not present.
Key Present
Returns true if the specified key is present in the object.
In the example above, if a 'name' key does exist in the input, it returns true.
This operation also works with attributes:
You can also use this validation operation as part of a filter:
The example above selects key:value pairs with value "Mariano" ⇒ {name: Mariano, name: Mariano}
Assert Present
Returns an exception if any of the specified keys are not found.
Reference Elements From an Incoming Mule Message
Often, you want to use the different elements from the Mule Message that arrive at the DataWeave Transformer in your transform. The following sections show you how to reference each of these.
The Payload of a Mule Message
You can take the Payload of the Mule message that reaches the DataWeave transformer and use it in your transform body.
You can also refer to sub elements of the payload through the dot syntax
payload.user.
You can optionally also define the payload as an input directive in the header, although this isn’t required.
Inbound Properties from a Mule Message
You can take Inbound Properties from the Mule message that arrives to the DataWeave transformer and use them in your transform body. To refer to one of these, simply call it through the matching Mule Expression Language (MEL) expression.
In MEL, there are two supported syntaxes to call for an inbound property:
inboundProperties.name
inboundProperties[’name’]
You can optionally also define the inbound property as a constant directive in the header, although this isn’t required.
%var inUname = inboundProperties['userName']
Outbound Properties from a Mule Message
You can take any Outbound Properties in the Mule message that arrives to the DataWeave transformer and use it in your transform body. To refer to it, simply call it through the matching Mule Expression Language (MEL) expression.
In MEL, there are two supported syntaxes to call an outbound property:
outboundProperties.name
outboundProperties[’name’]
You can optionally also define the outbound property as a constant directive in the header, although this isn’t required.
%var outUname = outboundProperties['userName']
Flow Variables from a Mule Message
You can take any Flow Variable in the Mule message that arrives at the DataWeave transformer and use it in your transform body. To refer to it, simply call it through the matching Mule Expression Language (MEL) expression.
In MEL, there are two supported syntaxes to call a flow variable:
flowVars.name
flowVars[’name’]
You can optionally also define the flow variable as a constant directive in the header, although this isn’t required.
%var uname = flowVars['userName']
Operators
Map
Using Map on an Array
Returns an array that is the result of applying a transformation function (lambda) to each of the elements. The lambda is invoked with two parameters: index and the value. If these parameters are not named, the index is defined by default as $$ and the value as $.
In the following example, custom names are defined for the index and value parameters of the map operation, and then both are used to construct the returned value. In this case, value is defined as firstName and its index in the array is defined as position.
Using Map on an Object
Returns an array with the values that result out of applying a transformation function (lambda) to each of the values in the object. The keys of the original object are all ignored by this operation and the object is treated as an array. To have access to the keys, you can use the operation mapObject instead. The lambda is invoked with two parameters: index and the value. If these parameters are not named, the index is defined by default as $$ and the value as $. The index refers to the position of a key:value pair when the object is treated as an array.
In the example above, as key and value are not defined, they’re identified by the placeholders $$ and $. For each key:value pair in the input, an object is created and placed in an array of objects. Each of these objects contains two properties: one of these directly uses the value, the other multiplies this value by a constant that is defined as a directive in the header.
The mapping below performs exactly the same transform, but it defines custom names for the properties of the operation, instead of using $ and $$. Here,
position is defined as referring to the array index, and
money to the value in that index.
Map Object
Similar to Map, but instead of processing only the values of an object, it processes both keys and values, and instead of returning an array with the results of processing these values through the lambda, it returns an object with the key:value pairs that result from processing both key and value of the object through the lambda.
The lambda is invoked with two parameters: key and the value. If these parameters are not named, the key is defined by default as $$ and the value as $.
In the example above, as key and value are not defined, they’re identified by the placeholders $$ and $. For each key:value pair in the input, the key is preserved and the value becomes an object with two properties: one of these is the original value, the other is the result of multiplying this value by a constant that is defined as a directive in the header.
The mapping below performs exactly the same transform, but it defines custom names for the properties of the operation, instead of using $ and $$. Here, 'category' is defined as referring to the original key in the object, and 'money' to the value in that key.
Pluck
Pluck is useful for mapping an object into an array. Pluck is an alternate mapping mechanism to mapObject. Like mapObject, pluck executes a lambda over every key:value pair in its processed object, but instead of returning an object, it returns an array, which may be built from either the values or the keys in the object.
The lambda is invoked with two parameters: key and the value. If these parameters are not named, the key is defined by default as $$ and the value as $.
Filter
Using Filter on an Object
Returns an object with the key:value pairs that pass the acceptance criteria defined in the lambda. If these parameters are not named, the key is defined by default as $$ and the value as $.
Remove
Using Remove on an Object
When running it on an object, it returns another object where the specified keys are removed.
The above example removes the key value pair that contains the key 'aa' from {aa: "a", bb: "b"} ⇒ {bb: "b"}
When or or.
AND
The expression and (in lower case) can be used to link multiple conditions, its use means that all of the linked conditions must evaluate to true for the expression as a whole to evaluate to true.
In the example above, currency is "EUR", unless the payload has BOTH conditions met.
OR
The expression or (in lowercase) can be used to link multiple conditions. Its use means that either one or all of the linked conditions must evaluate to true for the expression as a whole to evaluate to true.
In the example above, currency is "EUR", only when one of the conditions evaluates to true.
Concat
The concat operator is defined using two plus signs. You must have spaces on both sides of them.
Using Concat on an Object
Returns the resulting object of concatenating two existing objects.
The example above concatenates object {aa: a} and {cc: c} in a single one ⇒ {aa: a , cc: c}
Using Concat on an Array
When using arrays, it returns the resulting array of concatenating two existing arrays.
IS
Evaluates if a condition validates to true and returns a boolean value. Conditions may include
and and
or operators.
Contains
Evaluates if an array or list contains in at least one of its indexes a value that validateso true and returns a boolean value. You can search for a literal value, or match a regex too.
Using Contains on an Array
You can evaluate if any value in an array matches a given condition:
Using Contains on a String
You can also use contains to evaluate a substring from a larger string:
Instead of searching for a literal substring, you can also match it agains a regular expression:
AS (Type Coercion)
Coerce the given value to the specified type. DataWeave by default attempts to convert the type of a value before failing, so using this operator to convert is sometimes not required but still recommended.
Coerce to string
Any simple types can be coerced to string. If formatting is required (such as for a number or date) the format schema property can be used.
Date and number format schemas are based on Java DateTimeFormatter and DecimalFormat.
Coerce to number
A string can be coerced to number. If the given number has a specific format the schema property can be used.
Any format pattern accepted by DecimalFormat is allowed.
Coerce to date
Date types can be coerced from string or number.
Any format pattern accepted by DateTimeFormatter is allowed.
Type Of
Returns the type of a provided element (eg: '":string"' , '":number"' )
Flatten
If you have an array of arrays, this function can flatten it into a single simple array.
Zip and Unzip
If you have two or more separate lists, the zip function can be used to merge them together into a single list of consecutive n-tuples. Imagine two input lists each being one side of a zipper: similar to the interlocking teeth of a zipper, the zip function interdigitates each element from each input list, one element at a time.
Here is another example of the zip function with more than two input lists.
Unzip works similarly to zip except that the input is a single list consisting of two or more embedded lists of elements.
Size Of
Returns the number of elements in an array (or anything that can be converted to an array)
Reduce
Applies a reduction to the array. The lambda is invoked with two parameters: the accumulator ($$) and the value ($). Unless specified, the accumulator by default takes the first value of the array.
In some cases, you may want to not use the first element of the array as the initial value of the accumulator. To set the accumulator to be something else, you must define this in a lambda.
Join By
Merges an array into a single string value, using the provided string as a separator between elements.
Split By
Performs the opposite operation as Join By. It splits a string into an array of separate elements, looking for instances of the provided string and using it as a separator.
Order By
Returns the provided array ordered according to the value returned by the lambda. The lambda is invoked with two parameters: index and the value. If these parameters are not named, the index is defined by default as $$ and the value as $.
Group By
Partitions an array into a Object that contains Arrays, according to the discriminator lambda you define. The lambda is invoked with two parameters: index and the value. If these parameters are not named, the index is defined by default as $$ and the value as $.
Distinct By
Returns only unique values from an array that may have duplicates. The lambda is invoked with two parameters: index and value. If these parameters are not defined, the index is defined by default as $$ and the value as $.
Replace
Replaces a section of a string for another, in accordance to a regular expression, and returns a modified string.
Match
Match a string against a regular expression. Match returns an array that contains the entire matching expression, followed by all of the capture groups that match the provided regex.
In the example above, we see that the search regular expression describes an email address. It contains two capture groups, what’s before and what’s after the @. The result is an array of three elements: the first is the whole email address, the second matches one of the capture groups, the third matches the other one.
Scan
Returns an array with all of the matches in the given string. Each match is returned as an array that contains the complete match, as well as any capture groups there may be in your regular expression.
In the example above, we see that the search regular expression describes an email address. It contains two capture groups, what’s before and what’s after the @. The result is an array with two matches, as there are two email addresses in the input string. Each of these matches is an array of three elements, the first is the whole email address, the second matches one of the capture groups, the third matches the other one.
Similar
Evaluates if two values are similar, regardless of their type. For example, the string "1234" and the number 1234 aren’t equal, but they are recognized as similar.
Capitalize
Returns the provided string with every word starting with a capital letter and no underscores.
Basic Math Operations
Date Time Operations
Global MEL Functions
Your DataWeave code can call any function you define as a global Mule Expression Language (MEL) function, as long as it is correctly defined in the Mule Project where your Transform Message element sits.
Refer to Using DataWeave in Studio.
Object
Type ⇒ ':object'
Objects are represented as a collection of key:value pairs.
Object: { 'Key' : Value }
Key : 'Qualified Name' @('Qualified Name'= Value,…)
Qualified Name: 'namespace prefix#name' where the 'namespace prefix#' part is optional
Name: String that represents the name.
Special Types of Objects
Single Value Objects
If an Object has only one key:value pair, the enclosing curly brackets { } are not required:
Conditional Elements
Objects can define conditional key:value pairs based on a conditional expression.
This example outputs an additional field called "extension" only when the fileSystem property is present in payload (this field may contain any value, not just "true").
If absent:
Dynamic Elements
Dynamic elements allow you to add the result of an expression as key:value pairs of an object.
It is important to note that the expression between the parentheses should return an array of objects. All of objects in that array get merged together. They are also merged with the contained object. So the output looks like this:
String
Type ⇒ ':string'
A string can be defined by the use of double quotes or single quotes.
String interpolation
String interpolation allows you to embed variables or expressions directly in a'
Date time is the conjunction of 'Date' + 'Time'. Local timezone to use.
Period
Type ⇒ ':period'
Specifies a period of time. Examples |PT9M| ⇒ 9 minutes , |P1Y| ⇒ 1 Year
Accessors Expressions
Type ⇒ ':regex'
Regular Expressions are defined between /. For example /(\d+)/ for represents multiple numerical digits from 0-9. These may be used as arguments in certain operations that act upon strings, like Matches or Replace, or on operations that act upon objects and arrays, such as filters. as a Hint for Developers
In Anypoint Studio, it’s easy to view metadata that describes the input and output data of every building block you’re using. When defining a custom type for a particular input or output of your transform, this is represented in the DataWeave transformer’s metadata. Exposing metadata helps you understand what it is you’re integrating to in order to build up the rest of a system, as it lets you know what you need to provide and what you can expect in advance.
Java
Class
Java developers use the 'class' metadata key as hint for what class needs to be created and sent in. If this is not explicitly defined, DataWeave tries to infer from the context or it assigns it the default values:
java.util.HashMap for objects
java.util.ArrayList for lists
The above code defines your type as an instance of 'com.anypoint.df.pojo.User'.
Defining Types For Type Coercion
Format
The metadata 'format' key is used for formatting numbers and dates.
In Anypoint Studio, you can define several more values, like separators, quote characters and escape characters. See Using DataWeave in Studio.
Functions and Lambdas
There are two types of directives you can use to define a function, through
%var (as with variables) using a lambda, or through
%function.
Lambdas
Lambdas can be used inside operators such as map, mapObject, etc. or they can be assigned to a variable. When using lambdas with an operator, they can be either named or anonymous.
Functions
You can declare functions in the Header and these can be invoked at any point in the Body. You refer to them as you do to any variable or constant: using the form $<function-name>() passing an expression as argument. The result of the expression that is passed as an argument is used in the execution of the function body.
Existing Functions
Expressions that Call