"parameters": { "hadoop": { "zookeeper.quorum": [ "IP", "IP", "IP"] } } } } } The deployment has as inputs parameters defining the applied function and the input data. Both Map/Reduce function and data objects define a parameter, which specifies the location it is retrieved from. For instance, source the ``file://'' Uniform Resource Identifier (URI) indicates sending a directory structure from the local file system and the ``ftp://'' indicates that the data should be fetched from a File Transfer Protocol (FTP) resource. It is the framework's responsibility to materialize an instantiation of the desired environment along with the function and data. Object 4.37: Map/Reduce { "mapreduce": { "function": { "source": "file://.", "args": {} }, "data": { "source": "ftp:///...", "dest": "/data" }, "fault_tolerant": true, "backend": {"type": "hadoop"} } } Additional parameters include the and parameters. The former flag indicates if the fault\_tolerant backend deployment should operate in a fault tolerant mode. For instance, in the case of Hadoop, this may Map/Reduce mean configuring automatic failover of name nodes using Zookeeper. The parameter accepts an object backend describing the system providing the workflow. This may be a native deployment of Hadoop, or a Map/Reduce special instantiation using other frameworks such as Mesos. A function prototype is defined earlier. Key properties are that functions describe their input parameters and generated results. For input parameters, the and respectively describe the objects buildInputs systemBuildInputs that should be evaluated and system packages that should be present before this function can be installed. The attribute describes how to apply this function to its input data. Parameters affecting the evaluation of the eval function may be passed in as the attribute. The results of the function application can be accessed via the args object, which is a mapping from arbitrary keys (e.g., data, processed, model) to an object representing the outputs result.
"parameters": { "hadoop": { "zookeeper.quorum": [ "IP", "IP", "IP"] } } } } } The Map/Reduce deployment has as inputs parameters defining the applied function and the input data. Both function and data objects define a source parameter, which specifies the location it is retrieved from. For instance, the ``file://'' Uniform Resource Identifier (URI) indicates sending a directory structure from the local file system and the ``ftp://'' indicates that the data should be fetched from a File Transfer Protocol (FTP) resource. It is the framework's responsibility to materialize an instantiation of the desired environment along with the function and data. Object 4.37: Map/Reduce { "mapreduce": { "function": { "source": "file://.", "args": {} }, "data": { "source": "ftp:///...", "dest": "/data" }, "fault_tolerant": true, "backend": {"type": "hadoop"} } } Additional parameters include the fault\_tolerant and backend parameters. The former flag indicates if the Map/Reduce deployment should operate in a fault tolerant mode. For instance, in the case of Hadoop, this may mean configuring automatic failover of name nodes using Zookeeper. The backend parameter accepts an object describing the system providing the Map/Reduce workflow. This may be a native deployment of Hadoop, or a special instantiation using other frameworks such as Mesos. A function prototype is defined earlier. Key properties are that functions describe their input parameters and generated results. For input parameters, the buildInputs and systemBuildInputs respectively describe the objects that should be evaluated and system packages that should be present before this function can be installed. The eval attribute describes how to apply this function to its input data. Parameters affecting the evaluation of the function may be passed in as the args attribute. The results of the function application can be accessed via the outputs object, which is a mapping from arbitrary keys (e.g., data, processed, model) to an object representing the result.