Data Provider can also provide an abstraction of data being transformed earlier by another system that can be either a legacy system or another big data system. In this case, Data Provider would represent a Data Consumer of that other system. For example, a (big) streaming data source could be generated by another system operation on (big) data at rest. Data Provider activities include: Creating the metadata describing the data source(s), usage policies/access rights, and other relevant attributes Publishing the availability of the information and the means to access it Making the data accessible by other RA components using suitable programmable interface. Data Service Abstraction The Data Service Abstraction would typically include a registry so that the transformation functions can locate a Data provider, identify what comparable data it contains, understand what types of access allowed, understand what types of analysis are supported, where the data source is located, how to access the data, security requirements for the data, privacy requirements for the data, etc. As such, the interface would include the means for registering the data source, for querying the registry, and a standard set of data contained by the registry. Because the data can be too large to economically move across the network, the interface could also allow submi analysis request (as a software code implementing a certain algorithm for execution) the results returned to the requestor. Subject to data characteristics (such as, volume, velocity, and variety) and system design considerations, interfaces for exposing and accessing data would vary in their complexity and can include both push and pull software mechanisms. These mechanisms can include subscription to events, listening to data feeds, querying for specific data properties or content, and the ability to submit a code for execution to process the data in situ. Note that not all use of data service abstraction would always be automated, but rather might involve a human role logging into the system and providing directions where new data should be transferred (for example, via FTP). Transformation Provider Transformation Provider is the role of executing a generic “vertical system” data life cycle including data collection from various sources, multiple data transformations being implemented using both traditional and new technologies, and diverse data usage. The Transformation functions would typically be specific to the vertical application and therefore are not candidates for standardization. However, the metadata and the policies defined and exchanged between the transformation blocks could be standardized.
Data Provider can also provide an abstraction of data being transformed earlier by another system that can be either a legacy system or another big data system. In this case, Data Provider would represent a Data Consumer of that other system. For example, a (big) streaming data source could be generated by another system operation on (big) data at rest. Data Provider activities include: Creating the metadata describing the data source(s), usage policies/access rights, and other relevant attributes Publishing the availability of the information and the means to access it Making the data accessible by other RA components using suitable programmable interface. Data Service Abstraction The Data Service Abstraction would typically include a registry so that the transformation functions can locate a Data provider, identify what comparable data it contains, understand what types of access allowed, understand what types of analysis are supported, where the data source is located, how to access the data, security requirements for the data, privacy requirements for the data, etc. As such, the interface would include the means for registering the data source, for querying the registry, and a standard set of data contained by the registry. Because the data can be too large to economically move across the network, the interface could also allow submi analysis request (as a software code implementing a certain algorithm for execution) the results returned to the requestor. Subject to data characteristics (such as, volume, velocity, and variety) and system design considerations, interfaces for exposing and accessing data would vary in their complexity and can include both push and pull software mechanisms. These mechanisms can include subscription to events, listening to data feeds, querying for specific data properties or content, and the ability to submit a code for execution to process the data in situ. Note that not all use of data service abstraction would always be automated, but rather might involve a human role logging into the system and providing directions where new data should be transferred (for example, via FTP). Transformation Provider Transformation Provider is the role of executing a generic “vertical system” data life cycle including data collection from various sources, multiple data transformations being implemented using both traditional and new technologies, and diverse data usage. The Transformation functions would typically be specific to the vertical application and therefore are not candidates for standardization. However, the metadata and the policies defined and exchanged between the transformation blocks could be standardized.