Topic: The new approach to data transformation - alternative map-reduce, SQL, etc.
The majority of approaches and data processing systems, including SQL and map-reduce, are based on conversion of sets of elements (tables, collections, etc.) Operation dataful is represented as an activity graph on sets where one set is defined and calculated from others. The alternative approach uses use conversions of columns (functions). Thus, data transformations are represented as an activity graph with functions (instead of sets), and each function is defined and in runtime is calculated through other functions. Now there are three main operations with columns: Calculate is an analog map or select Link is a type join Accumulate is an analog reduce or group-by For example, the description could look so://to Define columns col1.calc (...); col2.link (...); col3.accu (...); colN.calc (...) ;//to Calculate all columns schema.eval (); Here each determination accepts the user function (as well as in map-reduce), and also pointers on other columns, given which will be for calculation of the data of the given column. This approach is implemented with the help here this library: Bistro: https://github.com/asavinov/bistro It can be included in systems where, for example, potentially difficult data handling is necessary for migration of the data, import-export, generation of reports, a DBMS, flow analysis, etc. It would be interesting to learn judgement concerning this approach generally and its implementations in library Bistro in particular. Whether it will work generally? To whom and where it could be most useful now and in the future? What it is possible to refine in the given implementation? Earlier this approach has been implemented as the web application for operation with tables (arguing (Angular 2) Estimate application Data Commandr for operation with that): Data Commandr: http://dc.conceptoriented.com