1

Topic: Aggregation of the data sql through kafka in mongodb

The task:
Data from devices registers in sql server each 5 seconds This data I can be with identical values to go.
It would be desirable to send this data in a flow kafka, to process them and to lay down in Mongodb already given without identical values
SQL the Example:

wrote:

Date Value
2/7/2018 OF 0:11:10 PM 13.12
07.02.2018 12:11:15 13.12
07.02.2018 12:11:20 13.12
07.02.2018 12:11:25 16.28
07.02.2018 12:11:25 16.28
07.02.2018 12:11:25 16.28
=>
MongoDB:
2/7/2018 OF 0:11:10 PM 13.12
07.02.2018 12:11:25 16.28

Prompt in what side to dig?

2

Re: Aggregation of the data sql through kafka in mongodb

What admissible time delay between an urgency of the data in a sequel and ?
Whether to begin with esteem about conjugation of a sequel and  approaches you.

3

Re: Aggregation of the data sql through kafka in mongodb

wrote:

what admissible time delay between an urgency of the data in a sequel and ?
Whether to begin with esteem about conjugation of a sequel and  approaches you.

Yes approaches, already it works and it is very good.
Time delay no more than 3 seconds
To me to understand what method to do aggregation, it is possible with the help  in the separate table sql and it already to send to a flow, or there are any possibilities at  who that knows?

4

Re: Aggregation of the data sql through kafka in mongodb

mr_max;
It would be logical  before  to deliver and read from it both in , and in . Before  to deliver , or on basis  , or what another  processing.

5

Re: Aggregation of the data sql through kafka in mongodb

Can simply create an index in a sequel?
With an option: IGNORE_DUP_KEY = ON

6

Re: Aggregation of the data sql through kafka in mongodb

wrote:

can simply create an index in a sequel?
With an option: IGNORE_DUP_KEY = ON

In SQL want, that all pattern was stored.
And there not absolutely identical values.
There fractional values
5.565
5.567
5.568
And such here values it would be possible .

7

Re: Aggregation of the data sql through kafka in mongodb

Whether

wrote:

conjugation of a sequel and  approaches you.

mr_max wrote:

Yes approaches, already it works and it is very good.

then to you above already advised... Look aside

8

Re: Aggregation of the data sql through kafka in mongodb

The grandfather;
But at me already   costs, which copies stupidly given of one basis in another.
Now how to adjust  on  as I offered or there is more refined method?

9

Re: Aggregation of the data sql through kafka in mongodb

mr_max;
As a variant, it is possible to consider Microsoft StreamInsight, time at you there all the same is MS SQL
https://msdn.microsoft.com/ru-ru/library/ee391416 (v=sql.111).aspx