1

Topic: Algorithm of sifting of the wrong data

There are certain sensors transferring measurements through a certain interval of time t +-x seconds. Results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. In the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. It would be desirable to eliminate the wrong data. But that that too difficult turns out. Thought  the data, and after to discard that quit on value and an interval. But thus the saltus of the data "disappears" at sensor control/switching-off transfer. That is again it is necessary on this variant that that to mold. Or nevertheless it does not turn out easier?

2

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: would Show examples that.

3

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. What exactly is measured by sensors and with what accuracy? When checking there transited last time AN> Results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. Curves happen different. Sensors produce measurements with a marker of time or simply bring down sequentially? And if transfer of sensors that after switching-on any time normally is possible it is necessary on warming up (an output on a nominal). At some instruments such transition  to a half an hour can be. AN> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often.  groups on three sensors. They at temperature change say lies or is simple after switching-on or depending on an input supply? AN> It would be desirable to eliminate the wrong data. But that that too difficult turns out. It is much better to have an estimation of an error of measurement.

4

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> It would be desirable to eliminate the wrong data. But that that too difficult turns out. At me two sentences: 1. Clustering 2. If value range some error of measurements it is possible to discard simply the data "strongly falling outside the limits 3 is in advance known and known. To apply  and to use confidential , but I badly remember it also to you it is necessary to understand most

5

Re: Algorithm of sifting of the wrong data

6

Re: Algorithm of sifting of the wrong data

Hello, kov_serg, you wrote: _> Hello, AlexNek, you wrote: AN>> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. _> what exactly is measured by sensors and with what accuracy? When checking transited last time in the core temperature. What checking of the house? AN>> results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. _> curves happen different. Sensors produce measurements with a marker of time or simply bring down sequentially? At me the data only for: the code of the sensor, time and value marker. _> and if transfer of sensors that after switching-on any time normally is possible it is necessary on warming up (an output on a nominal). All on batteries. Look approximately so, only without the display Searched in the beginning that for that similar with Z Wave, but anything decent and concerning the cheap did not find. _> At some instruments such transition  to a half an hour can be. AN>> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. _> Stavte groups on three sensors. At me not KIPOVSKY a lab... _> They at temperature change say lies or is simple after switching-on or depending on an input supply? Depending on what that of not clear hogwash. AN>> It would be desirable to eliminate the wrong data. But that that too difficult turns out. _> it is much better to have an estimation of an error of measurement. That's not the point. For example 1: 0 minutes - 10, 3-15, 5-13, 10-14. For 5 minute periods. 2: 0 minutes - 10, 3-12, 5min - 8, 10-14. For 5 minute periods.

7

Re: Algorithm of sifting of the wrong data

Hello, Kernan, you wrote: K> Hello, AlexNek, you wrote: AN>> It would be desirable to eliminate the wrong data. But that that too difficult turns out. K> at me two sentences: K> 1. Clustering by what criteria to break into clusters? K> 2. If value range some error of measurements it is possible to discard simply the data "strongly falling outside the limits is in advance known and known we assume  16. 24, and here 18 it for a present situation truly or not? K> 3. To apply  and to use confidential , but I badly remember it also to you it is necessary to understand most does not approach, in this case only one value fig. And at me two.

8

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. AN> results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. AN> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. AN> It would be desirable to eliminate the wrong data. But that that too difficult turns out. AN> thought  the data, and after to discard that quit on value and an interval. But thus the saltus of the data "disappears" at sensor control/switching-off transfer. That is again it is necessary on this variant that that to mold. AN> or nevertheless it does not turn out easier? So Ransac, not?

9

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. AN> results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. AN> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. AN> It would be desirable to eliminate the wrong data. But that that too difficult turns out. AN> thought  the data, and after to discard that quit on value and an interval. But thus the saltus of the data "disappears" at sensor control/switching-off transfer. That is again it is necessary on this variant that that to mold. AN> or nevertheless it does not turn out easier? As a variant: Dixon's Q test

10

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> Hello, kov_serg, you wrote: _>> Hello, AlexNek, you wrote: AN>>> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. _>> what exactly is measured by sensors and with what accuracy? When checking transited last time AN> in the core temperature. What checking of the house? Elementarily - nearby the normal thermometer suppose and compare indications. AN>>> results of measurement well enough can be described by curves. Though sensor transfer from place to place or generally its switch-off for some time is possible. _>> curves happen different. Sensors produce measurements with a marker of time or simply bring down sequentially? AN> at me the data only for: the code of the sensor, time and value marker. _>> and if transfer of sensors that after switching-on any time normally is possible it is necessary on warming up (an output on a nominal). AN> all on batteries. Look approximately so, only without display AN> Image: rst-02252-305x305.jpg AN> Searched in the beginning that for that similar with Z Wave, but anything decent and concerning the cheap did not find. _>> at some instruments such transition  to a half an hour can be. Here a heap of sources of heat, a battery of radio the unit (in  noises can raise capacity) AN>>> In the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. _>> Stavte groups on three sensors. AN> at me not KIPOVSKY a lab... _>> They at temperature change say lies or is simple after switching-on or depending on an input supply? AN> depending on what that of not clear hogwash. Try to localize a hogwash or change sensors if a hogwash in them. AN>>> It would be desirable to eliminate the wrong data. But that that too difficult turns out. _>> it is much better to have an estimation of an error of measurement. AN> matter is not in it. AN> for example AN> 1: 0 minutes - 10, 3-15, 5-13, 10-14. For 5 minute periods. AN> 2: 0 minutes - 10, 3-12, 5min - 8, 10-14. For 5 minute periods. Here it is not absolutely clear where there are sensors, there can be drafts, the sun can fall, or the cat came to get warm. As alternative: it is possible on the microcontroller  i2c temperature sensors (8 LM75 for example) to hang up and on infrared port to betray, elements the minimum will be.

11

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: It is a saw on tails failures? You interrogate sensors by turns in flow 3 minutes or at once all each 3 minutes?

12

Re: Algorithm of sifting of the wrong data

Hello, 3141566=Z, you wrote: Z> Hello, AlexNek, you wrote:... Z> So Ransac, not? It is necessary to look. https://ru.wikipedia.org/wiki/RANSAC though at once confuses point 2 "function M, allowing to calculate parameters  models P on a data set from n points"

13

Re: Algorithm of sifting of the wrong data

Hello, Spinifex, you wrote: S> Hello, AlexNek, you wrote:... S> As a variant: S> Dixon's Q test it is necessary to understand more close too

14

Re: Algorithm of sifting of the wrong data

Hello, kov_serg, you wrote: _>>> Hello, AlexNek, you wrote: AN>>>> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. _>>> what exactly is measured by sensors and with what accuracy? When checking transited last time AN>> in the core temperature. What checking of the house? _> it is elementary - nearby the normal thermometer suppose and compare indications. And without the thermometer clearly that some measurements are beaten out from the correct row. AN>>>> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. _>>> Stavte groups on three sensors. AN>> at me not KIPOVSKY a lab... _>>> They at temperature change say lies or is simple after switching-on or depending on an input supply? AN>> depending on what that of not clear hogwash. _> Try to localize a hogwash or change sensors if a hogwash in them. Well it is admissible a hogwash in "server", what it gives to me? Though "server" changed, the same. I need to "clean" basis simply. At record in basis to filter probably it does not turn out. AN>>>> It would be desirable to eliminate the wrong data. But that that too difficult turns out. _>>> it is much better to have an estimation of an error of measurement. AN>> matter is not in it. AN>> for example AN>> 1: 0 minutes - 10, 3-15, 5-13, 10-14. For 5 minute periods. AN>> 2: 0 minutes - 10, 3-12, 5min - 8, 10-14. For 5 minute periods. _> here it is not absolutely clear where there are sensors, there can be drafts, the sun can fall, or the cat came to get warm. _> as alternative: it is possible on the microcontroller  i2c temperature sensors (8 LM75 for example) to hang up and on infrared port to betray, elements the minimum will be. The normal box on street, on  and in rooms is necessary to me. Earlier "server" on  thought to make, but it turns out more expensively than to buy all ready, and trouble only with a software in this case.

15

Re: Algorithm of sifting of the wrong data

Hello, kov_serg, you wrote: _> Hello, AlexNek, you wrote: _> Image: sample.jpg _> It is a saw on tails failures? _> you interrogate sensors by turns in flow 3 minutes or at once all each 3 minutes? A saw one of failure types. I to sensors have no relation. "Server" - the same small box is engaged in all. "Server" the automatic machine sends given - here this data I and I receive. Each type of sensors has the interval. Is 3 minute, there are 7 minute. But here what iron is such and will be. Any  it is not foreseen.

16

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> Hello, kov_serg, you wrote: _>>>> Hello, AlexNek, you wrote: AN>>>>> There are certain sensors transferring measurements through a certain interval of time t +-x seconds. _>>>> what exactly is measured by sensors and with what accuracy? When checking transited last time AN>>> in the core temperature. What checking of the house? _>> it is elementary - nearby the normal thermometer suppose and compare indications. AN> and without the thermometer clearly that some measurements are beaten out from the correct row. Disassemble one sensor and on  look at characteristics. AN>>>>> in the course of the analysis of results it was clarified that sensors sometimes "say lies" thus the data  much more often or less often. _>>>> Stavte groups on three sensors. AN>>> at me not KIPOVSKY a lab... _>>>> They at temperature change say lies or is simple after switching-on or depending on an input supply? AN>>> depending on what that of not clear hogwash. _>> Try to localize a hogwash or change sensors if a hogwash in them. AN> Well it is admissible a hogwash in "server", what it gives to me? Though "server" changed, the same. AN> I need to "clean" basis simply. At record in basis to filter probably it does not turn out. It is possible to filter, but it is necessary to understand precisely that you want to throw out.  records also experiment with filters. The most simple .. Type filters y [i+1] =q*y [i] + (1-q) *x [i] where q=exp (ln (th)/n), n-number of measurements on which on  history will be  an accuracy threshold for example %0.1 th=0.001 we take energy of a signal (a dispersion) (x [i+1]-x [i]) ^2 as is smoothed for example on 4 measurements and on a threshold is defined when rings.... Generally a variant much. It is possible to construct models and to predict values, but at you most likely sensors . AN>>>>> it would be desirable to eliminate the wrong data. But that that too difficult turns out. _>>>> it is much better to have an estimation of an error of measurement. AN>>> matter is not in it. AN>>> for example AN>>> 1: 0 minutes - 10, 3-15, 5-13, 10-14. For 5 minute periods. AN>>> 2: 0 minutes - 10, 3-12, 5min - 8, 10-14. For 5 minute periods. _>> here it is not absolutely clear where there are sensors, there can be drafts, the sun can fall, or the cat came to get warm. _>> as alternative: it is possible on the microcontroller  i2c temperature sensors (8 LM75 for example) to hang up and on infrared port to betray, elements the minimum will be. AN> the normal box on street, on  and in rooms is necessary to me. "Server" on  earlier thought to make, but it turns out more expensively than to buy all ready, and trouble only with a software in this case. There are arduin and sensors sensors +-0.5S or most on any msp430 to collect and in hermetic  a box to thrust.

17

Re: Algorithm of sifting of the wrong data

AN> Here for example a typical case of failure Such sensation that at a given time gets to a data link a high-frequency component. The saw can goes when the refrigerator compressor turnes on? AN> sensors send signals on a radio channel Carrying frequency it is possible to twist?

18

Re: Algorithm of sifting of the wrong data

19

Re: Algorithm of sifting of the wrong data

Hello, Dym On, you wrote: AN>> Here for example a typical case of failure DO> Such sensation that at a given time gets to a data link a high-frequency component. The saw can goes when the refrigerator compressor turnes on? About exterior noises as that did not think. But periodicity of sending of the data regulates any microcontroller and to force down its noise thus as that it is improbable. Though there can be an error in its program which is initiated by a noise. AN>> sensors send signals on radio channel DO> Carrying frequency it is possible to twist? Most likely is not present, for certain quartz what  costs, and everywhere all will be identical for changing is problematic.

20

Re: Algorithm of sifting of the wrong data

21

Re: Algorithm of sifting of the wrong data

22

Re: Algorithm of sifting of the wrong data

Hello, AlexNek, you wrote: AN> Hello, 3141566=Z, you wrote: Z>> Hello, AlexNek, you wrote: AN>... Z>> So Ransac, not? AN> it is necessary to look. AN> https://ru.wikipedia.org/wiki/RANSAC AN> though at once confuses point 2 AN> "function M, allowing to calculate parameters  models P on a data set from n points" Apply the elementary model - how many that of low harmonics with  (parameters model) or  the low order., as the data as a whole smooth it seizes them. A saw throws out, naturally.