1

Topic: Reading of the big file

Hello colleagues, are such more a question (can even more ), solved  the task there is very big raster file, tens gigabyte, and it is necessary to do various filtrations (improving, separation of boundaries, determinations of objects and ), has been implemented in three flows one flow reads in in the ring buffer, other flow is engaged in data acquisition from this buffer handling and additions of results in other buffer from which the third flow wrote down results in a file. And in all this process the slowest part was this data reading from a file, read in the portions of the size from 1 to 10, using normal WinApi functions for operation with files, data handling was practically on the order faster and the processing flow was in waiting of a new portion of the data more. Whether probably there is what that methods to lift speed of reading, that can  a file in storage or what system functions.

2

Re: Reading of the big file

Hello, SL, you wrote: SL> And in all this process the slowest part was this data reading from a file, read in the portions of the size from 1 to 10, using normal WinApi functions for operation with files, data handling was practically on the order faster and the processing flow was in waiting of a new portion of the data more. Whether probably there is what that methods to lift speed of reading, that can  a file in storage or what system functions. There is such problem. Reading rests against speed of a disk swapping. We tell, for WD Black data flow rate between a disk and system of all 227MB/with. It is impossible to read faster, though as read/map. But it is possible to deliver SSD.

3

Re: Reading of the big file

Hello, V. Zudin, you wrote: SVZ> There is such problem. SVZ> reading rests against speed of a disk swapping. We tell, for WD Black data flow rate between a disk and system of all 227MB/with. It is impossible to read faster, though as read/map. SVZ> but it is possible to deliver SSD. RAID rescues the father of Russian democracy. And RAID from SSD it generally a song.

4

Re: Reading of the big file

Hello, SL, you wrote: SL> Hello colleagues, are such more a question (can even more ), solved  the task there is very big raster file, tens gigabyte. You simply incorrectly approached to the task. At you unsuccessful data view. It is not necessary to store a picture one big piece. The multi-level picture which is necessary to you can to be stored on set of separate computing nodes (for example google maps) and in the same place to be processed. And to produce necessary sections on demand.

5

Re: Reading of the big file

Hello, SL, you wrote: SL> Hello colleagues, are such more a question (can even more ), solved  the task there is very big raster file, tens gigabyte, and it is necessary to do various filtrations (improving, separation of boundaries, determinations of objects and ), has been implemented in three flows one flow reads in in the ring buffer, other flow is engaged in data acquisition from this buffer handling and additions of results in other buffer from which the third flow wrote down results in a file. SL> and in all this process the slowest part was this data reading from a file, read in the portions of the size from 1 to 10, using normal WinApi functions for operation with files, data handling was practically on the order faster and the processing flow was in waiting of a new portion of the data more. Whether probably there is what that methods to lift speed of reading, that can  a file in storage or what system functions. Physical restrictions of speed you will not overcome. And at level to try: - to use compression (tiff and if lossy compression - jpeg is admissible). If direct access to fragments-tajlam is necessary, to break the big file on , to compress them; - to track, that the file on a disk was not . If the file creates own , to rummage in win api on this subject; - a format. Whether he allows to read a flow, without driving a disk head once again there-here.