1

Topic: How quickly to read big files?

Wrote the wonderful program which reads files in the size ~500 (in a file ~ 10 million lines).  showed that about 2/3 operating times occupies I/O. Therefore I tried to optimize operation, testing 3 variants of the program (in brackets program runtime in two flows is specified):
1. A classical method With ++ std:: ifstream + getline () ( 161 505, 160 876 )
2. Display of ALL file in storage means Qt: QFile + QByteArray + QBuffer + Buffer.readLine () ( 273 488, 296 010 )
3. Map -  a file in storage means Qt: QFile + File.map () ( 157 117, 159 959 )
The one-continuous test yields the same results (but I initially am guided by multi-threaded operation).
At me a classical hard disk (not SSD) with the buffer 64 MB and read-ahead support. Whether there will be a gain of productivity from SSD a disk if to consider, what on section it is a lot of empty seat so files not  and therefore operation goes through the disk buffer?
What else there are artful methods of fast reading of the big files?

2

Re: How quickly to read big files?

You in the first need to know the iron. And to understand where there is that upper bound
When it is necessary to stop and not to torture a software the strange wishes.  speed
Copyings of your file in NULL or in/dev/null on . Also be repelled
From this number. For one-continuous application there will be a formula where this time will be divided
On actually I/O and operation of one round of your parcer.

3

Re: How quickly to read big files?

mayton wrote:

You in the first need to know the iron. And to understand where there is that upper bound
When it is necessary to stop and not to torture a software the strange wishes.  speed
Copyings of your file in NULL or in/dev/null on . Also be repelled
From this number. For one-continuous application there will be a formula where this time will be divided
On actually I/O and operation of one round of your parcer.

Fulfilled sampling of such piece of the code (copying from a disk in  and clearing  at an output from function):

QFile File (NameFile);
if (! File.open (QIODevice:: ReadOnly | QIODevice:: Text)) return;
QByteArray Data = File.readAll ();

Received runtime 158 836 and 165 079  . Iron + OS I means faster these digits on a current sheaf I will not spring.
Then a question about SSD: there will be a productivity gain at its usage or restriction goes at interface SATA level?

4

Re: How quickly to read big files?

AlekseySQL wrote:

Fulfilled sampling of such piece of the code (copying from a disk in

[spoiler ...]  a fillet zero

 1004842352 byte (s)
Current time: 19:06:54,13
It is copied files: 1.
Current time: 19:07:02,81 

Gigabyte for nine seconds. On  a disk of 5400 rpm.
~110MB/sek that is less than restrictions SATA-1 (~150/) [/spoiler]

5

Re: How quickly to read big files?

Basil A. Sidorov wrote:

it is passed...
[spoiler ...]  a fillet zero

 1004842352 byte (s)
Current time: 19:06:54,13
It is copied files: 1.
Current time: 19:07:02,81 

Gigabyte for nine seconds. On  a disk of 5400 rpm.
~110MB/sek that is less than restrictions SATA-1 (~150/) [/spoiler]

That your samplings on an ancient disk - not clearly mean.

6

Re: How quickly to read big files?

I descended bought this SSD. Laid out about 3500 blood. Well to you to tell children...
And all turned out! Ur!!!
Now read test of files from a disk in storage and its clearing is fulfilled for 41 395 and 41 146  (approximately in 4 times faster)!

7

Re: How quickly to read big files?

AlekseySQL wrote:

I Descended bought this SSD. Laid out about 3500 blood. Well to you to tell children...
And all turned out! Ur!!!
Now read test of files from a disk in storage and its clearing is fulfilled for 41 395 and 41 146  (approximately in 4 times faster)!

And if also hands to make.....

8

Re: How quickly to read big files?

You can test mine block_read.h . Wrote for fast reading of the big text files. A topic here
At examples of source codes in the same place you can look uniq-crc.cpp

9

Re: How quickly to read big files?

AlekseySQL wrote:

I Descended bought this SSD. Laid out about 3500 blood. Well to you to tell children...
And all turned out! Ur!!!
Now read test of files from a disk in storage and its clearing is fulfilled for 41 395 and 41 146  (approximately in 4 times faster)!

, it is a vicious way. smile))
There was a trifle... To write in Requirements that your software works only on SSD...

10

Re: How quickly to read big files?

mayton wrote:

Druzhishche, is a vicious way. smile))
There was a trifle... To write in Requirements that your software works only on SSD...

For this reason I at first test silt different algorithms without purchase of new iron.
By the way, for those 160 seconds at me text files in gross weight 16,2  were processed...

11

Re: How quickly to read big files?

AlekseySQL wrote:

That your samplings on an ancient disk - not clearly mean.

5400rpm it is not so ancient disk - you lagged behind life.
But the main thing at all it:

 REXX-Regina_3.9.1 5.00 5 Apr 2015
Read 1004842352 bytes in 11.06 seconds 

[spoiler the interpreter, note]

/* Stupid sample */
parse version version
say version
call time r
parse arg file
if stream (file, c, open read) = ' READY:'
then content = charin (file, stream (file, c, query size))
call stream file, c, close
say ' Read ' length (content) ' bytes in ' format (time (r), 2) ' seconds ' 

[/spoiler]

12

Re: How quickly to read big files?

Basil A. Sidorov wrote:

5400rpm it is not so ancient disk - you lagged behind life.
But the main thing at all it:

 REXX-Regina_3.9.1 5.00 5 Apr 2015
Read 1004842352 bytes in 11.06 seconds 

[spoiler the interpreter, note]

/* Stupid sample */
parse version version
say version
call time r
parse arg file
if stream (file, c, open read) = ' READY:'
then content = charin (file, stream (file, c, query size))
call stream file, c, close
say ' Read ' length (content) ' bytes in ' format (time (r), 2) ' seconds ' 

[/spoiler]

Again something not clear is written...

13

Re: How quickly to read big files?

AlekseySQL wrote:

Again something not clear is written...

semigigabyte Reading should not occupy neither hundred seconds, nor thirty.
Even if you do not have solid-state disk.
Practice shows, as does not occupy.

14

Re: How quickly to read big files?

Basil A. Sidorov wrote:

it is passed...
Semigigabyte reading should not occupy neither hundred seconds, nor thirty.
Even if you do not have solid-state disk.
Practice shows, as does not occupy.

I read not half-gigabyte, and 16,2 . Polgigabajta - the size of one file.