1

Topic: Acceleration disk reading by means of BtrFS

In Linakse in 2012 fixed standards of data storage at "new" file system BtrFS. One of possibilities  is data compression "on the fly" at reading / records. As in the project I buried at speed I/O decided to try this possibility. On SSD a disk it will have double advantage: spares a place + accelerates data reading. My text files are csv - files, dataful divided '; ', and the binary - sequentially written down data structures with fields of numerical types.
There is no compression:
The initial size of text files: 37,2
Speed of reading of text files: 136
The initial size of the binary data: 6,8
Compression zlib:
The size of text files: 8,8
Speed of reading of text files: 99
The size of the binary data: 3,2
Compression lzo:
The size of text files: 14
Speed of reading of text files: 74
The size of the binary data: 4
Already there was a new method of compression ZSTD with possibility of setting of numerical compression ratio. I yet did not test it, but I think that with its help it is possible to achieve still  speed of data reading. But even now it is visible that the size on a disk is reduced approximately in 2,5 times, and speed of reading increases approximately in 2 times.
So who will have a similar case, can try also possibilities of file systems from Linaksa.

2

Re: Acceleration disk reading by means of BtrFS

Forgot to write that at me 4-eh the nuclear processor i5-4690 (without hypertrading), therefore data reading from a disk in all three experiments was fulfilled 4  by parallel flows (and principal 5th flow waited for their end). In other words, there was no the free kernel which could be used for an automatic release of the data "on the fly".
+ Text files have the big size (approximately on 300 Mb), therefore the received acceleration it is impossible to carry everyone to more good operation with files of the small size. Acceleration turned out at the expense of compression of files on a disk.

3

Re: Acceleration disk reading by means of BtrFS

For this purpose not mandatory to format the device under btrfs. Library zlib  on all languages and you could read text files without getting them from archive.

4

Re: Acceleration disk reading by means of BtrFS

AlekseySQL wrote:

Speed of reading of text files: 74
The size of the binary data: 4

54 MB/WITH? You are assured, what it SSD, instead of HDD 20-year-old prescription?

5

Re: Acceleration disk reading by means of BtrFS

mayton wrote:

For this purpose not mandatory to format the device under btrfs. Library zlib  on all languages and you could read text files without getting them from archive.

What for to do what OS does for me?

6

Re: Acceleration disk reading by means of BtrFS

miksoft wrote:

it is passed...
54 MB/WITH? You are assured, what it SSD, instead of HDD 20-year-old prescription?

Speed of reading of the TEXT data, their size was checked much more.

7

Re: Acceleration disk reading by means of BtrFS

AlekseySQL wrote:

it is passed...
Speed of reading of the TEXT data, their size was checked much more.

Then that these mean 4 ?

8

Re: Acceleration disk reading by means of BtrFS

miksoft wrote:

it is passed...
Then that these mean 4 ?

The size of the binary data is resulted only for an estimation of a level of their compression: not oblate they occupy 6,8 GBYTES, and for different methods of compression - of 3,2 and 4 Gbytes.
By the way, if on garlic, at me not pure reading  files: the read lines , and values were transformed to number formats. Structures with the received numeric data were added in my array. So there is a small overhead projector at read operations of the text data. Therefore at 2,5 multiple reduction of data size to receive 2-uh the multiple magnification of speed it is possible, if dearching operation practically did not occupy CPU time. I.e. it is possible to try to lift still  compression, receiving a read performance gain.
But it I already will be  only on the binary data: to me speed of their reading is important.

9

Re: Acceleration disk reading by means of BtrFS

AlekseySQL wrote:

it is passed...
Speed of reading of the TEXT data, their size was checked much more.

I to you even will tell more. Old  supported sparse-files. It not archiving
And simply logical  files of the big size thus that they physically
Do not take a place. It is convenient at downloading of serials . Thus on samplings
Speed of a read or write can receive a fake. Speed of data reading from a spindle
And logical speed with which  gives units can differ on orders.
With your text files too there can be a miracle. Depending on that that for the data
Inside.
And with  - be not mistaken. It created for another matters. A principal point - a failure from LVM.
With it the volume manager  is not necessary. The file system is that.
And also copy-on-write. And options .

10

Re: Acceleration disk reading by means of BtrFS

"" it is cool sounds. Almost as "Tampax".
DoubleSpace about what does not speak?

11

Re: Acceleration disk reading by means of BtrFS

Siemargl wrote:

DoubleSpace about what does not speak?

And to it was Stacker...

12

Re: Acceleration disk reading by means of BtrFS

There is even more cool piece - ZFS
For understanding:
the Manual on administration of file systems ZFS Solaris

13

Re: Acceleration disk reading by means of BtrFS

rdb_dev wrote:

There is even more cool piece - ZFS
For understanding:
the Manual on administration of file systems ZFS Solaris

Yes, selecting from compressing  read and about ZFS, but the people write that for it it is necessary much . Besides support ZFS in Ubuntu appeared only in version 16.04, i.e. to the decision only two years, and I do not have time to work beta - a tester.

Siemargl wrote:

"Linaks" is cool sounds. Almost as "Tampax".

Really, the joke was tightened. I will write now Linux.

14

Re: Acceleration disk reading by means of BtrFS

By the way, when I resulted compression indexes lzo and zlib for binary files implied those files which already beforehand are compressed by my algorithm on repeating values. In other words, double compression was used: at first my algorithm, and then by means of . Today delivered experiment on compression with the help lzo structures which did not transit preliminary handling by my algorithm of compression and received more good indexes of compression lzo:
There is no compression:
The initial size of not OBLATE binary data: 16,3
The initial size of the oblate binary data: 6,8
Compression lzo:
The size of not OBLATE binary data: 6
The size of not OBLATE binary data: 4
It is visible that application of my algorithm of compression while is justified: the total size of files becomes less in one and a half time:
TO = 6 / 4 = 1,5
But if to consider an extra time necessary on "release" my algorithm it is possible all pluses come to naught. In other words, application of possibilities  is possible removes from my code superfluous algorithms without loss of speed of reading of the binary data.

15

Re: Acceleration disk reading by means of BtrFS

mayton wrote:

it is passed...
I to you even will tell more. Old  supported sparse-files. It not archiving
And simply logical  files of the big size thus that they physically
Do not take a place....

It was in Simpsons all is in NTFS.

16

Re: Acceleration disk reading by means of BtrFS

Foreheads - the fan of a penguin. I so think.

17

Re: Acceleration disk reading by means of BtrFS

mayton wrote:

Foreheads - the fan of a penguin. I so think.

So-so the fan, does not know that btrfs was never recommended for production and already deprecated and will be remote from .

18

Re: Acceleration disk reading by means of BtrFS

mayton wrote:

Foreheads - the fan of a penguin.

Well to whom does not happen smile
Another matter that a topic to a C ++ has no relation.

19

Re: Acceleration disk reading by means of BtrFS

NekZ;
It came shares idea. Here it is difficult to classify. Let itself tells about a C subject ++.

20

Re: Acceleration disk reading by means of BtrFS

mayton wrote:

For this purpose not mandatory to format the device under btrfs. Library zlib  on all languages and you could read text files without getting them from archive.

I, by the way, considered a question to use zip -  "on the fly". But, unfortunately, in Qt there is no operation with zip - archives, therefore the decision turns out not . At usage btrfs it is finite too not , but it turns out "on a freebie", therefore "the pocket does not pull".

21

Re: Acceleration disk reading by means of BtrFS

AlekseySQL wrote:

But, unfortunately, in Qt there is no operation with zip - archives, therefore the decision turns out not .

At you - the strange understanding .

22

Re: Acceleration disk reading by means of BtrFS

AlekseySQL wrote:

it is passed...
I, by the way, considered a question to use zip -  "on the fly". But, unfortunately, in Qt there is no operation with zip - archives, therefore the decision turns out not . At usage btrfs it is finite too not , but it turns out "on a freebie", therefore "the pocket does not pull".

What relation QT has to the files compressed by algorithm ZIP? If, for any reason, the religion does not allow to use library function zlib, what hinders to pull 7-zip utility CLI for all the ? Quite probably that on an official site 7-zip there is a library and its description API.

23

Re: Acceleration disk reading by means of BtrFS

rdb_dev wrote:

it is quite possible

Only not "quite probably", and LZMA SDK - regular loading since dense times.

24

Re: Acceleration disk reading by means of BtrFS

, after 20 + years of development, from easy submission, zlib suddenly ceased to be  smile

25

Re: Acceleration disk reading by means of BtrFS

He meant - a cross connect-Qt .