26

Re: What DB to select?

White Owl wrote:

And  that we have a special forum "DBMS Comparing"?

I widely enough have a question, including files and various formats of type dbf, xml... I do not think that these variants there can be considered.

27

Re: What DB to select?

AlekseySQL wrote:

In a binary format the data stores all fields (in structures), and all fields are read also. Therefore there was a thought to use a DB, not to read superfluous fields (and conversion from DB sampling in the necessary objects most likely as well as in the previous case practically will not decelerate operation).

Break structure on two: often used fields and remaining. Store in two different tables, and communication on a key or it is simple on an index, i.e. record number in the table.

AlekseySQL wrote:

What DB advise at similar volume? Whether approach for this purpose dbf - files (or they too are read entirely)? While criterion one - maximum speed.

You already made the fastest variant. No DB rescues you, since at DB usage there will be a superfluous loading on percents, and superfluous reading of a disk since the DB except the data contains auxiliary  can also: indexes etc.
think as is better to refine layout of the data in your DB, to minimize disk readings.
Can it is necessary enter any auxiliary . for acceleration of search of the necessary data.
Here it is necessary to be repelled, look from a specific target which operations are more often fulfilled and under them to do optimization.
PS If disk IO it is critical, it is possible to take faster disks, modern SSD read linearly 3 + Gb/second

28

Re: What DB to select?

AlekseySQL wrote:

it is passed...
I widely enough have a question, including files and various formats of type dbf, xml... I do not think that these variants there can be considered.

In your task xml does not give any advantages. It is the same csv by the nature of access.

29

Re: What DB to select?

Dima T wrote:

You already made the fastest variant. No DB rescues you, since at DB usage there will be a superfluous loading on percents, and superfluous reading of a disk since the DB except the data contains auxiliary  can also: indexes etc.

It is the incorrect information. Depends on the input data and from the demanded logic of handling.

30

Re: What DB to select?

Dima T wrote:

You already made the fastest variant. No DB rescues you, since at DB usage there will be a superfluous loading on percents, and superfluous reading of a disk since the DB except the data contains auxiliary  can also: indexes etc.
.

Well so it is the "auxiliary" information and allows to work more quickly dataful.

31

Re: What DB to select?

Apropos dbf. Most likely does not fly up.
Software solutions such as
Dbase, clipper, e. T. A C. There was during an epoch 32 bit gland and a software with all that it implies. It is impossible to open a file in the size more than 2 . And so forth.
If under dbf you meant another - that specify that.

32

Re: What DB to select?

Arm79 wrote:

it is passed...
It is the incorrect information. Depends on the input data and from the demanded logic of handling.

Generally - it agree. But, unfortunately, it does not uncover the HARDWARE of details of the solved task, therefore we start with its statement that the task is already solved optimally.

982183 wrote:

it is passed...
Well so it is the "auxiliary" information and allows to work more quickly dataful.

I so understand this superfluous for the given task.

33

Re: What DB to select?

To us are necessary though any digits.
SQL. Make  in the console:
$ cp file.csv/dev/null
Also inform us how many it occupied time in seconds.

34

Re: What DB to select?

mayton wrote:

To us are necessary though any digits.
SQL. Make  in the console:
$ cp file.csv/dev/null
Also inform us how many it occupied time in seconds.

The file in the size 352 was copied "in anywhere" about 1 second.

35

Re: What DB to select?

Dima T wrote:

PS If disk IO it is critical, it is possible to take faster disks, modern SSD read linearly 3 + Gb/second

If 3Gb/sek - it is 3 Gbyte/second for this purpose it is necessary the computer completely  that mother supported the interface of M. 2. And after that it is necessary to replace the processor + storage... So not a variant.

36

Re: What DB to select?

AlekseySQL wrote:

it is passed...
If 3Gb/sek - it is 3 Gbyte/second for this purpose it is necessary the computer completely  that mother supported the interface of M. 2. And after that it is necessary to replace the processor + storage... So not a variant.

Yes, speech about M.2 with support PCIe x4.
it is not mandatory, there are controlers-adapters PCIe - M.2.

37

Re: What DB to select?

Dima T wrote:

Yes, speech about M.2 with support PCIe x4.
it is not mandatory, there are controlers-adapters PCIe - M.2.

Thanks! Did not know about such and thought that such speeds of operation with a disk are not accessible to me.

38

Re: What DB to select?

Dima T wrote:

Yes, speech about M.2 with support PCIe x4.
it is not mandatory, there are controlers-adapters PCIe - M.2.

Where you were when I ssd with sata - the interface to myself bought recently?
I bought a disk of ~10 times in the smaller speed, twice in the smaller size and only ~1,5 times are cheaper, than there are disks for M.2...
p.s. I go for a drive on a floor from a grief smile

39

Re: What DB to select?

AlekseySQL wrote:

Thanks! I always thought that time SQL - requests allows to specify specific fields for sampling also reading happens only this field. What there was my surprise when I read it:

Only consider that the size of basis at  storage, can be much more, than at row-wise and this size will grow linearly at filling of records with any data, even NULL whereas some DB of row-wise storage are capable to optimize the sizes of records, storing only bit flags for NULL and fields,  to change (in a case with  a DBMS).

40

Re: What DB to select?

AlekseySQL wrote:

Where you were when I ssd with sata - the interface to myself bought recently?
I bought a disk of ~10 times in the smaller speed, twice in the smaller size and only ~1,5 times are cheaper, than there are disks for M.2...
p.s. I go for a drive on a floor from a grief smile

Is ssd - disks which connect directly to PCI Express 3.0 and the offered adapters demand presence PCI Express 3.0 .
And so, as I remember, on yandex - a market by means of filters I searched :
1. "On sale"
2. LGA - 1150
3. PCI Express 3.0
Also found only models Supermicro (at them specific appearance, therefore were remembered) at the inadequate prices. And now there there are models (from 5500 roubles) which support my processor and possess connector PCI Express 3.0.
So if "presses", it will be possible .

41

Re: What DB to select?

Basil A. Sidorov wrote:

databases with  storage
?

They here are not necessary most likely.
And your simplified understanding "is read only by all fields from the table" incorrectly.
All is much more difficult.

42

Re: What DB to select?

AlekseySQL wrote:

Is 50  the text information (csv - a format). For acceleration of reversal I overtook it in binary files (fulfilling conversions from the text in the necessary data types), receiving 33 .
What DB advise at similar volume? Whether approach for this purpose dbf - files (or they too are read entirely)? While criterion one - maximum speed.

How many there records, objects?
The volume in bytes to very few people is interesting at talk about a DB.

43

Re: What DB to select?

AlekseySQL wrote:

And so as I remember, on yandex - a market by means of filters I searched :
1. "On sale"
2. LGA - 1150
3. PCI Express 3.0
Also found only models Supermicro (at them specific appearance, therefore were remembered) at the inadequate prices. And now there there are models (from 5500 roubles) which support my processor and possess connector PCI Express 3.0.
So if "presses", it will be possible .

On a site of vendors analyzed these  and it appeared that PCI Express [u 3.0 [/u slot has speed x16, and the necessary slot with a speed x4 has old version PCI Express 2.0 . So a release: for M.2 it is necessary to change in addition: mother, the processor, storage.

44

Re: What DB to select?

MasterZiv wrote:

it is passed...
How many there records, objects?
The volume in bytes to very few people is interesting at talk about a DB.

In a file of the sizes of 352 MB 6 000 000 lines are allocated, each of which represents  record. In other words, one record in a text format weighs about 60 Byte (and in binary a bit less than ~40 Byte).
I will soon try speed at  storage in a file and here  result.

45

Re: What DB to select?

Is better you would describe character of the data in a file, allocation on values of columns, structure of record and prospective actions over the data.
So much time killed, and the reality is not present till now

46

Re: What DB to select?

AlekseySQL wrote:

In a file of the sizes of 352 MB 6 000 000 lines are allocated

It is all   in  even 32 bit processes. What for you generally
You arrange transfusion from a disk on a disk - absolutely not clearly.

47

Re: What DB to select?

AlekseySQL wrote:

it is passed...
On a site of vendors analyzed these  and it appeared that PCI Express [u 3.0 [/u slot has speed x16, and the necessary slot with a speed x4 has old version PCI Express 2.0 . So a release: for M.2 it is necessary to change in addition: mother, the processor, storage.

what for to change?
Well PCI the card will work in a mode 2.0 with 4 lines

48

Re: What DB to select?

Arm79 wrote:

Is better you would describe character of the data in a file, allocation on values of columns, structure of record and prospective actions over the data.
So much time killed, and the reality is not present till now

Please, wait for result of the test at "" storage. Probably all is already made, only is not tested.

49

Re: What DB to select?

Dimitry Sibiryakov wrote:

it is passed...
It is all   in  even 32 bit processes. What for you generally
You arrange transfusion from a disk on a disk - absolutely not clearly.

I prepare the data for the subsequent repeated reading.

50

Re: What DB to select?

The task as usual remained behind a frame