51

Re: What DB to select?

AlekseySQL wrote:

I Prepare the data for the subsequent repeated reading.

From the RAM to read much more conveniently, than from a disk.

52

Re: What DB to select?

wrote:

what for to change?
Well PCI the card will work in a mode 2.0 with 4 lines

Found interesting article about operation ssd - disks in various connectors. For me serial operations (the first schedule) are important. There it is visible that speed of a disk in PCI Express 2.0 slot stops on 1.2 GBYTES/SEK (for a disk, allowing to work for speed of 1.75 GBYTES/SEK).
How PCI Express 2.0 shows for a disk working on 3.2 GBYTES/SEK? Too it will be restricted to speed of 1.2 GBYTES/SEK?

53

Re: What DB to select?

AlekseySQL, overall performance of all system is equal to overall performance of its most feeble component.

54

Re: What DB to select?

wrote:

the Task as usual remained behind a frame

And how differently to justify immediate changeover of iron?
Not algorithms to be picked.
Only a hardcore.

55

Re: What DB to select?

AlekseySQL wrote:

it is passed...
Found interesting article about operation ssd - disks in various connectors. For me serial operations (the first schedule) are important. There it is visible that speed of a disk in PCI Express 2.0 slot stops on 1.2 GBYTES/SEK (for a disk, allowing to work for speed of 1.75 GBYTES/SEK).
How PCI Express 2.0 shows for a disk working on 3.2 GBYTES/SEK? Too it will be restricted to speed of 1.2 GBYTES/SEK?

Interesting article. In gland modern I dig a little, thought all PCI Express `  are identical, only lines add, and it appeared is more difficult arranged.
Itself recently thought to replace the SSD PCI-E 2.0 x2 on more bright. But I to sense from it have not enough, since all data in  is cached, therefore while postponed purchase.

56

Re: What DB to select?

AlekseySQL wrote:

As PCI Express 2.0 shows for a disk working on 3.2 GBYTES/SEK? Too it will be restricted to speed of 1.2 GBYTES/SEK?

I think yes, I so understood that 1.2 GBYTES/SEK are limit PCI Express 2.0 x4

57

Re: What DB to select?

MasterZiv wrote:

And your simplified understanding

Not mine - it is not necessary to attribute to me another's sins.
P.S.
- Here you remember, to that the notorious sine of forty five is equal?
- Root from two on two.
- And it though time was useful to you?
- Certainly. You asked - I answered.

58

Re: What DB to select?

How much to me changes a sclerosis, a line PCI-Express - ~ gigabyte a second.
P.S. . Does not change.

59

Re: What DB to select?

Basil A. Sidorov wrote:

How much to me changes a sclerosis, a line PCI-Express - ~ gigabyte a second.
P.S. . Does not change.

It depends on version PCI Express: each new version approximately in 2 times faster the previous. So logically, if PCI Express 3.0 can transfer 3.2 GBYTES/SEK (because there are such disks) PCI Express 2.0 should allow to transfer 1.6 GBYTES/SECONDS Why then in this article happened  on 1.2 Gbytes/sek - not clearly.

60

Re: What DB to select?

AlekseySQL wrote:

It depends on version PCI Express

I, generally, resulted a minimum.
Whatever ancient was version PCI-Express, on four lines to you is almost guaranteed ~800 + Mbyte/second

61

Re: What DB to select?

AlekseySQL wrote:

Why then in this article happened  on 1.2 Gbytes/sek - not clearly.

Even through the main waterpipe it is impossible to pump over cubic meter a second by means of the manual pump.
P.S.
At you any surprisingly naive representations.
At all I do not know - to be afflicted or smile...

62

Re: What DB to select?

Basil A. Sidorov wrote:

it is passed...
Even through the main waterpipe it is impossible to pump over cubic meter a second by means of the manual pump.
P.S.
At you any surprisingly naive representations.
At all I do not know - to be afflicted or smile...

Simply the people do not understand singularities of operation of the system bus and competitive usage lane' (bands) serial (serial) bus PCI-E. Moreover, some confuse till now the size of PCI-E slot to amount of bands really used in this slot and very few people reflects on that, how many is real bands at a specific chipset on the motherboard and whether these bands for switching of all necessary for them PCI-E suffices devices on a maximum of their possibilities.

63

Re: What DB to select?

Basil A. Sidorov wrote:

it is passed...
Even through the main waterpipe it is impossible to pump over cubic meter a second by means of the manual pump.
P.S.
At you any surprisingly naive representations.
At all I do not know - to be afflicted or smile...

The disk from article can work on 1,7 Gb, thus PSI EXxpress 3.0 allows to work to disks of 3,2 Gbytes/sek, PSI EXxpress 2.0 twice more slowly, than PSI EXxpress 3.0 (so passes a minimum of 1,6 Gbytes/sek). Therefore it is not clear, why the disk worked on 1.2 Gbytes/seconds

64

Re: What DB to select?

AlekseySQL wrote:

Therefore it is not clear, why the disk worked on 1.2 Gbytes/seconds

That you the difficult such...
If the allegory with the manual pump is so not clear, we try on another...
SSD small volume work more slowly big because there is a limit of throughput of the flash-memory and the channel of the controler. On each channel the certain memory size so the disk of small volume uses less channels is installed.
For the same reason, on one-continuous calls, throughput SSD can be noticeable more low, than for multi-threaded scenarios.
From all above enumerated, it is possible (in your case - it is necessary) to draw one simple output: restrictions in a date transmission chain operate by a principle "the most feeble link" - any component and resultant speed can restrict throughput cannot be above, than at the slowest element.
P.S.
I hope, you do not calculate what I will in details disassemble specifications of the specific device?

65

Re: What DB to select?

Basil A. Sidorov wrote:

SSD small volume work more slowly big because there is a limit of throughput of the flash-memory and the channel of the controler. On each channel the certain memory size so the disk of small volume uses less channels is installed.

In article it is used SSD Patriot Hellfire in capacity 240  which testing showed 2,6 Gbytes/sek on record and 1,5 Gbytes/sek on reading. So matter is not in unsuccessful implementation SSD of a disk which could not use all channels.

Basil A. Sidorov wrote:

For the same reason, on one-continuous calls, throughput SSD can be noticeable more low, than for multi-threaded scenarios.

Character of loading the same, as for PSI Express 3.0, allowed to carry out transmission of 1.7 GBYTES/SECONDS So and not in character business.

66

Re: What DB to select?

AlekseySQL wrote:

In article it is used SSD Patriot Hellfire in capacity 240  which testing showed 2,6 Gbytes/sek on record and 1,5 Gbytes/sek on reading.

it is good, I esteem for you:

iXBT wrote:

Testing Hellfire, we paid attention that the maximum speed on serial operations can be "squeezed out" of it only multi-threaded loading so it too should be taken into consideration on the future: theoretical throughput on that and theoretical that the "real" data received in different programs under different scenarios, will depend more not on it, and from these programs and scenarios - in that case, of course, when force majeure circumstances do not prevent.

In tests to which you refer, under the fastest result is (a standard mark): "32T10".
That is the given throughput is reached at loading of the drive from 32 flows and at enough deep water of queue of requests.
To understand "where a bottom", it is necessary to look on  with a mark "1T1", and it (in a second) about gigabyte and a quarter at reading and gigabyte - on record.

67

Re: What DB to select?

AlekseySQL wrote:

it is passed...
The file in the size 352 was copied "in anywhere" about 1 second.

1) I meant  speed of original 50 Gb of a file. Here - approximation - not a variant.
We can catch non-linear effects.
2) Show a cap of a file and the first some lines.

$ head-n 3 file50gb.csv

68

Re: What DB to select?

Dimitry Sibiryakov wrote:

it is passed...
From the RAM to read much more conveniently, than from a disk.

Friends. It seems to me we went a little not to that direction. We achieve fast reading of the table
With SSD but we do not ask ourselves a question and how it actually there got? ? It too
Swung ftp/wget - an ohm or something similar.

69

Re: What DB to select?

AlekseySQL wrote:

it is passed...
In a file of the sizes of 352 MB 6 000 000 lines are allocated, each of which represents  record. In other words, one record in a text format weighs about 60 Byte (and in binary a bit less than ~40 Byte).
I will soon try speed at  storage in a file and here  result.

6 million is a little, to you are not necessary  a DB for this purpose.
Any DBMS consults.
For example, SQLite. Or MySQL.

70

Re: What DB to select?

AlekseySQL wrote:

it is passed...
Please, wait for result of the test at "" storage. Probably all is already made, only is not tested.

Once again, to you are not necessary  a DBMS. They are necessary at data size of the task from billion records.
At you even 10 million is not present. On 3 orders it is less. Indexes and a cache easily to you make all your reading fast.
Any modern DBMS does it. Any copes with such small volume. Only it is necessary to put a head and hands
To designing of a DB and writing of requests.

71

Re: What DB to select?

AlekseySQL wrote:

Please, wait for result of the test at "" storage. Probably all is already made, only is not tested.

At me is 1 HDD and 1 SSD - disks. At conversion from txt - a format in bin - a format record was produced only on HDD, and after successful end the data was copied on SSD (not to reduce resource SSD unsuccessful tests). Data reading from bin - files it was produced 4  by flows (3 flows read with SSD and 1 flow read with HDD). The analysis of repetition was produced For each field with the previous value (or its increment) at the expense of what it was possible to fulfill data compression.
AOS (array of struct):
The size  bin - files (there is no compression): 23.3
Write time bin - files (only on HDD): 290
Read time bin - files (with HDD + SSD): 89

SOA (struct of array):

The size  bin - files (there is a compression): 6.8
Write time bin - files (only on HDD): 176
Read time bin - files (with HDD + SSD): 11.5
It is visible that the size of files decreased ~3 times, and speed of reading grew ~8 times. There was it because not all columns were read from a disk, and some have been passed with the help seekg (). I am happy with result.

72

Re: What DB to select?

Buy two units of storage on 32, and the problem is solved.
You want - make a RAM-disk, you want directly all data on address space display. .

73

Re: What DB to select?

AlekseySQL wrote:

not to reduce resource SSD unsuccessful tests

This superfluous. A resource of modern SSD on record 1 + a petabyte, i.e. yours 33 Gb it is possible to write down 30 + thousand times.

74

Re: What DB to select?

13th wrote:

Buy two units of storage on 32, and the problem is solved.
You want - make a RAM-disk, you want directly all data on address space display. .

I think that this task is not necessary similar expenditures.
By the way I consider actual a question on the single-valued or repeated reading of a file.
How many generally "run" it is necessary to make on a file to solve the task of the author?

75

Re: What DB to select?

mayton wrote:

it is passed...
I think that this task is not necessary similar expenditures.
By the way I consider actual a question on the single-valued or repeated reading of a file.
How many generally "run" it is necessary to make on a file to solve the task of the author?

At the moment files only are prepared for the subsequent algorithms. Therefore while I put experiments on the single-valued run on a file.