1

Topic: By operation with flows used storage STRONGLY grows

There are at me files dataful on months. I decided to regroup them on clients, i.e. to create files which store the data for all period for each separate client.
At first I, processing every month the initial data, opened and closed a file of the client with adding new in the end:

File-> open (FileName, std:: ios:: out | std:: ios:: binary | std:: ios:: app);

But it worked very long and I decided to open once a file of the client, after each month to do flush (), and to close only when on the client in flow of month there were no transactions. For this purpose I created such object:

using closerWriteFile = std:: function <void (std:: ofstream *)>;
using uptr_ofstream = std:: unique_ptr <std:: ofstream, closerWriteFile>;
uptr_ofstream OpenBinFileForWrite (cstring& FileName)
{
closerWriteFile&& Closer = std:: bind (CloseFileForWrite, _1, FileName, true);
uptr_ofstream File (new std:: ofstream, std:: move (Closer));
File-> open (FileName, std:: ios:: out | std:: ios:: binary | std:: ios:: trunc);//NOW AT DISCOVERY WE CLEAR THE FILE OF THE CLIENT
return std:: move (File);
}
void CloseFileForWrite (std:: ofstream* File)
{
File-> close ();
}
class OpenFiles
{
std:: map <string, uptr_ofstream> Files;
std:: map <string, uint32> Monthes;
mutable std:: mutex MutexFiles;
public:
uptr_ofstream&& Get (cstring& FileName, cuint32 CurrentMonth)
{
std:: lock_guard <std:: mutex> Guard (MutexFiles);
//remember working with data
Monthes [FileName] = CurrentMonth;
//initial find file in map
std:: map <string, uptr_ofstream>:: iterator it;
it = Files.find (FileName);
if (it! = Files.end ()) return std:: move (it-> second);
//add to map
Files [FileName] = OpenBinFileForWrite (FileName);
return std:: move (Files [FileName]);
}
void CloseOldFiles (cuint32 CurrentMonth)
{
if (CurrentMonth! = Consistency. GetOldestMonth ()) return;
std:: lock_guard <std:: mutex> Guard (MutexFiles);
std:: map <string, uptr_ofstream>:: const_iterator it;
for (const auto& Value: Monthes)
{
if (Value.second> = CurrentMonth) continue;
//find files
it = Files.find (Value.first);
if (it == Files.end ()) continue;
//delete this file from map
Files.erase (it);
}
}
} CurrentFiles;

Now I can write such function  to files:

void WriteDataForClient (const myLib::array<writeData>& ArrayOfClient;
const uint32 CurrentMonth;
const string& Client)
{
const string FileName = Settings.at ("SubDirectoryWithOutputFiles") + Client + ".bin";
//find file
uptr_ofstream&& File = CurrentFiles. Get (FileName, CurrentMonth);
//write file
ArrayOfClient. WriteArrayToBinFile (std:: move (File));//THE DATA HERE IS ADDED
//flush added data
File-> flush ();
}

Method CloseOldFiles (cuint32 CurrentMonth) is caused after handling of each month (files thus are closed that is visible in a debugger).
But why - that very strongly grows storage consumption. After all logically I do flush (), and all data should be dropped on a disk. In what a jamb?

2

Re: By operation with flows used storage STRONGLY grows

Get to itself a DB.

3

Re: By operation with flows used storage STRONGLY grows

a guest wrote:

Get to itself a DB.

Any DB - is an overhead charge which decelerate operation (I I work on a house computer and I do not have separate server).

4

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

Any DB - is an overhead charge which decelerate operation (I I work on a house computer and I do not have separate server).

Before to declare operation deceleration, whether it would be necessary to lead a benchmarking and to clarify so it actually, say, in a case a DB file-server (Access, SQLite...), and whether it is exact to you productivity in 1000000 operations of record-reading in a second is necessary.
After all so it is possible and to FVMas' to reach, and there all badly ended smile))
Use SQLite as a variant, building in application as statics-libu. It is possible to make it completely in-memory and then  on a disk when it is necessary, I do not think that the data at you so many that are not located in the modern volume RAM.
Well or washed down the LSM-tree

5

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

But why - that very strongly grows storage consumption.

At any problems with storage to you helps DrMemory. Very much I recommend.

6

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

But why - that very strongly grows storage consumption. After all logically I do flush (), and all data should be dropped on a disk. In what a jamb?

Files here at anything. The data is written to the buffer in the size 4-8  and on buffer filling is dropped on a disk, i.e. flush () generally does not influence in any way on the storage expenditure, it simply command forcedly to write down from the buffer on a disk without waiting fillings. An additional brake protecting from loss of the data in case of program embarkation.

7

Re: By operation with flows used storage STRONGLY grows

Dima T wrote:

Files here at anything. The data is written to the buffer in the size 4-8  and on buffer filling is dropped on a disk, i.e. flush () generally does not influence in any way on the storage expenditure, it simply command forcedly to write down from the buffer on a disk without waiting fillings. An additional brake protecting from loss of the data in case of program embarkation.

I will disagree: I wrote down in due time files on a disk without a method call flush (), simply closing them (forced thus becomes flush ()). And so file closing was o-o-very long business, and as a result I transferred it to a separate asynchronous flow to raise productivity. I do not think that these brakes were from - for records 4  (and remaining 100-500 MB of a file were written rather quickly).

8

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

it is passed...
I will disagree: I wrote down in due time files on a disk without a method call flush (), simply closing them (forced thus becomes flush ()). And so file closing was o-o-very long business, and as a result I transferred it to a separate asynchronous flow to raise productivity. I do not think that these brakes were from - for records 4  (and remaining 100-500 MB of a file were written rather quickly).

OS caches disk writing, it under it involves the free storage, but not storage of process. In process the small buffer if you do not trust - is source codes, here one of implementations .
You as defined the raised expenditure of storage: your process takes away a lot of storage or as a whole is occupied much?

9

Re: By operation with flows used storage STRONGLY grows

Dima T wrote:

OS caches disk writing, it under it involves the free storage, but not storage of process. In process the small buffer if you do not trust - is source codes, here one of implementations .
You as defined the raised expenditure of storage: your process takes away a lot of storage or as a whole is occupied much?

Process consumes 7  (at conversion of source files in the size 5 ). But after end of handling storage why - that is not liberated, so I will search for an error: any objects do not cause the .

10

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

I will search for an error

To listen to strangers of councils and to use DrMemory you you refuse on ideological or
To religious motives?

11

Re: By operation with flows used storage STRONGLY grows

All thanks, the error is found!
There was all the matter is that in structure with which I filled an array, the field with type std:: string has been added. It approximately three times increased the size of structure and as a consequence - the volume of consumed storage three times increased. Therefore the part of storage of the beginning  on a disk and program operation was strongly decelerated.
I will test tomorrow that faster: constant discovery / closing of files with  or storage of open files in collections and usage flush ().

12

Re: By operation with flows used storage STRONGLY grows

AlekseySQL wrote:

I will test tomorrow that faster: constant discovery / closing of files with  or storage of open files in collections and usage flush ().

Faster the second, and even faster the second without flush ()

13

Re: By operation with flows used storage STRONGLY grows

Dimitry Sibiryakov wrote:

to Listen to strangers of councils and to use DrMemory you you refuse on ideological or
To religious motives?

Thanks for councils.
By means of Vtune Amplifier I analyzed selection and storage clearing. It appeared that no leaks are present. In other words it not technical, but a logical error. Therefore automatic tools do not help.

14

Re: By operation with flows used storage STRONGLY grows

Dima T wrote:

Faster the second, and even faster the second without flush ()

If flows really write on 4 , then indeed. Tomorrow I will check up.

15

Re: By operation with flows used storage STRONGLY grows

It turned out that operation with discovery / closing of a file - the fastest. Most likely the matter is that an overhead charge for service of a collection with open flows above, than an overhead charge for discovery and closing of files.
And so the code is easier. As this code is fulfilled only once, I for simplicity.

16

Re: By operation with flows used storage STRONGLY grows

Dimitry Sibiryakov wrote:

it is passed...
At any problems with storage to you helps DrMemory. Very much I recommend.

They can catch:
1. Buffer overflow
2. Usage after free
?
And for limits of a stack array or a global array it finds outputs?

17

Re: By operation with flows used storage STRONGLY grows

And with Valgrind did not compare it?

18

Re: By operation with flows used storage STRONGLY grows

tip78 wrote:

they can catch:

Yes. And still leaks, the non-initialized reading, not released resources etc.
Comparing in Valgrindom is on a site. But personally for me Valgrinda have a deadly
The lack - does not work under Windows.

19

Re: By operation with flows used storage STRONGLY grows

And the small chest simply opened.
Missed. smile))

20

Re: By operation with flows used storage STRONGLY grows

tip78 wrote:

and with Valgrind did not compare it?

At me from Qt Creator "the Analyzer of storage Valgrind" wrote an error: "it was not possible to launch the program. A way or the rights are inadmissible?"
So I quickly on Intel -  skipped the tool.