1

Topic: To store a state for service + rabbitmq

Hello. There is a certain service, the bus (rabbitq+masstransit) and messages. Initially I assumed that if suddenly service fell, at restarting all raw messages it takes from the bus (since msg ack at us will not be the message anywhere does not get to). Specifying requirements, it was clarified that the organization of interaction at us is constructed as follows - the message that it is necessary to begin some operation comes, I answer that started to process request. Further service falling after which I should recover a state (the processed message () is possible. Queue any more does not store the initial message (was ack.) . Sootv. I should organize unpretentious storage, on type what  service of that processes etc.: id, service id, message body, process status something Can be necessary still. A question: as well as on what similar storage is better for organizing - mysql, nosql. There is something similar ? The task unpretentious, the decision for certain . Not one decade. It would not be desirable to invent a bicycle. In advance I thank.

2

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> Hello. S> There is a certain service, the bus (rabbitq+masstransit) and messages. Initially I assumed that if suddenly service fell, at restarting all raw messages it takes from the bus (since msg ack at us will not be the message anywhere does not get to). S> Specifying requirements, it was clarified that the organization of interaction at us is constructed as follows - the message that it is necessary to begin some operation comes, I answer that started to process request. Further service falling after which I should recover a state (the processed message ()) is possible. Queue any more does not store the initial message (was ack.) . Sootv. I should organize unpretentious storage, on type what  service of that processes etc.: S> id, service id, message body, process status S> something Can be necessary still. S> a question: as well as on what similar storage is better for organizing - mysql, nosql. There is something similar ? The task unpretentious, the decision for certain . Not one decade. It would not be desirable to invent a bicycle. S> in advance I thank. Amazonovsky SQS supports such pattern: - the sender sends the message in queue and forgets - the receiver takes the message from queue. Thus the message is not deleted, and is simply marked invisible to all remaining on specified by the receiver . - It is supposed that for this  the message will be processed by the receiver - if the receiver successfully processes the message it explicitly deletes it from queue - if it falls on what that to the reason - the message becomes visible for all on the outflow  and the receiver can take it on handling as rises I Suggest to check up, whether your bus here supports such pattern (many support). It solves a problem in a root without superfluous databases.

3

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> Amazonovsky SQS supports such pattern: I> - the sender sends the message in queue and forgets I> - the receiver takes the message from queue. Thus the message is not deleted, and is simply marked invisible to all remaining on specified by the receiver . I> - it is supposed that for this  the message will be processed by the receiver I> - if the receiver successfully processes the message it explicitly deletes it from queue I> - if it falls on what that to the reason - the message becomes visible for all on the outflow  and the receiver can take it on handling as rises I> I Suggest to check up, whether your bus here supports such pattern (many support). It solves a problem in a root without superfluous databases. At me other scenario - me comes the message (start opertaion). I answer it (ok, I will start). Further, the come message is removed from queue (I answered it), and its data is used for handling (process operation). The protocol at us such. The scenario above at me does not work.

4

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> At me other scenario - me comes the message (start opertaion). I answer it (ok, I will start). Further, the come message is removed from queue (I answered it), and its data is used for handling (process operation). S> the Protocol at us such. The scenario above at me does not work. I would suggest  to space apart operations "I answered" and "I really started to work".  the first service saved the message where  and answered "ok, I will start", and the second slowly would rake messages from  storages. Storage can be or queue on algorithm above, or the  a database - in this case it will be necessary to enter statuses. If  - one then it is simple enough: rakes all what not "processed". If them a little (flows, processes, machines) - that all becomes more interesting As it is necessary to consider an amount  and something to do with messages which have not been processed some times - that poison message did not break system.

5

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> At me other scenario - me comes the message (start opertaion). I answer it (ok, I will start). Further, the come message is removed from queue (I answered it), and its data is used for handling (process operation). S> the Protocol at us such. The scenario above at me does not work. It is possible not to remove the message from queue, and to answer the sender with other message. In this case to the sender can come ` ok, I will start ` some times.

6

Re: To store a state for service + rabbitmq

Hello, neFormal, you wrote: F> it is possible not to remove the message from queue, and to answer the sender with other message. F> in this case to the sender can come ` ok, I will start ` some times. I.e. to use an initial queue as temporal storage. A variant if the sender agrees to receive more than one ack on one message.

7

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> Hello, neFormal, you wrote: F>> it is possible not to remove the message from queue, and to answer the sender with other message. F>> in this case to the sender can come ` ok, I will start ` some times. I> i.e. to use an initial queue as temporal storage. A variant if the sender agrees to receive more than one ack on one message. I and was going to make - to use queue as storage. But I am afraid that on our protocol two ack not . Though it will be necessary to specify/try.

8

Re: To store a state for service + rabbitmq

Hello, neFormal, you wrote: F> Hello, Sharov, you wrote: S>> At me other scenario - me comes the message (start opertaion). I answer it (ok, I will start). Further, the come message is removed from queue (I answered it), and its data is used for handling (process operation). S>> the Protocol at us such. The scenario above at me does not work. F> it is possible not to remove the message from queue, and to answer the sender with other message. F> in this case to the sender can come ` ok, I will start ` some times. Somehow so (comments) see: var responseQueue = _publishEndpointCache. GetEndpoint ("resp"); _receiveDataBus = BusInitializer. Create ("rcv", sbc => {sbc. SetConcurrentConsumerLimit (10); sbc. SetCreateMissingQueues (true); responseQueue. Send ("received msg"); sbc. Subscribe (subs => subs. Handler <Message> (msg => {var j = new Runner (); j. Consume (msg, report); //I so understand that to those it is time while to here we do not reach, the message will be only accepted, but not ack, i.e. in queue//as soon as we quit this output agent the message will be repeatedly ack and is is final remote})); sbc. Subscribe (subs => subs. Handler <Dummy> (msg => Logger. Info ("Received DUMMY message")));}); My hypotheses in comments are true?

9

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> Hello, Sharov, you wrote: S>> At me other scenario - me comes the message (start opertaion). I answer it (ok, I will start). Further, the come message is removed from queue (I answered it), and its data is used for handling (process operation). S>> the Protocol at us such. The scenario above at me does not work. I> I would suggest  to space apart operations "I answered" and "I really started to work". They also are spaced apart that a problem and generated. I> Toest the first service saved the message where  and answered "ok, I will start", and the second slowly would rake messages from  storages. About structure and internal storage I also ask. I.e. it will be no more, on idea, than one table. It would be desirable to understand - what  to take, and whether is not present already . Decisions where it is possible to look. I> Storage can be or queue on algorithm above, or the  a database - in this case it will be necessary to enter statuses. If  - one then it is simple enough: rakes all what not "processed". If them a little (flows, processes, machines) - that all becomes more interesting And what interesting? At everyone  will be id, and before the beginning of operation he will write in  the uniq. id and the status ("processing"). I> As it is necessary to consider an amount  and something to do with messages which have not been processed some times - that poison message did not break system. Here this excellent remark.

10

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> My hypotheses in comments are true? Unobviously where the message from queue is deleted.

11

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> They also are spaced apart that a problem and generated. Here it is difficult to make comments without knowing a context. S> about structure and internal storage I also ask. I.e. it will be no more, on idea, than one table. It would be desirable to understand - what  to take, and whether is not present already . Decisions where it is possible to look. So who knows, what loading at you, specificity, whether it is necessary to delete the processed messages or to save for history. I>> Storage can be or queue on algorithm above, or the  a database - in this case it will be necessary to enter statuses. If  - one then it is simple enough: rakes all what not "processed". If them a little (flows, processes, machines) - that all becomes more interesting S> And what interesting? At everyone  will be id, and before the beginning of operation he will write in  the uniq. id and the status ("processing"). In this case anything interesting - only it is necessary to work attentively with  But here if one of  "dies for a long time", and the messages "occupied" with it it is necessary to process remaining, or an amount  the dynamic... Then this excellent remark is really interesting S> Here. Therefore that standard decisions is better  generally - in the first such unobvious trifles are already solved.

12

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> But here if one of  "dies for a long time", and these are the messages occupied with it it is necessary to process, or an amount  the dynamic... Then really interesting Here still there is an interesting moment at how many  to noticeable loading: and how many messages can process  parallely that in first to be loaded, in the second - not to be overloaded and not to leave in  because of too big kol-va is competitive processed messages. Here, alas, I did not find typical decisions (and I will be grateful, if throw) and managed bicycles with "potential weight" handlings of the message and "the maximum total of scales", processed on a specific server.

13

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> Hello, Sharov, you wrote: S>> About structure and internal storage I also ask. I.e. it will be no more, on idea, than one table. It would be desirable to understand - what  to take, and whether is not present already . Decisions where it is possible to look. I> so who knows, what loading at you, specificity, whether it is necessary to delete the processed messages or to save for history. Loading will be absolutely operetta. It is necessary to store stories. S>> And what the interesting? At everyone  will be id, and before the beginning of operation he will write in  the uniq. id and the status ("processing"). I> In this case anything interesting - only it is necessary to work attentively with  I> But here if one of  "dies for a long time", and the messages "occupied" with it it is necessary to process remaining, or an amount  the dynamic... Then really interesting One more really important and interesting remark. S>> Here this excellent remark. I> therefore that standard decisions is better  generally - in the first such unobvious trifles are already solved. Here therefore I also asked a question. And judging by kol-in subtleties, it is better to take already ready decision. It is necessary to understand what...

14

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: S>> My hypotheses in comments are true? I> it is unobvious where the message from queue is deleted. The queue manager deletes the message (sends ack.) after the lambda in Hander fulfills.

15

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> Loading will be absolutely operetta. It is necessary to store stories. S> One more really important and interesting remark. If I correctly understood, the case is a slow one-continuous handling of queue from small kol-va messages (less 10 million a year), and thus the initial queue cannot be used for for temporal storage  - I would suggest to add one more table in a current DB (provided that there similar loadings-obemy of the data) and not to be soared. If is not present - that to think over input therefore as they very strongly influence the decision in more details. - whether multi-threaded data handling is necessary - whether enough multi-threaded data handling by one machine - will be how much critical possibility of that that the message is processed twice and results - are re-recorded - how much  operation of handling of the message generally basically S> Here therefore I and asked a question. And judging by kol-in subtleties, it is better to take already ready decision. It is necessary to understand what... Yes here  depends on answers on  above very much even it will be probable that  a bicycle differently more favourably hemorrhoids with what adjustment a thread of the monster from number of "the general-purpose solvers of all problems of integration"

16

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: F>> it is possible not to remove the message from queue, and to answer the sender with other message. F>> in this case to the sender can come ` ok, I will start ` some times. S>//I so understand, what to those it is time while to here we do not reach, the message will be only accepted, but not ack, i.e. in queue S>//as soon as we quit this output agent the message will be repeatedly ack and is is final remote S> My hypotheses in comments are true? I can not tell for certain since I with it worked through wrappers, and they could and catch simply exceptions and do nack. But in docks there is a such: Messages can be returned to the queue using AMQP methods that feature a requeue parameter (basic.recover, basic.reject and basic.nack), or due to a channel closing while holding unacknowledged messages. Any of these scenarios caused messages to be requeued at the back of the queue for RabbitMQ releases earlier than 2.7.0. From RabbitMQ release 2.7.0, messages are always held in the queue in publication order, even in the presence of requeueing or channel closure. https://www.rabbitmq.com/semantics.html I here do not know, when happens ack, therefore without concept that will be if the output agent falls in a crust. After several failures (the amount is underlined in title ` x-retry-count `) the message gets in dead letter exchange (to a word about toxic messages): http://www.rabbitmq.com/dlx.html then all not last messages can be deleted or processed later. And generally I meant that the supplier of messages too has a queue (often with  id) from which he waits answers and acknowledgement. Here it is possible to send the to it ` i will start ` at the handling beginning, and to shift duties on resending and count of attempts on the supplier. But if works redeliver at rupture of connection to ack it will be better.

17

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: I> But here if one of  "dies for a long time", and the messages "occupied" with it it is necessary to process remaining, or an amount  the dynamic... Then it is really interesting if to trust docks the occupied messages after death will be returned in queue. If there is nobody to process generally on this case there is a lifetime  in queue after which it goes in dead letters. About dynamics too there is a section, but I did not get a grasp. Generally, I did not catch any problems with it. Loading was small and heels of output agents normally lived even at deaths door one of them.

18

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> the Question: as well as on what similar storage is better for organizing - mysql, nosql. There is something similar ? The task unpretentious, the decision for certain . Not one decade. It would not be desirable to invent a bicycle. There are some variants. As my preferences: 1) to Throw out on three letters RabbitMQ and simply to make queue of tasks. Working nodes periodically interrogate it, taking away jobs. The taken away jobs thus are marked at once as "in process". The logic with , falling, etc. is elementarily added if needed directly in request "give-me-following-task". Well and simple table MySQL/PostgreSQL will be scaled to hundreds tasks a second. Gourmets can use https://www.postgresql.org/docs/9.6/sta … otify.html or something similar for simplification of loading. 2) if so it would be desirable to eat a cactus we do the message "the task it is received" idempotent and it is combined with a template "the invisible message". I.e. The working node takes the message, without confirming it, and sends on a central node acknowledgement "ja-ja-work". In case of falling (after a while) the message again becomes visible and other node starts it to execute, repeatedly sending "ja-ja-work". In my practice, difficult protocols with a state on messages result only in  so I would begin with 1).

19

Re: To store a state for service + rabbitmq

Hello, neFormal, you wrote: F> if to trust docks the occupied messages after death will be returned in queue. The HARDWARE marked that most likely a little ack on one  in its case does not transit. Therefore input queue usage as temporal storage most likely not . Though that that the sender demands that acknowledgement that its message is accepted - at least atypically. Normally it has enough 200  from queue

20

Re: To store a state for service + rabbitmq

Hello, Cyberax, you wrote: Cs> 1) periodically it interrogate Working nodes, taking away jobs. HERE tell, whether you know standard decisions for loading equalization on working nodes?

21

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: Cs>> 1) periodically it interrogate Working nodes, taking away jobs. I> HERE tell, whether you know standard decisions for loading equalization on working nodes? In a case with poll-model all is simple - the working node knows the loading and it takes simple not new jobs, yet does not receive enough the free capacity. In the elementary case it is enough to look that the number of the active tasks is no more number CPU. Such approach is perfectly combined with heterogeneous clusters (welcome to EC2 Spot Fleet, by the way).

22

Re: To store a state for service + rabbitmq

Hello, itslave, you wrote: Cs>> 1) periodically it interrogate Working nodes, taking away jobs. I> HERE tell, whether you know standard decisions for loading equalization on working nodes? Still, if it would be desirable absolutely a telemarket it is possible to take AWS Batch. He incurs almost all operation on  challenging tasks.

23

Re: To store a state for service + rabbitmq

Hello, Cyberax, you wrote: the C> Such approach is perfectly combined with heterogeneous clusters (welcome to EC2 Spot Fleet, by the way). Thanks, I already without small 5 years as a C> In a case with poll-model all are simple - the working node knows the loading and it takes simple not new jobs, yet does not receive enough the free capacity. In the elementary case it is enough to look that the number of the active tasks is no more number CPU. I that in course. Only I about standard decisions asked -  bicycles is available with a prosperity. And with configuring of an amount of competitive tasks, and with different (simple enough) algorithms by capacity calculation: and on kol-vu processed messages, and on total "weight" of messages ( when  handlings of messages strongly varies), and on loading CPU.... Bothered

24

Re: To store a state for service + rabbitmq

Hello, neFormal, you wrote: F> Hello, itslave, you wrote: I>> But here if one of  "dies for a long time", and the messages "occupied" with it it is necessary to process remaining, or an amount  the dynamic... Then it is really interesting F> if to trust docks the occupied messages after death will be returned in queue. F> if there is nobody to process generally on this case there is a lifetime  in queue after which it goes in dead letters. F> about dynamics too there is a section, but I did not get a grasp. Generally, I did not catch any problems with it. Loading was small and heels of output agents normally lived even at deaths door one of them. In this part of arguing no queue is present - we considered the message from queue, made ack. Now it is a question of storage of messages where it is stored id . Also that will be, if it long time is not accessible. It is possible  to refuse identification, there will be one heap of raw messages on all.

25

Re: To store a state for service + rabbitmq

Hello, Sharov, you wrote: S> In this part of arguing no queue is present - we considered the message from queue, made ack. And what for to do ack? The supplier set the task in queue, received 200 from queue that its message is accepted - and all. On it it  comes to an end. Queue guarantees that delivers the message to the receiver, and that - that processes in due time. Give  more context, can it turns out to do without ack - then we come to simple, standard and checked decisions that in itself it is fine