1

Edit:

OK, so, people said it is unclear what I am asking. I am asking for feedback on this design. Here is an example user story:

As a group admin on the website I want to be notified when a user in my group uploads a file to the group.

Easiest solution would be that in the code handling the upload, we just directly create an email message in there and send it. However, this seems like it isn't really the appropriate level of separation of concerns, so instead we are thinking to have a separate worker process which does nothing but send notifications. So, the website in the upload code handles receiving the file, extracting some metadata from it (like filename) and writing this to the database. As soon as it is done handling the file upload it then does two things: Writes the details of the notification to be sent (such as subject, filename, etc...) to a dedicated "notification" table and also creates a message in a queue which the notification sending worker process monitors. The entire sequence is shown in the diagram below.

My questions are: Do you see any drawbacks in this design? Is there a better design? The team wants to use Azure Worker Roles, Queues and Table storage. Is it the right call to use these components or is this design unnecessarily complex? Quality attribute requirements are that it is easy to code, easy to maintain, easy to debug at runtime, auditable (history is available of when notifications were sent, etc...), monitor-able. Any other quality attributes you think we should be designing for?

Original:

We are creating a cloud application (in Azure) in which there are at least 2 components. The first is the "source" component (for example a UI / website) in which some action happens or some condition is met that triggers a second component or "worker" to perform some job. These jobs have details or metadata associated with them which we plan to store in Azure Table Storage. Here is the pattern we are considering:

enter image description here

Steps:

  1. Condition for job met.
  2. Source writes job details to table.
  3. Source puts job in queue.

Asynchronously:

  1. Worker accepts job from queue.
  2. Worker Records DateTimeStarted in table.
  3. Queue marks job marked as "in progress".
  4. Worker performs job.
  5. Worker updates table with details (including DateTimeCompleted).
  6. Worker reports completion to queue.
  7. Job deleted from queue.

Please comment and let me know if I have this right, or if there is some better pattern. For example sake, consider the work to be "sending a notification" such as an email whose template fields are filled from the "details" mentioned in the pattern.

  • 3
    I don't see a question here. – Robert Harvey May 16 '14 at 21:56
1
+50

I can't answer whether you should do this or not, but I can try to help with your decision-making process.

First, what you are proposing is a very standard mechanism for asynchronously processing data. So, you are doing something very conventional, and that is good. I use AWS and have done this a lot with SQS and SNS. I can't speak to what Azure offers. AWS gives an example of this pattern for batch processing.

Here are some advantages to this approach:

  • The initial request can be faster because there is less work being done. There is some time for putting the message in the queue, but it is probably faster than saving to disk and sending an email.

  • You can make the notifications more fault tolerant, or at the least, not interfere with the initial request which prompted that notification. If your email service goes down for a day, you can keep the message in the queue and the worker can try it again the next day.

  • It will scale well - both in terms of users, and as new features arise.

There are a few downsides:

  • There is now additional complexity. It would be simpler to just send the email.

  • Correctly handling errors can more complicated. What do you do if you can't send an email? Retry? Delete the message from the queue and forget? This error handling is important, but your use-case decides how sophisticated you need it to be.

  • You probably want another web service handling this, which adds another component. Though, on small-scale projects, I have put the worker thread in the same web service.


Separation of Concerns

You mentioned Separation of Concerns as one reason for your design. Regardless of how you end up architecting this, you should still separate concerns at a class level. Whatever class is receiving this data can call an "notifier" interface. You might have different implementations of this over time: send the email in the same thread, send the email in a different thread without a queue, put the email in a queue. Each time you change your architecture, you keep the servicing code the same.

  • Thanks David! Great answer!! Part of what I am taking from your answer is that you didn't see anything glaringly defective about my design. Let me explain part of it in further detail and let me know what you think. – Infin8Loop May 24 '14 at 1:46
  • The "Table" component in the sequence diagram is something in Azure called "Table Storage". Table storage is like a database table. It isn't a relational database. You can't have joins between tables. In fact, there will be a relational database in this pattern as well. It will hold the users and business data and such, and both the Worker and the Source will use this relational database. – Infin8Loop May 24 '14 at 1:50
  • The "Table" will be used strictly for operational and logging data. Objects stored in the table can have different columns or schema but they are non-hierarchical. Do you think this separation of business data from operational data is appropriate and does it sound like it is appropriate to use a relational store for the business data and this other "table store" for the operational data? Thanks. – Infin8Loop May 24 '14 at 4:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.