As someone who has created quite a few of these mailers, the queue getting stuck on a single piece of mail and hanging indefinitely is incredibly common. As time has gone on my solutions have become simpler and more pragmatic, since additional complexity breads additional problems.
For example, if I was going to design an emailer today:
-Grab the email from a database save it to a file (likely one or several XML files) and place it in an "Outgoing" directory (ye olde file system).
- Then have a process which grabs an atomic lock (only one running at a time!), gets the directory listings, and launches the actual "sender" for every file individually (concurrently).
- When the launcher launches the sender it records the PIDs of the process against the actual emails/XML files internally.
- After a set wait period if any processes are still running, the launcher kills them, and moves the email/XML into a "Failed" directory which we monitor independantly.
- Every email which is sent gets moved to an "Archive" directory by the sender process, and we monitor that to see if no emails have been archived for a long time (e.g. 30 minutes).
You can accomplish the same thing using a database (Outgoing, Achive, and Failed tables), but frankly with so many awesome file system tools already around it doesn't make sense to reinvent that wheel. Plus people intuitively understand that if a file is sitting in the "Outgoing" or "Failed" directories then it hasn't been sent yet (just like your client would!).
I strongly approve. That's how I implemented the backend of my medical records exchange stack.
File system based queues. Point-to-point data interchange, so no concurrency; your notion of imprinted work tasks with PIDs is a good idea.
I used a "pull" model. A thread would take work from one directory and drop into another. Poor man's workflow. Worked great. Super easy to monitor and troubleshoot.
Using Java, implementing the cross platform file locking (so a downstream process wouldn't pull a task before it was ready) took some finesse, a small caveat.
For example, if I was going to design an emailer today:
-Grab the email from a database save it to a file (likely one or several XML files) and place it in an "Outgoing" directory (ye olde file system).
- Then have a process which grabs an atomic lock (only one running at a time!), gets the directory listings, and launches the actual "sender" for every file individually (concurrently).
- When the launcher launches the sender it records the PIDs of the process against the actual emails/XML files internally.
- After a set wait period if any processes are still running, the launcher kills them, and moves the email/XML into a "Failed" directory which we monitor independantly.
- Every email which is sent gets moved to an "Archive" directory by the sender process, and we monitor that to see if no emails have been archived for a long time (e.g. 30 minutes).
You can accomplish the same thing using a database (Outgoing, Achive, and Failed tables), but frankly with so many awesome file system tools already around it doesn't make sense to reinvent that wheel. Plus people intuitively understand that if a file is sitting in the "Outgoing" or "Failed" directories then it hasn't been sent yet (just like your client would!).