Hacker Newsnew | past | comments | ask | show | jobs | submitlogin

This discussion is, at least it seems, mostly about multiple heterogenous consumers; not homogenous consumers/replicas/horizontal scaling. So, if Slack sends a queue message for every DM that's sent, the difference between having 1 consumer that updates the database and 1 consumer that sends a push notification, versus having 2 consumers that both only update the database.

The idea of having multiple homogenous consumers shouldn't be controversial; that's just horizontal scaling. And, well, at least until a few hours ago I also would have said that the idea of having multiple heterogenous consumers is also uncontroversially bad. But I guess everyone has "their way" of doing things.

Its also important to note that there's a third situation I see somewhat often: maybe call it homogenous delegated consumers, whereby you've got messages like '{"type":"SendDM", "content": {}}'. Or maybe: '{"type":"SendDM", "action": "UpdateDB", "content":{}}'. The consumers are still homogenous, they all run the same code, but they may internally delegate the message to do different things depending on enums within the message. This is pretty ok; its different because at least you'd never have a consumer hit message and be like "I don't want this take it back".

Though I'd caution against it; just understand that its something of a 'hack' to make one queue act like N queues, and that's ok if you're small and have a good grasp on the problem domain. The big issue it will inevitably run into is: some queue message "kinds" will take a lot longer to process than others; and so if you're e.g. overloading a queue to handle both a simple email send and a much more complex asynchronous database update, you'll inevitably get delayed emails. Absolutely inevitable. But, it can work for a time.



Guidelines | FAQ | Lists | API | Security | Legal | Apply to YC | Contact

Search: