Today's Insights:

1. Introduction to Application Interarction Patterns

2. What is a Webhook?

3. What is Polling?

4. What is Publish/Subscribe?

5. Industry Example from discord.com

Webhooks, Polling, Pub/Sub:
Which to Use?

In the rapidly evolving world of software development, the ability of applications to communicate effectively remains a cornerstone of successful technology strategies. Whether it's updating data in real-time, reducing server load, or maintaining system scalability, choosing the right interaction pattern can make a significant difference. This issue of our newsletter delves into three primary methods of application interaction: Webhooks, Polling, and Publish/Subscribe (Pub/Sub). Each of these patterns offers distinct advantages and challenges, making them suitable for different scenarios. By understanding these methods, developers and architects can make informed decisions that optimize performance and efficiency in their projects. Let’s explore how these technologies work, their use cases, and weigh their pros and cons to better grasp their impact on modern software solutions.

What is a Webhook?

A webhook is an HTTP callback that is triggered by specific events within a web application or server. It allows web apps to send real-time data to other applications or services as soon as an event occurs. The basic concept of a webhook involves setting up an endpoint (URL) to receive HTTP POST requests. When a specified event happens, the source application makes an HTTP request to the endpoint configured with the webhook, sending data immediately related to the event.

Typical Use Cases and Examples

Webhooks are commonly used to integrate different applications or services. For instance, a webhook might automatically notify a payment gateway to release funds when a transaction is marked as 'complete' in an e-commerce platform. Another example is triggering an email or SMS notification when a new user signs up on a website.

Advantages and Disadvantages of Webhooks

Webhooks offer the significant advantage of real-time communication, enabling immediate data transfer that ensures systems are updated without delay, thus enhancing responsiveness and operational efficiency by eliminating the need for frequent polling. However, they depend heavily on the availability of the receiver's system to handle requests at the time of the event. This reliance can pose a risk if the receiving system experiences downtime or connectivity issues, potentially leading to data loss or delays. Furthermore, implementing webhooks can increase the complexity of a system’s architecture and lead to higher server loads, as they necessitate continuous readiness to accept and process incoming HTTP requests.

What is Polling?

Polling is a communication protocol in which a client repeatedly sends HTTP requests to a server to check for updates at regular intervals. This technique is used when a client needs to stay informed about changes without the server actively notifying it. The basic concept of polling involves the client periodically sending a request to the server to inquire if new data or updates are available.

Typical Use Cases and Examples

Polling is commonly used in scenarios where real-time updates are not critical but timely information is still necessary. For example, an application may poll a server every few minutes to check for updates in user status or to retrieve new emails. Another typical use case is in dashboard applications that need to display the latest data, such as traffic or weather conditions, where updates are fetched at set intervals.

Advantages and Disadvantages of Polling

Polling offers the advantage of simplicity and control over polling frequency, making it relatively easy to implement and adjust based on specific needs, which is ideal for scenarios where high sophistication in real-time updates isn’t crucial. However, it can be quite inefficient as it involves making repeated requests that may not always retrieve new data, leading to unnecessary data traffic and increased server load. Furthermore, the delayed updates due to the interval between polls can make it unsuitable for applications that require instant data synchronization. This method also tends to increase the server load, especially during peak times, which might affect overall system performance.

What is Publish/Subscribe (Pub/Sub)?

Publish/Subscribe, or Pub/Sub, is a messaging pattern where messages are sent by publishers to topics, instead of directly to receivers. Subscribers listen to specific topics and receive messages asynchronously as they are published. The primary concept of Pub/Sub is to decouple the production of information from its consumption, ensuring that publishers and subscribers are independent of each other.

Typical Use Cases and Examples

Pub/Sub is widely used in scenarios where messages need to be distributed to multiple consumers asynchronously. For instance, in real-time chat applications, messages can be published to a topic and all subscribers to that topic receive the messages immediately. It's also used in event-driven architectures, such as when updates in a database should trigger actions in various parts of an application without direct coupling between them.

Advantages and Disadvantages of Publish/Subscribe

Pub/Sub offers the advantage of asynchronous communication and scalability, making it highly effective for systems where the publisher doesn't need to wait for subscriber processes to complete. This model supports a high degree of scalability due to the decoupling of service components and can manage varying loads effectively. However, managing a Pub/Sub system can be complex, especially in large-scale environments where managing topic subscriptions and ensuring message integrity can become challenging. Additionally, since messages are broadcasted to all subscribers indiscriminately, there can be concerns over data redundancy and the efficiency of the system when the number of subscribers is very large. This can lead to increased resource consumption and potential performance bottlenecks.

Industry Example from discord.com

Stanislav Vishnevskiy, CTO and Co-Founder of Discord, explains how the platform utilizes the Publish/Subscribe (Pub/Sub) model to effectively handle the challenges of massive user traffic. In the realm of real-time messaging, Discord showcases an exemplary use of the Publish/Subscribe (Pub/Sub) model to manage massive scale. Operating with over 5 million concurrent users, Discord's infrastructure relies on a Pub/Sub system where messages are published to a "guild" and instantly propagated to all connected users. This model allowed Discord to handle millions of events per second efficiently, despite the challenges of high traffic and data volume. Their implementation emphasizes the scalability and real-time capabilities of Pub/Sub, while innovations like the Manifold and FastGlobal libraries address potential bottlenecks in message distribution and data access, ensuring that the system remains responsive and stable even under extreme loads.

Source: How Discord Scaled Elixir to 5,000,000 Concurrent Users