Architecture and Design 101: Resiliency Patterns — Retry Mechanism

In this article, I would like to walk you through how to implement the Retry Mechanism using Resilience4j and Spring Boot.

Resilience4j is an open-source, lightweight Java library that helps you build resilient and fault-tolerant applications.

The Resilience4j Retry module helps you handle transient failures like loss of network connectivity, timeout issues, temporarily unavailable resources, and retry operations that may fail initially when dealing with remote sources. It provides a flexible and configurable way to define retry policies for specific operations or methods.

The retry module comes with various configuration parameters, such as the maximum number of retries, backoff intervals between retries, and the type of exceptions that should trigger a retry mechanism. This enables you to fine-tune the retry behavior based on your specific use cases and requirements.

Retry enables you to handle transient errors efficiently. Transient errors are temporary and usually, the operation is likely to succeed if retried.

If retries are to be applied, the operation must exhibit idempotent characteristics. For instance, if the remote service successfully receives and processes our initial request but encounters an issue while sending the response, a subsequent retry mustn't lead the service to treat the request as new or produce an unexpected error. This consideration is particularly important in scenarios such as money transfers in banking, where the integrity of the transaction is paramount.

Key Features Offered by Resilience4j Retry Module

• Simple Retry: You need to specify both the maximum number of retry attempts and the duration to wait between retries for a given operation.
• Conditional Retry: Incorporate conditional logic to assess whether a retry should be initiated, contingent on the outcome of the preceding retry attempt.
• Retrying based on the Exceptions: Configure the Retry module to retry operations based on specific checked exceptions.
• Fallback Mechanism: Establish a backup plan that will be used if all other attempts at retrying are unsuccessful. This enables you to handle failures gracefully by offering alternative logic or default values.
• Backoff Techniques: Use a variety of backoff techniques to manage the time between retries. Resilience4j Retry supports randomized and exponential interval backoff strategies.

When to use retry?

• Network Operations: When making network requests to external services, transient issues such as timeouts or intermittent connectivity problems can occur.
• Database operations: Database operations may encounter transient issues, such as connection timeouts or deadlocks.
• File System Operations: Reading or writing to a file system may encounter transient issues due to concurrent access or temporary unavailability.
• External Service Integration: Integrating with external services (e.g., payment gateways, third-party APIs) may result in transient failures due to network issues or temporary unavailability of the service.
• Message Queues: Sending messages to and receiving messages from a message broker.
• Concurrency Issues: Concurrent access to shared resources or data structures may lead to race conditions or conflicts.
• Throttling and Rate Limiting: Services enforcing rate limiting or throttling may temporarily reject requests that exceed allowed limits.

Resilience4J Configurations

Below are the various configurations provided by Resilience4J.

https://resilience4j.readme.io/docs/retry

Implementing Retry with Spring Boot and Resilience4j

Let us create a spring boot application to demonstrate the retry pattern. Create a spring boot application using spring initializr with Gradle.

Resilience4j allows implementation in both programmatic and configuration-based approaches, providing flexibility for developers to choose the method that best fits their requirements.

Below are the steps involved in implementing Retry pattern.

Step 1: Add the below resilience4j dependency to your build.gradle file.

implementation "io.github.resilience4j:resilience4j-spring-boot3:2.1.0"

Step 2: Define retry configurations for each instance. Below is the sample configuration.

Below is the simple configuration.

resilience4j:
  retry:
    instances:
      getCustomerDetailsByCustomerId:
        maxAttempts: 4
        waitDuration: 1s
        retryExceptions:
          - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - com.com.techmonks.resilience.retry.exception.DataNotFoundException

The above Resilience4j Retry configuration sets up a retry mechanism for an operation getCustomerDetailsByCustomerId. The configuration includes the following settings:

• maxAttempts: 4: Specifies the maximum number of retry attempts, which is set to 4 in this case.
• waitDuration: 3sdefines the wait duration between retry attempts, which is set to 1 second.
• retryExceptions: Specifies the exceptions that should trigger a retry. In this configuration, HttpClientErrorException an exception from the Spring Web Client library will trigger a retry.
• ignoreExceptions: Lists the exceptions that should be ignored, meaning that they will not trigger a retry. In this case, the Customer NotFoundException exception will be ignored.

It is a recommended practice to specify retry exceptions and ignore exceptions as part of the configurations. It enables you to retry certain exceptions while ignoring others selectively.

Step 3: Annotate your method with the @Retry annotation.

    @Retry(name = "getCustomerDetailsByCustomerId")
    public Customer getCustomerDetailsByCustomerId(String customerId) throws InterruptedException {
        System.out.println("Retrieve customer Information");
        return this.customerApi.getCustomerById(customerId);
    }

You are ready with the basic configurations needed for retry functionality. Now, let us delve into more detailed configurations.

Retry Examples

Let’s explore the retry examples using the configuration-based approach.

Simple Retry with Specific Exceptions and Ignore others

if you closely observe the above configuration, we enabled the retry only for HttpClientErrorException. Now, let us simulate the exception by passing the invalid URL and verifying whether it is working as intended or not.

In the customerApi, we are triggering the issue by sending the request to an invalid URL.

 private String customerExternalApiUrl = "https://{{guid}}.mockapi.io/api/v1/customers";

 public Customer getCustomerById(String customerId) {
        return webClientBuilder.build().method(HttpMethod.GET)
                .uri(customerExternalApiUrl + customerId)
                .retrieve()
                .bodyToMono(Customer.class)
                .block();
    }

The above code will throw an exception due to missing forward slash in the URL. After running the application, you will see the console log as below.

As you see, the application retried for 4 times as configured for WebClientResponseException. if any other exception throws, it won’t trigger the retry.

Let us update the logic to throw DataNotFoundException and verify the logs.

In the above logs, we see that the retry mechanism is not triggered for DataNotFoundException.

Conditional Retry — Retry on Result and Exception Predicate

With the conditional retry, you can dynamically decide whether the retry should be performed or not based on custom rules.

Retry based on the Result Predicate

To retry based on the result, you need to update the configuration as below.

resilience4j:
  retry:
    instances:
      getCustomerDetailsByCustomerId:
        maxAttempts: 4
        waitDuration: 1s
        retryExceptions:
          - org.springframework.web.reactive.function.client.WebClientResponseException
        ignoreExceptions:
          - com.techmonks.resilience.retry.exception.DataNotFoundException
        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate

After updating the configurations, let us add the Result Predicate logic. Below is the sample implementation.

package com.techmonks.resilience.retry.predicate;

import com.techmonks.resilience.retry.Customer;

import java.util.function.Predicate;

public class CustomerResultPredicate implements Predicate<Customer> {
    @Override
    public boolean test(Customer customer) {
        //TODO: Custom rules goes here
        return customer.getCustomerId().equals("1");
    }
}

After adding the result predicate, let us run the application and verify the results.

When the customer ID matches with “1”, as we defined in the result predicate, retry functionality gets triggered. Predicates provide you with more fine-grained control of the retry functionality based on your application's custom requirements.

Retry based on Exception Predicate

Exception Predicates work very similar to result predicates.

Below is the application configuration.

resilience4j:
  retry:
    instances:
      getCustomerDetailsByCustomerId:
        baseConfig: default
        maxAttempts: 4
        waitDuration: 1s
#        retryExceptions:
#          - org.springframework.web.reactive.function.client.WebClientResponseException
#        ignoreExceptions:
#          - com.techmonks.resilience.retry.exception.DataNotFoundException
        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate

Below is the sample exception predicate.

package com.techmonks.resilience.retry.predicate;

import com.techmonks.resilience.retry.exception.DataNotFoundException;

import java.util.function.Predicate;

public class ExceptionPredicate implements Predicate<Throwable> {
    @Override
    public boolean test(Throwable throwable) {
        System.out.println("Exception predicate triggered.");
        return throwable instanceof DataNotFoundException;
    }
}

By using a custom exception predicate, you can define specific conditions based on the type of exception thrown to determine whether a retry should be performed or not. If the test method returns true, it indicates that a retry should be performed.

Below is the console log after adding the exception predicate.

Retry with an Exponential Backoff

The exponential backoff strategy enables you to configure the incremental duration between each retry attempt. This gives sufficient time for remote service to heal and recover from the issues and enables you to handle issues in a more resilient way.

Below is the sample configuration.

resilience4j:
  retry:
    instances:
      getCustomerDetailsByCustomerId:
        baseConfig: default
        maxAttempts: 4
        waitDuration: 1s
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
#        retryExceptions:
#          - org.springframework.web.reactive.function.client.WebClientResponseException
#        ignoreExceptions:
#          - com.techmonks.resilience.retry.exception.DataNotFoundException
        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate

To implement exponential backoff, two values are defined: an initial wait time and a multiplier. In this approach, the wait time between successive retry attempts grows exponentially due to the specified multiplier. For instance, with an initial wait time of 1 second and a multiplier of 2, the retries would occur at intervals of 1s, 2s, 4s, 8s, 16s, and so forth. This method is particularly recommended when the client operates as a background job or a daemon.

The exponential backoff strategy is frequently employed to mitigate the risk of overwhelming the system and to allow sufficient time for recovery from temporary failures. By progressively extending the wait duration between retry attempts, the retry mechanism enhances resilience and diminishes the probability of exacerbating the existing failures.

Global and Method/Instance Specific Configurations

You can define the global retry configuration by creating a default template that can be easily reused across multiple retry instances. You can override the global configurations at the instance level if you would like to override the default values.

Below is the sample configuration.

resilience4j:
  retry:
    configs:
      default:
        maxAttempts: 3
        waitDuration: 1s
        retryExceptions:
           - org.springframework.web.reactive.function.client.WebClientResponseException
           - org.springframework.web.client.HttpServerErrorException
        ignoreExceptions:
          - com.techmonks.resilience.retry.exception.DataNotFoundException
#        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
#        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate
    instances:
      getCustomerDetailsByCustomerId:
        baseConfig: default
        maxAttempts: 4
        waitDuration: 1s
        enableExponentialBackoff: true
        exponentialBackoffMultiplier: 2
        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
#        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate

Retry with Fallback Mechanism

The fallback mechanism enables you to return the default values when all the retry attempts are failed instead of returning the error. The fallback method should be placed within the same class and have the same method signature as the retrying method, with an additional target exception parameter.

Example

package com.techmonks.resilience.retry;

import com.techmonks.resilience.retry.exception.DataNotFoundException;
import io.github.resilience4j.retry.RetryRegistry;
import io.github.resilience4j.retry.annotation.Retry;
import jakarta.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

@Slf4j
@Service
public class CustomerService {

    private final CustomerApi customerApi;

    private static final String GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID = "getCustomerDetailsByCustomerId";

    private final RetryRegistry retryRegistry;

    public CustomerService(CustomerApi customerApi, RetryRegistry retryRegistry) {
        this.customerApi = customerApi;
        this.retryRegistry = retryRegistry;
    }

    @PostConstruct
    public void postConstruct() {
        io.github.resilience4j.retry.Retry.EventPublisher eventPublisher =
                retryRegistry.retry(GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID).getEventPublisher();
        eventPublisher.onEvent(event -> log.info("Get Customer Details by Customer Id - On Event. Details: " + event));
        eventPublisher.onSuccess(event -> log.info("Get Customer Details by Customer Id - On Success. Details: " + event));
        eventPublisher.onError(event -> log.info("Get Customer Details by Customer Id - On Error. Details: " + event));
        eventPublisher.onRetry(event -> log.info("Get Customer Details by Customer Id - On Retry. Details: " + event));
        eventPublisher.onIgnoredError(event -> log.info("Get Customer Details by Customer Id - On Ignored Error. Details: " + event));
    }

    @Retry(name = GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID, fallbackMethod = "getCustomerDetailsByCustomerIdFallback")
    public Customer getCustomerDetailsByCustomerId(String customerId) {
        log.info("Retrieve customer Information");
        return this.customerApi.getCustomerById(customerId);
    }

    public Customer getCustomerDetailsByCustomerIdFallback(String customerId, DataNotFoundException dataNotFoundException) {
        log.info("Fallback triggered");
        log.info("Exception " + dataNotFoundException.getMessage());
        return new Customer("default", "default");
    }
}

Updated application.yml configurations to trigger the fallback mechanism.

resilience4j:
  retry:
    configs:
      default:
        maxAttempts: 3
        waitDuration: 1s
#        retryExceptions:
#           - org.springframework.web.reactive.function.client.WebClientResponseException
#           - org.springframework.web.client.HttpServerErrorException
#        ignoreExceptions:
#          - com.techmonks.resilience.retry.exception.DataNotFoundException
#        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate
    instances:
      getCustomerDetailsByCustomerId:
        baseConfig: default
        maxAttempts: 4
        waitDuration: 1s
#        enableExponentialBackoff: true
#        exponentialBackoffMultiplier: 2
        resultPredicate: com.techmonks.resilience.retry.predicate.CustomerResultPredicate
#        exceptionPredicate: com.techmonks.resilience.retry.predicate.ExceptionPredicate

In the above code, the fallback method gets triggered when the customerApi throws DataNotFoundException. fallback strategy helps when you would like to return the results from the cache or provide a default behavior. It’s important to note that if there are multiple fallback methods, the one with the closest match to the thrown exception will be invoked.

Below are logs after the fallback implementation.

Retry Event Listeners

Event Listeners publishes multiple events that occur during the retry process. You can write a subscription logic to perform certain actions, such as logging and monitoring.

Example:

package com.techmonks.resilience.retry;

import io.github.resilience4j.retry.RetryRegistry;
import io.github.resilience4j.retry.annotation.Retry;
import jakarta.annotation.PostConstruct;
import lombok.extern.slf4j.Slf4j;
import org.springframework.stereotype.Service;

@Slf4j
@Service
public class CustomerService {

    private final CustomerApi customerApi;

    private static final String GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID = "getCustomerDetailsByCustomerId";

    private final RetryRegistry retryRegistry;

    public CustomerService(CustomerApi customerApi, RetryRegistry retryRegistry) {
        this.customerApi = customerApi;
        this.retryRegistry = retryRegistry;
    }

    @PostConstruct
    public void postConstruct() {
        io.github.resilience4j.retry.Retry.EventPublisher eventPublisher =
                retryRegistry.retry(GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID).getEventPublisher();
        eventPublisher.onEvent(event -> log.info("Get Customer Details by Customer Id - On Event. Event Details: " + event));
        eventPublisher.onError(event -> log.info("Get Customer Details by Customer Id - On Error. Event Details: " + event));
        eventPublisher.onRetry(event -> log.info("Get Customer Details by Customer Id - On Retry. Event Details: " + event));
        eventPublisher.onSuccess(event -> log.info("Get Customer Details by Customer Id - On Success. Event Details: " + event));
        eventPublisher.onIgnoredError(event -> log.info("Get Customer Details by Customer Id - On Ignored Error. Event Details: " + event));
    }

    @Retry(name = GET_CUSTOMER_DETAILS_BY_CUSTOMER_ID)
    public Customer getCustomerDetailsByCustomerId(String customerId) {
        log.info("Retrieve customer Information");
        return this.customerApi.getCustomerById(customerId);
    }
}

In the above implementation, the eventPublisher is obtained from the retryRegistry for the “getCustomerDetailsByCustomerId” retry instance in the postConstruct method.

Below is the list of the events available as part of eventPublisher.

• onEvent: Activates when any event transpires during the retry process.
• onSuccess: Initiates when the retry operation achieves success.
• onError: Activates when an error occurs, signifying the failure of the retry attempt.
• onRetry: Executes when a retry is attempted.
• onIgnoredError: This takes effect when an error event is disregarded as per the configured Retry settings.

Retry Implementation Using Programmatic Approach

Below is an example to define the retry configurations using a programmatic approach.

package com.techmonks.resilience.configuration;

import com.techmonks.resilience.retry.exception.DataNotFoundException;
import io.github.resilience4j.retry.Retry;
import io.github.resilience4j.retry.RetryConfig;
import io.github.resilience4j.retry.RetryRegistry;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.web.client.HttpClientErrorException;
import org.springframework.web.client.HttpServerErrorException;

import java.time.Duration;

@Configuration
public class RetryConfiguration {
    private final RetryRegistry retryRegistry;

    public RetryConfiguration(RetryRegistry retryRegistry) {
        this.retryRegistry = retryRegistry;
    }

    @Bean
    public Retry getCustomerDetailsByCustomerIdRetryConfig() {
        RetryConfig customerDetailsRetryConfig = RetryConfig.custom()
                .maxAttempts(3)
                .waitDuration(Duration.ofSeconds(2))
                .retryExceptions(HttpClientErrorException.class, HttpServerErrorException.class)
                .ignoreExceptions(DataNotFoundException.class)
                .build();

        return retryRegistry.retry("getCustomerDetailsByCustomerId", customerDetailsRetryConfig);
    }
}

The custom configuration is defined using the RetryConfig.custom() builder along with the required configurations. The Retry instance is created using the retryRegistry.retry() method, which takes the name of the Retry instance and the custom configuration as parameters.

Conclusion

Resiliance4J is a very powerful and lightweight package that enables developers to build resilient applications. In this article, we covered the concepts pertinent to the retry module certain extent.

As always, you can find the complete source code on GitHub.

That’s all for today!

Thank you for taking the time to read this article. I hope you have enjoyed it. If you enjoyed it and would like to stay updated on various technology topics, please consider subscribing for more insightful content.

references:

Retry