Circuit Breaker

Description

The circuit breaker is a simple state machine:

circuit-breaker

The circuit breaker starts closed. In this state, the circuit breaker maintains a rolling window of recent invocations. For each invocation, the rolling window tracks whether it finished successfully or failed.

The rolling window must be full to take any state transition decision. For example, if the rolling window has size 10, a closed circuit breaker always allows at least 10 invocations.

If the rolling window contains a number of failures higher than the configured ratio, a closed circuit breaker moves to open. When the circuit breaker is open, invocations are not allowed to proceed. Instead, the circuit breaker fails fast and throws CircuitBreakerOpenException.

For example, if the rolling window has size 10 and the failure rate is 0.5, it means that 5 invocations out of the most recent 10 invocations must fail for the circuit breaker to move to open.

After some time, an open circuit breaker moves to half-open to determine whether failing fast is still appropriate. A half-open circuit breaker allows some configured number of probe attempts to proceed. If all of them succeed, the circuit breaker moves to closed and invocations are allowed again. If some probe invocations fail, the circuit breaker moves back to open and invocations are prevented.

Successes and failures are determined from the method result. More specifically:

  1. If the guarded method returns normally, it is treated as success.

  2. Otherwise, if the guarded method throws an exception whose type is assignable to any of the skipOn types (see skipOn), it is treated as success.

  3. Otherwise, if the guarded method throws an exception whose type is assignable to any of the failOn types (see failOn), it is treated as failure.

  4. Otherwise, it is treated as success.

Lifecycle

Circuit breaker needs to maintain some state between invocations: the number of recent successful invocations, the number of recent failed invocations, and so on. This state is a singleton, irrespective of the lifecycle of the bean that uses the @CircuitBreaker annotation.

More specifically, the circuit breaker state is uniquely identified by the combination of the bean class (java.lang.Class) and the method object (java.lang.reflect.Method) representing the guarded method.

For example, if there’s a guarded method doWork on a bean which is @RequestScoped, each request will have its own instance of the bean, but all invocations of doWork will share the same circuit breaker state.

Interactions with Other Strategies

See How to Use Multiple Strategies for an overview of how fault tolerance strategies are nested.

If @Fallback is used with @CircuitBreaker, the fallback method or handler may be invoked if a CircuitBreakerOpenException is thrown, depending on the fallback configuration.

If @Retry is used with @RateLimit, each retry attempt is processed by the circuit breaker as an independent invocation. If CircuitBreakerOpenException is thrown, the execution may be retried, depending on how retry is configured.

Configuration

There are 7 configuration options, corresponding to the 7 members of the @CircuitBreaker annotation.

requestVolumeThreshold

Type: int

Default: 20

The size of the rolling window. That is, the number of recent consecutive invocations tracked by a closed circuit breaker.

failureRatio

Type: double

Default: 0.5

The ratio of failures in the rolling window that causes a closed circuit breaker to move to open.

delay + delayUnit

Type: long + ChronoUnit

Default: 5000 millis, or 5 seconds

The delay after which an open circuit breaker moves to half-open.

successThreshold

Type: int

Default: 1

The number of probe invocations allowed when the circuit breaker is half-open. If they all succeed, the circuit breaker moves to closed, otherwise it moves back to open.

failOn

Type: Class<? extends Throwable>[]

Default: {Throwable.class}

Set of exception types that the circuit breaker considers failures.

skipOn

Type: Class<? extends Throwable>[]

Default: {}

Set of exception types that the circuit breaker considers success. This configuration takes precedence over failOn.

Metrics

Circuit breaker exposes the following metrics:

Name

ft.circuitbreaker.calls.total

Type

Counter

Unit

None

Description

The number of times the circuit breaker logic was run. This is usually once per method call, but may be more than once if the method call is retried.

Tags

  • method - the fully qualified method name

  • circuitBreakerResult = [success|failure|circuitBreakerOpen] - the result of the method call, as considered by the circuit breaker

    • success - the method ran and was successful

    • failure - the method ran and failed

    • circuitBreakerOpen - the method did not run because the circuit breaker was in open or half-open state

Name

ft.circuitbreaker.state.total

Type

Gauge<Long>

Unit

Nanoseconds

Description

Amount of time the circuit breaker has spent in each state

Tags

  • method - the fully qualified method name

  • state = [open|closed|halfOpen] - the circuit breaker state

Notes

Although this metric is a Gauge, its value increases monotonically.

Name

ft.circuitbreaker.opened.total

Type

Counter

Unit

None

Description

Number of times the circuit breaker has moved from closed state to open state

Tags

  • method - the fully qualified method name

Name

ft.circuitbreaker.state.current
This is an additional feature of SmallRye Fault Tolerance and is not specified by MicroProfile Fault Tolerance.

Type

Gauge<Long> (0 or 1)

Unit

None

Description

Whether the circuit breaker is currently in given state (1) or not (0)

Tags

  • method - the fully qualified method name

  • state = [open|closed|halfOpen] - the circuit breaker state

See the Metrics reference guide for general metrics information.

Extra Features

Circuit Breaker Maintenance

This is an additional feature of SmallRye Fault Tolerance and is not specified by MicroProfile Fault Tolerance.

It is sometimes useful to see the circuit breaker status from within the application, or reset it to the initial state. This is possible in two steps:

  1. Give the circuit breaker a name by annotating the guarded method with @CircuitBreakerName:

    @ApplicationScoped
public class MyService {
    @CircuitBreaker
    @CircuitBreakerName("hello-cb") (1)
    public String hello() {
        ...
    }
}
    1 The circuit breaker guarding the MyService.hello method is given a name hello-cb.

  2. Inject CircuitBreakerMaintenance and call its methods:

    @ApplicationScoped
public class Example {
    @Inject
    CircuitBreakerMaintenance maintenance;

    public void test() {
        System.out.println("Circuit breaker state: "
            + maintenance.currentState("hello-cb")); (1)
        maintenance.resetAll(); (2)
    }
}
    1 Obtains current circuit breaker state.
    2 Resets all circuit breakers to the initial state.

The CircuitBreakerMaintenance interface provides 4 methods:

  1. currentState(name): returns current state of given circuit breaker. The return type CircuitBreakerState is an enum with 3 values: CLOSED, OPEN, HALF_OPEN.

  2. onStateChange(name, callback): registers a callback that will be called when given circuit breaker changes state.

  3. reset(name): resets given circuit breaker to the initial state.

  4. resetAll(): resets all circuit breakers in the application to the initial state.

See the javadoc of those methods for more information.

Inspecting Exception Cause Chains

This is an additional feature of SmallRye Fault Tolerance and is not specified by MicroProfile Fault Tolerance.
This feature only works in the non-compatible mode.

The @CircuitBreaker annotation can specify that certain exceptions should be treated as failures (failOn) and others as successes (skipOn). The specification limits this to inspecting the actual exception that was thrown. However, in many usecases, exceptions are wrapped and the exception the user wants to decide on is only present in the cause chain.

For that reason, in the non-compatible mode, if the actual thrown exception isn’t known failure or known success, SmallRye Fault Tolerance inspects the cause chain. To be specific, in case a @CircuitBreaker method throws an exception, the decision process is:

  1. If the exception is assignable to one of the skipOn exceptions, the circuit breaker treats it as a success.

  2. Otherwise, if the exception is assignable to one of the failOn exceptions, the circuit breaker treats it as a failure.

  3. Otherwise, if the cause chain of the exception contains an exception assignable to one of the skipOn exceptions, the circuit breaker treats it as a success.

  4. Otherwise, if the cause chain of the exception contains an exception assignable to one of the failOn exceptions, the circuit breaker treats it as a failure.

  5. Otherwise, the exception is treated as a success.

For example, say we have this method:

@CircuitBreaker(requestVolumeThreshold = 10,
    skipOn = ExpectedOutcomeException.class,
    failOn = IOException.class)
public Result doSomething() {
    ...
}

If doSomething throws an ExpectedOutcomeException, the circuit breaker treats it as a success. If doSomething throws an IOException, the circuit breaker treats it as a failure. If doSomething throws a WrapperException whose cause is ExpectedOutcomeException, the circuit breaker treats it as a success. If doSomething throws a WrapperException whose cause is IOException, the circuit breaker treats it as a failure.

Comparing with the @CircuitBreaker specification, SmallRye Fault Tolerance inserts 2 more steps into the decision process that inspect the cause chain. Note that these steps are executed if and only if the thrown exception matches neither failOn nor skipOn. If the thrown exception matches either of them, the cause chain is not inspected at all.