Documentation

Feature Flags for Backup Providers

Learn how to use feature flags to prevent downtime by switching to a backup service provider in real time.

With any web application, it's rarely a matter of if an external vendor will have performance issues or go offline so much as when they will. For example, when an application relies entirely on a single email provider, when that provider is down or suffering from degraded performance, the application can't send password resets or other critical emails reliably. That can lead to frustrated users, increased support requests, or countless other problems.

To avoid this, it's often useful to maintain a backup provider that can quickly be enabled for critical services. While there are several ways to manage primary and backup providers, using circuit breaker feature flags can be one of the most efficient solutions because it enables making adjustments between providers in real-time without having to deploy new code or even restart the application.

While this guide focuses on maintaining a permanent backup option, much of this approach works similarly for minimizing or eliminating downtime when swapping back-end services as well. We'll look at an example of how Flipper can be used to determine the right email provider for a few scenarios where a primary provider is completely offline, experiencing degraded performance, or when you just prefer to split email delivery across two services.

Why set up backup providers?

The case in favor of using backup providers revolves around minimizing downtime and interruptions, and so the justification for having a backup provider for critical services is pretty basic. Any external service can become unreachable for a variety of reasons. Downtime. API changes. Bugs. And even going out of business.

Given the relative simplicity of the reasons why any team would want a backup provider, the more useful question here will be to address why someone wouldn't want to set up a backup provider.

Backup providers can provide resilience, but they also create new code paths and complexity. Similarly, if the backup provider isn't tested regularly, there's no guarantee that it will work reliably when you need it most. In some cases, testing the backup provider can be handled with automated tests, but in other cases, some tools may need more manual verification that the backup provider is working as expected.

For example, if we're talking about email backup providers, it's entirely possible that the API request to send the email works, but the DKIM, SPF, or DMARC settings could be out-of-date. So while it would pass a basic automated test, the provider could still fail to get the emails to inboxes. Or if the provider modifies the headers or structure of their emails, that can create unintended delivery consequences as well.

Alternatively, if a backup provider hasn't been used in a while, it's possible the account was locked due to failed payment or an expired credit card. That can't be monitored with automated tests, and that means someone has to be staying on top of billing for a provider the team may be barely using most of the time.

Suffice it to say that while backup providers can be helpful, they do increase the amount of work we need to do in order to ensure we can rely on the backup when the time comes. For larger apps, the case may be easy to justify, but for newer applications or those with smaller audiences, the resiliency may not be worth it.

Where can backup providers help?

We've touched on email, but backup provider can help reduce interruptions in plenty of additional scenarios. For the most part, the best candidates are the services where the end user likely wouldn't even notice the difference–things like email, SMS, payment processors, or CDN's. And when a framework like Rails enables swapping a provider by only changing some configuration values, like with ActionMailer or ActiveJob, that's a good indicator the provider might be better off with a backup as well.

Let's walk through some scenarios where having a backup provider can have a significant impact on reducing service interruptions. As we explore scenarios, we'll assume that the default state is to send 100% of requests to the primary provider. Then each scenario adjusts according to the needs of that scenario.

1. Primary Provider is Fully Offline

When the primary provider is fully offline or otherwise not working at all, having a backup provider comes through big time by making it trivial to switch all traffic to the backup provider for as long as needed. This is the simplest and most critical benefit of using backup providers, but it's far from the only benefit.

2. Primary Provider has Degraded Performance

In other cases, the primary provider may not be entirely broken but may be experiencing delays or other types of intermittent issues that aren't quite show-stoppers. In those cases, we may want to adjust which provider receives which types of traffic based on more nuanced decisions.

For example, if our primary email provider is experiencing sending delays of five or ten minutes, all emails are still being delivered—eventually. But for critical and time-sensitive emails like password resets, that delay could significantly increase support requests and/or dissatisfaction. So in that case, we may only want to re-route critical emails to the backup provider.

Alternatively, if our application has a freemium model, we may be alright leaving free customers with the primary provider and instead sending only our paying customers to the backup provider. This can be especially true if the backup provider has significantly higher costs.

3. Balance Traffic Across Providers

Sometimes backup providers aren't just about being the last line of defense. We can also use backup providers in order to spread the workload across multiple providers in order to not entirely beholden to the whims of a single provider. The added bonus with this approach means that all of the providers are always in rotation, and that means we can be more confident if we every need to temporarily shift all of the traffic to one provider or the other.

Using Rails with Multiple Email Providers

Before we get into the specifics of using Flipper to dynamically choose an email provider at delivery time, let's get the lay of the land for configuring email providers in Rails. We have several factors to consider, and we'll also discuss the pros and cons of relying purely on Rails for switching providers vs. using a solution like Flipper.

Credentials & Settings

The latest versions of rails offer encrypted credentials for storing secure configuration values as well as unencrypted YAML for custom configuration values that can be loaded using Rails::Application.config_for to convert it from YAML into a Ruby hash. With these options, we can define everything we need to use multiple email providers, but we can't easily switch between providers at runtime without a little more work.

Also, we'd have to restart the application in order to pick up configuration-only changes. That's a little less convenient, but it's still better than having to wait for the primary provider to recover to 100%.

Environment Variables

Rails can also access environment variables like ENV["email_back_provider"] and use those to change values, but that too requires the application to be reloaded. The upside is that some hosting providers that make it easy to specify environment variables via a web interface would make it relatively simple to update an environment variable value. Since they (usually) automatically reload the application after changes to environment variables, it would just be a matter of specifying a different value, and the application would restart automatically.

Environment Configuration Files

So far, we've looked at ways to store and access values related to configuring the providers, but we haven't looked at how we set those values within our codebase. For the most part, these should already be at least somewhat familiar to anyone who has worked with Rails.

Rails also provides environment configuration files so we can specify settings on a per-environment basis. The configuration for application.rb, development.rb, test.rb, production.rb, and other environments could be used to determine a provider, but that limits changes to when the application is loaded. So we'd still have to ensure our application restarts in order to pick up changes.

Moreover, we're not looking to make environment-specific adjustments when it comes to backup providers because all of the value of having a backup provider flows directly to production. So logically, the environment configuration files aren't the ideal place to manage which provider we're using.

ApplicationMailer

If we'd like to be able to dynamically determine an email provider, the simplest solution involves updating our ApplicationMailer class so that it choose the provider automatically. By adding a before_action callback, we could design it choose the sending provider at the last moment.

class ApplicationMailer < ActionMailer::Base
  before_action :assign_email_provider

  # ...

  private

  def assign_email_provider
    self.smtp_settings = if Flipper.enabled?(:backup_email_provider)
    { # ... Backup Provider Settings }     
    else
    { # ... Primary Provider Settings }     
    end
  end
end

For each of those settings, we can update any of the relevant ActionMailer settings. It's not the exact implementation I'd choose in the long run, but it would work. We'll look at a more complete and well-designed solution shortly.

Individual Mailers & Emails

In addition to selecting the email provider entirely through ApplicationMailer, we can also choose the provider for individual mailers or even specify unique options for each individual email. To override the provider for an entire mailer, we could use a similar approach as above from the ApplicationMailer example, but to override the sender for individual emails, we'd have to dynamically override the delivery options.

Flipping Providers Efficiently

While the built-in options from Rails can work, most would require at least a restart of the application in order to pick up the new settings. Similarly, these approaches are mostly all-or-nothing. The options that don't require a restart would require a significant amount of code to be sprinkled around the mailers since they can't easily distinguish which provider to use on their own. With Flipper, however, we can perform the relevant changes in a variety of ways in real time. Plus we can design everything to be well-contained and easier to follow.

While the earlier ApplicationMailer is pretty close conceptually, it would be nice to keep the Flipper.enabled? checks to a minimum. And if they can be tucked away in a class, even better. So if we wanted to update the application mailer, we need to specify :delivery_method and :<delivery_method>_settings (ex. :smtp_settings, :sendmail_settings, etc.)

class ApplicationMailer < ActionMailer::Base
  before_action :assign_email_provider

  default from: "from@example.com"
  layout "mailer"

  private

  def assign_email_provider
    # The logic to know which email provider to use is entirely in
    #   our EmailProvider class. This approach ignores considering
    #   whether the specific recipient or email should be routed
    #   routed differently.
    provider = EmailProvider.current

    # ex. :smtp, :sendmail, :file, :test, etc.
    self.delivery_method = provider.delivery_method.to_sym

    # Get name of delivery-method-dependent settings attribute 
    #   assignment. 
    # ex. .smtp_settings=, .sendmail_settings=, .file_settings=
    settings_method = "#{provider.delivery_method}_settings="

    # Assign provider settings to relevant settings attribute
    self.public_send(settings_method.to_sym, provider.settings)
  end
end

Then we could design our EmailProvider to retrieve settings from credentials and set it up so that EmailProvider.current will always give us the relevant email provider. There's endless ways to approach this, but here's an example with some tests to help serve as a starting point that can be customized for your application. (An example of the credentials file format is included at the end for reference.)

class EmailProvider
  # It's convenient if we're able to directly compare 
  #   providers for testing and related logic
  include Comparable

  attr_accessor :delivery_method, :settings

  # Initialize the two values that we need to configure
  #   the delivery method for our mailers.
  def initialize(delivery_method:, settings:)
    @delivery_method = delivery_method
    @settings = settings
  end

  # We can use the hash to check for equality, but
  #   it also streamlines debugging
  def to_h
    {
      delivery_method: delivery_method,
      settings: settings
    }
  end

  def <=>(other)
    self.to_h <=> other.to_h
  end

  class << self
    # Designed to support checking specific ators, but
    #   our current logic doesn't take advantage of it.
    def current(...)
      if Flipper.enabled?(:backup_email_provider, ...)
    secondary
    else
        primary
      end
    end

    def primary
      new(**settings_for(:primary))
    end

    def secondary
      new(**settings_for(:secondary))
    end

    def settings_for(email_provider)
      # Loading the values from credentials the same way 
      #   that they'd be used to configure mailers makes it nice
      Rails.application.credentials.dig(
        :email_provider, 
        email_provider.to_sym
      )
    end
  end
end

The other advantage to encapsulating our email provider logic is that we can focus our tests primarily on ensuring that EmailProvider.current always gives us what we expect. This comes in handy since testing mailers will set delivery_method to :test. So we can't simply test that a mailer instance's delivery method corresponds to the correct provider. So with the encapsulation, we can be very thorough in terms of testing the provider switching, but then we can focus a test mailer on running once with the backup provider enabled and once with it disabled will give us the test coverage we need on the ApplicationMailer object.

require "test_helper"

class EmailProviderTest < ActiveSupport::TestCase
  User = Struct.new(:id) do
    def flipper_id
      "User;#{id}"
    end
  end

  setup do
    @user = User.new(1)
  end

  test "defaults to primary provider" do
    Flipper.disable(:backup_email_provider)
    assert_equal EmailProvider.primary, EmailProvider.current
  end

  test "uses secondary provider when enable" do
    Flipper.enable(:backup_email_provider)
    assert_equal EmailProvider.secondary, EmailProvider.current
  end

  test "supports checking against actors" do
    Flipper.disable(:backup_email_provider)
    assert_equal EmailProvider.primary,
      EmailProvider.current(@user)

    Flipper.enable(:backup_email_provider)
    assert_equal EmailProvider.secondary,
      EmailProvider.current(@user)
  end

  test "uses correct providers for users individual actor" do
    other_user = User.new(0)
    Flipper.enable(:backup_email_provider, @user)
    assert_equal EmailProvider.primary,
      EmailProvider.current(other_user)
    assert_equal EmailProvider.secondary, 
      EmailProvider.current(@user)
  end
end

An example of the values in the credentials file. These correlate directly to the ActionMailer configuration options for the various delivery methods.

email_provider:
  primary:
    delivery_method: 'smtp'
    settings:
      address:         'smtp.primary.example.com'
      domain:          'example.com'
      user_name:       '<username>'
      password:        '<password>'
      authentication:  'plain'
      enable_starttls: 'true'
  secondary:
    delivery_method: 'sendmail'
    settings:
      location:  '/usr/sbin/sendmail'
      arguments:
        - '-i'

A stock mailer for testing that the ApplicationMailer before_action doesn't cause any errors.

class BackupsMailer < ApplicationMailer
  def example
    @greeting = "Hi"

    mail to: "to@example.org"
  end
end

We test the example mailer once with the flag enabled and once with the flag disabled. The test will still use the :test delivery method, but ensuring that it succeeds with each path gives us coverage while relying primarily on our EmailProvider tests for the underlying logic.

class BackupsMailerTest < ActionMailer::TestCase
  test "primary" do
    Flipper.disable(:email_backup_provider)
    mail = BackupsMailer.example
    assert_equal "Primary", mail.subject
    assert_equal ["to@example.org"], mail.to
    assert_equal ["from@example.com"], mail.from
    assert_match "Hi", mail.body.encoded
  end

  test "primary via backup provider" do
    Flipper.enable(:email_backup_provider)
    mail = BackupsMailer.example
    assert_equal "Primary", mail.subject
    assert_equal ["to@example.org"], mail.to
    assert_equal ["from@example.com"], mail.from
    assert_match "Hi", mail.body.encoded
  end
end

With this logic, it simplifies the process of switching to a backup email provider, but the same concepts will work with most back-end services. While this implementation would only work with the flag fully-enabled, fully-disabled, or enabled for a percentage of time, it could be extended to take the recipient into account based the recipient's attributes that determine whether they should receive priority treatment or not.

Keep in mind that emails with multiple recipients could lead to scenarios where some recipient should receive priority treatment and some shouldn't. So in those cases, it can be handy to choose the mail server based on if any recipients should receive priority treatment.