Injecting Secrets from Secrets Manager with Dependency Injection

In a recent AWS blog post, Dror Helper talked about using Secrets Manager to manage app configuration. This article is a great start and provides context for using the service for .NET application configuration.

We’ve been using Secrets Manager pretty heavily as we migrate from on premise to AWS and move to a more secure configuration and align with modern best practices such as the 12 Factor Application. Since we’ve leveraged Secrets Manager so heavily, we’ve written an internal open source ( Innersource) project and publish it as a package to our internal NuGet feed. This NuGet package incorporates several best practices such as leveraging .NET dependency injection and caching using the AWSSDK.SecretsManager.Caching library.

Oddly enough, Dror’s post and an update to our library happened to coincide. We had been using a strongly typed object with JSON serialization when retrieved from secrets manager. While this pattern works great for retrieving secrets on demand when they are needed during runtime, it didn’t support the standard application bootstrap process with appsettings.json and storing in the IConfiguration collection which was requested by one of our application teams.

Options Pattern #

Now that you have some background, let’s talk about how we implemented our changes using a couple of methods starting with a simple implementation of the options pattern. In the first code snippet you will see an example from appsettings.json. You can see we are using the standard AWS structure and adding a SecretsManager complex property to the AWS object. The SecretIdentifiers property is custom and contains a list of secrets we want to retrieve and load into our options class.

{
  "AWS": {
    "Region": "us-east-1",
    "SecretsManager": {
      "SecretIdentifiers": ["MySecret"]
    }
  }
}

Here is our options class, as you can see we are following the structure of the SecretsManager property in the appsettings.json.

public class SecretsManagerOptions
{
  public List<string> SecretIdentifiers { get; set; } = new();
}

Don’t forget the get and set methods on the property or the options won’t map the values properly.

In the Program.cs we grab the options from the configuration using the Microsoft.Extensions.Configuration extension Get method put that into a local smOptions variable. We then add that object to the service collection as a singleton, making it available to our application via dependency injection.

SecretsManagerOptions smOptions =
  _configuration.GetSection("AWS:SecretsManager").Get<SecretsManagerOptions>();

serviceCollection.AddSingleton<SecretsManagerOptions>(smOptions);

This is great as it loads the configuration from the configuration file, but it still requires the application to retrieve the actual secrets at runtime when you need them and the options aren’t available via IConfiguration.

Initializing the Configuration Provider #

This limitation lead us to investigate the ConfigurationProvider pattern per Dror’s blog post and the NuGet package Kralizek Secrets Manager. Using both of these as inspiration we set up our ConfigurationProvider, our ConfigurationProviderOptions, various extensions and an all important service factory. I’ll get to the service factory in just a bit.

Starting with the Program.cs file of the app consuming the package, we’ll walk through how we bootstrap our application with IHostBuilder. (This example is from a console application, a web application would be similar.)

builder
  .ConfigureAppConfiguration((config) =>
  {
    config.AddJsonFile($"appsettings.json", false);
    var settings = config.Build();
    var profileName = settings.GetSection("AWS:Profile").Value;
    config.AddSecretsManager(
      new JsonUtilities(),
      new SecretsManagerServiceFactory().Create(profileName),
      options => options.ReadFromConfigurationSection());
  })
  .ConfigureServices((context, services) =>
  {
    ...
  });

We load our appsettings.json file first to make sure our initial configuration. You can continue to layer in environment specific appsettings as you normally would. Right after the appsettings.json is added, we call config.Build() so that we can access the named profile specified in the configuration file (if the consuming app chooses). Whether or not to use a named profile is entirely up to the consuming app.

Next, we leverage our AddSecretsManager extension method from an IConfigurationBuilder extension class. This extension method accepts a JsonUtilities class (used to properly parse our secret after it is retrieved) as well as an instance of our SecretsManagerServiceFactory mentioned before. This factory is responsible for creating instances of an AwsSecretsManagerClient, SecretsCache, and our custom SecretsManagerService.

Using the factory pattern here is important to understand because when configuring the consuming application, the .NET dependency injection process has not yet resolved the critical services we need to retrieve our application configuration from Secrets Manager. We are also passing the named profile because in our enterprise, developers leverage AWS SSO to obtain tokens. The AWS .NET SDK understands the proper credential search hierarchy and resolves the tokens via those named profiles so there is no need to manage AWS credentials by hand or in the code.

An options class is then passed into the AddSecretsManager extension method which instructs the provider which secrets to load into IConfiguration. We can then use the ConfigureServices extension method to wire up the required services for our application.

Loading the Configuration #

Now that we have the basics down, let’s dive a bit deeper so you can see how the AddSecretsManager extension method works to retrieve the secrets and add them to the IConfiguration collection. Since our provider class implements the .NET ConfigurationProvider class we need to override the Load method.

public override void Load()
{
  foreach (var secretId in Options.SecretIdentifiers)
  {
    var secretResult = _secretsManagerService.GetSecretJsonAsync(secretId).Result;
    if (_jsonUtilities.Parse(secretResult))
    {
      var values = _jsonUtilities.ExtractValues(secretId);

      foreach (var (key, value) in values)
      {
        Data[key] = value;
      }
    }
    else
    {
      Data[secretId] = secretResult;
    }
  }
}

We loop through the Secrets listed in our appsettings.json (via our options class) and retrieve them from Secrets Manager via our custom SecretsManagerService. We then use our JsonUtilities class to parse and extract the secret values and put them into the Data dictionary located on the parent ConfigurationProvider class, thereby including our secrets stored in SecretsManager in IConfiguration.

Kralizek NuGet Package #

Let me briefly talk about the Kralizek Secrets Manager package I mentioned above and why I didn’t just use it. First of all, we already had a secrets manager NuGet package in use and we were just looking to expand the functionality. Secondly, it took me a while and multiple iterations to understand what I wanted to implement, Kralizek seemed overly complicated at the beginning and I was looking for a simpler solution. Lastly, I didn’t really like how it injected and expected an Access Key and Secret for authentication. When running in production we use IAM Execution Roles and Instance Profiles so we don’t need to inject the tokens at runtime and as I stated before, our developers generally don’t manage Access Keys and Secrets “by hand”.

All of that said, if an open source solution works in your environment and it is easy to plug in and implmenent, there is no reason to develop your own.

Layering Configuration Sources #

Using the configuration provider is a powerfull tool and allows the layering of configuration from different sources such as different file types, databases, environment variables, command-line arguments and any other sources you can imagine and makes them available to your entire application consistently. Adding Secrets manager to these layers adds additional flexibility AND security in your application.

Conclusion #

Secrets Management is a very important consideration these days as applications move to the cloud and I’m excited to see how our teams leverage this tool as well as some others we are developing for our teams.

Innersource Resources #

If you are interested in Innersource, here are some additional resources.

• Innersource Commons
• Introduction to Innersource
• What is Innersource - Gitlabs