SpringBoot Allow more control over property overriding for profile-specific files

We have dev, int, and prod environments with matching profiles. These environments share a lot of the same configuration because these environments all live in AWS, so we also have an aws profile. We set spring.active.profiles to dev, int, or prod.

We're trying to use the spring.profiles.group feature:

# application.yaml
spring:
  profiles:
    group:
      dev: aws
      int: aws
      prod: aws

# application-aws.yaml
... # aws-specific configuration

# application-dev.yaml
... # dev-specific configuration

However, spring.profiles.group isn't meeting our needs. In this example, the aws profile will override the environment-specific profile. This overriding is undesirable because we believe that a more specific profile should override a more general profile.

We're aware of the following solutions, but we don't find them ideal either:

Multi-document files

Multi-document files would allow us to control override order by placing the environment-specific profiles at the bottom of the file. This placement would allow the environment-specific profiles to override the aws profile. However, we find multi-document files more difficult to read and maintain, but maybe we're just not used to multi-document files. We've noticed that much of the Spring Boot documentation uses multi-document files. Does Spring Boot encourage use of multi-document files over profile-specific files because multi-document files allow more control in overriding? Multi-document files would address our issue.

spring.active.profiles

We could set spring.active.profiles to aws,dev on application start-up, but ideally, we'd only have to set spring.active.profiles to dev. We also have other profiles, and setting all of these profiles in spring.active.profiles across many applications seems unnecessary.

spring.config.location

It looks like we could use spring.config.location to control override order, but this solution isn't ideal because we'd have to define spring.config.location as an environment property as required by Spring Boot. If we have to set an environment property, we may as well just use spring.active.profiles.

Include environment-specific profiles outside of our packaged JAR file

All of our profiles are packaged within the JAR file as part of our classpath. We don't want to also have to include our environment-specific profiles outside of our JAR files. We'd find it easier to use spring.active.profiles.

Potential solutions

Allow users to disable overriding and throw an exception if they attempt to set the same property twice

Obvious value exists in allowing properties outside of the packaged JAR file to override properties packaged inside the JAR file, but less value exists in property overriding within the JAR file. Users should have full control over application properties within the classpath, so why is property overriding necessary or desired? In many cases, users might unintentionally override their properties. If users could disable overriding for properties packaged inside the JAR file, property load order wouldn't matter.

Place environment-specific profiles in the classpath /config package and the AWS profile in the classpath root

Spring Boot documentation says that files in the classpath /config package should override files in the classpath root, but this functionality doesn't seem to work for us. Maybe because this functionality doesn't apply to profile-specific files or because this functionality doesn't apply to spring.profiles.group. If this functionality worked or a similar functionality existed, it would have an advantage in that we could separate our environment-specific profiles from our more general profiles. We also have other profiles, and it would make a lot of sense to group the environment-specific profiles (ci-cd, dev, int, prod, etc.) separately from the more general profiles (aws, etc.), such as placing them in separate directories.

Comment From: ttddyy

Hi @Slooz

I have a similar issue and I am recommending my team a programmatic approach. Create an EnvironmentPostProcessor which inserts the shared config(application-aws.yaml in this case) to the property sources in the lowest order of the property resolution. This way any env specific configurations override the shared configuration. Also, this solution is independent of profile management, so applications are free to use multi doc, profile, groups, or config location if they need to.

Here is sample impl:

public class MyEnvironmentProcessor implements EnvironmentPostProcessor {

    // Boot injects DeferredLog since logging system is not initialized yet.
    private final Log log;

    public StarterPropertiesEnvironmentProcessor(Log log) {
        this.log = log;
    }

    @Override
    public void postProcessEnvironment(ConfigurableEnvironment environment, SpringApplication application) {
        try {
            ClassPathResource resource = new ClassPathResource("application-aws.yaml");
            YamlPropertySourceLoader loader = new YamlPropertySourceLoader();
            List<PropertySource<?>> props = loader.load("sharedConfig: [application-aws.yaml]", resource);
            for (PropertySource<?> prop : props) {
                environment.getPropertySources().addLast(prop);  // lowest priority
            }
        } catch (IOException ex) {
            // throw exception to stop app or just log and move on...
        }
    }

}

Then, in META-INF/spring.factories

org.springframework.boot.env.EnvironmentPostProcessor=com.example.MyEnvironmentProcessor

Comment From: mbhave

@Slooz Maybe something like this could work

spring:
  profiles:
    group:
      dev: aws,development
      int: aws,integration
      prod: aws,production

Then when the dev profile is active, application-development.yml would take precedence over application-aws.yml.

I am also not sure why classpath:/config/ doesn't work for you. If you could provide a small sample that replicates that, that would be helpful.

Comment From: philwebb

Another option might be to use spring.config.import to import your aws config directly from your application-<profile>.yaml. You'd probably still need multi-document files to get the ordering correct, but they'd be easier to maintain than putting everything in one file.

E.g. application-dev.yaml

spring:
  config:
    import: aws-common.yaml
---
# dev specific configuration

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: Slooz

Thank you for your replies.

@ttddyy, we'd prefer not to use a programmatic approach. We believe simpler solutions exist.

@mbhave,

We're going to use your proposed solution, but with a modification. dev: ...,development seems unintuitive, so we're going to go with the following:

spring:
  profiles:
    group:
      dev-profile-group: aws,dev
      int-profile-group: aws,int
      prod-profile-group: aws,prod

This solution still looks a little weird, because dev-profile-group is also a profile, but this solution seems like the cleanest way to allow us to use profile groups while ordering the profiles in our desired way.

It's quite simple to verify that classpath:/config/ doesn't work. I'm not sure if you want me to create a sample project on GitHub, but the following replicates the behavior: application/yaml:

spring:
  profiles:
    group:
      dev: aws

resources/config/dev.yaml:

logging:
  level:
    root: info

resources/aws.yaml:

logging:
  level:
    root: warn

If classpath:/config/ files should always override classpath:/ files, then the dev profile's info logging level should override the aws profile's warn logging level, but Spring Boot uses the warn logging level. Again, maybe because this behavior doesn't apply when using profile groups.

@philwebb, that solution would also work, but spring.config.import isn't ideal because then aws isn't treated as a profile, which prevents us from using aws with the @Profile annotation. Instead of @Profile("aws"), we'd have to use @Profile({"dev", "int", "prod"}).

We still think that allowing users to disable overriding would be the cleanest solution. If we could disable overriding, profile ordering wouldn't matter. I find it difficult to follow some property files. Many developers will just add properties to application.yaml and override properties as needed in profile-specific files. This practices makes property configuration difficult to understand. By throwing an error on overriding, developers would have to think about and decide when and where to set a property. An error would also prevent developers from accidentally overriding a property.

Comment From: mbhave

Thanks for the example, @Slooz.

If classpath:/config/ files should always override classpath:/ files, then the dev profile's info logging level should override the aws profile's warn logging level, but Spring Boot uses the warn logging level.

This is not the case because, by default, classpath:/ and classpath:/config/ are considered to be part of the same "location group". There is an example of how a location group is processed here.

Comment From: Slooz

@mbhave I see. I thought otherwise because of the following: https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.files

Comment From: mbhave

I am going to close this issue because there is a solution that works. I don't think we can throw an exception if the same property is set in multiple files because there can be a valid reason to do depending on how profiles are activated.

Comment From: Slooz

@mbhave @philwebb I don't think Spring Boot should close this issue just because there are valid reasons to override properties. There are also valid reasons for Spring Boot to offer a strict mode that throws an exception if a user sets the same property twice. This would force users to structure their application properties in a way that's easier to understand and maintain as well as prevent users from unintentionally overriding properties. If you don't think this functionality has value or is worth implementing, I'm fine with you closing this issue. Thanks.