SpringBoot Override of YML lists in externalized configuration files does not work or is documented wrongly

Consider a @ConfigurationProperties class like the following

@ConfigurationProperties
@Getter
@Setter
@Component
public class MyConfigurationClass {

  private List<String> myProp = new ArrayList<>(Arrays.asList("foo", "bar", "blub")); 

}

The spring boot documentation (https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.typesafe-configuration-properties.merging-complex-types) states that you can override list properties using YML lists or comma-separated lists.

Consider my application.yml

myProp
  - machine
  - articles
  - blub

Now consider a profile file application-dev.yml

myProp
 - machine
 - articles

I would exepect that the list of myProp now holds 2 items: machine and articles.

But this is not true unfortunately. The dev profile is not able to override the list property completly but only replaces the respective indices. The list property still holds 3 items, machine, articles and blub.

This is the logic I use to get the value for the list property:

[...]
                int index = 0;
                while (true) {
                    String listValue = this.configurableSpringEnvironment.getProperty(configProperty.getName().concat("[" + index + "]"));

                    if (listValue == null) {
                        break;
                    } else {
                        if (index > 0) {
                            currentValue.append(",");
                        }
                        currentValue.append(listValue);
                        index += 1;
                    }
                }
                currentValue.append("]");
                return currentValue.toString();
[...]

Either the documentation is wrong (it states the following: "For YAML, both comma-separated lists and YAML lists can be used for completely overriding the contents of the list") OR you forgot a very important fact that was part of your previous documentation: https://docs.spring.io/spring-boot/docs/1.5.6.RELEASE/reference/html/boot-features-external-config.html#boot-features-external-config-loading-yaml. --> Extra care is required when configuring lists that way as overriding will not work as you would expect. In the example above, when my.servers is redefined in several places, the individual elements are targeted for override, not the list. To make sure that a PropertySource with higher precedence can override the list, you need to define it as a single property:

I know that the new documentation is about a list of complex objects and the old documentation is about a list with "simple" values. But there is no part in the new documentation that states that lists for simple types act differently than lists of complex types.

Comment From: wilkinsona

Assuming I have understood the problem correctly, I think this should work as you want it to. We have some tests that verify this is the case, for example:

https://github.com/spring-projects/spring-boot/blob/f0c05a28a3a7426335d7c21bf158840df2df481c/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/properties/bind/CollectionBinderTests.java#L156-L172

https://github.com/spring-projects/spring-boot/blob/f0c05a28a3a7426335d7c21bf158840df2df481c/spring-boot-project/spring-boot/src/test/java/org/springframework/boot/context/properties/bind/CollectionBinderTests.java#L174-L183

It's not clear where the unwanted blub is coming from in your case as it's present both in the property's default value and in application.yml. I'm also not sure where "the logic I use to get the value for the list property" comes into play. To remove these uncertainties, please provide a minimal sample that reproduces the problem and that uses a different value for the third element in the list in the two possible sources (its default and application.yml).

Comment From: Walnussbaer

I don't think it is necessary to use a different value for the third element.

But let me summarize what the problem is:

This is my application.yml

myProp
  - machine
  - articles
  - blub

This is my application-dev.yml, It has higher precedence because it is an active profile when starting the spring boot app.

myProp
 - machine
 - articles

Now when you want to query the value in custom logic, you have to use something like this: springEnvironment.getProperty(...)

For lists, you have to query earch list item individually by using it's index (https://docs.spring.io/spring-boot/docs/current/reference/htmlsingle/#features.external-config.yaml). Note that using springEnvironment.getProperty("my-prop") returns null when using the yml list syntax in properties files, So I must use the index syntax.

Bascially what my logic does is the following: Iterate over all available indices to get all list values. If an index returns null, that means for me that I have reached the end of the list.

So this is what happens in my code:

this.configurableSpringEnvironment.getProperty("my-prop[0]") --> returns machine --> correct 👍 
this.configurableSpringEnvironment.getProperty("my-prop[1]") --> returns articles --> correct 👍 
this.configurableSpringEnvironment.getProperty("my-prop[2]") --> returns NOT null but blub--> NOT correct 👎 

Why is the last one not correct: because application-dev.yml only contains a list with 2 entries and not 3.

Does that clarify my problem?

Comment From: Walnussbaer

UPDATE:

It seems like that the binding for the @ConfigurationProperties class works as expected. I just checked it in the debugger.

It only contains the 2 list items, machine and articles.

But the springEnvironment.getProperty() returns false (?) values (as described in my previous post).

Comment From: wilkinsona

Thanks, I understand now.

The documentation that you've referenced is referring to type-safe binding of configuration properties. The behavior where a list is only populated from a single source is specific to this binding and is what you get when you use @ConfigurationProperties. When you're accessing the environment directly, as you are doing, you aren't using the type-safe binding of configuration properties so none of its behavior applies.

If you want to mimic the same behavior when accessing the environment directly, you will have to drop down a level and work with the environment's individual property sources in order. Once you find one that contains a my-prop[] property, you'd then use it exclusively to determine all of the list's values. This is what IndexedElementsBinder does:

https://github.com/spring-projects/spring-boot/blob/9411a4ce99de39dbba3987ade25ce384cd447925/spring-boot-project/spring-boot/src/main/java/org/springframework/boot/context/properties/bind/IndexedElementsBinder.java#L66-L75

The return short-circuits the binding so that only the first source the contains items for the list is used.

If you're seeing the myProp field of a MyConfigurationClass instance that's been set by @ConfigurationProperties binding contain values from multiple sources then you have found a bug. If you can provide a minimal example of it occurring we can investigate it and fix it. On the other hand, if your problem only occurs when working with the environment directly, you'll have to update your code so that it behaves in a similar manner to Boot's configuration property binding support.

Comment From: wilkinsona

I missed your most-recent comment while writing the above. Thanks for confirming that @ConfigurationProperties is working as expected. In that case, I'm going to close this one. As you're working with the environment directly and have dropped beneath Boot's configuration properties support that builds on top of the environment, you'll have to mimic some of its functionality.

Comment From: Walnussbaer

@wilkinsona Thank you very much for the detailed explanation. I thought that the getProperty method of the SpringEnvironment already takes into account the precedences of property sources. But that doesn't seem to be case in this special scenario. Wouldn't that be better/ a reason for a new issue?

Because usually the getProperty() method returns the value with the highest precedence the spring environment contains.

For a better understanding: I have built a Config Editor for our project that enables us to alter the PropertySources and thus our @ConfigurationProperties bindings during runtime.

Comment From: wilkinsona

Multiple indexed property keys mapping onto the same underlying collection-based value is a feature of type-safe configuration property binding. The Environment from Spring Framework knows nothing about this feature. As far as it's concerned my-prop[0], my-prop[1], and my-prop[2] are three different and entirely unrelated properties.

I thought that the getProperty method of the Spring Environment already takes into account the precedences of property sources

It does, however it's taken into account on a property-by-property basis.

Comment From: Walnussbaer

All right, that makes sense.

I think the best solution for me is to use reflection on @ConfigurationProperties classes when trying to query the current value of a configuration property in a generic way (instead of using the spring environment which I use at the moment). This way I can leverage all the features of the type-safe binding of spring boot regarding @ConfigurationProperties classes.

Thanks again for the discussion!