To make is easier to work with configuration in .properties files and .yml files and map the configuration entries to metadata entries for validation purposes, we'd like to add something to spring-boot-configuration-metadata that shows how to do it based on the rules described on the wiki.

We're not 100% sure what form the input to the helper method should take. @kdvolder @martinlippert, can you please help to guide us by providing some info on how you might use such a method?

Comment From: kdvolder

I have a number of ideas on actually multiple helper methods that could be useful depending on context. The reason their could be more than one method is that you essentially may need to 'run' the semantics forwards and backwards depending on what the tools are trying to do. Essentially there are two use-cases:

  • for things like completions one needs to be able to 'generate' alternate forms of a key
  • for validation one needs to able to 'recognize equivalency' between two given keys (or at least between a given key typed by the user and the form of the key expressed in the metadata).

For the 'generative form' the input can be the key as it appears in the metadata and the output a collection of 'aliases' that can be used in properties / yaml file by users.

For the 'equivalency test' I think the best approach would be (if it is possible, and I think it is under the Boot 2.x rules) to compute a 'canonical' value of the key. This should return a value that can be used as a lookup key in a hashmap. I.e. input is some key from a user's document (or from the metadata), and the output is a value that can be used to efficiently look up information for that key in a hashmap.

This serves two purposes at once. You can use it for 'equivalency tests' simply by comparing the canoncual forms of two keys to one another. And as bonus you can also use it to store and retrieve information efficiently (which is important when trying to understand/navigaget nested yaml structure to determine completion context, for example).

Comment From: kdvolder

It would also be useful to have a third helper method that, given the name of a getter or setter method in a Java bean returns the key as it would appear in the properties metadata. The rules around how to derive kebab form of the key from setter and getter methods are also somewhat complex and really not spelled out clearly anywhere (as far as I know).

Comment From: kdvolder

A fourth useful mehod idea...

Compute the 'recommended' property name (segment) from a (Enum) field name. I.e. when there is a enum with name like SOMETHING_WITH_UNDERSCORES what is the recommended 'spelling' of this name to use in a property (e.g. when it used as the key in property bound to a map, as for example, arises in something jackson serialization and deserialization options.

Comment From: kdvolder

Another sugestion: a helper method that accepts an identifier in any of the 'supported' formats (kebab / snake / camel) and converts it to the 'recommended' kebab form).

And one more: a helper method that accepts an identifier in any of the supported formats and returns a collection of all the valid supported aliases.

Comment From: kdvolder

Thought about this a bit more and perhaps we can sanitize the list of all the above suggestions a bit to something a bit more manageable.

The only method I am sure of will be useful at the moment would be the one that computes the 'canonical' form of the key as explained earlier. I.e. given a key as the user may type in a document, compute a value derived from this that can be used to store and lookup metadata in a hash table.

All the other methods I suggested, I'm not so sure about at the moment. As I start working on https://github.com/spring-projects/sts4/issues/331 I may discover in due course if there's others that would be useful. But for the time being, its probably not worth for you to invest effort into developing them until there's really clear use for them.

It seems to me that the requested 'toCanonicalForm' method is actually not that hard to implement so perhaps you don't even need to implement it at all.

What would be helpful however is if someone could help me answer some questions / confusions I have at the moment after reading through:

https://github.com/spring-projects/spring-boot/wiki/Relaxed-Binding-2.0

Question 1: What is meant precisely by 'special character'.

It says in a number of places:

properties are bound by removing any special characters and converting to lowercase

But its not 100% clear what is consider as a 'special' characters in this context. Is that any non-alphanumeric character? What about the '.' characters? In the examples It looks like the '.' are always being retained so... they are not 'special'?

Question 2

Escaping rules in yaml keys strike me as a little strange...

In particular this example confuses me (which probably means I'm misunderstanding something about how things really work).

spring:
  my-example:
    '[foo.baz]': bar
    '[abc xyz]': def

Why can we not simply type something like this:

spring:
  my-example:
    foo.baz: bar
    'abc xyz': def

I can understand there's a need to escape the space in a yaml key. But I don't think there is a need to escape a '.' inside of yaml key is there? Yaml is perfectly happy to allow for dots inside the keys. I also do not understand why we should put '[]' around something with a '.' to identifiy it as 'belonging together' when it should be clear already from the parsed yaml that it belongs together as it is contained within a single yaml key.

I do get however why these brackets might be necessary in the .properties format to keep a key like 'foo.baz' from being split at the '.'. 

spring.my-example[foo.baz]: bar

But in yaml... there's no confusion about the meaning of the '.' so why is it necessary to add extra '[]'?

Question 3

Not a question from the document. But just something I don't understand... and maybe somewhat related to Question 2.

Why does this work in a .properties file:

logging.level.some.package=debug

I was expecting it should rather be written like this:

logging.level[some.package]=debug

I.e. to avoid interpreting the '.' in the package name being interpreted as stepping into a map rather than being a part of the key.

Anyhow... I'm probably not quite grasping the mechanics of how this works correctly. Someone can help me understand?

Comment From: philwebb

Question 1: What is meant precisely by 'special character'.

Most of the actual source of truth can be found in the code for ConfigurationPropertyName. In this case it means anything that isn't a -z, 0-9. The . is not mentioned because it has already been used to split the key at this point.

To give a more concrete example, given the name person.first-name. The element first-name is bound to a setter or constructor arg that matches firstname. The matching here is not case-sensitive. Another example of a slightly different form is PERSON_FIRSTNAME in an environment variable. Here the split occurs on _, giving ['PERSON', 'FIRSTNAME'] and the binding would look for a setter or constructor arg that matches FIRSTNAME.

Escaping rules in yaml keys strike me as a little strange...

The YAML format never lasts that long, it's actually mapped to something more like the properties format as it's loaded. The reason for this is we need something that can work with Spring's Environment abstraction.

So given this file:

spring:
  my-example:
    '[foo.baz]': bar
    '[abc xyz]': def

What ends up in the Environment is:

spring.my-example.[foo.baz]=bar
spring.my-example.[abc xyz]=def

I don't think there is a need to escape a '.' inside of yaml key is there?

There's unfortunately some complexity around this due to the flattening mentioned above and the fact that you can bind complex object inside a Map.

A specific example would be Map<String,Person> where Person has nested Name items with firstName and lastName fields. You can map that using:

person.springboot.name.fist-name=Spring
person.springboot.name.last-name=Boot

To keep things simple the Map binder always takes the first element as the key then tries to bind the rest to the value. So here you'd have a Map with a single key named springboot and a Person object with a nested Name object containing the fields firstName of Spring and lastName of Boot.

If your key has dots in it we need the escaping to know that it needs to be consumed in its entirety. For example, say you wanted a key of spring.boot you'd need to write:

person.[spring.boot].name.fist-name=Spring
person.[spring.boot].name.last-name=Boot

Without that we'd be looking for boot field on the Person object.

There is one exception to this rule. If the Map value is a scalar type like an Enum then I think we're greedy with the Key. E.g. with Map<String,LogLevel> I pretty sure you can write:

logging.level.org.springframework.boot=info

Not a question from the document. But just something I don't understand... and maybe somewhat related to Question 2

This is because the logging level is an enum type so we are greedy with they key. There are a lot of examples of map binding in the MapBinderTests that might be worth a look at. Take a look at MapBinder.getEntryName to see this logic in action.

Comment From: kdvolder

Thanks Phil, I think I understand the reason for the weird escapes using '[' around yaml keys.

The rule still strikes me as a bit weird though. From a user's point of view it seems totally unnessary and seems really like something that the 'mapping' (i.e boot transformation function that converts from yaml to properties format) could handle that better. I.e. what I mean is that if I write something like this:

spring:
  my-example:
    foo.baz: bar

Then the mapping that transforms this into the properties format could just be smart enough to detect that there is a '.' inside of the 'foo.baz' key, so to avoid splitting it up and breaking what seems to be clearly the user's intent (i.e. 'foo.baz' is a single map key), the mapping should automatically add the '[]' around that key and produce something like this:

spring.my-example[foo.baz]: bar

If you agree with that (?), I wonder if you think that perhaps you might actually 'fix' this weirdness in a future release?

TBH, I am not sure I'd want to bend the tooling around this weird situation and treat yaml map keys as something potentially 'composite' (this is definitely not how our editor interprets them at the moment, it just treats them always as 'atomic' values and the only 'escaping' done on its values is what the yaml parser does. I would probably want to keep things simple in that way. Especially so if you agree that this double escaping weirdness could actually be thought of as bug in spring boot, i.e. something that mat actually be changed to be more logical in the future. (But if you think it is unlikely to ever get changed, then I may need to think long and hard about how our editor interprets yaml keys as atomic now, and how/if to change this).

About the logging.level special case...

There is one exception to this rule. If the Map value is a scalar type like an Enum then I think we're greedy with the Key. E.g. with Map I pretty sure you can write:

Yes, I'm sure too. I just didn't understand why :-)

. If the Map value is a scalar type like an Enum then I think we're greedy with the Key.

How do we know when a type is 'scalar' ? I.e. can you give a more precise characterisation of the 'special case' rule. How about lists for example? Since you can write these in comma separated format... they are 'scalar'? Or are they not?

What about a POJO class like 'Person'? Do we have to inspect the class to determine whether it has setter / getter methods and looks like it might be some kind of bean? (A potential complication for the tooling, is that we may not be able to inspect a type if its not on the classpath, which does happen in some cases).

Finallly... how important is it that the tools should understand/support the special case logic?

I.e. would it be okay for example if we flagged a warning for something like

logging.level.org.springframework.boot=info

because 'springframework' is not something that looks like a valid property of the type of values stored in the logging.level Map?

I.e. should user be encouraged to write this:

logging.level[org.springframework.boot]=info

Or should we (tooling/editor) really work hard at handling this special case?

Personally I think putting brackets to make it clear that 'this stuff is one key' seems like a good practice and probably something to encourage.

BTW: I think our current editor implementaton does support this special case and I know it works fine for 'logging.level' specifically. I'm just not 100% sure we have the 'trigger condition' for the special case exactly right so it might not actually work correctly in all cases.

Comment From: philwebb

seems really like something that the 'mapping' (i.e boot transformation function that converts from yaml to properties format) could handle that better

That's quite a nice idea but it might break existing users. It's often quite convenient to not use the nested form at all. I.e:

spring:
  my-example:
    foo: bar

Can be written:

spring.my-example.foo: bar

if you think it is unlikely to ever get changed, then I may need to think long and hard about how our editor interprets yaml keys as atomic now, and how/if to change this

I can't see an obvious way to know which form the user really wants so I think it's pretty unlikely that we can change anything in the near term.

How do we know when a type is 'scalar'

That's a very hard question to answer without a running application. I'm not sure you can tell at design-time what the actual binding rules will be a runtime. I have a feeling we might need to look at the meta-data JSON file that we provide to see if we can provide extra information.

I.e. would it be okay for example if we flagged a warning for something like 'logging.level.org.springframework.boot=info'

I think we shouldn't log a warning for such cases since our documentation has actively encouraged that form. The [...] notation is pretty ugly so it seems a shame to insist on it.

My gut instinct is that we should try and make the IDE implementation as simple as possible but I'm not sure how. I think we'll need some input from @snicoll and since he has quite a bit of knowledge on how other IDEs deal with it.

Comment From: kdvolder

Can be written:

spring.my-example.foo: bar

Yes, I see, if you insist on this being 'correct' and equivalent to the nested form, then there really isn't a way to avoid the explicit escapes for when you do want the '.' to be a actual part of the key (instead of being interpreted... since it really could be either one.

The [...] notation is pretty ugly so it seems a shame to insist on it.

I guess its a matter of taste. I personally think it is nicer to see the brackets clearly demarkating the logger/package name, than to have to mentally parse the structure of something like this:

logging.level.org.springframework.boot=info

Not all the '.' in the above have the same meaning. And I think that is actually confusing, not just to the tooling but also to any human trying to read it. For example, as human, it is not possible to mentally parse this property correctly and understand that org.springframework.boot is meant to be 'atomic' unless I actualy know a lot about the types of the properties and what they are bound to.

The form with explicit '[..]' on the other hand, does not require this knowledge and it is immediately clear at a cursory glance that the dots in 'org.springframework.boot' are not to be interpreted as 'structure traversals' but just as literal dots.

So really, I would always prefer to use the explicit '[]' here. It leads to simpler rules which are both easier for tooling and for humans to understand.

Comment From: kdvolder

So really, I would always prefer to use the explicit '[]' here. It leads to simpler rules which are both easier for tooling and for humans to understand.

In addition if we accept that tooling is allowed to give a warning when the '[]' are not used expliclty, then, basically my questions about 'when does this special case really apply' and 'how do we know the type a property is bound to is scalar'.... kind of become moot. (Which I guess is just another example on how 'insisting' on explict brackets to escape the '.' in all cases... makes everything simpler).

Comment From: wilkinsona

https://github.com/spring-projects/sts4/issues/331 has been closed so it looks like this is no longer needed.