Spring Boot allows configuration of the most common configuration options for Cassandra driver 4 and allows customization of the session builder and DriverConfigLoader.

Configuring additional driver config options (e.g. driver profiles, additional settings) requires quite some code to do so. It would be convenient to use the Driver's config file format for an easier configuration mechanism since the majority of configuration examples use config files as using config files is the preferred mechanism.

Ideally, a config file can be specified through a property (e.g. spring.data.cassandra.config-file) that gets included if the property is set after defaultOverrides and before Spring Boot's config. Doing so allows picking up the config file. It also allows overriding Cassandra settings using system properties and the specified config file may override Spring Boot's properties which aligns nicely with the typical ordering of PropertySources.

Reference

  • https://docs.datastax.com/en/developer/java-driver/4.9/manual/core/configuration/#concepts
  • https://docs.datastax.com/en/developer/java-driver/4.9/manual/core/pooling/
  • https://docs.datastax.com/en/developer/java-driver/4.9/manual/core/load_balancing/

Comment From: snicoll

and the specified config file may override Spring Boot's properties which aligns nicely with the typical ordering of PropertySources.

I think that's inconsistent with "it gets included if the property is set after defaultOverrides and before Spring Boot's config.". Specifying the config file as a system property or in application.properties will give it the same precedence. For that reason, I think the file should come first and take precedence over anything else that has been defined.

Comment From: snicoll

I've started hacking on this and I am not sure what I am doing wrong, perhaps @adutra could help me?

It looks like all values in the file are considered a String so "basic.request.timeout = 5 seconds" for instance leads to 

basic.request.timeout has type STRING rather than NUMBER

Also, I am having second thought about introducing a property to a file. Users may want to use a different format for the file or different precedences rules. Maybe we need to reconsider our options here?

Comment From: adutra

@snicoll I see here that you are trying to read the value of basic.request.timeout as a long with Config.getLong().

However the value for that key in the file is the string 100 milliseconds; this particular string can only be parsed as a string (obviously) or a Duration, not as a long. The following lines would both work:

driverConfig.getProfile("first").getString(DefaultDriverOption.REQUEST_TIMEOUT)
driverConfig.getProfile("first").getDuration(DefaultDriverOption.REQUEST_TIMEOUT)

I recommend going with getDuration since this setting expects its value to be convertible to a valid Duration instance.

Comment From: snicoll

Thanks @adutra, that fixed it.

I am having a hard time figuring out what the precedence should be. From my experiment, there isn't any satisfactory outcome.

Spring's Environment takes precedence over the file

This looks the more natural option to me after having discussed this one with @wilkinsona. Whatever you put in the environment wins over a property in the file. Given our configurable Environment this is the most logical way of doing things (i.e. a user may set a spring.data.cassandra.contact-points as an env property to override the Cassandra cluster location).

I've tried to try this using:

private Config createConfig(CassandraProperties properties) {
    Config config = mapConfig(properties);
    Resource configFile = properties.getConfig();
    if (configFile != null) {
        config.withFallback(loadConfig(configFile));
    }
    return  config;
}

Two problems with this approach:

  • Profiles are not loaded. I don't understand if that's a limitation of the API (you can't have a profile in a fallback Config) or something in the code above is wrong
  • Any default in CassandraProperties takes precedence over a value in the file. That's really counter-intuitive as if there is nothing in the environment for the property, you'd expect the property in the file to win. Fixing this would be cumbersome as the whole point of CassandraProperties is that we don't have to care if the user has set the property or if the default should be used, if any).

File takes precedence over Spring's Environment

This was my first approach to ignore the precedence problem I've described previously. If you specify a file, it will take precedence on anything. That also means that you can't really use the file to specify some defaults that you may want to override using the specific property.

Only file is used when set

If a file is used, we use that and only that, ignoring any other property we currently handle. This sounds like a hard sell for a Spring Boot user who's used to the auto-completion experience of application.properties in the IDE.

What do you think? I am missing something?

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: adutra

I am having a hard time figuring out what the precedence should be. From my experiment, there isn't any satisfactory outcome.

I'm not sure I have enough knowledge of Spring internals and philosophy to give a valid input in this discussion. But anyways, here are my 2 cents:

As an example, other libraries, such as Quarkus-Cassandra or the Cassandra Kafka Sink, went with the idea of a configuration stack generally composed of the following elements (the first ones taking precedence over the last ones):

  1. Settings defined via system properties;
  2. Settings defined through the library-specific configuration mechanism (such as Spring Boot's CassandraProperties);
  3. Settings defined in any application.conf or application.json file, if such files are found on the classpath. This would typically be used to configure advanced stuff that the library does not provide first-class support for.
  4. Driver default settings.

Regarding the handling of config files in layer 3 (advanced stuff) – since this seems to be the core of the current discussion:

  1. In the examples I cited, the files must follow TypeSafe conventions and be named either application.conf or application.json – this is usually not configurable. In fact, I would recommend against providing a way to change the names for these files: TypeSafe Config is already too complex as it is and besides, it does offer a means to change these names already. I personally think that introducing a spring.data.cassandra.config-file setting for this would be counter-productive – but happy to be convinced otherwise.
  2. Any application.conf file is expected to be in TypeSafe Config's HOCON format and should contain a valid driver config section inside the datastax-java-driver key, just as you would do if configuring the driver directly. Note that the file can in theory contain configuration for other libraries, such as Akka – this wouldn't be a problem as long as the driver can find its stuff in there.
  3. Same for application.json which is similarly expected to be parseable by TypeSafe Config.
  4. Note that TypeSafe also recognizes application.properties as a valid file name by default – but this creates a conflict. The libraries I mentioned, therefore, exclude this name from layer 3 since this name is already recognized by them at layer 2, and you wouldn't want to have that file parsed twice (once in layer 2, once in layer 3).

To sum up, I would urge you to go with the approach where any Spring-specific stuff takes precedence over config files. This seems to make more sense imho and also would be in sync with what we did for other frameworks.

I've tried to try this using:

java private Config createConfig(CassandraProperties properties) { Config config = mapConfig(properties); Resource configFile = properties.getConfig(); if (configFile != null) { config.withFallback(loadConfig(configFile)); } return config; }

If you ever follow my layered suggestion above, I would rather expect something like:

private Config createConfig(CassandraProperties properties) {
  return
  // layer 1: system properties – unless Spring already handles these
  ConfigFactory.defaultOverrides()
      // layer 2: SpringBoot properties from CassandraProperties
      .withFallback(mapSpringProperties(properties))
      // layer 3: application.conf and application.json
      .withFallback(
          ConfigFactory.parseResources("application.conf")
              .withFallback(ConfigFactory.parseResources("application.json")))
      // layer 4: driver defaults
      .withFallback(ConfigFactory.defaultReferenceUnresolved())
      .resolve();
}
  • Profiles are not loaded. I don't understand if that's a limitation of the API (you can't have a profile in a fallback Config) or something in the code above is wrong

Are you referring to Spring profiles or driver execution profiles? If the latter, we should probably investigate what's happening as this doesn't seem normal.

Also note that in the examples I cited, execution profiles are considered "advanced". IOW, users wishing to use driver execution profiles currently need to do it in layer 3. I think it is fair to expect the same from Spring Boot users.

Comment From: snicoll

In the examples I cited, the files must follow TypeSafe conventions and be named either application.conf or application.json – this is usually not configurable.

I don't understand what you mean by that. The work I have on this branch uses ConfigFactory.parseURL(config.getURL()). I don't understand how this should have an opinion as how the file should be named.

I personally think that introducing a spring.data.cassandra.config-file setting for this would be counter-productive – but happy to be convinced otherwise.

Not sure what you find counter-productive. Spring Boot already has an Environment abstraction with property precedence and profiles support. As such it should be preferred over any other mechanism. The discussion in this issue is that Spring Boot does not provide a mapping for all properties so user need a mechanism to access those if they want to. Given the unfortunate naming class of Cassandra's environment mechanism and ours, I am not keen to provide out-of-the-box support for something called application.*. What if Spring Boot decides in a future release to support .json format?

To sum up, I would urge you to go with the approach where any Spring-specific stuff takes precedence over config files. This seems to make more sense imho and also would be in sync with what we did for other frameworks.

Yep, we've reached the same conclusion. This prevents us to document defaults values in the metadata. IMO this is a major step backwards.

Are you referring to Spring profiles or driver execution profiles? If the latter, we should probably investigate what's happening as this doesn't seem normal.

I was talking about Cassandra profiles but maybe this was a glitch on my end.

Comment From: snicoll

I've updated my branch with the proposal (see https://github.com/snicoll/spring-boot/commit/2f7844bfedeaf61e054013ed48074092e03dfa34). It remains that the test doesn't pass as default in CassandraProperties are preventing updates from the file. What we can do is remove those defaults and then document them in manual metadata. It's a maintenance burden on our end we could soften by writing a test for it.

Let me know what you think and we can follow-up accordingly.

Comment From: spring-projects-issues

If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.

Comment From: ittays

// moved

Comment From: snicoll

Did you mean to post that on #31025?

Comment From: ittays

Oh, yes! I'll move the comment. Thanks!