SpringBoot Starter dependencies promoting misuse Maven's transitive dependency mechanisms

From Spring Boot's own documentation (as well as the many examples of use I've seen both within Spring Boot's own examples and elsewhere), it is clear that starter dependencies are intended to transitively bring in numerous other dependencies to be used at both runtime and compile. Per Spring Boot's documentation:

Starters are a set of convenient dependency descriptors that you can include in your application. You get a one-stop-shop for all the Spring and related technology that you need, without having to hunt through sample code and copy paste loads of dependency descriptors. For example, if you want to get started using Spring and JPA for database access, just include the spring-boot-starter-data-jpa dependency in your project, and you are good to go.

The starters contain a lot of the dependencies that you need to get a project up and running quickly and with a consistent, supported set of managed transitive dependencies.

However, according to the Maven dependency documentation it is intended that all compile dependencies be explicitly listed, rather than transitively used at compile time:

it is intended that [transitive compile dependencies] should be runtime scope instead, so that all compile dependencies must be explicitly listed - however, there is the case where the library you depend on extends a class from another library, forcing you to have available at compile time. For this reason, compile time dependencies remain as compile scope even when they are transitive.

Using this mechanism to transitively bring in compile-scoped dependencies therefore seems to be at odds with the intent of how Maven officially intends them to be used. One place that makes this abundantly clear is the Maven dependency:analyze plugin goal, which displays warnings when the Maven starter dependencies are used directly. For instance, running mvn dependency:analyze on Spring Boot's own "Getting Started" example generates the following output:

[WARNING] Used undeclared dependencies found:
[WARNING]    org.springframework:spring-web:jar:4.3.6.RELEASE:compile
[WARNING]    org.springframework.boot:spring-boot-test:jar:1.5.1.RELEASE:test
[WARNING]    org.springframework.boot:spring-boot-test-autoconfigure:jar:1.5.1.RELEASE:test
[WARNING]    org.springframework:spring-test:jar:4.3.6.RELEASE:test
[WARNING]    org.springframework.boot:spring-boot:jar:1.5.1.RELEASE:compile
[WARNING]    org.hamcrest:hamcrest-library:jar:1.3:test
[WARNING]    org.springframework:spring-context:jar:4.3.6.RELEASE:compile
[WARNING]    junit:junit:jar:4.12:test
[WARNING]    org.springframework.boot:spring-boot-autoconfigure:jar:1.5.1.RELEASE:compile
[WARNING]    org.springframework:spring-beans:jar:4.3.6.RELEASE:compile
[WARNING] Unused declared dependencies found:
[WARNING]    org.springframework.boot:spring-boot-starter-web:jar:1.5.1.RELEASE:compile
[WARNING]    org.springframework.boot:spring-boot-starter-test:jar:1.5.1.RELEASE:test
[WARNING]    org.springframework.boot:spring-boot-starter-actuator:jar:1.5.1.RELEASE:compile

Why was the Spring Boot starter pattern was designed in such a way to be directly contrary to the intent of the underlying build system? Is there a reason it was felt that this misuse is "OK"? Is there a "correct" way that this functionality — conveniently providing a set of dependencies at set known versions with a single simple Maven configuration — would better be implemented?

Note that I first asked this question on StackOverflow per the Spring Boot question guidelines, but I received no responses and the question got closed as "too broad". — https://stackoverflow.com/questions/27994153

Comment From: wilkinsona

Why was the Spring Boot starter pattern was designed in such a way to be directly contrary to the intent of the underlying build system?

IMO, you've read too much into a single sentence in Maven's docs. I think it's telling that to be even warned you have to opt in.

Is there a reason it was felt that this misuse is "OK"?

Misuse is rather subjective. I don't consider the starters to be a misuse. If you do, then you are free to declare all of your compile dependencies explicitly.

Is there a "correct" way that this functionality — conveniently providing a set of dependencies at set known versions with a single simple Maven configuration — would better be implemented?

I don't believe so, which could be one very good reason why transitive compile dependencies remain in the compile scope and no warnings are generated by default.

Comment From: mjustin

IMO, you've read too much into a single sentence in Maven's docs. I think it's telling that to be even warned you have to opt in.

It's literally saying that the intent is that "all compile dependencies must be explicitly listed", so I don't think saying it's the intent it's reading too much into that sentence. Now perhaps that's an erroneous statement in their documentation, or one that is no longer applicable. But it's definitely the stated intent that compile dependencies be explicitly listed.

What is not obviously stated is the rationale behind that, and the risks involved in varying from that intent.

Comment From: snicoll

(not sure that it helps, but I am also on the Maven PMC).

I agree with @wilkinsona - You are overreading this and your point of view lack of practicability IMO. The dependency:analyze goal was a mean to offer users a way to detect that your compilation classpath only contains what you actually need. This sentence in the doc is merely there to explain why the compile scope is transitive as well.

Back to what Spring Boot does, rather than talking about "misuse" (that's cold), can you take a step back and see why we've done things that way? (that's the practicability part).

First, if you want to list your dependencies yourself, go ahead we're fine with that. It will be more work (and maybe a cleaner build from your point of view) and every single feature Spring Boot has to offer will work exactly the same way. In other words, if you don't like our starters, don't use them.

Back to the feature, it is a way to guide you with opinions and what others out there are doing. It's not a silver bullet but if you follow the path it will save you a lot of time, especially when you're starting a new project or when you are upgrading your technical stack (~= the Spring Boot version).

In retrospective, you're having a Maven question and not a Spring Boot questions because tons of projects out there not using Spring Boot do not list all their compile dependencies themselves. I am not sure what can be added but we're on gitter if you want to chat.

Comment From: mjustin

@snicoll,

Thanks for your explanation. "Misuse" may be a bit harsh of a term on my part, and I could have done more to explain the reason for asking the question. I will attempt to lay out my thinking on this issue (and my apologies in advance for the length, but I figure better a long complete explanation than talking past each other).

To start with, I think Spring Boot's goal of providing a standard set of consistent, known-working dependencies is a great one, and I'm not questioning that. In addition to the code correctness factor, it also makes it much easier to make use of the libraries used within a particular starter — particularly when using an IDE that auto-imports/auto-completes such things. So from a practicality standpoint (as you put it), I completely agree with what Spring Boot is achieving here.

To restate my interpretation of that sentence in the documentation, it seems that if the Maven designers could have, they would made transitive compile-scoped dependencies be brought in at a runtime scope instead. The consequence of this would be that to use any such dependencies directly from the code at compile time would have to be explicitly declared in the POM as compile scope.

However, they couldn't make transitive compile dependencies runtime scope since a dependency might be using the transitive dependency in a way that requires it to be the compile scope. As a concrete example, say there is a compile time dependency a, which itself has a compile time depedency b. a has a class A which extends class B in dependency b. If you try to extend class A but didn't have class B on your compile-time classpath, you'd get a compilation error. Therefore, dependency b needs to be brought in as a compile-time dependency when bringing in dependency a as a compile time dependency, even though no classes in it are used directly at compile time.

I am not a Maven designer, so I can only make guesses and inferences as to why Maven has this opinion on dependencies. One advantage I've seen is that such transitive dependencies are frequently implementation details of the underlying library. The fact that a library used a different one under the covers is ideally not something you'd want to depend on. The library designers could very reasonably remove the underlying library dependency (or change its version) if they change their internal implementation. So by depending on a library that you're not declaring, you're depending on the implementation details of another library, and may break yourself when you go to upgrade dependencies. Granted, the fix is straightforward, but it still feels less than ideal.

That all said, the Spring Boot starter POMs feel like a slightly different beast than this. In their case, one of their main purposes is to bundle up a list of dependencies that the app can make use of. So it's a very intentional exposure of dependencies, rather than an incidental one. Due to this, I think your statement that I'm "having a Maven question and not a Spring Boot questions because tons of projects out there not using Spring Boot do not list all their compile dependencies themselves" is actually not completely applicable. The key difference here is that this is clearly the intent of the starter projects, whereas I haven't run across other libraries that are explicitly designed with the purpose of exposing its dependencies.

Design ideology aside, the pragmatic question is what (if any) consequences there are for using transitive compile dependencies in this manner. It seems unlikely that Maven will be changing how transitive compile dependencies work, so there's probably no real risk of the tool changing to be more like what it claims to be intending. I would say that the only real consequences involved are around dependency verification tools, with misleading output as to potential issues. If I wanted to use "maven dependency:analyze" to check that I'm not inadvertantly using any dependencies that I'm not intending to, it would return the starters as "unused declared" dependencies, and any transitively used dependencies as "used undeclared" depencencies. Thus any acutal issues that I'm trying to find would be impossible to distinguish from the noise. Now I can mark the starter dependencies as "ignoredUnusedDeclaredDependencies" and explicitly declare the transitive dependencies (or mark them as "ignoredUsedUndeclaredDependencies"). However, as soon as I do that I'm losing one of the primary benefits of the Spring Boot starter.

Ultimately, my goals in raising this question are: 1. Improving my understanding of why the tools have made certain design decisions 2. Determining whether it would be ideal if changes were made to the Maven and/or Spring Boot approaches. 3. Making it so analyzing a project's dependencies would allow transitive compile dependencies from Spring Boot starter dependencies, but not other types of dependencies.

For instance, would it be better if: 1. There was a way to have Maven be aware of dependencies like the starter ones where the goal is to make the dependencies available at compile time? 2. The Starter project dependencies were added via some other Maven mechanism, like a custom plugin 3. The dependency:analyze goal (and related goals) could be made aware of the special nature of Spring starter dependencies, and not treat their intended use cases as warnings.

Comment From: bclozel

Dependency management is certainly an interesting topic and something the Spring Boot team takes seriously. Now Andy is leading the Spring IO Platform (effectively managing a lot of dependencies in the Spring portfolio and enforcing good dependency hygiene in projects), and Stephane is a Maven PMC + our release manager/build gatekeeper.

Both said that this merely points to the maven compile scope being transitive and that you're reading too much into this. The Spring Boot build is expressing dependencies as they're meant to be; like you're saying yourself, this is a problem from the dependency analysis point of view.

Dependency verification tools are only providing a view on a tiny part of the actual dependency management problem; we have to do much more: backwards compatibility, license, project maintenance, community, support, runtime compatibility, etc.

We've provided our opinions to the best of our knowledge and experience and discussing this even more is not time well spent for the Spring Boot project.

Comment From: kohlerm

In short maven's compile time dependency model is broken. :smiling_imp:

It pollutes the class path with unnecessary dependencies, which is bad for compilation speed, and confuses people.. I have meet quit a few developers who believed that you always need all transitive dependencies for(Java) compilation. In almost all cases you only need three library you import from. The inheritance example mentioned is rather bad design IMHO,

Comment From: mjustin

Per @bclozel's point about this being viewed as an analysis problem and not an issue with Spring Boot, I have created ticket MDEP-557 for the Maven Dependency Plugin to handle this scenario.

Comment From: wlei07

Agree with @mjustin and @kohlerm And really suprised by Spring community's defensive mentility. People are giving suggestions and hoping it better, and you guys are NOT welcoming and pushing them away...

Comment From: wilkinsona

I don't think we pushed anyone away. We didn't agree that we've misused Maven's transitive dependency mechanisms, but three different members of the team spent their time commenting on the issue, explaining why we've done what we have done with the tools that Maven makes available to us.

For the situation to improve, changes to Maven or at least its dependency analysis tool are required. mjustin took the time to open an issue to explore that which helps to move things forwards in a constructive direction. Ironically, you've reacted with a 👎 to that when it's the first step to make things better, particularly as it won't just benefit Spring Boot but also other projects such a JUnit that offers a dependency that aggregates a number of dependencies in a similar manner to Boot's starters.

Comment From: wlei07

Well maven community also disagree with you guys, see how long your ticket has been opened and their comments: https://issues.apache.org/jira/browse/MDEP-557

I hope you guys are really creative and help us solve the dependency hell in our project with all the transitive dependencies... With due respect, Spring is the best framework, and I hope you guys keep it that way...

Or probably it was really my mistake, by the name "starters", it was intended for a project at starting phase, and when a project is already started and being mature, they should really stop using starters as you suggested: don't use them.

Comment From: wilkinsona

I don't see any evidence that the Maven community disagrees. It may not be a priority for them, but I don't see anyone saying it's a bad idea nor has the issue been closed. I think it's more likely that it's just not a priority for them. They have plenty on their plates and they're largely volunteers.

I hope you guys are really creative and help us solve the dependency hell in our project with all the transitive dependencies...

Unfortunately, we're not in a position to solve the dependency hell problems as they need to be solved at the level of the build tool.

Gradle has gone some way to help with this as they separate API dependencies (that are added to the compile classpath of consuming projects) and implementation dependencies (that are only added to the runtime classpath of consuming projects). On the producing side, both API and implementation dependencies are on the compile classpath.

IIRC, allowing producers and consumers to have different views of a module's dependencies is something that may be coming in Maven 4. Should this happen, we may be able to revisit Spring Boot's starters and how the dependencies are declared, but I suspect we may struggle to reach consensus on what should be a compile dependency and what should be a runtime dependency from a consumer's perspective and the status quo may still prevail.

by the name "starters", it was intended for a project at starting phase, and when a project is already started and being mature, they should really stop using starters as you suggested

The suggestion was not to use the starters if you don't like the transitive nature of their compile dependencies. That suggestion still stands as it's the only option that's available at this time. We don't suggest that people stop using the starters purely because a project is mature. If they're working well for you, please continue using them. If they're not and you believe that moving to explicitly declared dependencies will better suit your needs, please do that.