1. Create test-A and test-B ssh key with this command, ssh-keygen -m PEM -t rsa -b 4096 -f ./test-X.
  2. Update test-A public key to a private repo-A as DeployKey, and test-B public key to repo-B as DeployKey*.
  3. Use this app as the config server client, for example test-app-0 and test-app-1.
  4. Make repo-A as default repository, repo-B as the one and the only one pattern repo, with pattern "test-app-1/*".
  5. Start the config server, test-app-0, and test-app-1.
  6. The end-to-end is not working from the perspective of test-app-0 and test-app-1, it cannot get the configurations of config server.
  7. Check the logs of config server, there is some exception like ERROR: Repository not found.

Full-stack as below.

org.eclipse.jgit.api.errors.InvalidRemoteException: Invalid remote: origin
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:251)
at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:306)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.cloneToBasedir(JGitEnvironmentRepository.java:616)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.copyRepository(JGitEnvironmentRepository.java:591)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.createGitClient(JGitEnvironmentRepository.java:574)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:271)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:249)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.getLocations(MultipleJGitEnvironmentRepository.java:139)
at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:173)
at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:64)
at org.springframework.cloud.config.server.config.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:72)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:85)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:44)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:71)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:60)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:291)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:376)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:497)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at com.microsoft.azure.spring.service.common.log.ServletLoggingUtil.doFilterWrapped(ServletLoggingUtil.java:45)
at com.microsoft.azure.spring.service.filter.RequestAndResponseLoggingFilter.doFilterInternal(RequestAndResponseLoggingFilter.java:30)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:280)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:260)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: git@github.com:Incarnation-p-lee/spring-cloud-service-config.git: ERROR: Repository not found.\n
at org.eclipse.jgit.transport.TransportGitSsh.cleanNotFound(TransportGitSsh.java:218)
at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:306)
at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:170)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:137)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1271)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:243)
... 105 common frames omitted\n","message":"Error occured cloning to base directory.","exceptionClass":"InvalidRemoteException"

Comment From: ryanjbaxter

The exception is coming from the health check, can you set management.health.config.enabled=false and try again?

Comment From: Incarnation-p-lee

@ryanjbaxter Sure, let me have a try.

Comment From: Incarnation-p-lee

Any idea about this ?

Comment From: Incarnation-p-lee

Is there any update about this issue ?

Comment From: ryanjbaxter

Can you please try 2021.0.1?

Comment From: Incarnation-p-lee

Sure, let me have a try and back to you later.

Comment From: habuma

FWIW, when I generate a key using the incantation in the original message at the top of this issue, it doesn't work for me as either a regular, user-level key nor as a deployment key. But if I use the following command line to generate the key, it works fine either as a user key or as a deployment key: ssh-keygen -t ecdsa -b 256 -m PEM -f ./test2

More details: When I use the key generated with ssh-keygen -m PEM -t rsa -b 4096 -f ./test-X, the logs provide more info than shown here. Specifically I get a line that says:

    at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: git@github.com:habuma/test-config.git: ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.

Perhaps that is the issue?

Comment From: ryanjbaxter

Yeah reading that blog post it seems like you need to use SHA2 when using RSA. Did you try using -m RFC4716?

Comment From: leonard520

@ryanjbaxter Just want to double confirm from Config Server 3.1.1 doc, when did ecdsa start to be supported?

It is important that an entry for the Git server be present in the ~/.ssh/known_hosts file and that it is in ssh-rsa format. Other formats (such as ecdsa-sha2-nistp256) are not supported

Comment From: ryanjbaxter

Ah yes, I forgot about that 🤦‍♂️ it wouldnt work

Comment From: leonard520

Ah yes, I forgot about that 🤦‍♂️ it wouldnt work

@ryanjbaxter Sorry, a bit confusing here. Do you mean that Config Server doesn't support ecdsa? But seems @habuma provided a method to configure a key generate by ecdsa in Config Server and made it work? Do I miss something?

Comment From: habuma

Indeed, creating an ecdsa key (with PEM formatting) worked. Not sure if this is a surprise for everyone, but I was able to make it work.

Comment From: leonard520

@ryanjbaxter Does it mean that Config Server doesn't officially support ecdsa but it can work in this case?

Comment From: ryanjbaxter

I can't say for sure without looking into it

Comment From: leonard520

By further looking into the issue, I think there are two issues. 1. GitHub retires sha1 and Config Server doesn't support sha2 yet. Seems a higher version of jsch fix the problem. I have tried v0.2.0 and it works with rsa. Is there some plan for Config Server to upgrade the version? More discussion here https://stackoverflow.com/questions/71489256/spring-cloud-config-server-github-sha-1-error

  1. If user have several repos, and each repo has its own ssh credential, Config Server will fail. Please check the below test result. Test materials: Private repo A in my account Private repo B in my account Private repo C in other's account Account key A in my account Account key B in my account Private repo C in other's account Deploy key A in repo A Deploy key B in repo B

Case A: repo A with account key A as main repo, works Case B: repo B with account key B as main repo, works Case C: repo C with account key C as main repo, works Case D: repo A with deploy key A as main repo, works Case E: repo B with deploy key B as main repo, works Case F: repo A with account key A as main repo, repo B with account key B as additional repo, works Case G: repo A with deploy key A as main repo, repo B with deploy key B as additional repo, NO works Case H: repo A with account key A as main repo, repo C with account key C as additional repo, NO works

It seems that Config Server can't work with multiple repos. The reason case F works is because the two account keys belong to me and either of them can be access to the two repos.

@ryanjbaxter Would you please help to check the second issue as well? Thanks a lot!

Comment From: ryanjbaxter

For 1 we are going to move to Apache MINA https://github.com/spring-cloud/spring-cloud-config/issues/1901

Im unsure on 2, we would need to look into that. I am kind of strapped for time at the moment so I am not sure when I will get to it.

Comment From: leonard520

By further looking into the issue, I think there are two issues.

  1. GitHub retires sha1 and Config Server doesn't support sha2 yet. Seems a higher version of jsch fix the problem. I have tried v0.2.0 and it works with rsa. Is there some plan for Config Server to upgrade the version? More discussion here https://stackoverflow.com/questions/71489256/spring-cloud-config-server-github-sha-1-error
  2. If user have several repos, and each repo has its own ssh credential, Config Server will fail. Please check the below test result. Test materials: Private repo A in my account Private repo B in my account Private repo C in other's account Account key A in my account Account key B in my account Private repo C in other's account Deploy key A in repo A Deploy key B in repo B

Case A: repo A with account key A as main repo, works Case B: repo B with account key B as main repo, works Case C: repo C with account key C as main repo, works Case D: repo A with deploy key A as main repo, works Case E: repo B with deploy key B as main repo, works Case F: repo A with account key A as main repo, repo B with account key B as additional repo, works Case G: repo A with deploy key A as main repo, repo B with deploy key B as additional repo, NO works Case H: repo A with account key A as main repo, repo C with account key C as additional repo, NO works

It seems that Config Server can't work with multiple repos. The reason case F works is because the two account keys belong to me and either of them can be access to the two repos.

@ryanjbaxter Would you please help to check the second issue as well? Thanks a lot!

I think the problem is caused by in extractNestedProperties https://github.com/spring-cloud/spring-cloud-config/blob/d8f6341f336e8da78eaab424ca3a3e82431f83db/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/ssh/SshUriPropertyProcessor.java#L59 It uses hostname as sshUriPropertyMap's key so in this case, two different github repos will be overridden to use one ssh configuration.

Comment From: ryanjbaxter

Maybe we can prefix the name with the repo name?

Comment From: leonard520

Maybe we can prefix the name with the repo name?

Yes, I think it can work.

Comment From: ryanjbaxter

The problem is that Jsch which we use to make SSH connections has no concept of a repo. Can you create a separate issue for that one, its unrelated to this issue.

Comment From: leonard520

Here is the issue to track the second problem https://github.com/spring-cloud/spring-cloud-config/issues/2105