- Create test-A and test-B ssh key with this command, ssh-keygen -m PEM -t rsa -b 4096 -f ./test-X.
- Update test-A public key to a private repo-A as DeployKey, and test-B public key to repo-B as DeployKey*.
- Use this app as the config server client, for example test-app-0 and test-app-1.
- Make repo-A as default repository, repo-B as the one and the only one pattern repo, with pattern "test-app-1/*".
- Start the config server, test-app-0, and test-app-1.
- The end-to-end is not working from the perspective of test-app-0 and test-app-1, it cannot get the configurations of config server.
- Check the logs of config server, there is some exception like
ERROR: Repository not found
.
Full-stack as below.
org.eclipse.jgit.api.errors.InvalidRemoteException: Invalid remote: origin
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:251)
at org.eclipse.jgit.api.CloneCommand.fetch(CloneCommand.java:306)
at org.eclipse.jgit.api.CloneCommand.call(CloneCommand.java:200)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.cloneToBasedir(JGitEnvironmentRepository.java:616)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.copyRepository(JGitEnvironmentRepository.java:591)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.createGitClient(JGitEnvironmentRepository.java:574)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:271)
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:249)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.getLocations(MultipleJGitEnvironmentRepository.java:139)
at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55)
at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:173)
at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:64)
at org.springframework.cloud.config.server.config.ConfigServerHealthIndicator.doHealthCheck(ConfigServerHealthIndicator.java:72)
at org.springframework.boot.actuate.health.AbstractHealthIndicator.health(AbstractHealthIndicator.java:82)
at org.springframework.boot.actuate.health.HealthIndicator.getHealth(HealthIndicator.java:37)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:85)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.getHealth(HealthEndpointWebExtension.java:44)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:99)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getAggregateHealth(HealthEndpointSupport.java:110)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getContribution(HealthEndpointSupport.java:96)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:74)
at org.springframework.boot.actuate.health.HealthEndpointSupport.getHealth(HealthEndpointSupport.java:61)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:71)
at org.springframework.boot.actuate.health.HealthEndpointWebExtension.health(HealthEndpointWebExtension.java:60)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.util.ReflectionUtils.invokeMethod(ReflectionUtils.java:282)
at org.springframework.boot.actuate.endpoint.invoke.reflect.ReflectiveOperationInvoker.invoke(ReflectiveOperationInvoker.java:74)
at org.springframework.boot.actuate.endpoint.annotation.AbstractDiscoveredOperation.invoke(AbstractDiscoveredOperation.java:60)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$ServletWebOperationAdapter.handle(AbstractWebMvcEndpointHandlerMapping.java:291)
at org.springframework.boot.actuate.endpoint.web.servlet.AbstractWebMvcEndpointHandlerMapping$OperationHandler.handle(AbstractWebMvcEndpointHandlerMapping.java:376)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.base/java.lang.reflect.Method.invoke(Method.java:566)
at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:205)
at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:150)
at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:117)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:808)
at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1067)
at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:963)
at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1006)
at org.springframework.web.servlet.FrameworkServlet.doGet(FrameworkServlet.java:898)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:497)
at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:883)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:584)
at io.undertow.servlet.handlers.ServletHandler.handleRequest(ServletHandler.java:74)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:129)
at com.microsoft.azure.spring.service.common.log.ServletLoggingUtil.doFilterWrapped(ServletLoggingUtil.java:45)
at com.microsoft.azure.spring.service.filter.RequestAndResponseLoggingFilter.doFilterInternal(RequestAndResponseLoggingFilter.java:30)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.RequestContextFilter.doFilterInternal(RequestContextFilter.java:100)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.FormContentFilter.doFilterInternal(FormContentFilter.java:93)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.boot.actuate.metrics.web.servlet.WebMvcMetricsFilter.doFilterInternal(WebMvcMetricsFilter.java:96)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at org.springframework.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:201)
at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
at io.undertow.servlet.core.ManagedFilter.doFilter(ManagedFilter.java:61)
at io.undertow.servlet.handlers.FilterHandler$FilterChainImpl.doFilter(FilterHandler.java:131)
at io.undertow.servlet.handlers.FilterHandler.handleRequest(FilterHandler.java:84)
at io.undertow.servlet.handlers.security.ServletSecurityRoleHandler.handleRequest(ServletSecurityRoleHandler.java:62)
at io.undertow.servlet.handlers.ServletChain$1.handleRequest(ServletChain.java:68)
at io.undertow.servlet.handlers.ServletDispatchingHandler.handleRequest(ServletDispatchingHandler.java:36)
at io.undertow.servlet.handlers.RedirectDirHandler.handleRequest(RedirectDirHandler.java:68)
at io.undertow.servlet.handlers.security.SSLInformationAssociationHandler.handleRequest(SSLInformationAssociationHandler.java:117)
at io.undertow.servlet.handlers.security.ServletAuthenticationCallHandler.handleRequest(ServletAuthenticationCallHandler.java:57)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.security.handlers.AbstractConfidentialityHandler.handleRequest(AbstractConfidentialityHandler.java:46)
at io.undertow.servlet.handlers.security.ServletConfidentialityConstraintHandler.handleRequest(ServletConfidentialityConstraintHandler.java:64)
at io.undertow.security.handlers.AuthenticationMechanismsHandler.handleRequest(AuthenticationMechanismsHandler.java:60)
at io.undertow.servlet.handlers.security.CachedAuthenticatedSessionHandler.handleRequest(CachedAuthenticatedSessionHandler.java:77)
at io.undertow.security.handlers.AbstractSecurityContextAssociationHandler.handleRequest(AbstractSecurityContextAssociationHandler.java:43)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.SendErrorPageHandler.handleRequest(SendErrorPageHandler.java:52)
at io.undertow.server.handlers.PredicateHandler.handleRequest(PredicateHandler.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.handleFirstRequest(ServletInitialHandler.java:280)
at io.undertow.servlet.handlers.ServletInitialHandler.access$100(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:134)
at io.undertow.servlet.handlers.ServletInitialHandler$2.call(ServletInitialHandler.java:131)
at io.undertow.servlet.core.ServletRequestContextThreadSetupAction$1.call(ServletRequestContextThreadSetupAction.java:48)
at io.undertow.servlet.core.ContextClassLoaderSetupAction$1.call(ContextClassLoaderSetupAction.java:43)
at io.undertow.servlet.handlers.ServletInitialHandler.dispatchRequest(ServletInitialHandler.java:260)
at io.undertow.servlet.handlers.ServletInitialHandler.access$000(ServletInitialHandler.java:79)
at io.undertow.servlet.handlers.ServletInitialHandler$1.handleRequest(ServletInitialHandler.java:100)
at io.undertow.server.Connectors.executeRootHandler(Connectors.java:387)
at io.undertow.server.HttpServerExchange$1.run(HttpServerExchange.java:852)
at org.jboss.threads.ContextClassLoaderSavingRunnable.run(ContextClassLoaderSavingRunnable.java:35)
at org.jboss.threads.EnhancedQueueExecutor.safeRun(EnhancedQueueExecutor.java:2019)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.doRunTask(EnhancedQueueExecutor.java:1558)
at org.jboss.threads.EnhancedQueueExecutor$ThreadBody.run(EnhancedQueueExecutor.java:1449)
at org.xnio.XnioWorker$WorkerThreadFactory$1$1.run(XnioWorker.java:1280)
at java.base/java.lang.Thread.run(Thread.java:829)\nCaused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: git@github.com:Incarnation-p-lee/spring-cloud-service-config.git: ERROR: Repository not found.\n
at org.eclipse.jgit.transport.TransportGitSsh.cleanNotFound(TransportGitSsh.java:218)
at org.eclipse.jgit.transport.TransportGitSsh$SshFetchConnection.<init>(TransportGitSsh.java:306)
at org.eclipse.jgit.transport.TransportGitSsh.openFetch(TransportGitSsh.java:170)
at org.eclipse.jgit.transport.FetchProcess.executeImp(FetchProcess.java:137)
at org.eclipse.jgit.transport.FetchProcess.execute(FetchProcess.java:123)
at org.eclipse.jgit.transport.Transport.fetch(Transport.java:1271)
at org.eclipse.jgit.api.FetchCommand.call(FetchCommand.java:243)
... 105 common frames omitted\n","message":"Error occured cloning to base directory.","exceptionClass":"InvalidRemoteException"
Comment From: ryanjbaxter
The exception is coming from the health check, can you set management.health.config.enabled=false
and try again?
Comment From: Incarnation-p-lee
@ryanjbaxter Sure, let me have a try.
Comment From: Incarnation-p-lee
Any idea about this ?
Comment From: Incarnation-p-lee
Is there any update about this issue ?
Comment From: ryanjbaxter
Can you please try 2021.0.1?
Comment From: Incarnation-p-lee
Sure, let me have a try and back to you later.
Comment From: habuma
FWIW, when I generate a key using the incantation in the original message at the top of this issue, it doesn't work for me as either a regular, user-level key nor as a deployment key. But if I use the following command line to generate the key, it works fine either as a user key or as a deployment key: ssh-keygen -t ecdsa -b 256 -m PEM -f ./test2
More details: When I use the key generated with ssh-keygen -m PEM -t rsa -b 4096 -f ./test-X
, the logs provide more info than shown here. Specifically I get a line that says:
at java.base/java.lang.Thread.run(Thread.java:833) ~[na:na]
Caused by: org.eclipse.jgit.errors.NoRemoteRepositoryException: git@github.com:habuma/test-config.git: ERROR: You're using an RSA key with SHA-1, which is no longer allowed. Please use a newer client or a different key type.
Please see https://github.blog/2021-09-01-improving-git-protocol-security-github/ for more information.
Perhaps that is the issue?
Comment From: ryanjbaxter
Yeah reading that blog post it seems like you need to use SHA2 when using RSA. Did you try using -m RFC4716
?
Comment From: leonard520
@ryanjbaxter Just want to double confirm from Config Server 3.1.1 doc, when did ecdsa
start to be supported?
It is important that an entry for the Git server be present in the ~/.ssh/known_hosts file and that it is in ssh-rsa format. Other formats (such as ecdsa-sha2-nistp256) are not supported
Comment From: ryanjbaxter
Ah yes, I forgot about that 🤦♂️ it wouldnt work
Comment From: leonard520
Ah yes, I forgot about that 🤦♂️ it wouldnt work
@ryanjbaxter Sorry, a bit confusing here. Do you mean that Config Server doesn't support ecdsa
? But seems @habuma provided a method to configure a key generate by ecdsa in Config Server and made it work? Do I miss something?
Comment From: habuma
Indeed, creating an ecdsa key (with PEM formatting) worked. Not sure if this is a surprise for everyone, but I was able to make it work.
Comment From: leonard520
@ryanjbaxter Does it mean that Config Server doesn't officially support ecdsa
but it can work in this case?
Comment From: ryanjbaxter
I can't say for sure without looking into it
Comment From: leonard520
By further looking into the issue, I think there are two issues.
1. GitHub retires sha1 and Config Server doesn't support sha2 yet. Seems a higher version of jsch
fix the problem. I have tried v0.2.0 and it works with rsa
. Is there some plan for Config Server to upgrade the version?
More discussion here https://stackoverflow.com/questions/71489256/spring-cloud-config-server-github-sha-1-error
- If user have several repos, and each repo has its own ssh credential, Config Server will fail. Please check the below test result. Test materials: Private repo A in my account Private repo B in my account Private repo C in other's account Account key A in my account Account key B in my account Private repo C in other's account Deploy key A in repo A Deploy key B in repo B
Case A: repo A with account key A as main repo, works Case B: repo B with account key B as main repo, works Case C: repo C with account key C as main repo, works Case D: repo A with deploy key A as main repo, works Case E: repo B with deploy key B as main repo, works Case F: repo A with account key A as main repo, repo B with account key B as additional repo, works Case G: repo A with deploy key A as main repo, repo B with deploy key B as additional repo, NO works Case H: repo A with account key A as main repo, repo C with account key C as additional repo, NO works
It seems that Config Server can't work with multiple repos. The reason case F works is because the two account keys belong to me and either of them can be access to the two repos.
@ryanjbaxter Would you please help to check the second issue as well? Thanks a lot!
Comment From: ryanjbaxter
For 1 we are going to move to Apache MINA https://github.com/spring-cloud/spring-cloud-config/issues/1901
Im unsure on 2, we would need to look into that. I am kind of strapped for time at the moment so I am not sure when I will get to it.
Comment From: leonard520
By further looking into the issue, I think there are two issues.
- GitHub retires sha1 and Config Server doesn't support sha2 yet. Seems a higher version of
jsch
fix the problem. I have tried v0.2.0 and it works withrsa
. Is there some plan for Config Server to upgrade the version? More discussion here https://stackoverflow.com/questions/71489256/spring-cloud-config-server-github-sha-1-error- If user have several repos, and each repo has its own ssh credential, Config Server will fail. Please check the below test result. Test materials: Private repo A in my account Private repo B in my account Private repo C in other's account Account key A in my account Account key B in my account Private repo C in other's account Deploy key A in repo A Deploy key B in repo B
Case A: repo A with account key A as main repo, works Case B: repo B with account key B as main repo, works Case C: repo C with account key C as main repo, works Case D: repo A with deploy key A as main repo, works Case E: repo B with deploy key B as main repo, works Case F: repo A with account key A as main repo, repo B with account key B as additional repo, works Case G: repo A with deploy key A as main repo, repo B with deploy key B as additional repo, NO works Case H: repo A with account key A as main repo, repo C with account key C as additional repo, NO works
It seems that Config Server can't work with multiple repos. The reason case F works is because the two account keys belong to me and either of them can be access to the two repos.
@ryanjbaxter Would you please help to check the second issue as well? Thanks a lot!
I think the problem is caused by in extractNestedProperties
https://github.com/spring-cloud/spring-cloud-config/blob/d8f6341f336e8da78eaab424ca3a3e82431f83db/spring-cloud-config-server/src/main/java/org/springframework/cloud/config/server/ssh/SshUriPropertyProcessor.java#L59
It uses hostname
as sshUriPropertyMap
's key so in this case, two different github repos will be overridden to use one ssh configuration.
Comment From: ryanjbaxter
Maybe we can prefix the name with the repo name?
Comment From: leonard520
Maybe we can prefix the name with the repo name?
Yes, I think it can work.
Comment From: ryanjbaxter
The problem is that Jsch which we use to make SSH connections has no concept of a repo. Can you create a separate issue for that one, its unrelated to this issue.
Comment From: leonard520
Here is the issue to track the second problem https://github.com/spring-cloud/spring-cloud-config/issues/2105