Spring Cloud Config Issue with git pull in spring cloud config server when using basedir and running two instances of Config Server

I am writing my Spring cloud config server with git as a backend store. I have two instances of config server and below is my application.yml for config server. I am defining a basedir property for my git configuration so that git creates the local copy at this location.(We have a use case where we don't want config server to write a local git copy onto temporary location of OS which it do by default). Since I am using this basedir property and have two instances of config server running simultaneously, I am observing git lock issue on index.lock file may be because of concurrent read/write onto the basedir location of local copy.

spring:
  profiles:
    active: git
  cloud:
    config:
      server:
        git:
          refreshRate: 30
          uri: <my git uri>
          username: ${username}
          password: ${password}
          searchPaths: /*,/*/*,/*/*/*
          basedir: /project/configserver/cloned_copy
          force-pull: true
          order: 1

Below is the exception trace that I am getting on few of the config request calls:

java.lang.IllegalStateException: Cannot load environment at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:298) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:249) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.getLocations(MultipleJGitEnvironmentRepository.java:139) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.MultipleJGitEnvironmentRepository.findOne(MultipleJGitEnvironmentRepository.java:173) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:73) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:57) Caused by: org.eclipse.jgit.api.errors.JGitInternalException: Cannot lock /project/configserver/cloned_copy/.git/index at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:330) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.checkout(JGitEnvironmentRepository.java:426) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:282) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] ... 15 more Caused by: org.eclipse.jgit.errors.LockFailedException: Cannot lock /project/configserver/cloned_copy/.git/index at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:250) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:327) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.eclipse.jgit.dircache.DirCache.lock(DirCache.java:293) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.eclipse.jgit.lib.Repository.lockDirCache(Repository.java:1192) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:265) ~[org.eclipse.jgit-4.6.0.201612231935-r.jar:4.6.0.201612231935-r] at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.checkout(JGitEnvironmentRepository.java:426) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:282) ~[spring-cloud-config-server-3.0.5.jar:3.0.5] ... 15 more

Can someone suggest what extra needs to be done so as to avoid this locking issue on git index.lock file.

Question has been posted on stackoverflow too: https://stackoverflow.com/questions/72881154/issue-with-git-pull-in-spring-cloud-config-server-when-using-basedir-and-running

Comment From: ryanjbaxter

I am not sure what else we can do here to avoid this. Is there a reason both instances need to use the same base path?

Comment From: sohitjain02

@ryanjbaxter yes both instances will be using. I have application.yml with this basedir property defined. This will be wrapped in my jar file and this will be deployed on the server. We have two instances for our config server so this jar will be deployed to both instances. Now whenever a client demands property it will come to our config server so config server URl will be accessed for that and config server will try to use the git environment repository to return the config value. Now the request can go to any of the two instances and there can be multiple requests that can be served at the same time. Hence, both will try to read same basedir path (the local copy of git repository).

Also, the reason is , we understand that spring cloud config server always tries to return git configs from the locally cloned git copy (either copy created at basedir location or from temp directory if basedir is not defined). So in order to ensure 100% resiliency in case the remote git server goes down then, client applications are able to get the configs from locally cloned copy. If both instances have different locally cloned copies, there can be a case where both local copies may be out of sync and also temp directory is prone to getting wiped out through OS processes.

Comment From: ryanjbaxter

The problem is you have two different instances manipulating the git repo simultaneously.

What I would suggest is each instance have its own directory specified, you can do this by specifying the property via java options when running the jar. This avoids the problem all together.

I dont see how this would result in out of sync config, because the config server will fetch updates from the remote git repo before serving the request. I guess you could have one instance fetch updates and then the git server go down and the second instance now has an out of date local clone but I would imagine this is going to be incredibly rare.

Comment From: sohitjain02

@ryanjbaxter thanks. This worked by specifying different basedir's for both of the instances. Each instance will now clone/pull from their respective basedir paths.