Modified index could not be written Could not delete file /git-repo/.git/index.lock
Hi everyone, we experience in our spring-cloud-config server following issue. I tried to search all issues here but none has mentioned anything regarding corrupted index.lock in local copy of the configuration repository. Our spring cloud server config properties
spring:
cloud:
config:
server:
git:
uri: ${GIT_ENDPOINT:https://our.bitbucket.server/scm/ic/configuration-repo.git}
username: ${GIT_USER}
password: ${GIT_PWD}
refresh-rate: 10
force-pull: true
clone-on-start: true
default-label: ${DEFAULT_BRANCH:develop}
delete-untracked-branches: true
basedir: /git-repo // it runs inside docker and we create extra directory with persmissios here RUN mkdir -p /git-repo RUN chmod 777 /git-repo, but it does have the same effect as no defining baseDir thus fallback to /tmp
We use force pull strategy, we tried as some suggestions provide the dedicated baseDir rather than default /tmp but the issues still persist
2021-04-19 06:31:39.748 ERROR 1 --- [o-8080-exec-572] o.e.jgit.internal.storage.file.LockFile : Unlocking LockFile '/git-repo/.git/index.lock' failed
java.io.IOException: Could not delete file /git-repo/.git/index.lock
at org.eclipse.jgit.util.FileUtils.delete(FileUtils.java:219) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.internal.storage.file.LockFile.unlock(LockFile.java:520) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.internal.storage.file.LockFile.commit(LockFile.java:456) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCache.commit(DirCache.java:740) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.BaseDirCacheEditor.commit(BaseDirCacheEditor.java:278) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCacheBuilder.commit(DirCacheBuilder.java:1) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:603) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:484) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:275) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.checkout(JGitEnvironmentRepository.java:423) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:279) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:244) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:64) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.EnvironmentEncryptorEnvironmentRepository.findOne(EnvironmentEncryptorEnvironmentRepository.java:61) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.resource.ResourceController.retrieve(ResourceController.java:138) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.resource.ResourceController.retrieve(ResourceController.java:106) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at jdk.internal.reflect.GeneratedMethodAccessor84.invoke(Unknown Source) ~[na:na]
org.eclipse.jgit.errors.IndexWriteException: Modified index could not be written
at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:604) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:484) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:275) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.checkout(JGitEnvironmentRepository.java:423) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.refresh(JGitEnvironmentRepository.java:279) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.JGitEnvironmentRepository.getLocations(JGitEnvironmentRepository.java:244) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.AbstractScmEnvironmentRepository.findOne(AbstractScmEnvironmentRepository.java:55) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.CompositeEnvironmentRepository.findOne(CompositeEnvironmentRepository.java:64) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.environment.EnvironmentEncryptorEnvironmentRepository.findOne(EnvironmentEncryptorEnvironmentRepository.java:61) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.resource.ResourceController.retrieve(ResourceController.java:138) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
at org.springframework.cloud.config.server.resource.ResourceController.retrieve(ResourceController.java:106) ~[spring-cloud-config-server-3.0.2.jar:3.0.2]
pom.xml excerpt
<dependencies>
<dependency>
<groupId>org.springframework.boot</groupId>
<artifactId>spring-boot-starter-actuator</artifactId>
</dependency>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-config-server</artifactId>
</dependency>
</dependencies>
<dependencyManagement>
<dependencies>
<dependency>
<groupId>org.springframework.cloud</groupId>
<artifactId>spring-cloud-dependencies</artifactId>
<version>2020.0.1</version>
<type>pom</type>
<scope>import</scope>
</dependency>
</dependencies>
</dependencyManagement>
Thank you in advance for any suggestion
Comment From: marecica2
I'm not sure if it relates somehow to Bitbucket merge strategy.
We are using -no-ff merge strategy on the configuration repository and we dont allow rewriting history neither force pushes into develop and master branches
Merge commit
--no-ff
DEFAULT
(default)
Always create a new merge commit and update the target branch to it, even if the source branch is already up to date with the target branch.
We do have after merging PRs as well lots of breaking CheckoutConflictException like this
org.eclipse.jgit.errors.CheckoutConflictException: Checkout conflict with files:
NP/public/schema.json
public/locales/en/resource.json
public/locales/nl/resource.json
at org.eclipse.jgit.dircache.DirCacheCheckout.doCheckout(DirCacheCheckout.java:516) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.dircache.DirCacheCheckout.checkout(DirCacheCheckout.java:484) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
at org.eclipse.jgit.api.CheckoutCommand.call(CheckoutCommand.java:275) ~[org.eclipse.jgit-5.1.3.201810200350-r.jar:5.1.3.201810200350-r]
Comment From: spencergibb
Can you recreate this issue outside of a docker image?
Comment From: spring-cloud-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: marecica2
Can you recreate this issue outside of a docker image?
Unfortunately not. We run it inside k8s cluster and all changes on the underlying properties repository Our workaround for this issue is a custom aspectJ interceptor that does hard reset manually, which works but of couser isn't optimal solution. Is it known to happen only inside docker? Thank you for investigation
` @Component @Aspect @RequiredArgsConstructor @Slf4j public class ConflictAspect {
final private EnvironmentRepository repository;
@Around("execution(* org.springframework.cloud.config.server.resource.ResourceController.retrieve(..)) && args(name, profile, label, request, resolvePlaceholders)")
public Object interceptCall(ProceedingJoinPoint pjp, String name, String profile, String label,
ServletWebRequest request, boolean resolvePlaceholders) throws Throwable {
try {
return pjp.proceed();
} catch (Throwable err) {
if (err instanceof NoSuchResourceException) {
log.warn(err.getMessage());
throw err;
}
log.warn("Intercepted exception while retrieving git content", err);
try {
log.info("Trying to reset the repo..");
repository.reset(label);
log.info("Done repository reset");
} catch (Exception e) {
log.error("Unable to reset the repository");
throw e;
}
log.info("Retrying retrieval after repo reset...");
Object result = pjp.proceed();
log.info("Done.");
return result;
}
}
} `
Comment From: ryanjbaxter
Could you try this with Spring Cloud 2021.0.x or Spring Cloud 2022.0.x?
Comment From: spring-cloud-issues
If you would like us to look at this issue, please provide the requested information. If the information is not provided within the next 7 days this issue will be closed.
Comment From: spring-cloud-issues
Closing due to lack of requested feedback. If you would like us to look at this issue, please provide the requested information and we will re-open the issue.