SpringBoot Auto configuration for data-elasticsearch does not configure mapper that can handle MapDocument payload

Using default data-elasticsearch auto configuration the mapper used to convert payloads is not configured to handle org.springframework.data.elasticsearch.core.document.MapDocument which leads to errors when adding data.

jakarta.json.JsonException: Cannot find a serializer for type org.springframework.data.elasticsearch.core.document.MapDocument. Consider using a full-featured JsonpMapper
    at co.elastic.clients.json.SimpleJsonpMapper.serialize(SimpleJsonpMapper.java:113) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.json.JsonpUtils.serialize(JsonpUtils.java:146) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.elasticsearch.core.IndexRequest.serialize(IndexRequest.java:290) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.json.JsonpMapperBase$JsonpSerializableSerializer.serialize(JsonpMapperBase.java:111) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.json.JsonpMapperBase$JsonpSerializableSerializer.serialize(JsonpMapperBase.java:108) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.json.SimpleJsonpMapper.serialize(SimpleJsonpMapper.java:110) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.transport.rest_client.RestClientTransport.prepareLowLevelRequest(RestClientTransport.java:214) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.transport.rest_client.RestClientTransport.performRequest(RestClientTransport.java:145) ~[elasticsearch-java-8.4.3.jar:na]
    at co.elastic.clients.elasticsearch.ElasticsearchClient.index(ElasticsearchClient.java:946) ~[elasticsearch-java-8.4.3.jar:na]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.lambda$doIndex$6(ElasticsearchTemplate.java:213) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.execute(ElasticsearchTemplate.java:538) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.client.elc.ElasticsearchTemplate.doIndex(ElasticsearchTemplate.java:213) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.core.AbstractElasticsearchTemplate.save(AbstractElasticsearchTemplate.java:204) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository.lambda$save$5(SimpleElasticsearchRepository.java:172) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository.executeAndRefresh(SimpleElasticsearchRepository.java:344) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at org.springframework.data.elasticsearch.repository.support.SimpleElasticsearchRepository.save(SimpleElasticsearchRepository.java:172) ~[spring-data-elasticsearch-5.0.0-RC1.jar:5.0.0-RC1]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[na:na]
    at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77) ~[na:na]
    at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) ~[na:na]
    at java.base/java.lang.reflect.Method.invoke(Method.java:568) ~[na:na]

To work around this issue it is possible to provide ElasticsearchConfiguration as follows

@Configuration
static class Config extends ElasticsearchConfiguration {
    @Override
    public ClientConfiguration clientConfiguration() {
        return ClientConfiguration.localhost(); // or any other config
    }
}

The above however has the downside, that it breaks when when configuration is created via AOT.

Field repository in com.example.data.elasticsearch.CLR required a bean named 'elasticsearchTemplate' that could not be found.

The injection point has the following annotations:
    - @org.springframework.beans.factory.annotation.Autowired(required=true)

Action:

Consider defining a bean named 'elasticsearchTemplate' in your configuration.

The behaviour can be reproduced using the data-elasticsearch sample from this fork. Just add/remove the configuration in DataElasticsearchApplication

@sothawo maybe you can help with more insights on required/default mapping configuration.

Comment From: sothawo

I don't know how the ElasticsearchTransport is initialized in autocnfiguration, but the SimpleJsonpMapper seems not to be meant for mapping the application's data (https://github.com/elastic/elasticsearch-java/blob/main/java-client/src/main/java/co/elastic/clients/json/SimpleJsonpMapper.java#L29-L35):

/**
 * A simple implementation of <code>JsonpMapper</code> that only handles classes of the Java API client.
 * <p>
 * To handle application classes serialization and deserialization, consider using <code>JacksonJsonpMapper</code> or
 * <code>JsonbJsonpMapper</code>.
 */
public class SimpleJsonpMapper extends JsonpMapperBase {

In Spring Data Elasticsearch, when setting up the ElasticsearchTransport I explicitly pass in a JacksonJsonpMapper and had no problems with that.

As for that AOT error, I'm not into that, but the ElasticsearchConfiguration should provide that.

Comment From: wilkinsona

Boot will auto-configure the "best" JsonpMapper that it can. In order of preference, those are:

• JacksonJsonpMapper
• JsonbJsonpMapper
• SimpleJsonpMapper

The smoke test is left with the simple mapper as the other two could not be auto-configured:

   ElasticsearchClientConfigurations.JacksonJsonpMapperConfiguration:
      Did not match:
         - @ConditionalOnBean (types: com.fasterxml.jackson.databind.ObjectMapper; SearchStrategy: all) did not find any beans of type com.fasterxml.jackson.databind.ObjectMapper (OnBeanCondition)

   ElasticsearchClientConfigurations.JsonbJsonpMapperConfiguration:
      Did not match:
         - @ConditionalOnBean did not find required type 'jakarta.json.bind.Jsonb' (OnBeanCondition)
         - @ConditionalOnBean (types: jakarta.json.bind.Jsonb; SearchStrategy: all) did not find any beans of type jakarta.json.bind.Jsonb (OnBeanCondition)

Auto-configuration of an ObjectMapper requires spring-web to be on the classpath as, unfortunately, that's where Framework's MappingJackson2ObjectMapperBuilder lives. Given that spring-data-elasticsearch already depends on Jackson, I think it would make sense for us to make sure that a dependency on spring-boot-starter-data-elasticsearch gives an app all of the dependencies necessary for Boot to auto-configure Jackson, probably by adding spring-boot-starter-json to spring-boot-starter-data-elasticsearch.

Comment From: wilkinsona

The above however has the downside, that it breaks when when configuration is created via AOT

This looks like a Framework bug. elasticsearchTemplate is an alias for the elastichsearchOperations bean. This alias is lost during AOT processing. I've opened https://github.com/spring-projects/spring-framework/issues/29391.

Comment From: wilkinsona

I've tried running the smoke test with a dependency on spring-boot-starter-json and without the ElasticsearchConfiguration sub-class. It fails to start but with a different error:

  .   ____          _            __ _ _
 /\\ / ___'_ __ _ _(_)_ __  __ _ \ \ \ \
( ( )\___ | '_ | '_| | '_ \/ _` | \ \ \ \
 \\/  ___)| |_)| | | | | || (_| |  ) ) ) )
  '  |____| .__|_| |_|_| |_\__, | / / / /
 =========|_|==============|___/=/_/_/_/
 :: Spring Boot ::       (v3.0.0-SNAPSHOT)

2022-10-27T13:27:55.653+01:00  INFO 39218 --- [           main] c.e.d.e.DataElasticsearchApplication     : Starting AOT-processed DataElasticsearchApplication using Java 17.0.4 on wilkinsona-a01.vmware.com with PID 39218 (/Users/awilkinson/dev/spring-projects/spring-aot-smoke-tests/data-elasticsearch/build/libs/data-elasticsearch.jar started by awilkinson in /Users/awilkinson/dev/spring-projects/spring-aot-smoke-tests)
2022-10-27T13:27:55.674+01:00  INFO 39218 --- [           main] c.e.d.e.DataElasticsearchApplication     : No active profile set, falling back to 1 default profile: "default"
2022-10-27T13:27:57.438+01:00  WARN 39218 --- [           main] .m.SimpleElasticsearchPersistentProperty : Unsupported type 'class java.lang.String' for date property 'date'.
2022-10-27T13:27:58.676+01:00  WARN 39218 --- [           main] o.s.c.support.GenericApplicationContext  : Exception encountered during context initialization - cancelling refresh attempt: org.springframework.beans.factory.BeanCreationException: Error creating bean with name 'elasticsearchAuditingHandler': Cannot create inner bean '(inner bean)#710b18a6' of type [org.springframework.data.elasticsearch.config.PersistentEntitiesFactoryBean] while setting constructor argument
2022-10-27T13:27:58.738+01:00 ERROR 39218 --- [           main] o.s.b.d.LoggingFailureAnalysisReporter   : 

***************************
APPLICATION FAILED TO START
***************************

Description:

A component required a bean named '(inner bean)#710b18a6' that could not be found.


Action:

Consider defining a bean named '(inner bean)#710b18a6' in your configuration.

It looks like there's something in the auditing support with which AOT processing does not cope. That will have to be addressed in Spring Data Elastichsearch and/or Spring Framework.

Comment From: sothawo

https://github.com/christophstrobl/spring-aot-smoke-tests/blob/issue/data-elasticsearch-sample/data-elasticsearch/src/main/java/com/example/data/elasticsearch/Conference.java#L41 looks strange to me, a String that has a store type of Date?

Comment From: christophstrobl

@sothawo might be worth having a look at what as been done in data-mongo (9b1365) to get rid of the inner beans for auditing. I think a similar arrangement should work here too.

Comment From: sothawo

I'll check, but won't find time before tomorrow evening

Comment From: sothawo

@christophstrobl I am trying to get this running locally in order to test changes. I have installed a modified version of Spring Data Elasticsearch (version 5.0.0-PJ) in my local .m2 repository with a normal ./mvnw install but I fail to get this pulled in the gradle setup of the spring-aot-smoke-tests project I cloned from your Github account.

I added

repositories {
  mavenLocal()
}

and set the explicit dependency implementation("org.springframework.data:spring-data-elasticsearch:5.0.0-PJ") but still the RC1 version is pulled in. And I didn't find a property that I could set in ext[] to override the version of Spring Data Elasticsearch.

How can I set this up to have it running with my local version?

Comment From: sothawo

Ok, got it running now with:

repositories {
  mavenLocal()
}

dependencies {
  constraints {
    implementation("org.springframework.data:spring-data-elasticsearch:5.0.0-PJ!!")
  }
}