Spring Security OAuth2AuthenticationException: Provided token [xxxxx] isn't active

Summary

Using scdf 2.3.0 on k8s, I am getting a org.springframework.security.oauth2.core.OAuth2AuthenticationException: Provided token [] isn't active

launching

#!/bin/bash

KCHOST=http://localhost:8844
REALM=scdf23
CLIENT_ID=dataflow
CLIENT_SECRET=xxxx
UNAME=mk
PASSWORD=xxxx

ACCESS_TOKEN=`curl -s\
  -d "client_id=$CLIENT_ID" -d "client_secret=$CLIENT_SECRET" \
  -d "username=$UNAME" -d "password=$PASSWORD" \
  -d "grant_type=password" \
  "$KCHOST/auth/realms/$REALM/protocol/openid-connect/token"  | jq -r '.access_token' `

echo $ACCESS_TOKEN

and

java  -jar spring-cloud-dataflow-shell-2.3.0.RELEASE.jar --dataflow.uri=http://scdf2-server-data-flow-server:8080  --dataflow.username=dataflow --dataflow.password=xxxxx --dataflow.credentials-provider-command=./token.sh --dataflow.skip-ssl-validation=true   --debug

error shows:

WARNING - Problem connecting to the Spring Cloud Data Flow Server:
"Unable to access Data Flow Server at 'http://scdf2-server-data-flow-server:8080': 'org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized'. Unauthorized, did you forget to
Please double check your startup parameters and either restart the Data Flow Shell (with any missing configuration including security etc.) or target the Data Flow Server using the 'dataflow config server

Actual Behavior

Using scdf 2.3.0 shell, it seems that connect to server using oauth and keycloak.

scdf logging shows:

020-01-23 11:33:21.106 DEBUG 1 --- [nio-8080-exec-5] .s.a.DefaultAuthenticationEventPublisher : No event was found for the exception org.springframework.security.oauth2.core.OAuth2AuthenticationException
2020-01-23 11:33:21.106 DEBUG 1 --- [nio-8080-exec-5] .o.s.r.w.BearerTokenAuthenticationFilter : Authentication request for failed!
org.springframework.security.oauth2.core.OAuth2AuthenticationException: Provided token [eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJab1JGc0ZLVlQ4WHQ0ZlJSYzZUczRBT0N1LU9hVHdjdjFpMWpmQkZiQk1BIn0.eyJqdGkiOiJiOWM5YWExYy0yNDc3LTRmMzYtYWU0Ni1mYmIwYTRiYWNlOWIiLCJleHAiOjE1Nzk3NzI3NzYsIm5iZiI6MCwiaWF0IjoxNTc5NzcyNDc2LCJpc3MiOiJodHRwOi8va2V5Y2xvYWstaHR0cDo4ODQ0L2F1dGgvcmVhbG1zL3NjZGYyMyIsImF1ZCI6ImFjY291bnQiLCJzdWIiOiI1MWFkMzIzNC1mZDU1LTQ3MGEtOTkwOS05ZGU3NzlhNzI4OWEiLCJ0eXAiOiJCZWFyZXIiLCJhenAiOiJkYXRhZmxvdyIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6IjIyYzZkODNiLTM2NzctNGYwYi04NzI1LTdmNDllMDA5ZTg4OSIsImFjciI6IjEiLCJyZWFsbV9hY2Nlc3MiOnsicm9sZXMiOlsib2ZmbGluZV9hY2Nlc3MiLCJ1bWFfYXV0aG9yaXphdGlvbiJdfSwicmVzb3VyY2VfYWNjZXNzIjp7ImRhdGFmbG93Ijp7InJvbGVzIjpbInVtYV9wcm90ZWN0aW9uIiwic2NkZi1yb2xlIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6Im9wZW5pZCBkYXRhZmxvdy5tYW5hZ2UgZGF0YWZsb3cuY3JlYXRlIGRhdGFmbG93LnZpZXcgZGF0YWZsb3cuZGVwbG95IGRhdGFmbG93LnNjaGVkdWxlIHByb2ZpbGUgZGF0YWZsb3cubW9kaWZ5IGRhdGFmbG93LmRlc3Ryb3kgZW1haWwiLCJlbWFpbF92ZXJpZmllZCI6ZmFsc2UsImNsaWVudEhvc3QiOiIxMC40NS4wLjUiLCJjbGllbnRJZCI6ImRhdGFmbG93IiwidXNlcl9uYW1lIjoic2VydmljZS1hY2NvdW50LWRhdGFmbG93IiwicHJlZmVycmVkX3VzZXJuYW1lIjoic2VydmljZS1hY2NvdW50LWRhdGFmbG93IiwiY2xpZW50QWRkcmVzcyI6IjEwLjQ1LjAuNSJ9.Jv_o6hvwq2tTckfrzgdSTgXJSFmoJW3m4whBR1afCjIEIUjY8Tmaay-u0zZ8VSNqo-VpzdqQ5pqrN337uhRIOOkJSpgKV7hbE8I4c83Ug_Kt6BLwhuXipPk03CNvtaaO3DPtIH88bwEHEaSJIFg7_mQ3dtmeIid_qUzq8ontnEMh3IPyjaaLGrB5mHUV0JcA8oVvgfjuFrC1fTajAK3vDLhsnirBI3NKBX0gRxmMxT343Z8OKZVcO9k96O0zHHXkKZ7CzBwO_AeVFl_17YsAG9JK0zjJ0qEWDgcEKXfwJyxKc3H2yaDutFB2pGfpP4RceqMRG3F4N3ukUus6HEoTkQ] isn't active
        at org.springframework.security.oauth2.server.resource.authentication.OpaqueTokenAuthenticationProvider.authenticate(OpaqueTokenAuthenticationProvider.java:99)
        at org.springframework.security.authentication.ProviderManager.authenticate(ProviderManager.java:175)
        at org.springframework.security.oauth2.server.resource.web.BearerTokenAuthenticationFilter.doFilterInternal(BearerTokenAuthenticationFilter.java:124)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.ui.DefaultLogoutPageGeneratingFilter.doFilterInternal(DefaultLogoutPageGeneratingFilter.java:52)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.ui.DefaultLoginPageGeneratingFilter.doFilter(DefaultLoginPageGeneratingFilter.java:216)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:200)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.oauth2.client.web.OAuth2AuthorizationRequestRedirectFilter.doFilterInternal(OAuth2AuthorizationRequestRedirectFilter.java:160)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.authentication.logout.LogoutFilter.doFilter(LogoutFilter.java:116)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.header.HeaderWriterFilter.doHeadersAfter(HeaderWriterFilter.java:92)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:77)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:105)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:334)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:56)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:119)

and from the shell:

Welcome to the Spring Cloud Data Flow shell. For assistance hit TAB or type "help".
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - HTTP GET http://scdf2-server-data-flow-server:8080/security/info
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - Accept=[application/json, application/*+json]
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - Response 200 OK
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - Reading to [org.springframework.cloud.dataflow.rest.resource.security.SecurityInfoResource]
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - HTTP GET http://scdf2-server-data-flow-server:8080
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - Accept=[application/json, application/*+json]
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.HttpLogging:147 - Response 401 UNAUTHORIZED
2020-01-23T11:32:56+0100 DEBUG Spring Shell o.s.w.c.HttpMessageConverterExtractor:102 - Reading to [org.springframework.hateoas.mediatype.vnderrors.VndErrors]
WARNING - Problem connecting to the Spring Cloud Data Flow Server:
"Unable to access Data Flow Server at 'http://scdf2-server-data-flow-server:8080': 'org.springframework.web.client.HttpClientErrorException$Unauthorized: 401 Unauthorized'. Unauthorized, did you forget to
Please double check your startup parameters and either restart the Data Flow Shell (with any missing configuration including security etc.) or target the Data Flow Server using the 'dataflow config server

Expected Behavior

Get a valid token with the user client...

Configuration

scdf 2.3.0, keycloak 8.0.1 I think, Spring Boot 2.2.2.RELEASE -> (Spring Security 5.2.1.RELEASE)

Comment From: jgrandja

@eskuai Was your intention to log this issue to the spring-cloud-dataflow repo?

Looking at the stacktrace when OpaqueTokenAuthenticationProvider.authenticate() calls the Introspection Endpoint on the provider (Keycloak?), the provider is asserting that the token in not active. This does not look like an issue with Spring Security or Spring DataFlow.

To confirm, after you obtain the token from the curl command, try to access the endpoint using Postman (or curl) with the same token. I suspect you will get the same error.

Comment From: eskuai

Hi @jgrandja

I've opened this issue in scdf github, and scdf team says that this problem is related on my keycloak config.

Can you explain me as simple as possible why can it happens , for example, scdf and skipper using same from keycloak token, is it means that two process can use the same token?

I need help to understand that what is this issue and why can happens,and try to back with more details to scdf team.

I don't know what meaning token is not active. Who decide if token is active or not? Process or keycloak? Can i Say that a token can be reuse?

Is it possible that a dashboard use a Lot of tokens for a one web session? Scdf create a lot of sessions for same user and client.

What can i test to get info for scdf team or fix ir it is my mistake with kc config?

Please , if you can help me

Comment From: eskuai

And, if i use curl with a token, i can call to scdf for a time... An then, 401... Token is not active. Timeout for token? Kc config token time to live?

Comment From: jgrandja

This issue certainly seems related to your Keycloak config.

I don't know what meaning token is not active. Who decide if token is active or not? Process or keycloak?

The provider is always the source of truth. It mints tokens and validates tokens. So your Keycloak instance for some reason is returning "not active" from the Introspection endpoint.

You should report this to Keycloak and have them help you troubleshoot this issue. You can report here.

I also found this script that you can use to test on your own. You can modify the script to obtain the token and then pass the token to the Introspection Endpoint to see what it returns.

I'm going to close this issue since it's related to Keycoak.

Comment From: eskuai

HI @jgrandja

For your curiosity only,

./tokenintrospect.sh 
Handling connection for 8844
Response={ "access_token": "eyJhbGciOiJSUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICJab1JGc0ZLVlQ4WHQ0ZlJSYzZUczRBT0N1LU9hVHdjdjFpMWpmQkZiQk1BIn0.eyJqdGkiOiI0YWEyYTg5Yi04ODFlLTQ5ZmYtOTVmZi0wNzA3MjQxOTZjZDYiLCJleHAiOjE1Nzk4ODQwMjgsIm5iZiI6MCwiaWF0IjoxNTc5ODgzNDI4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0Ojg4NDQvYXV0aC9yZWFsbXMvc2NkZjIzIiwiYXVkIjoiYWNjb3VudCIsInN1YiI6IjQwZTllNzM4LWExMWMtNDUwOC1hYjMyLWUxODYwZWE3NDc4MiIsInR5cCI6IkJlYXJlciIsImF6cCI6ImRhdGFmbG93IiwiYXV0aF90aW1lIjowLCJzZXNzaW9uX3N0YXRlIjoiYzkxYmI3YTQtNzdkMC00NjBhLWI5ZGUtNGY4NWE2MzY1YzZhIiwiYWNyIjoiMSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiZGF0YWZsb3ciOnsicm9sZXMiOlsic2NkZi1yb2xlIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImRhdGFmbG93Lm1hbmFnZSBkYXRhZmxvdy5jcmVhdGUgZGF0YWZsb3cudmlldyBkYXRhZmxvdy5kZXBsb3kgZGF0YWZsb3cuc2NoZWR1bGUgcHJvZmlsZSBkYXRhZmxvdy5tb2RpZnkgZGF0YWZsb3cuZGVzdHJveSBlbWFpbCIsImVtYWlsX3ZlcmlmaWVkIjpmYWxzZSwidXNlcl9uYW1lIjoibWsiLCJwcmVmZXJyZWRfdXNlcm5hbWUiOiJtayJ9.gdA71Z-O4sTlA6chTBf_CiT4AQCslpI3TdHkKUcSEbYs5QHn_FoLXPDX9B7E6lFQWZnEfYn8UST2dL8WRUskRS3t2qDNdJDyLDNeOjJzUG12ZxmRdjis2AI3qvtL-5J6ucDUT9z2---eqGvdTGQF3wl7KZMAn6dlJ7Bt8XbxaNDEIe9HNDIBWMUcEIL4uqFHsRcyezVe4FCanH17kzXnxP6DDbb_yz2DJp2xkuKiZDbITLvoHdEPolcDtcsTfLOARvOxstegT-aKwxlNoLwezBlNHYmR3XgsnILfYhF4MnzqDLPd9z-3OzDtiToTqEazuSrsN_W5GmqQ8YmS5_W5Uw", "expires_in": 600, "refresh_expires_in": 3600, "refresh_token": "eyJhbGciOiJIUzI1NiIsInR5cCIgOiAiSldUIiwia2lkIiA6ICIzMGI0NmE4Mi02N2FlLTQzZTItYjE0MS03OGVhYzIzY2UwYmIifQ.eyJqdGkiOiI4NDBlODU1NS0yZDRiLTRmYmEtOWE3My1mMGFlNzVhYjgwYmUiLCJleHAiOjE1Nzk4ODcwMjgsIm5iZiI6MCwiaWF0IjoxNTc5ODgzNDI4LCJpc3MiOiJodHRwOi8vbG9jYWxob3N0Ojg4NDQvYXV0aC9yZWFsbXMvc2NkZjIzIiwiYXVkIjoiaHR0cDovL2xvY2FsaG9zdDo4ODQ0L2F1dGgvcmVhbG1zL3NjZGYyMyIsInN1YiI6IjQwZTllNzM4LWExMWMtNDUwOC1hYjMyLWUxODYwZWE3NDc4MiIsInR5cCI6IlJlZnJlc2giLCJhenAiOiJkYXRhZmxvdyIsImF1dGhfdGltZSI6MCwic2Vzc2lvbl9zdGF0ZSI6ImM5MWJiN2E0LTc3ZDAtNDYwYS1iOWRlLTRmODVhNjM2NWM2YSIsInJlYWxtX2FjY2VzcyI6eyJyb2xlcyI6WyJvZmZsaW5lX2FjY2VzcyIsInVtYV9hdXRob3JpemF0aW9uIl19LCJyZXNvdXJjZV9hY2Nlc3MiOnsiZGF0YWZsb3ciOnsicm9sZXMiOlsic2NkZi1yb2xlIl19LCJhY2NvdW50Ijp7InJvbGVzIjpbIm1hbmFnZS1hY2NvdW50IiwibWFuYWdlLWFjY291bnQtbGlua3MiLCJ2aWV3LXByb2ZpbGUiXX19LCJzY29wZSI6ImRhdGFmbG93Lm1hbmFnZSBkYXRhZmxvdy5jcmVhdGUgZGF0YWZsb3cudmlldyBkYXRhZmxvdy5kZXBsb3kgZGF0YWZsb3cuc2NoZWR1bGUgcHJvZmlsZSBkYXRhZmxvdy5tb2RpZnkgZGF0YWZsb3cuZGVzdHJveSBlbWFpbCJ9.YVvP69FhTaqnjhI6txz0RGS_kdrofJGo-SOJxQlzyUU", "token_type": "bearer", "not-before-policy": 1579801147, "session_state": "c91bb7a4-77d0-460a-b9de-4f85a6365c6a", "scope": "dataflow.manage dataflow.create dataflow.view dataflow.deploy dataflow.schedule profile dataflow.modify dataflow.destroy email" }
Handling connection for 8844
{
  "jti": "4aa2a89b-881e-49ff-95ff-070724196cd6",
  "exp": 1579884028,
  "nbf": 0,
  "iat": 1579883428,
  "iss": "http://localhost:8844/auth/realms/scdf23",
  "aud": "account",
  "sub": "40e9e738-a11c-4508-ab32-e1860ea74782",
  "typ": "Bearer",
  "azp": "dataflow",
  "auth_time": 0,
  "session_state": "c91bb7a4-77d0-460a-b9de-4f85a6365c6a",
  "preferred_username": "mk",
  "email_verified": false,
  "acr": "1",
  "realm_access": {
    "roles": [
      "offline_access",
      "uma_authorization"
    ]
  },
  "resource_access": {
    "dataflow": {
      "roles": [
        "scdf-role"
      ]
    },
    "account": {
      "roles": [
        "manage-account",
        "manage-account-links",
        "view-profile"
      ]
    }
  },
  "scope": "dataflow.manage dataflow.create dataflow.view dataflow.deploy dataflow.schedule profile dataflow.modify dataflow.destroy email",
  "user_name": "mk",
  "client_id": "dataflow",
  "username": "mk",
  "active": true
}