Some request of a webclient instance does not resolve the ip address of hostname correct after it has changed by the dns

[stefan-g]

The web client instance connects (sometimes) to the old ip address of a domain name after the ip of the domain has changed by the dns. It looks like that the dns cache is not refreshed in all socket connections as it should. Only a restart of the service solves the problem.

Expected Behavior

When a ip address of a domain name changes, the ip of all socket addresses should be refreshed after the configured ttl time.

Actual Behavior

In some request the webclient connects to the old ip adress after it has changed by the dns server.

Steps to Reproduce

I can't reproduce it locally.

WebClient instance is cached in a spring service object and defined like that:

    HttpClient httpClient = HttpClient.create(resourceFactory.getConnectionProvider())
                                      .metrics(true, Function.identity())
                                      .resolver(nameResolverSpec -> nameResolverSpec.cacheMaxTimeToLive(Duration.ofSeconds(60)))
                                      .option(ChannelOption.CONNECT_TIMEOUT_MILLIS, 5000)
                                      .responseTimeout(Duration.ofMinutes(2));
    return WebClient.builder()
                    .clientConnector(new ReactorClientHttpConnector(httpClient))
                    .build();

network address cache is set at application startup like that:

  public static void main(String[] args) {
    Security.setProperty("networkaddress.cache.ttl", "60");
    Security.setProperty("networkaddress.cache.negative.ttl", "0");

    SpringApplication.run(Application.class, args);
  }

To monitor the ip-address i have defined a scheduler job that checks every hour (by calling InetAddress.getByName("https://MyHostname")) if the ip of the hostname has changed. InetAddress.getByName returns the correct ip address when it has changed. It looks to me that the wrong ip is not cached by InetAddress.

Possible Solution

?

Your Environment

Java 17 openjdk
Spring boot dependencies 3.2.8
Reactor-netty-core 1.1.21
Netty 4.1.111.Final
Linux os

[violetagg]

@stefan-g When the IP is changed you need to dispose the current connection pool, because it holds open connections to the old IP. You can do it with https://projectreactor.io/docs/netty/release/api/reactor/netty/resources/ConnectionProvider.html#disposeWhen-java.net.SocketAddress-

[stefan-g]

@violetagg Thanks for your answer. If i configure maxIdleTime and maxLifeTime of the ConnectionProvider to a shorter time should the open connections be closed?

[violetagg]

@stefan-g yes they will

[stefan-g]

Would it not make sense that the connection pool checks if the ip address of a domain name has changed when the connection is acquired? Could be a configurable feature

[violetagg]

@stefan-g We can think about some configuration but this has to be tightly coupled with DNS resolution configuration. Example let's say there is no expiration of the DNS cache but we check every time whether the IP is changed. Wdyt?
May be the DNS cache expiration can trigger something.

[stefan-g]

@violetagg i think that would make sense because if ttl of the networkaddress cache is too high InetAddress would also not get the new ip address.

The use case i have is as follow:
We have a data center that switches ip from time to time (normally during night time) between 2 locations. To prevent any down time on our side if have to define a short maxLifeTime (what most of the time is not needed) or monitor an ip change and call disposeWhen() as you mentioned.

[violetagg]

@stefan-g Let me mark this as an enhancement. If you would like to provide a PR for this feature, it would be great!

[stefan-g]

Probably not familiar enough with netty connection pool to provide a useful pr

[samueldlightfoot]

@violetagg What feature specifically are you thinking here? I may be able to help out.

[violetagg]

@samueldlightfoot I cannot find anything that can notify us for the cache expiration, so the only option will be to configure the DNS resolver's cache ttl and the connection pool's maxLifeTime with one and the same value.

[violetagg]

@samueldlightfoot Another thing that needs attention is that the fact that DNS resolver's cache ttl is reached, doesn't mean that the next result of the DNS resolution will be another IP, in most cases it will be the same IP.