Setting up the AllowedHostsFilter for a server behind a load balancer

The AllowedHostsFilter documentation says:

You may find that some of your routes can not be used properly with allowed hosts filtering. This is commonly the case for load balancer health checks, which often use the IP address of the server as the host name

I wonder, and what’s the case when everything is behind a load balancer which implies the server is always called by the ip:port?

I would think that the filter is still useful while adding allowing all calls using the ip:port, or, is it safe to just disable it?