Skip to content

Breaking change report - EDS #48373

@kimmikimmi

Description

@kimmikimmi

Is this the right place to submit this?

  • This is not a security vulnerability or a crashing bug
  • This is not a question about how to use Istio

Bug Description

We bumped up istio version from 1.14.3 to 1.20, and observed newly launched pod's ips are not properly resolved by old pods (registered as ServiceEntry).

$ kubectl --context pm -n service-ratelimit get pods -o wide
NAME                                        READY   STATUS    RESTARTS   AGE     IP              NODE                                               NOMINATED NODE   READINESS GATES
service-ratelimit-canary-666d66cc75-cmbnw   6/6     Running   0          12m     IPa   ip-....ap-northeast-2.compute.internal    <none>           <none>
service-ratelimit-canary-666d66cc75-vbbc4   6/6     Running   0          2m6s    IPb     ip-....ap-northeast-2.compute.internal    <none>           <none>

$ istioctl --context ph pc endpoint url-shortener-admin-staging-7c8f9d8d5-dmc5z.url-shortener-admin | grep 'IPb'
No result

I think this is because of the change, from this change, registering IPs from newly launched ip depends on PILOT_SEND_UNHEALTHY_ENDPOINTS flag, and this is false by default.

		if oie, exists := emap[nie.Address]; exists {
				// If endpoint exists already, we should push if it's health status changes.
				if oie.HealthStatus != nie.HealthStatus {
					needPush = true
				}
				newIstioEndpoints = append(newIstioEndpoints, nie)
			} else {
				// If the endpoint does not exist in shards that means it is a
				// new endpoint. Always send new endpoints even if they are not healthy.
				// This is OK since we disable panic threshold when SendUnhealthyEndpoints is enabled.
				// Without SendUnhealthyEndpoints we do not need this; headless services will trigger the push in the Kubernetes controller.
				if features.SendUnhealthyEndpoints.Load() {
					needPush = true
				}
				newIstioEndpoints = append(newIstioEndpoints, nie)
			}

Version

1.20, but happens from 1.19.4

Additional Information

We use ServiceEntry's workloadSelector to discover mapped ip to mesh internal domain.
No response

Metadata

Metadata

Assignees

No one assigned

    Type

    No type

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions