PKS cluster creation fails with "Error: Timed out sending ‘get_state’ to worker node after 45 seconds"
search cancel

PKS cluster creation fails with "Error: Timed out sending ‘get_state’ to worker node after 45 seconds"

book

Article ID: 345569

calendar_today

Updated On:

Products

VMware

Issue/Introduction

Symptoms:
  • PKS cluster creation is failing with Error: Timed out sending ‘get_state’ to worker node after 45 seconds.

  • You see that Master and Worker nodes are created and all the processes are running fine.

  • You are using 172.17.x.x/36 or one of the PKS Reserved subnet for PKS Management plane.

  • When you run Bosh task xxx for cluster cluster creation, you see the output similar to:

Task 204 | 18:08:12 | Preparing deployment: Preparing deployment
Task 204 | 18:08:14 | Warning: DNS address not available for the link provider instance: pivotal-container-s-ab25c4850e38
Task 204 | 18:08:14 | Warning: DNS address not available for the link provider instance: pivotal-container-s0-ab25c4850e38
Task 204 | 18:08:14 | Warning: DNS address not available for the link provider instance: pivotal-container-s-ab25c4850e38
Task 204 | 18:08:26 | Preparing deployment: Preparing deployment (00:00:14)
Task 204 | 18:08:33 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 204 | 18:08:33 | Creating missing vms: master/33c9b266-30e7-4f0f-9bd1-f51e2aed9577 (0)
Task 204 | 18:08:33 | Creating missing vms: worker/898c20e9-5962-415e-90ca-0e1e32bb662f (0)
Task 204 | 18:09:40 | Creating missing vms: master/33c9b266-30e7-4f0f-9bd1-f51e2aed9577 (0) (00:01:07)
Task 204 | 18:09:46 | Creating missing vms: worker/898c20e9-5962-415e-90ca-0e1e32bb662f (0) (00:01:13)
Task 204 | 18:09:47 | Updating instance master: master/33c9b266-30e7-4f0f-9bd1-f51e2aed9577 (0) (canary)(00:02:19)
Task 204 | 18:12:06 | Updating instance worker: worker/898c20e9-5962-415e-90ca-0e1e32bb662f (0) (canary)(00:05:09)
                  L Error: Timed out sending ‘get_state’ to 040fdd68-b97f-46e6-ad69-93e2bc598380 after 45 seconds
Task 204 | 18:17:15 | Error: Timed out sending ‘get_state’ to 040fdd68-b97f-46e6-ad69-93e2bc598380 after 45 seconds

 


Environment

VMware PKS 1.x

Cause

This issue occurs when your PKS Management plane subnet conflicts with the Reserved IP Blocks.This restriction applies to all virtual machines (VMs) deployed during the PKS installation process, including the PKS control plane, Ops Manager, BOSH Director, and Harbor Registry. If you use them already, then Kubernetes worker nodes cannot reach Harbor or internal Kubernetes services and cluster creation will fail.

Resolution

Ensure to not to use the below Reserved Subnets any where in your PKS environment. If you use them already, you need to re-deploy the PKS environment by choosing the proper subnets. For more information, see Plan IP Blocks section under Planning, Preparing, and Configuring NSX-T for PKS.

The Docker daemon on the Kubernetes worker node uses the subnet in the following CIDR range. Do not use IP addresses in the following CIDR range:

  • 172.17.0.1/16

  • 172.18.0.1/16

  • 172.19.0.1/16

  • 172.20.0.1/16

  • 172.21.0.1/16

  • 172.22.0.1/16

If PKS is deployed with Harbor, Harbor uses the following CIDR ranges for its internal Docker bridges. Do not use IP addresses in the following CIDR range:

  • 172.18.0.0/16

  • 172.19.0.0/16

  • 172.20.0.0/16

  • 172.21.0.0/16

  • 172.22.0.0/16

Each Kubernetes cluster uses the following subnet for Kubernetes services. Do not use the following IP block for the Nodes IP Block:

  • 10.100.200.0/24