PKS upgrade failed with error "2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image"
Article ID: 316814
Updated On:
Products
VMware
Issue/Introduction
Symptoms:
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
- PKS upgrade from 1.2.7 to 1.3.1 failed
- The upgrade is halted after upgrading two of the nodes in the cluster
- You see the following Bosh task reports errors:
Task 13565 | 23:39:22 | Preparing package compilation: Finding packages to compile (00:00:00)
Task 13565 | 23:39:25 | Updating instance worker: worker/4427fda7-d962-4ec5-a594-b3526c05394a (3) (canary) (00:02:34)31m
L Error: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m
Task 13565 | 23:41:59 | [31mError: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m
Task 13565 | 23:39:25 | Updating instance worker: worker/4427fda7-d962-4ec5-a594-b3526c05394a (3) (canary) (00:02:34)31m
L Error: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m
Task 13565 | 23:41:59 | [31mError: Action Failed get_task: Task 16e218ba-5984-4fd8-6633-52ea7a790bf7 result: 2 of 5 post-start scripts failed. Failed Jobs: telemetry-agent-image, kubelet. Successful Jobs: bosh-dns, sink-resources-images, wavefront-proxy-images.[0m
- You see messages similar to the following in the post-start.stderr.log:
worker/4427fda7-d962-4ec5-a594-b3526c05394a: /var/vcap/sys/log/telemetry-agent-image/post-start.stderr.log
Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-suspend.service.8.gz: no space left on device
Error processing tar file(exit status 1): open /usr/share/man/man1/watch.1.gz: no space left on device
Error processing tar file(exit status 1): symlink tset.1.gz /usr/share/man/man1/reset.1.gz: no space left on device
post-start.stderr.log (END)
Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-suspend.service.8.gz: no space left on device
Error processing tar file(exit status 1): open /usr/share/man/man1/watch.1.gz: no space left on device
Error processing tar file(exit status 1): symlink tset.1.gz /usr/share/man/man1/reset.1.gz: no space left on device
post-start.stderr.log (END)
- You see an error similar to the following when running the docker-load command:
docker load -i /var/vcap/packages/telemetry-agent-image/pkstelemetrybot_telemetry-agent:fda6005.tar
8241afc74c6f: Loading layer [==================================================>] 120.8MB/120.8MB
Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-remount-fs.service.8.gz: no space left on device
8241afc74c6f: Loading layer [==================================================>] 120.8MB/120.8MB
Error processing tar file(exit status 1): open /usr/share/man/man8/systemd-remount-fs.service.8.gz: no space left on device
Note: The preceding log excerpts are only examples. Date, time, and environmental variables may vary depending on your environment.
Environment
VMware PKS 1.x
Cause
- The local docker registry doesn't have enough free space
- Docker image registry is filled with dangling images
Resolution
To resolve the issue, identify the dangling images in the registry and remove them
To identify the dangling images, please follow the instructions noted in How to identify and remove dangling registry images in docker registry (68151) .
To identify the dangling images, please follow the instructions noted in How to identify and remove dangling registry images in docker registry (68151) .
Additional Information
Feedback
Yes
No
Powered by
