top of page

vCenter Web UI not accessible Error no healthy upstream.

Suddenly our vCenter in Production Environment stop working and had an web ui message "no healthy upstream"



Log Analysis:


From the logs, we confirmed this is because of vmdird being in read-only state. We found out that vsphere-ui is not starting because pre start command is failing on "create_and_assign_vsphere_client_solution_user_role":

2023-01-20T02:27:38.208Z Wa(03)+ host-1972   File "/usr/lib/vmware-vsphere-ui/firstboot/vsphere_ui_prestart.py", line 80, in <module>

2023-01-20T02:27:38.208Z Wa(03)+ host-1972     raise ex

2023-01-20T02:27:38.208Z Wa(03)+ host-1972   File "/usr/lib/vmware-vsphere-ui/firstboot/vsphere_ui_prestart.py", line 77, in <module>

2023-01-20T02:27:38.208Z Wa(03)+ host-1972     create_and_assign_vsphere_client_solution_user_role(VSPHERE_UI_FIRSTBOOT_CONFIG_DIR)

2023-01-20T02:27:38.208Z Wa(03)+ host-1972   File "/usr/lib/vmware-vsphere-ui/firstboot/solution_user_permission_utils.py", line 59, in create_and_assign_vsphere_client_solution_user_role


Checking Authorization we find that updating of role is failing due to ldap failed command:

2023-01-20T15:27:38.189+13:00 [dataservice-1  ERROR com.vmware.cis.core.authz.accesscontrol.impl.AuthorizationServiceInternalImpl  opId=ac7e7b11-deb5-4687-9608-42285a1bef4c] Store Exception

com.vmware.cis.core.authz.exception.StoreException: java.lang.Exception: Failed to add entry to lotus

From vmdird logs, ldap modify operations are failing because vmdird is in read only mode:

2023-01-20T15:27:38.194769+13:00 err vmdird  t@139636651300608: InternalModifyEntry: VdirExecutePostModifyCommitPlugins - code(9114)

2023-01-20T15:27:38.194851+13:00 err vmdird  t@139636651300608: VmDirSendLdapResult: Request (Modify), Error (LDAP_UNWILLING_TO_PERFORM(53)), Message (Server in read-only mode), (0) socket (127.0.0.1)


Issue:

• vSphere-ui in stopped state


Findings:

• We found out that the vsphere-ui is in stopped state.


Fix:

• We run vdcadmintool to confirm the vmdird state

• Login to VCSA Appliance through SSH then Run this /usr/lib/vmware-vmdir/bin/vdcadmintool and select Number 6 to get the vmdir state  


Run ./fixpsc.sh script on the problematic vCenter with the healthy vCenter as reference. After this, we were able to start vsphere-ui service.

• Run one more time this /usr/lib/vmware-vmdir/bin/vdcadmintool and select Number 6 to get the vmdir state

 • Now we can see that the VmDir State is Normal

• At this point of time you can now access your vCenter Web UI successfully


Note: This is my personal and owned experience that I love to share in the VMware Community. I hope you appreciate it, thank you

Comments


bottom of page