vCenter Appliance v6.5a “503 Service Unavailable” Error

I recently ran into the “503 Service Unavailable” message when trying to access vCenter in my homelab and thought this fix might be of interest to others. Please be sure to discuss this procedure with support prior to running any of the commands below in your environment. Also be sure to run a backup or snapshot as well prior to following the procedure below.

Symptoms

On the webclient, the web page shows the error: 503 Service Unavailable (Failed to connect to endpoint). . .
PowerCLI cannot connect to vCenter.

vpxd on vCenter Server 6.5 intermittently crashes with the following error stack in the vpxd log file:

2016-11-28T02:09:36.418Z error vpxd[7F27BD4A9700] [Originator@6876 sub=Default opID=HB-host-83@226512-71ec21d0] [VdbStatement] SQLError was thrown: “ODBC error: (23505) – ERROR: duplicate key value violates unique constraint “pk_vpx_vm_virtual_device”;
–> Error while executing the query” is returned when executing SQL statement “INSERT INTO VPX_VM_VIRTUAL_DEVICE (ID, DEVICE_KEY, CONN_AL_GU_CONTROL_FLAG, CONN_CONNECTED_FLAG, CONN_START_CONNECTED_FLAG, CONN_STATUS, DEVICE_INFO_SUMMARY, DEVICE_INFO_LABEL, DEVICE_CONTROLLER_KEY, DEVICE_UNIT_NUMBER, DEVICE_TYPE, DEVICE_TEXT) VALUES (?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?)”

Note:
The VMware vCenter Server 6.5 vpxd logs are located in the %ALLUSERSPROFILE%\VMWare\vCenterServer\logs\vmware-vpx folder.
The VMware vCenter Server Appliance vpxd 6.5 logs are located in the /var/log/vmware/vmware-vpx folder.

Resolution
Caution: Before making database modifications, ensure there is a working backup of the database or snapshot.

If the embedded postgres database is being used, locate the duplicate key in the postgresql logs:

postgresql.log
UTC 583a9c81.5814 127750 VCDB vc ERROR: duplicate key value violates unique constraint “pk_vpx_vm_virtual_device”
UTC 583a9c81.5814 127750 VCDB vc DETAIL: Key (id, device_key)=(361, 3002) already exists.

Note: The postgresql.log can be found in var/log/vmware/vpostgres

To connect to the embedded postgres database:

/opt/vmware/vpostgres/current/bin/psql -d VCDB -U postgres

To remove the duplicate key run the below command:
(modify the id and device_key according to your vpxd log below!):

DELETE FROM vc.vpx_vm_virtual_device where id=’361′ and device_key=’3002′;