vCenter Fix – The Host is already connected to vCenter

Recently I worked on an issue where a coworker changed the IP and hostname of a ESXi Host without removing it from vCenter first. The old host was removed from vCenter but it was while the host was disconnected so the host itself didnt know it was removed. When he tried to add it back to vCenter it popped an error “The host is already connected to vCenter”. We spent a ton of time troubleshooting this and I kept finding articles talking about the very issue BUT they ALL say to run a script that should be located on the host at /opt/vmware/uninstallers/VMware-fdm-uninstall.sh. That all sounded great but apparently in ESXi 6.7 that file either doesnt exist or was moved and I simply could not find it. After doing even more research it seemed to have to do with some settings left behind for vpxa config that prevents you from adding the host.


Here is what the old file looked like:

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>




I got brave and renamed the file and I was able to get the host to reconnect to vCenter AND it re-ceated the vpxa.cfg file BUT in vCenter the host had an error that the httpnfc service could not start because its config was missing… grrr…

This is the file that vCenter generated when I renamed the file and connected the host to vCenter successfully:

      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
  <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I went back to look at the original “bad” vpxa.cfg and compared to it the newly generated one and there were a ton of differences. I then took the stuff that was missing out of the new file from the old one and added it, restarted the vpxa service and it started for a few seconds and then stopped… super frustrating! I felt that this may have been my fault so the next thing i did was I removed the extra junk so I was back to the small file that vCenter created. I then removed the host from vCenter. This removed the vCenter info from the vpxa.cfg file.


This is the config file after I removed from vCenter:

<ConfigRoot>
  <vmacore>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
  </vmacore>
  <vpxa>
    <hostIp>HOSTIP</hostIp>
    <hostKey/>
    <hostPort>443</hostPort>
    <serverIp/>
    <serverPort>-1</serverPort>
  </vpxa>

I then took everything in the original file (minus the sections in the new file) starting at and pasted it above the new sections. This time when I restarted vpxa it stayed running and I re added the host to vCenter and this time it added successfully and without errors!!! Problem Solved!

This is what the final file looked like (very similar to the very first one!)

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I wanted to share this in case anyone else runs into the issue. Main take away here is make sure you remove objects from vCenter while they are still online if at all possible.

Tagged : / / / / /