September 9, 2020 - AllTheSystems

Recently I worked on an issue where a coworker changed the IP and hostname of a ESXi Host without removing it from vCenter first. The old host was removed from vCenter but it was while the host was disconnected so the host itself didnt know it was removed. When he tried to add it back to vCenter it popped an error “The host is already connected to vCenter”. We spent a ton of time troubleshooting this and I kept finding articles talking about the very issue BUT they ALL say to run a script that should be located on the host at /opt/vmware/uninstallers/VMware-fdm-uninstall.sh. That all sounded great but apparently in ESXi 6.7 that file either doesnt exist or was moved and I simply could not find it. After doing even more research it seemed to have to do with some settings left behind for vpxa config that prevents you from adding the host.

Here is what the old file looked like:

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I got brave and renamed the file and I was able to get the host to reconnect to vCenter AND it re-ceated the vpxa.cfg file BUT in vCenter the host had an error that the httpnfc service could not start because its config was missing… grrr…

This is the file that vCenter generated when I renamed the file and connected the host to vCenter successfully:

      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
  <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I went back to look at the original “bad” vpxa.cfg and compared to it the newly generated one and there were a ton of differences. I then took the stuff that was missing out of the new file from the old one and added it, restarted the vpxa service and it started for a few seconds and then stopped… super frustrating! I felt that this may have been my fault so the next thing i did was I removed the extra junk so I was back to the small file that vCenter created. I then removed the host from vCenter. This removed the vCenter info from the vpxa.cfg file.

This is the config file after I removed from vCenter:

<ConfigRoot>
  <vmacore>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
  </vmacore>
  <vpxa>
    <hostIp>HOSTIP</hostIp>
    <hostKey/>
    <hostPort>443</hostPort>
    <serverIp/>
    <serverPort>-1</serverPort>
  </vpxa>

I then took everything in the original file (minus the sections in the new file) starting at and pasted it above the new sections. This time when I restarted vpxa it stayed running and I re added the host to vCenter and this time it added successfully and without errors!!! Problem Solved!

This is what the final file looked like (very similar to the very first one!)

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I wanted to share this in case anyone else runs into the issue. Main take away here is make sure you remove objects from vCenter while they are still online if at all possible.

While windows update will update the VCRedist packages you have installed it will not remove the old versions. Here is a script that can be used from the SCCM scripts section. To create approve and add SCCM Scripts see this post. The script will first check to see if chocolatey is installed and will attempt to install it if not found. After that as long as chocolatey was installed successfully it will move on to looking for installs of VCRedist and will uninstall all versions found and use chocolatey to install the latest version (it will install both x64 and x86). See comments in code for a few details like where you can add or remove versions to look for.

#Start checking for Chocolatey
try {
	invoke-command -scriptblock {choco} -erroraction stop
    write-host "Has Choco. all is good!"
    $Choco_installed = $true
}

catch {
	Write-Host "Needs Choco.  Trying install..." 
	try {
		invoke-command -scriptblock {Set-ExecutionPolicy Bypass -Scope Process -Force; Invoke-Expression ((New-Object System.Net.WebClient).DownloadString('https://chocolatey.org/install.ps1'))} -ErrorAction Stop
        $Choco_installed = $true
    }
	catch {
        write-host "Install Failed"
        $Choco_installed = $false
	}
		
}
Set-ExecutionPolicy -ExecutionPolicy Bypass -Scope Process -force;


#Checks to see if the above was successful
if ($Choco_installed -eq $true) {
    #this is where you can set the versions of VCRedist to look for
    $VCredistVersions = @()
    $VCredistVersions = (
        "2005",
        "2008",
        "2010",
        "2013",
        "2015",
        "2019"
    )

#Loop through each version
    foreach ($Version in $VCredistVersions) {
        if (get-wmiobject -Class Win32_Product| where {$_.name -like "*Microsoft Visual C++ $Version Redistributable*"}| select name,localpackage) {
            Write-Host "Found Microsoft Visual C++ $version Redistributable.  Removing old versions and installing latest..." -ForegroundColor Yellow
            $Packages = get-wmiobject -Class Win32_Product| where {$_.name -like "*Microsoft Visual C++ $Version Redistributable*"}| select name,localpackage -ErrorAction Stop
                    foreach ($Package in $Packages) {
                        $packagename = $Package.localpackage    
#Run the actual uninstall                        
cmd.exe /c "msiexec /x $packagename /qn"
                        Write-Host "Successfully uninstalled $packagename!" -ForegroundColor Green
                    }
                    #Install latest version
                    choco update vcredist$version -y -f
        } 
    }
}

Tagged : SCCM powershell / sccm script / VCRedist update powershell / Visual C++ Redist update

AllTheSystems

Day: September 9, 2020

vCenter Fix – The Host is already connected to vCenter

SCCM Script – Visual C++ Redistributable updater