vCenter Archives - AllTheSystems

For the most part you can use vCenter and powercli to get almost anything you need done for your virtual environment. However, sometimes you might run into a monitoring or management tool that does not have the ability to talk to vCenter. In this case we need to give the tool a local ESXi account, but we don’t to be handing out root password to automated systems. Adding a local user account can be done through the web ui for each ESXi host. If you have more than a few this could take a while. To solve this we can use our good friend powershell. To do add these users we will need to connect to each host using ssh. I tried just using the built-in ssh that windows/powershell can do but it just didn’t work right. To get over this I used a tool called kitty this is a modified putty that lets you run it from command line. You can use plink as well but I already had kitty available and some code I had written for a different task. Another thing you need before starting is vmware powercli. You can see this post on how to set that up. The code below will connect to vCenter, look for the host(s) we define, then for each of the hosts it will enable ssh via powercli, connect via kitty (ssh) and run our commands to create the user and set read-only permission, finally it will disable ssh.

#Check for Creds and ask for them if they aren't found
if (!($Creds)) {$Creds = get-credential -Message "Enter your vCenter Admin Creds in domain\username format"}

# Connect to vCenter
Connect-VIServer -server yourvCenterIPorName -Credential $creds -force

#get all hosts.  You can use an array here instead, but this will get all of the hosts connected to your vCenter
$esxHosts = Get-VMHost 


foreach ($esxHost in $esxHosts) {
    
    $esxHost = $esxHost.name
    #Start SSH Service
    Get-VMHostService -VMHost $esxHost | Where-Object {$_.Key -eq "TSM-SSH" } | Start-VMHostService

    #run commands to create user and set read-only permissions
.\kitty.exe -kload .\kex.ktx root@$esxHost -pass "ESXiPassword" -cmd '
esxcli system account add -d=''read-only local user'' -i=''username'' -p=''Password'' -c=''Password''
esxcli system permission set -i=''username'' -r=''ReadOnly''
esxcli system permission list
exit
'

    
    #Stop SSH Service
    Get-VMHostService -VMHost $esxHost | Where-Object {$_.Key -eq "TSM-SSH" } | Stop-VMHostService -Confirm:$false


}

Remove a local user

If you need to remove the user we can modify the above script a bit to remove the user from all the hosts. That looks like this:

#Check for Creds and ask for them if they aren't found
if (!($Creds)) {$Creds = get-credential -Message "Enter your vCenter Admin Creds in domain\username format"}

# Connect to vCenter
Connect-VIServer -server yourvCenterIPorName -Credential $creds -force

#get all hosts.  You can use an array here instead, but this will get all of the hosts connected to your vCenter
$esxHosts = Get-VMHost 


foreach ($esxHost in $esxHosts) {
    
    $esxHost = $esxHost.name
    #Start SSH Service
    Get-VMHostService -VMHost $esxHost | Where-Object {$_.Key -eq "TSM-SSH" } | Start-VMHostService

#run commands to remove the user
.\kitty.exe -kload .\kex.ktx root@$esxHost -pass "ESXiPassword" -cmd '
 esxcli system account remove -i=''username''
 exit
 '

   
    #Stop SSH Service
    Get-VMHostService -VMHost $esxHost | Where-Object {$_.Key -eq "TSM-SSH" } | Stop-VMHostService -Confirm:$false


}

Tagged : powershell / vcenter

Recently I worked on an issue where a coworker changed the IP and hostname of a ESXi Host without removing it from vCenter first. The old host was removed from vCenter but it was while the host was disconnected so the host itself didnt know it was removed. When he tried to add it back to vCenter it popped an error “The host is already connected to vCenter”. We spent a ton of time troubleshooting this and I kept finding articles talking about the very issue BUT they ALL say to run a script that should be located on the host at /opt/vmware/uninstallers/VMware-fdm-uninstall.sh. That all sounded great but apparently in ESXi 6.7 that file either doesnt exist or was moved and I simply could not find it. After doing even more research it seemed to have to do with some settings left behind for vpxa config that prevents you from adding the host.

Here is what the old file looked like:

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I got brave and renamed the file and I was able to get the host to reconnect to vCenter AND it re-ceated the vpxa.cfg file BUT in vCenter the host had an error that the httpnfc service could not start because its config was missing… grrr…

This is the file that vCenter generated when I renamed the file and connected the host to vCenter successfully:

      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
  <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I went back to look at the original “bad” vpxa.cfg and compared to it the newly generated one and there were a ton of differences. I then took the stuff that was missing out of the new file from the old one and added it, restarted the vpxa service and it started for a few seconds and then stopped… super frustrating! I felt that this may have been my fault so the next thing i did was I removed the extra junk so I was back to the small file that vCenter created. I then removed the host from vCenter. This removed the vCenter info from the vpxa.cfg file.

This is the config file after I removed from vCenter:

<ConfigRoot>
  <vmacore>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
  </vmacore>
  <vpxa>
    <hostIp>HOSTIP</hostIp>
    <hostKey/>
    <hostPort>443</hostPort>
    <serverIp/>
    <serverPort>-1</serverPort>
  </vpxa>

I then took everything in the original file (minus the sections in the new file) starting at and pasted it above the new sections. This time when I restarted vpxa it stayed running and I re added the host to vCenter and this time it added successfully and without errors!!! Problem Solved!

This is what the final file looked like (very similar to the very first one!)

<config>
  <dvs>
    <portSyncBatchLimit>100</portSyncBatchLimit>
  </dvs>
  <httpNfc>
    <accessMode>proxyAuto</accessMode>
    <enabled>true</enabled>
  </httpNfc>
  <level id="SoapAdapter.HTTPService">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService</logName>
  </level>
  <level id="SoapAdapter.HTTPService.HttpConnection">
    <logLevel>info</logLevel>
    <logName>SoapAdapter.HTTPService.HttpConnection</logName>
  </level>
  <log>
    <level>verbose</level>
    <maxFileNum>10</maxFileNum>
    <maxFileSize>1048576</maxFileSize>
    <memoryLevel>verbose</memoryLevel>
    <outputToConsole>false</outputToConsole>
    <outputToFiles>false</outputToFiles>
    <outputToSyslog>true</outputToSyslog>
    <syslog>
      <facility>local4</facility>
      <ident>Vpxa</ident>
      <logHeaderFile>/var/run/vmware/vpxaLogHeader.txt</logHeaderFile>
    </syslog>
  </log>
  <nfc>
    <loglevel>error</loglevel>
  </nfc>
  <task>
    <completedMaxEntries>1000</completedMaxEntries>
    <maxThreads>98</maxThreads>
    <minCompletedLifetime>120</minCompletedLifetime>
  </task>
  <trace>
    <mutex>
      <profiledMutexes>InvtLock</profiledMutexes>
    </mutex>
    <vmomi>
      <calls>false</calls>
    </vmomi>
  </trace>
  <vmacore>
    <http>
      <defaultClientPoolConnectionsPerServer>300</defaultClientPoolConnectionsPerServer>
    </http>
    <soap>
      <sessionTimeout>1440</sessionTimeout>
    </soap>
    <ssl>
      <doVersionCheck>false</doVersionCheck>
    </ssl>
    <threadPool>
      <IoMax>9</IoMax>
      <TaskMax>4</TaskMax>
      <ThreadStackSizeKb>128</ThreadStackSizeKb>
      <threadNamePrefix>vpxa</threadNamePrefix>
    </threadPool>
  </vmacore>
    <vpxa>
    <bundleVersion>1000000</bundleVersion>
    <datastorePrincipal>UN</datastorePrincipal>
    <hostIp>HOSTIP</hostIp>
    <hostKey>HOSTKEY</hostKey>
    <hostPort>443</hostPort>
    <licenseExpiryNotificationThreshold>15</licenseExpiryNotificationThreshold>
    <memoryCheckerTimeInSecs>30</memoryCheckerTimeInSecs>
    <serverIp>VCENTERIP</serverIp>
    <serverPort>902</serverPort>
  </vpxa>
  <workingDir>/var/log/vmware/vpx</workingDir>

I wanted to share this in case anyone else runs into the issue. Main take away here is make sure you remove objects from vCenter while they are still online if at all possible.

Tagged : esxi / reinstall vcenter agent 6.7 / the host is already connected to vcenter / vcenter / vcenter 6.7 / VMware-fdm-uninstall.sh missing

AllTheSystems

Category: vCenter

Add local ESXi users

Remove a local user

vCenter Fix – The Host is already connected to vCenter