RTSA - on the road to becoming a sysadmin

About: The purpose of this document is to explain an unpleasant behaviour observed with the vCSA 5.5 appliance, in which an user's expired password cannot be changed by the user himself at the prompt informing him of the password's expiration, but instead requires an administrative reset; it must be noted that prompt directly offers the user the apparent possibility of changing his password, despite this action actually not being possible, as described below.

Note on applicability: This behaviour has been encountered with the 5.5 vCSA while using the VSPHERE.LOCAL user store; the 6.0 version has yet to be tested.

Description of the issue: The user goes to login to his vCenter account and is presented with a pop-up titled Reset password which informs him that Your account password has expired and you need to change it before you can sign in., at which point the user proceeds to entering his old password and providing the new password (and its confirmation). The result is that an error bar is generated in the pop-up window, informing him that The specified principal ([userName redacted]) is invalid. - the password is not changed, as such, and the user is unable able to access his vCenter account.

Explanation: No VMware support article has been found detailing this problem (this statement was up-to-date at the end of 2016), so the vCSA's logfiles were consulted: /var/log/messages proved inconclusive (it simply stated that the user's password had expired, something that was already known), but searching around the /var/log directory led to the identification of the quite relevant /var/log/vmware/sso/ssoAdminServer.log file:

ssoAdminServer.log

[vCSA name redacted]:~ # tail -n52 /var/log/vmware/sso/ssoAdminServer.log
[2016-12-02 07:53:58,062 pool-2-thread-3  INFO  com.vmware.identity.vlsi.RoleBasedAuthorizer] User Anonymous is authorized for method call 'ServiceInstance.retrieveServiceContent'
[2016-12-02 07:53:58,073 pool-2-thread-5  INFO  com.vmware.identity.vlsi.RoleBasedAuthorizer] User Anonymous is authorized for method call 'IdentitySourceManagementService.getSystemDomainName'
[2016-12-02 07:53:58,073 pool-1-thread-4  INFO  com.vmware.identity.admin.vlsi.IdentitySourceManagementServiceImpl] [User Anonymous] Getting system domain name
[2016-12-02 07:53:58,077 pool-1-thread-4  INFO  com.vmware.identity.admin.vlsi.IdentitySourceManagementServiceImpl] Vmodl method 'IdentitySourceManagementService.getSystemDomainName' return value is 'vsphere.local'
[2016-12-02 07:53:58,088 pool-2-thread-4  INFO  com.vmware.identity.vlsi.RoleBasedAuthorizer] User Anonymous is authorized for method call 'ServiceInstance.retrieveServiceContent'
[2016-12-02 07:53:58,099 pool-2-thread-1  INFO  com.vmware.identity.vlsi.RoleBasedAuthorizer] User Anonymous is authorized for method call 'PrincipalManagementService.resetLocalUserPassword'
[2016-12-02 07:53:58,099 pool-1-thread-5  INFO  com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl] [User Anonymous] Resetting password of local user '[userName redacted]'.
[2016-12-02 07:53:58,113 pool-1-thread-5  ERROR com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl] Error in resetLocalPersonUserPassword (change password). Check if user actually exists. Idm client exception.com.vmware.identity.idm.InvalidPrincipalException: Invalid login credential for user [userName redacted]@vsphere.local
[2016-12-02 07:53:58,113 pool-1-thread-5  INFO  com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl] The specified principal ([userName redacted]) is invalid.
com.vmware.vim.sso.admin.exception.InvalidPrincipalException: The specified principal ([userName redacted]) is invalid.
  at com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl.resetLocalPersonUserPassword(PrincipalManagementImpl.java:2327)
  at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$18.call(PrincipalManagementServiceImpl.java:494)
  at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl$18.call(PrincipalManagementServiceImpl.java:481)
  at com.vmware.identity.admin.vlsi.util.VmodlEnhancer.invokeVmodlMethod(VmodlEnhancer.java:153)
  at com.vmware.identity.admin.vlsi.PrincipalManagementServiceImpl.resetLocalUserPassword(PrincipalManagementServiceImpl.java:481)
  at sun.reflect.GeneratedMethodAccessor1365.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at com.vmware.vim.vmomi.server.impl.InvocationTask.run(InvocationTask.java:76)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Unknown Source)
Caused by: com.vmware.identity.idm.InvalidPrincipalException: Invalid login credential for user [userName redacted]@vsphere.local
  at com.vmware.identity.idm.server.ServerUtils.getRemoteException(ServerUtils.java:101)
  at com.vmware.identity.idm.server.IdentityManager.changeUserPassword(IdentityManager.java:5540)
  at sun.reflect.GeneratedMethodAccessor1228.invoke(Unknown Source)
  at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
  at java.lang.reflect.Method.invoke(Unknown Source)
  at sun.rmi.server.UnicastServerRef.dispatch(Unknown Source)
  at sun.rmi.transport.Transport$2.run(Unknown Source)
  at sun.rmi.transport.Transport$2.run(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.Transport.serviceCall(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport.handleMessages(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.access$400(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler$1.run(Unknown Source)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
  at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
  at java.lang.Thread.run(Unknown Source)
  at sun.rmi.transport.StreamRemoteCall.exceptionReceivedFromServer(Unknown Source)
  at sun.rmi.transport.StreamRemoteCall.executeCall(Unknown Source)
  at sun.rmi.server.UnicastRef.invoke(Unknown Source)
  at java.rmi.server.RemoteObjectInvocationHandler.invokeRemoteMethod(Unknown Source)
  at java.rmi.server.RemoteObjectInvocationHandler.invoke(Unknown Source)
  at com.sun.proxy.$Proxy69.changeUserPassword(Unknown Source)
  at com.vmware.identity.idm.client.CasIdmClient.changeUserPassword(CasIdmClient.java:2447)
  at com.vmware.identity.admin.server.ims.impl.PrincipalManagementImpl.resetLocalPersonUserPassword(PrincipalManagementImpl.java:2317)
  ... 11 more

As the logfile suggests, the SSO service seems to not actually authenticate the account for which the password has expired, but instead uses a generic User Anonymous principal for querying the target account, which it identifies as having an expired password (this is not detailed in the actual logfile and would likely happen in the background, or is possibly being recorded into the /var/log/messages log instead, as can be seen in the below example; the timestamps of the log excerpt do seem to suggest so), at which time the generic principal calls the function for resetting an user's password.

/var/log/messages

[vCSA name redacted]:~ # tail /var/log/messages
2016-12-02T07:53:58+00:00 [vCSA name redacted] vmdird: t@140199453239040: Modify Entry (CN=[userName redacted],CN=Users,DC=vsphere,DC=local)
2016-12-02T07:53:58+00:00 [vCSA name redacted] vmdird: t@140199453239040: User account control - (cn=[userName redacted],cn=users,dc=vsphere,dc=local): (800000) flag set, new value=(800000)
2016-12-02T07:53:58+00:00 [vCSA name redacted] vmdird: t@140199453239040: Lockout policy check - password expired. (cn=[userName redacted],cn=users,dc=vsphere,dc=local)

The invocation fails as might be expected given the generic identity - meantime, in the UI, the actual user is presented with a pop-up window which suggests that he can perform the password change, so he enters his old password and the new password and its confirmation. He hits OK and is told that his account is actually invalid, which makes no apparent sense. However, the described behaviour and separation between the UI and the underlying SSO service can be confirmed by providing a bogus current password instead of the actual one - the UI reacts exactly as when the user entered his proper password, which means that the user's input has no effect on what is happening in the background.

Consequently, it would seem that this behaviour describes an SSO bug in VMware's 5.5 vCSA (possibly 6.0 as well).

Solution: The straightforward solution is for the user to contact a VMware administrator, who will log in using his own account (or the administrator@vsphere.local one) and reset the user's password. Once logged in with the password provided him by the administrator, the user will have the ability to change it to something meaningful for himself via the regular password change mechanism.