Unexpected error from ACE/Agent API.

In following this guide for configuring a CentOS 6 system to authenticate with RSA SecurID I came across an unusual error message that had me scratching my head:

Unexpected error from ACE/Agent API.

The problem stemmed from having an incorrect value in the /var/ace/sdopts.rec file for CLIENT_IP. For some reason I had put the IP address of the RSA authentication server in there. CLIENT_IP is the IP address of the RSA client, or rather, the machine you’re working on. The client uses whatever’s in that file to report to the RSA server what its IP address is. If the RSA server gets an invalid IP response from the client, it won’t authenticate.

SELinux issues

Much blood and tears were shed in dealing with getting SELinux to exist harmoniously with RSA SecurID. The problem was exacerbated my the fact that there is a lot of half solutions and misinformation floating out there on the internet. This will hopefully help fix that.

The message entry does not exist for Message ID: 1001

At this point acetest worked beautifully but I could not use an RSA passcode to SSH into the system. Digging into the log revealed this error message:

sshd[2135]: ACEAGENT: The message entry does not exist for Message ID: 1001

Thanks to this post, I realized it was due to selinux. Modifying the selinux config information to allow /var/ace to be read, per the commands below, seemed to fix the issue.

setenforce 0
chcon -Rv --type=sshd_t /var/ace/
setenforce 1

But alas! The solution was not a very good one. The commands above have two problems with them: first, the chcon command is temporary and does not survive selinux policy relabels; second, it assigns the type sshd_t, which does allow SSH to access it, but revokes RSA SecurID’s ability to write to the directory. This is a problem if you ever need to clear node secrets. The server will initiate the wipe but the client will not be able to modify that directory, resulting in node secret mismatches.

I finally decided to RTFM and landed on this documentation page, which explained the issue I was having: selinux mislabeling. The proper solution to this problem is use a label that both SecurID and SSHD can write / read to. Thanks to this SELinux Manpage (it really pays to RTFM!) I discovered that the label I want is var_auth_t (the default label applied when creating /var/ace is var_t, which SSH can’t read.) 

To survive relabeling, use the semanage command, which is not installed by default. Thanks to this site I learned I must install policycoreutils-pithon:

yum install policycoreutils-python

Once semanage is installed, use it to change the label for /var/ace and everything inside it to var_auth_t, then apply the changes with restorecon:

semanage fcontext -a -t var_auth_t "/var/ace(/.*)?"
restorecon -R -v /var/ace

Finally, both RSA SecurID and OpenSSH can read what they need to and authentication is successful.

First acetest succeeds but subsequent ones fail

If you followed the bad advice of relabeling /var/ace to sshd_t you might run across a very frustrating issue where acetest would succeed, but any attempts to SSH into the box or even run acetest again would fail. The error message on the RSA SecurID server was

Node secret mismatch: cleared on server but not on agent

The problem is due to the improper SELinux labeling mentioned above. The fix is the same:

yum install policycoreutils-python
semanage fcontext -a -t var_auth_t "/var/ace(/.*)?"
restorecon -R -v /var/ace

SSH access denied even with successful acetest

If acetest succeeds and you’ve loaded the module into PAM but still get access denied, it could be due to your SSH configuration. Ensure the following options are set:

ChallengeResponseAuthentication yes 
UsePrivilegeSeparation no

Victory.