Ticket History #8038: Kadmind/kadmin.local issues after migration from version 1.12.2 to 1.13

[andrei.maslennikov@gmail.com - Tue Nov 18 12:58:36 2014]:

> ..I have attempted to migrate the K5 binaries from 1.12.22 to 1.13 (both
> builds were done on Centos 6.6/x86_64). While the KDC is working correctly
> and I can authenticate, the "kadmind" daemon is failing to start. It is
> simply hanging and eating 100% of CPU.
>
> As well, when it is invoked, "kadmin.local" goes into the same 100% CPU
> loop at the very start, with a message "Authenticating as principal
> root/admin@.. with password" and does not return the prompt. [Whereas I
> can manually authenticate with "kinit root/admin" without any problem].
>
> If I roll back to 1.12.2, everything works again like a charm..
>
> The binaries were built with the vanilla 1.13 source. All the conf and
> db files were left untouched. Stracing does provide no clue. KDC log
> contains no errors. Kadmind log is empty.
>
> Had anyone observed anything similar migrating to 1.13?

I do not think we have heard of similar issues from other people migrating to 1.13.

The strace log (or even potentially an ltrace log) may be more informative for us than it was for
you; are you interested in providing some sample output (proably use -s 0 to avoid revealing
secret data)?

[andrei.maslennikov@gmail.com - Wed Nov 19 04:09:49 2014]:

> Hello Benjamin, and many hanks for looking into this!
>
> Please grab this tar.gz file containing the strace and ltrace logs for
> kadmin.local,
> both for working 1.12.2 and hanging 1.13, both binary builds for 1.12.2 and
> 1.13,
> kdc.conf and krb5.conf:

Many thanks for the very comprehensive listing.

Do you by any chance have something like a password quality module or some other out-of-tree
plugin module installed?

I think the next step may be to attach gdb to the spinning/hung kadmin.local and get a
backtrace.

[andrei.maslennikov@gmail.com - Thu Nov 20 04:04:03 2014]:

> Hello, it seems we are getting closer to the origin of the loop!

Yes, indeed.

> I.e. it looks like kadmin.local goes mad while comparing strings or looking
> for encryption type "arcfour-hmac".

We did change some code relating to how the keysalt list is processed, so that's likely the culprit
here.

> Thus the issue is turning around "arcfour-hmac". My current list of
> encryption types includes this
> family, and I have to keep them there as the principals already possess the
> corresponding keys:

This is not actually true; the "supported_enctypes" variable is rather misnamed -- it controls the
generation of new keys on the KDC (e.g., from user-supplied passwords) by specifying which
enctypes will be generated, but has no effect on the operation of existing keys.  (There is a
separate variable, "permitted_enctypes", which does have that effect.)

> "arcfour-hmac:normal arcfour-hmac:norealm arcfour-hmac:onlyrealm
> arcfour-hmac-md5:normal"

The arcfour family of enctypes do not actually use a salt, ever, so there is no difference between
the :normal, :norealm, and :onlyrealm forms.  I am not yet sure if this duplication is causing the
spinning, but it is certainly unnecessary in your configuration.

> (If I recall it correctly, "arcfour-hmac" is used on Windows clients, and
> we have some
> Win machines that get the authentication from K5MIT and not from AD).

Windows XP and Windows Server 2003 only support the arcfour and single-DES encryption types.
Newer versions of windows also support the more modern AES encryption types (though perhaps
Windows Server 2008 would need some manual configuration to start using them; this is not
entirely clear to me).  If your windows clients are, e.g., Windows 7 and above, they can handle
AES enctypes just fine.  You will probably need to go and rekey some things to make a full
transition, but the mere presence of such windows clients does not require you to keep arcfour
around.  The arcfour enctypes are likely to be deprecated in the next year or two.

> What could be done about it?

I will fire up a test KDC with the problematic supported_enctypes list and look into a fix.

> Cheers - Andrei.
>
> PS I have mentioned that the Builds directory of my yesterday's tar file was
> in fact containing two times 1.12.2 builds and not 1.12.2 and 1.13. I have
> hence replaced the tar.file with the one correcting the correct Builds
> subdir,
> sorry for this mistake. You may again grab it at:
>
>        http://afsita.masl.eu/k5/k5.debug.tgz

Thanks, but I think just the bit about supported_enctypes will be enough for now.

Thanks for following up on this issue.

-Ben

Avoid infinite loop on duplicate keysalts

When duplicate suppression was requested, we would enter an
infinite loop upon encountering a duplicate entry, a bug
introduced in commit 0918990bf1d8560d74473fc0e41d08d433da1a15
and thus present in release 1.13.

Rework the conditional to avoid the loop, at the expense of
additional indentation for some of the code.

https://github.com/krb5/krb5/commit/c828e7cb137de3559f026dcc552a52162d9ca5cd
Author: Ben Kaduk <kaduk@mit.edu>
Commit: c828e7cb137de3559f026dcc552a52162d9ca5cd
Branch: master
 src/lib/kadm5/str_conv.c |   21 ++++++++++-----------
 1 files changed, 10 insertions(+), 11 deletions(-)