Skip Menu |
 

Date: Wed, 19 Apr 2017 00:20:29 -0400 (EDT)
From: Richard Silverman <res@qoxp.net>
To: krb5-bugs@mit.edu
Subject: duplicate caching of some cross-realm TGTs
Download (untitled) / with headers
text/plain 1.4KiB
Hello,

Under certain circumstances, we're seeing duplicate caching of some
cross-realm tickets. Specifically:

1 Kerberos principal user@A tries to establish a GSSAPI security context
with GSS hostbased-service name host@foo.

2 The client tries its own realm first for "host/foo@A"; the service
principal should be there, but it's missing.

3 The A KDC refers the client to realm B.

4 The B KDC refers the client back to realm A, and the attempt ends in a
KDC policy error from A due to the lineage check.

In this case for us (MIT Kerberos 1.15.1), another copy of the krbtgt/B@A
referral TGT from step 3 is added to the ccache every time. With a process
that, say, ssh'es to thousands of hosts periodically, the ccache grows to
an impractical size.

Note that the second referral TGT from step 4 is not cached at all, let
alone repeatedly -- which also seems odd but is not an immediate problem.

One might wonder whether the looping referrals are themselves a
mistake. Actually these optimistic referrals are intentional, addressing
certain aspects of our infrastructure; the missing host principal is the
real breakage here. But it's really not relevant; the duplicate caching is
a problem on its own.

I'm attaching a patch that fixes the issue. Do note the accompanying
commit message, as it has some more comments. As it indicates, I'm not
sure this is the right fix -- so please consider it a first stab at a
solution.

Thanks,

Richard E. Silverman

Message body is not shown because sender requested not to inline it.

Date: Wed, 19 Apr 2017 07:54:33 -0400 (EDT)
From: Richard Silverman <res@qoxp.net>
To: krb5 <rt@krbdev.mit.edu>
Subject: Re: [krbdev.mit.edu #8579] AutoReply: duplicate caching of some cross-realm TGTs
RT-Send-Cc:
I’m attaching an updated patch, which fixes logging and a memory leak.

--
Richard

Message body is not shown because sender requested not to inline it.

Download (untitled) / with headers
text/plain 2.2KiB
Okay, I see the problem. In the traditional model of credential
caching, the upper layer checks the cache for a credential, doesn't
find it, goes out and gets it, and puts it in the cache. In a
concurrent scenario a few processes might write duplicate entries to
the cache, but that's (hopefully) rare and pretty harmless, so under
this model we don't need the ccache layer to do any duplicate
suppression.

During a TGS request the KDC might hand us a credential we didn't ask
for, such as a referral or an alternate TGS cred. Caching these
credentials breaks the model--since we never wanted these credentials
at the upper layer, the upper layer never checks for them. That also
calls into question the value of caching those unsolicited responses--
they won't be useful if the same higher-level operation is repeated
(or even a similar one, such as a request resulting in a refferal to
the same realm). They would only save a TGS request if an unrelated
operation happens to ask for that cross-realm TGT principal.

So I think my preferred solution for this scenario is to change
get_cred.c not to cache answers it didn't ask for. (This
unfortunately isn't quite as simple as removing code due to the
current design of step_get_tgt(), but it shouldn't be too hard.)
Handling duplicates inside the ccache layer has a few problems:

* FILE ccaches are essentially append-only, so suppressing duplicates
by removing the old entry can't be implemented efficiently in the most
common ccache type. (Heimdal implements removal by overwriting
selected parts of the cred entry so that it becomes invisible to
traversal, but that doesn't prevent the ccache from growing
unacceptably large.)

* Suppressing duplicates by ignoring the new credential isn't the
behavior we want for ccache config entries. We would ideally like to
be able to change the value of config entries that already exist by
writing a new credential entry, although that doesn't currently work
(krb5_cc_get_config() stops when it sees the old entry).

* Copying a moderately-sized ccache would become O(n^2) as we do a
full read of the ccache for each entry.

In your patch, you noted a krb5_cc_remove_cred() call inside
krb5_cc_store_cred(). That is an accidental leftover; see
http://krbdev.mit.edu/rt/Ticket/Display.html?id=7906
Date: Mon, 24 Apr 2017 14:56:06 -0400 (EDT)
From: Richard Silverman <res@qoxp.net>
To: Greg Hudson via RT <rt-comment@krbdev.mit.edu>
Subject: Re: [krbdev.mit.edu #8579] duplicate caching of some cross-realm TGTs
RT-Send-Cc:
Show quoted text
> So I think my preferred solution for this scenario is to change
> get_cred.c not to cache answers it didn't ask for.

This makes sense to me, and it also (I think) solves another problem I’ve run into that I’ve dubbed “ccache poisoining.” If a client receives an inaccurate referral and caches it, the cached referral can prevent the client from following an available successful path for a different service ticket later on. Of course, the incorrect referral is the root problem, but these things happen in complex multi-platform/realm arrangements, so it’s nice to contain the breakage.

--
Richard
From: Sam Hartman <hartmans@mit.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #8579] duplicate caching of some cross-realm TGTs
Date: Mon, 24 Apr 2017 16:13:52 -0400
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.8KiB
Show quoted text
>>>>> "Greg" == Greg Hudson via RT <rt-comment@krbdev.mit.edu> writes:

Show quoted text
Greg> During a TGS request the KDC might hand us a credential we
Greg> didn't ask for, such as a referral or an alternate TGS cred.
Greg> Caching these credentials breaks the model--since we never
Greg> wanted these credentials at the upper layer, the upper layer
Greg> never checks for them. That also calls into question the
Greg> value of caching those unsolicited responses-- they won't be
Greg> useful if the same higher-level operation is repeated (or even
Greg> a similar one, such as a request resulting in a refferal to
Greg> the same realm). They would only save a TGS request if an
Greg> unrelated operation happens to ask for that cross-realm TGT
Greg> principal.

There's one thing I'd recommend thinking about at least.
What happens in a client-driven cross-realm situation. That is, what
happens if the client has a well-populated domain_realms section in its
config file.
Will you change what tickets are retained? If so, do we care?
That's the only case where I believe cross-realm tickets are
particularly likely to be used.

I like the security properties of your solution better than the current
behavior though.
Currently, we interpret a TGS referral from a KDC as meaning two things:

1) Use the referred realm along the path to the requested principal.

2) This ticket is a valid TGT for any contact to the referred realm.

We may not use that ticket generally, but if we cache it, an attacker
might be able to convince us to do so.

However, if we don't cache the intermediate ticket, we're very close to
revising the second assertion from the KDC to "use this TGT for this one
operation."
That's more conservative in a way that I like. I admit that with the
KDCs I'm aware of, it doesn't buy you anything in practice.
For client-driven cross-realm scenarios, I believe we should cache the
TGTs we ask for, but not alternate TGTs. If we cache alternate TGTs, we
could have the same kind of scenario where we repeatedly cache an
alternate TGT because the overall TGS operation fails.
From: Sam Hartman <hartmans@mit.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #8579] duplicate caching of some cross-realm TGTs
Date: Mon, 24 Apr 2017 18:00:08 -0400
RT-Send-Cc:
Show quoted text
>>>>> "Greg" == Greg Hudson via RT <rt-comment@krbdev.mit.edu> writes:

Show quoted text
Greg> For client-driven cross-realm scenarios, I believe we should
Greg> cache the TGTs we ask for, but not alternate TGTs. If we
Greg> cache alternate TGTs, we could have the same kind of scenario
Greg> where we repeatedly cache an alternate TGT because the overall
Greg> TGS operation fails.

Agreed.
From: ghudson@mit.edu
Subject: git commit

Don't cache referral and alternate TGT replies

During a TGS request, if we get a TGT response that we didn't directly
ask for (a referral TGT or an alternate TGT), don't cache it. It
would have limited value in the cache as similar operations won't look
for that TGT. If the overall TGS operation fails and is repeated, we
could wind up caching the same entry multiple times, which doesn't
work well with our current ccache implementations.

https://github.com/krb5/krb5/commit/1dc619624421002b1e64d3b8c7e270508381b3e6
Author: Greg Hudson <ghudson@mit.edu>
Commit: 1dc619624421002b1e64d3b8c7e270508381b3e6
Branch: master
src/lib/krb5/krb/get_creds.c | 13 +++----------
src/tests/t_crossrealm.py | 35 +++++++++++++++++++++++++++++++++++
src/tests/t_referral.py | 11 ++++++++---
3 files changed, 46 insertions(+), 13 deletions(-)
From: ghudson@mit.edu
Subject: git commit

Remove stray exit in t_referral.py

Commit 1dc619624421002b1e64d3b8c7e270508381b3e6 included a stray
"exit(0)" for debugging. Remove it.

https://github.com/krb5/krb5/commit/73e40a81df07d035d1a86b871012d022d38fb9e3
Author: Greg Hudson <ghudson@mit.edu>
Commit: 73e40a81df07d035d1a86b871012d022d38fb9e3
Branch: master
src/tests/t_referral.py | 1 -
1 files changed, 0 insertions(+), 1 deletions(-)