Skip Menu |
 

Subject: GSSAPI context being destroyed when ticket cache renewed
Download (untitled) / with headers
text/plain 1.7KiB
Hello,

In helping a user on the openldap-software list track down why
replication (SASL/GSSAPI based) wasn't working correctly on his servers,
we found that the problem was the way in which MIT Kerberos handles
ticket refreshes. Basically, the way the replication work is we have a
master and a replica. The replica makes a persistent bind to the master
with SASL/GSSAPI. On my servers using Heimdal kerberos, this works
fine, and can last for days. However, on his servers using MIT
Kerberos, once the ticket cache was refreshed (we both use k5start by
Russ Allbery to do this), the GSSAPI context was destroyed, meaning that
traffic could no longer be encrypted between the master and the replica.
Replication would stop, and the master and replicas would both get into
odd states after that.

Another user reported having the same problems with MIT kerberos:

--On Thursday, August 31, 2006 9:41 PM +0100 simon@sxw.org.uk wrote:

Show quoted text
>
> If you're using MIT Kerberos, I strongly suspect that the problem you're
> seeing is due to the behaviour of its GSSAPI library. With MIT's
> implementation a gssapi context is only valid for the lifetime of the
> initiator's credential. So, when your clients credentials expire, the
> security context also expires, and the client and server both start
> failing to decrypt each others packets. We first noticed this problem
> because slapd would crash when the decryption failure occurred - this
> problem no longer exists - but MIT Kerberos's behaviour has not changed.
>
> Just renewing your credentials won't help, as the new credentials will
> only be used when establishing a new security context, which only happens
> when a new connection is opened.


It would be great to have this fixed!

--Quanah
From: Sam Hartman <hartmans@mit.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
Date: Fri, 01 Sep 2006 00:55:40 -0400
RT-Send-Cc:
I strongly suspect that the context is ending when it expires and that
SASL needs to do a better job of catching this error and reporting a
connection problem.
Date: Thu, 31 Aug 2006 22:08:12 -0700
From: Quanah Gibson-Mount <quanah@stanford.edu>
To: rt-comment@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:


--On Friday, September 01, 2006 12:55 AM -0400 Sam Hartman via RT
<rt-comment@krbdev.mit.edu> wrote:

Show quoted text
> I strongly suspect that the context is ending when it expires and that
> SASL needs to do a better job of catching this error and reporting a
> connection problem.

Well, this is very different than how Heimdal's behavior operates, and
quite frankly I prefer what Heimdal does. I see no reason to make an
*already* established & encrypted connection break just because the context
has ended.

--Quanah


--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Date: Wed, 06 Sep 2006 13:43:27 -0700
From: Quanah Gibson-Mount <quanah@stanford.edu>
To: rt-comment@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:


--On Friday, September 01, 2006 12:55 AM -0400 Sam Hartman via RT
<rt-comment@krbdev.mit.edu> wrote:

Show quoted text
> I strongly suspect that the context is ending when it expires and that
> SASL needs to do a better job of catching this error and reporting a
> connection problem.


Just to be clear, the problem happens when the ticket cache is refreshed.
I.e., the tickets for the existing SASL/GSSAPI connection hadn't actually
yet expired, just the ticket cache was refreshed with new tickets. I can
understand why the SASL/GSSAPI context would be closed out on *expiration*
but I think a refresh shouldn't have this effect. ;)


--Quanah


--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
From: Russ Allbery <rra@stanford.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
Date: Wed, 06 Sep 2006 14:10:21 -0700
RT-Send-Cc:
Quanah Gibson-Mount via RT <rt-comment@krbdev.mit.edu> writes:
Show quoted text
> Sam Hartman via RT <rt-comment@krbdev.mit.edu> wrote:

Show quoted text
>> I strongly suspect that the context is ending when it expires and that
>> SASL needs to do a better job of catching this error and reporting a
>> connection problem.

Show quoted text
> Just to be clear, the problem happens when the ticket cache is
> refreshed. I.e., the tickets for the existing SASL/GSSAPI connection
> hadn't actually yet expired, just the ticket cache was refreshed with
> new tickets. I can understand why the SASL/GSSAPI context would be
> closed out on *expiration* but I think a refresh shouldn't have this
> effect. ;)

This makes me wonder what in GSS-API is looking at the ticket cache. I
would have thought that once the GSS-API context was established and
authentication was finished, there wouldn't be further need to look at the
Kerberos ticket cache, but apparently that's not correct?

--
Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
Date: Wed, 06 Sep 2006 17:47:17 -0400
From: Jeffrey Altman <jaltman@mit.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:
Russ Allbery via RT wrote:

Show quoted text
>> Just to be clear, the problem happens when the ticket cache is
>> refreshed. I.e., the tickets for the existing SASL/GSSAPI connection
>> hadn't actually yet expired, just the ticket cache was refreshed with
>> new tickets. I can understand why the SASL/GSSAPI context would be
>> closed out on *expiration* but I think a refresh shouldn't have this
>> effect. ;)

If it is possible, can you post a stack trace at the point the context
is deemed to be invalid?

That would help a lot.
Date: Wed, 06 Sep 2006 15:07:37 -0700
From: Quanah Gibson-Mount <quanah@stanford.edu>
To: rt-comment@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.3KiB


--On Wednesday, September 06, 2006 5:45 PM -0400 Jeffrey Altman via RT
<rt-comment@krbdev.mit.edu> wrote:

Show quoted text
> Russ Allbery via RT wrote:
>
>>> Just to be clear, the problem happens when the ticket cache is
>>> refreshed. I.e., the tickets for the existing SASL/GSSAPI connection
>>> hadn't actually yet expired, just the ticket cache was refreshed with
>>> new tickets. I can understand why the SASL/GSSAPI context would be
>>> closed out on *expiration* but I think a refresh shouldn't have this
>>> effect. ;)
>
> If it is possible, can you post a stack trace at the point the context
> is deemed to be invalid?
>
> That would help a lot.

Hm, after going back through the thread, I can't tell specifically if it is
actually the refresh or the expiration that caused the problem, because the
user set it to a 5 minute ticket with a 4 minute refresh to demonstrate the
issue.

I myself do not use MIT kerberos for my OpenLDAP servers, so reproducing
this in my environment would take a bit of work. I'm currently lacking the
internal development environment where I'd usually test such things. :/

I can get in contact with the user who reported the issue, and see what
additional data they can gather, if you like.

--Quanah


--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
Date: Wed, 06 Sep 2006 18:18:50 -0400
From: Jeffrey Altman <jaltman@mit.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:
Quanah Gibson-Mount via RT wrote:
Show quoted text
> I can get in contact with the user who reported the issue, and see what
> additional data they can gather, if you like.

We need to know exactly what is wrong if we are going to have a chance
of improving the behavior. I don't see any place in the code that the
credential cache would be examined once the context is established.

Jeffrey Altman
To: rt@krbdev.mit.edu
From: Simon Wilkinson <simon@sxw.org.uk>
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
Date: Fri, 8 Sep 2006 10:32:59 +0100
RT-Send-Cc:
As the person quoted right at the beginning, I should probably
contribute my findings here.

I don't believe that ticket refresh is an issue. I can quite happily
refresh, destroy, or replace my Kerberos credentials from under a
running GSSAPI context, without causing that context to break.

The issue (if there is an issue) is that Heimdal and MIT's behaviour
differ when the initiator's credentials do actually expire. Heimdal
allows the context to continue to be used for wrapping operations
past expiry - MIT expires the context, and calls to wrap() or unwrap
() fail. This difference in behaviour is an issue when using SASL
applications with security layers, as the only way to renew the
context is to reconnect to the server. In addition, many applications
have inadequate error handling around their security layer
implementations.

I suspect that the current MIT behaviour is correct. Whilst there's
no explicit language in RFC2743, it suggests that the length of time
for which the context will be valid depends on credential lifetime.

Simon.
Date: Fri, 08 Sep 2006 09:08:54 -0700
From: Quanah Gibson-Mount <quanah@stanford.edu>
To: rt-comment@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.4KiB


--On Friday, September 08, 2006 5:33 AM -0400 Simon Wilkinson via RT
<rt-comment@krbdev.mit.edu> wrote:

Show quoted text
> As the person quoted right at the beginning, I should probably
> contribute my findings here.
>
> I don't believe that ticket refresh is an issue. I can quite happily
> refresh, destroy, or replace my Kerberos credentials from under a
> running GSSAPI context, without causing that context to break.
>
> The issue (if there is an issue) is that Heimdal and MIT's behaviour
> differ when the initiator's credentials do actually expire. Heimdal
> allows the context to continue to be used for wrapping operations
> past expiry - MIT expires the context, and calls to wrap() or unwrap
> () fail. This difference in behaviour is an issue when using SASL
> applications with security layers, as the only way to renew the
> context is to reconnect to the server. In addition, many applications
> have inadequate error handling around their security layer
> implementations.
>
> I suspect that the current MIT behaviour is correct. Whilst there's
> no explicit language in RFC2743, it suggests that the length of time
> for which the context will be valid depends on credential lifetime.


Thanks Simon for the follow-up. So it sounds like then, the error here
really is inside cyrus-sasl then?

--Quanah

--
Quanah Gibson-Mount
Principal Software Developer
ITS/Shared Application Services
Stanford University
GnuPG Public Key: http://www.stanford.edu/~quanah/pgp.html
From: Russ Allbery <rra@stanford.edu>
To: rt@krbdev.mit.edu
Subject: Re: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
Date: Fri, 08 Sep 2006 14:01:28 -0700
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.2KiB
Quanah Gibson-Mount via RT <rt-comment@krbdev.mit.edu> writes:

Show quoted text
> Thanks Simon for the follow-up. So it sounds like then, the error here
> really is inside cyrus-sasl then?

There is at least *some* error inside Cyrus SASL. The behavior that we're
seeing (in a different context than the one Quanah originally raised) is
that Cyrus SASL will go into a tight loop inside the library logging
messages about expired contexts without ever returning to the application.
That's clearly broken. I just haven't been able to find where the
brokenness is yet (mostly because I haven't had a chance to look in
depth).

Whether there's also a separate error in Kerberos is a different question.
It's looking to me like there's actually (arguable) incorrect behavior in
Heimdal, in that once a Kerberos ticket expires, I think a strong argument
can be made that the products of that ticket, such as the session key used
to provide confidentiality, are no longer valid either.

I don't know what that would mean for, say, a version of ssh that did
integrity protection using GSSAPI, though. Having your login session go
away because your original ticket expired might be technically correct but
sounds rather bad.

--
Russ Allbery (rra@stanford.edu) <http://www.eyrie.org/~eagle/>
Subject: [krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed
Date: Fri, 8 Sep 2006 10:56:28 -0600
From: "Moore, Patrick" <pcmoore@sandia.gov>
To: krb5-bugs@mit.edu
RT-Send-Cc:
[krbdev.mit.edu #4222] GSSAPI context being destroyed when ticket cache renewed

If you have the ability to hack the code, I believe you can ignore the GSS_S_CONTEXT_EXPIRED after your calls to gss_wrap() or gss_unwrap()  -- relying on the MIT implementation to complete the function before checking the context lifetime and returning the error.  I  hope they continue to code it this way, as I know of some applications that rely on this undocumented feature.

 

This is the same as #6739 which we resolved in 1.9 and 1.8.3.