Ticket History #5712: Random issue reported by Kevin

# Wed Sep 05 17:10:50 2007 kpkoch (Kevin Koch) - Ticket created

Download (untitled) / with headers
text/plain 833B

NIM was built from the trunk on 9/4/2007 to verify that previously
reported bugs were still fixed after a hack was replaced with a real
solution.

During testing, a new problem was observed when using SecureCRT and an
open NIM GUI.

Several identities were defined; only one is legitimate for connecting
to Athena.

When modifying an identity, it would always pop to the top of the list
in the Basic view. Then SecureCRT would attempt to connect using
topmost identity instead of using the default identity. Even though
the topmost identity and the default identity had valid credentials,
NIM would prompt for a password for the topmost identity.

Attempts to reproduce the problem from a freshly rebooted virtual
machine or host XP PC failed, even though the host PC was rebooted
after the newly built NIM was installed.

# Wed Sep 05 17:32:11 2007 jaltman (Jeffrey Altman) - Subject changed from (no value) to 'Random issue reported by Kevin'

# Wed Sep 05 17:32:12 2007 jaltman (Jeffrey Altman) - Status changed from 'new' to 'open'

# Wed Sep 05 17:32:47 2007 jaltman (Jeffrey Altman) - Data error added

# Wed Sep 05 17:33:04 2007 jaltman (Jeffrey Altman) - AdminCc asanka (asanka@mit.edu) added

# Wed Sep 05 17:33:05 2007 jaltman (Jeffrey Altman) - AdminCc jaltman (Jeffrey Altman) added

# Wed Sep 05 17:33:53 2007 jaltman (Jeffrey Altman) - Ticket 5712 RefersTo ticket 5603.

# Wed Sep 05 17:37:09 2007 Jeffrey Altman <jaltman@secure-endpoints.com> - Correspondence added

Date:	Wed, 05 Sep 2007 17:39:08 -0400
From:	Jeffrey Altman <jaltman@secure-endpoints.com>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #5712]
RT-Send-Cc:

Download (untitled) / with headers
text/plain 1.5KiB

Kevin Koch via RT wrote:

Show quoted text

> NIM was built from the trunk on 9/4/2007 to verify that previously
> reported bugs were still fixed after a hack was replaced with a real
> solution.
>
> During testing, a new problem was observed when using SecureCRT and an
> open NIM GUI.
>
> Several identities were defined; only one is legitimate for connecting
> to Athena.
>
> When modifying an identity, it would always pop to the top of the list
> in the Basic view. Then SecureCRT would attempt to connect using
> topmost identity instead of using the default identity. Even though
> the topmost identity and the default identity had valid credentials,
> NIM would prompt for a password for the topmost identity.
>
> Attempts to reproduce the problem from a freshly rebooted virtual
> machine or host XP PC failed, even though the host PC was rebooted
> after the newly built NIM was installed.

It would be useful if you would debug the problem instead of reporting
symptoms viewed from 10,000ft. You still have an open SecureCRT ticket.
(RT 5603) You haven't described how SecureCRT is interacting with
Kerberos and NIM. Its obviously making a gssapi call or it could be
manipulating the credential cache configuration. Which is it?

If its making a gssapi call, is it requesting a specific identity or
asking for the identity in the default ccache?

If its manipulating the ccache, is it changing the default ccname?

What do you mean by "modifying an identity"? This is an extremely
ambiguous statement. The default identity is listed first in the view
so if the ccache is "default", then it will be first.

Download smime.p7s
application/x-pkcs7-signature 3.2KiB

Message body not shown because it is not plain text.

# Wed Sep 05 17:44:41 2007 jaltman (Jeffrey Altman) - Correspondence added

Download (untitled) / with headers
text/plain 134B

please indicate which version of SecureCRT you are using. If you are
using a version other than 5.1, provide links for downloading.

# Wed Sep 05 19:28:43 2007 jaltman (Jeffrey Altman) - Correspondence added

Download (untitled) / with headers
text/plain 1.5KiB

Debugging SecureCRT.exe. Start Visual Studio devenv.exe or windbg.exe.
Open SecureCRT.exe as the current project. Open source file lshfunc.c
and place break points within acquire_tkt_no_princ().

What this shows is that SecureCRT does not call gss_acquire_cred()
directly. Instead it calls gss_init_sec_context() with
GSS_C_NO_CREDENTIAL. This results in a call to gss_acquire_cred() with
desired_name equal to GSS_C_NO_CREDENTIAL which in turn calls
acquire_init_cred() with the same which results in a call to
krb5int_cc_default().

Now things get interesting. krb5int_cc_default() loads leashw32.dll and
searches for the function pLeash_AcquireInitialTicketsIfNeeded() which
when called without a desired principal name checks the current default
ccache for a valid TGT and if not found queries NIM to prompt the user.
The user can response with any principal name and NIM will return the
ccache name for the cache that holds the identity specified by the user.

Once the ccache name is determined, the ccache name is stored in the
krb5_context. The result is that subsequent changes to the default
ccache name in NIM (or manually in the registry) will not affect
SecureCRT until it is restarted. The call to krb5_cc_set_default_name()
is performed within krb5int_cc_default().

Next time you see a problem. Attach a debugger to SecureCRT.exe and
debug what is happening. Then you can file a report showing where the
actual problem is describing function calls and parameters.

As far as I can tell with SecureCRT 5.1 as downloaded from MIT, it is
working as designed.

# Thu Sep 06 09:46:32 2007 kpkoch (Kevin Koch) - Comments added

From:	"Kevin Koch" <kpkoch@MIT.EDU>
To:	<rt-comment@krbdev.mit.edu>
Subject:	RE: [krbdev.mit.edu #5712] Random issue reported by Kevin
Date:	Thu, 6 Sep 2007 09:46:22 -0400
RT-Send-Cc:

Download (untitled) / with headers
text/plain 388B

5.1 build 263.

Kevin

Show quoted text

-----Original Message-----
From: RT User,,,, [mailto:rt@MIT.EDU] On Behalf Of Jeffrey Altman via RT
Sent: Wednesday, September 05, 2007 5:45 PM
To: kpkoch@MIT.EDU
Subject: [krbdev.mit.edu #5712] Random issue reported by Kevin

please indicate which version of SecureCRT you are using. If you are
using a version other than 5.1, provide links for downloading.

# Thu Sep 06 09:50:48 2007 kpkoch (Kevin Koch) - Comments added

From:	"Kevin Koch" <kpkoch@MIT.EDU>
To:	<rt-comment@krbdev.mit.edu>
Subject:	RE: [krbdev.mit.edu #5712]
Date:	Thu, 6 Sep 2007 09:50:38 -0400
RT-Send-Cc:

Download (untitled) / with headers
text/plain 2.2KiB

Jeff --

From the bottom line of your message, maybe the problem is that the most
recently 'modified' identity pops to the top of the list, even if it isn't
the default. Usually 'modified' meant obtaining new credentials. This
would be consistent with your later email concluding that SecureCRT is
behaving as expected.

I could reproduce the problem for a while and then for the rest of the day
yesterday couldn't. Very frustrating.

Kevin

Show quoted text

-----Original Message-----
From: 0000-Admin [mailto:daemon@MIT.EDU] On Behalf Of Jeffrey Altman via RT
Sent: Wednesday, September 05, 2007 5:37 PM
To: kpkoch@mit.edu
Subject: Re: [krbdev.mit.edu #5712]

Kevin Koch via RT wrote:

> NIM was built from the trunk on 9/4/2007 to verify that previously
> reported bugs were still fixed after a hack was replaced with a real
> solution.
>
> During testing, a new problem was observed when using SecureCRT and an
> open NIM GUI.
>
> Several identities were defined; only one is legitimate for connecting
> to Athena.
>
> When modifying an identity, it would always pop to the top of the list
> in the Basic view. Then SecureCRT would attempt to connect using
> topmost identity instead of using the default identity. Even though
> the topmost identity and the default identity had valid credentials,
> NIM would prompt for a password for the topmost identity.
>
> Attempts to reproduce the problem from a freshly rebooted virtual
> machine or host XP PC failed, even though the host PC was rebooted
> after the newly built NIM was installed.

It would be useful if you would debug the problem instead of reporting
symptoms viewed from 10,000ft. You still have an open SecureCRT ticket.
(RT 5603) You haven't described how SecureCRT is interacting with
Kerberos and NIM. Its obviously making a gssapi call or it could be
manipulating the credential cache configuration. Which is it?

If its making a gssapi call, is it requesting a specific identity or
asking for the identity in the default ccache?

If its manipulating the ccache, is it changing the default ccname?

What do you mean by "modifying an identity"? This is an extremely
ambiguous statement. The default identity is listed first in the view
so if the ccache is "default", then it will be first.

# Thu Sep 06 09:51:42 2007 kpkoch (Kevin Koch) - Comments added

From:	"Kevin Koch" <kpkoch@MIT.EDU>
To:	<rt-comment@krbdev.mit.edu>
Subject:	RE: [krbdev.mit.edu #5712] Random issue reported by Kevin
Date:	Thu, 6 Sep 2007 09:51:32 -0400
RT-Send-Cc:

Download (untitled) / with headers
text/plain 1.8KiB

Jeff --

That was very helpful.

Thanks.

Kevin

Show quoted text

-----Original Message-----
From: RT User,,,, [mailto:rt@MIT.EDU] On Behalf Of Jeffrey Altman via RT
Sent: Wednesday, September 05, 2007 7:29 PM
To: kpkoch@MIT.EDU
Subject: [krbdev.mit.edu #5712] Random issue reported by Kevin

Debugging SecureCRT.exe. Start Visual Studio devenv.exe or windbg.exe.
Open SecureCRT.exe as the current project. Open source file lshfunc.c
and place break points within acquire_tkt_no_princ().

What this shows is that SecureCRT does not call gss_acquire_cred()
directly. Instead it calls gss_init_sec_context() with
GSS_C_NO_CREDENTIAL. This results in a call to gss_acquire_cred() with
desired_name equal to GSS_C_NO_CREDENTIAL which in turn calls
acquire_init_cred() with the same which results in a call to
krb5int_cc_default().

Now things get interesting. krb5int_cc_default() loads leashw32.dll and
searches for the function pLeash_AcquireInitialTicketsIfNeeded() which
when called without a desired principal name checks the current default
ccache for a valid TGT and if not found queries NIM to prompt the user.
The user can response with any principal name and NIM will return the
ccache name for the cache that holds the identity specified by the user.

Once the ccache name is determined, the ccache name is stored in the
krb5_context. The result is that subsequent changes to the default
ccache name in NIM (or manually in the registry) will not affect
SecureCRT until it is restarted. The call to krb5_cc_set_default_name()
is performed within krb5int_cc_default().

Next time you see a problem. Attach a debugger to SecureCRT.exe and
debug what is happening. Then you can file a report showing where the
actual problem is describing function calls and parameters.

As far as I can tell with SecureCRT 5.1 as downloaded from MIT, it is
working as designed.

# Thu Sep 06 10:10:10 2007 Jeffrey Altman <jaltman@secure-endpoints.com> - Correspondence added

Date:	Thu, 06 Sep 2007 10:12:07 -0400
From:	Jeffrey Altman <jaltman@secure-endpoints.com>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #5712]
RT-Send-Cc:

Download (untitled) / with headers
text/plain 736B

Kevin Koch via RT wrote:

Show quoted text

> Jeff --
>
> From the bottom line of your message, maybe the problem is that the most
> recently 'modified' identity pops to the top of the list, even if it isn't
> the default. Usually 'modified' meant obtaining new credentials. This
> would be consistent with your later email concluding that SecureCRT is
> behaving as expected.
>
> I could reproduce the problem for a while and then for the rest of the day
> yesterday couldn't. Very frustrating.
>
> Kevin

Is the identity that you are obtaining credentials for alphabetically
first based upon the current sort order?

Identities with creds come first, then identities without creds.
If you click on the column header you change the sort order.

Download smime.p7s
application/x-pkcs7-signature 3.2KiB

Message body not shown because it is not plain text.

# Thu Sep 06 10:42:29 2007 kpkoch (Kevin Koch) - Comments added

Download (untitled) / with headers
text/plain 764B

Today I see this behavior on XP:

Define two identities: one which can be used to connect to Athena and
testprinc@SUCHDAMAGE.ORG (password foobarbaz). Set testprinc to
expire in 5 minutes and not be renewable. Wait for it to expire.

Open the NIM GUI. In the basic view, click the Identity column header
to sort the real identity to the top of the list. Close the GUI.

Open SecureCRT and connect to Athena. It works. Disconnect from
Athena and close SecureCRT.

Open the NIM GUI, sort the Badic view identity list so that testprinc
is at the top and close the GUI.

Open SecureCRT and connect to Athena. I get a GSSAPI error.

Another clue? The failure when testprinc is at the top of the Basic
view identity list does not happen if the GUI is open.

# Thu Sep 06 11:00:12 2007 jaltman (Jeffrey Altman) - Correspondence added

Date:	Thu, 06 Sep 2007 11:02:16 -0400
From:	Jeffrey Altman <jaltman@mit.edu>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #5712] Random issue reported by Kevin
RT-Send-Cc:

Download (untitled) / with headers
text/plain 1.3KiB

Kevin Koch via RT wrote:

Show quoted text

> Today I see this behavior on XP:
>
> Define two identities: one which can be used to connect to Athena and
> testprinc@SUCHDAMAGE.ORG (password foobarbaz). Set testprinc to
> expire in 5 minutes and not be renewable. Wait for it to expire.
>
> Open the NIM GUI. In the basic view, click the Identity column header
> to sort the real identity to the top of the list. Close the GUI.
>
> Open SecureCRT and connect to Athena. It works. Disconnect from
> Athena and close SecureCRT.
>
> Open the NIM GUI, sort the Badic view identity list so that testprinc
> is at the top and close the GUI.
>
> Open SecureCRT and connect to Athena. I get a GSSAPI error.

You are getting an error with a ticket that is near expiration.
Perhaps your VM clock and the server clock are off by a couple of
minutes and the service ticket is being rejected. In this case
the ticket appears to be valid to the client according to its
clock and is invalid on the server.

Attach a debugger to SecureCRT.exe and debug the issue.

These descriptions of symptoms do not help anyone because
they can't be used to reproduce the issue.

Show quoted text

> Another clue? The failure when testprinc is at the top of the Basic
> view identity list does not happen if the GUI is open.

A clue to what? The only clues are in the code. Debug the code.

# Thu Sep 06 11:06:51 2007 kpkoch (Kevin Koch) - Correspondence added

Download (untitled) / with headers
text/plain 525B

I forgot to add that the real identity is the default identity in all
cases. Never is testprinc the default identity.

Show quoted text

> You are getting an error with a ticket that is near expiration.
> Perhaps your VM clock and the server clock are off by a couple of
> minutes and the service ticket is being rejected. In this case
> the ticket appears to be valid to the client according to its
> clock and is invalid on the server.

testprinc is expired. Not near expiration, expired.

This is happening on the host PC, not in a VM.

# Thu Sep 06 11:34:29 2007 jaltman (Jeffrey Altman) - Correspondence added

Date:	Thu, 06 Sep 2007 11:36:36 -0400
From:	Jeffrey Altman <jaltman@mit.edu>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #5712] Random issue reported by Kevin
RT-Send-Cc:

Download (untitled) / with headers
text/plain 1.8KiB

Kevin Koch via RT wrote:

Show quoted text

> I forgot to add that the real identity is the default identity in all
> cases. Never is testprinc the default identity.
>

>> You are getting an error with a ticket that is near expiration.
>> Perhaps your VM clock and the server clock are off by a couple of
>> minutes and the service ticket is being rejected. In this case
>> the ticket appears to be valid to the client according to its
>> clock and is invalid on the server.

>
> testprinc is expired. Not near expiration, expired.
>
> This is happening on the host PC, not in a VM.

Attach a debugger and figure out what ccache is being given to
SecureCRT when it asks for one. SecureCRT is only going to use
the ccache that it is given the first time it asks for the default
ccache. This ccache is going to come from the krb5int_cc_default()
function. It is not going to come from NIM unless krb5int_cc_default()
determines via a call to leashw32.dll that there are no valid
credentials in the ccache that is default for the SecureCRT.exe
process.

Note that each krb5_context has its own notion of the default ccache.
Once a krb5_context is assigned a default ccache the value set by
NIM becomes irrelevant. The NIM default ccache is communicated to
krb5int_cc_default() via the registry.

HKCU\SOFTWARE\MIT\Kerberos5\ "ccname"

If that value and NIM disagree about what the default ccache is
that is a problem. If not, the problem is not in NIM.

When NIM is queried it is always to prompt the user. The obtain
new creds dialog is displayed. The last identity the user obtained
creds for will be displayed as the default. The user can enter
any Kerberos principal of her choosing. If she wants an identity
that already has credentials she can simply enter the name and
press 'Ok'. NIM will return the ccache name for that identity to
the requesting application.

# Tue Sep 11 09:43:24 2007 kpkoch (Kevin Koch) - Correspondence added

Download (untitled) / with headers
text/plain 710B

Now I am confused.

The original issues list for the KfW release team had a report of NIM
returning the default testprinc@SUCHDAMAGE.ORG to SecureCRT when
SecureCRT's username was kpkoch. In a private message, Sam basically
said 'NIM will return creds for the default identity.' That model of
what NIM does seemed to work until the post-KfW 3.2.1 changes.

If SecureCRT doesn't ask for a particular identity, are you saying
there are conditions under which NIM will return something other than
the default identity? Is this different from the 3.2.1 behavior?

[jaltman - Thu Sep 6 11:34:29 2007]:

Show quoted text

> Attach a debugger and figure out what ccache is being given to
> SecureCRT when it asks for one. ...

# Tue Sep 11 09:52:40 2007 jaltman (Jeffrey Altman) - Correspondence added

Date:	Tue, 11 Sep 2007 09:55:17 -0400
From:	Jeffrey Altman <jaltman@mit.edu>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #5712] Random issue reported by Kevin
RT-Send-Cc:

Download (untitled) / with headers
text/plain 945B

Kevin Koch via RT wrote:

Show quoted text

> Now I am confused.
>
> The original issues list for the KfW release team had a report of NIM
> returning the default testprinc@SUCHDAMAGE.ORG to SecureCRT when
> SecureCRT's username was kpkoch. In a private message, Sam basically
> said 'NIM will return creds for the default identity.' That model of
> what NIM does seemed to work until the post-KfW 3.2.1 changes.
>
> If SecureCRT doesn't ask for a particular identity, are you saying
> there are conditions under which NIM will return something other than
> the default identity? Is this different from the 3.2.1 behavior?

I am saying that you should read the code. Obtaining the "default
identity" does not involve NIM. By the time that NIM (or Leash) is
queried the user is being prompted and the user can choose to provide
credentials for whatever identity she wants to.

Attach a debugger, set break points, and walk the code.

Jeffrey Altman

# Tue Dec 31 16:44:13 2013 tlyu (Taylor Yu) - Queue changed from krb5 to kfw