Ticket History #8614: Assertion failure when repeatedly loading libgssapi

# Thu Sep 21 11:02:49 2017 Dmitri Matveev <matveev1@yahoo.com> - Ticket created

Date:	Thu, 21 Sep 2017 06:29:39 +0000 (UTC)
From:	Dmitri Matveev <matveev1@yahoo.com>
To:	"krb5-bugs@mit.edu" <krb5-bugs@mit.edu>
Subject:	Error with SAP NW 7.51 krb5int_key_register: Assertion `destructors_set[keynum] == 0' failed

Download (untitled) / with headers
text/plain 3.5KiB

Download (untitled) / with headers
text/html 8.3KiB

Hello,

We use MIT Kerberos krb5-1.15.1 via SSO for SAP ABAP authentication user.

OS version RHEL 7.3 with last kernel patch

Linux sapfs2ci 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux

After update krb5* to last version:

krb5-libs.x86_64 1.15.1-8.el7
krb5-workstation.x86_64 1.15.1-8.el7

Ticket is received from AD an work properly:

sapfs2ci:fs2adm 65> /usr/bin/kinit -k -V SAP/S41@CO.METINVEST.COM
Using default cache: /tmp/krb5cc_1039
Using principal: SAP/S41@CO.METINVEST.COM
Authenticated to Kerberos v5

sapfs2ci:fs2adm 66> /usr/bin/klist -f
Ticket cache: FILE:/tmp/krb5cc_1039
Default principal: SAP/S41@CO.METINVEST.COM

Valid starting Expires Service principal
09/21/2017 09:27:51 09/21/2017 19:27:51 krbtgt/CO.METINVEST.COM@CO.METINVEST.COM
renew until 09/22/2017 09:27:51, Flags: RIA

We have problem with start SAP APP server (used last SAP kernel 749 Linux GNU SLES-11 x86_64 cc4.8.5 use-pr170902 path 315)

In logs stderr0 (SAP APP)^

dw.sapFS2_D00: threads.c:357: krb5int_key_register: Assertion `destructors_set[keynum] == 0' failed

In log main SAP process disp+work:

N SncInit(): found snc/gssapi_lib=/usr/local/lib/libgssapi_krb5.so
N File "/usr/local/lib/libgssapi_krb5.so" dynamically loaded as GSS-API v2 library.
M si_code = -6
M Sending process ID = 28307
M Real user ID of sending process = 1039
M ------------------ C-STACK ----------------------
dw.sapFS2_D00[S](LinStackBacktrace+0xa9)[0x654518]
dw.sapFS2_D00[S](LinStack+0x3e)[0x657484]
dw.sapFS2_D00[S](CTrcStack2+0x4b)[0x6532dd]
dw.sapFS2_D00[S](SigIGenAction+0x730)[0x2553940]
<signal handler called> [0x7f5689fee5e0]
libc.so.6[S](__GI_raise+0x37)[0x7f5689c511f7]
libc.so.6[S](__GI_abort+0x148)[0x7f5689c528e8]
libc.so.6[S](__assert_fail_base+0x126)[0x7f5689c4a266]
libc.so.6[S](__GI___assert_fail+0x42)[0x7f5689c4a312]
libkrb5support.so.0[S](krb5int_key_register+0xf4)[0x7f53a7478694]
libgssapi_krb5.so[gssapi_krb5.c:1091](gss_krb5int_lib_init+0x2a)[0x7f567eece41a]
libgssapi_krb5.so[g_initialize.c:120](gssint_mechglue_init__aux+0x1f)[0x7f567eebec8f]
libpthread.so.0[S](__GI___pthread_once+0x50)[0x7f5689febe20]
libgssapi_krb5.so[g_initialize.c:156](gssint_mechglue_initialize_library+0x17)[0x7f567eec3297]
libgssapi_krb5.so[g_initialize.c:277](gss_indicate_mechs+0x4d)[0x7f567eec331d]
dw.sapFS2_D00[S](SncPDLInit+0x49a)[0x1b9dee6]
dw.sapFS2_D00[S](SncInit+0xa5c)[0x1ba0012]
dw.sapFS2_D00[S](SncInitU+0x49)[0x1ba1168]
dw.sapFS2_D00[S](ThSncInit(unsigned char)+0x8b)[0x51fce3]
dw.sapFS2_D00[S](ThInit()+0x1794)[0x4eebf3]
dw.sapFS2_D00[S](ThStart()+0x35c)[0x4ef6c6]
dw.sapFS2_D00[S](DpMain+0x39b)[0x229311b]
libc.so.6[S](__libc_start_main+0xf5)[0x7f5689c3dc05]
M -------------------------------------------------
M caught SIGABRT (6)

Downgrade version krb5-libs.x86_64 to 1.14.1 not helped

Downgrade version SAP KERNEL not helped

Downgrade version RHEL KERNEL not helped

We have many SAP system worked on krb5-1.14.1 for a SAP SSO

# Thu Sep 21 16:53:19 2017 ghudson@mit.edu (Greg Hudson) - Correspondence added

Download (untitled) / with headers
text/plain 849B

I believe this happens when two different versions of
libgssapi_krb5.so are loaded into the same process (in this case,
probably the one under /usr/lib and the one in /usr/local/lib), but
they both wind up using the same instance of libkrb5support.so. The
initializer runs in both copies of libgssapi_krb5, and tries to re-
register the same thread-specific data key, triggering the assertion
failure.

Debugging these kinds of problems can be difficult, as it requires
tracking down what is loading each version of libgssapi_krb5.so. In
this case it seems like SAP is loading the version from
/usr/local/lib, but I don't know what is loading the one from
/usr/lib.

(I'm also a little confused as to how a process could manage to load
two different versions of libgssapi_krb5.so as we haven't changed its
soname since the 1.2 release.)

# Fri Sep 22 02:15:44 2017 Dmitri Matveev <matveev1@yahoo.com> - Comments added

Date:	Fri, 22 Sep 2017 06:15:37 +0000 (UTC)
From:	Dmitri Matveev <matveev1@yahoo.com>
To:	"rt-comment@krbdev.mit.edu" <rt-comment@krbdev.mit.edu>
Subject:	Re: [krbdev.mit.edu #8614] Error with SAP NW 7.51 krb5int_key_register: Assertion `destructors_set[keynum] == 0' failed
RT-Send-Cc:

Download (untitled) / with headers
text/plain 4.5KiB

Download (untitled) / with headers
text/html 10.9KiB

Hello, Dmitri!

Yes, problem is an inconsistent version lib in /usr/lib64/ and /usr/local/lib/

ls -la /usr/lib64/libkrb5support.so.0.1
-rwxr-xr-x. 1 root root 58728 Apr 28 22:49 /usr/lib64/libkrb5support.so.0.1

ls -la /usr/local/lib/libkrb5support.so.0.1
-rw-r--r--. 1 root root 181632 Sep 21 09:49 /usr/local/lib/libkrb5support.so.0.1

ls -la /usr/lib64/libgssapi_krb5.so.2.2
-rwxr-xr-x. 1 root root 320776 Apr 28 22:49 /usr/lib64/libgssapi_krb5.so.2.2

ls -la /usr/local/lib/libgssapi_krb5.so.2.2
-rw-r--r--. 1 root root 1929168 Sep 21 09:49 /usr/local/lib/libgssapi_krb5.so.2.2

I try set profile parameter SAP APP to:

snc/gssapi_lib = /usr/lib64/libgssapi_krb5.so.2

and SAP APP start with SSO

but previous version 1.14.1 we set this parameter to:

snc/gssapi_lib = /usr/local/lib/libgssapi_krb5.so

I'm not sure now if the correct library is working.

On Thursday, September 21, 2017 6:03 PM, Dmitri Matveev via RT <rt-comment@krbdev.mit.edu> wrote:

Hello,We use MIT Kerberos krb5-1.15.1 via SSO for SAP ABAP authentication user. OS version RHEL 7.3 with last kernel patchLinux sapfs2ci 3.10.0-693.2.2.el7.x86_64 #1 SMP Sat Sep 9 03:55:24 EDT 2017 x86_64 x86_64 x86_64 GNU/Linux After update krb5* to last version:krb5-libs.x86_64 1.15.1-8.el7

krb5-workstation.x86_64 1.15.1-8.el7

Ticket is received from AD an work properly: sapfs2ci:fs2adm 65> /usr/bin/kinit -k -V SAP/S41@CO.METINVEST.COM

Using default cache: /tmp/krb5cc_1039

Using principal: SAP/S41@CO.METINVEST.COM

Authenticated to Kerberos v5

sapfs2ci:fs2adm 66> /usr/bin/klist -f

Ticket cache: FILE:/tmp/krb5cc_1039

Default principal: SAP/S41@CO.METINVEST.COMValid starting Expires Service principal

09/21/2017 09:27:51 09/21/2017 19:27:51 krbtgt/CO.METINVEST.COM@CO.METINVEST.COM

renew until 09/22/2017 09:27:51, Flags: RIA

We have problem with start SAP APP server (used last SAP kernel 749 Linux GNU SLES-11 x86_64 cc4.8.5 use-pr170902 path 315)In logs stderr0 (SAP APP)^

dw.sapFS2_D00: threads.c:357: krb5int_key_register: Assertion `destructors_set[keynum] == 0' failed

In log main SAP process disp+work:

N SncInit(): found snc/gssapi_lib=/usr/local/lib/libgssapi_krb5.so

N File "/usr/local/lib/libgssapi_krb5.so" dynamically loaded as GSS-API v2 library.

M si_code = -6

M Sending process ID = 28307

M Real user ID of sending process = 1039

M ------------------ C-STACK ----------------------

dw.sapFS2_D00[S](LinStackBacktrace+0xa9)[0x654518]

dw.sapFS2_D00[S](LinStack+0x3e)[0x657484]

dw.sapFS2_D00[S](CTrcStack2+0x4b)[0x6532dd]

dw.sapFS2_D00[S](SigIGenAction+0x730)[0x2553940]

<signal handler called> [0x7f5689fee5e0]

libc.so.6[S](__GI_raise+0x37)[0x7f5689c511f7]

libc.so.6[S](__GI_abort+0x148)[0x7f5689c528e8]

libc.so.6[S](__assert_fail_base+0x126)[0x7f5689c4a266]

libc.so.6[S](__GI___assert_fail+0x42)[0x7f5689c4a312]

libkrb5support.so.0[S](krb5int_key_register+0xf4)[0x7f53a7478694]

libgssapi_krb5.so[gssapi_krb5.c:1091](gss_krb5int_lib_init+0x2a)[0x7f567eece41a]

libgssapi_krb5.so[g_initialize.c:120](gssint_mechglue_init__aux+0x1f)[0x7f567eebec8f]

libpthread.so.0[S](__GI___pthread_once+0x50)[0x7f5689febe20]

libgssapi_krb5.so[g_initialize.c:156](gssint_mechglue_initialize_library+0x17)[0x7f567eec3297]

libgssapi_krb5.so[g_initialize.c:277](gss_indicate_mechs+0x4d)[0x7f567eec331d]

dw.sapFS2_D00[S](SncPDLInit+0x49a)[0x1b9dee6]

dw.sapFS2_D00[S](SncInit+0xa5c)[0x1ba0012]

dw.sapFS2_D00[S](SncInitU+0x49)[0x1ba1168]

dw.sapFS2_D00[S](ThSncInit(unsigned char)+0x8b)[0x51fce3]

dw.sapFS2_D00[S](ThInit()+0x1794)[0x4eebf3]

dw.sapFS2_D00[S](ThStart()+0x35c)[0x4ef6c6]

dw.sapFS2_D00[S](DpMain+0x39b)[0x229311b]

libc.so.6[S](__libc_start_main+0xf5)[0x7f5689c3dc05]

M -------------------------------------------------

M caught SIGABRT (6)

Downgrade version krb5-libs.x86_64 to 1.14.1 not helpedDowngrade version SAP KERNEL not helpedDowngrade version RHEL KERNEL not helped

We have many SAP system worked on krb5-1.14.1 for a SAP SSO

_______________________________________________
krb5-bugs mailing list
krb5-bugs@mit.edu
https://mailman.mit.edu/mailman/listinfo/krb5-bugs

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Status changed from 'new' to 'resolved'

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Taken

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Tags pullup added

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Target_Version 1.18-next added

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Target_Version 1.17-next added

# Tue Oct 20 19:27:56 2020 ghudson@mit.edu (Greg Hudson) - Correspondence added

Subject:	git commit
From:	ghudson@mit.edu

Download (untitled) / with headers
text/plain 1008B

Unregister thread key in SPNEGO finalization

Commit d160bc733a3dbeb6d84f4e175234ff18738d9f66 (ticket 7045) added a
new thread key K5_KEY_GSS_SPNEGO_STATUS and registered it in SPNEGO
library initialization, but neglected to unregister it in
finalization. As a result, loading, unloading, and reloading
libgssapi_krb5 could throw an assertion failure if libkrb5support
remained loaded. Unregister the key in SPNEGO finalization and add a
test case.

Reported and investigated by Adam Dabrowski.

https://github.com/krb5/krb5/commit/07ff54d0bb85109df114612bbbfa6559f4a1e0cb
Author: Greg Hudson <ghudson@mit.edu>
Commit: 07ff54d0bb85109df114612bbbfa6559f4a1e0cb
Branch: master
.gitignore | 1 +
src/lib/gssapi/spnego/spnego_mech.c | 1 +
src/tests/gssapi/Makefile.in | 34 ++++++++------
src/tests/gssapi/deps | 2 +
src/tests/gssapi/reload.c | 83 +++++++++++++++++++++++++++++++++++
5 files changed, 106 insertions(+), 15 deletions(-)

# Tue Nov 03 13:14:13 2020 ghudson@mit.edu (Greg Hudson) - Correspondence added

Subject:	git commit
From:	ghudson@mit.edu

Download (untitled) / with headers
text/plain 1KiB

Unregister thread key in SPNEGO finalization

Commit d160bc733a3dbeb6d84f4e175234ff18738d9f66 (ticket 7045) added a
new thread key K5_KEY_GSS_SPNEGO_STATUS and registered it in SPNEGO
library initialization, but neglected to unregister it in
finalization. As a result, loading, unloading, and reloading
libgssapi_krb5 could throw an assertion failure if libkrb5support
remained loaded. Unregister the key in SPNEGO finalization and add a
test case.

Reported and investigated by Adam Dabrowski.

(cherry picked from commit 07ff54d0bb85109df114612bbbfa6559f4a1e0cb)

https://github.com/krb5/krb5/commit/0d092a739429610af4824c460171ac529601d1f5
Author: Greg Hudson <ghudson@mit.edu>
Commit: 0d092a739429610af4824c460171ac529601d1f5
Branch: krb5-1.18
.gitignore | 1 +
src/lib/gssapi/spnego/spnego_mech.c | 1 +
src/tests/gssapi/Makefile.in | 11 +++--
src/tests/gssapi/deps | 2 +
src/tests/gssapi/reload.c | 83 +++++++++++++++++++++++++++++++++++
5 files changed, 94 insertions(+), 4 deletions(-)

# Tue Nov 03 13:14:14 2020 ghudson@mit.edu (Greg Hudson) - Correspondence added

From:	ghudson@mit.edu
Subject:	git commit

Download (untitled) / with headers
text/plain 1KiB

Unregister thread key in SPNEGO finalization

Commit d160bc733a3dbeb6d84f4e175234ff18738d9f66 (ticket 7045) added a
new thread key K5_KEY_GSS_SPNEGO_STATUS and registered it in SPNEGO
library initialization, but neglected to unregister it in
finalization. As a result, loading, unloading, and reloading
libgssapi_krb5 could throw an assertion failure if libkrb5support
remained loaded. Unregister the key in SPNEGO finalization and add a
test case.

Reported and investigated by Adam Dabrowski.

(cherry picked from commit 07ff54d0bb85109df114612bbbfa6559f4a1e0cb)

https://github.com/krb5/krb5/commit/994f5f5107676aa127c680f91df3f682b89cfd88
Author: Greg Hudson <ghudson@mit.edu>
Commit: 994f5f5107676aa127c680f91df3f682b89cfd88
Branch: krb5-1.17
.gitignore | 1 +
src/lib/gssapi/spnego/spnego_mech.c | 1 +
src/tests/gssapi/Makefile.in | 11 +++--
src/tests/gssapi/deps | 2 +
src/tests/gssapi/reload.c | 83 +++++++++++++++++++++++++++++++++++
5 files changed, 94 insertions(+), 4 deletions(-)

# Tue Nov 03 14:24:07 2020 ghudson@mit.edu (Greg Hudson) - Tags pullup deleted

# Tue Nov 03 14:43:12 2020 ghudson@mit.edu (Greg Hudson) - Version_Fixed 1.17.2 added

# Tue Nov 03 14:43:12 2020 ghudson@mit.edu (Greg Hudson) - Version_Fixed 1.18.3 added

# Mon Nov 16 22:51:13 2020 ghudson@mit.edu (Greg Hudson) - Subject changed from 'Error with SAP NW 7.51 krb5int_key_register: Assertion `destructors_set[keynum] == 0' failed' to 'Assertion failure when repeatedly loading libgssapi_krb5'

Ticket History #8614: Assertion failure when repeatedly loading libgssapi_krb5