Skip Menu |
 

Date: Mon, 17 Dec 2012 14:18:08 +0100
From: Joschi Brauchle <joschi.brauchle@tum.de>
To: krb5-bugs@mit.edu
Subject: Indefinite FD polling
Download (untitled) / with headers
text/plain 7.6KiB
Show quoted text
>Confidential: no
>Synopsis: Indefinite polling on (possibly non-existent) FD causes
hang and user lockout
Show quoted text
>Severity: serious
>Priority: medium
>Category: krb5-libs
>Class: sw-bug
>Release: 1.10.2
>Environment:
OS: openSUSE 12.2
System: Linux st-brauchle 3.4.11-2.16-desktop #1 SMP PREEMPT Wed Sep 26
17:05:00 UTC 2012 (259fc87) x86_64 x86_64 x86_64 GNU/Linux
Architecture: x86_64

Show quoted text
>Description:
After ticket expired, user ran 'kinit'. Then, rpc.gssd process was at
100%. Strace on process shows indefinite polling of FD.

This the GDB bt:
------------
#0 0x00007fa04f857104 in __GI___poll (fds=0x7fa051ce70b8, nfds=1,
timeout=-1242459) at ../sysdeps/unix/sysv/linux/poll.c:83
#1 0x00007fa04fda0134 in service_fds () from /usr/lib64/libkrb5.so.3
#2 0x00007fa04fda0f8c in k5_sendto () from /usr/lib64/libkrb5.so.3
#3 0x00007fa04fda140c in krb5_sendto_kdc () from /usr/lib64/libkrb5.so.3
#4 0x00007fa04fd776b5 in krb5_tkt_creds_get () from /usr/lib64/libkrb5.so.3
#5 0x00007fa04fd7781d in krb5_get_credentials () from
/usr/lib64/libkrb5.so.3
#6 0x00007fa04e6bcc1d in get_credentials.isra.0 () from
/usr/lib64/libgssapi_krb5.so
#7 0x00007fa04e6d2eaa in krb5_gss_init_sec_context_ext () from
/usr/lib64/libgssapi_krb5.so
#8 0x00007fa04e6d3541 in krb5_gss_init_sec_context () from
/usr/lib64/libgssapi_krb5.so
#9 0x00007fa04e6c4d86 in gss_init_sec_context () from
/usr/lib64/libgssapi_krb5.so
#10 0x00007fa04fffcbad in gss_init_sec_context
(minor_status=minor_status@entry=0x7fffe9b73d70,
claimant_cred_handle=0x7fa051caf9b0,
context_handle=context_handle@entry=0x7fa051cc1f68,
target_name=0x7fa051cbf6f0, req_mech_type=<optimized out>, req_flags=2,
time_req=time_req@entry=0,
input_chan_bindings=input_chan_bindings@entry=0x0,
input_token=input_token@entry=0x0,
actual_mech_type=actual_mech_type@entry=0x0,
output_token=output_token@entry=0x7fffe9b73d80,
ret_flags=ret_flags@entry=0x7fffe9b73d74, time_rec=time_rec@entry=0x0)
at g_init_sec_context.c:160
#11 0x00007fa05021fed7 in authgss_refresh
(auth=auth@entry=0x7fa051cae090) at auth_gss.c:422
#12 0x00007fa0502202b9 in authgss_create
(clnt=clnt@entry=0x7fa051cabcb0, name=0x7fa051cbc100,
sec=sec@entry=0x7fffe9b73ef0) at auth_gss.c:201
#13 0x00007fa0502203cf in authgss_create_default
(clnt=clnt@entry=0x7fa051cabcb0, service=0x7fa051ca8880
"nfs@gemini.lnt.ei.tum.de",
sec=sec@entry=0x7fffe9b73ef0) at auth_gss.c:233
#14 0x00007fa050655c53 in create_auth_rpc_client
(clp=clp@entry=0x7fa051ca6360,
clnt_return=clnt_return@entry=0x7fffe9b74388,
auth_return=auth_return@entry=0x7fffe9b74390, uid=uid@entry=10011,
authtype=authtype@entry=0) at gssd_proc.c:889
#15 0x00007fa050656180 in process_krb5_upcall
(clp=clp@entry=0x7fa051ca6360, uid=10011, fd=12,
tgtname=tgtname@entry=0x0, service=service@entry=0x0)
at gssd_proc.c:1014
#16 0x00007fa050656903 in handle_gssd_upcall
(clp=clp@entry=0x7fa051ca6360) at gssd_proc.c:1228
#17 0x00007fa05065487c in scan_poll_results (ret=1) at gssd_main_loop.c:84
#18 gssd_run () at gssd_main_loop.c:221
#19 0x00007fa0506535ef in main (argc=<optimized out>, argv=<optimized
Show quoted text
out>) at gssd.c:194
78 in ../sysdeps/unix/sysv/linux/poll.c
$1 = {fd = 16, events = 1, revents = 1}
93 in ../sysdeps/unix/sysv/linux/poll.c
0x00007fa04fda0134 in service_fds () from /usr/lib64/libkrb5.so.3
Single stepping until exit from function service_fds,
which has no line number information.
__GI___errno_location () at errno-loc.c:27
27 errno-loc.c: No such file or directory.
28 in errno-loc.c
0x00007fa04fda013b in service_fds () from /usr/lib64/libkrb5.so.3
Single stepping until exit from function service_fds,
which has no line number information.
__memcpy_ssse3_back () at ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:60
60 ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S: No such file or
directory.
74 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
75 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
95 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
96 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
128 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
130 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
131 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
132 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
134 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
135 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
136 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
137 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
138 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
139 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
140 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
141 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
142 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
143 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
144 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
145 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
146 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
147 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
148 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
149 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
150 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
845 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
847 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
848 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
849 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
851 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
852 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
853 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
855 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
856 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
857 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
859 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
860 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
861 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
863 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
864 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
865 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
867 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
868 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
869 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
871 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
872 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
873 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
875 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
876 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
877 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
879 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
880 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
881 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
Run till exit from #0 __memcpy_ssse3_back () at
../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S:881
0x00007fa04fda0123 in service_fds () from /usr/lib64/libkrb5.so.3
881 in ../sysdeps/x86_64/multiarch/memcpy-ssse3-back.S
Run till exit from #0 0x00007fa04fda0123 in service_fds () from
/usr/lib64/libkrb5.so.3

Program received signal SIGINT, Interrupt.
0x00007fa04f857104 in __GI___poll (fds=0x7fa051ce70b8, nfds=1,
timeout=-1378045) at ../sysdeps/unix/sysv/linux/poll.c:83
83 ../sysdeps/unix/sysv/linux/poll.c: No such file or directory.
A debugging session is active.

Inferior 1 [process 3228] will be detached.

Quit anyway? (y or n) Detaching from program: /usr/sbin/rpc.gssd,
process 3228
------------

Note that there is a very similar bug report #7454 at
http://krbdev.mit.edu/rt/Ticket/Display.html?id=7454, the proposed fix
from
https://github.com/krb5/krb5/commit/9eb2b4dfc136da326e54081ae18cb4d648c6500d
has *already been applied*. Hence, there must be another problem with
the FDs in the service_fds/k5_sendto function.


Show quoted text
>How-To-Repeat:
Not reproducible yet
Download smime.p7s
application/pkcs7-signature 4.4KiB

Message body not shown because it is not plain text.

If you can run a build with symbols so that you can get more information
the next time you observe this, that will be helpful in tracking this
down. In particular:

* Which call site of service_fds() is spinning (line numbers in the stack
trace will suffice)
* What are the contents of the selstate structure
* What are the contents of the conns list

Thanks.
Date: Tue, 18 Dec 2012 13:15:56 +0100
From: Joschi Brauchle <joschi.brauchle@tum.de>
To: rt-comment@krbdev.mit.edu
Subject: [krbdev.mit.edu #7508] Indefinite FD polling
RT-Send-Cc:
On 12/17/2012 05:58 PM, Greg Hudson via RT wrote:
Show quoted text
> If you can run a build with symbols so that you can get more information
> the next time you observe this, that will be helpful in tracking this
> down.

Sure, actually I thought I had symbols installed. Sorry 'bout that.

Anyways, they are installed now and I will try to reproduce the problem
with short ticket lifetimes + renewal.
Download smime.p7s
application/pkcs7-signature 4.4KiB

Message body not shown because it is not plain text.

From: ghudson@mit.edu
Subject: git commit

Fix spin loop reading from KDC TCP socket

In the k5_sendto code for reading from a TCP socket, detect
end-of-stream when reading the length. Otherwise we can get stuck in
an infinite loop of poll() and read().

[ghudson@mit.edu: commit message]

https://github.com/krb5/krb5/commit/53e5c850e05f011e9e7f25c2032aec51d8b352a9
Author: Viktor Dukhovni <viktor@twosigma.com>
Committer: Greg Hudson <ghudson@mit.edu>
Commit: 53e5c850e05f011e9e7f25c2032aec51d8b352a9
Branch: master
src/lib/krb5/os/sendto_kdc.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)
From: tlyu@mit.edu
Subject: git commit

Fix spin loop reading from KDC TCP socket

In the k5_sendto code for reading from a TCP socket, detect
end-of-stream when reading the length. Otherwise we can get stuck in
an infinite loop of poll() and read().

[ghudson@mit.edu: commit message]

(back ported from commit 53e5c850e05f011e9e7f25c2032aec51d8b352a9)

https://github.com/krb5/krb5/commit/a68c4fb6c498d2507a060db1deb44e0a5a42cd31
Author: Tom Yu <tlyu@mit.edu>
Commit: a68c4fb6c498d2507a060db1deb44e0a5a42cd31
Branch: krb5-1.11
src/lib/krb5/os/sendto_kdc.c | 4 ++--
1 files changed, 2 insertions(+), 2 deletions(-)