Skip Menu |
 

Date: Wed, 21 Nov 2012 13:27:03 +0100
From: Johannes Schilling <of82ecuq@stud.informatik.uni-erlangen.de>
To: krb5-bugs@mit.edu
Subject: select on set of all bad fds
CC: problems@stud.informatik.uni-erlangen.de
Download (untitled) / with headers
text/plain 1.1KiB
Submitter-Id: new
Originator: Johannes Schilling
Confidential: no
Synopsis: select on set of all bad fds
Severity: non-critical
Priority: medium
Category: krb5-libs
Class: sw-bug
Release: 1.10.1+dfsg
Environment: debian wheezy, various machines. all have the debian
wheezy 1.10.1+dfsg kerberos packages installed. // TODO: wollen die mehr?
System: Linux 3.0.31 #1 SMP x86_64 GNU/Linux
Machine: x86_64
Description: in src/lib/krb5/os/sendto_kdc.c:service_fds we iterate
over all fds in conns, doing a "continue" if the current fd is <0
(INVALID_SOCKET). now from time to time it happened that all these fds
were bad, so select() immediately returned, but nothing was done inside
the handling loop, so select() is called again, immediately returning,
... ssh was still functional, but sshds with >500 cpu-hours at 600h
uptime are not the nicest of things.
How-To-Repeat: this is where i'm not exactly sure. setup sshd with
kerberos and wait.
Fix: the attached patch checks if any fds are ok, and if not
returns 1, the error-code for which is also used if select failed, and
which is checked by the calling function so things can be
retried/aborted/...

Message body is not shown because sender requested not to inline it.

Do you really mean select()? I would expect krb5 1.10.x on Debian to use
poll(). That's not really important, though.

If there are no valid fds in the connection state list, then selstate-
Show quoted text
>nfds should be 0 and the loop should terminate. I can't find any bugs
which would violate that invariant. If you see this happening again, can
you examine the contents of the selstate structure?
Date: Wed, 28 Nov 2012 19:42:14 +0100
From: Alexander Wuerstlein <snalwuer@cip.informatik.uni-erlangen.de>
To: rt@krbdev.mit.edu
CC: CIP Support <problems@stud.informatik.uni-erlangen.de>
Subject: Re: [krbdev.mit.edu #7454] select on set of all bad fds
RT-Send-Cc:
Download (untitled) / with headers
text/plain 15.2KiB
On 12-11-28 17:46, Greg Hudson via RT <rt-comment@krbdev.mit.edu> wrote:
Show quoted text
> Do you really mean select()? I would expect krb5 1.10.x on Debian to use
> poll(). That's not really important, though.

Yes, but you are right, its poll(), here is what strace -p on such a process says:
poll([{fd=4, events=POLLIN}, {fd=10, events=POLLIN}], 2, 1779582387) = 2 ([{fd=4, revents=POLLIN}, {fd=10, revents=POLLIN}])
(repeated ad infinitum)

Show quoted text
> If there are no valid fds in the connection state list, then selstate-
> >nfds should be 0 and the loop should terminate. I can't find any bugs
> which would violate that invariant. If you see this happening again, can
> you examine the contents of the selstate structure?

For the full gdb session, see below.

In sendto_kdc.c line 1169, after cm_select_or_poll, there is a number of
fds in selstate, as you can see below. But the 'state->fd' of the only
connection in 'conns' is INVALID_SOCKET (see line 1179), so the for-loop
always calls continue and never calls state->service() to handle data on
the fds. Our patch detects if all connections in 'conns' are in the
INVALID_SOCKET state and exits.


Ciao,

Alexander Wuerstlein.


----

(gdb) bt
#0 0x00007f4a58e1ea88 in *__GI___poll (fds=fds@entry=0x7f4a5ad38ab8, nfds=2, timeout=<optimized out>, timeout@entry=1778794454)
at ../sysdeps/unix/sysv/linux/poll.c:83
#1 0x00007f4a59358004 in cm_select_or_poll (sret=<synthetic pointer>, out=0x7f4a5ad38ab8, in=0x7f4a5ad36aa0) at ../../../../src/lib/krb5/os/sendto_kdc.c:530
#2 service_fds (context=context@entry=0x7f4a5ad252e0, selstate=selstate@entry=0x7f4a5ad36aa0, interval=interval@entry=1, conns=conns@entry=0x7f4a5ad3be80,
seltemp=seltemp@entry=0x7f4a5ad38ab8, msg_handler=msg_handler@entry=0x7f4a59357880 <check_for_svc_unavailable>,
msg_handler_data=msg_handler_data@entry=0x7fff80e65c48, winner_out=winner_out@entry=0x7fff80e65b18) at ../../../../src/lib/krb5/os/sendto_kdc.c:1163
#3 0x00007f4a59358e5c in k5_sendto (context=context@entry=0x7f4a5ad252e0, message=message@entry=0x7fff80e65cd0, servers=servers@entry=0x7fff80e65c50,
socktype1=socktype1@entry=2, socktype2=1, callback_info=callback_info@entry=0x0, reply=reply@entry=0x7fff80e65ce0, remoteaddr=remoteaddr@entry=0x0,
remoteaddrlen=remoteaddrlen@entry=0x0, server_used=server_used@entry=0x7fff80e65c4c,
msg_handler=msg_handler@entry=0x7f4a59357880 <check_for_svc_unavailable>, msg_handler_data=msg_handler_data@entry=0x7fff80e65c48)
at ../../../../src/lib/krb5/os/sendto_kdc.c:1290
#4 0x00007f4a593592dc in krb5_sendto_kdc (context=context@entry=0x7f4a5ad252e0, message=message@entry=0x7fff80e65cd0, realm=realm@entry=0x7fff80e65cf0,
reply=reply@entry=0x7fff80e65ce0, use_master=use_master@entry=0x7fff80e65e08, tcp_only=tcp_only@entry=0) at ../../../../src/lib/krb5/os/sendto_kdc.c:339
#5 0x00007f4a59331efb in init_creds_get (context=context@entry=0x7f4a5ad252e0, ctx=0x7f4a5ad25b90, use_master=use_master@entry=0x7fff80e65e08)
at ../../../../src/lib/krb5/krb/get_in_tkt.c:578
#6 0x00007f4a5933202d in krb5int_get_init_creds (context=context@entry=0x7f4a5ad252e0, creds=creds@entry=0x7f4a5ad25a30, client=client@entry=0x7f4a5ad25980,
prompter=prompter@entry=0x7f4a576ae9e0, prompter_data=prompter_data@entry=0x7f4a5ad21ed0, start_time=start_time@entry=0,
in_tkt_service=in_tkt_service@entry=0x0, options=options@entry=0x7f4a5ad25ab0, gak_fct=gak_fct@entry=0x7f4a59332f80 <krb5_get_as_key_password>,
gak_data=gak_data@entry=0x7fff80e65e30, use_master=use_master@entry=0x7fff80e65e08, as_reply=as_reply@entry=0x7fff80e65e18)
at ../../../../src/lib/krb5/krb/get_in_tkt.c:1649
#7 0x00007f4a59333632 in krb5_get_init_creds_password (context=0x7f4a5ad252e0, creds=0x7f4a5ad25a30, client=0x7f4a5ad25980, password=<optimized out>,
prompter=0x7f4a576ae9e0, data=0x7f4a5ad21ed0, start_time=0, in_tkt_service=0x0, options=0x7f4a5ad25ab0) at ../../../../src/lib/krb5/krb/gic_pwd.c:291
#8 0x00007f4a576ac3e3 in ?? () from /lib/x86_64-linux-gnu/security/pam_krb5.so
#9 0x00007f4a576acb21 in ?? () from /lib/x86_64-linux-gnu/security/pam_krb5.so
#10 0x00007f4a576aef86 in pam_sm_authenticate () from /lib/x86_64-linux-gnu/security/pam_krb5.so
#11 0x00007f4a5a4471f5 in ?? () from /lib/x86_64-linux-gnu/libpam.so.0
#12 0x00007f4a5a446a78 in pam_authenticate () from /lib/x86_64-linux-gnu/libpam.so.0
#13 0x00007f4a5aaafb0d in ?? ()
#14 0x00007f4a5aaafda1 in ?? ()
#15 0x00007f4a5aaa3a55 in ?? ()
#16 0x00007f4a5aaa67ba in ?? ()
#17 0x00007f4a5aaa75dd in ?? ()
#18 0x00007f4a5aa8f103 in main ()
(gdb) p selstate
No symbol "selstate" in current context.
(gdb) list
78 in ../sysdeps/unix/sysv/linux/poll.c
(gdb) finish
Run till exit from #0 0x00007f4a58e1ea88 in *__GI___poll (fds=fds@entry=0x7f4a5ad38ab8, nfds=2, timeout=<optimized out>, timeout@entry=1778794454)
at ../sysdeps/unix/sysv/linux/poll.c:83
0x00007f4a59358004 in cm_select_or_poll (sret=<synthetic pointer>, out=0x7f4a5ad38ab8, in=0x7f4a5ad36aa0) at ../../../../src/lib/krb5/os/sendto_kdc.c:530
530 ../../../../src/lib/krb5/os/sendto_kdc.c: No such file or directory.
Value returned is $1 = 2
(gdb) finish
Run till exit from #0 0x00007f4a59358004 in cm_select_or_poll (sret=<synthetic pointer>, out=0x7f4a5ad38ab8, in=0x7f4a5ad36aa0)
at ../../../../src/lib/krb5/os/sendto_kdc.c:530
1169 in ../../../../src/lib/krb5/os/sendto_kdc.c
(gdb) p selstate
$2 = (struct select_state *) 0x7f4a5ad36aa0
(gdb) p *selstate
$3 = {fds = {{fd = 4, events = 1, revents = 26222}, {fd = 10, events = 1, revents = 29295}, {fd = 11, events = 5, revents = 30068}, {fd = 1702194477,
events = 25454, revents = 25960}, {fd = -1066876818, events = 71, revents = 1}, {fd = 1359020033, events = 128, revents = -31996}, {fd = -1066918212,
events = 71, revents = 28}, {fd = 1359020033, events = 128, revents = 8208}, {fd = 171443713, events = 0, revents = 542}, {fd = -16796891,
events = 21022, revents = -16142}, {fd = 65679, events = 1, revents = 20737}, {fd = -2096889728, events = 956, revents = -16382}, {fd = 65656,
events = 1, revents = 20737}, {fd = -2096889728, events = 3260, revents = -16284}, {fd = 65697, events = 1, revents = 513}, {fd = -2096889814,
events = 159, revents = 1}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0,
events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1634497125, events = 26478, revents = 28261}, {fd = 528, events = 0, revents = 0}, {
fd = 80, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 4, events = -7348, revents = 0}, {fd = -2132387920, events = 32767,
revents = 0}, {fd = 1496736238, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1, events = 1, revents = 0}, {fd = 0,
events = 0, revents = 0}, {fd = 1, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 74849, events = 0, revents = 0}, {
fd = 1523804080, events = 32586, revents = 0}, {fd = 1143887169, events = 32581, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 48, events = 0, revents = 0}, {fd = 74801, events = 0, revents = 0}, {fd = 0, events = 32586, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 1494028800, events = 32586, revents = 0}, {fd = 74705, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {
fd = 1494034560, events = 32586, revents = 0}, {fd = -1, events = 2, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 1523805376, events = 32586, revents = 0}, {fd = -1, events = -1, revents = -1}, {fd = 0, events = 0, revents = 0}, {
fd = 1523805392, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0},
{fd = -1, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1523804976, events = 32586, revents = 0}, {fd = 1494029568, events = 32586,
revents = 0}, {fd = 0, events = -1, revents = -1}, {fd = 0, events = 0, revents = 0} <repeats 18 times>, {fd = 16193, events = 0, revents = 0}, {
fd = 1494042040, events = 32586, revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1523805520, events = 32586, revents = 0}, {
fd = 1523805520, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 18 times>, {fd = 1494028800, events = 32586, revents = 0}, {
fd = 74225, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0,
events = 0, revents = 0}, {fd = 1494028800, events = 32586, revents = 0}, {fd = 15953, events = 0, revents = 0}, {fd = 1494040248, events = 32586,
revents = 0}, {fd = 1494040248, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 95 times>, {fd = 73393, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 21 times>, {fd = 73217, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0} <repeats 13 times>, {fd = 73105, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 23 times>, {fd = 14689, events = 0,
revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1523807024, events = 32586,
revents = 0}, {fd = 1523807024, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 23 times>, {fd = 14465, events = 0,
revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1523807248, events = 32586,
revents = 0}, {fd = 1523807248, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 716 times>}, nfds = 2, end_time = {
tv_sec = 1351610530, tv_usec = 66304}}
(gdb) p seltemp
$4 = (struct select_state *) 0x7f4a5ad38ab8
(gdb) p * seltemp
$5 = {fds = {{fd = 4, events = 1, revents = 1}, {fd = 10, events = 1, revents = 1}, {fd = 11, events = 5, revents = 30068}, {fd = 1702194477, events = 25454,
revents = 25960}, {fd = -1066876818, events = 71, revents = 1}, {fd = 1359020033, events = 128, revents = -31996}, {fd = -1066918212, events = 71,
revents = 28}, {fd = 1359020033, events = 128, revents = 8208}, {fd = 171443713, events = 0, revents = 542}, {fd = -16796891, events = 21022,
revents = -16142}, {fd = 65679, events = 1, revents = 20737}, {fd = -2096889728, events = 956, revents = -16382}, {fd = 65656, events = 1,
revents = 20737}, {fd = -2096889728, events = 3260, revents = -16284}, {fd = 65697, events = 1, revents = 513}, {fd = -2096889814, events = 159,
revents = 1}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1634497125, events = 26478, revents = 28261}, {fd = 528, events = 0, revents = 0}, {fd = 80,
events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 4, events = -7348, revents = 0}, {fd = -2132387920, events = 32767, revents = 0}, {
fd = 1496736238, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1, events = 1, revents = 0}, {fd = 0, events = 0, revents = 0},
{fd = 1, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 74849, events = 0, revents = 0}, {fd = 1523804080, events = 32586,
revents = 0}, {fd = 1143887169, events = 32581, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 48,
events = 0, revents = 0}, {fd = 74801, events = 0, revents = 0}, {fd = 0, events = 32586, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0,
events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1494028800,
events = 32586, revents = 0}, {fd = 74705, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1494034560, events = 32586, revents = 0},
{fd = -1, events = 2, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1523805376, events = 32586, revents = 0},
{fd = -1, events = -1, revents = -1}, {fd = 0, events = 0, revents = 0}, {fd = 1523805392, events = 32586, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = -1, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0}, {fd = 1523804976, events = 32586, revents = 0}, {fd = 1494029568, events = 32586, revents = 0}, {fd = 0, events = -1, revents = -1}, {
fd = 0, events = 0, revents = 0} <repeats 18 times>, {fd = 16193, events = 0, revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {
fd = 1494042040, events = 32586, revents = 0}, {fd = 1523805520, events = 32586, revents = 0}, {fd = 1523805520, events = 32586, revents = 0}, {fd = 0,
events = 0, revents = 0} <repeats 18 times>, {fd = 1494028800, events = 32586, revents = 0}, {fd = 74225, events = 0, revents = 0}, {fd = 0,
events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0}, {fd = 1494028800,
events = 32586, revents = 0}, {fd = 15953, events = 0, revents = 0}, {fd = 1494040248, events = 32586, revents = 0}, {fd = 1494040248, events = 32586,
revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 95 times>, {fd = 73393, events = 0, revents = 0}, {fd = 0, events = 0,
revents = 0} <repeats 21 times>, {fd = 73217, events = 0, revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 13 times>, {fd = 73105, events = 0,
revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 23 times>, {fd = 14689, events = 0, revents = 0}, {fd = 1494042040, events = 32586,
revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1523807024, events = 32586, revents = 0}, {fd = 1523807024, events = 32586,
revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 23 times>, {fd = 14465, events = 0, revents = 0}, {fd = 1494042040, events = 32586,
revents = 0}, {fd = 1494042040, events = 32586, revents = 0}, {fd = 1523807248, events = 32586, revents = 0}, {fd = 1523807248, events = 32586,
revents = 0}, {fd = 0, events = 0, revents = 0} <repeats 716 times>}, nfds = 2, end_time = {tv_sec = 1351610530, tv_usec = 66304}}
(gdb) p selret
$6 = 2
(gdb) p * conns
$10 = {fd = -1, err = 22, state = FAILED, is_udp = 0, service = 0x7f4a593581c0 <service_tcp_fd>, socktype = 1, family = 2, addrlen = 16, addr = {
ss_family = 2, __ss_align = 0, __ss_padding = '\000' <repeats 111 times>}, x = {out = {sgbuf = {{iov_base = 0x7f4a5ad3bf54, iov_len = 4}, {
iov_base = 0x7f4a5ad36640, iov_len = 200}}, sgp = 0x7f4a5ad3bf28, sg_count = 2, msg_len_buf = "\000\000", <incomplete sequence \310>}, in = {
bufsizebytes_read = 0, bufsize = 0, buf = 0x0, pos = 0x0, bufsizebytes = "\000\000\000", n_left = 0}}, callback_buffer = {magic = 0, length = 0,
data = 0x0}, server_index = 1, next = 0x0}
Thanks for the detail. At first I was perplexed, but then I realized
that line 1290 is passing the wrong list pointer to service_fds()--it's
passing only a subset of the connection list corresponding to sel_state.
I will commit a fix shortly.
From: ghudson@mit.edu
Subject: SVN Commit

Fix spin-loop bug in k5_sendto_kdc

In the second part of the first pass over the server list, we passed
the wrong list pointer to service_fds, causing it to see only a subset
of the server entries corresponding to sel_state. This could cause
service_fds to spin if an event is reported on an fd not in the
subset.

https://github.com/krb5/krb5/commit/2b06a22f7fd8ec01fb27a7335125290b8ceb6f18
Author: Greg Hudson <ghudson@mit.edu>
Commit: 2b06a22f7fd8ec01fb27a7335125290b8ceb6f18
Branch: master
src/lib/krb5/os/sendto_kdc.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)
From: tlyu@mit.edu
Subject: SVN Commit

Fix spin-loop bug in k5_sendto_kdc

In the second part of the first pass over the server list, we passed
the wrong list pointer to service_fds, causing it to see only a subset
of the server entries corresponding to sel_state. This could cause
service_fds to spin if an event is reported on an fd not in the
subset.

(cherry picked from commit 2b06a22f7fd8ec01fb27a7335125290b8ceb6f18)

https://github.com/krb5/krb5/commit/9eb2b4dfc136da326e54081ae18cb4d648c6500d
Author: Greg Hudson <ghudson@mit.edu>
Committer: Tom Yu <tlyu@mit.edu>
Commit: 9eb2b4dfc136da326e54081ae18cb4d648c6500d
Branch: krb5-1.11
src/lib/krb5/os/sendto_kdc.c | 2 +-
1 files changed, 1 insertions(+), 1 deletions(-)