Skip Menu |
 

To: krb5-bugs@mit.edu
Subject: socket leak in sendto_kdc.c, start_connection()
From: Bill Dodd <bdodd@austin.ibm.com>
Date: 24 Feb 2004 17:56:29 -0600

In start_connection(), if the connect() fails (e.g. with ECONNREFUSED),
an error is returned, but the socket is not closed.

To observe the leak, set udp_preference_limit to 1 in krb5.conf and
run kdc5_hammer with a large repeat count against a kdc that only
listens on UDP. Observe the open files/sockets with lsof. A contrived
scenario to be sure, but it can be seen in more legitimate cases as
well.

Observed on 1.3.2-beta5, but it exists in all 1.3.X releases.

Patch follows:

*** sendto_kdc.c.orig Fri Dec 5 19:30:42 2003
--- sendto_kdc.c Tue Feb 24 14:37:47 2004
***************
*** 563,566 ****
--- 563,568 ----
} else {
dprint("connect failed: %m\n", SOCKET_ERRNO);
+ dprint("closing fd %d\n", fd);
+ (void) closesocket(fd);
state->err = SOCKET_ERRNO;
state->state = FAILED;
Date: Tue, 24 Feb 2004 21:27:48 -0500
Subject: Re: [krbdev.mit.edu #2277] socket leak in sendto_kdc.c, start_connection()
From: Ken Raeburn <raeburn@MIT.EDU>
To: rt-comment@krbdev.mit.edu
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.1KiB
On Tuesday, Feb 24, 2004, at 18:56 US/Eastern, Bill Dodd via RT wrote:

Show quoted text
> In start_connection(), if the connect() fails (e.g. with ECONNREFUSED),
> an error is returned, but the socket is not closed.

Under what circumstances (aside from, maybe, running the client and KDC
on the same host) can you get back ECONNREFUSED from a connect call on
a non-blocking socket? Seems to me that to get and process the ICMP
message from another host, the process would have to, well, block. I
suppose, if the scheduler switched to another process during the
connect call, it's possible a response from another nearby host could
come in before the process continued, but it seems like a tricky race
condition more than a reliable leak.

It's definitely a bug, I'm just trying to get a sense of how serious...

Show quoted text
> To observe the leak, set udp_preference_limit to 1 in krb5.conf and
> run kdc5_hammer with a large repeat count against a kdc that only
> listens on UDP. Observe the open files/sockets with lsof. A contrived
> scenario to be sure, but it can be seen in more legitimate cases as
> well.

Were you testing this with a KDC on the same host, or another host?

Ken
Date: Tue, 24 Feb 2004 23:35:30 -0500
Subject: [krbdev.mit.edu #2277] socket leak in sendto_kdc.c, start_connection()
From: Ken Raeburn <raeburn@MIT.EDU>
To: rt@krbdev.mit.edu
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.4KiB
[resend; my original message didn't get sent to the submitter]

Begin forwarded message:

Show quoted text
> From: Ken Raeburn <raeburn@mit.edu>
> Date: Tue Feb 24, 2004 21:27:48 US/Eastern
> To: rt-comment@krbdev.mit.edu
> Subject: Re: [krbdev.mit.edu #2277] socket leak in sendto_kdc.c,
> start_connection()
>
> On Tuesday, Feb 24, 2004, at 18:56 US/Eastern, Bill Dodd via RT wrote:
>
>> In start_connection(), if the connect() fails (e.g. with
>> ECONNREFUSED),
>> an error is returned, but the socket is not closed.
>
> Under what circumstances (aside from, maybe, running the client and
> KDC on the same host) can you get back ECONNREFUSED from a connect
> call on a non-blocking socket? Seems to me that to get and process
> the ICMP message from another host, the process would have to, well,
> block. I suppose, if the scheduler switched to another process during
> the connect call, it's possible a response from another nearby host
> could come in before the process continued, but it seems like a tricky
> race condition more than a reliable leak.
>
> It's definitely a bug, I'm just trying to get a sense of how serious...
>
>> To observe the leak, set udp_preference_limit to 1 in krb5.conf and
>> run kdc5_hammer with a large repeat count against a kdc that only
>> listens on UDP. Observe the open files/sockets with lsof. A contrived
>> scenario to be sure, but it can be seen in more legitimate cases as
>> well.
>
> Were you testing this with a KDC on the same host, or another host?
>
> Ken
>
To: rt-comment@krbdev.mit.edu
Cc: raeburn@mit.edu, krb5-prs@mit.edu
Subject: Re: [krbdev.mit.edu #2277] socket leak in sendto_kdc.c, start_connection()
From: Bill Dodd <bdodd@austin.ibm.com>
Date: 25 Feb 2004 12:13:09 -0600
RT-Send-Cc:
Download (untitled) / with headers
text/plain 2.1KiB

Show quoted text
>>>>> "Ken" == Ken Raeburn via <RT" <rt-comment@krbdev.mit.edu>> writes:

Show quoted text
Ken> [resend; my original message didn't get sent to the submitter]
Ken> Begin forwarded message:

Show quoted text
>> From: Ken Raeburn <raeburn@mit.edu>
>> Date: Tue Feb 24, 2004 21:27:48 US/Eastern
>> To: rt-comment@krbdev.mit.edu
>> Subject: Re: [krbdev.mit.edu #2277] socket leak in sendto_kdc.c,
>> start_connection()
>>
>> On Tuesday, Feb 24, 2004, at 18:56 US/Eastern, Bill Dodd via RT wrote:
>>
>>> In start_connection(), if the connect() fails (e.g. with
>>> ECONNREFUSED),
>>> an error is returned, but the socket is not closed.
>>
>> Under what circumstances (aside from, maybe, running the client and
>> KDC on the same host) can you get back ECONNREFUSED from a connect
>> call on a non-blocking socket?

Ken,

Yes, the client and KDC were on the same host when I saw the
ECONNREFUSED. Sorry, I should have said that in my original note. I
was thinking there could be other conditions/errors where you could go
down that error path, but as you say, that might be a pretty rare
condition.


Show quoted text
>> Seems to me that to get and process
>> the ICMP message from another host, the process would have to, well,
>> block. I suppose, if the scheduler switched to another process during
>> the connect call, it's possible a response from another nearby host
>> could come in before the process continued, but it seems like a tricky
>> race condition more than a reliable leak.
>>
>> It's definitely a bug, I'm just trying to get a sense of how serious...
>>
>>> To observe the leak, set udp_preference_limit to 1 in krb5.conf and
>>> run kdc5_hammer with a large repeat count against a kdc that only
>>> listens on UDP. Observe the open files/sockets with lsof. A contrived
>>> scenario to be sure, but it can be seen in more legitimate cases as
>>> well.
>>
>> Were you testing this with a KDC on the same host, or another host?
>>
>> Ken
>>

I just re-ran the same scenario with the client on a different machine
than the KDC and hit a different, more severe problem. The client is
dying with a "broken pipe" error in writev() in service_tcp_fd().
This is on AIX. I'll have to debug this some more.

Thanks for following up,
-bill
From: raeburn@mit.edu
Subject: CVS Commit
* sendto_kdc.c (start_connection): Close socket if connect() call fails for an
unexpected reason.


To generate a diff of this commit:



cvs diff -r5.374 -r5.375 krb5/src/lib/krb5/os/ChangeLog
cvs diff -r5.65 -r5.66 krb5/src/lib/krb5/os/sendto_kdc.c
From: tlyu@mit.edu
Subject: CVS Commit
pullup from trunk


To generate a diff of this commit:



cvs diff -r5.343.2.13 -r5.343.2.14 krb5/src/lib/krb5/os/ChangeLog
cvs diff -r5.62.2.1 -r5.62.2.2 krb5/src/lib/krb5/os/sendto_kdc.c