Ticket History #778: Source address problem under Linux when multiple interface addresses are present

From chuckie@ramirez.hilander.com Tue Oct 26 20:50:40 1999
Received: from MIT.EDU (PACIFIC-CARRIER-ANNEX.MIT.EDU [18.69.0.28]) by rt-11.MIT.EDU (8.7.5/8.7.3) with SMTP id UAA11955 for <bugs@RT-11.MIT.EDU>; Tue, 26 Oct 1999 20:50:35 -0400
Received: from [208.203.138.71] by MIT.EDU with SMTP
id AA15615; Tue, 26 Oct 99 20:50:50 EDT
Received: from chuckie by ramirez.hilander.com with local (Exim 3.03 #1)
id 11gHIP-0000RI-00
for krb5-bugs@mit.edu; Wed, 27 Oct 1999 00:50:33 +0000
Message-Id: <E11gHIP-0000RI-00@ramirez.hilander.com>
Date: Wed, 27 Oct 1999 00:50:33 +0000
From: ahp@hilander.com
Sender: "Alec H. Peterson,,,," <chuckie@ramirez.hilander.com>
Reply-To: ahp@hilander.com
To: krb5-bugs@MIT.EDU
Subject: Source address problems with Linux and multiple addresses on an interface
X-Send-Pr-Version: 3.99

Alec H. Peterson - ahp@hilander.com
Staff Scientist
Centergate Research Group - http://www.centergate.com
"Technology so advanced, even we don't understand it."

System: Linux ramirez.hilander.com 2.2.12-20 #1 Mon Sep 27 10:25:54 EDT 1999 i586 unknown
Architecture: i586

The KDC uses a single socket to handle all incoming UDP requests (well,
one per listening port anyway). This is a problem for machines with
multiple interface addresses, especially with Linux 2.2 kernels. In those
kernels it seems to pick an interface address at random to use (perhaps to
help with load sharing or something). So, when the KDC sends reply packets
it may not be using the same source address as the client was using (for
its destination address).


The most reliable way to reproduce it is to put the KDC on a server that
has multiple addresses on an interface and address the server from the
client with one of the 'secondary' addresses on the interface. This
behavior will be most reliably exhibited with a Linux 2.0 kernel, since
it will always source packets from the primary interface address (the
2.2 kernel is more random).

This can be addressed by changing the model to
use listen()/accept() and thus giving the KDC an individual socket for each
client. This may be more cumbersome to manage (more stuff to select() on)
but it would address the problem. I just looked through the sendto()
manpage briefly and there doesn't seem to be an easy way to set the source
address, although I might just be missing something.


From: Ken Raeburn <raeburn@MIT.EDU>
To: ahp@hilander.com
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple addresses on an interface
Date: 26 Oct 1999 21:59:54 -0400

ahp@hilander.com writes:
In those
kernels it seems to pick an interface address at random to use (perhaps to
help with load sharing or something).

Actually, I believe it's for routing -- it's probably picking the
interface indicated by the routing table as the best way to reach the
destination, and using the (primary) address of that interface. One
benefit of this is, if routing is symmetric, that'll give a source
address which the local kernel thinks is the best way for the remote
system to respond. (No, routing isn't always symmetric these days,
and there are other issues the kernel knows nothing about, but if it's
not given any other direction, it's a decent heuristic.)

So, when the KDC sends reply packets
it may not be using the same source address as the client was using (for
its destination address).

Yep, known problem. :-(

It gets worse -- with the coming of IPv6, it's entirely likely that a
single interface will have multiple addresses. And that's ignoring
virtual hosts, dynamically added/removed interfaces, etc.

This can be addressed by changing the model to
use listen()/accept() and thus giving the KDC an individual socket for each
client. This may be more cumbersome to manage (more stuff to select() on)
but it would address the problem. I just looked through the sendto()
manpage briefly and there doesn't seem to be an easy way to set the source
address, although I might just be missing something.

For this approach, the bind() call is probably best, one socket per
local sending address. One of the proposed revised IPv6-compatible
socket APIs with IPv6 support appears to have an option to set the
destination address, but of course we want to fix this for pre-IPv6
systems too.

It should be possible to use recvmsg(MSG_PEEK) or BSD4.4
setsockopt(IP_RECVDSTADDR) to get the destination address of an
incoming packet, so we could still use one receiving socket for UDP
traffic. Perhaps in addition to selecting on a bunch of bound
sockets, so as to handle interfaces added after the program starts.


Anyways, pardon my rambling. It's in our bugs database now, and we'll
look into it. I don't know if it'll get into the coming bugfix
release, but I expect we'll have dealt with it in time for 1.2.
Though if you feel like contributing a clean fix for the problem, it'd
get taken care of sooner....

Ken

From: "Alec H. Peterson" <ahp@hilander.com>
To: Ken Raeburn <raeburn@MIT.EDU>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple
addresses on an interface
Date: Tue, 26 Oct 1999 20:19:31 -0600

This is a multi-part message in MIME format.
--------------6A0B9D2A5BA7C02405E5C17C
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

Ken Raeburn wrote:

Well it could be an advantage if you're the one initiating the connection
:-)


Yikes...


Yup, the bind() approach is what I did. Works fine, although it could get
messy on machines that have lots of virtual servers (then again, should you
really be running virtual servers on your kerberos server? :-)


H'mmm, didn't see that option in the Linux man pages, guess it wouldn't be
too portable....


No problem. I've got a patch that's attached tot his e-mail. It works for
me, although I haven't tested it much yet...

Unrelated question: Has anybody implemented patches to the KDC that takes a
SecurID response for the users's secret instead of a regular password (and
then obviously asks the SecurID server to verify it). I'm planning to write
the mods myself, but if somebody else has done it no need to re-invent the
wheel...

Alec

--
Alec H. Peterson - ahp@hilander.com
Staff Scientist
CenterGate Research Group - http://www.centergate.com
"Technology so advanced, even _we_ don't understand it!"
--------------6A0B9D2A5BA7C02405E5C17C
Content-Type: text/plain; charset=us-ascii;
name="network.c.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="network.c.patch"

*** kdc/network.c Mon May 10 20:35:13 1999
--- kdc/network.c.new Tue Oct 26 19:11:51 1999
***************
*** 28,33 ****
--- 28,35 ----
#include "kdc_util.h"
#include "extern.h"
#include "kdc5_err.h"
+ #include <sys/ioctl.h>
+ #include <net/if.h>

#include <ctype.h>
#ifdef KRB5_USE_INET
***************
*** 45,50 ****
--- 47,53 ----
static int *udp_port_fds = (int *) NULL;
static u_short *udp_port_nums = (u_short *) NULL;
static int n_udp_ports = 0;
+ static int n_sockets = 0;
static int max_udp_ports = 0;
static fd_set select_fds;
static int select_nfds;
***************
*** 89,98 ****
--- 92,119 ----
const char *prog;
{
struct sockaddr_in sin;
+ struct sockaddr_in *ptr;
krb5_error_code retval;
u_short port;
char *cp;
int i;
+ int t;
+ int j;
+
+ struct ifconf info;
+ struct ifreq card[32];
+ int sock;
+
+ memset(&info,0,sizeof(info));
+ memset(&card,0,sizeof(card));
+
+ sock = socket(AF_INET,SOCK_DGRAM,0);
+ info.ifc_len = sizeof(card);
+ info.ifc_ifcu.ifcu_buf = (char *)&card;
+ ioctl(sock,SIOCGIFCONF,&info);
+ close(sock);
+
+ t = (info.ifc_len / sizeof(struct ifreq));

FD_ZERO(&select_fds);
select_nfds = 0;
***************
*** 116,138 ****
}

for (i=0; i<n_udp_ports; i++) {
! if ((udp_port_fds[i] = socket(PF_INET, SOCK_DGRAM, 0)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot create server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! sin.sin_port = htons(udp_port_nums[i]);
! if (bind(udp_port_fds[i], (struct sockaddr *) &sin,
sizeof(sin)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot bind server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! FD_SET(udp_port_fds[i], &select_fds);
! if (udp_port_fds[i]+1 > select_nfds)
! select_nfds = udp_port_fds[i]+1;
}

return 0;
--- 137,168 ----
}

for (i=0; i<n_udp_ports; i++) {
! for (j = 0; j < t; j++)
! {
! if(card[j].ifr_ifru.ifru_addr.sa_family != AF_INET) continue;
! ptr = (struct sockaddr_in *)&card[j].ifr_addr;
! if(ptr->sin_addr.s_addr == 0) continue;
! if ((udp_port_fds[n_sockets] = socket(PF_INET, SOCK_DGRAM, 0)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot create server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! sin.sin_port = htons(udp_port_nums[i]);
!
! sin.sin_addr.s_addr = ptr->sin_addr.s_addr;
! if (bind(udp_port_fds[n_sockets], (struct sockaddr *) &sin,
sizeof(sin)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot bind server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! FD_SET(udp_port_fds[n_sockets], &select_fds);
! if (udp_port_fds[n_sockets]+1 > select_nfds)
! select_nfds = udp_port_fds[n_sockets]+1;
! n_sockets++;
! }
}

return 0;
***************
*** 216,222 ****
com_err(prog, errno, "while selecting for network input");
continue;
}
! for (i=0; i<n_udp_ports; i++) {
if (FD_ISSET(udp_port_fds[i], &readfds)) {
process_packet(udp_port_fds[i], prog, udp_port_nums[i]);
nfound--;
--- 246,252 ----
com_err(prog, errno, "while selecting for network input");
continue;
}
! for (i=0; i<n_sockets; i++) {
if (FD_ISSET(udp_port_fds[i], &readfds)) {
process_packet(udp_port_fds[i], prog, udp_port_nums[i]);
nfound--;

--------------6A0B9D2A5BA7C02405E5C17C--


From: Ken Raeburn <raeburn@MIT.EDU>
To: "Alec H. Peterson" <ahp@hilander.com>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple addresses on an interface
Date: 27 Oct 1999 22:11:41 -0400


No, IP_RECVDSTADDR would be mostly for BSD4.4-derived systems like
NetBSD and FreeBSD. Probably not very useful, since we could get the
same info more portably with recvmsg. If we wanted to do it that way
for some reason.


For the most part, it looks good, though I haven't tested it at all.
I see a couple possible problems, though; would you mind changing it a
little and sending it to me again?

Take a look at lib/krb5/os/localaddr.c. Simply calling
krb5_os_localaddr might be best, if it's practical, though it requires
freeing up memory afterwards. Some of the comments in that file also
describe why an array of ifreq structures may not work for some
systems. (It actually becomes important for IPv6, and I've already
written some IPv6 support for that routine that should be in 1.2. I
don't know how it'll work for Linux.)

It'd help to log the IP address in the failure log messages, at least
for the bind call.


Yes, some support was put in at Cygnus, but the changes haven't been
folded back into the MIT distribution. If you go to
www.cryptography.org, you can get the "KerbNet" distribution.

If you'd like to work on integrating this code into the MIT code base,
I'd like to get it distributed. In that case, I should probably put
you on the "krbdev" developers' mailing list, point you to the current
snapshots rather than the release sources, and get you in touch with
Frank Cusack, who was also interested in doing some work on the
hardware preauth support.

From: "Alec H. Peterson" <ahp@hilander.com>
To: Ken Raeburn <raeburn@MIT.EDU>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple
addresses on an interface
Date: Wed, 27 Oct 1999 21:13:02 -0600

Ken Raeburn wrote:

Sure.


H'mmm, interesting, I'll look at that localaddr call...


Good idea.


Nice, I'll check it out.


Sounds like a plan to me.

I'll make another crack at that patch and send it along.

Alec

--
Alec H. Peterson - ahp@hilander.com
Staff Scientist
CenterGate Research Group - http://www.centergate.com
"Technology so advanced, even _we_ don't understand it!"

From: "Alec H. Peterson" <ahp@hilander.com>
To: Ken Raeburn <raeburn@MIT.EDU>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple
addresses on an interface
Date: Thu, 28 Oct 1999 13:12:30 -0600

This is a multi-part message in MIME format.
--------------B87A291AED9BEB4EBC0CE70A
Content-Type: text/plain; charset=us-ascii
Content-Transfer-Encoding: 7bit

OK, here's another crack at the patch. It still uses an array, but I think
it addresses your concern about sizeof(struct ifreq) (cribbed some code from
c_localaddr.c)

Alec

--
Alec H. Peterson - ahp@hilander.com
Staff Scientist
CenterGate Research Group - http://www.centergate.com
"Technology so advanced, even _we_ don't understand it!"
--------------B87A291AED9BEB4EBC0CE70A
Content-Type: text/plain; charset=us-ascii;
name="network.c.patch"
Content-Transfer-Encoding: 7bit
Content-Disposition: inline;
filename="network.c.patch"

*** kdc/network.c Mon May 10 20:35:13 1999
--- kdc/network.c.new Thu Oct 28 12:07:21 1999
***************
*** 28,33 ****
--- 28,35 ----
#include "kdc_util.h"
#include "extern.h"
#include "kdc5_err.h"
+ #include <sys/ioctl.h>
+ #include <net/if.h>

#include <ctype.h>
#ifdef KRB5_USE_INET
***************
*** 40,50 ****
--- 42,65 ----
#endif
#include <arpa/inet.h>

+ #ifdef HAVE_SA_LEN
+ #ifndef max
+ #define max(a,b) ((a) > (b) ? (a) : (b))
+ #endif
+ #define ifreq_size(i) max(sizeof(struct ifreq),\
+ sizeof((i).ifr_name)+(i).ifr_addr.sa_len)
+ #else
+ #define ifreq_size(i) sizeof(struct ifreq)
+ #endif /* HAVE_SA_LEN*/
+
+ #define NUM_INTERFACES 32
+
extern int errno;

static int *udp_port_fds = (int *) NULL;
static u_short *udp_port_nums = (u_short *) NULL;
static int n_udp_ports = 0;
+ static int n_sockets = 0;
static int max_udp_ports = 0;
static fd_set select_fds;
static int select_nfds;
***************
*** 89,98 ****
--- 104,131 ----
const char *prog;
{
struct sockaddr_in sin;
+ struct sockaddr_in *ptr;
krb5_error_code retval;
u_short port;
char *cp;
int i;
+ int t;
+ int j;
+
+ struct ifconf info;
+ struct ifreq card[NUM_INTERFACES];
+ int sock;
+
+ memset(&info,0,sizeof(info));
+ memset(&card,0,sizeof(card));
+
+ sock = socket(AF_INET,SOCK_DGRAM,0);
+ info.ifc_len = ifreq_size(*card) * NUM_INTERFACES;
+ info.ifc_ifcu.ifcu_buf = (char *)&card;
+ ioctl(sock,SIOCGIFCONF,&info);
+ close(sock);
+
+ t = (info.ifc_len / sizeof(struct ifreq));

FD_ZERO(&select_fds);
select_nfds = 0;
***************
*** 116,138 ****
}

for (i=0; i<n_udp_ports; i++) {
! if ((udp_port_fds[i] = socket(PF_INET, SOCK_DGRAM, 0)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot create server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! sin.sin_port = htons(udp_port_nums[i]);
! if (bind(udp_port_fds[i], (struct sockaddr *) &sin,
sizeof(sin)) == -1) {
retval = errno;
! com_err(prog, 0, "Cannot bind server socket on port %d",
! udp_port_nums[i]);
return(retval);
! }
! FD_SET(udp_port_fds[i], &select_fds);
! if (udp_port_fds[i]+1 > select_nfds)
! select_nfds = udp_port_fds[i]+1;
}

return 0;
--- 149,180 ----
}

for (i=0; i<n_udp_ports; i++) {
! for (j = 0; j < t; j++)
! {
! if(card[j].ifr_ifru.ifru_addr.sa_family != AF_INET) continue;
! ptr = (struct sockaddr_in *)&card[j].ifr_addr;
! if(ptr->sin_addr.s_addr == 0) continue;
! if ((udp_port_fds[n_sockets] = socket(PF_INET, SOCK_DGRAM, 0)) == -1) {
retval = errno;
com_err(prog, 0, "Cannot create server socket on port %d",
udp_port_nums[i]);
return(retval);
! }
! sin.sin_port = htons(udp_port_nums[i]);
!
! sin.sin_addr.s_addr = ptr->sin_addr.s_addr;
! if (bind(udp_port_fds[n_sockets], (struct sockaddr *) &sin,
sizeof(sin)) == -1) {
retval = errno;
! com_err(prog, 0, "Cannot bind server socket on port %d with address %s",
! udp_port_nums[i], inet_ntoa(sin.sin_addr));
return(retval);
! }
! FD_SET(udp_port_fds[n_sockets], &select_fds);
! if (udp_port_fds[n_sockets]+1 > select_nfds)
! select_nfds = udp_port_fds[n_sockets]+1;
! n_sockets++;
! }
}

return 0;
***************
*** 216,222 ****
com_err(prog, errno, "while selecting for network input");
continue;
}
! for (i=0; i<n_udp_ports; i++) {
if (FD_ISSET(udp_port_fds[i], &readfds)) {
process_packet(udp_port_fds[i], prog, udp_port_nums[i]);
nfound--;
--- 258,264 ----
com_err(prog, errno, "while selecting for network input");
continue;
}
! for (i=0; i<n_sockets; i++) {
if (FD_ISSET(udp_port_fds[i], &readfds)) {
process_packet(udp_port_fds[i], prog, udp_port_nums[i]);
nfound--;

--------------B87A291AED9BEB4EBC0CE70A--


From: Ken Raeburn <raeburn@MIT.EDU>
To: "Alec H. Peterson" <ahp@hilander.com>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple addresses on an interface
Date: 28 Oct 1999 17:14:59 -0400


("c_localaddr.c"?)

That doesn't look right to me. The ifreq_size macro uses one of the
fields filled in by the ioctl call to figure out how big the structure
is, and how many bytes further you need to look in your buffer to find
the next ifreq structure. I don't understand how your setting of
ifc_len before the ioctl is going to help.

The code in localaddr.c uses the ifreq_size value to increment a byte
offset into the buffer, which is used to compute the structure
pointer.

for (i = 0; i < n; i+= ifreq_size(*ifr) ) {
...
ifr = (struct ifreq *)((caddr_t) ifc.ifc_buf+i);

The ifc_len value really should be exactly the size of the buffer
space provided, not a computed value that could be larger, or the
syscall could overrun the buffer.

Ken

From: "Alec H. Peterson" <ahp@hilander.com>
To: Ken Raeburn <raeburn@MIT.EDU>
Cc: krb5-bugs@MIT.EDU
Subject: Re: krb5-kdc/778: Source address problems with Linux and multiple
addresses on an interface
Date: Thu, 28 Oct 1999 16:27:01 -0600

Ken Raeburn wrote:

Yeah, localaddr.c is just a stub that calls a function in c_localaddr.c
under the crypto directory.


Ahhhh, OK, I get it now, which is why the array won't work right (because it
pre-allocates)...

Alec

--
Alec H. Peterson - ahp@hilander.com
Staff Scientist
CenterGate Research Group - http://www.centergate.com
"Technology so advanced, even _we_ don't understand it!"