Skip Menu |
 

From: "Basch, Richard" <Richard.Basch@gs.com>
To: "'krb5-bugs@mit.edu'" <krb5-bugs@mit.edu>
Subject: krb5- 1.15 KDC network performance issue
Date: Sat, 24 Dec 2016 05:57:00 +0000
CC: "gs-kerberos-eng@internal.email.gs.com" <gs-kerberos-eng@internal.email.gs.com>, "Harrison, Kayla C." <kayla.harrison@gs.com>
Download (untitled) / with headers
text/plain 1.8KiB

There appears to be a change in behavior in krb5-1.15 vs krb5-1.14 which is showing a significant network performance issue.

 

Scenario:

-          RHEL 6 host

-          Additional V4 IP addresses defined (VIPs tied to virtual interfaces, e.g. eth0:1 or to lo:1)

-          No IPv6 addresses setup (not that I think this matters)

-          kdc_listen = 0.0.0.0:88

 

Kinit to the machine’s real IP performs sub-second (<1ms)

Kinit to the machine’s VIP takes about 1.0s

 

strace –tt suggests the delay is in sendmsg(); recvmsg() appears to be behaving quickly.

 

Here is the interesting thing... if I setup “kdc_listen = VIP:88 0.0.0.0:88”, it performs well. The only reason I don’t want to define VIP:88 in kdc.conf is that VIP may be moved from machine to machine (failover).

 

Likewise, with krb5-1.14, with kdc_ports = 88, there were no issues.

 

Any suggestions/assistance?

 

 

______________________________________________________________________________

Richard Basch

VP, Technology - Critical Infrastructure

Goldman, Sachs & Co

30 Hudson St.  7th Floor | Jersey City, NJ 07302
richard.basch@gs.com  | +1 (917) 343-4071

 

P Save a tree: Please don't print this mail unless necessary.

 

The Goldman Sachs Group, Inc. All rights reserved.

See http://www.gs.com/disclaimer/global_email for important risk disclosures, conflicts of interest and other terms and conditions relating to this e-mail and your reliance on information contained in it.  This message may contain confidential or privileged information.  If you are not the intended recipient, please advise us immediately and delete this message.  See http://www.gs.com/disclaimer/email for further information on confidentiality and the risks of non-secure electronic communication.  If you cannot access these links, please notify us by reply message and we will send the contents to you.

 

Some questions:

* What was the configuration in 1.14?

* If you just write "kdc_listen = 88", or use whatever KDC network
configuration you had in 1.14, does everything work correctly?
From: "Basch, Richard" <Richard.Basch@gs.com>
To: "rt-comment@krbdev.mit.edu" <rt-comment@krbdev.mit.edu>
Subject: RE: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue
Date: Sat, 24 Dec 2016 17:07:16 +0000
RT-Send-Cc:
The only difference in kdc.conf is converting *_port(s) = port to *_listen = 0.0.0.0:port.

It falls to start if I try the 1.14 kdc.conf because it tries to bind to the wildcard IPv6 address and says address protocol family not supported in the log. I had to be explicit with kdc_listen to bind to 0.0.0.0:88. Actually I normally would have multiple ports defined in both 1.14 and 1.15, but I already tried with a single port with 1.15 (with no change in behavior).


Sent from my mobile phone.

Richard Basch
VP, Technology
Goldman Sachs and Co.

From: Greg Hudson via RT <rt-comment@krbdev.mit.edu>
Sent: Saturday, December 24, 2016 11:10:14 AM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue
 
Some questions:

* What was the configuration in 1.14?

* If you just write "kdc_listen = 88", or use whatever KDC network
configuration you had in 1.14, does everything work correctly?
Okay, that sounds like two bugs we need to fix:

1. If a wildcard address is explicitly specified, we don't set pktinfo
on the socket. The one-second latency you observed likely occurs
because the KDC sends the reply on the wrong source address and the
client doesn't receive it.

2. If no address is specified and the machine doesn't have any IPv6
addresses (not even a loopback address), the KDC fails to start after
failing to bind to the IPv6 wildcard address. The symmetric problem
might exist for a machine with only IPv6 addresses.
From: Richard Basch <basch@alum.mit.edu>
Subject: Re: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue
Date: Sat, 24 Dec 2016 23:52:01 -0500
CC: Richard Basch <basch@mit.edu>, gs-kerberos-eng@ny.email.gs.com
To: Richard Basch <Richard.Basch@gs.com>, Greg Hudson via RT <rt-comment@krbdev.mit.edu>
RT-Send-Cc:
Download (untitled) / with headers
text/plain 2.6KiB
Based on a preliminary analysis of net-server.c, it seems that perhaps pktinfo should always be enabled on UDP sockets. It looks like it never causes a failure (at most a warning) and can only help. For instance, if you bind to 127.0.0.1 but the interface is defined as /8, it should still be able to respond even if a packet comes in via 127.1.2.3.

The following patch appears to fix the performance issue.

diff --git a/src/lib/apputils/net-server.c b/src/lib/apputils/net-server.c
index 171ecc4..4d5ed61 100644
--- a/src/lib/apputils/net-server.c
+++ b/src/lib/apputils/net-server.c
@@ -752,8 +752,8 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
        goto cleanup;
    }

-    /* Try to turn on pktinfo for UDP wildcard sockets. */
-    if (ba->type == UDP && ba->address == NULL) {
+    /* Try to turn on pktinfo for UDP sockets. */
+    if (ba->type == UDP) {
        krb5_klog_syslog(LOG_DEBUG, _("Setting pktinfo on socket %s"),
                         paddr(sock_address));
        ret = set_pktinfo(sock, sock_address->sa_family);
@@ -762,9 +762,10 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
                    _("Cannot request packet info for UDP socket address "
                      "%s port %d"), paddr(sock_address), ba->port);
            krb5_klog_syslog(LOG_INFO, _("System does not support pktinfo yet "
-                                         "binding to a wildcard address.  "
+                                         "binding to %s.  "
                                         "Packets are not guaranteed to "
-                                         "return on the received address."));
+                                         "return on the received address."),
+                             paddr(sock_address));
        }
    }

Show quoted text
On Dec 24, 2016, at 10:57 PM, Basch, Richard <Richard.Basch@gs.com> wrote:



-----Original Message-----
From: Greg Hudson via RT [mailto:rt-comment@krbdev.mit.edu]
Sent: Saturday, December 24, 2016 1:01 PM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue

Okay, that sounds like two bugs we need to fix:

1. If a wildcard address is explicitly specified, we don't set pktinfo on the socket.  The one-second latency you observed likely occurs because the KDC sends the reply on the wrong source address and the client doesn't receive it.

2. If no address is specified and the machine doesn't have any IPv6 addresses (not even a loopback address), the KDC fails to start after failing to bind to the IPv6 wildcard address.  The symmetric problem might exist for a machine with only IPv6 addresses.

Show quoted text
> Based on a preliminary analysis of net-server.c, it seems that perhaps
> pktinfo should always be enabled on UDP sockets. It looks like it
> never causes a failure (at most a warning) and can only help. For
> instance, if you bind to 127.0.0.1 but the interface is defined as
> /8, it should still be able to respond even if a packet comes in
> via 127.1.2.3.

Does that work? I wasn't aware that you could receive packets at
multiple addresses except by binding to the wildcard address.

Regardless, setting pktinfo on all UDP sockets is probably the simplest
solution to that half of the problem. I'm not sure if we want to log
the pktinfo-not-supported message for all addresses, though.
From: Richard Basch <basch@alum.mit.edu>
Subject: Re: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue
Date: Sun, 25 Dec 2016 22:08:05 -0500
CC: Richard Basch <basch@mit.edu>, gs-kerberos-eng@ny.email.gs.com
To: Richard Basch <Richard.Basch@gs.com>, Greg Hudson via RT <rt-comment@krbdev.mit.edu>
RT-Send-Cc:
Download (untitled) / with headers
text/plain 6.2KiB
Download (untitled) / with headers
text/html 15.4KiB
The following patch may address both issues� I tested the earlier patch, but not this one (other than to make sure it compiles cleanly). Hopefully, this will allow the more generic wildcard addressing even when one of the protocol families isn�t bound on any interface. Only the section �@@ -885,11 +886,26 @@ setup_addresses� is new in this patch.


diff --git a/src/lib/apputils/net-server.c b/src/lib/apputils/net-server.c
index 171ecc4..13366f5 100644
--- a/src/lib/apputils/net-server.c
+++ b/src/lib/apputils/net-server.c
@@ -752,8 +752,8 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
        goto cleanup;
    }

-    /* Try to turn on pktinfo for UDP wildcard sockets. */
-    if (ba->type == UDP && ba->address == NULL) {
+    /* Try to turn on pktinfo for UDP sockets. */
+    if (ba->type == UDP) {
        krb5_klog_syslog(LOG_DEBUG, _("Setting pktinfo on socket %s"),
                         paddr(sock_address));
        ret = set_pktinfo(sock, sock_address->sa_family);
@@ -762,9 +762,10 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
                    _("Cannot request packet info for UDP socket address "
                      "%s port %d"), paddr(sock_address), ba->port);
            krb5_klog_syslog(LOG_INFO, _("System does not support pktinfo yet "
-                                         "binding to a wildcard address.  "
+                                         "binding to %s.  "
                                         "Packets are not guaranteed to "
-                                         "return on the received address."));
+                                         "return on the received address."),
+                             paddr(sock_address));
        }
    }

@@ -885,11 +886,26 @@ setup_addresses(struct socksetup *data)
                               verto_callbacks[addr.type],
                               bind_conn_types[addr.type]);
            if (ret) {
-                krb5_klog_syslog(LOG_ERR,
-                                 _("Failed setting up a %s socket (for %s)"),
-                                 bind_type_names[addr.type],
-                                 paddr(ai->ai_addr));
-                goto cleanup;
+                /*
+                 * If a wildcard address resolves to multiple protocol
+                 * families, we will only warn about the bind failure.
+                 * For other conditions, error.
+                 */
+                if ( (ai_list->ai_next == NULL) ||
+                     (ai->ai_family == AF_INET && ((struct sockaddr_in *)ai->ai_addr)->sin_addr.s_addr != INADDR_ANY) ||
+                     (ai->ai_family == AF_INET6 && !IN6_ARE_ADDR_EQUAL(&((struct sockaddr_in6 *)ai->ai_addr)->sin6_addr, &in6addr_any)) ) {
+                    krb5_klog_syslog(LOG_ERR,
+                                     _("Failed setting up a %s socket (for %s)"),
+                                     bind_type_names[addr.type],
+                                     paddr(ai->ai_addr));
+                    goto cleanup;
+                } else {
+                    krb5_klog_syslog(LOG_WARNING,
+                                     _("Failed setting up a %s socket (for %s)"),
+                                     bind_type_names[addr.type],
+                                     paddr(ai->ai_addr));
+                    continue;
+                }
            }
        }

Show quoted text
On Dec 24, 2016, at 11:52 PM, Richard Basch <basch@alum.mit.edu> wrote:

Based on a preliminary analysis of net-server.c, it seems that perhaps pktinfo should always be enabled on UDP sockets. It looks like it never causes a failure (at most a warning) and can only help. For instance, if you bind to 127.0.0.1 but the interface is defined as /8, it should still be able to respond even if a packet comes in via 127.1.2.3.

The following patch appears to fix the performance issue.

diff --git a/src/lib/apputils/net-server.c b/src/lib/apputils/net-server.c
index 171ecc4..4d5ed61 100644
--- a/src/lib/apputils/net-server.c
+++ b/src/lib/apputils/net-server.c
@@ -752,8 +752,8 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
        goto cleanup;
    }

-    /* Try to turn on pktinfo for UDP wildcard sockets. */
-    if (ba->type == UDP && ba->address == NULL) {
+    /* Try to turn on pktinfo for UDP sockets. */
+    if (ba->type == UDP) {
        krb5_klog_syslog(LOG_DEBUG, _("Setting pktinfo on socket %s"),
                         paddr(sock_address));
        ret = set_pktinfo(sock, sock_address->sa_family);
@@ -762,9 +762,10 @@ setup_socket(struct socksetup *data, struct bind_address *ba,
                    _("Cannot request packet info for UDP socket address "
                      "%s port %d"), paddr(sock_address), ba->port);
            krb5_klog_syslog(LOG_INFO, _("System does not support pktinfo yet "
-                                         "binding to a wildcard address.  "
+                                         "binding to %s.  "
                                         "Packets are not guaranteed to "
-                                         "return on the received address."));
+                                         "return on the received address."),
+                             paddr(sock_address));
        }
    }

On Dec 24, 2016, at 10:57 PM, Basch, Richard <Richard.Basch@gs.com> wrote:



-----Original Message-----
From: Greg Hudson via RT [mailto:rt-comment@krbdev.mit.edu]
Sent: Saturday, December 24, 2016 1:01 PM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue

Okay, that sounds like two bugs we need to fix:

1. If a wildcard address is explicitly specified, we don't set pktinfo on the socket.  The one-second latency you observed likely occurs because the KDC sends the reply on the wrong source address and the client doesn't receive it.

2. If no address is specified and the machine doesn't have any IPv6 addresses (not even a loopback address), the KDC fails to start after failing to bind to the IPv6 wildcard address.  The symmetric problem might exist for a machine with only IPv6 addresses.


From: Richard Basch <basch@alum.mit.edu>
Subject: Re: [krbdev.mit.edu #8530] krb5- 1.15 KDC network performance issue
Date: Sun, 25 Dec 2016 22:59:01 -0500
To: Greg Hudson via RT <rt-comment@krbdev.mit.edu>
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.1KiB
Forgot to answer your question…

I did a performance test with a kinit against a 127.x.x.x that wasn’t 127.0.0.1 and wasn’t explicitly defined on any interface and it worked. Unfortunately, I didn’t perform a baseline test without the patch.


Show quoted text
> On Dec 25, 2016, at 9:26 PM, Greg Hudson via RT <rt-comment@krbdev.mit.edu> wrote:
>
>> Based on a preliminary analysis of net-server.c, it seems that perhaps
>> pktinfo should always be enabled on UDP sockets. It looks like it
>> never causes a failure (at most a warning) and can only help. For
>> instance, if you bind to 127.0.0.1 but the interface is defined as
>> /8, it should still be able to respond even if a packet comes in
>> via 127.1.2.3.
>
> Does that work? I wasn't aware that you could receive packets at
> multiple addresses except by binding to the wildcard address.
>
> Regardless, setting pktinfo on all UDP sockets is probably the simplest
> solution to that half of the problem. I'm not sure if we want to log
> the pktinfo-not-supported message for all addresses, though.
> _______________________________________________
> krb5-bugs mailing list
> krb5-bugs@mit.edu
> https://mailman.mit.edu/mailman/listinfo/krb5-bugs
Our pktinfo support explicitly doesn't operate on non-wildcard sockets
(see udppktinfo.c:is_bound_to_wildcard()), and now that I think about
it, there may have been some issues trying to use pktinfo on non-
wildcard sockets on FreeBSD. So I don't think that test was conclusive.

Also, do you know if the kernel on these machines was built without IPv6
support? I think an EAFNOSUPPORT error would actually come from
socket(), not bind().
From: "Basch, Richard" <Richard.Basch@gs.com>
To: "'rt-comment@krbdev.mit.edu'" <rt-comment@krbdev.mit.edu>
Subject: RE: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo
Date: Mon, 26 Dec 2016 20:04:28 +0000
RT-Send-Cc:
/etc/sysconfig/network is configured with "NETWORKING_IPV6=no", but I was able to "modprobe ipv6 ; lsmod | grep ipv6". And I 100% agree it is coming from socket()... I said bind not able to come up with the words "opening a socket" at that late hour, but did not mean bind(2); I agree the error occurs during socket(2).


Show quoted text
-----Original Message-----
From: Greg Hudson via RT [mailto:rt-comment@krbdev.mit.edu]
Sent: Monday, December 26, 2016 2:28 PM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo

Our pktinfo support explicitly doesn't operate on non-wildcard sockets (see udppktinfo.c:is_bound_to_wildcard()), and now that I think about it, there may have been some issues trying to use pktinfo on non- wildcard sockets on FreeBSD. So I don't think that test was conclusive.

Also, do you know if the kernel on these machines was built without IPv6 support? I think an EAFNOSUPPORT error would actually come from socket(), not bind().
I opened https://github.com/krb5/krb5/pull/587 with fixes the two issues
discussed in this ticket.
Show quoted text
> I opened https://github.com/krb5/krb5/pull/587 with fixes the two
> issues
> discussed in this ticket.

Richard, do you think you could spare the time to test those patches
against your system configuration?
From: "Basch, Richard" <Richard.Basch@gs.com>
To: "'rt-comment@krbdev.mit.edu'" <rt-comment@krbdev.mit.edu>
Subject: RE: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo
Date: Sun, 8 Jan 2017 17:05:52 +0000
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.4KiB
I will be testing this weekend. Visually, the patch looks correct, though I wonder if pktinfo should be restricted to wildcard-only sockets.

If you bind to 127.0.0.1 but define the interface as 127.0.0.1/8, will it return on the correct address if you direct the query to 127.1.1.1. Certainly, this works with "ping", etc. so the traffic is definitely seen. It's an edge-case which most likely has impacts on systems responding to traffic directed to PtP links and Loopback addresses which do not use /32 masks and I think it is a condition which might not work without pktinfo being attempted. Personally, I suggest just attempting pktinfo for all UDP and not worry if it can't get the info since it will merely use the bound address in that case.


Show quoted text
-----Original Message-----
From: Greg Hudson via RT [mailto:rt-comment@krbdev.mit.edu]
Sent: Friday, January 06, 2017 11:49 AM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo


Richard, do you think you could spare the time to test those patches against your system configuration?
From: "Basch, Richard" <Richard.Basch@gs.com>
To: "'rt-comment@krbdev.mit.edu'" <rt-comment@krbdev.mit.edu>
Subject: RE: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo
Date: Mon, 9 Jan 2017 02:45:23 +0000
RT-Send-Cc:
Download (untitled) / with headers
text/plain 1.8KiB
The GitHub pull works.

As I mentioned earlier, I think pktinfo might be appropriate for all UDP (i.e. I think there are other edge conditions, such as multicast, PtP and loopback addresses which won't be handled properly).

Show quoted text
-----Original Message-----
From: Basch, Richard [Tech]
Sent: Sunday, January 08, 2017 12:06 PM
To: 'rt-comment@krbdev.mit.edu'
Subject: RE: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo

I will be testing this weekend. Visually, the patch looks correct, though I wonder if pktinfo should be restricted to wildcard-only sockets.

If you bind to 127.0.0.1 but define the interface as 127.0.0.1/8, will it return on the correct address if you direct the query to 127.1.1.1. Certainly, this works with "ping", etc. so the traffic is definitely seen. It's an edge-case which most likely has impacts on systems responding to traffic directed to PtP links and Loopback addresses which do not use /32 masks and I think it is a condition which might not work without pktinfo being attempted. Personally, I suggest just attempting pktinfo for all UDP and not worry if it can't get the info since it will merely use the bound address in that case.


-----Original Message-----
From: Greg Hudson via RT [mailto:rt-comment@krbdev.mit.edu]
Sent: Friday, January 06, 2017 11:49 AM
To: Basch, Richard [Tech]
Subject: [krbdev.mit.edu #8530] KDC/kadmind explicit wildcard listener addresses do not use pktinfo


Richard, do you think you could spare the time to test those patches against your system configuration?
From: ghudson@mit.edu
Subject: git commit

Use pktinfo for explicit UDP wildcard listeners

In net-server.c, use pktinfo on UDP server sockets if they are bound
to wildcard addresses, whether that is explicit or implicit in the
address specification.

https://github.com/krb5/krb5/commit/d005beaa72c70bc28b2b0b49b9d83eff160ca8f1
Author: Greg Hudson <ghudson@mit.edu>
Commit: d005beaa72c70bc28b2b0b49b9d83eff160ca8f1
Branch: master
src/lib/apputils/net-server.c | 13 ++++++++++++-
1 files changed, 12 insertions(+), 1 deletions(-)
From: tlyu@mit.edu
Subject: git commit

Use pktinfo for explicit UDP wildcard listeners

In net-server.c, use pktinfo on UDP server sockets if they are bound
to wildcard addresses, whether that is explicit or implicit in the
address specification.

(cherry picked from commit d005beaa72c70bc28b2b0b49b9d83eff160ca8f1)

https://github.com/krb5/krb5/commit/e23d062471bf9071072aaf2df39054508fe74cc1
Author: Greg Hudson <ghudson@mit.edu>
Committer: Tom Yu <tlyu@mit.edu>
Commit: e23d062471bf9071072aaf2df39054508fe74cc1
Branch: krb5-1.15
src/lib/apputils/net-server.c | 13 ++++++++++++-
1 files changed, 12 insertions(+), 1 deletions(-)