Ticket History #559: BUG: krb5-1.0.x aclocal.m4

From schwim@whatmore.Stanford.EDU Mon Mar 16 21:37:05 1998
Received: from MIT.EDU (PACIFIC-CARRIER-ANNEX.MIT.EDU [18.69.0.28]) by rt-11.MIT.EDU (8.7.5/8.7.3) with SMTP id VAA12973 for <bugs@RT-11.MIT.EDU>; Mon, 16 Mar 1998 21:37:04 -0500
Received: from whatmore.Stanford.EDU by MIT.EDU with SMTP
id AA07709; Mon, 16 Mar 98 21:37:34 EST
Received: (from schwim@localhost)
by whatmore.Stanford.EDU (8.8.8/8.8.8) id SAA25686;
Mon, 16 Mar 1998 18:37:01 -0800 (PST)
Message-Id: <199803170237.SAA25686@whatmore.Stanford.EDU>
Date: Mon, 16 Mar 1998 18:37:01 -0800 (PST)
From: Larry Schwimmer <schwim@whatmore.Stanford.EDU>
To: krb5-bugs@MIT.EDU
Cc: schwim@leland.Stanford.EDU
Subject: BUG: krb5-1.0.x aclocal.m4


From: "Theodore Y. Ts'o" <tytso@MIT.EDU>
To: krb5-bugs@MIT.EDU, Larry Schwimmer <schwim@whatmore.Stanford.EDU>
Cc: gnats-admin@RT-11.MIT.EDU, krb5-prs@RT-11.MIT.EDU
Subject: Re: pending/559: BUG: krb5-1.0.x aclocal.m4
Date: Mon, 16 Mar 1998 23:00:05 -0500

Date: Mon, 16 Mar 1998 18:37:01 -0800 (PST)
From: Larry Schwimmer <schwim@whatmore.Stanford.EDU>

configure does not know how to check the network libraries
correctly. The relevant code is in aclocal.m4. It currently uses
an incomplete kludge.

We have a more complete kludge in our tree now...

The socket and nsl libraries are optional libraries which
should not be used for socket applications. libc contains all the
necessary calls.

This is not true, unfortunately. At least, not on all versions of Unix.
Some systems, most notably Solaris, require the use of these two
libraries if you want to use any part of the socket API at all.

Here's the current kludge which we're using in our development tree, for
your information, edification, and hopefully not too-much-horror....

- Ted

dnl
dnl The following was written by jhawk@mit.edu
dnl
dnl AC_LIBRARY_NET: Id: net.m4,v 1.4 1997/10/25 20:49:53 jhawk Exp
dnl
dnl This test is for network applications that need socket() and
dnl gethostbyname() -ish functions. Under Solaris, those applications need to
dnl link with "-lsocket -lnsl". Under IRIX, they should *not* link with
dnl "-lsocket" because libsocket.a breaks a number of things (for instance:
dnl gethostbyname() under IRIX 5.2, and snoop sockets under most versions of
dnl IRIX).
dnl
dnl Unfortunately, many application developers are not aware of this, and
dnl mistakenly write tests that cause -lsocket to be used under IRIX. It is
dnl also easy to write tests that cause -lnsl to be used under operating
dnl systems where neither are necessary (or useful), such as SunOS 4.1.4, which
dnl uses -lnsl for TLI.
dnl
dnl This test exists so that every application developer does not test this in
dnl a different, and subtly broken fashion.
dnl
dnl It has been argued that this test should be broken up into two seperate
dnl tests, one for the resolver libraries, and one for the libraries necessary
dnl for using Sockets API. Unfortunately, the two are carefully intertwined and
dnl allowing the autoconf user to use them independantly potentially results in
dnl unfortunate ordering dependancies -- as such, such component macros would
dnl have to carefully use indirection and be aware if the other components were
dnl executed. Since other autoconf macros do not go to this trouble, and almost
dnl no applications use sockets without the resolver, this complexity has not
dnl been implemented.
dnl
dnl The check for libresolv is in case you are attempting to link statically
dnl and happen to have a libresolv.a lying around (and no libnsl.a).
dnl
AC_DEFUN(AC_LIBRARY_NET, [
# Most operating systems have gethostbyname() in the default searched
# libraries (i.e. libc):
AC_CHECK_FUNC(gethostbyname, ,
# Some OSes (eg. Solaris) place it in libnsl:
AC_CHECK_LIB(nsl, gethostbyname, ,
# Some strange OSes (SINIX) have it in libsocket:
AC_CHECK_LIB(socket, gethostbyname, ,
# Unfortunately libsocket sometimes depends on libnsl.
# AC_CHECK_LIB's API is essentially broken so the following
# ugliness is necessary:
AC_CHECK_LIB(socket, gethostbyname,
LIBS="-lsocket -lnsl $LIBS",
AC_CHECK_LIB(resolv, gethostbyname),
-lnsl)
)
)
)
AC_CHECK_FUNC(socket, , AC_CHECK_LIB(socket, socket, ,
AC_CHECK_LIB(socket, socket, LIBS="-lsocket -lnsl $LIBS", , -lnsl)))
])

From: Larry Schwimmer <schwim@whatmore.Stanford.EDU>
To: Cc: krb5-bugs@MIT.EDU, gnats-admin@RT-11.MIT.EDU, krb5-prs@RT-11.MIT.EDU
Subject: Re: pending/559: BUG: krb5-1.0.x aclocal.m4
Date: Mon, 16 Mar 1998 20:11:28 -0800 (PST)

You (Theodore Y. Ts'o) write:

Good to hear. (-: (I know -- this is an annoying case to
handle. Didn't mean to be an annoyance. I called it a kludge
because it hard-coded the OS.)


I was referring to IRIX in that statement. We support
Solaris, HP-UX, AIX, Linux, Digital UNIX, and IRIX here. So I'm
familiar with Solaris and the joyous necessity of -lsocket -lnsl.
It's the primary reason why programs get built wrong on IRIX.

When we upgraded our kerberos clients last Winter break, we
ended up breaking logins until we figured out the problem. (Our test
machine worked just fine since the people testing it were in the local
password file.) I'm in the process of cleaning up our local patch set
and building the 1.0.5 distribution and wanted to be sure the problem
is fixed. It was ... unpleasant.

yours,
Larry

State-Changed-From-To: open-closed
State-Changed-By: tytso
State-Changed-When: Tue Mar 17 16:58:40 1998
State-Changed-Why: Problem already addressed in development tree

Submitter-Id: net
Originator: Larry Schwimmer
Confidential: no
Synopsis: configure should not check for nsl on IRIX
Severity: serious
Priority: medium
Category: krb5-build
Class: sw-bug
Release: 1.0.5 and earlier
Environment:

IRIX

Description:

configure does not know how to check the network libraries
correctly. The relevant code is in aclocal.m4. It currently uses
an incomplete kludge.

The socket and nsl libraries are optional libraries which
should not be used for socket applications. libc contains all the
necessary calls.

aclocal.m4 contains a kludge which causes it not to check
for libsocket on IRIX. But nsl is still checked. Linking with libnsl
on IRIX breaks calls to NIS, which means that programs like login,
klogind, and telnetd break.

How-To-Repeat:

Configure krb5-1.0.x on an IRIX system which has libnsl
installed.

Fix:

I'd send a patch to aclocal.m4, but it does not work with the
standard autoconf. aclocal.m4 refers to AC_LOCALDIR, which is not
defined in the normal autoconf. The autoconf in util/autoconf refers
to it, but it would be nice if aclocal.m4 was fixed so that it worked
with the stock autoconf.

Anyhow, this is the problem code in the WITH_NETLIB macro of
aclocal.m4:

[if test "`(uname) 2>/dev/null`" != IRIX ; then
AC_CHECK_LIB(socket,main)
fi
AC_CHECK_LIB(nsl,main)]

One could move the nsl check into the if section, but an
approach which does not hard-code OS's is preferable. This is the
code I normally use to handle socket and nsl libraries:

AC_CHECK_FUNC(gethostbyname)
if test $ac_cv_func_gethostbyname = no; then
AC_CHECK_LIB(nsl, gethostbyname)
fi
AC_CHECK_FUNC(connect)
if test $ac_cv_func_connect = no; then
AC_CHECK_LIB(socket, connect)
fi

yours,
Larry Schwimmer
schwim@leland.stanford.edu
Leland Systems Group