Ticket History #2760: fopen(3) returning NULL and errno==0

-------------------- Start of forwarded message --------------------
To: krbdev@mit.edu
Date: Tue, 26 Oct 2004 18:54:07 -0400
From: "Roland C. Dowdeswell" <elric@imrryr.org>
Message-Id: <20041026225407.6098E174F8@arioch.imrryr.org>
Subject: fopen(3) returning NULL (with workaround.)

So, I've experienced a few core dumps with the kerberos libraries
on Solaris 7. After a bit of examination, I determined that on
Solaris 7 [and I presume earlier versions as well], it is possible
for fopen(3) to return NULL yet set errno to zero[1]. In
krb5_ktfileint_open(), if fopen(3) returns NULL, the function will
return errno. This has the unfortunate effect of seeming like
success to callers who later try to use the NULL FILE *.

Anyway, here's a small patch which just returns ENFILE if errno == 0.

[1] Solaris 7's fopen(3) returns with errno == 0, if it can't
get a fd < 256. In the same situation Solaris 8 sets errno
to ENFILE.

--
Roland Dowdeswell http://www.Imrryr.ORG/~elric/

===================================================================
RCS file: /ms/dev/sec/mitkrb5/cvs-dirs/mitkrb5-1.3.1/mitkrb5/src/lib/krb5/keytab/kt_file.c,v
retrieving revision 1.1
retrieving revision 1.2
diff -u -r1.1 -r1.2
--- src/lib/krb5/keytab/kt_file.c 29 Mar 2004 19:57:38 -0000 1.1
+++ src/lib/krb5/keytab/kt_file.c 23 Sep 2004 04:24:22 -0000 1.2
@@ -1008,10 +1008,10 @@
krb5_create_secure_file(context, KTFILENAME(id));
KTFILEP(id) = fopen(KTFILENAME(id), fopen_mode_rbplus);
if (!KTFILEP(id))
- return errno;
+ return errno?errno:ENFILE;
writevno = 1;
} else /* some other error */
- return errno;
+ return errno?errno:ENFILE;
}
if ((kerror = krb5_lock_file(context, fileno(KTFILEP(id)), mode))) {
(void) fclose(KTFILEP(id));