Ticket History #6565: HP-UX IA64 wrong endian

# Mon Sep 14 21:45:49 2009 "Arlene Berry" <aberry@likewise.com> - Ticket created

Subject:	HP-UX IA64 wrong endian
Date:	Mon, 14 Sep 2009 19:17:15 -0400
From:	"Arlene Berry" <aberry@likewise.com>
To:	<krb5-bugs@mit.edu>

Download (untitled) / with headers
text/plain 1.4KiB

Download (untitled) / with headers
text/html 6.6KiB

For krb5-1.7 all IA64 platforms are identified as little endian but HP-UX is big endian. It’s causing the 1.7 arcfour string to key function to use the wrong endian and it’s resulting in KRB5KDC_ERR_PREAUTH_FAILED errors. The endian is also incorrect for 1.6.3 but its arcfour string to key function works anyway. Here’s what I did to fix it for us:

Index: krb5/src/include/k5-platform.h

===================================================================

--- krb5/src/include/k5-platform.h (revision 37170)

+++ krb5/src/include/k5-platform.h (revision 37171)

@@ -476,10 +476,10 @@

As far as I know, only PDP11 and ARM (which we don't handle here)

have strange byte orders where an 8-byte value isn't laid out as

either 12345678 or 87654321. */

-# if defined(__i386__) || defined(_MIPSEL) || defined(__alpha__) || defined(__ia64__)

+# if defined(__i386__) || defined(_MIPSEL) || defined(__alpha__) || (defined(__ia64__) && !defined(__hpux))

# define K5_LE

# endif

-# if defined(__hppa__) || defined(__rs6000__) || defined(__sparc__) || defined(_MIPSEB) || defined(__m68k__) || defined(__sparc64__) || defined(__ppc__) || defined(__ppc64__)

+# if defined(__hppa__) || defined(__rs6000__) || defined(__sparc__) || defined(_MIPSEB) || defined(__m68k__) || defined(__sparc64__) || defined(__ppc__) || defined(__ppc64__) || (defined(__hpux) && defined(__ia64))

# define K5_BE

# endif

#endif

# Mon Sep 14 22:36:14 2009 raeburn (Ken Raeburn) - Correspondence added

From:	Ken Raeburn <raeburn@MIT.EDU>
To:	rt@krbdev.mit.edu
Subject:	Re: [krbdev.mit.edu #6565] HP-UX IA64 wrong endian
Date:	Mon, 14 Sep 2009 22:36:11 -0400
RT-Send-Cc:

Download (untitled) / with headers
text/plain 578B

On Sep 14, 2009, at 21:45, Arlene Berry via RT wrote:

Show quoted text

> For krb5-1.7 all IA64 platforms are identified as little endian but
> HP-UX is big endian.

I wondered if there should be some startup-time sanity check of that,
just to be safe.

The current GCC sources indicate that it'll define __BIG_ENDIAN__ on
big-endian IA64 targets. It may be good to add __BIG_ENDIAN__ and
__LITTLE_ENDIAN__ to the list of macros we test for; certainly several
GCC configurations seem to define one or the other for several
processors that can go either way.

I'll take a look....

Ken

# Tue Sep 15 02:17:18 2009 raeburn (Ken Raeburn) - Given to raeburn (Ken Raeburn)

# Tue Sep 15 02:17:18 2009 raeburn (Ken Raeburn) - Status changed from 'new' to 'review'

# Tue Sep 15 02:17:18 2009 raeburn (Ken Raeburn) - Correspondence added

From:	raeburn@mit.edu
Subject:	SVN Commit

Download (untitled) / with headers
text/plain 329B

Check __BIG_ENDIAN__ and __LITTLE_ENDIAN__ to determine endianness.
In fallback code, check ia64 platforms for hpux vs everything else;
HP-UX uses big-endian mode.

https://github.com/krb5/krb5/commit/66f4ea3db7a9c1eb278b035fdae8ebea7636edb9
Commit By: raeburn
Revision: 22761
Changed Files:
U trunk/src/include/k5-platform.h

# Tue Sep 15 15:28:32 2009 raeburn (Ken Raeburn) - Correspondence added

From:	raeburn@mit.edu
Subject:	SVN Commit

Download (untitled) / with headers
text/plain 203B

Fix stupid logic bug in last version.

https://github.com/krb5/krb5/commit/2b607418f2d745472a0f9d70e6ddc61f21ee9a01
Commit By: raeburn
Revision: 22766
Changed Files:
U trunk/src/include/k5-platform.h

# Mon Jan 04 23:38:56 2010 tlyu (Taylor Yu) - Version_Fixed 1.8 added

# Sun Sep 24 20:39:55 2017 ghudson@mit.edu (Greg Hudson) - Status changed from 'review' to 'resolved'