Ticket History #7860: libdb2 tests hang

Hi,

While updating the MIT Kerberos page for BLFS (www.linuxfromscratch.org/blfs), I found that "make check" generated a
few errors.

The first (and most important) one is:
----------
make: Entering directory
`/sources/mitkrb/krb5-1.11.4/src/plugins/kdb/db2/libdb2/test'
gcc -L.. -L../../../../../lib -Wl,--enable-new-dtags -Wl,-rpath
-Wl,/usr/lib -g -O2  -o dbtest dbtest.o  -ldb
LD_LIBRARY_PATH=`echo -L.. -L../../../../../lib | sed -e "s/-L//g" -e
"s/ /:/g"`; export LD_LIBRARY_PATH;  srcdir=. TMPDIR=.  /bin/sh ./run.test
Test 1: btree, hash: small key, small data pairs
test1: type btree: failed
make: Leaving directory
`/sources/mitkrb/krb5-1.11.4/src/plugins/kdb/db2/libdb2/test'
----------

I found the reason for that: the dictionary used is the one installed
for cracklib, which contains shell meta character and spaces. Those are
interpreted by the "for i in `cat $TMP1`" in run.test, so that what is
sent to the database is not what was originally in the file. Changing
the line:
"sed -e '/^$/d' < $DICT | cat -v | sed -e ${1}q"
to:
"sed -e '/[[:space:]*?/d' -e '/^$/d' < $DICT | cat -v | sed -e ${1}q"
allows the first test to pass.

But then, the second test hangs and has to be interrupted:
-------------
make: Entering directory
`/sources/mitkrb/krb5-1.11.4/src/plugins/kdb/db2/libdb2/test'
LD_LIBRARY_PATH=`echo -L.. -L../../../../../lib | sed -e "s/-L//g" -e
"s/ /:/g"`; export LD_LIBRARY_PATH;  srcdir=. TMPDIR=.  /bin/sh ./run.test
Test 1: btree, hash: small key, small data pairs
Test 1: recno: small key, small data pairs
Test 2: btree, hash: small key, medium data pairs
Test 2: recno: small key, medium data pairs^C
------------

I tried to debug this by putting some printf at various places in the
code, I found that:
- the call to dbp->close(dbp) does not return (in dbtest)
- in rec_close.c, the instructions:
                while (status == RET_SUCCESS) {
                        iov[0].iov_base = data.data;
                        iov[0].iov_len = data.size;
                        if (writev(t->bt_rfd, iov, 2) != data.size + 1)
                                return (RET_ERROR);
                        status = (dbp->seq)(dbp, &key, &data, R_NEXT);
                }
enter an infinite loop, but the file __dbtest keeps the same size.

Going farther without assistance is too hard... I would need at least some pointers to docs about the structure of the database.

System information. The box is a virtual machine under qemu:
Linux virtuallfs 3.10.10-lfs-1 #1 SMP Tue Feb 4 17:16:25 CET 2014 x86_64
GNU/Linux
gcc (GCC) 4.8.1
Copyright (C) 2013 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  There is NO
warranty; not even for MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.

The build is as follows:
cd src &&
sed -e "s@python2.5/Python.h@& python2.7/Python.h@g" \
    -e "s@-lpython2.5]@&,\n
AC_CHECK_LIB(python2.7,main,[PYTHON_LIB=-lpython2.7])@g" \
    -i configure.in &&
sed -e "s@interp->result@Tcl_GetStringResult(interp)@g" \
    -i kadmin/testing/util/tcl_kadm5.c &&
autoconf &&
./configure CPPFLAGS="-I/usr/include/et -I/usr/include/ss" \
            --prefix=/usr                                  \
            --sysconfdir=/etc                              \
            --localstatedir=/var/lib                       \
            --with-system-et                               \
            --with-system-ss                               \
            --enable-dns-for-realm &&
make

There are no krb5 libraries on the system (first install). The same
behavior is observed for both versions 1.11.4 et 1.12.1 (1.12.1 is built
without the CPPFLAGS).

Regards
Pierre
NB : the other error is a minor defect in the krb testsuite of
"tests/dejagnu". I can send another report on request.

At this point I am pretty sure that this is actually a compiler bug,
because:

* I can't reproduce the problem with the head revision of gcc.

* With gcc 4.8.1, I was able to construct simplified test cases (like
the attached file) which clearly don't have strict aliasing violations,
and only take advantage of C99 6.3.2.3 paragraph 7.

I would speculate that in the attached test case, the gcc 4.8.1
optimizer sees list dereferenced before the conditional, and ent->prev
in the else clause, and erroneously concludes that if the two pointers
are the same, a strict aliasing violation must have occurred and the
behavior is undefined. But since the dereference only happens when the
pointers are not equal, that deduction is invalid.

As for TAILQ, the macros in db-queue.h appear to be free of any aliasing
or type-punning issues. More recent versions of queue.h (including the
one we copied into include/k5-queue.h) define additional TAILQ functions
including:

#define TAILQ_PREV(elm, headname, field) \
(*(((struct headname *)((elm)->field.tqe_prev))->tqh_last))

which is pretty clearly a strict aliasing violation since tqe_prev could
point to the first member of an element structure rather than a list
head. Therefore, we should avoid using TAILQ_PREV,
TAILQ_FOREACH_REVERSE, and TAILQ_FOREACH_REVERSE_SAFE if we want to
avoid unpleasant surprises. TAILQs aren't really designed for reverse
iteration anyway.

I will adjust the commit message on my CIRCLEQ->TAILQ patch to reflect
my belief that it's working around a compiler bug, and push it.

I will revert d21a86e47a7cda29225013e08d060095b94b2ee7 and make an
alternate change to deal with special shell characters in the system
dictionary, since the built-in word list is not long enough for all of
the tests.

#include <stdio.h>

struct entry {
struct entry *next;
struct entry *prev;
};

struct entrylist {
struct entry *first;
};

static void __attribute__((noinline))
test(struct entrylist *list)
{
struct entry *ent = list->first;

if (ent->prev == (void *)list)
list->first = ent->next;
else
ent->prev->next = ent->next;

ent->next = list->first;
ent->prev = (void *)list;
list->first = ent;
}

int
main()
{
struct entrylist list;
struct entry ent;

list.first = (void *)&list;
ent.next = (void *)&list;
ent.prev = (void *)&list;
list.first = &ent;
test(&list);
printf("list: %p\n", &list);
printf("ent: %p\n", &ent);
printf("next: %p\n", ent.next);
}