Ah, bummer! Was hoping for something easy. The last bit of context I can provide that I realized was absent in my initial report is that from the apache side, we see this most often (all of the time? Difficult to say that with certainty) while telling apache to do a "graceful reload or restart", which involves a managed ramp down / killing of child processes, and those children that are throwing this.

Throwing a hypothesis out there without much context on this project, if I'm reading your message correctly: potentially https://github.com/krb5/krb5/blob/81e47875e3de0e52fbb11d61ef30a9406497af73/src/lib/gssapi/krb5/gssapi_krb5.c#L1117-L1119 is a good place to look? If so, I'm wondering if the init function's registration of those variables (https://github.com/krb5/krb5/blob/81e47875e3de0e52fbb11d61ef30a9406497af73/src/lib/gssapi/krb5/gssapi_krb5.c#L1072-L1081) could either be interrupted (in my head I'm imagining by a sigkill?), or have thrown an error (leading to some of the keys being unregistered)

I'm very rarely handing out in c-land, so the idea of interrupted execution may be completely off base, but the error handling still stands out to me as having potential?

On Thu, Jan 9, 2020 at 8:01 PM Greg Hudson via RT <rt@krbdev.mit.edu> wrote:
We do see rare reports of assertion failures in the krb5int_key functions,
which handle an internal table of thread-specific data keys.

Ticket 8614 (which you linked to from the stackoverflow answer) happens because
krb5int_key_register() is called on a key that is marked as already registered.
A candidate explanation there is two different versions of libgssapi_krb5 in
the same process, both calling into the same libkrb5support, although I'm not
sure that's right--although it's easy enough to have multiple versions of
libgssapi_krb5 installed on a machine, they should all have the same soname
(since we haven't changed it in a long time), which I think would make it
difficult to load more than one version into a process.

The failure reported here is the inverse: krb5int_key_delete() is called on a
key that isn't marked as registered. krb5int_key_delete() is invoked from the
library finalizer of libgssapi_krb5 (also the finalizer of the krb5 version of
libcom_err, but typically the e2fsprogs version of com_err is used on Linux).
Although it's possible for the finalizer to run without the initializer having
run, there is a check for that. So I don't have any good theories.