Ticket History #7721: Primary KDC lookups happen sooner than necessary

Currently, the fallback to primary is accomplished by checking whether the primary was used in krb5_sendto_kdc(), communicating this back up the chain through k5_init_creds_get() and k5_get_init_creds() to krb5_get_init_creds_password().  If a failure is observed other than four specific codes (indicating a KDC connectivity or password entry issue), krb5_get_init_creds_password() retries the whole AS exchange with use_primary set.  If k5_init_creds_get() makes multiple requests; the use_primary value communicated up the chain is for the last request sent.

When evaluating alternative designs, it is important to consider whether the fallback must retry only the final request or (as it currently does) the entire exchange.  Retrying the final request would be sufficient for many multi-request exchanges; for example, if encrypted timestamp failed because of a recent password change, retrying the preauthenticated request would solve the problem.  If the account was recently created, retrying the final request might need to be iterated, as both the unauthenticated and preauthenticated requests would fail against an outdated replica KDC.  In more complicated scenarios we could see a failure due to an earlier request using a replica KDC.  Some examples include:

* When using SPAKE, if the SPAKEChallenge was sent by an replica KDC with an outdated long-term key, we would only see a failure after the SPAKEResponse is sent.

* If an AS request requires a WRONG_REALM referral and the referral target changes because the account is moved, we would only see a failure after the old realm is contacted.

Note that for these cases the current design can erroneously conclude that a fallback wouldn't be productive, because it only looks at the use_primary value for the final request.  This problem could be rectified easily enough by making k5_init_creds_get() communicate the logical AND of use_primary results to the caller.

Some candidate designs:

* krb5_sendto_kdc() could evaluate the KDC error code and retry against the primary KDC internally.  This would work for simple scenarios but not more complex ones.  This design requires somewhat expansive knowledge of error codes to know when not to retry--not just the ones indicating unrecoverable failures but ones such as PREAUTH_REQUIRED which are handled as part of multi-request exchanges.  (In theory the set of non-fallback error codes is unbounded due to preauth errors such as KDC_ERR_DH_KEY_PARAMETERS_NOT_ACCEPTED, but retrying on such errors might be an acceptable inefficiency.)

* krb5_sendto_kdc() could perform the lookup only when it's cheap (no use of DNS), leaving *use_primary as 0 otherwise.  We would retry in more failure cases but wouldn't make any extra DNS requests on success.  This design would require changes to k5_locate_kdc(), and would require deciding whether locate plugins are assumed to be cheap or expensive (until a refresh of the locate interface, which we want for other reasons).

* krb5_sendto_kdc() could communicate the server it used back to the caller (e.g. by transferring ownership of servers.servers[server_used]).  The caller could then query it with k5_kdc_is_primary().  Properly handling the complicated cases would require maintaining an array of {realm, server} objects and querying all of them until one is determined not to be primary.

* krb5_sendto_kdc() could have a state object which persists after the call.  We could then call krb5_sendto_kdc() again with the same state object, and it could determine whether its prior answer was from a primary KDC.  Properly handling the complicated cases would require maintaining an array of state objects.  This design might require implementing https://k5wiki.kerberos.org/wiki/Projects/Rule_based_sendto_kdc_loop .