Resent-To: rt-comment@krbdev.MIT.EDU
Resent-From: Tom Yu <tlyu@MIT.EDU>
Resent-Date: Wed, 28 Aug 2013 15:17:23 -0400
Resent-Message-ID: <ldvvc2p7hh8.fsf@cathode-dark-space.mit.edu>
Date: Wed, 28 Aug 2013 00:25:04 -0400
From: Richard Basch <basch@alum.MIT.EDU>
Subject: RE: [krbdev.mit.edu #7695] AutoReply: krb5-1.11.3/1.10.6 - full resync may fail and still result in ulog being updated
In-Reply-To: 
To: "'Richard Basch'" <basch@alum.MIT.EDU>, krbdev@mit.edu
CC: richard.basch@gs.com
Message-ID: <069e01cea3a6$92217d50$b66477f0$@mit.edu>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii
Content-Language: en-us
Content-Transfer-Encoding: 7BIT
Thread-Index: Ac6cYhyM71adIhhoSuehsJ/d2YadNQEq7cowAG8bI8AAAdDQ4AA0+1zQ
References: <rt-7695@krbdev.mit.edu> <rt-7695-38935.16.4884902896994@krbdev.mit.edu> <045301cea10e$750307b0$5f091710$@mit.edu>
RT-Send-Cc: 
Content-Length: 5765

One more fix for 1.11 related to full resync (and this set of fixes is
"final"; I have confirmed functionality in my test environment).

1.10:
https://github.com/rbasch/krb5/commit/fc7aabea9bf0abb0712a8509c38d4382474361
c3

1.11:
https://github.com/rbasch/krb5/commit/f6237998bf7b20ea898d8b1ac2b30255caad89
d8
https://github.com/rbasch/krb5/commit/affc746f296869d25c49ee2eabc843c60470ac
df
https://github.com/rbasch/krb5/commit/906d18fe56849ee59a114c31e5242a749166bc
f5

Basically, there was a significant performance hit with the 1.11 DB reloads,
which I finally tracked down to the ulog locking for each record being
inserted during the database during a DB reload (causing a timeout even with
a 30 minute timeout; the reload then completed in 2-3 minutes after the data
transfer after making the last change).

In essence, 1.11 required 3 patches... 1.10 only 1 patch.

The other problem with 1.11 was I found it would pick up improper/stale dump
files especially if the serial number had been reset to a lower value. So in
essence, my patch for 1.11 is in 3 parts:
- Reset the ulog to the new information only after the new database is
promoted.
- Perform a more thorough check whether a conditional dump is required (the
quick check was flawed)
- Only perform a ulog_lock in krb5_db_put_principal if it is on the master
KDC (whether you adopt this technique or simply do not lock if iproprole is
a specific value is debatable), but during a full db reload, you should not
perform a ulog_lock for each record being loaded.

This set seems to solve all the problems I have observed.


-----Original Message-----
From: Richard Basch [mailto:basch@alum.mit.edu] 
Sent: Monday, August 26, 2013 11:16 PM
To: 'Richard Basch'; 'krbdev@mit.edu'
Cc: 'richard.basch@gs.com'
Subject: RE: [krbdev.mit.edu #7695] AutoReply: krb5-1.11.3/1.10.6 - full
resync may fail and still result in ulog being updated

I believe these are final... They compile and I believe they will fix the
issues (I will be testing for a couple days, but "no news is good news").

1.10:
https://github.com/rbasch/krb5/commit/fc7aabea9bf0abb0712a8509c38d4382474361
c3

1.11:
https://github.com/rbasch/krb5/commit/f6237998bf7b20ea898d8b1ac2b30255caad89
d8
https://github.com/rbasch/krb5/commit/affc746f296869d25c49ee2eabc843c60470ac
df


-----Original Message-----
From: Richard Basch [mailto:basch@alum.mit.edu] 
Sent: Monday, August 26, 2013 10:12 PM
To: 'Richard Basch'; 'krbdev@mit.edu'
Cc: 'richard.basch@gs.com'
Subject: RE: [krbdev.mit.edu #7695] AutoReply: krb5-1.11.3/1.10.6 - full
resync may fail and still result in ulog being updated

There is a second bug... I don't believe the patch I provided is wrong, but
I believe to be effective in my test environment, a second fix is required,
specifically to the dump side of kdb5_util, where it might not properly
determine whether to create a new dump file (the quick serial number check
is flawed).

Stay tuned for another git reference (though this one should only apply to
1.11 since 1.10 doesn't use the same conditional dump optimization logic).


-----Original Message-----
From: Richard Basch [mailto:basch@alum.mit.edu] 
Sent: Saturday, August 24, 2013 5:11 PM
To: krbdev@mit.edu
Cc: richard.basch@gs.com; 'Richard Basch'
Subject: RE: [krbdev.mit.edu #7695] AutoReply: krb5-1.11.3/1.10.6 - full
resync may fail and still result in ulog being updated

Instead of the supplied patch, refer to the following patches instead:

1.11:
https://github.com/rbash/krb5/commit/affc746f296869d25c49ee2eabc843c60470ac
df

1.10:
https://github.com/rbasch/krb5/commit/fc7aabea9bf0abb0712a8509c38d4382474361
c3

I am relatively confident in the above since they move the ulog update to
after the db promotion, ensuring everything is ok first. This should avoid
all the issues which previously existed.

There is a remote chance something might prevent the ulog update from taking
place but the db might be updated. I am not quite sure what the right answer
is in this case, but certainly the other way round of updating the ulog
before the database matches is wrong. There are pros and cons to resetting
the ulog prior to the db load, but in either scenario, the final state
should not be set until after the db is loaded & promoted.


-----Original Message-----
From: krb5 [mailto:rt@krbdev.mit.edu] 
Sent: Sunday, August 18, 2013 6:26 PM
To: basch@alum.mit.edu
Subject: [krbdev.mit.edu #7695] AutoReply: krb5-1.11.3/1.10.6 - full resync
may fail and still result in ulog being updated 


Greetings,

This message has been automatically generated in response to the
creation of a trouble ticket regarding:
	"krb5-1.11.3/1.10.6 - full resync may fail and still result in ulog
being updated", 
a summary of which appears below.

There is no need to reply to this message right now.  Your ticket has been
assigned an ID of [krbdev.mit.edu #7695].

Please include the string:

         [krbdev.mit.edu #7695]

in the subject line of all future correspondence about this issue. To do so,

you may reply to this message.

                        Thank you,
                        

-------------------------------------------------------------------------
In my test environment, even with krb5-1.11.3, I noticed a database reload
(full resync) may still fail and result in the ulog being updated with the
new serial number, resulting in an inconsistent environment.

 
I have another patch available which seems to fix the condition.
Specifically, I have seen the condition occur with an accompanying log
message:

./kdb5_util returned a bad exit status (2)

 
krb5-1.11

https://github.com/rbasch/krb5/commit/83c34de8a740711961d05fde150cc8b16e68f9
e1

 
krb5-1.10

https://github.com/rbasch/krb5/commit/638b2e299157b1c2e637cb992bc07cf9f55985
94