Vanishing records in deferred-write database524806 Aug 23, 2007 11:20 AM
Upon upgrading to 3.2.43, one of our project's unit tests revealed an issue with records disappearing 'spontaneously' from a deferred-write database (a few seconds after a reinsertion at a previous key value). I was able to boil it down to a small test case:
It only happens with deferred-write enabled, and does not happen in 3.2.23.
- Gordon @ IA
As you'll see, about 5 seconds after a record is put back at a previously-deleted key, it disappears -- as if some delayed background cleanup from the previous delete has affected the new insert.As you'll see, about 5 seconds after a record is put back at a previously-deleted key, it disappears -- as if some delayed background cleanup from the previous delete has affected the new insert.
import java.io.File; import org.archive.util.FileUtils; import com.sleepycat.je.Database; import com.sleepycat.je.DatabaseConfig; import com.sleepycat.je.DatabaseEntry; import com.sleepycat.je.DatabaseException; import com.sleepycat.je.Environment; import com.sleepycat.je.EnvironmentConfig; public class DisappearingDeferredWriteRecord { public static void main(String[] args) throws DatabaseException, InterruptedException { EnvironmentConfig envConfig = new EnvironmentConfig(); envConfig.setTransactional(true); envConfig.setAllowCreate(true); File envDir = new File(DisappearingDeferredWriteRecord.class.getSimpleName()); if (envDir.exists()) { FileUtils.deleteDir(envDir); } envDir.mkdirs(); Environment env = new Environment(envDir, envConfig); DatabaseConfig dbConfig = new DatabaseConfig(); dbConfig.setTransactional(false); dbConfig.setAllowCreate(true); dbConfig.setDeferredWrite(true); // <-NECESSARY Database db = env.openDatabase(null, "db", dbConfig); DatabaseEntry empty = new DatabaseEntry(new byte[0]); db.put(null,new DatabaseEntry("0".getBytes()),empty); System.out.println("expect 1: "+db.count()); db.delete(null, new DatabaseEntry("0".getBytes())); System.out.println("expect 0: "+db.count()); db.put(null,new DatabaseEntry("1".getBytes()),empty); System.out.println("expect 1: "+db.count()); db.put(null,new DatabaseEntry("0".getBytes()),empty); System.out.println("expect 2: "+db.count()); // db.sync(); // does help, if done here Thread.sleep(5000); // <-WAIT MUST BE ~ 5 SECONDS OR MORE System.out.println("expect 2: "+db.count()); // "1" !! db.sync(); // doesn't help, here System.out.println("expect 2: "+db.count()); // "1" !! db.close(); env.close(); } }
It only happens with deferred-write enabled, and does not happen in 3.2.23.
- Gordon @ IA
This content has been marked as final. Show 6 replies
1. Re: Vanishing records in deferred-write database512799 Aug 23, 2007 11:57 AM (in response to 524806)Hi Gordon,
Thanks for reporting this. I just did a quick run and didn't see the error. We will investigate and report back soon.
Ron
My results:
expect 0: 0
expect 1: 1
expect 2: 2
expect 2: 2
expect 2: 2
2. Re: Vanishing records in deferred-write databaseGreybird-Oracle Aug 23, 2007 3:23 PM (in response to 512799)Gordon,
It does fail for me, and I suspect it's timing dependent and the result of concurrent daemon thread activity. Just letting you know that we reproduced it. I'll post more status later today.
--mark
3. Re: Vanishing records in deferred-write database524806 Aug 23, 2007 4:49 PM (in response to Greybird-Oracle)I did trim the delay to the barest minimum, 5000ms, that reliably reproduced here. (Even 4900ms was iffy.) So if not seeing it on a specific machine, I'd first try upping that. Perhaps execution timings or thread scheduling is a big factor. (Our initial test failure was using a 32-bit 1.5 JVM on an athlon64; this minimal test case has been triggering the error on a 1.6 JVM core duo for me.)
- Gordon @ IA
4. Re: Vanishing records in deferred-write databaseGreybird-Oracle Aug 24, 2007 3:55 PM (in response to 524806)I have a status update on this problem. We have a fix, a new test case, and we are running stress tests now. We expect to be able to publish a release with this fix during the first part of next week.
Gordon, if you want to try the fix earlier, please send me email. Thanks again for narrowing the problem down and giving us a reproducible case -- that helped a lot.
--mark
5. Re: Vanishing records in deferred-write database524806 Aug 24, 2007 6:12 PM (in response to Greybird-Oracle)Would be happy to test sooner if you'd like but our need is not urgent -- we can stay with 3.2.23 for now. Thanks for the quick response.
- Gordon
6. Re: Vanishing records in deferred-write databaseGreybird-Oracle Aug 28, 2007 10:09 PM (in response to 524806)Hi Gordon,
This bug has been fixed in JE 3.2.44:
DB Java Edition 3.2.44 is now available
Thanks again for reporting it and for the reproducible test case.
--mark