Random Images



kyoto-cabinet  support transactions (search "transaction" for details).

That means that we can survive for some time without after-crash replay of application level WAL

  • We still write application level WAL transaction-log, but that is very easy
  • the replay and consistency check is more complex, we will implement that late

Kyoto cabinet:

  • either N*readers (no writers)
  • or 1 writer (no other writers and no other readers: the writer can write of course)

In short, this means that it is enough to use just 1 process to access to the database, no gains from using multiple processes (of course on an  8 CPU machine 7 CPU-s can handle crypto-signatures, and 1 handle the database, so we utilize all processors, and can do 1000+ real transactions  per second)

DB Usage:

  • most of the time reading would happen to lookup data (from dirty-cache and DB ), and generate signed certificates, update the "dirty-cache"-data in-memory (which can be hash, lists, trees, etc...) and write application-WAL to disk
    • note: we write app-level WAL transaction-log (at the very minimum, the drafts and certificates) even if we don't have after-crash recovery from our application-WAL.
  • Then, after "closing the batch" say appr every 20-30msec (or max 80-100 msec at high load) after the WAL is flush()-ed, all the "dirty-cache"-data from memory must be commited to the DB (these are all idempotent key=value writes), but during that time readers should NOT operate (at least much easier if readers don't operate)
    • a few years ago Marcell implemented an ugly hack-prototype of that "transaction batching" in Lua and tokyocabinet
    • the condition to "close the batch" and start writing was something like: 100msec or 1024 transactions, but it was closed prematurely if a new transaction attempted to access "dirty-data" (data which was not yet commited). This might be overcautious.

Created by: admin. Last Modification: 2012-03-05 (Mon) 09:33:22 CET by admin.