Co-authored-by: James Mills <1290234+prologic@users.noreply.github.com>
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Co-authored-by: Tai Groot <tai@taigrr.com>
Reviewed-on: https://git.mills.io/prologic/bitcask/pulls/160
Co-authored-by: James Mills <james@mills.io>
Co-committed-by: James Mills <james@mills.io>
Added Sift and ScanSift functions for review without tests (for now)
fix docstrings
Added tests for Sift and ScanSift
Note this also fixes a bug in the Scan() function where the RMutex is not locked, allowing a potential race condition
closes#231
Reviewed-on: https://git.mills.io/prologic/bitcask/pulls/232
Co-authored-by: Tai Groot <tai@taigrr.com>
Co-committed-by: Tai Groot <tai@taigrr.com>
Fixes#228
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/bitcask/pulls/229
Co-authored-by: James Mills <james@mills.io>
Co-committed-by: James Mills <james@mills.io>
[ADDED] new tests for TTL expiration race condition, see #216
[REMOVED] removes cleanup / automatic expiration from get() function to resolve#216
Reviewed-on: https://git.mills.io/prologic/bitcask/pulls/227
Co-authored-by: Tai Groot <tai@taigrr.com>
Co-committed-by: Tai Groot <tai@taigrr.com>
* Add test case for Locking after Merge
* retain lock file after merge
* remove replacing lock file (not needed)
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Co-authored-by: yash <yash.chandra@grabpay.com>
* add failing test case to highlight the race condition on bug
note : the test "TestLock" is non deterministic, its outcome depends
on the sequence of instructions yielded by the go scheduler on each run.
There are two values, "goroutines" and "succesfulLockCount", which can
be edited to see how the test performs.
With the committed value, resp "20" and "50", I had a 100% failure on
my local machine, running linux (Ubuntu 20.04).
Sample test output :
$ go test . -run TestLock
--- FAIL: TestLock (0.17s)
lock_test.go:91: [runner 14] lockCounter was > 1 on 5 occasions, max seen value was 2
lock_test.go:91: [runner 03] lockCounter was > 1 on 2 occasions, max seen value was 3
lock_test.go:91: [runner 02] lockCounter was > 1 on 3 occasions, max seen value was 3
lock_test.go:91: [runner 00] lockCounter was > 1 on 1 occasions, max seen value was 2
lock_test.go:91: [runner 12] lockCounter was > 1 on 7 occasions, max seen value was 3
lock_test.go:91: [runner 01] lockCounter was > 1 on 8 occasions, max seen value was 2
lock_test.go:91: [runner 04] lockCounter was > 1 on 6 occasions, max seen value was 4
lock_test.go:91: [runner 13] lockCounter was > 1 on 1 occasions, max seen value was 2
lock_test.go:91: [runner 17] lockCounter was > 1 on 4 occasions, max seen value was 2
lock_test.go:91: [runner 10] lockCounter was > 1 on 3 occasions, max seen value was 2
lock_test.go:91: [runner 08] lockCounter was > 1 on 6 occasions, max seen value was 2
lock_test.go:91: [runner 09] lockCounter was > 1 on 4 occasions, max seen value was 2
lock_test.go:91: [runner 05] lockCounter was > 1 on 1 occasions, max seen value was 2
lock_test.go:91: [runner 19] lockCounter was > 1 on 3 occasions, max seen value was 3
lock_test.go:91: [runner 07] lockCounter was > 1 on 4 occasions, max seen value was 3
lock_test.go:91: [runner 11] lockCounter was > 1 on 9 occasions, max seen value was 2
lock_test.go:91: [runner 15] lockCounter was > 1 on 1 occasions, max seen value was 3
lock_test.go:91: [runner 16] lockCounter was > 1 on 1 occasions, max seen value was 3
FAIL
FAIL github.com/prologic/bitcask 0.176s
FAIL
* flock: create a wrapper module, local to bitcask, around gofrs.Flock
the racy TestLock has been moved to bitcask/flock
* flock: add test for expected regular locking behavior
* flock: replace gofrs/flock with local implementation
* update go.sum
* Add build constraint for flock_unix.go
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
* new merge approach
* code refactor
* comment added
* isMerging flag added to allow 1 merge operation at a time
* get api modified. merge updated (no recursive read locks)
Co-authored-by: yash <yash.chandra@grabpay.com>
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
* live backup first commit
* exclude lock file in backup
* create path if not exist for backup
Co-authored-by: yash <yash.chandra@grabpay.com>
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
* Add configuration options for FileMode
Add two additional configuration values, and their corresponding default values:
* DirFileModeBeforeUmask - Dir FileMode is used on all directories created. DefaultDirFileModeBeforeUmask is 0700.
* FileFileModeBeforeUmask - File FileMode is used on all files created, except for the "lock" file (managed by the Flock library). DefaultFileFileModeBeforeUmask is 0600.
When using these bits of configuration, keep in mind these FileMode values are set BEFORE any umask rules are applied. For example, if the user's umask is 022, setting DirFileFileModeBeforeUmask to 777 will result in directories with FileMode set to 755 (this umask prevents the write bit from being applied to group and world permissions).
* moving defer statements after checking for errors
use os.ModePerm const instead of os.FileMode(777)
* fix spelling/grammar
* skip these tests for Windows as they appear to break - Windows is less POSIX-y than it claims
* ignore "lock" file for default case too -- this was incorrectly passing before including this, as my local dev station has umask 022
* internal/data: comment exported functions
* internal/data: make smaller codec exported api surface
* make key and value sizes serializing bubble up to everything
* Makefile setup & go mod tidy
* Add Unit Test for testing a corrupted config
* Add Unit Test for testing errors from .Stats()
* Refactor Datafile into an interface and add Unit Tests for testing Merge() errors
* Refactor indexer into an interface and add Unit Tests for .Close() errors
* Add Unit Tests for .Delete() errors
* Add Unit Tests for testing Put/Get errors
* Add Unit Test for testing Open errors (bad path for example)
* Refactor out bitcask.writeConfig
* Add more tests for config errors
* Add unit test for options that might error
* Add more test cases for close errors
* Add test case for rotating datafiles
* Fix a possible data race in .Stats()
* Add test case for checksum errors
* Add test case for Sync errors with Put and WithSync enabled
* Refactor and use testify.mock for mocks and generate mocks for all interfaces
* Refactor TestCloseErrors
* Refactored TestDeleteErrors
* Refactored TestGetErrors
* Refactored TestPutErrors
* Refactored TestMergeErrors and fixed a bug with .Fold()
* Add test case for Scan() errors
* Apparently only Scan() can return nil Node()s?
* Add Unit Test for testing WithSync() option
* Add Unit Test for testing re-indexing
* Add Unit Test for testing re-indexing with deleted keys (tombstone values)
* avoid unnecessary use of encoder/decoder to decrease memory allocations
* add an optional configurable mempool to avoid extra allocs
* add doc.go with examples
* Added new API Stats() and Prune()
* Improved Merge() logic to also prune old key/values and actually reclaim disk space
* Added backward compat for the old Merge() function
* Refactor indexing of keys to items (hints)
* Remove redundant TestOpenMerge
* Add unit test for Stats()
* Improve TestMerge()
