Commit Graph

44 Commits

Author SHA1 Message Date
James Mills
90dd53c573
Rename all Go module paths 2021-07-10 17:47:38 +10:00
Yash Suresh Chandra
e7c6490762
Purge api added to remove expired keys (#204)
* purge api added

* merged with master, import order fix

* purge api renamed to RunGC

Co-authored-by: yash <yash.chandra@grabpay.com>
2021-06-02 06:47:30 +10:00
Yash Suresh Chandra
5c6ceadac1
Add support for keys with ttl (#177)
* ttl support first commit

* imports fix

* put api args correction

* put options added

* upgrade method added

* upgrade log added

* v0 to v1 migration script added

* error assertion added

* temp migration dir fix

Co-authored-by: yash <yash.chandra@grabpay.com>
2020-12-21 17:41:43 +10:00
James Mills
8a60b5a370
Fix a bug when MaxValueSize == 0 on Merge operations 2020-12-18 07:35:16 +10:00
James Mills
0ab7d79246
Add support for unlimited key/value sizes 2020-12-12 02:16:36 +10:00
yashschandra
158f6d9888
Get space that can be reclaimed (#189)
* get reclaimable space added

* import order fix

Co-authored-by: yash <yash.chandra@grabpay.com>
2020-12-01 06:07:00 +10:00
yashschandra
f4357e6f18
local live backup support (#185)
* live backup first commit

* exclude lock file in backup

* create path if not exist for backup

Co-authored-by: yash <yash.chandra@grabpay.com>
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
2020-11-30 07:49:02 +10:00
Bryan Stenson
295301a44c
Add configuration options for FileMode (#183)
* Add configuration options for FileMode

Add two additional configuration values, and their corresponding default values:

* DirFileModeBeforeUmask - Dir FileMode is used on all directories created.  DefaultDirFileModeBeforeUmask is 0700.
* FileFileModeBeforeUmask - File FileMode is used on all files created, except for the "lock" file (managed by the Flock library).  DefaultFileFileModeBeforeUmask is 0600.

When using these bits of configuration, keep in mind these FileMode values are set BEFORE any umask rules are applied.  For example, if the user's umask is 022, setting DirFileFileModeBeforeUmask to 777 will result in directories with FileMode set to 755 (this umask prevents the write bit from being applied to group and world permissions).

* moving defer statements after checking for errors

use os.ModePerm const instead of os.FileMode(777)

* fix spelling/grammar

* skip these tests for Windows as they appear to break - Windows is less POSIX-y than it claims

* ignore "lock" file for default case too -- this was incorrectly passing before including this, as my local dev station has umask 022
2020-11-05 08:06:45 +10:00
shiniao
643e578585
Refactored Save function for config (#179)
* Refactored Save function for config

* Refactored Save function for config
2020-10-29 21:47:08 +10:00
chenbc
afdf956e7a
Fix recoverDatafile error covering (#162) 2020-06-17 15:06:30 +10:00
Ignacio Hagopian
8dca9cd2a7
Auto recovery (#153)
* implement autorepair

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* fix misspell

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* Update internal/data/recover.go

Co-authored-by: James Mills <prologic@shortcircuit.net.au>

* Update internal/utils.go

Co-authored-by: James Mills <prologic@shortcircuit.net.au>

* Update internal/data/recover.go

Co-authored-by: James Mills <prologic@shortcircuit.net.au>

* skip failing test on windows

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
2020-05-08 03:48:36 +10:00
Ignacio Hagopian
7b24d87695
don't allow empty keys (#151)
* bitcask: don't allow empty keys

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>

* go mod tidy

Signed-off-by: Ignacio Hagopian <jsign.uy@gmail.com>
2020-05-06 10:47:41 +10:00
James Mills
65e9317d26
Fix glfmt/golint issues 2019-10-14 16:55:47 +10:00
Ignacio Hagopian
498ea4069c codebeat: Code quality improvement (#103)
* codebeat: improve & bugfix

* codebeat: refactor to improve readability

* bugfix

* bugfix

* internal/data/codec: improve code coverage
2019-09-24 07:19:07 +10:00
Ignacio Hagopian
f17187a5c7
Test for data corruption in datafile decoding (#99)
* internal/data: move codec to own subpackage

* internal/data/codec: check & test nil Entry Decode

* internal/data/decoder: test for short prefix error

* internal/data/codec: test invalid key & value sizes

* internal/data/codec: check & test for truncated data

* interna/data/codec: use assert for tests
2019-09-16 09:29:08 -03:00
Ignacio Hagopian
5be114adab
Makefile setup & key/value coherent datatypes & refactoring (#98)
* internal/data: comment exported functions

* internal/data: make smaller codec exported api surface

* make key and value sizes serializing bubble up to everything

* Makefile setup & go mod tidy
2019-09-12 10:44:26 -03:00
Ignacio Hagopian
7e0fa151f7 fix test compilation (#97) 2019-09-10 06:25:09 +10:00
James Mills
d59d5ad8c2
Improves Test Coverage by covering error cases (#95)
* Add Unit  Test for testing a corrupted config

* Add Unit Test for testing errors from .Stats()

* Refactor  Datafile into an interface and add Unit Tests for testing Merge() errors

* Refactor indexer into an interface and add Unit Tests for .Close() errors

* Add Unit Tests for .Delete() errors

* Add Unit Tests for  testing Put/Get errors

* Add Unit Test for testing Open errors (bad path for example)

* Refactor out bitcask.writeConfig

* Add more tests for config errors

* Add unit test for options that might error

* Add more test cases for close errors

* Add test case for rotating datafiles

* Fix a possible data race in .Stats()

* Add test case for checksum errors

* Add test case for Sync errors with Put and WithSync enabled

* Refactor and use testify.mock for mocks and generate mocks for all interfaces

* Refactor TestCloseErrors

* Refactored TestDeleteErrors

* Refactored TestGetErrors

* Refactored TestPutErrors

* Refactored TestMergeErrors and fixed a bug with .Fold()

* Add test case for Scan() errors

* Apparently only Scan() can return nil Node()s?
2019-09-09 07:18:38 +10:00
Ignacio Hagopian
0d3a9213ed cmd/bitcask: recovery tool (#92)
* cmd/bitcask: recovery tool

* refactor configuration & use it in recover tool
2019-09-07 07:57:30 +10:00
Ignacio Hagopian
a2b5ae2287 fix: check of persisted index values (#91) 2019-09-04 22:42:32 +10:00
Ignacio Hagopian
93cc1d409f codec_index: check sizes, new tests for data corruption & refactor (#84)
* bitcask/codec_index: check key and data sizes

* codec_index: tests for key and data size overflows

* codec_index: simplify internal funcs for unused returns
2019-09-04 12:26:26 +10:00
Ignacio Hagopian
8041a4c1e7 Refactor and general tests for codec index (#83)
* codec_index: unexport const fields

* codec_index: unexport internal functions and doc exported ones

* codec_index: rename func & return errors for corruption

* codec_index: new test for ReadIndex, WriteIndex, and read corruption

* Update internal/codec_index.go

Co-Authored-By: James Mills <1290234+prologic@users.noreply.github.com>

* Update internal/codec_index.go

Co-Authored-By: James Mills <1290234+prologic@users.noreply.github.com>
2019-09-03 08:19:35 +10:00
James Mills
abbbeb8e1d
Replace keydir with ART trie (#75)
* Replace keydir with ART trie

* Address some review feedback

* Address review feedback (consts)
2019-09-02 08:38:56 +10:00
James Mills
36bc134b22
Fix a bug wit the decoder passing the wrong value for the value's offset into the buffer (#77) 2019-08-31 08:35:17 +10:00
Ignacio Hagopian
8f56cffd86 codec: collapse write and save extra alloc (#64) 2019-08-12 06:47:26 +10:00
James Mills
7204a33512
Fix and cleanup some unnecessary internal sub-packages and duplication 2019-08-08 22:28:25 +10:00
James Mills
c7d101d34f
Fixed missing model import for codec with undefined Entry 2019-08-08 20:14:27 +10:00
Awn
af43cfa8f1 Remove merge function (#60)
* tidy: clean up some leftovers

Fixes #56
Fixes #57
Fixes #58

* api: remove standalone merge function

Fixes #55
2019-08-08 19:51:45 +10:00
Ignacio Hagopian
1f10b4026d inline hash function to save extra alloc (#53) 2019-08-08 09:52:31 +10:00
Ignacio Hagopian
fd179b4a86 custom high-performance encoder implementation (#52) 2019-08-08 09:21:46 +10:00
James Mills
755b1879b5
Use []byte byte slices as keys directly avoiding serialing string(s) (#46) (#51) 2019-08-08 08:14:48 +10:00
James Mills
d0c913ccee
Revert "Use []byte byte slices as keys directly avoiding serialing string(s) (#46)" (#50)
This reverts commit 3c1808cad3f19c23c6e4aacd4cfbbc4a04da1c08.
2019-08-08 08:06:38 +10:00
James Mills
3c1808cad3
Use []byte byte slices as keys directly avoiding serialing string(s) (#46) 2019-08-08 07:59:11 +10:00
Ignacio Hagopian
a407905ae2 Improve Get/Put performance with optional mempooling (#36)
* avoid unnecessary use of encoder/decoder to decrease memory allocations

* add an optional configurable mempool to avoid extra allocs

* add doc.go with examples
2019-08-05 07:23:07 +10:00
Ignacio Hagopian
8aa66c66da keydir: avoid defers (#34) 2019-08-01 19:18:05 +10:00
James Mills
51bac21c0a
Improves Merge() operation by also pruning old key/value pairs (#29)
* Added new API Stats() and Prune()

* Improved Merge() logic to also prune old key/values and actually reclaim disk space

* Added backward compat for the old Merge() function

* Refactor indexing of keys to items (hints)

* Remove redundant TestOpenMerge

* Add unit test for Stats()

* Improve TestMerge()
2019-07-27 07:52:25 +10:00
panyun
fd2023ee38 Fix issue(db file Merge issue in windows env): (#15)
* Fix issue(windows env):
1. Run my program firstly, it will create my.db folder normaly.
2. Close my program and restart it again, I will receive an error.

* Fix coding format issue

* Change the last fixing with @prologic suggestion.
2019-07-23 13:27:14 +10:00
James Mills
47ad6601f3
Update generated protobuf code 2019-07-19 20:49:49 +10:00
James Mills
53dc013215
Optimized and increased read performance by ~2-3x by memory mapping the read-only datafiles 2019-03-23 13:35:55 +10:00
James Mills
711d08ce91
Increased read performance by ~3-4x by removing another unnecessary I/O operation (Seek) 2019-03-23 12:14:15 +10:00
James Mills
c593bc966f
Refactor Datafile.Size() 2019-03-22 17:33:24 +10:00
James Mills
352c32ee12
Add Len() to exported API (extended API) 2019-03-21 10:47:50 +10:00
James Mills
aaea7273c3
Add Keys() to exported API (extended API) 2019-03-21 10:41:56 +10:00
James Mills
1298240f53
Unexport some internal implemtnation details 2019-03-18 17:31:31 +10:00