Repository navigation
use double hashing instead of linear probing - #7418
Conversation
Codecov Report❌ Patch coverage is
Additional details and impacted files@@ Coverage Diff @@
## truehash #7418 +/- ##
============================================
- Coverage 99.12% 99.08% -0.05%
============================================
Files 86 86
Lines 16665 16686 +21
============================================
+ Hits 16519 16533 +14
- Misses 146 153 +7 ☔ View full report in Codecov by Sentry. 🚀 New features to boost your workflow:
|
|
interesting atime result data! Thanks for sharing. Error in value[[3L]](cond) :
Error in revparse_single(object, branch): Error in 'git2r_revparse_single': Requested object could not be found
when trying to checkout 3178f10af7860223b7bea0fba519104d6a9d55cais |
|
atime CI failure can probably be fixed by merging master into this branch. |
The |
|
No obvious timing issues in HEAD=cuckoo_hash Generated via commit aba17d5 Download link for the artifact containing the test results: ↓ atime-results.zip
|
* Implement the hash table * memrecycle(): replace TRUELENGTH marks with a hash * rbindlist(): replace 1/2 TRUELENGTH with hashing Also avoid crashing when creating a 0-size hash. * rbindlist(): replace 2/2 TRUELENGTH with hashing This may likely require a dynamically growing hash of TRUELENGTHs instead of the current pre-allocation approach with a very conservative over-estimate. * chmatchMain(): replace TRUELENGTH marks with hash * copySharedColumns(): hash instead of TRUELENGTH * combineFactorLevels(): hash instead of TRUELENGTH * anySpecialStatic(): hash instead of TRUELENGTH * forder(): hash instead of TRUELENGTH The hash needs O(n) memory (actually 2*n/load_factor entries) which isn't great. * Remove savetl() * Add codecov suppressions * Dynamically grow the hash table with bound unknown In forder() and rbindlist(), there is no good upper boundary on the number of elements in the hash known ahead of time. Grow the hash table dynamically. Since the R/W locks are far too slow and OpenMP atomics are too limited, rely on strategically placed flushes, which isn't really a solution. * Minor hash improvements Use only 28 bits of the pointer (lower 32 but discard the lowest 4). Inline the linear search by advancing the pointer instead of repeatedly computing and dividing the hash value. Average improvement of 10%. * dhash: no need to keep previous table The hash can only be enlarged from (1) a single-thread context, or (2) under a critical section, so there is no need to worry about other threads getting a use-after-free due to a reallocation. This should halve the memory use by the hash table. * use double hashing instead of linear probing (#7418) * add lookup or insert * use lookup or insert * use lookup_or_insert * really use lookup or insert * use cuckoo hashing * add rehash * use power of 2 and mask instead of modulo * mix instead of multiplication * use different mixes * change multipliers * use double hashing * remove xor folding * Fix allocation non-overflow precondition * Set the default load factor * Inline hash_rehash() * update comments * Leave overflow checking to R_alloc * internal_error() is not covered --------- Co-authored-by: Ivan K <krylov.r00t@gmail.com> * replace dhashtab with hashtab in rbindlist * Use hashtab in forder() Since range_str() runs a parallel OpenMP loop that may update the hash table in a critical section, use a special form of hash_set that returns the newly reallocated hash table instead of overwriting it in place. * Drop dhashtab * rbindlist: initial hash size = upperBoundUniqueNames * Don't bother cleaning the hash before an error * Avoid setting same key&value twice * chmatch: hash x instead of table (#7454) * add hash x branch in chmatch * adapt kick-in threshold * make chin branch more explicit * Use linear probing instead of double hashing (#7455) * use linear probing instead of double hashing * remove mask from struct * fix comment * rbindlist(): better initial allocation size Also adjust the minimal hash table size to avoid shift overflow on size=0 and free=0 on size=1. * dogroups: replace remaining uses of SETLENGTH * hashtab: switch to C allocator to avoid longjumps PROTECT() the corresponding EXTPTRSXP while used. Introduce a separate hash_set_shared() operation that avoids long jumps. Deallocate the previous hash table when growing a non-shared hash table. * range_str: use hash_set_shared in OpenMP region Explicitly check the return value and update the shared pointer when necessary. If a reallocation attempt fails, signal an error when possible. * range_str: allocate temporaries on the R heap Avoid memory leaks from potential long jumps in hash_set(). * memrecycle: allocate temporary on the R heap This avoids a memory leak in case growVector or hash_set fails. * chmatchdup: allocate temporaries on the R heap Prevent memory leak in case hash_set() causes a long jump. * rbindlist: drop 'uniq' Turns out it wasn't used. * rbindlist: use hash_set_shared Where C-heap allocations also exist, catch and handle potential allocation failures from the hash table. * range_str: propagate rehashed marks to caller * fix stale comment * NEWS item Co-authored-by: Benjamin Schwendinger <benjaminschwe@gmail.com> Co-authored-by: HughParsonage <hugh.parsonage@gmail.com> Co-authored-by: Jan Gorecki <J.Gorecki@wit.edu.pl> * glci: update status expectations for R-devel * copyAsGrowable: use the correct number of arguments --------- Co-authored-by: Benjamin Schwendinger <benjaminschwe@gmail.com> Co-authored-by: Benjamin Schwendinger <52290390+ben-schwen@users.noreply.github.com> Co-authored-by: HughParsonage <hugh.parsonage@gmail.com> Co-authored-by: Jan Gorecki <J.Gorecki@wit.edu.pl>



So I implemented double hashing and adjusted the table size to the power of 2 (both suggestions from Ivan in #6694)
knuthis Ivans linear probing with knuth hashdoubleis the first try of double hashing where some mixing of XOR happens to spare multiplicationsnoxoris an easier variation without mixingkhashis using thekhash.hfromklibWhat really made (double and noxor) competitive with
masterandkhashwas probably parallelizing the probing inchmatchafter building the hashtable (khashalso uses this). What is left is to benchmarkchmatchdupwhich is probably slower and either needs adjustment of the hash table to also save a list of elements or at least cache the 3x lookups.Details