HybridHashMap

Hash table/map with open-addressing and hybrid storage, storing keys of type K and values of type V. Setting V to void turns the map into a set.

Keys are immutable except for when they are classes in which case they are head-const (through bin reinterpretation to KeyValueType), This can be overridden by setting keyEqualPred to, for instance, a == b for class keys.

Uses quadratic probing (using triangular numbers) unless usePrimeCapacity in which case a simpler probing is used.

Deletion/Removal of elements is lazy via the sentinel bitmap _store.holesPtr or through assignment of reserved value of KeyType when KeyType has hole/tombstone-sentinel-support via trait isHoleable. The terms "tombstone" and "sentinel" are used at, for instance, https://engineering.fb.com/2019/04/25/developer-tools/f14/ and https://www.youtube.com/watch?v=ncHmEUmJZf4&t=2837s.

Element iteration via - either byKey, byValue or byKeyValue over HybridHashMap and - byElement over HybridHashSet respects taking the container argument either as an l-value or r-value using detected using auto ref-qualified parameter introspected using (__traits(isRef, y)). In the r-value case no reference counting is needed. In the l-value case setting borrowChecked to true adds run-time support for dynamic Rust-style ownership and borrowing between the range and the container.

@safe
struct HybridHashMap (
K
V = void
alias hasher = hashOf
string keyEqualPred = defaultKeyEqualPredOf!(K)
Allocator = Mallocator
Options options = Options.init
) if (
isNullable!K &&
isAllocator!Allocator
) {}

Constructors

this
this(T element)

Make with the element element.

Destructor

~this
~this()

Destruct.

Postblit

this(this)
this(this)

No copying.

Members

Aliases

ElementType
alias ElementType = T
Undocumented in source.
KeyType
alias KeyType = K

Type of key stored.

Nullifier
alias Nullifier = typeof(K.init.nullifier)
Undocumented in source.
StoreK
alias StoreK = void*
Undocumented in source.
StoreK
alias StoreK = void*
Undocumented in source.
StoreK
alias StoreK = K
Undocumented in source.
T
alias T = K
Undocumented in source.
ValueType
alias ValueType = V

Type of value stored.

count
alias count = length

Get element count.

keyEqualPredFn
alias keyEqualPredFn = binaryFun!keyEqualPred
Undocumented in source.
keyEqualPredFn
alias keyEqualPredFn = keyEqualPred
Undocumented in source.

Enums

InsertionStatus
enum InsertionStatus

Map insertion status.

InsertionStatus
enum InsertionStatus

Set insertion status.

Functions

averageProbeCount
double averageProbeCount()
clear
void clear()

Empty.

contains
bool contains(SomeKey key)

Check if element is stored.

containsUsingLinearSearch
bool containsUsingLinearSearch(SomeKey key)

Check if element is stored.

containsWithHoleMoving
bool containsWithHoleMoving(K key)

Check if element is stored. Move found element to a hole if possible.

dup
typeof(this) dup()
get
inout(V) get(K key, inout(V) defaultValue)

Get value of key or defaultValue if key not present (and therefore nothrow).

getKeyRef
const(K) getKeyRef(SomeKey key, const(K) defaultKey)

Get reference to key-part of stored element at key, if present, otherwise return defaultKey.

insert
InsertionStatus insert(T element)

Insert element, being either a key-value (map-case) or a just a key (set-case).

insert
InsertionStatus insert(K key, V value)

Insert or replace value at key.

insertAndReturnElement
T insertAndReturnElement(SomeElement element)

Insert element, being either a key-value (map-case) or a just a key (set-case).

insertN
void insertN(R elements)

Insert elements, all being either a key-value (map-case) or a just a key (set-case).

opBinaryRight
const(K)* opBinaryRight(SomeKey key)
Undocumented in source. Be warned that the author may not have intended to support it.
opBinaryRight
inout(V)* opBinaryRight(SomeKey key)
Undocumented in source. Be warned that the author may not have intended to support it.
opEquals
bool opEquals(typeof(this) rhs)

Equality.

opIndex
inout(V) opIndex(SomeKey key)

Indexing.

opIndexAssign
V opIndexAssign(V value, K key)

Supports the syntax aa[key] = value;.

opIndexOpAssign
V opIndexOpAssign(Rhs rhs, K key)
Undocumented in source. Be warned that the author may not have intended to support it.
opOpAssign
typeof(this) opOpAssign(SomeKey key)
Undocumented in source. Be warned that the author may not have intended to support it.
rawStore
inout(T)[] rawStore()

Unsafe access to raw store.

rehashingRemoveN
size_t rehashingRemoveN(Keys keys)

Remove all elements matching keys followed by a rehash.

remove
bool remove(SomeKey key)

Remove element.

reserveExtra
void reserveExtra(size_t extraCapacity)

Reserve room for extraCapacity number of extra buckets.

totalProbeCount
size_t totalProbeCount()
tryGetElementFromCtorParams
const(Class) tryGetElementFromCtorParams(Params params)

Try to retrieve class-element of type Class constructed with parameters params.

Manifest constants

growScaleP
enum growScaleP;

Numerator for grow scale.

growScaleQ
enum growScaleQ;

Denominator for grow scale.

hasAddressLikeKey
enum hasAddressLikeKey;

Is true iff K is an address, in which case holes/tombstones are represented by a specific value holeKeyConstant.

hasHoleableKey
enum hasHoleableKey;
Undocumented in source.
hasHoleableKey
enum hasHoleableKey;
Undocumented in source.
hasHoleableKey
enum hasHoleableKey;
Undocumented in source.
hasHoleableKey
enum hasHoleableKey;
Undocumented in source.
hasHoleableKey
enum hasHoleableKey;
Undocumented in source.
hasNullableKey
enum hasNullableKey;
Undocumented in source.
hasValue
enum hasValue;

In the hash map case, V is non-void, and a value is stored alongside the key of type K.

holeKeyAddress
enum holeKeyAddress;
Undocumented in source.
holeKeyOffset
enum holeKeyOffset;
Undocumented in source.
isBorrowChecked
enum isBorrowChecked;

Is true iff this is borrow-checked.

nullKeyElement
enum nullKeyElement;
Undocumented in source.
nullKeyElement
enum nullKeyElement;
Undocumented in source.
usePrimeCapacity
enum usePrimeCapacity;
Undocumented in source.

Properties

binCount
size_t binCount [@property getter]

Get bin count (capacity).

empty
bool empty [@property getter]

Check if empty.

length
size_t length [@property getter]

Get length.

Static functions

holeKeyConstant
K holeKeyConstant()
holeKeyConstant
K holeKeyConstant()
Undocumented in source.
holeKeyConstant
K holeKeyConstant()
Undocumented in source. Be warned that the author may not have intended to support it.
isHoleKeyConstant
bool isHoleKeyConstant(K key)
isHoleKeyConstant
bool isHoleKeyConstant(K key)
Undocumented in source. Be warned that the author may not have intended to support it.
isHoleKeyConstant
bool isHoleKeyConstant(K key)
Undocumented in source.
isOccupiedBin
bool isOccupiedBin(T bin)
Undocumented in source. Be warned that the author may not have intended to support it.
isValidKey
bool isValidKey(SomeKey key)

Is true if key is valid.

keyOf
inout(K) keyOf(inout(SomeElement) element)

Get key part of element.

keyOf
inout(K) keyOf(inout(SomeElement) element)

Element type.

keyOf
inout(K) keyOf(inout(KeyValueType) element)

Get key part.

keyOf
inout(SomeElement) keyOf(inout(SomeElement) element)

Get key part of element.

valueOf
inout(V) valueOf(inout(T) element)

Get value part of element.

valueOf
inout(V) valueOf(inout(T) element)

Element type.

withCapacity
typeof(this) withCapacity(size_t minimumCapacity)

Make with room for storing at least minimumCapacity number of elements.

withElements
typeof(this) withElements(R elements)

Make with the elements elements. TODO: Replace with makeWithElements.

Static variables

borrowedErrorMessage
auto borrowedErrorMessage;
Undocumented in source.

Structs

T
struct T

Mutable element reference with mutable constant key and value.

Templates

isScopedKeyType
template isScopedKeyType(SomeKey)

Is true if an instance of SomeKey that can be implictly cast to K.

Variables

assumeNonFullHaystack
enum bool assumeNonFullHaystack;

Setting this true doesn't give measurable speedups so set it to false for now.

growInPlaceFlag
enum bool growInPlaceFlag;

Is true iff in-place rehashing during growth should be performed.

Parameters

K

key type

V

value type

hasher

hash function or std.digest Hash

Allocator

memory allocator for bin array

See Also

https://engineering.fb.com/2019/04/25/developer-tools/f14/

https://github.com/abseil/abseil-cpp/blob/master/absl/container/flat_hash_map.h

https://www.sebastiansylvan.com/post/robin-hood-hashing-should-be-your-default-hash-table-implementation/

https://arxiv.org/abs/1605.04031

https://github.com/Tessil/robin-map

https://github.com/martinus/robin-hood-hashing

https://probablydance.com/2017/02/26/i-wrote-the-fastest-hashtable/

https://en.wikipedia.org/wiki/Lazy_deletion

https://forum.dlang.org/post/ejqhcsvdyyqtntkgzgae@forum.dlang.org https://gankro.github.io/blah/hashbrown-insert/

Test: dmd -version=show -preview=dip1000 -preview=in -vcolumns -preview=fieldwise -debug -g -unittest -checkaction=context -allinst -main -unittest -I../.. -i -run hybrid_hashmap.d

TODO: Pass nullValue and holeValue explicitly as members of Options instead of relying on traits isNullable. The search and remove usages of isNullable, nullValue and holeValue (enum) members.

TODO: Support set/map union via ~ and ~= operator. See https://issues.dlang.org/show_bug.cgi?id=15682.

TODO: Support non-nullable keys via extra bit-set _nullsPtr similar to _store.holesPtr TODO: Don’t allocate _store.holesPtr when we don’t need removal such as for compiler (dmd) symbol tables. Use GrowOnlyFlag. TODO: Factor out array store into container/store.d

TODO: Add support for assignment from AA that calls withElements

TODO: Add test for real class types as keys to test/container and test also with normal AA

TODO: Add class type erasure layer to void* for the relevant members of HashMap store and factor out that store to container/store.d

TODO: Make sure key classes and pointers are shifted down by alignement as is done in AlignedAddress.toHash before hashed

TODO: Replace withCapacity with void capacity(size_t) like D arrays.

TODO: Tests fails when linearSearchMaxSize is set to 0

TODO: A test fails with when -preview=fieldwise is not passed.

TODO: Add support for opApply by copying opApply members in ~/.dub/packages/emsi_containers-0.8.0/emsi_containers/src/containers/hashmap.d

TODO: Implement foreach (const k, const v; _) support. See https://forum.dlang.org/post/tj700o$1412$1@digitalmars.com

TODO: Disable pragma(inline, true) and rebenchmark

TODO: Robin-hood case introspect key type for storage of parts of hash alongside nullValue and holeValue. Typically for address-types that doesn't need scanning. Example is Address for implementation of GC. This is a D showcase for code that is difficult to write in C++.

TODO: group nxt.probing functions in Prober struct given as type template param to HybridHashMap

TODO: Make load factor dependent on current capacity or length and perhaps also type and hash-function to get memory efficiency when it matters. Similar to what is recommended in https://ticki.github.io/blog/horrible/.

TODO: For copyable types replace auto ref with logic that only passes by ref when it's faster to do so. See_Also: https://github.com/dlang/dmd/pull/11000. When -preview=in has been made the default this complexity can be removed. Or replace ref with in.

TODO: Use mmap allocator when _store.elts.sizeof is larger than at least 8 pages

TODO: Use StoreK in store and cast between it and KeyType

TODO: Fix bug in growInPlaceWithCapacity and benchmark

TODO: Modify existing unittest for struct Rel { const string name; }

TODO: Use allocator.dispose() instead of allocator.deallocate() as in https://github.com/dlang-community/containers

TODO: if hash-function is cast(size_t)(classInstance) always use prime length and shift pointer before hash based on alignof (might not be needed when module prime) to maximize memory locality when adding successively allocated pointers

TODO: Add extractElement that moves it out similar to http://en.cppreference.com/w/cpp/container/unordered_set/extract

TODO: Add merge or union algorithm here or into container/common.d. See also: http://en.cppreference.com/w/cpp/container/unordered_set/merge. this algorithm moves elements from source if they are not already in this

TODO: Robin-Hood-hashing

TODO: Enable borrowChecked unconditionally in version (debug) if and when opPostMove is implemented, in which case opPostMove() should assert false if this is borrowed. See: https://github.com/dlang/DIPs/pull/109

TODO: Save one word by making _store.elts.length be inferred by prime_modulo.primeConstants[_primeIndex] if this is not too costly.

TODO: Only add one extra element to capacity when assumeNonFullHaystack is true

TODO: Remove use of static if (__traits(isCopyable, ...)) and `static if (__traits(isPOD, ...))` in cases where compiler can handle more moves

Meta