Hermes
Yet another concurrent, lightweight, fast and efficient In-Memory Key-Value Store alternative.
This was a sub-component of a pet IR system in Go.
I thought this is helpful enough and can stand on its own and can help alleviate disk I/O operations (database, file, etc.) in a quite-not-so-small-but-single-machine kind of systems.
What makes Hermes unique is its use of the ff:
- LRFU cache (which makes for a swift transition between LRU and LFU).
- Bloom filter implementation (Cuckoo Filter, to allow for item deletion in every eviction done by the policy) to avoid caching one-hit-wonders (disabled by default, please see the toml file to enable it) for memory efficiency.
- HAMT data structure (which makes it memory efficient at a nearly O(1) operation)
Use this in your app like so (as embedded):
package main
import (
"fmt"
"github.com/jtejido/hermes/hermes"
"github.com/jtejido/hermes/config"
"github.com/BurntSushi/toml"
"sync"
"strconv"
)
func main() {
var wg sync.WaitGroup
var ctx hermes.Context
var con config.Config
if _, err := toml.DecodeFile("src/github.com/jtejido/hermes/config.toml", &con); err != nil {
fmt.Println(err)
return
}
c := hermes.NewCache(&con)
for i := 0; i <= 100000; i++ {
wg.Add(1)
go func(i int) {
c.Set(strconv.Itoa(i), []byte(strconv.Itoa(i)))
v, err := c.Get(ctx, strconv.Itoa(i))
if err == nil {
fmt.Printf("key: %s, value: %s is still here\n", strconv.Itoa(i), v)
} else {
fmt.Printf("%d not found\n", i)
}
wg.Done()
}(i)
wg.Wait()
}
}
It depends on a .toml file for its setting (thus the need for a 3rd party dependency), you can roll your own if you wish, as long as you can marshall/unmarshall it to the right config type (see config/config.go for these types).
It was supposed to be a complete app, after modifying the default configuration, and can be run to accept keys and values via REST API right away.
If you happened to have gotten this via go get github.com/jtejido/hermes, it already created a compiled binary on your designated bin folder.
Type this for the options:
./hermes -h
The commands for running it is as follows:
-lambda float
Lambda used for LRFU. (default 0.65)
-filter
Bloom Filter enabled?. (default false)
-filter-items uint
Maximum number of items to be stored in filter. (default 1000000)
-logfile string
Location of the logfile. (default "access.log")
-maxmemory int
Maximum amount of data in the cache in MB. (default 256)
-port int
The public port to use. (default 9090) --take note that apps should use this
-address int
The port for peers to listen on. (default 10050)
-shards int
Number of shards for the cache. (default 1024)
-version
Hermes version.
Aside from the above options, and checking the .toml file, you should be aware that if you're planning to use it as a distributed cache, there is a setting in .toml file called peers, which, by default, the address should be part of this array; if setting up another instance of hermes on a different address, that address should also be included here (of course that instance should have a different address, e.g. 10051), which means you can add as many as you wish, as long as they're all added on the peers array.
Before using it (using any languages), you should know that the value it stores is in []byte, which means you'll have to do the serialization/deserialization on your end (any key-value stores does that).
To put something in, use curl to send PUT request:
curl -v -XPUT localhost:9090/hermes/api/v1/cache/example3 -d "yey"
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9090 (#0)
> PUT /hermes/api/v1/cache/example3 HTTP/1.1
> Host: localhost:9090
> User-Agent: curl/7.43.0
> Accept: */*
> Content-Length: 4
> Content-Type: application/x-www-form-urlencoded
>
* upload completely sent off: 4 out of 4 bytes
< HTTP/1.1 201 Created
< Date: Sat, 27 Oct 2018 21:38:52 GMT
< Content-Length: 0
<
* Connection #0 to host localhost left intact
An access.log file will be created in the same folder.
To get the items, you'll have do it via GET:
curl -v -XGET localhost:9090/hermes/api/v1/cache/example
* Trying 127.0.0.1...
* Connected to localhost (127.0.0.1) port 9090 (#0)
> GET /hermes/api/v1/cache/example HTTP/1.1
> Host: localhost:9090
> User-Agent: curl/7.43.0
> Accept: */*
>
< HTTP/1.1 200 OK
< Content-Type: application/json
< Date: Sat, 27 Oct 2018 21:43:12 GMT
< Content-Length: 16
<
* Connection #0 to host localhost left intact
{"Value":"yey"}
The values will be in json form (more stuff will be added soon), which means you'll decode it first before deserializing it back for your own app.
others are:
others are:
curl -v -XDELETE localhost:9090/hermes/api/v1/cache/example
curl -v -XGET localhost:9090/hermes/api/v1/stats // also json {hit, miss, collission, delhit and delmiss}
curl -v -XGET localhost:9090/hermes/api/v1/clear
If you wish to run it as a service or daemon, use daemonize, or if you can use takama's library, modify the source, and re-build it.