Mirrors/prologic-msgbus
6
0
Fork 0
mirror of https://git.mills.io/prologic/msgbus.git synced 2024-06-25 16:28:57 +00:00
Code Issues Projects Releases Wiki Activity
185 Commits 3 Branches 22 Tags 18 MiB
006c37a7c2
Commit Graph

49 Commits

Author SHA1 Message Date
James Mills
7d7e34e753
Fix panic updating metrics when not enabled 2022-04-05 16:09:34 +10:00
James Mills
7f1a30c6b0 Fix a bug in client not tracking topic index when using indexing (#36) 
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/36
 2022-04-04 02:36:56 +00:00
James Mills
e7f0248805 Set a unique client id per subscriber connection (#35) 
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/35
 2022-04-04 01:56:56 +00:00
James Mills
01ab56f9b3 Fix Subscribe() deadlock (#34) 
Fixes #139

This was caused by full subscriber buffers.

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/34
 2022-04-04 01:15:49 +00:00
James Mills
6bfb669347 Add support for a write-ahead-log (WAL) to persist messages (#33) 
Closes #31

Adds support for a write-ahead-log (WAL) for messages per queue/topic. This is now the new default behaviour and adds a new CLI flag `-l/--log-path` and Env var `LOG_PATH` to configure where the logs are stored.

On startup, the message bus will refill the queues with the contents of messages from persisted log files with the most recent `-Q/--max-queue-size` number of items.

That is, on startup/crash the queues/topics will always contain the same messages as if the message bus had never restarted or crashed in the first place.

This has a benefit of actually making the per-topic sequence number _actually_ monotic increasing integers and something that can be relied upon when indexing into a queue/topic for subscribers with the `-i/--index` / `Index` option.

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/33
 2022-04-03 15:59:38 +00:00
James Mills
429361c85c Fix case for invalid Index to handle Topic resets (msgbus restarts or crashes) (#30) 
Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/30
 2022-04-02 22:23:01 +00:00
James Mills
14b5443f50 Fix panic on nil message (#29) 
Fixed a bug I found in `saltyd`:

```
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 2022/04/02 21:49:31 http: panic serving 10.0.2.82:50050: interface conversion: interface {} is nil, not msgbus.Message
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | goroutine 1839 [running]:
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | net/http.(*conn).serve.func1()
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:1825 +0xbf
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | panic({0xc8c2c0, 0xc000095e60})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/runtime/panic.go:844 +0x258
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | git.mills.io/prologic/msgbus.(*MessageBus).Subscribe.func1({0x0?, 0x0?})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/git.mills.io/prologic/msgbus@v0.1.13/msgbus.go:487 +0x205
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | git.mills.io/prologic/msgbus.(*Queue).ForEach(0xc00015abe0, 0xc00043b020)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/git.mills.io/prologic/msgbus@v0.1.13/queue.go:126 +0x110
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | git.mills.io/prologic/msgbus.(*MessageBus).Subscribe(0xc0004fa000, {0xc0003088f0, 0xf}, {0xc0003aae05, 0x20}, {0xc0005330a8, 0x1, 0x40d987?})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/git.mills.io/prologic/msgbus@v0.1.13/msgbus.go:486 +0x5ad
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | git.mills.io/prologic/msgbus.(*Client).Start(0xc0000ac300)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/git.mills.io/prologic/msgbus@v0.1.13/msgbus.go:742 +0xb5
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | git.mills.io/prologic/msgbus.(*MessageBus).ServeHTTP(0xc0004fa000, {0x2b1ea08, 0xc0003b4000}, 0xc0002a8200)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/git.mills.io/prologic/msgbus@v0.1.13/msgbus.go:617 +0x758
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | go.mills.io/saltyim/internal.(*Server).initRoutes.func1({0x2b1ea08?, 0xc0003b4000?}, 0xd8732a?, {0x1d?, 0x0?, 0x0?})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/src/internal/handlers.go:53 +0x37
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | go.mills.io/saltyim/internal.NewServer.func1.1({0x2b1ea08, 0xc0003b4000}, 0xc0003aa200?, {0xc0000ca160, 0x1, 0x1})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/src/internal/server.go:388 +0x26d
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | github.com/julienschmidt/httprouter.(*Router).ServeHTTP(0xc00024e900, {0x2b1ea08, 0xc0003b4000}, 0xc0002a8200)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/github.com/julienschmidt/httprouter@v1.3.0/router.go:387 +0x82b
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | go.mills.io/saltyim/internal.(*Router).ServeHTTP(0xc44960?, {0x2b1ea08?, 0xc0003b4000?}, 0x4?)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/src/internal/router.go:144 +0x25
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | github.com/justinas/nosurf.(*CSRFHandler).handleSuccess(...)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/github.com/justinas/nosurf@v1.1.1/handler.go:187
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | github.com/justinas/nosurf.(*CSRFHandler).ServeHTTP(0xc0002245a0, {0x2b1ea08, 0xc0003b4000}, 0x4?)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/github.com/justinas/nosurf@v1.1.1/handler.go:144 +0x5c4
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | github.com/NYTimes/gziphandler.GzipHandlerWithOpts.func1.1({0x2b1f0f8, 0xc0000ca060}, 0x4c4801?)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/github.com/!n!y!times/gziphandler@v1.1.1/gzip.go:338 +0x262
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | net/http.HandlerFunc.ServeHTTP(0x2?, {0x2b1f0f8?, 0xc0000ca060?}, 0xc0003d7380?)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:2084 +0x2f
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | github.com/unrolled/logger.(*Logger).Handler.func1({0x2b1f7e8?, 0xc0001980e0}, 0xc0002a8100)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/go/pkg/mod/github.com/unrolled/logger@v0.0.0-20201216141554-31a3694fe979/logger.go:80 +0xf4
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | net/http.HandlerFunc.ServeHTTP(0xc0003aa226?, {0x2b1f7e8?, 0xc0001980e0?}, 0xc000188400?)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:2084 +0x2f
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | net/http.serverHandler.ServeHTTP({0xc00007c210?}, {0x2b1f7e8, 0xc0001980e0}, 0xc0002a8100)
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:2916 +0x43b
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | net/http.(*conn).serve(0xc000790000, {0x2b20228, 0xc0002ab830})
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:1966 +0x5d7
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | created by net/http.(*Server).Serve
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | 	/usr/local/go/src/net/http/server.go:3071 +0x4db
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | time="2022-04-02T21:49:52Z" level=debug msg="pong latency of 10.0.2.82:49826: 1.671603ms"
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | time="2022-04-02T21:50:20Z" level=info msg="synced store"
saltyim_saltyd.1.w8s8tw903qnj@dm3.mills.io    | [saltyd] 2022/04/02 21:50:32 (172.69.42.134) "GET /avatar/d3d52221e8da5a8ae012f4e2db0631c181f4156f0edcde5cffa25b347c7ceda8 HTTP/1.1" 304 0 108.078µs
```

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/29
 2022-04-02 21:59:10 +00:00
James Mills
7b71102aa8 Add support for subscribers to start from an index (#26) 
Closes #20

Co-authored-by: James Mills <prologic@shortcircuit.net.au>
Co-authored-by: xuu <xuu@noreply@mills.io>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/26
 2022-04-02 14:05:15 +00:00
James Mills
929cfd3a36
Fix response for empty topic/queue by responding with HTTP 204 No Content instead of 404 Not Found (Fixes #25) 2022-03-31 20:55:03 +10:00
xuu
7181b6df1b feat: Add Content-Encoding support for brotli, gzip, and deflate (#24) 
feat: Add Content-Encoding support for brotli, gzip, and deflate
Co-authored-by: Jon Lundy <jon@xuu.cc>
Reviewed-on: https://git.mills.io/prologic/msgbus/pulls/24
Co-authored-by: xuu <xuu@noreply@mills.io>
Co-committed-by: xuu <xuu@noreply@mills.io>
 2022-03-29 22:03:38 +00:00
James Mills
6a840a26b4
Revert "change gorilla/websocket to nhooyr.io/websocket" 
This reverts commit ec8dbec63232c1c78881a19a18e11f4dd9b63935.
 2022-03-27 12:42:56 +10:00
nilsocket
ec8dbec632
change gorilla/websocket to nhooyr.io/websocket 2022-03-26 23:15:34 +05:30
James Mills
09d2e16ed2
Add Access-Control-Allow-Origin: * to all responses 2022-03-25 11:27:23 +10:00
James Mills
d6d0a1e95b
Cleanup logging 2022-03-21 02:12:31 +10:00
James Mills
c42b2c5462
Fix more data races 2022-03-21 01:10:02 +10:00
James Mills
84e7cb8a95
Code cleanup 2022-03-20 22:28:49 +10:00
James Mills
8d19831a86
Migrate to Drone CI (cloud), CodeCov and Go11Modules 2018-12-31 18:05:23 +10:00
James Mills
ca70888049
Adjusted some defaults 2018-05-19 19:44:41 -07:00
James Mills
c640130f11
Improve API to be more RESTful 2018-05-14 03:04:45 -07:00
James Mills
81e35b8e18
Fixes several performance issues found in profiling and load testing. (#12) 
* Fixed drop rate for subscriber listeners by adding buffering to channels (configurable)

* Added optinoal pprof support

* Added profile target for running profiled benchmarks
 2018-05-14 02:12:58 -07:00
James Mills
60dd187cdd
Improves test coverage (#7) 
* Added GET / (empty) test

* Added POST /hello test

* Added PUT /hello test

* Added simple e2e test (POST & GET)

* Add Subscriber test (fixed some race conditions)
 2018-05-10 02:46:25 -07:00
James Mills
dd992ab5d8
Added configurable bounded queues with a deque data structure with added metrics (#6) 
Adds configurable bounded queues with a deque data structure with added metrics
 2018-05-09 23:25:13 -07:00
Alexander Emelin
22efa3afc8 use New instead of NewMessageBus for main message bus constructor (#5) 
* use New instead of NewMessageBus for main message bus constructor

* update readme: New as message bus constructor
 2018-05-08 01:50:56 -07:00
James Mills
8da5d41e65
Fixed subscribers gauge 2018-05-08 00:18:34 -07:00
James Mills
6ce5fb7104
Don't restrict size of messages at the websocket layer 2018-05-08 00:15:23 -07:00
James Mills
f169926a0c
Add metrics for queue depths 2018-05-07 21:51:54 -07:00
James Mills
0215f6f9a8
Fix debug logging 2018-05-06 14:04:51 -07:00
James Mills
1e0200788a
Add extra logging and fix a concurrency bug 2018-05-06 09:34:52 -07:00
James Mills
fba29f436b
Fixed leaking Subscriber(s) on unexpected closure from clients 2018-05-05 14:27:25 -07:00
James Mills
29053c2314
Improved metrics 2018-05-05 01:29:08 -07:00
James Mills
850ebb6f5f
Don't block on full subscriber channels (dead?) 2018-05-03 11:04:15 -07:00
James Mills
32f92b201f
Add support for control messages and various client improvements 2018-05-03 00:57:52 -07:00
James Mills
4eec89138b
Add extra counters for fetched, delivered and dropped messages 2018-05-02 12:50:12 -07:00
James Mills
3dbba632c7
Fixed tests 2018-05-02 01:24:30 -07:00
James Mills
f20b6cd8b9
Added basic metrics 2018-05-02 00:41:14 -07:00
James Mills
4b954ba4b4
Switched to using the github.com/gorilla/websocket library 2018-05-01 23:19:04 -07:00
James Mills
07699203ab
Add feature to msgbus sub CLI to executre commands on messages for convenient shell scriptiong 2018-04-30 23:09:12 -07:00
James Mills
931120defd
Various error handling improvements 2018-04-06 01:27:25 -07:00
James Mills
85f61cdddd
Guard against concurrent map access to topics 2018-04-02 14:38:33 -07:00
James Mills
f1ed5bd63c
Minor cleanup 2018-03-29 10:44:02 -07:00
James Mills
2aa716aaab
Switch logging to logrus 2018-03-25 17:03:56 -07:00
James Mills
baf9ecbf55
Set application/json content-types for most responses 2018-03-25 16:27:39 -07:00
James Mills
f546c64e24
Comment out logging 2018-03-03 19:58:58 +00:00
James Mills
dc7243ffa9
Per topic sequences with ttl 2017-08-14 00:34:12 -07:00
James Mills
215e226f69
Simplify a lot of the code and wrap up MessageBus into a Middleware that plays with standard net/http 2017-08-09 02:54:11 -07:00
James Mills
a56c04cbbe
Remove ack crap (rethining this) 2017-08-07 01:05:47 -07:00
James Mills
c76ef2f61d
Client wrapper and reconnecting client 
See #2
 2017-08-07 01:00:35 -07:00
James Mills
d4e65d0c45
Various improvements in conjunction with https://github.com/prologic/autodock/tree/go-port 2017-08-06 16:31:32 -07:00
James Mills
537969e9f2
Initial Commit 2017-06-03 16:16:17 +01:00