Pior Bastida
3dbda60333
Improve performance of enqueueing tasks ( #946 )
...
* Improve performance of enqueueing tasks
Add an in-memory cache to keep track of all the queues. Use this cache
to avoid sending an SADD since after the first call, that extra network
call isn't necessary.
The cache will expire every 10 secs so for cases where the queue is
deleted from asynq:queues set, it can be added again next time a task is
enqueued to it.
* Use sync.Map to simplify the conditional SADD
* Cleanup queuePublished in RemoveQueue
---------
Co-authored-by: Yousif <753751+yousifh@users.noreply.github.com>
2024-10-30 08:25:35 +03:00
guoguangwu
e4b8663154
chore: unnecessary use of fmt.Sprintf
2023-07-07 20:45:42 -07:00
Emanuel Bennici
0275df8df4
Update redis/go-redis to v9
...
Version v9 implements the support for Redis v7 and has some
other improvements.
2023-04-17 22:30:33 -07:00
Erwan Leboucher
5c723f597e
Correct the error message to cancel an active tasks
2022-04-13 06:08:46 -07:00
Ken Hibino
de139cc18e
Update RDB.RunTask to schedule aggregating task
2022-04-11 16:55:43 -07:00
Ken Hibino
74db013ab9
Add RDB.RunAllAggregatingTasks
2022-04-11 16:55:43 -07:00
Ken Hibino
725105ca03
Update RDB.ArchiveTask to archive aggregating task
2022-04-11 16:55:43 -07:00
Ken Hibino
d8f31e45f1
Add RDB.ArchiveAllAggregatingTasks
2022-04-11 16:55:43 -07:00
Ken Hibino
9023cbf4be
Update RDB.DeleteTask to handle aggregating task
2022-04-11 16:55:43 -07:00
Ken Hibino
9279c09125
Add RDB.DeleteAllAggregatingTasks
2022-04-11 16:55:43 -07:00
Ken Hibino
bc27126670
Fix memory usage lua script
2022-04-11 16:55:43 -07:00
Ken Hibino
0cfa7f47ba
Fix memory_usage lua script
2022-04-11 16:55:43 -07:00
Ken Hibino
71bd8f0535
Add RDB.ListAggregating
2022-04-11 16:55:43 -07:00
Ken Hibino
0149396bae
Add RDB.GroupStats for inspecting groups
2022-04-11 16:55:43 -07:00
Ken Hibino
47af17cfb4
Fix RDB.CurrentStats to report the correct queue size
2022-04-11 16:55:43 -07:00
Ken Hibino
652939dd3a
Update memory usage redis lua script to account for groups
2022-04-11 16:55:43 -07:00
Ken Hibino
efe3c74037
Show number of groups and aggregating task count in QueueInfo
2022-04-11 16:55:43 -07:00
Ken Hibino
d4006894ad
Remove base.DeadlinesKey
2022-02-19 06:15:44 -08:00
Ken Hibino
aa26f3819e
Fix flaky tests
2022-01-05 09:07:42 -08:00
Ken Hibino
82d18e3d91
Record total tasks processed/failed
2021-12-16 16:53:02 -08:00
Ken Hibino
99a6750656
Add Latency field to QueueInfo
2021-12-11 10:29:43 -08:00
Ken Hibino
6e7106c8f2
Record time when task moved to pending state
2021-12-11 10:29:43 -08:00
Ken Hibino
f4ddac4dcc
Introduce Task Results
...
* Added Retention Option to specify retention TTL for tasks
* Added ResultWriter as a client interface to write result data for the associated task
2021-11-06 15:18:42 -07:00
Ken Hibino
2261c7c9a0
Change TaskMessage.ID type from uuid.UUID to string
2021-11-06 15:18:42 -07:00
Jason White
b3ef9e91a9
Upgrade go-redis/redis to version 8
2021-09-02 05:56:02 -07:00
Ken Hibino
e6355bf3f5
Use approximate memory usage for QueueInfo
2021-08-09 08:58:44 -07:00
Ken Hibino
99c7ebeef2
Add migration command in CLI
2021-06-29 16:34:21 -07:00
Ken Hibino
e01c6379c8
Fix lua script for redis-cluster mode
2021-06-29 16:34:21 -07:00
Ken Hibino
68dd6d9a9d
(fix): Clear unique lock when task is deleted via Inspector
2021-06-29 16:34:21 -07:00
Ken Hibino
68839dc9d3
Fix lua scripts for redis cluster
2021-06-29 16:34:21 -07:00
Ken Hibino
8922d2423a
Define RDB.GetTaskInfo
2021-06-29 16:34:21 -07:00
Ken Hibino
b9c2572203
Refactor redis keys and store messages in protobuf
...
Changes:
- Task messages are stored under "asynq:{<qname>}:t:<task_id>" key in redis, value is a HASH type and message are stored under "msg" key in the hash. The hash also stores "deadline", "timeout".
- Redis LIST and ZSET stores task message IDs
- Task messages are serialized using protocol buffer
2021-06-29 16:34:21 -07:00
Ken Hibino
ffe9aa74b3
Add errors.RedisCommandError type
2021-06-29 16:34:21 -07:00
Ken Hibino
d2d4029aba
Update RDB.CurrentStats and RDB.HistoricalStats with specific errors
2021-06-29 16:34:21 -07:00
Ken Hibino
76bd865ebc
Update RDB.RemoveQueue with specific error types
2021-06-29 16:34:21 -07:00
Ken Hibino
136d1c9ea9
Update rdb.List* methods with specific errors
2021-06-29 16:34:21 -07:00
Ken Hibino
52e04355d3
Return QueueNotFoundError from DeleteAll* methods
2021-06-29 16:34:21 -07:00
Ken Hibino
cde3e57c6c
Update RDB.RunAll* methods with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
dd66acef1b
Return QueueNotFoundError from ArchiveAll* methods
2021-06-29 16:34:21 -07:00
Ken Hibino
30a3d9641a
Update tests for RDB.DeleteTask and RDB.ArchiveTask
2021-06-29 16:34:21 -07:00
Ken Hibino
961582cba6
Update RDB.RunTask with more specific errors
2021-06-29 16:34:21 -07:00
Ken Hibino
430dbb298e
Update RDB.DeleteTask with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
675826be5f
Update RDB.ArchiveAll methods with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
62f4e46b73
Update RDB.ArchiveAllPendingTasks with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
12a90f6a8d
Update RDB.ArchiveTask with custom errors
2021-06-29 16:34:21 -07:00
Ken Hibino
4d65024bd7
Update rdb.ArchiveTask with more specific error types
2021-06-29 16:34:21 -07:00
Ken Hibino
76486b5cb4
Rename error types
2021-06-29 16:34:21 -07:00
Ken Hibino
cb5bdf245c
Update RDB.ArchiveTask with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
267493ccef
Update RDB.RunTask with task state
2021-06-29 16:34:21 -07:00
Ken Hibino
7af3981929
Refactor redis keys and store messages in protobuf
...
Changes:
- Task messages are stored under "asynq:{<qname>}:t:<task_id>" key in redis, value is a HASH type and message are stored under "msg" key in the hash. The hash also stores "deadline", "timeout".
- Redis LIST and ZSET stores task message IDs
- Task messages are serialized using protocol buffer
2021-06-29 16:34:21 -07:00
