go-pkg/asynq
2
0
Fork 0
mirror of https://github.com/hibiken/asynq.git synced 2024-11-15 03:48:44 +08:00
Code Issues Projects Releases Wiki Activity
908 Commits 12 Branches 52 Tags 11 MiB
3dbda60333
Commit Graph

137 Commits

Author SHA1 Message Date
Pior Bastida
3dbda60333
Improve performance of enqueueing tasks (#946) 
* Improve performance of enqueueing tasks

Add an in-memory cache to keep track of all the queues. Use this cache
to avoid sending an SADD since after the first call, that extra network
call isn't necessary.

The cache will expire every 10 secs so for cases where the queue is
deleted from asynq:queues set, it can be added again next time a task is
enqueued to it.

* Use sync.Map to simplify the conditional SADD

* Cleanup queuePublished in RemoveQueue

---------

Co-authored-by: Yousif <753751+yousifh@users.noreply.github.com>
 2024-10-30 08:25:35 +03:00
Pior Bastida
03f4799712
Run golangci-lint in CI (#927) 
* Setup golangci-lint in CI and local-dev

* Fix linting error or locally disable linter
 2024-10-26 08:48:12 +03:00
Harrison Miller
0dc670d7d8
Archived tasks that are trimmed from the set are deleted (#743) 
* fixed trimmed archive tasks not being deleted.

* improved test case.

* changed ZRANGEBYSCORE to ZRANGE with BYSCORE option.

---------

Co-authored-by: Harrison <harrison@Harrisons-MacBook-Pro.local>
Co-authored-by: Harrison Miller <harrison.miller@MBP-Harrison-Miller-M2.local>
 2024-10-19 09:18:09 +03:00
Tedja
d04888e748
feature: configurable janitor interval and deletion batch size (#715) 
* feature: configurable janitor interval and deletion batch size

* warn user when they set a big number of janitor batch size

* Update CHANGELOG.md

---------

Co-authored-by: Agung Hariadi Tedja <agung.tedja@kumparan.com>
 2024-05-06 14:11:52 +08:00
Phước Trung
c72bfef094 fix unit test 
Signed-off-by: Mohammed Sohail <sohailsameja@gmail.com>
 2023-04-17 22:30:33 -07:00
Emanuel Bennici
0275df8df4 Update redis/go-redis to v9 
Version v9 implements the support for Redis v7 and has some
other improvements.
 2023-04-17 22:30:33 -07:00
Ken Hibino
e939b5d166 Rename asynqtest package to testutil 2022-04-11 16:55:43 -07:00
Ken Hibino
1acd62c760 Move test helpers to asynqtest package 2022-04-11 16:55:43 -07:00
Ken Hibino
eb064c2bab Fix AggregationCheck with unlimited size to clear group name from 
all-groups set
 2022-04-11 16:55:43 -07:00
Ken Hibino
efe3c74037 Show number of groups and aggregating task count in QueueInfo 2022-04-11 16:55:43 -07:00
Ken Hibino
74d2eea4e0 Clear group if aggregation set empties the group 2022-04-11 16:55:43 -07:00
Ken Hibino
60a4dc1401 Add test for DeleteAggregationSet error case 2022-04-11 16:55:43 -07:00
Ken Hibino
4b716780ef Rewrite test for DeleteAggregationSet function with a new pattern 2022-04-11 16:55:43 -07:00
Ken Hibino
e63f41fb24 Fix DeleteAggregationSet 2022-04-11 16:55:43 -07:00
Ken Hibino
1c388baf06 Implement RDB.ReclaimStaleAggregationSets 2022-04-11 16:55:43 -07:00
Ken Hibino
3551d3334c Use zset for aggregation set to preserve score 2022-04-11 16:55:43 -07:00
Ken Hibino
8b16ede8bc Declare ReclaimStaleAggregationSets 2022-04-11 16:55:43 -07:00
Ken Hibino
888b5590fb Make GroupMaxSize and GroupMaxDelay config optional 2022-04-11 16:55:43 -07:00
Ken Hibino
4b35eb0e1a Fix RDB.AggregationCheck when run against an empty group 2022-04-11 16:55:43 -07:00
Ken Hibino
b29fe58434 Implement RDB.ListGroups 2022-04-11 16:55:43 -07:00
Ken Hibino
7849b1114c Implement RDB.DeleteAggregationSet 2022-04-11 16:55:43 -07:00
Ken Hibino
99c00bffeb Implement RDB.AggregationCheck 2022-04-11 16:55:43 -07:00
Ken Hibino
ed69667e86 Update ForwardIfReady test with group 2022-04-11 16:55:43 -07:00
Ken Hibino
61854ea1dc Update RDB.ForwardIfReady to forward to group if groupKey is specified 2022-04-11 16:55:43 -07:00
Ken Hibino
8b582899ad Add RDB.AddToGroup and RDB.AddToGroupUnique methods 2022-04-11 16:55:43 -07:00
Ken Hibino
7e5efb0e30 Drop GT option from RDB.ExtendLease 
GT option in ZAdd is supported for redis v6.2.0 or above.
This Change fixes redis version compatibility (currently v4.0+)
 2022-02-20 06:20:38 -08:00
Ken Hibino
59927509d8 Remove timeout and deadline fields under task key 2022-02-19 06:15:44 -08:00
Ken Hibino
dfae8638e1 Update RDB methods to work with lease 2022-02-19 06:15:44 -08:00
Ken Hibino
87dc392c7f Add RDB.ExtendLease method 2022-02-19 06:15:44 -08:00
Ken Hibino
bc2f1986d7 Update ListDeadlineExceeded to ListLeaseExpired 2022-02-19 06:15:44 -08:00
Ken Hibino
b8cb579407 Update RDB methods to use lease instead of deadlines set 2022-02-19 06:15:44 -08:00
Ken Hibino
d865d89900 Update RDB.Dequeue to insert task ID to lease set 2022-02-19 06:15:44 -08:00
Ken Hibino
aa26f3819e
Fix flaky tests 2022-01-05 09:07:42 -08:00
Ken Hibino
8dfabfccb3 Fix build 2021-12-19 07:06:37 -08:00
Ken Hibino
82d18e3d91
Record total tasks processed/failed 2021-12-16 16:53:02 -08:00
Ken Hibino
99a6750656 Add Latency field to QueueInfo 2021-12-11 10:29:43 -08:00
Ken Hibino
e7c1c3ad6f Use clock in RDB 2021-12-11 10:29:43 -08:00
Ken Hibino
6e7106c8f2 Record time when task moved to pending state 2021-12-11 10:29:43 -08:00
Ken Hibino
9f2c321e98
Add EnqueueContext method to Client 2021-11-15 16:34:26 -08:00
Ken Hibino
f4ddac4dcc Introduce Task Results 
* Added Retention Option to specify retention TTL for tasks
* Added ResultWriter as a client interface to write result data for the associated task
 2021-11-06 15:18:42 -07:00
Ken Hibino
4638405cbd Fix flaky test 2021-11-06 15:18:42 -07:00
Ken Hibino
dbdd9c6d5f Update RDB Enqueue and Schedule methods to check for task ID conflict 2021-11-06 15:18:42 -07:00
Ken Hibino
2261c7c9a0 Change TaskMessage.ID type from uuid.UUID to string 2021-11-06 15:18:42 -07:00
Jason White
b3ef9e91a9
Upgrade go-redis/redis to version 8 2021-09-02 05:56:02 -07:00
Ken Hibino
f0db219f6a
Add IsFailure to Config 
With this IsFailure config, users can provide a predicate function to 
determine whether the error returned from Handler counts as a failure.
 2021-09-01 06:00:54 -07:00
Ken Hibino
68dd6d9a9d (fix): Clear unique lock when task is deleted via Inspector 2021-06-29 16:34:21 -07:00
Ken Hibino
6cce31a134 Fix recoverer test 2021-06-29 16:34:21 -07:00
Ken Hibino
87264b66f3 Record last_failed_at time on Retry or Archive event 2021-06-29 16:34:21 -07:00
Ken Hibino
b9c2572203 Refactor redis keys and store messages in protobuf 
Changes:
- Task messages are stored under "asynq:{<qname>}:t:<task_id>" key in redis, value is a HASH type and message are stored under "msg" key in the hash. The hash also stores "deadline", "timeout".
- Redis LIST and ZSET stores task message IDs
- Task messages are serialized using protocol buffer
 2021-06-29 16:34:21 -07:00
Ken Hibino
a19ad19382 Update RDB.Dequeue with new errors package 2021-06-29 16:34:21 -07:00