go-pkg/asynq
2
0
Fork 0
mirror of https://github.com/hibiken/asynq.git synced 2024-11-10 11:31:58 +08:00
Code Issues Projects Releases Wiki Activity
791 Commits 14 Branches 52 Tags 11 MiB
4c8432e0ce
Commit Graph

332 Commits

Author SHA1 Message Date
Ken Hibino
c5100a9c23 Add a method to list running servers to Inspector 2021-01-14 06:43:44 -08:00
Ken Hibino
196d66f221 Fix ListSchedulerEnqueueEvents to list recent events first 2021-01-14 06:43:44 -08:00
Ken Hibino
f4dd8fe962 Add ListScheduelerEnqueueEvents to Inspector 2021-01-14 06:43:44 -08:00
Ken Hibino
f9c0673116 Add SchedulerEntries method to Inspector 2021-01-14 06:43:44 -08:00
Ken Hibino
e9239260ae Add DeleteQueue method to Inspector 
- Added ErrQueueNotFound and ErrQueueNotEmpty type to indicate the kind
  of an error returned from the method.
 2021-01-14 06:43:44 -08:00
MinJae Kwon
c4dc993241
fix: resolve go vet lint 2020-12-20 06:09:51 -08:00
Ken Hibino
942345ee80 v0.13.0 2020-10-13 06:33:47 -07:00
Ken Hibino
96b2318300 Add EnqueueErrorHandler option to SchedulerOpts 2020-10-13 06:31:47 -07:00
Ken Hibino
8312515e64 Update Option interface 
- Added `String()`, `Type()`, and `Value()` methods to the interface to
  aid with debugging and error handling.
 2020-10-13 06:31:47 -07:00
Ken Hibino
50e7f38365 Add Scheduler 
- Renamed previously called scheduler to forwarder to resolve name
  conflicts
 2020-10-13 06:31:47 -07:00
Ken Hibino
c43fb21a0a Minor test updates 2020-09-12 12:59:03 -07:00
Ken Hibino
69d7ec725a Close redis client after each test run 2020-09-12 12:59:03 -07:00
Ken Hibino
c26b7469bd Display cluster info in stats command when --cluster flag is passed 2020-09-12 12:59:03 -07:00
Ken Hibino
29e542e591 Rename Enqueue methods in Inspector to Run 2020-09-12 12:59:03 -07:00
Ken Hibino
a891ce5568 Rename InProgress to Active 2020-09-12 12:59:03 -07:00
Ken Hibino
c8c47fcbf0 Rename Enqueued to Pending 2020-09-12 12:59:03 -07:00
Ken Hibino
8076b5ae50 Use different redis db number for rdb package tests 2020-09-12 12:59:03 -07:00
Ken Hibino
a88325cb96 Add ClusterNodes and ClusterKeySlot in Inspector 2020-09-12 12:59:03 -07:00
Ken Hibino
6b96459881 Add test flags to run tests using redis cluster 2020-09-12 12:59:03 -07:00
Ken Hibino
572eb338d5 Fix flaky ProcessorRetry test 2020-09-12 12:59:03 -07:00
Ken Hibino
27f4027447 Add RedisClusterClientOpt to connect to redis cluster 2020-09-12 12:59:03 -07:00
Ken Hibino
ee1afd12f5 Fix done lua script 
If UniqueKey is an empty string, do not provide the key to Lua script
because that will cause CROSSSLOT error in redis cluster (since it
doesn't have any hash tag).
 2020-09-12 12:59:03 -07:00
Ken Hibino
3ac548e97c Fix dequeue Lua script to use a single hash tag 2020-09-12 12:59:03 -07:00
Ken Hibino
f38f94b947 Restructure CLI commands with subcommands 2020-09-12 12:59:03 -07:00
Ken Hibino
118ef27bf2 Update RemoveQueue in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
96f23d88cd Add more processor tests 2020-09-12 12:59:03 -07:00
Ken Hibino
2f226dfb84 Update ListServers and ListWorkers methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
aa2676bb57 Update Broker interface 2020-09-12 12:59:03 -07:00
Ken Hibino
f59de9ac56 Update all delete methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
996a6c0ead Update all kill methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
47e9ba4eba Update enqueue methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
dbf140a767 Update all list methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
5f82b4b365 Update HistoricalStats method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
44a3d177f0 Update Pause and Unpause methods in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
24b13bd865 Update CurrentStats method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
d25090c669 Add AllQueues method to RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
b5caefd663 Remove stale benchmark test 2020-09-12 12:59:03 -07:00
Ken Hibino
becd26479b Update WriteServerState and ClearServerState in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
4b81b91d3e Minor fix 2020-09-12 12:59:03 -07:00
Ken Hibino
a873d488ee Update ListDeadlineExceeded in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
650d7fdbe9 Update CheckAndEnqueue method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
f6d504939e Update Requeue method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
74f08795f8 Update Kill method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
35b2b1782e Update Retry method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
f63dcce0c0 Update Done method in RDB 2020-09-12 12:59:03 -07:00
Ken Hibino
565f86ee4f Update Dequeue command in rdb 2020-09-12 12:59:03 -07:00
Ken Hibino
94aa878060 Update Enqueue and Schedule commands in rdb 2020-09-12 12:59:03 -07:00
Ken Hibino
50b6034bf9 Move unique key generator function to base 2020-09-12 12:59:03 -07:00
Ken Hibino
154113d0d0 Update base package to generate redis keys with hashtag 2020-09-12 12:59:03 -07:00
Ken Hibino
a913e6d73f Add healthchecker to check broker connection 2020-07-28 22:45:57 -07:00
Ken Hibino
92d77bbc6e Minor comment fix 2020-07-28 22:45:57 -07:00
Ken Hibino
a28f61f313 Add Inspector type 2020-07-28 22:45:57 -07:00
Ken Hibino
8b60e6a268 Replace github.com/rs/xid with github.com/google/uuid 2020-07-06 05:48:31 -07:00
Ken Hibino
486dcd799b Add version command to CLI 2020-07-06 05:48:31 -07:00
Ken Hibino
199bf4d66a Minor code cleanup 2020-07-06 05:48:31 -07:00
Ken Hibino
7e942ec241 Use int64 type for Timeout and Deadline in TaskMessage 2020-07-06 05:48:31 -07:00
Ken Hibino
feee87adda Add recoverer 2020-07-06 05:48:31 -07:00
Ken Hibino
7657f560ec Add RDB.ListDeadlineExceeded 2020-07-06 05:48:31 -07:00
Ken Hibino
7c7de0d8e0 Fix processor 2020-07-06 05:48:31 -07:00
Ken Hibino
4e8ac151ae Update processor to adapt for deadlines set change 
- Processor dequeues tasks only when it's available to process
- Processor retries a task when its context's Done channel is closed
 2020-07-06 05:48:31 -07:00
Ken Hibino
08b71672aa Update RDB.Requeue to remove message from deadlines set 2020-07-06 05:48:31 -07:00
Ken Hibino
92af00f9fd Update RDB.Dequeue to return deadline as time.Time 2020-07-06 05:48:31 -07:00
Ken Hibino
113451ce6a Update RDB.Kill to remove message from deadlines set 2020-07-06 05:48:31 -07:00
Ken Hibino
9cd9f3d6b4 Update RDB.Retry to remove message from deadlines set 2020-07-06 05:48:31 -07:00
Ken Hibino
7b9119c703 Update RDB.Done to remove message from deadlines set 2020-07-06 05:48:31 -07:00
Ken Hibino
9b05dea394 Update RDB.Dequeue to return message and deadline 2020-07-06 05:48:31 -07:00
Ken Hibino
6cc5bafaba Add task message to deadlines set on dequeue 
Updated dequeueCmd to decode the message and compute its deadline and add
the message to the Deadline set.
 2020-07-06 05:48:31 -07:00
Ken Hibino
0527b93432 Change TaskMessage Timeout and Deadline to int 
* This change breaks existing tasks in Redis
 2020-07-06 05:48:31 -07:00
Ken Hibino
5dddc35d7c Add redis key for deadlines in base package 2020-07-06 05:48:31 -07:00
Ken Hibino
7f30fa2bb6 Fix requeue logic in processor 2020-06-13 06:22:32 -07:00
Ken Hibino
a2abeedaa0 Fix JSON number ovewflow issue 2020-06-12 06:29:36 -07:00
Ken Hibino
06c4a1c7f8 Limit the number of tasks moved by CheckAndEnqueue to prevent a long 
running script
 2020-06-08 06:06:18 -07:00
Ken Hibino
4e800a7f68 Update stats command to show queue paused status 2020-06-08 06:06:18 -07:00
Ken Hibino
363cfedb49 Update Dequeue operation to skip paused queues 2020-06-08 06:06:18 -07:00
Ken Hibino
4595bd41c3 Add Pause and Unpause methods to rdb 2020-06-08 06:06:18 -07:00
Ken Hibino
a38f628f3b Refactor server state management 2020-05-31 06:41:19 -07:00
Ken Hibino
fa3082e5bb Change LogLevel to satisfy flag.Value interface 2020-05-16 08:00:35 -07:00
Ken Hibino
d13f7e900f Allow setting minimum log level for logger 2020-05-16 08:00:35 -07:00
Ken Hibino
b63476ddc8 Simplify Logger interface 2020-05-16 08:00:35 -07:00
Ken Hibino
5161b9368a Clean up tests 2020-05-02 07:05:26 -07:00
Ken Hibino
7c7f8e5f30 Move Broker interface to base package 2020-04-19 08:51:17 -07:00
Ken Hibino
46ab4417dd Add test to simulate situation where redis is down 2020-04-19 08:51:17 -07:00
Ken Hibino
c688b8f4f9 Fix test for base package 2020-04-19 08:51:17 -07:00
Ken Hibino
239ef27a6e Update doc comments 2020-04-19 08:51:17 -07:00
Ken Hibino
b086e88a47 Rename ps command to servers 2020-04-19 08:51:17 -07:00
Ken Hibino
aafd8a5b74 Rename internal ProcessState to ServerState 2020-04-19 08:51:17 -07:00
Ken Hibino
b14c73809e Refactor server state 2020-04-19 08:51:17 -07:00
Ken Hibino
9b4438347e Fix comment 2020-03-21 11:44:26 -07:00
Ken Hibino
c33dd447ac Allow client to enqueue a task with unique option 
Changes:

- Added Unique option for clients
- Require go v.13 or above (to use new errors wrapping functions)
- Fixed adding queue key to all-queues set (asynq:queues) when scheduling.
 2020-03-21 11:40:40 -07:00
Ken Hibino
8d2b9d6be7 Add comments to exported types and functions from internal/log package 2020-03-13 21:04:45 -07:00
Bo-Yi Wu
30b68728d4 chore(lint): fix from gofmt -s 2020-03-13 20:01:39 -07:00
Ken Hibino
0bc6eba021 Allow custom logger to be used in Background 2020-03-12 08:40:37 -07:00
Ken Hibino
d664d68fa4 Extract out log package 2020-03-09 07:17:52 -07:00
Ken Hibino
a425f54d23 [ci skip] Remove todo comment 2020-03-09 06:09:07 -07:00
Ken Hibino
3c722386b0 Add Deadline option when enqueuing tasks 
Deadline option sets the deadline for the given task's context deadline.
 2020-03-08 17:12:42 -07:00
Ken Hibino
26e399bc9c Add workers command to asynqmon 2020-02-22 21:21:55 -08:00
Ken Hibino
e21fe3bd79 Add ListWorkers to RDB 2020-02-22 21:21:55 -08:00
Ken Hibino
37c6c73d9b Add test for cancelation pubsub 2020-02-22 15:16:24 -08:00
Ken Hibino
5775a5818d Update WriteProcessState to write both ProcessInfo and WorkerInfo to 
redis
 2020-02-22 08:56:51 -08:00
Ken Hibino
e3c9aae8c7 Include worker stats to ProcessState 2020-02-22 08:56:51 -08:00
Ken Hibino
9e02a91808 Add workers key in base package 2020-02-22 08:56:51 -08:00
Ken Hibino
830020eb39 Add ProcessState type to base package 2020-02-22 08:56:51 -08:00
Ken Hibino
2bcaea52ce Make heartbeater goroutine a monitor goroutine for process info. 2020-02-16 18:22:59 -08:00
Ken Hibino
d33ca98648 Use int as priority value type. 
Do not use unsigned int for merely non-negative quantities as it
complicates simple arithmetic.
 2020-02-13 06:23:51 -08:00
Ken Hibino
cd55a0e316 Add cancel command to asynqmon cli 2020-02-13 06:23:51 -08:00
Ken Hibino
6685827147 Add redis pubsub subscriber for cancelation 2020-02-13 06:23:51 -08:00
Ken Hibino
39459b4412 Allow user to specify timeout per task 2020-02-13 06:23:51 -08:00
Ken Hibino
d9327cf24b Avoid creating script struct on every rdb method invocation 2020-02-09 09:34:41 -08:00
Ken Hibino
96f06ac89b Add ps command to asynqmon 2020-02-02 20:56:33 -08:00
Ken Hibino
d03fa34eaf Add hearbeater 2020-02-01 09:35:49 -08:00
Ken Hibino
6fbaa2ed6c (fix): RestoreUnfinished to select correct queue 2020-01-26 16:05:46 -08:00
Ken Hibino
166497748b (fix): Requeue to select correct queue 2020-01-26 16:05:46 -08:00
Ken Hibino
31123fd42a Paginate tasks with asynqmon ls command 
Changes:
* Added --page and --size flags to ls command
* By default, the command will show first 30 tasks from the specified
queue
 2020-01-26 13:12:01 -08:00
Ken Hibino
fd8eb51440 [ci skip] Add badges to readme 2020-01-16 21:13:34 -08:00
Ken Hibino
cb2ebf18ac [performance] Skip the overhead of json decoding when scheduling to one 
queue
 2020-01-14 20:46:47 -08:00
Ken Hibino
5a6f737589 [performance] Use BRPOPLPUSH if one queue is used 2020-01-14 20:46:47 -08:00
Ken Hibino
858b0325bd Add rmq command to asynqmon 2020-01-14 06:02:00 -08:00
Ken Hibino
874d8e8843 Add RDB.RemoveQueue method 2020-01-14 06:02:00 -08:00
Ken Hibino
97316d6766 Fix flaky tests 
Some tests were failing due to mismatch in Score in ZSetEntry.
Changed ZSetEntry Score to float64 type so that we can use
cmpopts.EquateApprox to allow for margin when comparing.
 2020-01-11 10:09:15 -08:00
Ken Hibino
2631672575 Allow filtering results of asynqmon ls enqueued by providing queue 
name
 2020-01-11 10:09:15 -08:00
Ken Hibino
cf78a12866 Add "Queue" column to the output of asynqmon ls 2020-01-11 10:09:15 -08:00
Ken Hibino
c5b215e3b9 Update RDB.ListEnqueued to list tasks from all queues 2020-01-11 10:09:15 -08:00
Ken Hibino
89843ac565 Change RDB.CurrentStats to be multi-queue aware 2020-01-11 10:09:15 -08:00
Ken Hibino
67f381269a Maintain a set of queue names in redis set 2020-01-11 10:09:15 -08:00
Ken Hibino
390eb13149 Remove stale field in processor struct 2020-01-11 10:09:15 -08:00
Ken Hibino
718336ff44 Update RDB.Enqueue* methods to be multi queue aware 2020-01-11 10:09:15 -08:00
Ken Hibino
4f5d115b3e [ci skip] Upgrade github.com/google/go-cmp to v0.4.0 2020-01-07 21:55:18 -08:00
Ken Hibino
53d0902808 Change RDB.Dequeue to query multiple queues 2020-01-07 21:55:18 -08:00
Ken Hibino
2af9eb2c88 Delete stale code 2020-01-07 21:55:18 -08:00
Ken Hibino
28d698c24e Update CheckAndEnqueue to enqueue tasks to specified queue 2020-01-07 21:55:18 -08:00
Ken Hibino
ca78b92078 Add Queue option to allow user to specify queue from client 
Added base.QueueKey method to get redis key for given queue name.
Changed asynqtest.GetEnqueuedMessages to optionally take queue name.
 2020-01-07 21:55:18 -08:00
Ken Hibino
f3a23b9b12 Make Task type immutable 
This change makes it impossible to mutate payload within Handler or
RetryDelayFunc.
 2020-01-05 09:55:39 -08:00
Ken Hibino
294a695b98 Add history command to asynqmon tool 2020-01-04 12:22:00 -08:00
Ken Hibino
c62833540c Add license comment to all src files 2020-01-02 18:13:16 -08:00
Ken Hibino
4376dc1e9d Add benchmark tests for rdb 2019-12-31 08:24:03 -08:00
Ken Hibino
b7c0c5d3aa Handle mutated task in RDB's Done, Retry, Kill methods 
It is possible that user mutates the task's payload in Handler
(Although doc says the task in handler is read-only). Prevent
ending up in an inconsistent state by handling the case where
user mutates the task.
 2019-12-31 08:24:03 -08:00
Ken Hibino
ae0c2f9ca5 Use asynqtest helpers in asynq package tests 2019-12-29 10:05:02 -08:00
Ken Hibino
807729d36a Create internal asynqtest package for test helpers 2019-12-29 10:05:02 -08:00
Ken Hibino
49d6ab5df0 Clean up rdb package 2019-12-29 10:05:02 -08:00
Ken Hibino
4ceb49cfd1 Minor improvements 2019-12-28 06:41:39 -08:00
Ken Hibino
8d3d30da8f Add KillAllRetryTasks and KillAllScheduledTasks method to RDB 2019-12-27 15:17:45 -08:00
Ken Hibino
5b98b8eb62 Add KillRetryTask, KillScheduledTask methods to RDB 2019-12-27 15:17:45 -08:00
Ken Hibino
fb24d158ae fix: CurrentState without processed/failed data 2019-12-25 21:30:21 -08:00
Ken Hibino
cde9d41580 Add processed and failed count to Stats 2019-12-25 20:17:00 -08:00
Ken Hibino
698fdcfcb2 fix: Change max dead tasks count to 100 2019-12-25 19:57:23 -08:00
Ken Hibino
b2161a481d Record processed and failure daily count 2019-12-24 20:05:46 -08:00
Ken Hibino
eea919d117 Add RedisInfo method to RDB 2019-12-22 09:09:57 -08:00
Ken Hibino
3fd248615b Create base internal package 2019-12-22 07:40:25 -08:00
Ken Hibino
33e9da953d Log warning and info messages when unfinished tasks get aborted 2019-12-18 18:55:08 -08:00
Ken Hibino
69b46a7f0d Use sync.Once 2019-12-17 20:34:56 -08:00
Ken Hibino
5ddba8ca98 Unblock processor shutdown process if processor is waiting for semaphore 
token
 2019-12-17 20:07:17 -08:00
Ken Hibino
911e600c41 Terminate background upon receiving SIGTERM or SIGINT 2019-12-16 20:19:58 -08:00
Ken Hibino
e5686894d3 Fix: Do not use lua cjson library to encode task to json 
Go and Lua json libraries encodes json differently (e.g. order of
key/value) and caused a bug when removing tasks that was previously
encoded by Lua json library and redis was receiving a string generated
by Go json library.
 2019-12-15 20:05:56 -08:00
Ken Hibino
442b33a6d2 Remove (*RDB).RetryLater in favor of Retry method 2019-12-15 17:18:43 -08:00
Ken Hibino
d84e8c0ff2 Modify (*RDB).Kill method to atomically move task from in_progress to 
dead queue
 2019-12-15 17:16:13 -08:00
Ken Hibino
1b1662bb12 Add Retry method to *RDB 
(*RDB).Retry method takes a TaskMessage and will atomically moves the
message from in_progress queue to retry queue. Additionally it
increments the Retried counter and assigns the error message to the
message.
 2019-12-15 16:15:07 -08:00
Ken Hibino
04836194f9 Clean up rdb package tests 2019-12-14 13:43:06 -08:00
Ken Hibino
9410a65338 Rename randomTask test helper to newTaskMessage 2019-12-12 19:45:03 -08:00
Ken Hibino
ca31cbf843 Add DeleteAll* methods to rdb 2019-12-12 06:43:42 -08:00
Ken Hibino
be746d01a5 Add delete methods to rdb 2019-12-11 20:15:30 -08:00
Ken Hibino
edcd70b221 Replace google/uuid package with rs/xid for more compact id 2019-12-11 07:41:38 -08:00
Ken Hibino
ef562e2efa Minor improvement 2019-12-10 21:48:19 -08:00
Ken Hibino
a96719413c Add enqall command to asynqmon CLI 2019-12-10 21:40:28 -08:00
Ken Hibino
0d74c518bf Add methods to rdb to enqueue all tasks from dead, retry and scheduled 
queue
 2019-12-10 20:28:31 -08:00
Ken Hibino
ea28d3cac1 Minor improvement 2019-12-09 20:37:30 -08:00
Ken Hibino
8830d23388 Rename rdb methods to enqueue scheduled, retry, and dead tasks 2019-12-09 19:33:07 -08:00
Ken Hibino
4179c72c05 Add enq command to asynqmon 2019-12-09 06:18:25 -08:00
Ken Hibino
bbd8fb901f [ci skip] Fix typos 2019-12-08 14:17:57 -08:00
Ken Hibino
680a2cf3df Add methods to rdb.RDB to enqueues a task from scheduled, retry, dead 
queues
 2019-12-08 14:12:02 -08:00
Ken Hibino
c1d0497182 Remove debug log 2019-12-06 22:30:11 -08:00
Ken Hibino
f3b9745228 Add tests for rdb list methods 2019-12-05 17:13:37 -08:00
Ken Hibino
437fb03bb3 Unexport redis key name constants from rdb package 2019-12-04 17:23:11 -08:00
Ken Hibino
918f33d37d Implement all "list tasks" methods in rdb 2019-12-04 17:10:39 -08:00
Ken Hibino
afacc31990 Minor improvements 2019-12-04 16:58:07 -08:00
Ken Hibino
998e761660 Use different redis db for each package tests 2019-12-04 07:39:16 -08:00
Ken Hibino
28bfb6d83a Rename to CheckAndEnqueue 2019-12-04 07:28:57 -08:00
Ken Hibino
4684f961c0 Refactor forwarding of scheduled tasks 2019-12-04 07:14:37 -08:00
Ken Hibino
318b24b3b8 Rename MoveAll to RestoreUnfinished 2019-12-04 06:50:52 -08:00
Ken Hibino
4531e90b9d Define Schedule and RetryLater method for RDB 2019-12-04 06:45:30 -08:00
Ken Hibino
985018e1b5 Rename rdb Remove to Done 2019-12-04 06:33:05 -08:00
Ken Hibino
39f177dabf Change rdb Dequeue signature 2019-12-04 06:25:58 -08:00
Ken Hibino
d4e442d04f Extract rdb to internal package 2019-12-03 22:13:48 -08:00