mirror of
https://github.com/hibiken/asynq.git
synced 2025-09-17 20:30:06 +08:00
Compare commits
37 Commits
Author | SHA1 | Date | |
---|---|---|---|
![]() |
93ff4e4149 | ||
![]() |
3ee97f327e | ||
![]() |
49c117f4d1 | ||
![]() |
5b53a2aee9 | ||
![]() |
26e399bc9c | ||
![]() |
e21fe3bd79 | ||
![]() |
37c6c73d9b | ||
![]() |
5775a5818d | ||
![]() |
e3c9aae8c7 | ||
![]() |
9e02a91808 | ||
![]() |
830020eb39 | ||
![]() |
f9a6c6156f | ||
![]() |
8540172306 | ||
![]() |
b44908fefd | ||
![]() |
e78d5e4171 | ||
![]() |
0cc486a975 | ||
![]() |
906f231e6c | ||
![]() |
39f237899b | ||
![]() |
3d9a222bb3 | ||
![]() |
2bcaea52ce | ||
![]() |
0856ef32e0 | ||
![]() |
70efb7b455 | ||
![]() |
d33ca98648 | ||
![]() |
bf31fcc3ec | ||
![]() |
cd55a0e316 | ||
![]() |
6685827147 | ||
![]() |
133bb6c2c6 | ||
![]() |
39459b4412 | ||
![]() |
6e14062325 | ||
![]() |
1b41c721b4 | ||
![]() |
79a04e52a3 | ||
![]() |
d9327cf24b | ||
![]() |
7e33c9ebe0 | ||
![]() |
a6b79eb7a0 | ||
![]() |
7af69c8d3c | ||
![]() |
6dd4c688f4 | ||
![]() |
024cfa9fe6 |
@@ -5,7 +5,10 @@ git:
|
||||
env:
|
||||
- GO111MODULE=on # go modules are the default
|
||||
go: [1.12.x, 1.13.x]
|
||||
script:
|
||||
- go test -race -v -coverprofile=coverage.txt -covermode=atomic ./...
|
||||
services:
|
||||
- redis-server
|
||||
after_success:
|
||||
- bash ./.travis/benchcmp.sh
|
||||
- bash <(curl -s https://codecov.io/bash)
|
||||
|
22
CHANGELOG.md
22
CHANGELOG.md
@@ -7,6 +7,28 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
|
||||
|
||||
## [Unreleased]
|
||||
|
||||
## [0.5.0] - 2020-02-23
|
||||
|
||||
### Changed
|
||||
|
||||
- `Client` API has changed. Use `Enqueue`, `EnqueueAt` and `EnqueueIn` to enqueue and schedule tasks.
|
||||
|
||||
### Added
|
||||
|
||||
- `asynqmon workers` was added to list all running workers information
|
||||
|
||||
## [0.4.0] - 2020-02-13
|
||||
|
||||
### Changed
|
||||
|
||||
- `Handler` interface has changed. `ProcessTask` method takes two arguments `context.Context` and `*asynq.Task`
|
||||
- `Queues` field in `Config` has change from `map[string]uint` to `map[string]int`
|
||||
|
||||
### Added
|
||||
|
||||
- `Client` can optionally schedule task with `asynq.Timeout(duration)` to specify timeout duration for task. Default is no timeout.
|
||||
- `asynqmon cancel [task id]` will send a cancelation signal to the goroutine processing the speicified task.
|
||||
|
||||
## [0.3.0] - 2020-02-04
|
||||
|
||||
### Added
|
||||
|
51
CONTRIBUTING.md
Normal file
51
CONTRIBUTING.md
Normal file
@@ -0,0 +1,51 @@
|
||||
# Contributing
|
||||
|
||||
Thanks for your interest in contributing to Asynq!
|
||||
We are open to, and grateful for, any contributions made by the community.
|
||||
|
||||
## Reporting Bugs
|
||||
|
||||
Have a look at our [issue tracker](https://github.com/hibiken/asynq/issues). If you can't find an issue (open or closed)
|
||||
describing your problem (or a very similar one) there, please open a new issue with
|
||||
the following details:
|
||||
|
||||
- Which versions of Go and Redis are you using?
|
||||
- What are you trying to accomplish?
|
||||
- What is the full error you are seeing?
|
||||
- How can we reproduce this?
|
||||
- Please quote as much of your code as needed to reproduce (best link to a
|
||||
public repository or Gist)
|
||||
|
||||
## Getting Help
|
||||
|
||||
We run a [Gitter
|
||||
channel](https://gitter.im/go-asynq/community) where you can ask questions and
|
||||
get help. Feel free to ask there before opening a GitHub issue.
|
||||
|
||||
## Submitting Feature Requests
|
||||
|
||||
If you can't find an issue (open or closed) describing your idea on our [issue
|
||||
tracker](https://github.com/hibiken/asynq/issues), open an issue. Adding answers to the following
|
||||
questions in your description is +1:
|
||||
|
||||
- What do you want to do, and how do you expect Asynq to support you with that?
|
||||
- How might this be added to Asynq?
|
||||
- What are possible alternatives?
|
||||
- Are there any disadvantages?
|
||||
|
||||
Thank you! We'll try to respond as quickly as possible.
|
||||
|
||||
## Contributing Code
|
||||
|
||||
1. Fork this repo
|
||||
2. Download your fork `git clone https://github.com/your-username/asynq && cd asynq`
|
||||
3. Create your branch `git checkout -b your-branch-name`
|
||||
4. Make and commit your changes
|
||||
5. Push the branch `git push origin your-branch-name`
|
||||
6. Create a new pull request
|
||||
|
||||
Please try to keep your pull request focused in scope and avoid including unrelated commits.
|
||||
|
||||
After you have submitted your pull request, we'll try to get back to you as soon as possible. We may suggest some changes or improvements.
|
||||
|
||||
Thank you for contributing!
|
43
README.md
43
README.md
@@ -5,6 +5,7 @@
|
||||
[](https://goreportcard.com/report/github.com/hibiken/asynq)
|
||||
[](https://godoc.org/github.com/hibiken/asynq)
|
||||
[](https://gitter.im/go-asynq/community)
|
||||
[](https://codecov.io/gh/hibiken/asynq)
|
||||
|
||||
Asynq is a simple Go library for queueing tasks and processing them in the background with workers.
|
||||
It is backed by Redis and it is designed to have a low barrier to entry. It should be integrated in your web stack easily.
|
||||
@@ -21,12 +22,12 @@ To install `asynq` library, run the following command:
|
||||
go get -u github.com/hibiken/asynq
|
||||
```
|
||||
|
||||
## Quick Start
|
||||
## Quickstart
|
||||
|
||||
First, make sure you are running a Redis server locally.
|
||||
|
||||
```sh
|
||||
redis-server
|
||||
$ redis-server
|
||||
```
|
||||
|
||||
To create and schedule tasks, use `Client` and provide a task and when to process the task.
|
||||
@@ -34,7 +35,7 @@ To create and schedule tasks, use `Client` and provide a task and when to proces
|
||||
```go
|
||||
func main() {
|
||||
r := &asynq.RedisClientOpt{
|
||||
Addr: "localhost:6379",
|
||||
Addr: "127.0.0.1:6379",
|
||||
}
|
||||
|
||||
client := asynq.NewClient(r)
|
||||
@@ -45,16 +46,21 @@ func main() {
|
||||
t2 := asynq.NewTask("send_reminder_email", map[string]interface{}{"user_id": 42})
|
||||
|
||||
// Process immediately
|
||||
err := client.Schedule(t1, time.Now())
|
||||
err := client.Enqueue(t1)
|
||||
|
||||
// Process 24 hrs later
|
||||
err = client.Schedule(t2, time.Now().Add(24 * time.Hour))
|
||||
err = client.EnqueueIn(24*time.Hour, t2)
|
||||
|
||||
// If processing fails, retry up to 10 times (Default is 25)
|
||||
err = client.Schedule(t1, time.Now(), asynq.Retry(10))
|
||||
// Process at specified time.
|
||||
t := time.Date(2020, time.March, 6, 10, 0, 0, 0, time.UTC)
|
||||
err = client.EnqueueAt(t, t2)
|
||||
|
||||
// Use custom queue called "critical"
|
||||
err = client.Schedule(t1, time.Now(), asynq.Queue("critical"))
|
||||
// Pass options to specify processing behavior for a given task.
|
||||
//
|
||||
// MaxRetry specifies the maximum number of times this task will be retried (Default is 25).
|
||||
// Queue specifies which queue to enqueue this task to (Default is "default").
|
||||
// Timeout specifies the the timeout for the task's context (Default is no timeout).
|
||||
err = client.Enqueue(t1, asynq.MaxRetry(10), asynq.Queue("critical"), asynq.Timeout(time.Minute))
|
||||
}
|
||||
```
|
||||
|
||||
@@ -63,15 +69,14 @@ To start the background workers, use `Background` and provide your `Handler` to
|
||||
```go
|
||||
func main() {
|
||||
r := &asynq.RedisClientOpt{
|
||||
Addr: "localhost:6379",
|
||||
Addr: "127.0.0.1:6379",
|
||||
}
|
||||
|
||||
bg := asynq.NewBackground(r, &asynq.Config{
|
||||
// Specify how many concurrent workers to use
|
||||
Concurrency: 10,
|
||||
// You can optionally create multiple queues
|
||||
// with different priority level
|
||||
Queues: map[string]uint{
|
||||
// Optionally specify multiple queues with different priority.
|
||||
Queues: map[string]int{
|
||||
"critical": 6,
|
||||
"default": 3,
|
||||
"low": 1,
|
||||
@@ -92,7 +97,7 @@ func main() {
|
||||
// If ProcessTask return a non-nil error or panics, the task
|
||||
// will be retried after delay.
|
||||
type Handler interface {
|
||||
ProcessTask(*Task) error
|
||||
ProcessTask(context.Context, *asynq.Task) error
|
||||
}
|
||||
```
|
||||
|
||||
@@ -113,13 +118,21 @@ Asynq ships with a command line tool to inspect the state of queues and tasks.
|
||||
|
||||
To install, run the following command:
|
||||
|
||||
go get github.com/hibiken/asynq/tools/asynqmon
|
||||
```sh
|
||||
go get -u github.com/hibiken/asynq/tools/asynqmon
|
||||
```
|
||||
|
||||
For details on how to use the tool, refer to the tool's [README](/tools/asynqmon/README.md).
|
||||
|
||||
## Contributing
|
||||
|
||||
We are open to, and grateful for, any contributions (Github issues/pull-requests, feedback on Gitter channel, etc) made by the community.
|
||||
Please see the [Contribution Guide](/CONTRIBUTING.md) before contributing.
|
||||
|
||||
## Acknowledgements
|
||||
|
||||
- [Sidekiq](https://github.com/mperham/sidekiq) : Many of the design ideas are taken from sidekiq and its Web UI
|
||||
- [RQ](https://github.com/rq/rq) : Client APIs are inspired by rq library.
|
||||
- [Cobra](https://github.com/spf13/cobra) : Asynqmon CLI is built with cobra
|
||||
|
||||
## License
|
||||
|
24
asynq.go
24
asynq.go
@@ -11,7 +11,7 @@ import (
|
||||
"github.com/go-redis/redis/v7"
|
||||
)
|
||||
|
||||
// Task represents a task to be performed.
|
||||
// Task represents a unit of work to be performed.
|
||||
type Task struct {
|
||||
// Type indicates the type of task to be performed.
|
||||
Type string
|
||||
@@ -20,10 +20,9 @@ type Task struct {
|
||||
Payload Payload
|
||||
}
|
||||
|
||||
// NewTask returns a new Task. The typename and payload argument set Type
|
||||
// and Payload field respectively.
|
||||
// NewTask returns a new Task given a type name and payload data.
|
||||
//
|
||||
// The payload must be serializable to JSON.
|
||||
// The payload values must be serializable.
|
||||
func NewTask(typename string, payload map[string]interface{}) *Task {
|
||||
return &Task{
|
||||
Type: typename,
|
||||
@@ -31,13 +30,11 @@ func NewTask(typename string, payload map[string]interface{}) *Task {
|
||||
}
|
||||
}
|
||||
|
||||
// RedisConnOpt is a discriminated union of redis-client-option types.
|
||||
// RedisConnOpt is a discriminated union of types that represent Redis connection configuration option.
|
||||
//
|
||||
// RedisConnOpt represents a sum of following types:
|
||||
//
|
||||
// RedisClientOpt | *RedisClientOpt | RedisFailoverClientOpt | *RedisFailoverClientOpt
|
||||
//
|
||||
// Passing unexpected type to a RedisConnOpt variable can cause panic.
|
||||
type RedisConnOpt interface{}
|
||||
|
||||
// RedisClientOpt is used to create a redis client that connects
|
||||
@@ -53,7 +50,7 @@ type RedisClientOpt struct {
|
||||
// Redis server password.
|
||||
Password string
|
||||
|
||||
// Redis DB to select after connecting to the server.
|
||||
// Redis DB to select after connecting to a server.
|
||||
// See: https://redis.io/commands/select.
|
||||
DB int
|
||||
|
||||
@@ -61,13 +58,13 @@ type RedisClientOpt struct {
|
||||
// Default is 10 connections per every CPU as reported by runtime.NumCPU.
|
||||
PoolSize int
|
||||
|
||||
// TLS Config used to connect to the server.
|
||||
// TLS Config used to connect to a server.
|
||||
// TLS will be negotiated only if this field is set.
|
||||
TLSConfig *tls.Config
|
||||
}
|
||||
|
||||
// RedisFailoverClientOpt is used to creates a redis client that talks
|
||||
// to redis sentinels for service discovery and has automatic failover
|
||||
// to redis sentinels for service discovery and has an automatic failover
|
||||
// capability.
|
||||
type RedisFailoverClientOpt struct {
|
||||
// Redis master name that monitored by sentinels.
|
||||
@@ -84,7 +81,7 @@ type RedisFailoverClientOpt struct {
|
||||
// Redis server password.
|
||||
Password string
|
||||
|
||||
// Redis DB to select after connecting to the server.
|
||||
// Redis DB to select after connecting to a server.
|
||||
// See: https://redis.io/commands/select.
|
||||
DB int
|
||||
|
||||
@@ -92,11 +89,14 @@ type RedisFailoverClientOpt struct {
|
||||
// Default is 10 connections per every CPU as reported by runtime.NumCPU.
|
||||
PoolSize int
|
||||
|
||||
// TLS Config used to connect to the server.
|
||||
// TLS Config used to connect to a server.
|
||||
// TLS will be negotiated only if this field is set.
|
||||
TLSConfig *tls.Config
|
||||
}
|
||||
|
||||
// createRedisClient returns a redis client given a redis connection configuration.
|
||||
//
|
||||
// Passing an unexpected type as a RedisConnOpt argument will cause panic.
|
||||
func createRedisClient(r RedisConnOpt) *redis.Client {
|
||||
switch r := r.(type) {
|
||||
case *RedisClientOpt:
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math"
|
||||
"math/rand"
|
||||
@@ -33,19 +34,24 @@ type Background struct {
|
||||
mu sync.Mutex
|
||||
running bool
|
||||
|
||||
pinfo *base.ProcessInfo
|
||||
ps *base.ProcessState
|
||||
|
||||
// wait group to wait for all goroutines to finish.
|
||||
wg sync.WaitGroup
|
||||
|
||||
rdb *rdb.RDB
|
||||
scheduler *scheduler
|
||||
processor *processor
|
||||
syncer *syncer
|
||||
heartbeater *heartbeater
|
||||
subscriber *subscriber
|
||||
}
|
||||
|
||||
// Config specifies the background-task processing behavior.
|
||||
type Config struct {
|
||||
// Maximum number of concurrent processing of tasks.
|
||||
//
|
||||
// If set to zero or negative value, NewBackground will overwrite the value to one.
|
||||
// If set to a zero or negative value, NewBackground will overwrite the value to one.
|
||||
Concurrency int
|
||||
|
||||
// Function to calculate retry delay for a failed task.
|
||||
@@ -57,15 +63,15 @@ type Config struct {
|
||||
// t is the task in question.
|
||||
RetryDelayFunc func(n int, e error, t *Task) time.Duration
|
||||
|
||||
// List of queues to process with given priority level. Keys are the names of the
|
||||
// queues and values are associated priority level.
|
||||
// List of queues to process with given priority value. Keys are the names of the
|
||||
// queues and values are associated priority value.
|
||||
//
|
||||
// If set to nil or not specified, the background will process only the "default" queue.
|
||||
//
|
||||
// Priority is treated as follows to avoid starving low priority queues.
|
||||
//
|
||||
// Example:
|
||||
// Queues: map[string]uint{
|
||||
// Queues: map[string]int{
|
||||
// "critical": 6,
|
||||
// "default": 3,
|
||||
// "low": 1,
|
||||
@@ -73,7 +79,9 @@ type Config struct {
|
||||
// With the above config and given that all queues are not empty, the tasks
|
||||
// in "critical", "default", "low" should be processed 60%, 30%, 10% of
|
||||
// the time respectively.
|
||||
Queues map[string]uint
|
||||
//
|
||||
// If a queue has a zero or negative priority value, the queue will be ignored.
|
||||
Queues map[string]int
|
||||
|
||||
// StrictPriority indicates whether the queue priority should be treated strictly.
|
||||
//
|
||||
@@ -90,7 +98,7 @@ func defaultDelayFunc(n int, e error, t *Task) time.Duration {
|
||||
return time.Duration(s) * time.Second
|
||||
}
|
||||
|
||||
var defaultQueueConfig = map[string]uint{
|
||||
var defaultQueueConfig = map[string]int{
|
||||
base.DefaultQueueName: 1,
|
||||
}
|
||||
|
||||
@@ -105,8 +113,13 @@ func NewBackground(r RedisConnOpt, cfg *Config) *Background {
|
||||
if delayFunc == nil {
|
||||
delayFunc = defaultDelayFunc
|
||||
}
|
||||
queues := cfg.Queues
|
||||
if queues == nil || len(queues) == 0 {
|
||||
queues := make(map[string]int)
|
||||
for qname, p := range cfg.Queues {
|
||||
if p > 0 {
|
||||
queues[qname] = p
|
||||
}
|
||||
}
|
||||
if len(queues) == 0 {
|
||||
queues = defaultQueueConfig
|
||||
}
|
||||
|
||||
@@ -116,20 +129,23 @@ func NewBackground(r RedisConnOpt, cfg *Config) *Background {
|
||||
}
|
||||
pid := os.Getpid()
|
||||
|
||||
pinfo := base.NewProcessInfo(host, pid, n, queues, cfg.StrictPriority)
|
||||
rdb := rdb.NewRDB(createRedisClient(r))
|
||||
syncRequestCh := make(chan *syncRequest)
|
||||
syncer := newSyncer(syncRequestCh, 5*time.Second)
|
||||
heartbeater := newHeartbeater(rdb, pinfo, 5*time.Second)
|
||||
ps := base.NewProcessState(host, pid, n, queues, cfg.StrictPriority)
|
||||
syncCh := make(chan *syncRequest)
|
||||
cancels := base.NewCancelations()
|
||||
syncer := newSyncer(syncCh, 5*time.Second)
|
||||
heartbeater := newHeartbeater(rdb, ps, 5*time.Second)
|
||||
scheduler := newScheduler(rdb, 5*time.Second, queues)
|
||||
processor := newProcessor(rdb, pinfo, delayFunc, syncRequestCh)
|
||||
processor := newProcessor(rdb, ps, delayFunc, syncCh, cancels)
|
||||
subscriber := newSubscriber(rdb, cancels)
|
||||
return &Background{
|
||||
pinfo: pinfo,
|
||||
rdb: rdb,
|
||||
ps: ps,
|
||||
scheduler: scheduler,
|
||||
processor: processor,
|
||||
syncer: syncer,
|
||||
heartbeater: heartbeater,
|
||||
subscriber: subscriber,
|
||||
}
|
||||
}
|
||||
|
||||
@@ -141,18 +157,18 @@ func NewBackground(r RedisConnOpt, cfg *Config) *Background {
|
||||
// If ProcessTask return a non-nil error or panics, the task
|
||||
// will be retried after delay.
|
||||
type Handler interface {
|
||||
ProcessTask(*Task) error
|
||||
ProcessTask(context.Context, *Task) error
|
||||
}
|
||||
|
||||
// The HandlerFunc type is an adapter to allow the use of
|
||||
// ordinary functions as a Handler. If f is a function
|
||||
// with the appropriate signature, HandlerFunc(f) is a
|
||||
// Handler that calls f.
|
||||
type HandlerFunc func(*Task) error
|
||||
type HandlerFunc func(context.Context, *Task) error
|
||||
|
||||
// ProcessTask calls fn(task)
|
||||
func (fn HandlerFunc) ProcessTask(task *Task) error {
|
||||
return fn(task)
|
||||
// ProcessTask calls fn(ctx, task)
|
||||
func (fn HandlerFunc) ProcessTask(ctx context.Context, task *Task) error {
|
||||
return fn(ctx, task)
|
||||
}
|
||||
|
||||
// Run starts the background-task processing and blocks until
|
||||
@@ -176,7 +192,7 @@ func (bg *Background) Run(handler Handler) {
|
||||
sig := <-sigs
|
||||
if sig == syscall.SIGTSTP {
|
||||
bg.processor.stop()
|
||||
bg.pinfo.SetState("stopped")
|
||||
bg.ps.SetStatus(base.StatusStopped)
|
||||
continue
|
||||
}
|
||||
break
|
||||
@@ -196,10 +212,11 @@ func (bg *Background) start(handler Handler) {
|
||||
bg.running = true
|
||||
bg.processor.handler = handler
|
||||
|
||||
bg.heartbeater.start()
|
||||
bg.syncer.start()
|
||||
bg.scheduler.start()
|
||||
bg.processor.start()
|
||||
bg.heartbeater.start(&bg.wg)
|
||||
bg.subscriber.start(&bg.wg)
|
||||
bg.syncer.start(&bg.wg)
|
||||
bg.scheduler.start(&bg.wg)
|
||||
bg.processor.start(&bg.wg)
|
||||
}
|
||||
|
||||
// stops the background-task processing.
|
||||
@@ -210,16 +227,19 @@ func (bg *Background) stop() {
|
||||
return
|
||||
}
|
||||
|
||||
// Note: The order of termination is important.
|
||||
// Sender goroutines should be terminated before the receiver goroutines.
|
||||
//
|
||||
// processor -> syncer (via syncCh)
|
||||
bg.scheduler.terminate()
|
||||
bg.processor.terminate()
|
||||
// Note: processor and all worker goroutines need to be exited
|
||||
// before shutting down syncer to avoid goroutine leak.
|
||||
bg.syncer.terminate()
|
||||
bg.subscriber.terminate()
|
||||
bg.heartbeater.terminate()
|
||||
|
||||
bg.rdb.ClearProcessInfo(bg.pinfo)
|
||||
bg.wg.Wait()
|
||||
|
||||
bg.rdb.Close()
|
||||
bg.processor.handler = nil
|
||||
bg.running = false
|
||||
|
||||
logger.info("Bye!")
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -27,31 +28,37 @@ func TestBackground(t *testing.T) {
|
||||
})
|
||||
|
||||
// no-op handler
|
||||
h := func(task *Task) error {
|
||||
h := func(ctx context.Context, task *Task) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
bg.start(HandlerFunc(h))
|
||||
|
||||
client.Schedule(NewTask("send_email", map[string]interface{}{"recipient_id": 123}), time.Now())
|
||||
err := client.Enqueue(NewTask("send_email", map[string]interface{}{"recipient_id": 123}))
|
||||
if err != nil {
|
||||
t.Errorf("could not enqueue a task: %v", err)
|
||||
}
|
||||
|
||||
client.Schedule(NewTask("send_email", map[string]interface{}{"recipient_id": 456}), time.Now().Add(time.Hour))
|
||||
err = client.EnqueueAt(time.Now().Add(time.Hour), NewTask("send_email", map[string]interface{}{"recipient_id": 456}))
|
||||
if err != nil {
|
||||
t.Errorf("could not enqueue a task: %v", err)
|
||||
}
|
||||
|
||||
bg.stop()
|
||||
}
|
||||
|
||||
func TestGCD(t *testing.T) {
|
||||
tests := []struct {
|
||||
input []uint
|
||||
want uint
|
||||
input []int
|
||||
want int
|
||||
}{
|
||||
{[]uint{6, 2, 12}, 2},
|
||||
{[]uint{3, 3, 3}, 3},
|
||||
{[]uint{6, 3, 1}, 1},
|
||||
{[]uint{1}, 1},
|
||||
{[]uint{1, 0, 2}, 1},
|
||||
{[]uint{8, 0, 4}, 4},
|
||||
{[]uint{9, 12, 18, 30}, 3},
|
||||
{[]int{6, 2, 12}, 2},
|
||||
{[]int{3, 3, 3}, 3},
|
||||
{[]int{6, 3, 1}, 1},
|
||||
{[]int{1}, 1},
|
||||
{[]int{1, 0, 2}, 1},
|
||||
{[]int{8, 0, 4}, 4},
|
||||
{[]int{9, 12, 18, 30}, 3},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
@@ -64,46 +71,46 @@ func TestGCD(t *testing.T) {
|
||||
|
||||
func TestNormalizeQueueCfg(t *testing.T) {
|
||||
tests := []struct {
|
||||
input map[string]uint
|
||||
want map[string]uint
|
||||
input map[string]int
|
||||
want map[string]int
|
||||
}{
|
||||
{
|
||||
input: map[string]uint{
|
||||
input: map[string]int{
|
||||
"high": 100,
|
||||
"default": 20,
|
||||
"low": 5,
|
||||
},
|
||||
want: map[string]uint{
|
||||
want: map[string]int{
|
||||
"high": 20,
|
||||
"default": 4,
|
||||
"low": 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
input: map[string]uint{
|
||||
input: map[string]int{
|
||||
"default": 10,
|
||||
},
|
||||
want: map[string]uint{
|
||||
want: map[string]int{
|
||||
"default": 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
input: map[string]uint{
|
||||
input: map[string]int{
|
||||
"critical": 5,
|
||||
"default": 1,
|
||||
},
|
||||
want: map[string]uint{
|
||||
want: map[string]int{
|
||||
"critical": 5,
|
||||
"default": 1,
|
||||
},
|
||||
},
|
||||
{
|
||||
input: map[string]uint{
|
||||
input: map[string]int{
|
||||
"critical": 6,
|
||||
"default": 3,
|
||||
"low": 0,
|
||||
},
|
||||
want: map[string]uint{
|
||||
want: map[string]int{
|
||||
"critical": 2,
|
||||
"default": 1,
|
||||
"low": 0,
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"sync"
|
||||
@@ -12,8 +13,7 @@ import (
|
||||
"time"
|
||||
)
|
||||
|
||||
// Simple E2E Benchmark testing with no scheduled tasks and
|
||||
// no retries.
|
||||
// Simple E2E Benchmark testing with no scheduled tasks and retries.
|
||||
func BenchmarkEndToEndSimple(b *testing.B) {
|
||||
const count = 100000
|
||||
for n := 0; n < b.N; n++ {
|
||||
@@ -33,12 +33,14 @@ func BenchmarkEndToEndSimple(b *testing.B) {
|
||||
// Create a bunch of tasks
|
||||
for i := 0; i < count; i++ {
|
||||
t := NewTask(fmt.Sprintf("task%d", i), map[string]interface{}{"data": i})
|
||||
client.Schedule(t, time.Now())
|
||||
if err := client.Enqueue(t); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(count)
|
||||
handler := func(t *Task) error {
|
||||
handler := func(ctx context.Context, t *Task) error {
|
||||
wg.Done()
|
||||
return nil
|
||||
}
|
||||
@@ -74,16 +76,20 @@ func BenchmarkEndToEnd(b *testing.B) {
|
||||
// Create a bunch of tasks
|
||||
for i := 0; i < count; i++ {
|
||||
t := NewTask(fmt.Sprintf("task%d", i), map[string]interface{}{"data": i})
|
||||
client.Schedule(t, time.Now())
|
||||
if err := client.Enqueue(t); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
for i := 0; i < count; i++ {
|
||||
t := NewTask(fmt.Sprintf("scheduled%d", i), map[string]interface{}{"data": i})
|
||||
client.Schedule(t, time.Now().Add(time.Second))
|
||||
if err := client.EnqueueAt(time.Now().Add(time.Second), t); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(count * 2)
|
||||
handler := func(t *Task) error {
|
||||
handler := func(ctx context.Context, t *Task) error {
|
||||
// randomly fail 1% of tasks
|
||||
if rand.Intn(100) == 1 {
|
||||
return fmt.Errorf(":(")
|
||||
@@ -101,3 +107,64 @@ func BenchmarkEndToEnd(b *testing.B) {
|
||||
b.StartTimer() // end teardown
|
||||
}
|
||||
}
|
||||
|
||||
// Simple E2E Benchmark testing with no scheduled tasks and retries with multiple queues.
|
||||
func BenchmarkEndToEndMultipleQueues(b *testing.B) {
|
||||
// number of tasks to create for each queue
|
||||
const (
|
||||
highCount = 20000
|
||||
defaultCount = 20000
|
||||
lowCount = 20000
|
||||
)
|
||||
for n := 0; n < b.N; n++ {
|
||||
b.StopTimer() // begin setup
|
||||
setup(b)
|
||||
redis := &RedisClientOpt{
|
||||
Addr: redisAddr,
|
||||
DB: redisDB,
|
||||
}
|
||||
client := NewClient(redis)
|
||||
bg := NewBackground(redis, &Config{
|
||||
Concurrency: 10,
|
||||
Queues: map[string]int{
|
||||
"high": 6,
|
||||
"default": 3,
|
||||
"low": 1,
|
||||
},
|
||||
})
|
||||
// Create a bunch of tasks
|
||||
for i := 0; i < highCount; i++ {
|
||||
t := NewTask(fmt.Sprintf("task%d", i), map[string]interface{}{"data": i})
|
||||
if err := client.Enqueue(t, Queue("high")); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
for i := 0; i < defaultCount; i++ {
|
||||
t := NewTask(fmt.Sprintf("task%d", i), map[string]interface{}{"data": i})
|
||||
if err := client.Enqueue(t); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
for i := 0; i < lowCount; i++ {
|
||||
t := NewTask(fmt.Sprintf("task%d", i), map[string]interface{}{"data": i})
|
||||
if err := client.Enqueue(t, Queue("low")); err != nil {
|
||||
b.Fatalf("could not enqueue a task: %v", err)
|
||||
}
|
||||
}
|
||||
|
||||
var wg sync.WaitGroup
|
||||
wg.Add(highCount + defaultCount + lowCount)
|
||||
handler := func(ctx context.Context, t *Task) error {
|
||||
wg.Done()
|
||||
return nil
|
||||
}
|
||||
b.StartTimer() // end setup
|
||||
|
||||
bg.start(HandlerFunc(handler))
|
||||
wg.Wait()
|
||||
|
||||
b.StopTimer() // begin teardown
|
||||
bg.stop()
|
||||
b.StartTimer() // end teardown
|
||||
}
|
||||
}
|
||||
|
64
client.go
64
client.go
@@ -34,8 +34,9 @@ type Option interface{}
|
||||
|
||||
// Internal option representations.
|
||||
type (
|
||||
retryOption int
|
||||
queueOption string
|
||||
retryOption int
|
||||
queueOption string
|
||||
timeoutOption time.Duration
|
||||
)
|
||||
|
||||
// MaxRetry returns an option to specify the max number of times
|
||||
@@ -56,15 +57,24 @@ func Queue(name string) Option {
|
||||
return queueOption(strings.ToLower(name))
|
||||
}
|
||||
|
||||
// Timeout returns an option to specify how long a task may run.
|
||||
//
|
||||
// Zero duration means no limit.
|
||||
func Timeout(d time.Duration) Option {
|
||||
return timeoutOption(d)
|
||||
}
|
||||
|
||||
type option struct {
|
||||
retry int
|
||||
queue string
|
||||
retry int
|
||||
queue string
|
||||
timeout time.Duration
|
||||
}
|
||||
|
||||
func composeOptions(opts ...Option) option {
|
||||
res := option{
|
||||
retry: defaultMaxRetry,
|
||||
queue: base.DefaultQueueName,
|
||||
retry: defaultMaxRetry,
|
||||
queue: base.DefaultQueueName,
|
||||
timeout: 0,
|
||||
}
|
||||
for _, opt := range opts {
|
||||
switch opt := opt.(type) {
|
||||
@@ -72,6 +82,8 @@ func composeOptions(opts ...Option) option {
|
||||
res.retry = int(opt)
|
||||
case queueOption:
|
||||
res.queue = string(opt)
|
||||
case timeoutOption:
|
||||
res.timeout = time.Duration(opt)
|
||||
default:
|
||||
// ignore unexpected option
|
||||
}
|
||||
@@ -84,14 +96,13 @@ const (
|
||||
defaultMaxRetry = 25
|
||||
)
|
||||
|
||||
// Schedule registers a task to be processed at the specified time.
|
||||
// EnqueueAt schedules task to be enqueued at the specified time.
|
||||
//
|
||||
// Schedule returns nil if the task is registered successfully,
|
||||
// otherwise returns a non-nil error.
|
||||
// EnqueueAt returns nil if the task is scheduled successfully, otherwise returns a non-nil error.
|
||||
//
|
||||
// opts specifies the behavior of task processing. If there are conflicting
|
||||
// Option values the last one overrides others.
|
||||
func (c *Client) Schedule(task *Task, processAt time.Time, opts ...Option) error {
|
||||
// The argument opts specifies the behavior of task processing.
|
||||
// If there are conflicting Option values the last one overrides others.
|
||||
func (c *Client) EnqueueAt(t time.Time, task *Task, opts ...Option) error {
|
||||
opt := composeOptions(opts...)
|
||||
msg := &base.TaskMessage{
|
||||
ID: xid.New(),
|
||||
@@ -99,13 +110,34 @@ func (c *Client) Schedule(task *Task, processAt time.Time, opts ...Option) error
|
||||
Payload: task.Payload.data,
|
||||
Queue: opt.queue,
|
||||
Retry: opt.retry,
|
||||
Timeout: opt.timeout.String(),
|
||||
}
|
||||
return c.enqueue(msg, processAt)
|
||||
return c.enqueue(msg, t)
|
||||
}
|
||||
|
||||
func (c *Client) enqueue(msg *base.TaskMessage, processAt time.Time) error {
|
||||
if time.Now().After(processAt) {
|
||||
// Enqueue enqueues task to be processed immediately.
|
||||
//
|
||||
// Enqueue returns nil if the task is enqueued successfully, otherwise returns a non-nil error.
|
||||
//
|
||||
// The argument opts specifies the behavior of task processing.
|
||||
// If there are conflicting Option values the last one overrides others.
|
||||
func (c *Client) Enqueue(task *Task, opts ...Option) error {
|
||||
return c.EnqueueAt(time.Now(), task, opts...)
|
||||
}
|
||||
|
||||
// EnqueueIn schedules task to be enqueued after the specified delay.
|
||||
//
|
||||
// EnqueueIn returns nil if the task is scheduled successfully, otherwise returns a non-nil error.
|
||||
//
|
||||
// The argument opts specifies the behavior of task processing.
|
||||
// If there are conflicting Option values the last one overrides others.
|
||||
func (c *Client) EnqueueIn(d time.Duration, task *Task, opts ...Option) error {
|
||||
return c.EnqueueAt(time.Now().Add(d), task, opts...)
|
||||
}
|
||||
|
||||
func (c *Client) enqueue(msg *base.TaskMessage, t time.Time) error {
|
||||
if time.Now().After(t) {
|
||||
return c.rdb.Enqueue(msg)
|
||||
}
|
||||
return c.rdb.Schedule(msg, processAt)
|
||||
return c.rdb.Schedule(msg, t)
|
||||
}
|
||||
|
342
client_test.go
342
client_test.go
@@ -13,15 +13,20 @@ import (
|
||||
"github.com/hibiken/asynq/internal/base"
|
||||
)
|
||||
|
||||
func TestClient(t *testing.T) {
|
||||
func TestClientEnqueueAt(t *testing.T) {
|
||||
r := setup(t)
|
||||
client := NewClient(&RedisClientOpt{
|
||||
Addr: "localhost:6379",
|
||||
DB: 14,
|
||||
client := NewClient(RedisClientOpt{
|
||||
Addr: redisAddr,
|
||||
DB: redisDB,
|
||||
})
|
||||
|
||||
task := NewTask("send_email", map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"})
|
||||
|
||||
var (
|
||||
now = time.Now()
|
||||
oneHourLater = now.Add(time.Hour)
|
||||
)
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
task *Task
|
||||
@@ -33,7 +38,7 @@ func TestClient(t *testing.T) {
|
||||
{
|
||||
desc: "Process task immediately",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
processAt: now,
|
||||
opts: []Option{},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
@@ -42,6 +47,7 @@ func TestClient(t *testing.T) {
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
@@ -50,7 +56,7 @@ func TestClient(t *testing.T) {
|
||||
{
|
||||
desc: "Schedule task to be processed in the future",
|
||||
task: task,
|
||||
processAt: time.Now().Add(2 * time.Hour),
|
||||
processAt: oneHourLater,
|
||||
opts: []Option{},
|
||||
wantEnqueued: nil, // db is flushed in setup so list does not exist hence nil
|
||||
wantScheduled: []h.ZSetEntry{
|
||||
@@ -60,113 +66,241 @@ func TestClient(t *testing.T) {
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
Score: float64(time.Now().Add(2 * time.Hour).Unix()),
|
||||
Score: float64(oneHourLater.Unix()),
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Process task immediately with a custom retry count",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
opts: []Option{
|
||||
MaxRetry(3),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 3,
|
||||
Queue: "default",
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
{
|
||||
desc: "Negative retry count",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
opts: []Option{
|
||||
MaxRetry(-2),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 0, // Retry count should be set to zero
|
||||
Queue: "default",
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
{
|
||||
desc: "Conflicting options",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
opts: []Option{
|
||||
MaxRetry(2),
|
||||
MaxRetry(10),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 10, // Last option takes precedence
|
||||
Queue: "default",
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
{
|
||||
desc: "With queue option",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
opts: []Option{
|
||||
Queue("custom"),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"custom": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "custom",
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
{
|
||||
desc: "Queue option should be case-insensitive",
|
||||
task: task,
|
||||
processAt: time.Now(),
|
||||
opts: []Option{
|
||||
Queue("HIGH"),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"high": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "high",
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r) // clean up db before each test case.
|
||||
|
||||
err := client.Schedule(tc.task, tc.processAt, tc.opts...)
|
||||
err := client.EnqueueAt(tc.processAt, tc.task, tc.opts...)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
continue
|
||||
}
|
||||
|
||||
for qname, want := range tc.wantEnqueued {
|
||||
gotEnqueued := h.GetEnqueuedMessages(t, r, qname)
|
||||
if diff := cmp.Diff(want, gotEnqueued, h.IgnoreIDOpt); diff != "" {
|
||||
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.QueueKey(qname), diff)
|
||||
}
|
||||
}
|
||||
|
||||
gotScheduled := h.GetScheduledEntries(t, r)
|
||||
if diff := cmp.Diff(tc.wantScheduled, gotScheduled, h.IgnoreIDOpt); diff != "" {
|
||||
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledQueue, diff)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnqueue(t *testing.T) {
|
||||
r := setup(t)
|
||||
client := NewClient(RedisClientOpt{
|
||||
Addr: redisAddr,
|
||||
DB: redisDB,
|
||||
})
|
||||
|
||||
task := NewTask("send_email", map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"})
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
task *Task
|
||||
opts []Option
|
||||
wantEnqueued map[string][]*base.TaskMessage
|
||||
}{
|
||||
{
|
||||
desc: "Process task immediately with a custom retry count",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
MaxRetry(3),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 3,
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Negative retry count",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
MaxRetry(-2),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 0, // Retry count should be set to zero
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Conflicting options",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
MaxRetry(2),
|
||||
MaxRetry(10),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: 10, // Last option takes precedence
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "With queue option",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
Queue("custom"),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"custom": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "custom",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Queue option should be case-insensitive",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
Queue("HIGH"),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"high": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "high",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Timeout option sets the timeout duration",
|
||||
task: task,
|
||||
opts: []Option{
|
||||
Timeout(20 * time.Second),
|
||||
},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "default",
|
||||
Timeout: (20 * time.Second).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r) // clean up db before each test case.
|
||||
|
||||
err := client.Enqueue(tc.task, tc.opts...)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
continue
|
||||
}
|
||||
|
||||
for qname, want := range tc.wantEnqueued {
|
||||
got := h.GetEnqueuedMessages(t, r, qname)
|
||||
if diff := cmp.Diff(want, got, h.IgnoreIDOpt); diff != "" {
|
||||
t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.QueueKey(qname), diff)
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestClientEnqueueIn(t *testing.T) {
|
||||
r := setup(t)
|
||||
client := NewClient(RedisClientOpt{
|
||||
Addr: redisAddr,
|
||||
DB: redisDB,
|
||||
})
|
||||
|
||||
task := NewTask("send_email", map[string]interface{}{"to": "customer@gmail.com", "from": "merchant@example.com"})
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
task *Task
|
||||
delay time.Duration
|
||||
opts []Option
|
||||
wantEnqueued map[string][]*base.TaskMessage
|
||||
wantScheduled []h.ZSetEntry
|
||||
}{
|
||||
{
|
||||
desc: "schedule a task to be enqueued in one hour",
|
||||
task: task,
|
||||
delay: time.Hour,
|
||||
opts: []Option{},
|
||||
wantEnqueued: nil, // db is flushed in setup so list does not exist hence nil
|
||||
wantScheduled: []h.ZSetEntry{
|
||||
{
|
||||
Msg: &base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
Score: float64(time.Now().Add(time.Hour).Unix()),
|
||||
},
|
||||
},
|
||||
},
|
||||
{
|
||||
desc: "Zero delay",
|
||||
task: task,
|
||||
delay: 0,
|
||||
opts: []Option{},
|
||||
wantEnqueued: map[string][]*base.TaskMessage{
|
||||
"default": []*base.TaskMessage{
|
||||
&base.TaskMessage{
|
||||
Type: task.Type,
|
||||
Payload: task.Payload.data,
|
||||
Retry: defaultMaxRetry,
|
||||
Queue: "default",
|
||||
Timeout: time.Duration(0).String(),
|
||||
},
|
||||
},
|
||||
},
|
||||
wantScheduled: nil, // db is flushed in setup so zset does not exist hence nil
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r) // clean up db before each test case.
|
||||
|
||||
err := client.EnqueueIn(tc.delay, tc.task, tc.opts...)
|
||||
if err != nil {
|
||||
t.Error(err)
|
||||
continue
|
||||
|
13
doc.go
13
doc.go
@@ -9,8 +9,8 @@ Asynq uses Redis as a message broker. To connect to redis server,
|
||||
specify the options using one of RedisConnOpt types.
|
||||
|
||||
redis = &asynq.RedisClientOpt{
|
||||
Addr: "localhost:6379",
|
||||
Password: "secretpassword",
|
||||
Addr: "127.0.0.1:6379",
|
||||
Password: "xxxxx",
|
||||
DB: 3,
|
||||
}
|
||||
|
||||
@@ -24,8 +24,11 @@ Task is created with two parameters: its type and payload.
|
||||
"send_email",
|
||||
map[string]interface{}{"user_id": 42})
|
||||
|
||||
// Schedule the task t to be processed a minute from now.
|
||||
err := client.Schedule(t, time.Now().Add(time.Minute))
|
||||
// Enqueue the task to be processed immediately.
|
||||
err := client.Enqueue(t)
|
||||
|
||||
// Schedule the task to be processed in one minute.
|
||||
err = client.EnqueueIn(time.Minute, t)
|
||||
|
||||
The Background is used to run the background task processing with a given
|
||||
handler.
|
||||
@@ -45,7 +48,7 @@ Example of a type that implements the Handler interface.
|
||||
// ...
|
||||
}
|
||||
|
||||
func (h *TaskHandler) ProcessTask(task *asynq.Task) error {
|
||||
func (h *TaskHandler) ProcessTask(ctx context.Context, task *asynq.Task) error {
|
||||
switch task.Type {
|
||||
case "send_email":
|
||||
id, err := task.Payload.GetInt("user_id")
|
||||
|
BIN
docs/assets/asynqmon_ps.gif
Normal file
BIN
docs/assets/asynqmon_ps.gif
Normal file
Binary file not shown.
After Width: | Height: | Size: 582 KiB |
10
go.mod
10
go.mod
@@ -3,20 +3,12 @@ module github.com/hibiken/asynq
|
||||
go 1.13
|
||||
|
||||
require (
|
||||
github.com/go-redis/redis/v7 v7.0.0-beta.4
|
||||
github.com/go-redis/redis/v7 v7.2.0
|
||||
github.com/google/go-cmp v0.4.0
|
||||
github.com/mitchellh/go-homedir v1.1.0
|
||||
github.com/pelletier/go-toml v1.6.0 // indirect
|
||||
github.com/rs/xid v1.2.1
|
||||
github.com/spf13/afero v1.2.2 // indirect
|
||||
github.com/spf13/cast v1.3.1
|
||||
github.com/spf13/cobra v0.0.5
|
||||
github.com/spf13/jwalterweatherman v1.1.0 // indirect
|
||||
github.com/spf13/pflag v1.0.5 // indirect
|
||||
github.com/spf13/viper v1.6.0
|
||||
go.uber.org/goleak v0.10.0
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e // indirect
|
||||
golang.org/x/text v0.3.2 // indirect
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
|
||||
gopkg.in/yaml.v2 v2.2.7 // indirect
|
||||
)
|
||||
|
158
go.sum
158
go.sum
@@ -1,178 +1,53 @@
|
||||
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
|
||||
github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
|
||||
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
|
||||
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
|
||||
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
|
||||
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
|
||||
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
|
||||
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
|
||||
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
|
||||
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
|
||||
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
|
||||
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
|
||||
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
|
||||
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
|
||||
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
|
||||
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
|
||||
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
|
||||
github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
|
||||
github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
|
||||
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
|
||||
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
|
||||
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
|
||||
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
|
||||
github.com/go-redis/redis/v7 v7.0.0-beta.4 h1:p6z7Pde69EGRWvlC++y8aFcaWegyrKHzOBGo0zUACTQ=
|
||||
github.com/go-redis/redis/v7 v7.0.0-beta.4/go.mod h1:xhhSbUMTsleRPur+Vgx9sUHtyN33bdjxY+9/0n9Ig8s=
|
||||
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
|
||||
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
|
||||
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
|
||||
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
|
||||
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/go-redis/redis/v7 v7.2.0 h1:CrCexy/jYWZjW0AyVoHlcJUeZN19VWlbepTh1Vq6dJs=
|
||||
github.com/go-redis/redis/v7 v7.2.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.1 h1:YF8+flBXS5eO826T4nzqPrxfhQThhXl0YzfuUPu4SBg=
|
||||
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
|
||||
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/google/go-cmp v0.4.0 h1:xsAVV57WRhGj6kEIi8ReJzQlHHqcBYCElAvkovg3B/4=
|
||||
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1 h1:EGx4pi6eqNxGaHF6qqu48+N2wcFQ5qg5FXgOdqsJ5d8=
|
||||
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
|
||||
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
|
||||
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
|
||||
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
|
||||
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
|
||||
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
|
||||
github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
|
||||
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
|
||||
github.com/inconshreveable/mousetrap v1.0.0 h1:Z8tu5sraLXCXIcARxBp/8cbvlwVa7Z1NHg9XEKhtSvM=
|
||||
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
|
||||
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
|
||||
github.com/jtolds/gls v4.20.0+incompatible h1:xdiiI2gbIgH/gLH7ADydsJ1uDOEzR8yvV7C0MuV77Wo=
|
||||
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
|
||||
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
|
||||
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
|
||||
github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
github.com/magiconair/properties v1.8.0 h1:LLgXmsheXeRoUOBOjtwPQCWIYqM/LU1ayDtDePerRcY=
|
||||
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
|
||||
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
|
||||
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
|
||||
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
|
||||
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
|
||||
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
|
||||
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
|
||||
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
|
||||
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.8.0 h1:VkHVNpR4iVnU8XQR6DBm8BqYjN7CRzw+xKUbVVbbW9w=
|
||||
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/gomega v1.5.0 h1:izbySO9zDPmjJ8rDjLvkA2zJHIo+HkYXHnf7eN7SSyo=
|
||||
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
|
||||
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
|
||||
github.com/pelletier/go-toml v1.6.0 h1:aetoXYr0Tv7xRU/V4B4IZJ2QcbtMUFoNb3ORp7TzIK4=
|
||||
github.com/pelletier/go-toml v1.6.0/go.mod h1:5N711Q9dKgbdkxHL+MEfF31hpT7l0S0s/t2kKREewys=
|
||||
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
|
||||
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
|
||||
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
|
||||
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
|
||||
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
|
||||
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
|
||||
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
|
||||
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
|
||||
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
|
||||
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
|
||||
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
|
||||
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
|
||||
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d h1:zE9ykElWQ6/NYmHa3jpm/yHnI4xSofP+UP6SpjHcSeM=
|
||||
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||
github.com/smartystreets/goconvey v1.6.4 h1:fv0U8FUIMPNf1L9lnHLvLhgicrIVChEkdzIKYqbNC9s=
|
||||
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
|
||||
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
|
||||
github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
|
||||
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
|
||||
github.com/spf13/afero v1.2.2 h1:5jhuqJyZCZf2JRofRvN/nIFgIWNzPa3/Vz8mYylgbWc=
|
||||
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
|
||||
github.com/spf13/cast v1.3.0 h1:oget//CVOEoFewqQxwr0Ej5yjygnqGkvggSE/gB35Q8=
|
||||
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
|
||||
github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
|
||||
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
|
||||
github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
|
||||
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
|
||||
github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
|
||||
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
|
||||
github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
|
||||
github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
|
||||
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
|
||||
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/viper v1.3.2 h1:VUFqw5KcqRf7i70GOzW7N+Q7+gxVBkSSqiXB12+JQ4M=
|
||||
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
|
||||
github.com/spf13/viper v1.6.0 h1:qSjVKzM2dmqQLutPN4Y0SEzDpAf7T6HHIT3E2Xr75Gg=
|
||||
github.com/spf13/viper v1.6.0/go.mod h1:t3iDnF5Jlj76alVNuyFBk5oUMCvsrkbvZK0WQdfDi5k=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.2.2 h1:bSDNvY7ZPG5RlJ8otE/7V6gMiyenm9RtJ7IUVIAoJ1w=
|
||||
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
|
||||
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
|
||||
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
|
||||
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
|
||||
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
|
||||
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
|
||||
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
|
||||
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
|
||||
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
|
||||
go.uber.org/goleak v0.10.0 h1:G3eWbSNIskeRqtsN/1uI5B+eP73y3JUuBsv9AZjehb4=
|
||||
go.uber.org/goleak v0.10.0/go.mod h1:VCZuO8V8mFPlL0F5J5GK1rtHV3DrFcQ1R8ryq7FK0aI=
|
||||
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
|
||||
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
|
||||
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd h1:nTDtHvHSdCn1m6ITfMRqtOd/9+7a3s8RBNOZ3eYZzJA=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190522155817-f3200d17e092 h1:4QSRKanuywn15aTZvI/mIDEgPQpswuFndXpOj3rKEco=
|
||||
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/net v0.0.0-20200202094626-16171245cfb2 h1:CCH4IOTTfewWjGOlSp+zGcjutRKlBEZQ6wTn8ozI/nI=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e h1:o3PsSEY8E4eXWkXrIP9YJALUkVZqzHJT5DOasTyn8Vs=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a h1:1n5lsVfiQW3yfsRGu98756EH1YthsFqr/5mxHduZW2A=
|
||||
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e h1:9vRrk9YW2BTzLP0VCB9ZDjU4cPqkg+IDWL7XgxA1yxQ=
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/text v0.3.0 h1:g61tztE5qeGQ89tm6NTjjM9VPIm088od1l6aSorWRWg=
|
||||
@@ -181,34 +56,19 @@ golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
|
||||
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543 h1:E7g+9GITq07hpfrRu66IVDexMakfv52eLZ2CXBWiKr4=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
|
||||
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
|
||||
google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
|
||||
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127 h1:qIbj1fsPNlZgppZ+VLlY7N33q108Sa+fhmuc+sWQYwY=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
|
||||
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
|
||||
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
|
||||
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
|
||||
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
|
||||
gopkg.in/yaml.v2 v2.2.1 h1:mUhvW9EsL+naU5Q3cakzfE91YhliOondGd6ZrsDBHQE=
|
||||
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.2 h1:ZCJp+EgiOT7lHqUV2J862kp8Qj64Jo6az82+3Td9dZw=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.7 h1:VUgggvou5XRW9mHwD/yXxIYSMtY0zoKQf/v226p2nyo=
|
||||
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
||||
|
18
heartbeat.go
18
heartbeat.go
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hibiken/asynq/internal/base"
|
||||
@@ -16,7 +17,7 @@ import (
|
||||
type heartbeater struct {
|
||||
rdb *rdb.RDB
|
||||
|
||||
pinfo *base.ProcessInfo
|
||||
ps *base.ProcessState
|
||||
|
||||
// channel to communicate back to the long running "heartbeater" goroutine.
|
||||
done chan struct{}
|
||||
@@ -25,10 +26,10 @@ type heartbeater struct {
|
||||
interval time.Duration
|
||||
}
|
||||
|
||||
func newHeartbeater(rdb *rdb.RDB, pinfo *base.ProcessInfo, interval time.Duration) *heartbeater {
|
||||
func newHeartbeater(rdb *rdb.RDB, ps *base.ProcessState, interval time.Duration) *heartbeater {
|
||||
return &heartbeater{
|
||||
rdb: rdb,
|
||||
pinfo: pinfo,
|
||||
ps: ps,
|
||||
done: make(chan struct{}),
|
||||
interval: interval,
|
||||
}
|
||||
@@ -40,14 +41,17 @@ func (h *heartbeater) terminate() {
|
||||
h.done <- struct{}{}
|
||||
}
|
||||
|
||||
func (h *heartbeater) start() {
|
||||
h.pinfo.SetStarted(time.Now())
|
||||
h.pinfo.SetState("running")
|
||||
func (h *heartbeater) start(wg *sync.WaitGroup) {
|
||||
h.ps.SetStarted(time.Now())
|
||||
h.ps.SetStatus(base.StatusRunning)
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
h.beat()
|
||||
for {
|
||||
select {
|
||||
case <-h.done:
|
||||
h.rdb.ClearProcessState(h.ps)
|
||||
logger.info("Heartbeater done")
|
||||
return
|
||||
case <-time.After(h.interval):
|
||||
@@ -60,7 +64,7 @@ func (h *heartbeater) start() {
|
||||
func (h *heartbeater) beat() {
|
||||
// Note: Set TTL to be long enough so that it won't expire before we write again
|
||||
// and short enough to expire quickly once the process is shut down or killed.
|
||||
err := h.rdb.WriteProcessInfo(h.pinfo, h.interval*2)
|
||||
err := h.rdb.WriteProcessState(h.ps, h.interval*2)
|
||||
if err != nil {
|
||||
logger.error("could not write heartbeat data: %v", err)
|
||||
}
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -23,10 +24,10 @@ func TestHeartbeater(t *testing.T) {
|
||||
interval time.Duration
|
||||
host string
|
||||
pid int
|
||||
queues map[string]uint
|
||||
queues map[string]int
|
||||
concurrency int
|
||||
}{
|
||||
{time.Second, "some.address.ec2.aws.com", 45678, map[string]uint{"default": 1}, 10},
|
||||
{time.Second, "localhost", 45678, map[string]int{"default": 1}, 10},
|
||||
}
|
||||
|
||||
timeCmpOpt := cmpopts.EquateApproxTime(10 * time.Millisecond)
|
||||
@@ -34,8 +35,11 @@ func TestHeartbeater(t *testing.T) {
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r)
|
||||
|
||||
pi := base.NewProcessInfo(tc.host, tc.pid, tc.concurrency, tc.queues, false)
|
||||
hb := newHeartbeater(rdbClient, pi, tc.interval)
|
||||
state := base.NewProcessState(tc.host, tc.pid, tc.concurrency, tc.queues, false)
|
||||
hb := newHeartbeater(rdbClient, state, tc.interval)
|
||||
|
||||
var wg sync.WaitGroup
|
||||
hb.start(&wg)
|
||||
|
||||
want := &base.ProcessInfo{
|
||||
Host: tc.host,
|
||||
@@ -43,42 +47,53 @@ func TestHeartbeater(t *testing.T) {
|
||||
Queues: tc.queues,
|
||||
Concurrency: tc.concurrency,
|
||||
Started: time.Now(),
|
||||
State: "running",
|
||||
Status: "running",
|
||||
}
|
||||
hb.start()
|
||||
|
||||
// allow for heartbeater to write to redis
|
||||
time.Sleep(tc.interval * 2)
|
||||
|
||||
got, err := rdbClient.ReadProcessInfo(tc.host, tc.pid)
|
||||
ps, err := rdbClient.ListProcesses()
|
||||
if err != nil {
|
||||
t.Errorf("could not read process status from redis: %v", err)
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
||||
if diff := cmp.Diff(want, got, timeCmpOpt, ignoreOpt); diff != "" {
|
||||
t.Errorf("redis stored process status %+v, want %+v; (-want, +got)\n%s", got, want, diff)
|
||||
if len(ps) != 1 {
|
||||
t.Errorf("(*RDB).ListProcesses returned %d process info, want 1", len(ps))
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
||||
// state change
|
||||
pi.SetState("stopped")
|
||||
if diff := cmp.Diff(want, ps[0], timeCmpOpt, ignoreOpt); diff != "" {
|
||||
t.Errorf("redis stored process status %+v, want %+v; (-want, +got)\n%s", ps[0], want, diff)
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
||||
// status change
|
||||
state.SetStatus(base.StatusStopped)
|
||||
|
||||
// allow for heartbeater to write to redis
|
||||
time.Sleep(tc.interval * 2)
|
||||
|
||||
want.State = "stopped"
|
||||
got, err = rdbClient.ReadProcessInfo(tc.host, tc.pid)
|
||||
want.Status = "stopped"
|
||||
ps, err = rdbClient.ListProcesses()
|
||||
if err != nil {
|
||||
t.Errorf("could not read process status from redis: %v", err)
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
||||
if diff := cmp.Diff(want, got, timeCmpOpt, ignoreOpt); diff != "" {
|
||||
t.Errorf("redis stored process status %+v, want %+v; (-want, +got)\n%s", got, want, diff)
|
||||
if len(ps) != 1 {
|
||||
t.Errorf("(*RDB).ListProcesses returned %d process info, want 1", len(ps))
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
||||
if diff := cmp.Diff(want, ps[0], timeCmpOpt, ignoreOpt); diff != "" {
|
||||
t.Errorf("redis stored process status %+v, want %+v; (-want, +got)\n%s", ps[0], want, diff)
|
||||
hb.terminate()
|
||||
continue
|
||||
}
|
||||
|
@@ -53,6 +53,22 @@ var SortProcessInfoOpt = cmp.Transformer("SortProcessInfo", func(in []*base.Proc
|
||||
return out
|
||||
})
|
||||
|
||||
// SortWorkerInfoOpt is a cmp.Option to sort base.WorkerInfo for comparing slice of worker info.
|
||||
var SortWorkerInfoOpt = cmp.Transformer("SortWorkerInfo", func(in []*base.WorkerInfo) []*base.WorkerInfo {
|
||||
out := append([]*base.WorkerInfo(nil), in...) // Copy input to avoid mutating it
|
||||
sort.Slice(out, func(i, j int) bool {
|
||||
return out[i].ID.String() < out[j].ID.String()
|
||||
})
|
||||
return out
|
||||
})
|
||||
|
||||
// SortStringSliceOpt is a cmp.Option to sort string slice.
|
||||
var SortStringSliceOpt = cmp.Transformer("SortStringSlice", func(in []string) []string {
|
||||
out := append([]string(nil), in...)
|
||||
sort.Strings(out)
|
||||
return out
|
||||
})
|
||||
|
||||
// IgnoreIDOpt is an cmp.Option to ignore ID field in task messages when comparing.
|
||||
var IgnoreIDOpt = cmpopts.IgnoreFields(base.TaskMessage{}, "ID")
|
||||
|
||||
|
@@ -6,6 +6,7 @@
|
||||
package base
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"strings"
|
||||
"sync"
|
||||
@@ -19,8 +20,10 @@ const DefaultQueueName = "default"
|
||||
|
||||
// Redis keys
|
||||
const (
|
||||
psPrefix = "asynq:ps:" // HASH
|
||||
AllProcesses = "asynq:ps" // ZSET
|
||||
psPrefix = "asynq:ps:" // STRING - asynq:ps:<host>:<pid>
|
||||
AllWorkers = "asynq:workers" // ZSET
|
||||
workersPrefix = "asynq:workers:" // HASH - asynq:workers:<host:<pid>
|
||||
processedPrefix = "asynq:processed:" // STRING - asynq:processed:<yyyy-mm-dd>
|
||||
failurePrefix = "asynq:failure:" // STRING - asynq:failure:<yyyy-mm-dd>
|
||||
QueuePrefix = "asynq:queues:" // LIST - asynq:queues:<qname>
|
||||
@@ -30,30 +33,34 @@ const (
|
||||
RetryQueue = "asynq:retry" // ZSET
|
||||
DeadQueue = "asynq:dead" // ZSET
|
||||
InProgressQueue = "asynq:in_progress" // LIST
|
||||
CancelChannel = "asynq:cancel" // PubSub channel
|
||||
)
|
||||
|
||||
// QueueKey returns a redis key string for the given queue name.
|
||||
// QueueKey returns a redis key for the given queue name.
|
||||
func QueueKey(qname string) string {
|
||||
return QueuePrefix + strings.ToLower(qname)
|
||||
}
|
||||
|
||||
// ProcessedKey returns a redis key string for processed count
|
||||
// for the given day.
|
||||
// ProcessedKey returns a redis key for processed count for the given day.
|
||||
func ProcessedKey(t time.Time) string {
|
||||
return processedPrefix + t.UTC().Format("2006-01-02")
|
||||
}
|
||||
|
||||
// FailureKey returns a redis key string for failure count
|
||||
// for the given day.
|
||||
// FailureKey returns a redis key for failure count for the given day.
|
||||
func FailureKey(t time.Time) string {
|
||||
return failurePrefix + t.UTC().Format("2006-01-02")
|
||||
}
|
||||
|
||||
// ProcessInfoKey returns a redis key string for process info.
|
||||
// ProcessInfoKey returns a redis key for process info.
|
||||
func ProcessInfoKey(hostname string, pid int) string {
|
||||
return fmt.Sprintf("%s%s:%d", psPrefix, hostname, pid)
|
||||
}
|
||||
|
||||
// WorkersKey returns a redis key for the workers given hostname and pid.
|
||||
func WorkersKey(hostname string, pid int) string {
|
||||
return fmt.Sprintf("%s%s:%d", workersPrefix, hostname, pid)
|
||||
}
|
||||
|
||||
// TaskMessage is the internal representation of a task with additional metadata fields.
|
||||
// Serialized data of this type gets written to redis.
|
||||
type TaskMessage struct {
|
||||
@@ -77,49 +84,220 @@ type TaskMessage struct {
|
||||
|
||||
// ErrorMsg holds the error message from the last failure.
|
||||
ErrorMsg string
|
||||
|
||||
// Timeout specifies how long a task may run.
|
||||
// The string value should be compatible with time.Duration.ParseDuration.
|
||||
//
|
||||
// Zero means no limit.
|
||||
Timeout string
|
||||
}
|
||||
|
||||
// ProcessInfo holds information about running background worker process.
|
||||
// ProcessState holds process level information.
|
||||
//
|
||||
// ProcessStates are safe for concurrent use by multiple goroutines.
|
||||
type ProcessState struct {
|
||||
mu sync.Mutex // guards all data fields
|
||||
concurrency int
|
||||
queues map[string]int
|
||||
strictPriority bool
|
||||
pid int
|
||||
host string
|
||||
status PStatus
|
||||
started time.Time
|
||||
workers map[string]*workerStats
|
||||
}
|
||||
|
||||
// PStatus represents status of a process.
|
||||
type PStatus int
|
||||
|
||||
const (
|
||||
// StatusIdle indicates process is in idle state.
|
||||
StatusIdle PStatus = iota
|
||||
|
||||
// StatusRunning indicates process is up and processing tasks.
|
||||
StatusRunning
|
||||
|
||||
// StatusStopped indicates process is up but not processing new tasks.
|
||||
StatusStopped
|
||||
)
|
||||
|
||||
var statuses = []string{
|
||||
"idle",
|
||||
"running",
|
||||
"stopped",
|
||||
}
|
||||
|
||||
func (s PStatus) String() string {
|
||||
if StatusIdle <= s && s <= StatusStopped {
|
||||
return statuses[s]
|
||||
}
|
||||
return "unknown status"
|
||||
}
|
||||
|
||||
type workerStats struct {
|
||||
msg *TaskMessage
|
||||
started time.Time
|
||||
}
|
||||
|
||||
// NewProcessState returns a new instance of ProcessState.
|
||||
func NewProcessState(host string, pid, concurrency int, queues map[string]int, strict bool) *ProcessState {
|
||||
return &ProcessState{
|
||||
host: host,
|
||||
pid: pid,
|
||||
concurrency: concurrency,
|
||||
queues: cloneQueueConfig(queues),
|
||||
strictPriority: strict,
|
||||
status: StatusIdle,
|
||||
workers: make(map[string]*workerStats),
|
||||
}
|
||||
}
|
||||
|
||||
// SetStatus updates the state of process.
|
||||
func (ps *ProcessState) SetStatus(status PStatus) {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
ps.status = status
|
||||
}
|
||||
|
||||
// SetStarted records when the process started processing.
|
||||
func (ps *ProcessState) SetStarted(t time.Time) {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
ps.started = t
|
||||
}
|
||||
|
||||
// AddWorkerStats records when a worker started and which task it's processing.
|
||||
func (ps *ProcessState) AddWorkerStats(msg *TaskMessage, started time.Time) {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
ps.workers[msg.ID.String()] = &workerStats{msg, started}
|
||||
}
|
||||
|
||||
// DeleteWorkerStats removes a worker's entry from the process state.
|
||||
func (ps *ProcessState) DeleteWorkerStats(msg *TaskMessage) {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
delete(ps.workers, msg.ID.String())
|
||||
}
|
||||
|
||||
// Get returns current state of process as a ProcessInfo.
|
||||
func (ps *ProcessState) Get() *ProcessInfo {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
return &ProcessInfo{
|
||||
Host: ps.host,
|
||||
PID: ps.pid,
|
||||
Concurrency: ps.concurrency,
|
||||
Queues: cloneQueueConfig(ps.queues),
|
||||
StrictPriority: ps.strictPriority,
|
||||
Status: ps.status.String(),
|
||||
Started: ps.started,
|
||||
ActiveWorkerCount: len(ps.workers),
|
||||
}
|
||||
}
|
||||
|
||||
// GetWorkers returns a list of currently running workers' info.
|
||||
func (ps *ProcessState) GetWorkers() []*WorkerInfo {
|
||||
ps.mu.Lock()
|
||||
defer ps.mu.Unlock()
|
||||
var res []*WorkerInfo
|
||||
for _, w := range ps.workers {
|
||||
res = append(res, &WorkerInfo{
|
||||
Host: ps.host,
|
||||
PID: ps.pid,
|
||||
ID: w.msg.ID,
|
||||
Type: w.msg.Type,
|
||||
Queue: w.msg.Queue,
|
||||
Payload: clonePayload(w.msg.Payload),
|
||||
Started: w.started,
|
||||
})
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func cloneQueueConfig(qcfg map[string]int) map[string]int {
|
||||
res := make(map[string]int)
|
||||
for qname, n := range qcfg {
|
||||
res[qname] = n
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func clonePayload(payload map[string]interface{}) map[string]interface{} {
|
||||
res := make(map[string]interface{})
|
||||
for k, v := range payload {
|
||||
res[k] = v
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// ProcessInfo holds information about a running background worker process.
|
||||
type ProcessInfo struct {
|
||||
mu sync.Mutex
|
||||
Concurrency int
|
||||
Queues map[string]uint
|
||||
StrictPriority bool
|
||||
PID int
|
||||
Host string
|
||||
State string
|
||||
PID int
|
||||
Concurrency int
|
||||
Queues map[string]int
|
||||
StrictPriority bool
|
||||
Status string
|
||||
Started time.Time
|
||||
ActiveWorkerCount int
|
||||
}
|
||||
|
||||
// NewProcessInfo returns a new instance of ProcessInfo.
|
||||
func NewProcessInfo(host string, pid, concurrency int, queues map[string]uint, strict bool) *ProcessInfo {
|
||||
return &ProcessInfo{
|
||||
Host: host,
|
||||
PID: pid,
|
||||
Concurrency: concurrency,
|
||||
Queues: queues,
|
||||
StrictPriority: strict,
|
||||
// WorkerInfo holds information about a running worker.
|
||||
type WorkerInfo struct {
|
||||
Host string
|
||||
PID int
|
||||
ID xid.ID
|
||||
Type string
|
||||
Queue string
|
||||
Payload map[string]interface{}
|
||||
Started time.Time
|
||||
}
|
||||
|
||||
// Cancelations is a collection that holds cancel functions for all in-progress tasks.
|
||||
//
|
||||
// Cancelations are safe for concurrent use by multipel goroutines.
|
||||
type Cancelations struct {
|
||||
mu sync.Mutex
|
||||
cancelFuncs map[string]context.CancelFunc
|
||||
}
|
||||
|
||||
// NewCancelations returns a Cancelations instance.
|
||||
func NewCancelations() *Cancelations {
|
||||
return &Cancelations{
|
||||
cancelFuncs: make(map[string]context.CancelFunc),
|
||||
}
|
||||
}
|
||||
|
||||
// SetState set the state field of the process info.
|
||||
func (p *ProcessInfo) SetState(state string) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.State = state
|
||||
// Add adds a new cancel func to the collection.
|
||||
func (c *Cancelations) Add(id string, fn context.CancelFunc) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
c.cancelFuncs[id] = fn
|
||||
}
|
||||
|
||||
// SetStarted set the started field of the process info.
|
||||
func (p *ProcessInfo) SetStarted(t time.Time) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.Started = t
|
||||
// Delete deletes a cancel func from the collection given an id.
|
||||
func (c *Cancelations) Delete(id string) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
delete(c.cancelFuncs, id)
|
||||
}
|
||||
|
||||
// IncrActiveWorkerCount increments active worker count by delta.
|
||||
func (p *ProcessInfo) IncrActiveWorkerCount(delta int) {
|
||||
p.mu.Lock()
|
||||
defer p.mu.Unlock()
|
||||
p.ActiveWorkerCount += delta
|
||||
// Get returns a cancel func given an id.
|
||||
func (c *Cancelations) Get(id string) (fn context.CancelFunc, ok bool) {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
fn, ok = c.cancelFuncs[id]
|
||||
return fn, ok
|
||||
}
|
||||
|
||||
// GetAll returns all cancel funcs.
|
||||
func (c *Cancelations) GetAll() []context.CancelFunc {
|
||||
c.mu.Lock()
|
||||
defer c.mu.Unlock()
|
||||
var res []context.CancelFunc
|
||||
for _, fn := range c.cancelFuncs {
|
||||
res = append(res, fn)
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
@@ -5,9 +5,14 @@
|
||||
package base
|
||||
|
||||
import (
|
||||
"context"
|
||||
"math/rand"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/google/go-cmp/cmp"
|
||||
"github.com/rs/xid"
|
||||
)
|
||||
|
||||
func TestQueueKey(t *testing.T) {
|
||||
@@ -75,34 +80,137 @@ func TestProcessInfoKey(t *testing.T) {
|
||||
for _, tc := range tests {
|
||||
got := ProcessInfoKey(tc.hostname, tc.pid)
|
||||
if got != tc.want {
|
||||
t.Errorf("ProcessInfoKey(%s, %d) = %s, want %s", tc.hostname, tc.pid, got, tc.want)
|
||||
t.Errorf("ProcessInfoKey(%q, %d) = %q, want %q", tc.hostname, tc.pid, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Note: Run this test with -race flag to check for data race.
|
||||
func TestProcessInfoSetter(t *testing.T) {
|
||||
pi := NewProcessInfo("localhost", 1234, 8, map[string]uint{"default": 1}, false)
|
||||
func TestWorkersKey(t *testing.T) {
|
||||
tests := []struct {
|
||||
hostname string
|
||||
pid int
|
||||
want string
|
||||
}{
|
||||
{"localhost", 9876, "asynq:workers:localhost:9876"},
|
||||
{"127.0.0.1", 1234, "asynq:workers:127.0.0.1:1234"},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
got := WorkersKey(tc.hostname, tc.pid)
|
||||
if got != tc.want {
|
||||
t.Errorf("WorkersKey(%q, %d) = %q, want = %q", tc.hostname, tc.pid, got, tc.want)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// Test for process state being accessed by multiple goroutines.
|
||||
// Run with -race flag to check for data race.
|
||||
func TestProcessStateConcurrentAccess(t *testing.T) {
|
||||
ps := NewProcessState("127.0.0.1", 1234, 10, map[string]int{"default": 1}, false)
|
||||
var wg sync.WaitGroup
|
||||
started := time.Now()
|
||||
msgs := []*TaskMessage{
|
||||
&TaskMessage{ID: xid.New(), Type: "type1", Payload: map[string]interface{}{"user_id": 42}},
|
||||
&TaskMessage{ID: xid.New(), Type: "type2"},
|
||||
&TaskMessage{ID: xid.New(), Type: "type3"},
|
||||
}
|
||||
|
||||
wg.Add(3)
|
||||
|
||||
// Simulate hearbeater calling SetStatus and SetStarted.
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
pi.SetState("runnning")
|
||||
wg.Done()
|
||||
defer wg.Done()
|
||||
ps.SetStarted(started)
|
||||
ps.SetStatus(StatusRunning)
|
||||
}()
|
||||
|
||||
go func() {
|
||||
pi.SetStarted(time.Now())
|
||||
pi.IncrActiveWorkerCount(1)
|
||||
wg.Done()
|
||||
}()
|
||||
// Simulate processor starting worker goroutines.
|
||||
for _, msg := range msgs {
|
||||
wg.Add(1)
|
||||
ps.AddWorkerStats(msg, time.Now())
|
||||
go func(msg *TaskMessage) {
|
||||
defer wg.Done()
|
||||
time.Sleep(time.Duration(rand.Intn(500)) * time.Millisecond)
|
||||
ps.DeleteWorkerStats(msg)
|
||||
}(msg)
|
||||
}
|
||||
|
||||
// Simulate hearbeater calling Get and GetWorkers
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
pi.SetState("stopped")
|
||||
wg.Done()
|
||||
for i := 0; i < 5; i++ {
|
||||
ps.Get()
|
||||
ps.GetWorkers()
|
||||
time.Sleep(time.Duration(rand.Intn(100)) * time.Millisecond)
|
||||
}
|
||||
}()
|
||||
|
||||
wg.Wait()
|
||||
|
||||
want := &ProcessInfo{
|
||||
Host: "127.0.0.1",
|
||||
PID: 1234,
|
||||
Concurrency: 10,
|
||||
Queues: map[string]int{"default": 1},
|
||||
StrictPriority: false,
|
||||
Status: "running",
|
||||
Started: started,
|
||||
ActiveWorkerCount: 0,
|
||||
}
|
||||
|
||||
got := ps.Get()
|
||||
if diff := cmp.Diff(want, got); diff != "" {
|
||||
t.Errorf("(*ProcessState).Get() = %+v, want %+v; (-want,+got)\n%s",
|
||||
got, want, diff)
|
||||
}
|
||||
}
|
||||
|
||||
// Test for cancelations being accessed by multiple goroutines.
|
||||
// Run with -race flag to check for data race.
|
||||
func TestCancelationsConcurrentAccess(t *testing.T) {
|
||||
c := NewCancelations()
|
||||
|
||||
_, cancel1 := context.WithCancel(context.Background())
|
||||
_, cancel2 := context.WithCancel(context.Background())
|
||||
_, cancel3 := context.WithCancel(context.Background())
|
||||
var key1, key2, key3 = "key1", "key2", "key3"
|
||||
|
||||
var wg sync.WaitGroup
|
||||
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
c.Add(key1, cancel1)
|
||||
}()
|
||||
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
c.Add(key2, cancel2)
|
||||
time.Sleep(200 * time.Millisecond)
|
||||
c.Delete(key2)
|
||||
}()
|
||||
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
c.Add(key3, cancel3)
|
||||
}()
|
||||
|
||||
wg.Wait()
|
||||
|
||||
_, ok := c.Get(key1)
|
||||
if !ok {
|
||||
t.Errorf("(*Cancelations).Get(%q) = _, false, want <function>, true", key1)
|
||||
}
|
||||
|
||||
_, ok = c.Get(key2)
|
||||
if ok {
|
||||
t.Errorf("(*Cancelations).Get(%q) = _, true, want <nil>, false", key2)
|
||||
}
|
||||
|
||||
funcs := c.GetAll()
|
||||
if len(funcs) != 2 {
|
||||
t.Errorf("(*Cancelations).GetAll() returns %d functions, want 2", len(funcs))
|
||||
}
|
||||
}
|
||||
|
@@ -86,49 +86,48 @@ type DeadTask struct {
|
||||
Queue string
|
||||
}
|
||||
|
||||
// KEYS[1] -> asynq:queues
|
||||
// KEYS[2] -> asynq:in_progress
|
||||
// KEYS[3] -> asynq:scheduled
|
||||
// KEYS[4] -> asynq:retry
|
||||
// KEYS[5] -> asynq:dead
|
||||
// KEYS[6] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[7] -> asynq:failure:<yyyy-mm-dd>
|
||||
var currentStatsCmd = redis.NewScript(`
|
||||
local res = {}
|
||||
local queues = redis.call("SMEMBERS", KEYS[1])
|
||||
for _, qkey in ipairs(queues) do
|
||||
table.insert(res, qkey)
|
||||
table.insert(res, redis.call("LLEN", qkey))
|
||||
end
|
||||
table.insert(res, KEYS[2])
|
||||
table.insert(res, redis.call("LLEN", KEYS[2]))
|
||||
table.insert(res, KEYS[3])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[3]))
|
||||
table.insert(res, KEYS[4])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[4]))
|
||||
table.insert(res, KEYS[5])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[5]))
|
||||
local pcount = 0
|
||||
local p = redis.call("GET", KEYS[6])
|
||||
if p then
|
||||
pcount = tonumber(p)
|
||||
end
|
||||
table.insert(res, "processed")
|
||||
table.insert(res, pcount)
|
||||
local fcount = 0
|
||||
local f = redis.call("GET", KEYS[7])
|
||||
if f then
|
||||
fcount = tonumber(f)
|
||||
end
|
||||
table.insert(res, "failed")
|
||||
table.insert(res, fcount)
|
||||
return res`)
|
||||
|
||||
// CurrentStats returns a current state of the queues.
|
||||
func (r *RDB) CurrentStats() (*Stats, error) {
|
||||
// KEYS[1] -> asynq:queues
|
||||
// KEYS[2] -> asynq:in_progress
|
||||
// KEYS[3] -> asynq:scheduled
|
||||
// KEYS[4] -> asynq:retry
|
||||
// KEYS[5] -> asynq:dead
|
||||
// KEYS[6] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[7] -> asynq:failure:<yyyy-mm-dd>
|
||||
script := redis.NewScript(`
|
||||
local res = {}
|
||||
local queues = redis.call("SMEMBERS", KEYS[1])
|
||||
for _, qkey in ipairs(queues) do
|
||||
table.insert(res, qkey)
|
||||
table.insert(res, redis.call("LLEN", qkey))
|
||||
end
|
||||
table.insert(res, KEYS[2])
|
||||
table.insert(res, redis.call("LLEN", KEYS[2]))
|
||||
table.insert(res, KEYS[3])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[3]))
|
||||
table.insert(res, KEYS[4])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[4]))
|
||||
table.insert(res, KEYS[5])
|
||||
table.insert(res, redis.call("ZCARD", KEYS[5]))
|
||||
local pcount = 0
|
||||
local p = redis.call("GET", KEYS[6])
|
||||
if p then
|
||||
pcount = tonumber(p)
|
||||
end
|
||||
table.insert(res, "processed")
|
||||
table.insert(res, pcount)
|
||||
local fcount = 0
|
||||
local f = redis.call("GET", KEYS[7])
|
||||
if f then
|
||||
fcount = tonumber(f)
|
||||
end
|
||||
table.insert(res, "failed")
|
||||
table.insert(res, fcount)
|
||||
return res
|
||||
`)
|
||||
|
||||
now := time.Now()
|
||||
res, err := script.Run(r.client, []string{
|
||||
res, err := currentStatsCmd.Run(r.client, []string{
|
||||
base.AllQueues,
|
||||
base.InProgressQueue,
|
||||
base.ScheduledQueue,
|
||||
@@ -173,6 +172,17 @@ func (r *RDB) CurrentStats() (*Stats, error) {
|
||||
return stats, nil
|
||||
}
|
||||
|
||||
var historicalStatsCmd = redis.NewScript(`
|
||||
local res = {}
|
||||
for _, key in ipairs(KEYS) do
|
||||
local n = redis.call("GET", key)
|
||||
if not n then
|
||||
n = 0
|
||||
end
|
||||
table.insert(res, tonumber(n))
|
||||
end
|
||||
return res`)
|
||||
|
||||
// HistoricalStats returns a list of stats from the last n days.
|
||||
func (r *RDB) HistoricalStats(n int) ([]*DailyStats, error) {
|
||||
if n < 1 {
|
||||
@@ -188,18 +198,7 @@ func (r *RDB) HistoricalStats(n int) ([]*DailyStats, error) {
|
||||
keys = append(keys, base.ProcessedKey(ts))
|
||||
keys = append(keys, base.FailureKey(ts))
|
||||
}
|
||||
script := redis.NewScript(`
|
||||
local res = {}
|
||||
for _, key in ipairs(KEYS) do
|
||||
local n = redis.call("GET", key)
|
||||
if not n then
|
||||
n = 0
|
||||
end
|
||||
table.insert(res, tonumber(n))
|
||||
end
|
||||
return res
|
||||
`)
|
||||
res, err := script.Run(r.client, keys, len(keys)).Result()
|
||||
res, err := historicalStatsCmd.Run(r.client, keys, len(keys)).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
@@ -246,10 +245,10 @@ func reverse(x []string) {
|
||||
// for the list operation.
|
||||
type Pagination struct {
|
||||
// Number of items in the page.
|
||||
Size uint
|
||||
Size int
|
||||
|
||||
// Page number starting from zero.
|
||||
Page uint
|
||||
Page int
|
||||
}
|
||||
|
||||
func (p Pagination) start() int64 {
|
||||
@@ -475,21 +474,21 @@ func (r *RDB) EnqueueAllDeadTasks() (int64, error) {
|
||||
return r.removeAndEnqueueAll(base.DeadQueue)
|
||||
}
|
||||
|
||||
func (r *RDB) removeAndEnqueue(zset, id string, score float64) (int64, error) {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
local qkey = ARGV[3] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
return 1
|
||||
end
|
||||
var removeAndEnqueueCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
local qkey = ARGV[3] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
return 1
|
||||
end
|
||||
return 0
|
||||
`)
|
||||
res, err := script.Run(r.client, []string{zset}, score, id, base.QueuePrefix).Result()
|
||||
end
|
||||
return 0`)
|
||||
|
||||
func (r *RDB) removeAndEnqueue(zset, id string, score float64) (int64, error) {
|
||||
res, err := removeAndEnqueueCmd.Run(r.client, []string{zset}, score, id, base.QueuePrefix).Result()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
@@ -500,18 +499,18 @@ func (r *RDB) removeAndEnqueue(zset, id string, score float64) (int64, error) {
|
||||
return n, nil
|
||||
}
|
||||
|
||||
var removeAndEnqueueAllCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[1] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
end
|
||||
return table.getn(msgs)`)
|
||||
|
||||
func (r *RDB) removeAndEnqueueAll(zset string) (int64, error) {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[1] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
end
|
||||
return table.getn(msgs)
|
||||
`)
|
||||
res, err := script.Run(r.client, []string{zset}, base.QueuePrefix).Result()
|
||||
res, err := removeAndEnqueueAllCmd.Run(r.client, []string{zset}, base.QueuePrefix).Result()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
@@ -562,31 +561,31 @@ func (r *RDB) KillAllScheduledTasks() (int64, error) {
|
||||
return r.removeAndKillAll(base.ScheduledQueue)
|
||||
}
|
||||
|
||||
func (r *RDB) removeAndKill(zset, id string, score float64) (int64, error) {
|
||||
// KEYS[1] -> ZSET to move task from (e.g., retry queue)
|
||||
// KEYS[2] -> asynq:dead
|
||||
// ARGV[1] -> score of the task to kill
|
||||
// ARGV[2] -> id of the task to kill
|
||||
// ARGV[3] -> current timestamp
|
||||
// ARGV[4] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[5] -> max number of tasks in dead queue (e.g., 100)
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], msg)
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[4])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[5])
|
||||
return 1
|
||||
end
|
||||
// KEYS[1] -> ZSET to move task from (e.g., retry queue)
|
||||
// KEYS[2] -> asynq:dead
|
||||
// ARGV[1] -> score of the task to kill
|
||||
// ARGV[2] -> id of the task to kill
|
||||
// ARGV[3] -> current timestamp
|
||||
// ARGV[4] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[5] -> max number of tasks in dead queue (e.g., 100)
|
||||
var removeAndKillCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], msg)
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[4])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[5])
|
||||
return 1
|
||||
end
|
||||
return 0
|
||||
`)
|
||||
end
|
||||
return 0`)
|
||||
|
||||
func (r *RDB) removeAndKill(zset, id string, score float64) (int64, error) {
|
||||
now := time.Now()
|
||||
limit := now.AddDate(0, 0, -deadExpirationInDays).Unix() // 90 days ago
|
||||
res, err := script.Run(r.client,
|
||||
res, err := removeAndKillCmd.Run(r.client,
|
||||
[]string{zset, base.DeadQueue},
|
||||
score, id, now.Unix(), limit, maxDeadTasks).Result()
|
||||
if err != nil {
|
||||
@@ -599,25 +598,25 @@ func (r *RDB) removeAndKill(zset, id string, score float64) (int64, error) {
|
||||
return n, nil
|
||||
}
|
||||
|
||||
// KEYS[1] -> ZSET to move task from (e.g., retry queue)
|
||||
// KEYS[2] -> asynq:dead
|
||||
// ARGV[1] -> current timestamp
|
||||
// ARGV[2] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[3] -> max number of tasks in dead queue (e.g., 100)
|
||||
var removeAndKillAllCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("ZADD", KEYS[2], ARGV[1], msg)
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[2])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[3])
|
||||
end
|
||||
return table.getn(msgs)`)
|
||||
|
||||
func (r *RDB) removeAndKillAll(zset string) (int64, error) {
|
||||
// KEYS[1] -> ZSET to move task from (e.g., retry queue)
|
||||
// KEYS[2] -> asynq:dead
|
||||
// ARGV[1] -> current timestamp
|
||||
// ARGV[2] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[3] -> max number of tasks in dead queue (e.g., 100)
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
redis.call("ZADD", KEYS[2], ARGV[1], msg)
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[2])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[3])
|
||||
end
|
||||
return table.getn(msgs)
|
||||
`)
|
||||
now := time.Now()
|
||||
limit := now.AddDate(0, 0, -deadExpirationInDays).Unix() // 90 days ago
|
||||
res, err := script.Run(r.client, []string{zset, base.DeadQueue},
|
||||
res, err := removeAndKillAllCmd.Run(r.client, []string{zset, base.DeadQueue},
|
||||
now.Unix(), limit, maxDeadTasks).Result()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
@@ -650,19 +649,19 @@ func (r *RDB) DeleteScheduledTask(id xid.ID, score int64) error {
|
||||
return r.deleteTask(base.ScheduledQueue, id.String(), float64(score))
|
||||
}
|
||||
|
||||
func (r *RDB) deleteTask(zset, id string, score float64) error {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
return 1
|
||||
end
|
||||
var deleteTaskCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], ARGV[1], ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
if decoded["ID"] == ARGV[2] then
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
return 1
|
||||
end
|
||||
return 0
|
||||
`)
|
||||
res, err := script.Run(r.client, []string{zset}, score, id).Result()
|
||||
end
|
||||
return 0`)
|
||||
|
||||
func (r *RDB) deleteTask(zset, id string, score float64) error {
|
||||
res, err := deleteTaskCmd.Run(r.client, []string{zset}, score, id).Result()
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
@@ -709,6 +708,27 @@ func (e *ErrQueueNotEmpty) Error() string {
|
||||
return fmt.Sprintf("queue %q is not empty", e.qname)
|
||||
}
|
||||
|
||||
// Skip checking whether queue is empty before removing.
|
||||
var removeQueueForceCmd = redis.NewScript(`
|
||||
local n = redis.call("SREM", KEYS[1], KEYS[2])
|
||||
if n == 0 then
|
||||
return redis.error_reply("LIST NOT FOUND")
|
||||
end
|
||||
redis.call("DEL", KEYS[2])
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// Checks whether queue is empty before removing.
|
||||
var removeQueueCmd = redis.NewScript(`
|
||||
local l = redis.call("LLEN", KEYS[2]) if l > 0 then
|
||||
return redis.error_reply("LIST NOT EMPTY")
|
||||
end
|
||||
local n = redis.call("SREM", KEYS[1], KEYS[2])
|
||||
if n == 0 then
|
||||
return redis.error_reply("LIST NOT FOUND")
|
||||
end
|
||||
redis.call("DEL", KEYS[2])
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// RemoveQueue removes the specified queue.
|
||||
//
|
||||
// If force is set to true, it will remove the queue regardless
|
||||
@@ -718,27 +738,9 @@ func (e *ErrQueueNotEmpty) Error() string {
|
||||
func (r *RDB) RemoveQueue(qname string, force bool) error {
|
||||
var script *redis.Script
|
||||
if force {
|
||||
script = redis.NewScript(`
|
||||
local n = redis.call("SREM", KEYS[1], KEYS[2])
|
||||
if n == 0 then
|
||||
return redis.error_reply("LIST NOT FOUND")
|
||||
end
|
||||
redis.call("DEL", KEYS[2])
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
script = removeQueueForceCmd
|
||||
} else {
|
||||
script = redis.NewScript(`
|
||||
local l = redis.call("LLEN", KEYS[2])
|
||||
if l > 0 then
|
||||
return redis.error_reply("LIST NOT EMPTY")
|
||||
end
|
||||
local n = redis.call("SREM", KEYS[1], KEYS[2])
|
||||
if n == 0 then
|
||||
return redis.error_reply("LIST NOT FOUND")
|
||||
end
|
||||
redis.call("DEL", KEYS[2])
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
script = removeQueueCmd
|
||||
}
|
||||
err := script.Run(r.client,
|
||||
[]string{base.AllQueues, base.QueueKey(qname)},
|
||||
@@ -756,23 +758,23 @@ func (r *RDB) RemoveQueue(qname string, force bool) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// Note: Script also removes stale keys.
|
||||
var listProcessesCmd = redis.NewScript(`
|
||||
local res = {}
|
||||
local now = tonumber(ARGV[1])
|
||||
local keys = redis.call("ZRANGEBYSCORE", KEYS[1], now, "+inf")
|
||||
for _, key in ipairs(keys) do
|
||||
local ps = redis.call("GET", key)
|
||||
if ps then
|
||||
table.insert(res, ps)
|
||||
end
|
||||
end
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[1], "-inf", now-1)
|
||||
return res`)
|
||||
|
||||
// ListProcesses returns the list of process statuses.
|
||||
func (r *RDB) ListProcesses() ([]*base.ProcessInfo, error) {
|
||||
// Note: Script also removes stale keys.
|
||||
script := redis.NewScript(`
|
||||
local res = {}
|
||||
local now = tonumber(ARGV[1])
|
||||
local keys = redis.call("ZRANGEBYSCORE", KEYS[1], now, "+inf")
|
||||
for _, key in ipairs(keys) do
|
||||
local ps = redis.call("GET", key)
|
||||
if ps then
|
||||
table.insert(res, ps)
|
||||
end
|
||||
end
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[1], "-inf", now-1)
|
||||
return res
|
||||
`)
|
||||
res, err := script.Run(r.client,
|
||||
res, err := listProcessesCmd.Run(r.client,
|
||||
[]string{base.AllProcesses}, time.Now().UTC().Unix()).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
@@ -792,3 +794,39 @@ func (r *RDB) ListProcesses() ([]*base.ProcessInfo, error) {
|
||||
}
|
||||
return processes, nil
|
||||
}
|
||||
|
||||
// Note: Script also removes stale keys.
|
||||
var listWorkersCmd = redis.NewScript(`
|
||||
local res = {}
|
||||
local now = tonumber(ARGV[1])
|
||||
local keys = redis.call("ZRANGEBYSCORE", KEYS[1], now, "+inf")
|
||||
for _, key in ipairs(keys) do
|
||||
local workers = redis.call("HVALS", key)
|
||||
for _, w in ipairs(workers) do
|
||||
table.insert(res, w)
|
||||
end
|
||||
end
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[1], "-inf", now-1)
|
||||
return res`)
|
||||
|
||||
// ListWorkers returns the list of worker stats.
|
||||
func (r *RDB) ListWorkers() ([]*base.WorkerInfo, error) {
|
||||
res, err := listWorkersCmd.Run(r.client, []string{base.AllWorkers}, time.Now().UTC().Unix()).Result()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
data, err := cast.ToStringSliceE(res)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var workers []*base.WorkerInfo
|
||||
for _, s := range data {
|
||||
var w base.WorkerInfo
|
||||
err := json.Unmarshal([]byte(s), &w)
|
||||
if err != nil {
|
||||
continue // skip bad data
|
||||
}
|
||||
workers = append(workers, &w)
|
||||
}
|
||||
return workers, nil
|
||||
}
|
||||
|
@@ -317,8 +317,8 @@ func TestListEnqueuedPagination(t *testing.T) {
|
||||
tests := []struct {
|
||||
desc string
|
||||
qname string
|
||||
page uint
|
||||
size uint
|
||||
page int
|
||||
size int
|
||||
wantSize int
|
||||
wantFirst string
|
||||
wantLast string
|
||||
@@ -418,8 +418,8 @@ func TestListInProgressPagination(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
page uint
|
||||
size uint
|
||||
page int
|
||||
size int
|
||||
wantSize int
|
||||
wantFirst string
|
||||
wantLast string
|
||||
@@ -524,8 +524,8 @@ func TestListScheduledPagination(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
page uint
|
||||
size uint
|
||||
page int
|
||||
size int
|
||||
wantSize int
|
||||
wantFirst string
|
||||
wantLast string
|
||||
@@ -667,8 +667,8 @@ func TestListRetryPagination(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
page uint
|
||||
size uint
|
||||
page int
|
||||
size int
|
||||
wantSize int
|
||||
wantFirst string
|
||||
wantLast string
|
||||
@@ -800,8 +800,8 @@ func TestListDeadPagination(t *testing.T) {
|
||||
|
||||
tests := []struct {
|
||||
desc string
|
||||
page uint
|
||||
size uint
|
||||
page int
|
||||
size int
|
||||
wantSize int
|
||||
wantFirst string
|
||||
wantLast string
|
||||
@@ -2054,32 +2054,51 @@ func TestRemoveQueueError(t *testing.T) {
|
||||
func TestListProcesses(t *testing.T) {
|
||||
r := setup(t)
|
||||
|
||||
ps1 := &base.ProcessInfo{
|
||||
started1 := time.Now().Add(-time.Hour)
|
||||
ps1 := base.NewProcessState("do.droplet1", 1234, 10, map[string]int{"default": 1}, false)
|
||||
ps1.SetStarted(started1)
|
||||
ps1.SetStatus(base.StatusRunning)
|
||||
info1 := &base.ProcessInfo{
|
||||
Concurrency: 10,
|
||||
Queues: map[string]uint{"default": 1},
|
||||
Queues: map[string]int{"default": 1},
|
||||
Host: "do.droplet1",
|
||||
PID: 1234,
|
||||
State: "running",
|
||||
Started: time.Now().Add(-time.Hour),
|
||||
ActiveWorkerCount: 5,
|
||||
Status: "running",
|
||||
Started: started1,
|
||||
ActiveWorkerCount: 0,
|
||||
}
|
||||
|
||||
ps2 := &base.ProcessInfo{
|
||||
started2 := time.Now().Add(-2 * time.Hour)
|
||||
ps2 := base.NewProcessState("do.droplet2", 9876, 20, map[string]int{"email": 1}, false)
|
||||
ps2.SetStarted(started2)
|
||||
ps2.SetStatus(base.StatusStopped)
|
||||
ps2.AddWorkerStats(h.NewTaskMessage("send_email", nil), time.Now())
|
||||
info2 := &base.ProcessInfo{
|
||||
Concurrency: 20,
|
||||
Queues: map[string]uint{"email": 1},
|
||||
Queues: map[string]int{"email": 1},
|
||||
Host: "do.droplet2",
|
||||
PID: 9876,
|
||||
State: "stopped",
|
||||
Started: time.Now().Add(-2 * time.Hour),
|
||||
ActiveWorkerCount: 20,
|
||||
Status: "stopped",
|
||||
Started: started2,
|
||||
ActiveWorkerCount: 1,
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
processes []*base.ProcessInfo
|
||||
processes []*base.ProcessState
|
||||
want []*base.ProcessInfo
|
||||
}{
|
||||
{processes: []*base.ProcessInfo{}},
|
||||
{processes: []*base.ProcessInfo{ps1}},
|
||||
{processes: []*base.ProcessInfo{ps1, ps2}},
|
||||
{
|
||||
processes: []*base.ProcessState{},
|
||||
want: []*base.ProcessInfo{},
|
||||
},
|
||||
{
|
||||
processes: []*base.ProcessState{ps1},
|
||||
want: []*base.ProcessInfo{info1},
|
||||
},
|
||||
{
|
||||
processes: []*base.ProcessState{ps1, ps2},
|
||||
want: []*base.ProcessInfo{info1, info2},
|
||||
},
|
||||
}
|
||||
|
||||
ignoreOpt := cmpopts.IgnoreUnexported(base.ProcessInfo{})
|
||||
@@ -2088,7 +2107,7 @@ func TestListProcesses(t *testing.T) {
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
for _, ps := range tc.processes {
|
||||
if err := r.WriteProcessInfo(ps, 5*time.Second); err != nil {
|
||||
if err := r.WriteProcessState(ps, 5*time.Second); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
}
|
||||
@@ -2097,9 +2116,74 @@ func TestListProcesses(t *testing.T) {
|
||||
if err != nil {
|
||||
t.Errorf("r.ListProcesses returned an error: %v", err)
|
||||
}
|
||||
if diff := cmp.Diff(tc.processes, got, h.SortProcessInfoOpt, ignoreOpt); diff != "" {
|
||||
if diff := cmp.Diff(tc.want, got, h.SortProcessInfoOpt, ignoreOpt); diff != "" {
|
||||
t.Errorf("r.ListProcesses returned %v, want %v; (-want,+got)\n%s",
|
||||
got, tc.processes, diff)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
func TestListWorkers(t *testing.T) {
|
||||
r := setup(t)
|
||||
|
||||
const (
|
||||
host = "127.0.0.1"
|
||||
pid = 4567
|
||||
)
|
||||
|
||||
m1 := h.NewTaskMessage("send_email", map[string]interface{}{"user_id": "abc123"})
|
||||
m2 := h.NewTaskMessage("gen_thumbnail", map[string]interface{}{"path": "some/path/to/image/file"})
|
||||
m3 := h.NewTaskMessage("reindex", map[string]interface{}{})
|
||||
t1 := time.Now().Add(-time.Second)
|
||||
t2 := time.Now().Add(-10 * time.Second)
|
||||
t3 := time.Now().Add(-time.Minute)
|
||||
|
||||
type workerStats struct {
|
||||
msg *base.TaskMessage
|
||||
started time.Time
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
workers []*workerStats
|
||||
want []*base.WorkerInfo
|
||||
}{
|
||||
{
|
||||
workers: []*workerStats{
|
||||
{m1, t1},
|
||||
{m2, t2},
|
||||
{m3, t3},
|
||||
},
|
||||
want: []*base.WorkerInfo{
|
||||
{Host: host, PID: pid, ID: m1.ID, Type: m1.Type, Queue: m1.Queue, Payload: m1.Payload, Started: t1},
|
||||
{Host: host, PID: pid, ID: m2.ID, Type: m2.Type, Queue: m2.Queue, Payload: m2.Payload, Started: t2},
|
||||
{Host: host, PID: pid, ID: m3.ID, Type: m3.Type, Queue: m3.Queue, Payload: m3.Payload, Started: t3},
|
||||
},
|
||||
},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
ps := base.NewProcessState(host, pid, 10, map[string]int{"default": 1}, false)
|
||||
|
||||
for _, w := range tc.workers {
|
||||
ps.AddWorkerStats(w.msg, w.started)
|
||||
}
|
||||
|
||||
err := r.WriteProcessState(ps, time.Minute)
|
||||
if err != nil {
|
||||
t.Errorf("could not write process state to redis: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
got, err := r.ListWorkers()
|
||||
if err != nil {
|
||||
t.Errorf("(*RDB).ListWorkers() returned an error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
if diff := cmp.Diff(tc.want, got, h.SortWorkerInfoOpt); diff != "" {
|
||||
t.Errorf("(*RDB).ListWorkers() = %v, want = %v; (-want,+got)\n%s", got, tc.want, diff)
|
||||
}
|
||||
}
|
||||
}
|
||||
|
@@ -41,6 +41,14 @@ func (r *RDB) Close() error {
|
||||
return r.client.Close()
|
||||
}
|
||||
|
||||
// KEYS[1] -> asynq:queues:<qname>
|
||||
// KEYS[2] -> asynq:queues
|
||||
// ARGV[1] -> task message data
|
||||
var enqueueCmd = redis.NewScript(`
|
||||
redis.call("LPUSH", KEYS[1], ARGV[1])
|
||||
redis.call("SADD", KEYS[2], KEYS[1])
|
||||
return 1`)
|
||||
|
||||
// Enqueue inserts the given task to the tail of the queue.
|
||||
func (r *RDB) Enqueue(msg *base.TaskMessage) error {
|
||||
bytes, err := json.Marshal(msg)
|
||||
@@ -48,23 +56,18 @@ func (r *RDB) Enqueue(msg *base.TaskMessage) error {
|
||||
return err
|
||||
}
|
||||
key := base.QueueKey(msg.Queue)
|
||||
script := redis.NewScript(`
|
||||
redis.call("LPUSH", KEYS[1], ARGV[1])
|
||||
redis.call("SADD", KEYS[2], KEYS[1])
|
||||
return 1
|
||||
`)
|
||||
return script.Run(r.client, []string{key, base.AllQueues}, string(bytes)).Err()
|
||||
return enqueueCmd.Run(r.client, []string{key, base.AllQueues}, bytes).Err()
|
||||
}
|
||||
|
||||
// Dequeue queries given queues in order and pops a task message if there
|
||||
// is one and returns it. If all queues are empty, ErrNoProcessableTask
|
||||
// error is returned.
|
||||
// Dequeue queries given queues in order and pops a task message if there is one and returns it.
|
||||
// If all queues are empty, ErrNoProcessableTask error is returned.
|
||||
func (r *RDB) Dequeue(qnames ...string) (*base.TaskMessage, error) {
|
||||
var data string
|
||||
var err error
|
||||
if len(qnames) == 1 {
|
||||
data, err = r.dequeueSingle(base.QueueKey(qnames[0]))
|
||||
} else {
|
||||
// TODO(hibiken): Take keys are argument and don't compute every time
|
||||
var keys []string
|
||||
for _, q := range qnames {
|
||||
keys = append(keys, base.QueueKey(q))
|
||||
@@ -90,72 +93,74 @@ func (r *RDB) dequeueSingle(queue string) (data string, err error) {
|
||||
return r.client.BRPopLPush(queue, base.InProgressQueue, time.Second).Result()
|
||||
}
|
||||
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// ARGV -> List of queues to query in order
|
||||
var dequeueCmd = redis.NewScript(`
|
||||
local res
|
||||
for _, qkey in ipairs(ARGV) do
|
||||
res = redis.call("RPOPLPUSH", qkey, KEYS[1])
|
||||
if res then
|
||||
return res
|
||||
end
|
||||
end
|
||||
return res`)
|
||||
|
||||
func (r *RDB) dequeue(queues ...string) (data string, err error) {
|
||||
var args []interface{}
|
||||
for _, qkey := range queues {
|
||||
args = append(args, qkey)
|
||||
}
|
||||
script := redis.NewScript(`
|
||||
local res
|
||||
for _, qkey in ipairs(ARGV) do
|
||||
res = redis.call("RPOPLPUSH", qkey, KEYS[1])
|
||||
if res then
|
||||
return res
|
||||
end
|
||||
end
|
||||
return res
|
||||
`)
|
||||
res, err := script.Run(r.client, []string{base.InProgressQueue}, args...).Result()
|
||||
res, err := dequeueCmd.Run(r.client, []string{base.InProgressQueue}, args...).Result()
|
||||
if err != nil {
|
||||
return "", err
|
||||
}
|
||||
return cast.ToStringE(res)
|
||||
}
|
||||
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:processed:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value
|
||||
// ARGV[2] -> stats expiration timestamp
|
||||
// Note: LREM count ZERO means "remove all elements equal to val"
|
||||
var doneCmd = redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
local n = redis.call("INCR", KEYS[2])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[2], ARGV[2])
|
||||
end
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
|
||||
// Done removes the task from in-progress queue to mark the task as done.
|
||||
func (r *RDB) Done(msg *base.TaskMessage) error {
|
||||
bytes, err := json.Marshal(msg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Note: LREM count ZERO means "remove all elements equal to val"
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:processed:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value
|
||||
// ARGV[2] -> stats expiration timestamp
|
||||
script := redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
local n = redis.call("INCR", KEYS[2])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[2], ARGV[2])
|
||||
end
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
now := time.Now()
|
||||
processedKey := base.ProcessedKey(now)
|
||||
expireAt := now.Add(statsTTL)
|
||||
return script.Run(r.client,
|
||||
return doneCmd.Run(r.client,
|
||||
[]string{base.InProgressQueue, processedKey},
|
||||
string(bytes), expireAt.Unix()).Err()
|
||||
bytes, expireAt.Unix()).Err()
|
||||
}
|
||||
|
||||
// Requeue moves the task from in-progress queue to the default
|
||||
// queue.
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:queues:<qname>
|
||||
// ARGV[1] -> base.TaskMessage value
|
||||
// Note: Use RPUSH to push to the head of the queue.
|
||||
var requeueCmd = redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("RPUSH", KEYS[2], ARGV[1])
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// Requeue moves the task from in-progress queue to the specified queue.
|
||||
func (r *RDB) Requeue(msg *base.TaskMessage) error {
|
||||
bytes, err := json.Marshal(msg)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Note: Use RPUSH to push to the head of the queue.
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:queues:default
|
||||
// ARGV[1] -> base.TaskMessage value
|
||||
script := redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("RPUSH", KEYS[2], ARGV[1])
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
return script.Run(r.client,
|
||||
return requeueCmd.Run(r.client,
|
||||
[]string{base.InProgressQueue, base.QueueKey(msg.Queue)},
|
||||
string(bytes)).Err()
|
||||
}
|
||||
@@ -171,6 +176,27 @@ func (r *RDB) Schedule(msg *base.TaskMessage, processAt time.Time) error {
|
||||
&redis.Z{Member: string(bytes), Score: score}).Err()
|
||||
}
|
||||
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:retry
|
||||
// KEYS[3] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[4] -> asynq:failure:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value to remove from base.InProgressQueue queue
|
||||
// ARGV[2] -> base.TaskMessage value to add to Retry queue
|
||||
// ARGV[3] -> retry_at UNIX timestamp
|
||||
// ARGV[4] -> stats expiration timestamp
|
||||
var retryCmd = redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
|
||||
local n = redis.call("INCR", KEYS[3])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[3], ARGV[4])
|
||||
end
|
||||
local m = redis.call("INCR", KEYS[4])
|
||||
if tonumber(m) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[4], ARGV[4])
|
||||
end
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// Retry moves the task from in-progress to retry queue, incrementing retry count
|
||||
// and assigning error message to the task message.
|
||||
func (r *RDB) Retry(msg *base.TaskMessage, processAt time.Time, errMsg string) error {
|
||||
@@ -185,32 +211,11 @@ func (r *RDB) Retry(msg *base.TaskMessage, processAt time.Time, errMsg string) e
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:retry
|
||||
// KEYS[3] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[4] -> asynq:failure:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value to remove from base.InProgressQueue queue
|
||||
// ARGV[2] -> base.TaskMessage value to add to Retry queue
|
||||
// ARGV[3] -> retry_at UNIX timestamp
|
||||
// ARGV[4] -> stats expiration timestamp
|
||||
script := redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
|
||||
local n = redis.call("INCR", KEYS[3])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[3], ARGV[4])
|
||||
end
|
||||
local m = redis.call("INCR", KEYS[4])
|
||||
if tonumber(m) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[4], ARGV[4])
|
||||
end
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
now := time.Now()
|
||||
processedKey := base.ProcessedKey(now)
|
||||
failureKey := base.FailureKey(now)
|
||||
expireAt := now.Add(statsTTL)
|
||||
return script.Run(r.client,
|
||||
return retryCmd.Run(r.client,
|
||||
[]string{base.InProgressQueue, base.RetryQueue, processedKey, failureKey},
|
||||
string(bytesToRemove), string(bytesToAdd), processAt.Unix(), expireAt.Unix()).Err()
|
||||
}
|
||||
@@ -220,6 +225,31 @@ const (
|
||||
deadExpirationInDays = 90
|
||||
)
|
||||
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:dead
|
||||
// KEYS[3] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[4] -> asynq.failure:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value to remove from base.InProgressQueue queue
|
||||
// ARGV[2] -> base.TaskMessage value to add to Dead queue
|
||||
// ARGV[3] -> died_at UNIX timestamp
|
||||
// ARGV[4] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[5] -> max number of tasks in dead queue (e.g., 100)
|
||||
// ARGV[6] -> stats expiration timestamp
|
||||
var killCmd = redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[4])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[5])
|
||||
local n = redis.call("INCR", KEYS[3])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[3], ARGV[6])
|
||||
end
|
||||
local m = redis.call("INCR", KEYS[4])
|
||||
if tonumber(m) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[4], ARGV[6])
|
||||
end
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// Kill sends the task to "dead" queue from in-progress queue, assigning
|
||||
// the error message to the task.
|
||||
// It also trims the set by timestamp and set size.
|
||||
@@ -239,50 +269,27 @@ func (r *RDB) Kill(msg *base.TaskMessage, errMsg string) error {
|
||||
processedKey := base.ProcessedKey(now)
|
||||
failureKey := base.FailureKey(now)
|
||||
expireAt := now.Add(statsTTL)
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// KEYS[2] -> asynq:dead
|
||||
// KEYS[3] -> asynq:processed:<yyyy-mm-dd>
|
||||
// KEYS[4] -> asynq.failure:<yyyy-mm-dd>
|
||||
// ARGV[1] -> base.TaskMessage value to remove from base.InProgressQueue queue
|
||||
// ARGV[2] -> base.TaskMessage value to add to Dead queue
|
||||
// ARGV[3] -> died_at UNIX timestamp
|
||||
// ARGV[4] -> cutoff timestamp (e.g., 90 days ago)
|
||||
// ARGV[5] -> max number of tasks in dead queue (e.g., 100)
|
||||
// ARGV[6] -> stats expiration timestamp
|
||||
script := redis.NewScript(`
|
||||
redis.call("LREM", KEYS[1], 0, ARGV[1])
|
||||
redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
|
||||
redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[4])
|
||||
redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[5])
|
||||
local n = redis.call("INCR", KEYS[3])
|
||||
if tonumber(n) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[3], ARGV[6])
|
||||
end
|
||||
local m = redis.call("INCR", KEYS[4])
|
||||
if tonumber(m) == 1 then
|
||||
redis.call("EXPIREAT", KEYS[4], ARGV[6])
|
||||
end
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
return script.Run(r.client,
|
||||
return killCmd.Run(r.client,
|
||||
[]string{base.InProgressQueue, base.DeadQueue, processedKey, failureKey},
|
||||
string(bytesToRemove), string(bytesToAdd), now.Unix(), limit, maxDeadTasks, expireAt.Unix()).Err()
|
||||
}
|
||||
|
||||
// RestoreUnfinished moves all tasks from in-progress list to the queue
|
||||
// KEYS[1] -> asynq:in_progress
|
||||
// ARGV[1] -> queue prefix
|
||||
var requeueAllCmd = redis.NewScript(`
|
||||
local msgs = redis.call("LRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[1] .. decoded["Queue"]
|
||||
redis.call("RPUSH", qkey, msg)
|
||||
redis.call("LREM", KEYS[1], 0, msg)
|
||||
end
|
||||
return table.getn(msgs)`)
|
||||
|
||||
// RequeueAll moves all tasks from in-progress list to the queue
|
||||
// and reports the number of tasks restored.
|
||||
func (r *RDB) RestoreUnfinished() (int64, error) {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("LRANGE", KEYS[1], 0, -1)
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[1] .. decoded["Queue"]
|
||||
redis.call("LREM", KEYS[1], 0, msg)
|
||||
redis.call("RPUSH", qkey, msg)
|
||||
end
|
||||
return table.getn(msgs)
|
||||
`)
|
||||
res, err := script.Run(r.client, []string{base.InProgressQueue}, base.QueuePrefix).Result()
|
||||
func (r *RDB) RequeueAll() (int64, error) {
|
||||
res, err := requeueAllCmd.Run(r.client, []string{base.InProgressQueue}, base.QueuePrefix).Result()
|
||||
if err != nil {
|
||||
return 0, err
|
||||
}
|
||||
@@ -313,86 +320,124 @@ func (r *RDB) CheckAndEnqueue(qnames ...string) error {
|
||||
return nil
|
||||
}
|
||||
|
||||
// KEYS[1] -> source queue (e.g. scheduled or retry queue)
|
||||
// ARGV[1] -> current unix time
|
||||
// ARGV[2] -> queue prefix
|
||||
var forwardCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[2] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
end
|
||||
return msgs`)
|
||||
|
||||
// forward moves all tasks with a score less than the current unix time
|
||||
// from the src zset.
|
||||
func (r *RDB) forward(src string) error {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
local decoded = cjson.decode(msg)
|
||||
local qkey = ARGV[2] .. decoded["Queue"]
|
||||
redis.call("LPUSH", qkey, msg)
|
||||
end
|
||||
return msgs
|
||||
`)
|
||||
now := float64(time.Now().Unix())
|
||||
return script.Run(r.client,
|
||||
return forwardCmd.Run(r.client,
|
||||
[]string{src}, now, base.QueuePrefix).Err()
|
||||
}
|
||||
|
||||
// KEYS[1] -> source queue (e.g. scheduled or retry queue)
|
||||
// KEYS[2] -> destination queue
|
||||
var forwardSingleCmd = redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("LPUSH", KEYS[2], msg)
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
end
|
||||
return msgs`)
|
||||
|
||||
// forwardSingle moves all tasks with a score less than the current unix time
|
||||
// from the src zset to dst list.
|
||||
func (r *RDB) forwardSingle(src, dst string) error {
|
||||
script := redis.NewScript(`
|
||||
local msgs = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
|
||||
for _, msg in ipairs(msgs) do
|
||||
redis.call("ZREM", KEYS[1], msg)
|
||||
redis.call("LPUSH", KEYS[2], msg)
|
||||
end
|
||||
return msgs
|
||||
`)
|
||||
now := float64(time.Now().Unix())
|
||||
return script.Run(r.client,
|
||||
return forwardSingleCmd.Run(r.client,
|
||||
[]string{src, dst}, now).Err()
|
||||
}
|
||||
|
||||
// WriteProcessInfo writes process information to redis with expiration
|
||||
// set to the value ttl.
|
||||
func (r *RDB) WriteProcessInfo(ps *base.ProcessInfo, ttl time.Duration) error {
|
||||
bytes, err := json.Marshal(ps)
|
||||
// KEYS[1] -> asynq:ps:<host:pid>
|
||||
// KEYS[2] -> asynq:ps
|
||||
// KEYS[3] -> asynq:workers<host:pid>
|
||||
// keys[4] -> asynq:workers
|
||||
// ARGV[1] -> expiration time
|
||||
// ARGV[2] -> TTL in seconds
|
||||
// ARGV[3] -> process info
|
||||
// ARGV[4:] -> alternate key-value pair of (worker id, worker data)
|
||||
// Note: Add key to ZSET with expiration time as score.
|
||||
// ref: https://github.com/antirez/redis/issues/135#issuecomment-2361996
|
||||
var writeProcessInfoCmd = redis.NewScript(`
|
||||
redis.call("SETEX", KEYS[1], ARGV[2], ARGV[3])
|
||||
redis.call("ZADD", KEYS[2], ARGV[1], KEYS[1])
|
||||
redis.call("DEL", KEYS[3])
|
||||
for i = 4, table.getn(ARGV)-1, 2 do
|
||||
redis.call("HSET", KEYS[3], ARGV[i], ARGV[i+1])
|
||||
end
|
||||
redis.call("EXPIRE", KEYS[3], ARGV[2])
|
||||
redis.call("ZADD", KEYS[4], ARGV[1], KEYS[3])
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// WriteProcessState writes process state data to redis with expiration set to the value ttl.
|
||||
func (r *RDB) WriteProcessState(ps *base.ProcessState, ttl time.Duration) error {
|
||||
info := ps.Get()
|
||||
bytes, err := json.Marshal(info)
|
||||
if err != nil {
|
||||
return err
|
||||
}
|
||||
// Note: Add key to ZSET with expiration time as score.
|
||||
// ref: https://github.com/antirez/redis/issues/135#issuecomment-2361996
|
||||
var args []interface{} // args to the lua script
|
||||
exp := time.Now().Add(ttl).UTC()
|
||||
key := base.ProcessInfoKey(ps.Host, ps.PID)
|
||||
// KEYS[1] -> asynq:ps
|
||||
// KEYS[2] -> asynq:ps:<host:pid>
|
||||
// ARGV[1] -> expiration time
|
||||
// ARGV[2] -> TTL in seconds
|
||||
// ARGV[3] -> process info
|
||||
script := redis.NewScript(`
|
||||
redis.call("ZADD", KEYS[1], ARGV[1], KEYS[2])
|
||||
redis.call("SETEX", KEYS[2], ARGV[2], ARGV[3])
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
return script.Run(r.client, []string{base.AllProcesses, key}, float64(exp.Unix()), ttl.Seconds(), string(bytes)).Err()
|
||||
workers := ps.GetWorkers()
|
||||
args = append(args, float64(exp.Unix()), ttl.Seconds(), bytes)
|
||||
for _, w := range workers {
|
||||
bytes, err := json.Marshal(w)
|
||||
if err != nil {
|
||||
continue // skip bad data
|
||||
}
|
||||
args = append(args, w.ID.String(), bytes)
|
||||
}
|
||||
pkey := base.ProcessInfoKey(info.Host, info.PID)
|
||||
wkey := base.WorkersKey(info.Host, info.PID)
|
||||
return writeProcessInfoCmd.Run(r.client,
|
||||
[]string{pkey, base.AllProcesses, wkey, base.AllWorkers},
|
||||
args...).Err()
|
||||
}
|
||||
|
||||
// ReadProcessInfo reads process information stored in redis.
|
||||
func (r *RDB) ReadProcessInfo(host string, pid int) (*base.ProcessInfo, error) {
|
||||
key := base.ProcessInfoKey(host, pid)
|
||||
data, err := r.client.Get(key).Result()
|
||||
// KEYS[1] -> asynq:ps
|
||||
// KEYS[2] -> asynq:ps:<host:pid>
|
||||
// KEYS[3] -> asynq:workers
|
||||
// KEYS[4] -> asynq:workers<host:pid>
|
||||
var clearProcessInfoCmd = redis.NewScript(`
|
||||
redis.call("ZREM", KEYS[1], KEYS[2])
|
||||
redis.call("DEL", KEYS[2])
|
||||
redis.call("ZREM", KEYS[3], KEYS[4])
|
||||
redis.call("DEL", KEYS[4])
|
||||
return redis.status_reply("OK")`)
|
||||
|
||||
// ClearProcessState deletes process state data from redis.
|
||||
func (r *RDB) ClearProcessState(ps *base.ProcessState) error {
|
||||
info := ps.Get()
|
||||
host, pid := info.Host, info.PID
|
||||
pkey := base.ProcessInfoKey(host, pid)
|
||||
wkey := base.WorkersKey(host, pid)
|
||||
return clearProcessInfoCmd.Run(r.client,
|
||||
[]string{base.AllProcesses, pkey, base.AllWorkers, wkey}).Err()
|
||||
}
|
||||
|
||||
// CancelationPubSub returns a pubsub for cancelation messages.
|
||||
func (r *RDB) CancelationPubSub() (*redis.PubSub, error) {
|
||||
pubsub := r.client.Subscribe(base.CancelChannel)
|
||||
_, err := pubsub.Receive()
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
var pinfo base.ProcessInfo
|
||||
err = json.Unmarshal([]byte(data), &pinfo)
|
||||
if err != nil {
|
||||
return nil, err
|
||||
}
|
||||
return &pinfo, nil
|
||||
return pubsub, nil
|
||||
}
|
||||
|
||||
// ClearProcessInfo deletes process information from redis.
|
||||
func (r *RDB) ClearProcessInfo(ps *base.ProcessInfo) error {
|
||||
key := base.ProcessInfoKey(ps.Host, ps.PID)
|
||||
script := redis.NewScript(`
|
||||
redis.call("ZREM", KEYS[1], KEYS[2])
|
||||
redis.call("DEL", KEYS[2])
|
||||
return redis.status_reply("OK")
|
||||
`)
|
||||
return script.Run(r.client, []string{base.AllProcesses, key}).Err()
|
||||
// PublishCancelation publish cancelation message to all subscribers.
|
||||
// The message is the ID for the task to be canceled.
|
||||
func (r *RDB) PublishCancelation(id string) error {
|
||||
return r.client.Publish(base.CancelChannel, id).Err()
|
||||
}
|
||||
|
@@ -5,8 +5,9 @@
|
||||
package rdb
|
||||
|
||||
import (
|
||||
"encoding/json"
|
||||
"fmt"
|
||||
"strconv"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -541,7 +542,7 @@ func TestKill(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestRestoreUnfinished(t *testing.T) {
|
||||
func TestRequeueAll(t *testing.T) {
|
||||
r := setup(t)
|
||||
t1 := h.NewTaskMessage("send_email", nil)
|
||||
t2 := h.NewTaskMessage("export_csv", nil)
|
||||
@@ -613,9 +614,9 @@ func TestRestoreUnfinished(t *testing.T) {
|
||||
h.SeedEnqueuedQueue(t, r.client, msgs, qname)
|
||||
}
|
||||
|
||||
got, err := r.RestoreUnfinished()
|
||||
got, err := r.RequeueAll()
|
||||
if got != tc.want || err != nil {
|
||||
t.Errorf("(*RDB).RestoreUnfinished() = %v %v, want %v nil", got, err, tc.want)
|
||||
t.Errorf("(*RDB).RequeueAll() = %v %v, want %v nil", got, err, tc.want)
|
||||
continue
|
||||
}
|
||||
|
||||
@@ -741,81 +742,278 @@ func TestCheckAndEnqueue(t *testing.T) {
|
||||
}
|
||||
}
|
||||
|
||||
func TestReadWriteClearProcessInfo(t *testing.T) {
|
||||
func TestWriteProcessState(t *testing.T) {
|
||||
r := setup(t)
|
||||
pinfo := &base.ProcessInfo{
|
||||
Concurrency: 10,
|
||||
Queues: map[string]uint{"default": 2, "email": 5, "low": 1},
|
||||
PID: 98765,
|
||||
host, pid := "localhost", 98765
|
||||
queues := map[string]int{"default": 2, "email": 5, "low": 1}
|
||||
|
||||
started := time.Now()
|
||||
ps := base.NewProcessState(host, pid, 10, queues, false)
|
||||
ps.SetStarted(started)
|
||||
ps.SetStatus(base.StatusRunning)
|
||||
ttl := 5 * time.Second
|
||||
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
err := r.WriteProcessState(ps, ttl)
|
||||
if err != nil {
|
||||
t.Errorf("r.WriteProcessState returned an error: %v", err)
|
||||
}
|
||||
|
||||
// Check ProcessInfo was written correctly
|
||||
pkey := base.ProcessInfoKey(host, pid)
|
||||
data := r.client.Get(pkey).Val()
|
||||
var got base.ProcessInfo
|
||||
err = json.Unmarshal([]byte(data), &got)
|
||||
if err != nil {
|
||||
t.Fatalf("could not decode json: %v", err)
|
||||
}
|
||||
want := base.ProcessInfo{
|
||||
Host: "localhost",
|
||||
State: "running",
|
||||
Started: time.Now(),
|
||||
ActiveWorkerCount: 1,
|
||||
PID: 98765,
|
||||
Concurrency: 10,
|
||||
Queues: map[string]int{"default": 2, "email": 5, "low": 1},
|
||||
StrictPriority: false,
|
||||
Status: "running",
|
||||
Started: started,
|
||||
ActiveWorkerCount: 0,
|
||||
}
|
||||
if diff := cmp.Diff(want, got); diff != "" {
|
||||
t.Errorf("persisted ProcessInfo was %v, want %v; (-want,+got)\n%s",
|
||||
got, want, diff)
|
||||
}
|
||||
// Check ProcessInfo TTL was set correctly
|
||||
gotTTL := r.client.TTL(pkey).Val()
|
||||
timeCmpOpt := cmpopts.EquateApproxTime(time.Second)
|
||||
if !cmp.Equal(ttl, gotTTL, timeCmpOpt) {
|
||||
t.Errorf("TTL of %q was %v, want %v", pkey, gotTTL, ttl)
|
||||
}
|
||||
// Check ProcessInfo key was added to the set correctly
|
||||
gotProcesses := r.client.ZRange(base.AllProcesses, 0, -1).Val()
|
||||
wantProcesses := []string{pkey}
|
||||
if diff := cmp.Diff(wantProcesses, gotProcesses); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllProcesses, gotProcesses, wantProcesses)
|
||||
}
|
||||
|
||||
tests := []struct {
|
||||
pi *base.ProcessInfo
|
||||
ttl time.Duration
|
||||
}{
|
||||
{pinfo, 5 * time.Second},
|
||||
// Check WorkersInfo was written correctly
|
||||
wkey := base.WorkersKey(host, pid)
|
||||
workerExist := r.client.Exists(wkey).Val()
|
||||
if workerExist != 0 {
|
||||
t.Errorf("%q key exists", wkey)
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
err := r.WriteProcessInfo(tc.pi, tc.ttl)
|
||||
if err != nil {
|
||||
t.Errorf("r.WriteProcessInfo returned an error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
got, err := r.ReadProcessInfo(tc.pi.Host, tc.pi.PID)
|
||||
if err != nil {
|
||||
t.Errorf("r.ReadProcessInfo returned an error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
ignoreOpt := cmpopts.IgnoreUnexported(base.ProcessInfo{})
|
||||
if diff := cmp.Diff(tc.pi, got, ignoreOpt); diff != "" {
|
||||
t.Errorf("r.ReadProcessInfo(%q, %d) = %+v, want %+v; (-want,+got)\n%s",
|
||||
tc.pi.Host, tc.pi.PID, got, tc.pi, diff)
|
||||
}
|
||||
|
||||
key := base.ProcessInfoKey(tc.pi.Host, tc.pi.PID)
|
||||
gotTTL := r.client.TTL(key).Val()
|
||||
if !cmp.Equal(tc.ttl, gotTTL, timeCmpOpt) {
|
||||
t.Errorf("redis TTL %q returned %v, want %v", key, gotTTL, tc.ttl)
|
||||
}
|
||||
|
||||
now := time.Now().UTC()
|
||||
allKeys, err := r.client.ZRangeByScore(base.AllProcesses, &redis.ZRangeBy{
|
||||
Min: strconv.Itoa(int(now.Unix())),
|
||||
Max: "+inf",
|
||||
}).Result()
|
||||
if err != nil {
|
||||
t.Errorf("redis ZRANGEBYSCORE %q %d +inf returned an error: %v",
|
||||
base.AllProcesses, now.Unix(), err)
|
||||
continue
|
||||
}
|
||||
|
||||
wantAllKeys := []string{key}
|
||||
if diff := cmp.Diff(wantAllKeys, allKeys); diff != "" {
|
||||
t.Errorf("all keys = %v, want %v; (-want,+got)\n%s", allKeys, wantAllKeys, diff)
|
||||
}
|
||||
|
||||
if err := r.ClearProcessInfo(tc.pi); err != nil {
|
||||
t.Errorf("r.ClearProcessInfo returned an error: %v", err)
|
||||
continue
|
||||
}
|
||||
|
||||
// 1 means key exists
|
||||
if r.client.Exists(key).Val() == 1 {
|
||||
t.Errorf("expected %q to be deleted", key)
|
||||
}
|
||||
|
||||
if r.client.ZCard(base.AllProcesses).Val() != 0 {
|
||||
t.Errorf("expected %q to be empty", base.AllProcesses)
|
||||
}
|
||||
|
||||
// Check WorkersInfo key was added to the set correctly
|
||||
gotWorkerKeys := r.client.ZRange(base.AllWorkers, 0, -1).Val()
|
||||
wantWorkerKeys := []string{wkey}
|
||||
if diff := cmp.Diff(wantWorkerKeys, gotWorkerKeys); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllWorkers, gotWorkerKeys, wantWorkerKeys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestWriteProcessStateWithWorkers(t *testing.T) {
|
||||
r := setup(t)
|
||||
host, pid := "localhost", 98765
|
||||
queues := map[string]int{"default": 2, "email": 5, "low": 1}
|
||||
concurrency := 10
|
||||
|
||||
started := time.Now().Add(-10 * time.Minute)
|
||||
w1Started := time.Now().Add(-time.Minute)
|
||||
w2Started := time.Now().Add(-time.Second)
|
||||
msg1 := h.NewTaskMessage("send_email", map[string]interface{}{"user_id": "123"})
|
||||
msg2 := h.NewTaskMessage("gen_thumbnail", map[string]interface{}{"path": "some/path/to/imgfile"})
|
||||
ps := base.NewProcessState(host, pid, concurrency, queues, false)
|
||||
ps.SetStarted(started)
|
||||
ps.SetStatus(base.StatusRunning)
|
||||
ps.AddWorkerStats(msg1, w1Started)
|
||||
ps.AddWorkerStats(msg2, w2Started)
|
||||
ttl := 5 * time.Second
|
||||
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
err := r.WriteProcessState(ps, ttl)
|
||||
if err != nil {
|
||||
t.Errorf("r.WriteProcessState returned an error: %v", err)
|
||||
}
|
||||
|
||||
// Check ProcessInfo was written correctly
|
||||
pkey := base.ProcessInfoKey(host, pid)
|
||||
data := r.client.Get(pkey).Val()
|
||||
var got base.ProcessInfo
|
||||
err = json.Unmarshal([]byte(data), &got)
|
||||
if err != nil {
|
||||
t.Fatalf("could not decode json: %v", err)
|
||||
}
|
||||
want := base.ProcessInfo{
|
||||
Host: host,
|
||||
PID: pid,
|
||||
Concurrency: concurrency,
|
||||
Queues: queues,
|
||||
StrictPriority: false,
|
||||
Status: "running",
|
||||
Started: started,
|
||||
ActiveWorkerCount: 2,
|
||||
}
|
||||
if diff := cmp.Diff(want, got); diff != "" {
|
||||
t.Errorf("persisted ProcessInfo was %v, want %v; (-want,+got)\n%s",
|
||||
got, want, diff)
|
||||
}
|
||||
// Check ProcessInfo TTL was set correctly
|
||||
gotTTL := r.client.TTL(pkey).Val()
|
||||
timeCmpOpt := cmpopts.EquateApproxTime(time.Second)
|
||||
if !cmp.Equal(ttl, gotTTL, timeCmpOpt) {
|
||||
t.Errorf("TTL of %q was %v, want %v", pkey, gotTTL, ttl)
|
||||
}
|
||||
// Check ProcessInfo key was added to the set correctly
|
||||
gotProcesses := r.client.ZRange(base.AllProcesses, 0, -1).Val()
|
||||
wantProcesses := []string{pkey}
|
||||
if diff := cmp.Diff(wantProcesses, gotProcesses); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllProcesses, gotProcesses, wantProcesses)
|
||||
}
|
||||
|
||||
// Check WorkersInfo was written correctly
|
||||
wkey := base.WorkersKey(host, pid)
|
||||
wdata := r.client.HGetAll(wkey).Val()
|
||||
if len(wdata) != 2 {
|
||||
t.Fatalf("HGETALL %q returned a hash of size %d, want 2", wkey, len(wdata))
|
||||
}
|
||||
gotWorkers := make(map[string]*base.WorkerInfo)
|
||||
for key, val := range wdata {
|
||||
var w base.WorkerInfo
|
||||
if err := json.Unmarshal([]byte(val), &w); err != nil {
|
||||
t.Fatalf("could not unmarshal worker's data: %v", err)
|
||||
}
|
||||
gotWorkers[key] = &w
|
||||
}
|
||||
wantWorkers := map[string]*base.WorkerInfo{
|
||||
msg1.ID.String(): &base.WorkerInfo{
|
||||
Host: host,
|
||||
PID: pid,
|
||||
ID: msg1.ID,
|
||||
Type: msg1.Type,
|
||||
Queue: msg1.Queue,
|
||||
Payload: msg1.Payload,
|
||||
Started: w1Started,
|
||||
},
|
||||
msg2.ID.String(): &base.WorkerInfo{
|
||||
Host: host,
|
||||
PID: pid,
|
||||
ID: msg2.ID,
|
||||
Type: msg2.Type,
|
||||
Queue: msg2.Queue,
|
||||
Payload: msg2.Payload,
|
||||
Started: w2Started,
|
||||
},
|
||||
}
|
||||
if diff := cmp.Diff(wantWorkers, gotWorkers); diff != "" {
|
||||
t.Errorf("persisted workers info was %v, want %v; (-want,+got)\n%s",
|
||||
gotWorkers, wantWorkers, diff)
|
||||
}
|
||||
|
||||
// Check WorkersInfo TTL was set correctly
|
||||
gotTTL = r.client.TTL(wkey).Val()
|
||||
if !cmp.Equal(ttl, gotTTL, timeCmpOpt) {
|
||||
t.Errorf("TTL of %q was %v, want %v", wkey, gotTTL, ttl)
|
||||
}
|
||||
// Check WorkersInfo key was added to the set correctly
|
||||
gotWorkerKeys := r.client.ZRange(base.AllWorkers, 0, -1).Val()
|
||||
wantWorkerKeys := []string{wkey}
|
||||
if diff := cmp.Diff(wantWorkerKeys, gotWorkerKeys); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllWorkers, gotWorkerKeys, wantWorkerKeys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestClearProcessState(t *testing.T) {
|
||||
r := setup(t)
|
||||
host, pid := "127.0.0.1", 1234
|
||||
|
||||
h.FlushDB(t, r.client)
|
||||
|
||||
pkey := base.ProcessInfoKey(host, pid)
|
||||
wkey := base.WorkersKey(host, pid)
|
||||
otherPKey := base.ProcessInfoKey("otherhost", 12345)
|
||||
otherWKey := base.WorkersKey("otherhost", 12345)
|
||||
// Populate the keys.
|
||||
if err := r.client.Set(pkey, "process-info", 0).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.client.HSet(wkey, "worker-key", "worker-info").Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.client.ZAdd(base.AllProcesses, &redis.Z{Member: pkey}).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.client.ZAdd(base.AllProcesses, &redis.Z{Member: otherPKey}).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.client.ZAdd(base.AllWorkers, &redis.Z{Member: wkey}).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
if err := r.client.ZAdd(base.AllWorkers, &redis.Z{Member: otherWKey}).Err(); err != nil {
|
||||
t.Fatal(err)
|
||||
}
|
||||
|
||||
ps := base.NewProcessState(host, pid, 10, map[string]int{"default": 1}, false)
|
||||
|
||||
err := r.ClearProcessState(ps)
|
||||
if err != nil {
|
||||
t.Fatalf("(*RDB).ClearProcessState failed: %v", err)
|
||||
}
|
||||
|
||||
// Check all keys are cleared
|
||||
if r.client.Exists(pkey).Val() != 0 {
|
||||
t.Errorf("Redis key %q exists", pkey)
|
||||
}
|
||||
if r.client.Exists(wkey).Val() != 0 {
|
||||
t.Errorf("Redis key %q exists", wkey)
|
||||
}
|
||||
gotProcessKeys := r.client.ZRange(base.AllProcesses, 0, -1).Val()
|
||||
wantProcessKeys := []string{otherPKey}
|
||||
if diff := cmp.Diff(wantProcessKeys, gotProcessKeys); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllProcesses, gotProcessKeys, wantProcessKeys)
|
||||
}
|
||||
gotWorkerKeys := r.client.ZRange(base.AllWorkers, 0, -1).Val()
|
||||
wantWorkerKeys := []string{otherWKey}
|
||||
if diff := cmp.Diff(wantWorkerKeys, gotWorkerKeys); diff != "" {
|
||||
t.Errorf("%q contained %v, want %v", base.AllWorkers, gotWorkerKeys, wantWorkerKeys)
|
||||
}
|
||||
}
|
||||
|
||||
func TestCancelationPubSub(t *testing.T) {
|
||||
r := setup(t)
|
||||
|
||||
pubsub, err := r.CancelationPubSub()
|
||||
if err != nil {
|
||||
t.Fatalf("(*RDB).CancelationPubSub() returned an error: %v", err)
|
||||
}
|
||||
|
||||
cancelCh := pubsub.Channel()
|
||||
|
||||
var (
|
||||
mu sync.Mutex
|
||||
received []string
|
||||
)
|
||||
|
||||
go func() {
|
||||
for msg := range cancelCh {
|
||||
mu.Lock()
|
||||
received = append(received, msg.Payload)
|
||||
mu.Unlock()
|
||||
}
|
||||
}()
|
||||
|
||||
publish := []string{"one", "two", "three"}
|
||||
|
||||
for _, msg := range publish {
|
||||
r.PublishCancelation(msg)
|
||||
}
|
||||
|
||||
// allow for message to reach subscribers.
|
||||
time.Sleep(time.Second)
|
||||
|
||||
pubsub.Close()
|
||||
|
||||
mu.Lock()
|
||||
if diff := cmp.Diff(publish, received, h.SortStringSliceOpt); diff != "" {
|
||||
t.Errorf("subscriber received %v, want %v; (-want,+got)\n%s", received, publish, diff)
|
||||
}
|
||||
mu.Unlock()
|
||||
}
|
||||
|
76
processor.go
76
processor.go
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"math/rand"
|
||||
"sort"
|
||||
@@ -19,11 +20,11 @@ import (
|
||||
type processor struct {
|
||||
rdb *rdb.RDB
|
||||
|
||||
pinfo *base.ProcessInfo
|
||||
ps *base.ProcessState
|
||||
|
||||
handler Handler
|
||||
|
||||
queueConfig map[string]uint
|
||||
queueConfig map[string]int
|
||||
|
||||
// orderedQueues is set only in strict-priority mode.
|
||||
orderedQueues []string
|
||||
@@ -50,30 +51,35 @@ type processor struct {
|
||||
|
||||
// quit channel communicates to the in-flight worker goroutines to stop.
|
||||
quit chan struct{}
|
||||
|
||||
// cancelations is a set of cancel functions for all in-progress tasks.
|
||||
cancelations *base.Cancelations
|
||||
}
|
||||
|
||||
type retryDelayFunc func(n int, err error, task *Task) time.Duration
|
||||
|
||||
// newProcessor constructs a new processor.
|
||||
func newProcessor(r *rdb.RDB, pinfo *base.ProcessInfo, fn retryDelayFunc, syncRequestCh chan<- *syncRequest) *processor {
|
||||
qcfg := normalizeQueueCfg(pinfo.Queues)
|
||||
func newProcessor(r *rdb.RDB, ps *base.ProcessState, fn retryDelayFunc, syncCh chan<- *syncRequest, c *base.Cancelations) *processor {
|
||||
info := ps.Get()
|
||||
qcfg := normalizeQueueCfg(info.Queues)
|
||||
orderedQueues := []string(nil)
|
||||
if pinfo.StrictPriority {
|
||||
if info.StrictPriority {
|
||||
orderedQueues = sortByPriority(qcfg)
|
||||
}
|
||||
return &processor{
|
||||
rdb: r,
|
||||
pinfo: pinfo,
|
||||
ps: ps,
|
||||
queueConfig: qcfg,
|
||||
orderedQueues: orderedQueues,
|
||||
retryDelayFunc: fn,
|
||||
syncRequestCh: syncRequestCh,
|
||||
syncRequestCh: syncCh,
|
||||
cancelations: c,
|
||||
errLogLimiter: rate.NewLimiter(rate.Every(3*time.Second), 1),
|
||||
sema: make(chan struct{}, pinfo.Concurrency),
|
||||
sema: make(chan struct{}, info.Concurrency),
|
||||
done: make(chan struct{}),
|
||||
abort: make(chan struct{}),
|
||||
quit: make(chan struct{}),
|
||||
handler: HandlerFunc(func(t *Task) error { return fmt.Errorf("handler not set") }),
|
||||
handler: HandlerFunc(func(ctx context.Context, t *Task) error { return fmt.Errorf("handler not set") }),
|
||||
}
|
||||
}
|
||||
|
||||
@@ -98,6 +104,12 @@ func (p *processor) terminate() {
|
||||
const timeout = 8 * time.Second
|
||||
time.AfterFunc(timeout, func() { close(p.quit) })
|
||||
logger.info("Waiting for all workers to finish...")
|
||||
|
||||
// send cancellation signal to all in-progress task handlers
|
||||
for _, cancel := range p.cancelations.GetAll() {
|
||||
cancel()
|
||||
}
|
||||
|
||||
// block until all workers have released the token
|
||||
for i := 0; i < cap(p.sema); i++ {
|
||||
p.sema <- struct{}{}
|
||||
@@ -106,11 +118,13 @@ func (p *processor) terminate() {
|
||||
p.restore() // move any unfinished tasks back to the queue.
|
||||
}
|
||||
|
||||
func (p *processor) start() {
|
||||
func (p *processor) start(wg *sync.WaitGroup) {
|
||||
// NOTE: The call to "restore" needs to complete before starting
|
||||
// the processor goroutine.
|
||||
p.restore()
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for {
|
||||
select {
|
||||
case <-p.done:
|
||||
@@ -151,17 +165,20 @@ func (p *processor) exec() {
|
||||
p.requeue(msg)
|
||||
return
|
||||
case p.sema <- struct{}{}: // acquire token
|
||||
p.pinfo.IncrActiveWorkerCount(1)
|
||||
p.ps.AddWorkerStats(msg, time.Now())
|
||||
go func() {
|
||||
defer func() {
|
||||
p.ps.DeleteWorkerStats(msg)
|
||||
<-p.sema /* release token */
|
||||
p.pinfo.IncrActiveWorkerCount(-1)
|
||||
}()
|
||||
|
||||
resCh := make(chan error, 1)
|
||||
task := NewTask(msg.Type, msg.Payload)
|
||||
ctx, cancel := createContext(msg)
|
||||
p.cancelations.Add(msg.ID.String(), cancel)
|
||||
go func() {
|
||||
resCh <- perform(p.handler, task)
|
||||
resCh <- perform(ctx, task, p.handler)
|
||||
p.cancelations.Delete(msg.ID.String())
|
||||
}()
|
||||
|
||||
select {
|
||||
@@ -191,7 +208,7 @@ func (p *processor) exec() {
|
||||
// restore moves all tasks from "in-progress" back to queue
|
||||
// to restore all unfinished tasks.
|
||||
func (p *processor) restore() {
|
||||
n, err := p.rdb.RestoreUnfinished()
|
||||
n, err := p.rdb.RequeueAll()
|
||||
if err != nil {
|
||||
logger.error("Could not restore unfinished tasks: %v", err)
|
||||
}
|
||||
@@ -282,13 +299,13 @@ func (p *processor) queues() []string {
|
||||
// perform calls the handler with the given task.
|
||||
// If the call returns without panic, it simply returns the value,
|
||||
// otherwise, it recovers from panic and returns an error.
|
||||
func perform(h Handler, task *Task) (err error) {
|
||||
func perform(ctx context.Context, task *Task, h Handler) (err error) {
|
||||
defer func() {
|
||||
if x := recover(); x != nil {
|
||||
err = fmt.Errorf("panic: %v", x)
|
||||
}
|
||||
}()
|
||||
return h.ProcessTask(task)
|
||||
return h.ProcessTask(ctx, task)
|
||||
}
|
||||
|
||||
// uniq dedupes elements and returns a slice of unique names of length l.
|
||||
@@ -310,7 +327,7 @@ func uniq(names []string, l int) []string {
|
||||
|
||||
// sortByPriority returns a list of queue names sorted by
|
||||
// their priority level in descending order.
|
||||
func sortByPriority(qcfg map[string]uint) []string {
|
||||
func sortByPriority(qcfg map[string]int) []string {
|
||||
var queues []*queue
|
||||
for qname, n := range qcfg {
|
||||
queues = append(queues, &queue{qname, n})
|
||||
@@ -325,7 +342,7 @@ func sortByPriority(qcfg map[string]uint) []string {
|
||||
|
||||
type queue struct {
|
||||
name string
|
||||
priority uint
|
||||
priority int
|
||||
}
|
||||
|
||||
type byPriority []*queue
|
||||
@@ -336,21 +353,21 @@ func (x byPriority) Swap(i, j int) { x[i], x[j] = x[j], x[i] }
|
||||
|
||||
// normalizeQueueCfg divides priority numbers by their
|
||||
// greatest common divisor.
|
||||
func normalizeQueueCfg(queueCfg map[string]uint) map[string]uint {
|
||||
var xs []uint
|
||||
func normalizeQueueCfg(queueCfg map[string]int) map[string]int {
|
||||
var xs []int
|
||||
for _, x := range queueCfg {
|
||||
xs = append(xs, x)
|
||||
}
|
||||
d := gcd(xs...)
|
||||
res := make(map[string]uint)
|
||||
res := make(map[string]int)
|
||||
for q, x := range queueCfg {
|
||||
res[q] = x / d
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
func gcd(xs ...uint) uint {
|
||||
fn := func(x, y uint) uint {
|
||||
func gcd(xs ...int) int {
|
||||
fn := func(x, y int) int {
|
||||
for y > 0 {
|
||||
x, y = y, x%y
|
||||
}
|
||||
@@ -365,3 +382,16 @@ func gcd(xs ...uint) uint {
|
||||
}
|
||||
return res
|
||||
}
|
||||
|
||||
// createContext returns a context and cancel function for a given task message.
|
||||
func createContext(msg *base.TaskMessage) (context.Context, context.CancelFunc) {
|
||||
timeout, err := time.ParseDuration(msg.Timeout)
|
||||
if err != nil {
|
||||
logger.error("cannot parse timeout duration for %+v", msg)
|
||||
return context.WithCancel(context.Background())
|
||||
}
|
||||
if timeout == 0 {
|
||||
return context.WithCancel(context.Background())
|
||||
}
|
||||
return context.WithTimeout(context.Background(), timeout)
|
||||
}
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"context"
|
||||
"fmt"
|
||||
"sort"
|
||||
"sync"
|
||||
@@ -59,17 +60,21 @@ func TestProcessorSuccess(t *testing.T) {
|
||||
// instantiate a new processor
|
||||
var mu sync.Mutex
|
||||
var processed []*Task
|
||||
handler := func(task *Task) error {
|
||||
handler := func(ctx context.Context, task *Task) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
processed = append(processed, task)
|
||||
return nil
|
||||
}
|
||||
pi := base.NewProcessInfo("localhost", 1234, 10, defaultQueueConfig, false)
|
||||
p := newProcessor(rdbClient, pi, defaultDelayFunc, nil)
|
||||
workerCh := make(chan int)
|
||||
go fakeHeartbeater(workerCh)
|
||||
ps := base.NewProcessState("localhost", 1234, 10, defaultQueueConfig, false)
|
||||
cancelations := base.NewCancelations()
|
||||
p := newProcessor(rdbClient, ps, defaultDelayFunc, nil, cancelations)
|
||||
p.handler = HandlerFunc(handler)
|
||||
|
||||
p.start()
|
||||
var wg sync.WaitGroup
|
||||
p.start(&wg)
|
||||
for _, msg := range tc.incoming {
|
||||
err := rdbClient.Enqueue(msg)
|
||||
if err != nil {
|
||||
@@ -79,6 +84,7 @@ func TestProcessorSuccess(t *testing.T) {
|
||||
}
|
||||
time.Sleep(tc.wait)
|
||||
p.terminate()
|
||||
close(workerCh)
|
||||
|
||||
if diff := cmp.Diff(tc.wantProcessed, processed, sortTaskOpt, cmp.AllowUnexported(Payload{})); diff != "" {
|
||||
t.Errorf("mismatch found in processed tasks; (-want, +got)\n%s", diff)
|
||||
@@ -146,14 +152,18 @@ func TestProcessorRetry(t *testing.T) {
|
||||
delayFunc := func(n int, e error, t *Task) time.Duration {
|
||||
return tc.delay
|
||||
}
|
||||
handler := func(task *Task) error {
|
||||
handler := func(ctx context.Context, task *Task) error {
|
||||
return fmt.Errorf(errMsg)
|
||||
}
|
||||
pi := base.NewProcessInfo("localhost", 1234, 10, defaultQueueConfig, false)
|
||||
p := newProcessor(rdbClient, pi, delayFunc, nil)
|
||||
workerCh := make(chan int)
|
||||
go fakeHeartbeater(workerCh)
|
||||
ps := base.NewProcessState("localhost", 1234, 10, defaultQueueConfig, false)
|
||||
cancelations := base.NewCancelations()
|
||||
p := newProcessor(rdbClient, ps, delayFunc, nil, cancelations)
|
||||
p.handler = HandlerFunc(handler)
|
||||
|
||||
p.start()
|
||||
var wg sync.WaitGroup
|
||||
p.start(&wg)
|
||||
for _, msg := range tc.incoming {
|
||||
err := rdbClient.Enqueue(msg)
|
||||
if err != nil {
|
||||
@@ -163,6 +173,7 @@ func TestProcessorRetry(t *testing.T) {
|
||||
}
|
||||
time.Sleep(tc.wait)
|
||||
p.terminate()
|
||||
close(workerCh)
|
||||
|
||||
cmpOpt := cmpopts.EquateApprox(0, float64(time.Second)) // allow up to second difference in zset score
|
||||
gotRetry := h.GetRetryEntries(t, r)
|
||||
@@ -189,11 +200,11 @@ func TestProcessorQueues(t *testing.T) {
|
||||
})
|
||||
|
||||
tests := []struct {
|
||||
queueCfg map[string]uint
|
||||
queueCfg map[string]int
|
||||
want []string
|
||||
}{
|
||||
{
|
||||
queueCfg: map[string]uint{
|
||||
queueCfg: map[string]int{
|
||||
"high": 6,
|
||||
"default": 3,
|
||||
"low": 1,
|
||||
@@ -201,7 +212,7 @@ func TestProcessorQueues(t *testing.T) {
|
||||
want: []string{"high", "default", "low"},
|
||||
},
|
||||
{
|
||||
queueCfg: map[string]uint{
|
||||
queueCfg: map[string]int{
|
||||
"default": 1,
|
||||
},
|
||||
want: []string{"default"},
|
||||
@@ -209,8 +220,9 @@ func TestProcessorQueues(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
pi := base.NewProcessInfo("localhost", 1234, 10, tc.queueCfg, false)
|
||||
p := newProcessor(nil, pi, defaultDelayFunc, nil)
|
||||
cancelations := base.NewCancelations()
|
||||
ps := base.NewProcessState("localhost", 1234, 10, tc.queueCfg, false)
|
||||
p := newProcessor(nil, ps, defaultDelayFunc, nil, cancelations)
|
||||
got := p.queues()
|
||||
if diff := cmp.Diff(tc.want, got, sortOpt); diff != "" {
|
||||
t.Errorf("with queue config: %v\n(*processor).queues() = %v, want %v\n(-want,+got):\n%s",
|
||||
@@ -264,25 +276,30 @@ func TestProcessorWithStrictPriority(t *testing.T) {
|
||||
// instantiate a new processor
|
||||
var mu sync.Mutex
|
||||
var processed []*Task
|
||||
handler := func(task *Task) error {
|
||||
handler := func(ctx context.Context, task *Task) error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
processed = append(processed, task)
|
||||
return nil
|
||||
}
|
||||
queueCfg := map[string]uint{
|
||||
queueCfg := map[string]int{
|
||||
"critical": 3,
|
||||
base.DefaultQueueName: 2,
|
||||
"low": 1,
|
||||
}
|
||||
// Note: Set concurrency to 1 to make sure tasks are processed one at a time.
|
||||
pi := base.NewProcessInfo("localhost", 1234, 1 /*concurrency */, queueCfg, true /* strict */)
|
||||
p := newProcessor(rdbClient, pi, defaultDelayFunc, nil)
|
||||
workerCh := make(chan int)
|
||||
go fakeHeartbeater(workerCh)
|
||||
cancelations := base.NewCancelations()
|
||||
ps := base.NewProcessState("localhost", 1234, 1 /* concurrency */, queueCfg, true /*strict*/)
|
||||
p := newProcessor(rdbClient, ps, defaultDelayFunc, nil, cancelations)
|
||||
p.handler = HandlerFunc(handler)
|
||||
|
||||
p.start()
|
||||
var wg sync.WaitGroup
|
||||
p.start(&wg)
|
||||
time.Sleep(tc.wait)
|
||||
p.terminate()
|
||||
close(workerCh)
|
||||
|
||||
if diff := cmp.Diff(tc.wantProcessed, processed, cmp.AllowUnexported(Payload{})); diff != "" {
|
||||
t.Errorf("mismatch found in processed tasks; (-want, +got)\n%s", diff)
|
||||
@@ -303,7 +320,7 @@ func TestPerform(t *testing.T) {
|
||||
}{
|
||||
{
|
||||
desc: "handler returns nil",
|
||||
handler: func(t *Task) error {
|
||||
handler: func(ctx context.Context, t *Task) error {
|
||||
return nil
|
||||
},
|
||||
task: NewTask("gen_thumbnail", map[string]interface{}{"src": "some/img/path"}),
|
||||
@@ -311,7 +328,7 @@ func TestPerform(t *testing.T) {
|
||||
},
|
||||
{
|
||||
desc: "handler returns error",
|
||||
handler: func(t *Task) error {
|
||||
handler: func(ctx context.Context, t *Task) error {
|
||||
return fmt.Errorf("something went wrong")
|
||||
},
|
||||
task: NewTask("gen_thumbnail", map[string]interface{}{"src": "some/img/path"}),
|
||||
@@ -319,7 +336,7 @@ func TestPerform(t *testing.T) {
|
||||
},
|
||||
{
|
||||
desc: "handler panics",
|
||||
handler: func(t *Task) error {
|
||||
handler: func(ctx context.Context, t *Task) error {
|
||||
panic("something went terribly wrong")
|
||||
},
|
||||
task: NewTask("gen_thumbnail", map[string]interface{}{"src": "some/img/path"}),
|
||||
@@ -328,7 +345,7 @@ func TestPerform(t *testing.T) {
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
got := perform(tc.handler, tc.task)
|
||||
got := perform(context.Background(), tc.task, tc.handler)
|
||||
if !tc.wantErr && got != nil {
|
||||
t.Errorf("%s: perform() = %v, want nil", tc.desc, got)
|
||||
continue
|
||||
@@ -339,3 +356,9 @@ func TestPerform(t *testing.T) {
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
// fake heartbeater to receive sends from the worker channel.
|
||||
func fakeHeartbeater(ch <-chan int) {
|
||||
for range ch {
|
||||
}
|
||||
}
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
@@ -23,7 +24,7 @@ type scheduler struct {
|
||||
qnames []string
|
||||
}
|
||||
|
||||
func newScheduler(r *rdb.RDB, avgInterval time.Duration, qcfg map[string]uint) *scheduler {
|
||||
func newScheduler(r *rdb.RDB, avgInterval time.Duration, qcfg map[string]int) *scheduler {
|
||||
var qnames []string
|
||||
for q := range qcfg {
|
||||
qnames = append(qnames, q)
|
||||
@@ -43,8 +44,10 @@ func (s *scheduler) terminate() {
|
||||
}
|
||||
|
||||
// start starts the "scheduler" goroutine.
|
||||
func (s *scheduler) start() {
|
||||
func (s *scheduler) start(wg *sync.WaitGroup) {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for {
|
||||
select {
|
||||
case <-s.done:
|
||||
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
@@ -69,7 +70,8 @@ func TestScheduler(t *testing.T) {
|
||||
h.SeedRetryQueue(t, r, tc.initRetry) // initialize retry queue
|
||||
h.SeedEnqueuedQueue(t, r, tc.initQueue) // initialize default queue
|
||||
|
||||
s.start()
|
||||
var wg sync.WaitGroup
|
||||
s.start(&wg)
|
||||
time.Sleep(tc.wait)
|
||||
s.terminate()
|
||||
|
||||
|
62
subscriber.go
Normal file
62
subscriber.go
Normal file
@@ -0,0 +1,62 @@
|
||||
// Copyright 2020 Kentaro Hibino. All rights reserved.
|
||||
// Use of this source code is governed by a MIT license
|
||||
// that can be found in the LICENSE file.
|
||||
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
|
||||
"github.com/hibiken/asynq/internal/base"
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
)
|
||||
|
||||
type subscriber struct {
|
||||
rdb *rdb.RDB
|
||||
|
||||
// channel to communicate back to the long running "subscriber" goroutine.
|
||||
done chan struct{}
|
||||
|
||||
// cancelations hold cancel functions for all in-progress tasks.
|
||||
cancelations *base.Cancelations
|
||||
}
|
||||
|
||||
func newSubscriber(rdb *rdb.RDB, cancelations *base.Cancelations) *subscriber {
|
||||
return &subscriber{
|
||||
rdb: rdb,
|
||||
done: make(chan struct{}),
|
||||
cancelations: cancelations,
|
||||
}
|
||||
}
|
||||
|
||||
func (s *subscriber) terminate() {
|
||||
logger.info("Subscriber shutting down...")
|
||||
// Signal the subscriber goroutine to stop.
|
||||
s.done <- struct{}{}
|
||||
}
|
||||
|
||||
func (s *subscriber) start(wg *sync.WaitGroup) {
|
||||
pubsub, err := s.rdb.CancelationPubSub()
|
||||
cancelCh := pubsub.Channel()
|
||||
if err != nil {
|
||||
logger.error("cannot subscribe to cancelation channel: %v", err)
|
||||
return
|
||||
}
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
for {
|
||||
select {
|
||||
case <-s.done:
|
||||
pubsub.Close()
|
||||
logger.info("Subscriber done")
|
||||
return
|
||||
case msg := <-cancelCh:
|
||||
cancel, ok := s.cancelations.Get(msg.Payload)
|
||||
if ok {
|
||||
cancel()
|
||||
}
|
||||
}
|
||||
}
|
||||
}()
|
||||
}
|
64
subscriber_test.go
Normal file
64
subscriber_test.go
Normal file
@@ -0,0 +1,64 @@
|
||||
// Copyright 2020 Kentaro Hibino. All rights reserved.
|
||||
// Use of this source code is governed by a MIT license
|
||||
// that can be found in the LICENSE file.
|
||||
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/hibiken/asynq/internal/base"
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
)
|
||||
|
||||
func TestSubscriber(t *testing.T) {
|
||||
r := setup(t)
|
||||
rdbClient := rdb.NewRDB(r)
|
||||
|
||||
tests := []struct {
|
||||
registeredID string // ID for which cancel func is registered
|
||||
publishID string // ID to be published
|
||||
wantCalled bool // whether cancel func should be called
|
||||
}{
|
||||
{"abc123", "abc123", true},
|
||||
{"abc456", "abc123", false},
|
||||
}
|
||||
|
||||
for _, tc := range tests {
|
||||
var mu sync.Mutex
|
||||
called := false
|
||||
fakeCancelFunc := func() {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
called = true
|
||||
}
|
||||
cancelations := base.NewCancelations()
|
||||
cancelations.Add(tc.registeredID, fakeCancelFunc)
|
||||
|
||||
subscriber := newSubscriber(rdbClient, cancelations)
|
||||
var wg sync.WaitGroup
|
||||
subscriber.start(&wg)
|
||||
|
||||
if err := rdbClient.PublishCancelation(tc.publishID); err != nil {
|
||||
subscriber.terminate()
|
||||
t.Fatalf("could not publish cancelation message: %v", err)
|
||||
}
|
||||
|
||||
// allow for redis to publish message
|
||||
time.Sleep(time.Second)
|
||||
|
||||
mu.Lock()
|
||||
if called != tc.wantCalled {
|
||||
if tc.wantCalled {
|
||||
t.Errorf("fakeCancelFunc was not called, want the function to be called")
|
||||
} else {
|
||||
t.Errorf("fakeCancelFunc was called, want the function to not be called")
|
||||
}
|
||||
}
|
||||
mu.Unlock()
|
||||
|
||||
subscriber.terminate()
|
||||
}
|
||||
}
|
@@ -5,6 +5,7 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"sync"
|
||||
"time"
|
||||
)
|
||||
|
||||
@@ -39,8 +40,10 @@ func (s *syncer) terminate() {
|
||||
s.done <- struct{}{}
|
||||
}
|
||||
|
||||
func (s *syncer) start() {
|
||||
func (s *syncer) start(wg *sync.WaitGroup) {
|
||||
wg.Add(1)
|
||||
go func() {
|
||||
defer wg.Done()
|
||||
var requests []*syncRequest
|
||||
for {
|
||||
select {
|
||||
|
@@ -5,10 +5,11 @@
|
||||
package asynq
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"sync"
|
||||
"testing"
|
||||
"time"
|
||||
|
||||
"github.com/go-redis/redis/v7"
|
||||
h "github.com/hibiken/asynq/internal/asynqtest"
|
||||
"github.com/hibiken/asynq/internal/base"
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
@@ -27,7 +28,8 @@ func TestSyncer(t *testing.T) {
|
||||
const interval = time.Second
|
||||
syncRequestCh := make(chan *syncRequest)
|
||||
syncer := newSyncer(syncRequestCh, interval)
|
||||
syncer.start()
|
||||
var wg sync.WaitGroup
|
||||
syncer.start(&wg)
|
||||
defer syncer.terminate()
|
||||
|
||||
for _, msg := range inProgress {
|
||||
@@ -48,53 +50,43 @@ func TestSyncer(t *testing.T) {
|
||||
}
|
||||
|
||||
func TestSyncerRetry(t *testing.T) {
|
||||
inProgress := []*base.TaskMessage{
|
||||
h.NewTaskMessage("send_email", nil),
|
||||
h.NewTaskMessage("reindex", nil),
|
||||
h.NewTaskMessage("gen_thumbnail", nil),
|
||||
}
|
||||
goodClient := setup(t)
|
||||
h.SeedInProgressQueue(t, goodClient, inProgress)
|
||||
|
||||
// Simulate the situation where redis server is down
|
||||
// by connecting to a wrong port.
|
||||
badClient := redis.NewClient(&redis.Options{
|
||||
Addr: "localhost:6390",
|
||||
})
|
||||
rdbClient := rdb.NewRDB(badClient)
|
||||
|
||||
const interval = time.Second
|
||||
syncRequestCh := make(chan *syncRequest)
|
||||
syncer := newSyncer(syncRequestCh, interval)
|
||||
syncer.start()
|
||||
|
||||
var wg sync.WaitGroup
|
||||
syncer.start(&wg)
|
||||
defer syncer.terminate()
|
||||
|
||||
for _, msg := range inProgress {
|
||||
m := msg
|
||||
syncRequestCh <- &syncRequest{
|
||||
fn: func() error {
|
||||
return rdbClient.Done(m)
|
||||
},
|
||||
var (
|
||||
mu sync.Mutex
|
||||
counter int
|
||||
)
|
||||
|
||||
// Increment the counter for each call.
|
||||
// Initial call will fail and second call will succeed.
|
||||
requestFunc := func() error {
|
||||
mu.Lock()
|
||||
defer mu.Unlock()
|
||||
if counter == 0 {
|
||||
counter++
|
||||
return fmt.Errorf("zero")
|
||||
}
|
||||
counter++
|
||||
return nil
|
||||
}
|
||||
|
||||
time.Sleep(2 * interval) // ensure that syncer runs at least once
|
||||
|
||||
// Sanity check to ensure that message was not successfully deleted
|
||||
// from in-progress list.
|
||||
gotInProgress := h.GetInProgressMessages(t, goodClient)
|
||||
if l := len(gotInProgress); l != len(inProgress) {
|
||||
t.Errorf("%q has length %d; want %d", base.InProgressQueue, l, len(inProgress))
|
||||
syncRequestCh <- &syncRequest{
|
||||
fn: requestFunc,
|
||||
errMsg: "error",
|
||||
}
|
||||
|
||||
// FIXME: This assignment introduces data race and running the test with -race will fail.
|
||||
// simualate failover.
|
||||
rdbClient = rdb.NewRDB(goodClient)
|
||||
// allow syncer to retry
|
||||
time.Sleep(3 * interval)
|
||||
|
||||
time.Sleep(2 * interval) // ensure that syncer runs at least once
|
||||
|
||||
gotInProgress = h.GetInProgressMessages(t, goodClient)
|
||||
if l := len(gotInProgress); l != 0 {
|
||||
t.Errorf("%q has length %d; want 0", base.InProgressQueue, l)
|
||||
mu.Lock()
|
||||
if counter != 2 {
|
||||
t.Errorf("counter = %d, want 2", counter)
|
||||
}
|
||||
mu.Unlock()
|
||||
}
|
||||
|
@@ -8,10 +8,12 @@ Asynqmon is a command line tool to monitor the tasks managed by `asynq` package.
|
||||
- [Quick Start](#quick-start)
|
||||
- [Stats](#stats)
|
||||
- [History](#history)
|
||||
- [Process Status](#process-status)
|
||||
- [List](#list)
|
||||
- [Enqueue](#enqueue)
|
||||
- [Delete](#delete)
|
||||
- [Kill](#kill)
|
||||
- [Cancel](#cancel)
|
||||
- [Config File](#config-file)
|
||||
|
||||
## Installation
|
||||
@@ -22,7 +24,7 @@ In order to use the tool, compile it using the following command:
|
||||
|
||||
This will create the asynqmon executable under your `$GOPATH/bin` directory.
|
||||
|
||||
## Quick Start
|
||||
## Quickstart
|
||||
|
||||
The tool has a few commands to inspect the state of tasks and queues.
|
||||
|
||||
@@ -56,6 +58,16 @@ Example:
|
||||
|
||||

|
||||
|
||||
### Process Status
|
||||
|
||||
PS (ProcessStatus) command shows the list of running worker processes.
|
||||
|
||||
Example:
|
||||
|
||||
asynqmon ps
|
||||
|
||||

|
||||
|
||||
### List
|
||||
|
||||
List command shows all tasks in the specified state in a table format
|
||||
@@ -122,6 +134,18 @@ Example:
|
||||
|
||||
Running the above command will move all **Retry** tasks to **Dead** state.
|
||||
|
||||
### Cancel
|
||||
|
||||
Command `cancel` takes a task ID and sends a cancelation signal to the goroutine processing the specified task.
|
||||
You can obtain the task ID by running `ls` command.
|
||||
|
||||
The task should be in "in-progress" state.
|
||||
Handler implementation needs to be context aware in order to actually stop processing.
|
||||
|
||||
Example:
|
||||
|
||||
asynqmon cancel bnogo8gt6toe23vhef0g
|
||||
|
||||
## Config File
|
||||
|
||||
You can use a config file to set default values for the flags.
|
||||
|
53
tools/asynqmon/cmd/cancel.go
Normal file
53
tools/asynqmon/cmd/cancel.go
Normal file
@@ -0,0 +1,53 @@
|
||||
// Copyright 2020 Kentaro Hibino. All rights reserved.
|
||||
// Use of this source code is governed by a MIT license
|
||||
// that can be found in the LICENSE file.
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"os"
|
||||
|
||||
"github.com/go-redis/redis/v7"
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// cancelCmd represents the cancel command
|
||||
var cancelCmd = &cobra.Command{
|
||||
Use: "cancel [task id]",
|
||||
Short: "Sends a cancelation signal to the goroutine processing the specified task",
|
||||
Long: `Cancel (asynqmon cancel) will send a cancelation signal to the goroutine processing
|
||||
the specified task.
|
||||
|
||||
The command takes one argument which specifies the task to cancel.
|
||||
The task should be in in-progress state.
|
||||
Identifier for a task should be obtained by running "asynqmon ls" command.
|
||||
|
||||
Handler implementation needs to be context aware for cancelation signal to
|
||||
actually cancel the processing.
|
||||
|
||||
Example: asynqmon cancel bnogo8gt6toe23vhef0g`,
|
||||
Args: cobra.ExactArgs(1),
|
||||
Run: cancel,
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(cancelCmd)
|
||||
}
|
||||
|
||||
func cancel(cmd *cobra.Command, args []string) {
|
||||
r := rdb.NewRDB(redis.NewClient(&redis.Options{
|
||||
Addr: viper.GetString("uri"),
|
||||
DB: viper.GetInt("db"),
|
||||
Password: viper.GetString("password"),
|
||||
}))
|
||||
|
||||
err := r.PublishCancelation(args[0])
|
||||
if err != nil {
|
||||
fmt.Printf("could not send cancelation signal: %v\n", err)
|
||||
os.Exit(1)
|
||||
}
|
||||
fmt.Printf("Successfully sent cancelation siganl for task %s\n", args[0])
|
||||
}
|
@@ -44,16 +44,24 @@ asynqmon ls enqueued:critical -> List tasks from critical queue
|
||||
}
|
||||
|
||||
// Flags
|
||||
var pageSize uint
|
||||
var pageNum uint
|
||||
var pageSize int
|
||||
var pageNum int
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(lsCmd)
|
||||
lsCmd.Flags().UintVar(&pageSize, "size", 30, "page size")
|
||||
lsCmd.Flags().UintVar(&pageNum, "page", 0, "page number - zero indexed (default 0)")
|
||||
lsCmd.Flags().IntVar(&pageSize, "size", 30, "page size")
|
||||
lsCmd.Flags().IntVar(&pageNum, "page", 0, "page number - zero indexed (default 0)")
|
||||
}
|
||||
|
||||
func ls(cmd *cobra.Command, args []string) {
|
||||
if pageSize < 0 {
|
||||
fmt.Println("page size cannot be negative.")
|
||||
os.Exit(1)
|
||||
}
|
||||
if pageNum < 0 {
|
||||
fmt.Println("page number cannot be negative.")
|
||||
os.Exit(1)
|
||||
}
|
||||
c := redis.NewClient(&redis.Options{
|
||||
Addr: viper.GetString("uri"),
|
||||
DB: viper.GetInt("db"),
|
||||
@@ -184,11 +192,16 @@ func listRetry(r *rdb.RDB) {
|
||||
fmt.Println("No retry tasks")
|
||||
return
|
||||
}
|
||||
cols := []string{"ID", "Type", "Payload", "Retry In", "Last Error", "Retried", "Max Retry", "Queue"}
|
||||
cols := []string{"ID", "Type", "Payload", "Next Retry", "Last Error", "Retried", "Max Retry", "Queue"}
|
||||
printRows := func(w io.Writer, tmpl string) {
|
||||
for _, t := range tasks {
|
||||
retryIn := fmt.Sprintf("%.0f seconds", t.ProcessAt.Sub(time.Now()).Seconds())
|
||||
fmt.Fprintf(w, tmpl, queryID(t.ID, t.Score, "r"), t.Type, t.Payload, retryIn, t.ErrorMsg, t.Retried, t.Retry, t.Queue)
|
||||
var nextRetry string
|
||||
if d := t.ProcessAt.Sub(time.Now()); d > 0 {
|
||||
nextRetry = fmt.Sprintf("in %v", d.Round(time.Second))
|
||||
} else {
|
||||
nextRetry = "right now"
|
||||
}
|
||||
fmt.Fprintf(w, tmpl, queryID(t.ID, t.Score, "r"), t.Type, t.Payload, nextRetry, t.ErrorMsg, t.Retried, t.Retry, t.Queue)
|
||||
}
|
||||
}
|
||||
printTable(cols, printRows)
|
||||
|
@@ -73,7 +73,7 @@ func ps(cmd *cobra.Command, args []string) {
|
||||
printRows := func(w io.Writer, tmpl string) {
|
||||
for _, ps := range processes {
|
||||
fmt.Fprintf(w, tmpl,
|
||||
ps.Host, ps.PID, ps.State,
|
||||
ps.Host, ps.PID, ps.Status,
|
||||
fmt.Sprintf("%d/%d", ps.ActiveWorkerCount, ps.Concurrency),
|
||||
formatQueues(ps.Queues), timeAgo(ps.Started))
|
||||
}
|
||||
@@ -87,11 +87,11 @@ func timeAgo(since time.Time) string {
|
||||
return fmt.Sprintf("%v ago", d)
|
||||
}
|
||||
|
||||
func formatQueues(qmap map[string]uint) string {
|
||||
func formatQueues(qmap map[string]int) string {
|
||||
// sort queues by priority and name
|
||||
type queue struct {
|
||||
name string
|
||||
priority uint
|
||||
priority int
|
||||
}
|
||||
var queues []*queue
|
||||
for qname, p := range qmap {
|
||||
|
75
tools/asynqmon/cmd/workers.go
Normal file
75
tools/asynqmon/cmd/workers.go
Normal file
@@ -0,0 +1,75 @@
|
||||
// Copyright 2020 Kentaro Hibino. All rights reserved.
|
||||
// Use of this source code is governed by a MIT license
|
||||
// that can be found in the LICENSE file.
|
||||
|
||||
package cmd
|
||||
|
||||
import (
|
||||
"fmt"
|
||||
"io"
|
||||
"os"
|
||||
"sort"
|
||||
|
||||
"github.com/go-redis/redis/v7"
|
||||
"github.com/hibiken/asynq/internal/rdb"
|
||||
"github.com/spf13/cobra"
|
||||
"github.com/spf13/viper"
|
||||
)
|
||||
|
||||
// workersCmd represents the workers command
|
||||
var workersCmd = &cobra.Command{
|
||||
Use: "workers",
|
||||
Short: "Shows all running workers information",
|
||||
Long: `Workers (asynqmon workers) will show all running workers information.
|
||||
|
||||
The command shows the follwoing for each worker:
|
||||
* Process in which the worker is running
|
||||
* ID of the task worker is processing
|
||||
* Type of the task worker is processing
|
||||
* Payload of the task worker is processing
|
||||
* Queue that the task was pulled from.
|
||||
* Time the worker started processing the task`,
|
||||
Args: cobra.NoArgs,
|
||||
Run: workers,
|
||||
}
|
||||
|
||||
func init() {
|
||||
rootCmd.AddCommand(workersCmd)
|
||||
}
|
||||
|
||||
func workers(cmd *cobra.Command, args []string) {
|
||||
r := rdb.NewRDB(redis.NewClient(&redis.Options{
|
||||
Addr: viper.GetString("uri"),
|
||||
DB: viper.GetInt("db"),
|
||||
Password: viper.GetString("password"),
|
||||
}))
|
||||
|
||||
workers, err := r.ListWorkers()
|
||||
if err != nil {
|
||||
fmt.Println(err)
|
||||
os.Exit(1)
|
||||
}
|
||||
|
||||
if len(workers) == 0 {
|
||||
fmt.Println("No workers")
|
||||
return
|
||||
}
|
||||
|
||||
// sort by started timestamp or ID.
|
||||
sort.Slice(workers, func(i, j int) bool {
|
||||
x, y := workers[i], workers[j]
|
||||
if x.Started != y.Started {
|
||||
return x.Started.Before(y.Started)
|
||||
}
|
||||
return x.ID.String() < y.ID.String()
|
||||
})
|
||||
|
||||
cols := []string{"Process", "ID", "Type", "Payload", "Queue", "Started"}
|
||||
printRows := func(w io.Writer, tmpl string) {
|
||||
for _, wk := range workers {
|
||||
fmt.Fprintf(w, tmpl,
|
||||
fmt.Sprintf("%s:%d", wk.Host, wk.PID), wk.ID, wk.Type, wk.Payload, wk.Queue, timeAgo(wk.Started))
|
||||
}
|
||||
}
|
||||
printTable(cols, printRows)
|
||||
}
|
14
tools/go.mod
Normal file
14
tools/go.mod
Normal file
@@ -0,0 +1,14 @@
|
||||
module github.com/hibiken/asynq/tools
|
||||
|
||||
go 1.13
|
||||
|
||||
require (
|
||||
github.com/go-redis/redis/v7 v7.2.0
|
||||
github.com/hibiken/asynq v0.4.0
|
||||
github.com/mitchellh/go-homedir v1.1.0
|
||||
github.com/rs/xid v1.2.1
|
||||
github.com/spf13/cobra v0.0.5
|
||||
github.com/spf13/viper v1.6.2
|
||||
)
|
||||
|
||||
replace github.com/hibiken/asynq => ./..
|
195
tools/go.sum
Normal file
195
tools/go.sum
Normal file
@@ -0,0 +1,195 @@
|
||||
cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
|
||||
github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
|
||||
github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
|
||||
github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
|
||||
github.com/alecthomas/units v0.0.0-20151022065526-2efee857e7cf/go.mod h1:ybxpYRFXyAe+OPACYpWeL0wqObRcbAqCMya13uyzqw0=
|
||||
github.com/armon/consul-api v0.0.0-20180202201655-eb2c6b5be1b6/go.mod h1:grANhF5doyWs3UAsr3K4I6qtAmlQcZDesFNEHPZAzj8=
|
||||
github.com/beorn7/perks v0.0.0-20180321164747-3a771d992973/go.mod h1:Dwedo/Wpr24TaqPxmxbtue+5NUziq4I4S80YR8gNf3Q=
|
||||
github.com/beorn7/perks v1.0.0/go.mod h1:KWe93zE9D1o94FZ5RNwFwVgaQK1VOXiVxmqh+CedLV8=
|
||||
github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
|
||||
github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
|
||||
github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
|
||||
github.com/coreos/etcd v3.3.10+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
|
||||
github.com/coreos/go-etcd v2.0.0+incompatible/go.mod h1:Jez6KQU2B/sWsbdaef3ED8NzMklzPG4d5KIOhIy30Tk=
|
||||
github.com/coreos/go-semver v0.2.0/go.mod h1:nnelYz7RCh+5ahJtPPxZlU+153eP4D4r3EedlOD2RNk=
|
||||
github.com/coreos/go-systemd v0.0.0-20190321100706-95778dfbb74e/go.mod h1:F5haX7vjVVG0kc13fIWeqUViNPyEJxv/OmvnBo0Yme4=
|
||||
github.com/coreos/pkg v0.0.0-20180928190104-399ea9e2e55f/go.mod h1:E3G3o1h8I7cfcXa63jLwjI0eiQQMgzzUDFVpN/nH/eA=
|
||||
github.com/cpuguy83/go-md2man v1.0.10/go.mod h1:SmD6nW6nTyfqj6ABTjUi3V3JVMnlJmwcJI5acqYI6dE=
|
||||
github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
|
||||
github.com/dgrijalva/jwt-go v3.2.0+incompatible/go.mod h1:E3ru+11k8xSBh+hMPgOLZmtrrCbhqsmaPHjLKYnJCaQ=
|
||||
github.com/dgryski/go-sip13 v0.0.0-20181026042036-e10d5fee7954/go.mod h1:vAd38F8PWV+bWy6jNmig1y/TA+kYO4g3RSRF0IAv0no=
|
||||
github.com/fsnotify/fsnotify v1.4.7 h1:IXs+QLmnXW2CcXuY+8Mzv/fWEsPGWxqefPtCP5CnV9I=
|
||||
github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
|
||||
github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
|
||||
github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
|
||||
github.com/go-logfmt/logfmt v0.3.0/go.mod h1:Qt1PoO58o5twSAckw1HlFXLmHsOX5/0LbT9GBnD5lWE=
|
||||
github.com/go-logfmt/logfmt v0.4.0/go.mod h1:3RMwSq7FuexP4Kalkev3ejPJsZTpXXBr9+V4qmtdjCk=
|
||||
github.com/go-redis/redis v6.15.7+incompatible h1:3skhDh95XQMpnqeqNftPkQD9jL9e5e36z/1SUm6dy1U=
|
||||
github.com/go-redis/redis/v7 v7.0.0-beta.4/go.mod h1:xhhSbUMTsleRPur+Vgx9sUHtyN33bdjxY+9/0n9Ig8s=
|
||||
github.com/go-redis/redis/v7 v7.1.0 h1:I4C4a8UGbFejiVjtYVTRVOiMIJ5pm5Yru6ibvDX/OS0=
|
||||
github.com/go-redis/redis/v7 v7.1.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
|
||||
github.com/go-redis/redis/v7 v7.2.0 h1:CrCexy/jYWZjW0AyVoHlcJUeZN19VWlbepTh1Vq6dJs=
|
||||
github.com/go-redis/redis/v7 v7.2.0/go.mod h1:JDNMw23GTyLNC4GZu9njt15ctBQVn7xjRfnwdHj/Dcg=
|
||||
github.com/go-stack/stack v1.8.0/go.mod h1:v0f6uXyyMGvRgIKkXu+yp6POWl0qKG85gN/melR3HDY=
|
||||
github.com/gogo/protobuf v1.1.1/go.mod h1:r8qH/GZQm5c6nD/R0oafs1akxWv10x8SbQlK7atdtwQ=
|
||||
github.com/gogo/protobuf v1.2.1/go.mod h1:hp+jE20tsWTFYpLwKvXlhS1hjn+gTNwPg2I6zVXpSg4=
|
||||
github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
|
||||
github.com/golang/groupcache v0.0.0-20190129154638-5b532d6fd5ef/go.mod h1:cIg4eruTrX1D+g88fzRXU5OdNfaM+9IcxsU14FzY7Hc=
|
||||
github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
|
||||
github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.1/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
|
||||
github.com/google/btree v1.0.0/go.mod h1:lNA+9X1NB3Zf8V7Ke586lFgjr2dZNuvo3lPJSGZ5JPQ=
|
||||
github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
|
||||
github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
|
||||
github.com/gopherjs/gopherjs v0.0.0-20181017120253-0766667cb4d1/go.mod h1:wJfORRmW1u3UXTncJ5qlYoELFm8eSnnEO6hX4iZ3EWY=
|
||||
github.com/gorilla/websocket v1.4.0/go.mod h1:E7qHFY5m1UJ88s3WnNqhKjPHQ0heANvMoAMk2YaljkQ=
|
||||
github.com/grpc-ecosystem/go-grpc-middleware v1.0.0/go.mod h1:FiyG127CGDf3tlThmgyCl78X/SZQqEOJBCDaAfeWzPs=
|
||||
github.com/grpc-ecosystem/go-grpc-prometheus v1.2.0/go.mod h1:8NvIoxWQoOIhqOTXgfV/d3M/q6VIi02HzZEHgUlZvzk=
|
||||
github.com/grpc-ecosystem/grpc-gateway v1.9.0/go.mod h1:vNeuVxBJEsws4ogUvrchl83t/GYV9WGTSLVdBhOQFDY=
|
||||
github.com/hashicorp/hcl v1.0.0 h1:0Anlzjpi4vEasTeNFn2mLJgTSwt0+6sfsiTG8qcWGx4=
|
||||
github.com/hashicorp/hcl v1.0.0/go.mod h1:E5yfLk+7swimpb2L/Alb/PJmXilQ/rhwaUYs4T20WEQ=
|
||||
github.com/hibiken/asynq v0.4.0 h1:NvAfYX0DRe04WgGMKRg5oX7bs6ktv2fu9YwB6O356FI=
|
||||
github.com/hibiken/asynq v0.4.0/go.mod h1:dtrVkxCsGPVhVNHMDXAH7lFq64kbj43+G6lt4FQZfW4=
|
||||
github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
|
||||
github.com/inconshreveable/mousetrap v1.0.0/go.mod h1:PxqpIevigyE2G7u3NXJIT2ANytuPF1OarO4DADm73n8=
|
||||
github.com/jonboulle/clockwork v0.1.0/go.mod h1:Ii8DK3G1RaLaWxj9trq07+26W01tbo22gdxWY5EU2bo=
|
||||
github.com/jtolds/gls v4.20.0+incompatible/go.mod h1:QJZ7F/aHp+rZTRtaJ1ow/lLfFfVYBRgL+9YlvaHOwJU=
|
||||
github.com/julienschmidt/httprouter v1.2.0/go.mod h1:SYymIcj16QtmaHHD7aYtjjsJG7VTCxuUUipMqKk8s4w=
|
||||
github.com/kisielk/errcheck v1.1.0/go.mod h1:EZBBE59ingxPouuu3KfxchcWSUPOHkagtvWXihfKN4Q=
|
||||
github.com/kisielk/gotool v1.0.0/go.mod h1:XhKaO+MFFWcvkIS/tQcRk01m1F5IRFswLeQ+oQHNcck=
|
||||
github.com/konsorten/go-windows-terminal-sequences v1.0.1/go.mod h1:T0+1ngSBFLxvqU3pZ+m/2kptfBszLMUkC4ZK/EgS/cQ=
|
||||
github.com/kr/logfmt v0.0.0-20140226030751-b84e30acd515/go.mod h1:+0opPa2QZZtGFBFZlji/RkVcI2GknAs/DXo4wKdlNEc=
|
||||
github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
|
||||
github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
|
||||
github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
|
||||
github.com/magiconair/properties v1.8.0/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
|
||||
github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
|
||||
github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
|
||||
github.com/mitchellh/go-homedir v1.1.0 h1:lukF9ziXFxDFPkA1vsr5zpc1XuPDn/wFntq5mG+4E0Y=
|
||||
github.com/mitchellh/go-homedir v1.1.0/go.mod h1:SfyaCUpYCn1Vlf4IUYiD9fPX4A5wJrkLzIz1N1q0pr0=
|
||||
github.com/mitchellh/mapstructure v1.1.2 h1:fmNYVwqnSfB9mZU6OS2O6GsXM+wcskZDuKQzvN1EDeE=
|
||||
github.com/mitchellh/mapstructure v1.1.2/go.mod h1:FVVH3fgwuzCH5S8UJGiWEs2h04kUh9fWfEaFds41c1Y=
|
||||
github.com/mwitkow/go-conntrack v0.0.0-20161129095857-cc309e4a2223/go.mod h1:qRWi+5nqEBWmkhHvq77mSJWrCKwh8bxhgT7d/eI7P4U=
|
||||
github.com/oklog/ulid v1.3.1/go.mod h1:CirwcVhetQ6Lv90oh/F+FBtV6XMibvdAFo93nm5qn4U=
|
||||
github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.8.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/ginkgo v1.10.1/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
|
||||
github.com/onsi/gomega v1.5.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/onsi/gomega v1.7.0/go.mod h1:ex+gbHU/CVuBBDIJjb2X0qEXbFg53c61hWP/1CpauHY=
|
||||
github.com/pelletier/go-toml v1.2.0 h1:T5zMGML61Wp+FlcbWjRDT7yAxhJNAiPPLOFECq181zc=
|
||||
github.com/pelletier/go-toml v1.2.0/go.mod h1:5z9KED0ma1S8pY6P1sdut58dfprrGBbd/94hg7ilaic=
|
||||
github.com/pelletier/go-toml v1.6.0 h1:aetoXYr0Tv7xRU/V4B4IZJ2QcbtMUFoNb3ORp7TzIK4=
|
||||
github.com/pelletier/go-toml v1.6.0/go.mod h1:5N711Q9dKgbdkxHL+MEfF31hpT7l0S0s/t2kKREewys=
|
||||
github.com/pkg/errors v0.8.0/go.mod h1:bwawxfHBFNV+L2hUp1rHADufV3IMtnDRdf1r5NINEl0=
|
||||
github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
|
||||
github.com/prometheus/client_golang v0.9.1/go.mod h1:7SWBe2y4D6OKWSNQJUaRYU/AaXPKyh/dDVn+NZz0KFw=
|
||||
github.com/prometheus/client_golang v0.9.3/go.mod h1:/TN21ttK/J9q6uSwhBd54HahCDft0ttaMvbicHlPoso=
|
||||
github.com/prometheus/client_model v0.0.0-20180712105110-5c3871d89910/go.mod h1:MbSGuTsp3dbXC40dX6PRTWyKYBIrTGTE9sqQNg2J8bo=
|
||||
github.com/prometheus/client_model v0.0.0-20190129233127-fd36f4220a90/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
|
||||
github.com/prometheus/common v0.0.0-20181113130724-41aa239b4cce/go.mod h1:daVV7qP5qjZbuso7PdcryaAu0sAZbrN9i7WWcTMWvro=
|
||||
github.com/prometheus/common v0.4.0/go.mod h1:TNfzLD0ON7rHzMJeJkieUDPYmFC7Snx/y86RQel1bk4=
|
||||
github.com/prometheus/procfs v0.0.0-20181005140218-185b4288413d/go.mod h1:c3At6R/oaqEKCNdg8wHV1ftS6bRYblBhIjjI8uT2IGk=
|
||||
github.com/prometheus/procfs v0.0.0-20190507164030-5867b95ac084/go.mod h1:TjEm7ze935MbeOT/UhFTIMYKhuLP4wbCsTZCD3I8kEA=
|
||||
github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
|
||||
github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
|
||||
github.com/rs/xid v1.2.1 h1:mhH9Nq+C1fY2l1XIpgxIiUOfNpRBYH1kKcr+qfKgjRc=
|
||||
github.com/rs/xid v1.2.1/go.mod h1:+uKXf+4Djp6Md1KODXJxgGQPKngRmWyn10oCKFzNHOQ=
|
||||
github.com/russross/blackfriday v1.5.2/go.mod h1:JO/DiYxRf+HjHt06OyowR9PTA263kcR/rfWxYHBV53g=
|
||||
github.com/sirupsen/logrus v1.2.0/go.mod h1:LxeOpSwHxABJmUn/MG1IvRgCAasNZTLOkJPxbbu5VWo=
|
||||
github.com/smartystreets/assertions v0.0.0-20180927180507-b2de0cb4f26d/go.mod h1:OnSkiWE9lh6wB0YB77sQom3nweQdgAjqCqsofrRNTgc=
|
||||
github.com/smartystreets/goconvey v1.6.4/go.mod h1:syvi0/a8iFYH4r/RixwvyeAJjdLS9QV7WQ/tjFTllLA=
|
||||
github.com/soheilhy/cmux v0.1.4/go.mod h1:IM3LyeVVIOuxMH7sFAkER9+bJ4dT7Ms6E4xg4kGIyLM=
|
||||
github.com/spaolacci/murmur3 v0.0.0-20180118202830-f09979ecbc72/go.mod h1:JwIasOWyU6f++ZhiEuf87xNszmSA2myDM2Kzu9HwQUA=
|
||||
github.com/spf13/afero v1.1.2 h1:m8/z1t7/fwjysjQRYbP0RD+bUIF/8tJwPdEZsI83ACI=
|
||||
github.com/spf13/afero v1.1.2/go.mod h1:j4pytiNVoe2o6bmDsKpLACNPDBIoEAkihy7loJ1B0CQ=
|
||||
github.com/spf13/afero v1.2.2 h1:5jhuqJyZCZf2JRofRvN/nIFgIWNzPa3/Vz8mYylgbWc=
|
||||
github.com/spf13/afero v1.2.2/go.mod h1:9ZxEEn6pIJ8Rxe320qSDBk6AsU0r9pR7Q4OcevTdifk=
|
||||
github.com/spf13/cast v1.3.0/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
|
||||
github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
|
||||
github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
|
||||
github.com/spf13/cobra v0.0.5 h1:f0B+LkLX6DtmRH1isoNA9VTtNUK9K8xYd28JNNfOv/s=
|
||||
github.com/spf13/cobra v0.0.5/go.mod h1:3K3wKZymM7VvHMDS9+Akkh4K60UwM26emMESw8tLCHU=
|
||||
github.com/spf13/jwalterweatherman v1.0.0 h1:XHEdyB+EcvlqZamSM4ZOMGlc93t6AcsBEu9Gc1vn7yk=
|
||||
github.com/spf13/jwalterweatherman v1.0.0/go.mod h1:cQK4TGJAtQXfYWX+Ddv3mKDzgVb68N+wFjFa4jdeBTo=
|
||||
github.com/spf13/jwalterweatherman v1.1.0 h1:ue6voC5bR5F8YxI5S67j9i582FU4Qvo2bmqnqMYADFk=
|
||||
github.com/spf13/jwalterweatherman v1.1.0/go.mod h1:aNWZUN0dPAAO/Ljvb5BEdw96iTZ0EXowPYD95IqWIGo=
|
||||
github.com/spf13/pflag v1.0.3 h1:zPAT6CGy6wXeQ7NtTnaTerfKOsV6V6F8agHXFiazDkg=
|
||||
github.com/spf13/pflag v1.0.3/go.mod h1:DYY7MBk1bdzusC3SYhjObp+wFpr4gzcvqqNjLnInEg4=
|
||||
github.com/spf13/pflag v1.0.5 h1:iy+VFUOCP1a+8yFto/drg2CJ5u0yRoB7fZw3DKv/JXA=
|
||||
github.com/spf13/pflag v1.0.5/go.mod h1:McXfInJRrz4CZXVZOBLb0bTZqETkiAhM9Iw0y3An2Bg=
|
||||
github.com/spf13/viper v1.3.2/go.mod h1:ZiWeW+zYFKm7srdB9IoDzzZXaJaI5eL9QjNiN/DMA2s=
|
||||
github.com/spf13/viper v1.6.0/go.mod h1:t3iDnF5Jlj76alVNuyFBk5oUMCvsrkbvZK0WQdfDi5k=
|
||||
github.com/spf13/viper v1.6.2 h1:7aKfF+e8/k68gda3LOjo5RxiUqddoFxVq4BKBPrxk5E=
|
||||
github.com/spf13/viper v1.6.2/go.mod h1:t3iDnF5Jlj76alVNuyFBk5oUMCvsrkbvZK0WQdfDi5k=
|
||||
github.com/stretchr/objx v0.1.1/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
|
||||
github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
|
||||
github.com/subosito/gotenv v1.2.0 h1:Slr1R9HxAlEKefgq5jn9U+DnETlIUa6HfgEzj0g5d7s=
|
||||
github.com/subosito/gotenv v1.2.0/go.mod h1:N0PQaV/YGNqwC0u51sEeR/aUtSLEXKX9iv69rRypqCw=
|
||||
github.com/tmc/grpc-websocket-proxy v0.0.0-20190109142713-0ad062ec5ee5/go.mod h1:ncp9v5uamzpCO7NfCPTXjqaC+bZgJeR0sMTm6dMHP7U=
|
||||
github.com/ugorji/go v1.1.4/go.mod h1:uQMGLiO92mf5W77hV/PUCpI3pbzQx3CRekS0kk+RGrc=
|
||||
github.com/ugorji/go/codec v0.0.0-20181204163529-d75b2dcb6bc8/go.mod h1:VFNgLljTbGfSG7qAOspJ7OScBnGdDN/yBr0sguwnwf0=
|
||||
github.com/xiang90/probing v0.0.0-20190116061207-43a291ad63a2/go.mod h1:UETIi67q53MR2AWcXfiuqkDkRtnGDLqkBTpCHuJHxtU=
|
||||
github.com/xordataexchange/crypt v0.0.3-0.20170626215501-b2862e3d0a77/go.mod h1:aYKd//L2LvnjZzWKhF00oedf4jCCReLcmhLdhm1A27Q=
|
||||
go.etcd.io/bbolt v1.3.2/go.mod h1:IbVyRI1SCnLcuJnV2u8VeU0CEYM7e686BmAb1XKL+uU=
|
||||
go.uber.org/atomic v1.4.0/go.mod h1:gD2HeocX3+yG+ygLZcrzQJaqmWj9AIm7n08wl/qW/PE=
|
||||
go.uber.org/goleak v0.10.0/go.mod h1:VCZuO8V8mFPlL0F5J5GK1rtHV3DrFcQ1R8ryq7FK0aI=
|
||||
go.uber.org/multierr v1.1.0/go.mod h1:wR5kodmAFQ0UK8QlbwjlSNy0Z68gJhDJUG5sjR94q/0=
|
||||
go.uber.org/zap v1.10.0/go.mod h1:vwi/ZaCAaUcBkycHslxD9B2zi4UTXhF60s6SWpuDF0Q=
|
||||
golang.org/x/crypto v0.0.0-20180904163835-0709b304e793/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20181203042331-505ab145d0a9/go.mod h1:6SG95UA2DQfeDnfUPMdvaQW0Q7yPrPDi9nlGo2tz2b4=
|
||||
golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
|
||||
golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
|
||||
golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
|
||||
golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181114220301-adae6a3d119a/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20181220203305-927f97764cc3/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
|
||||
golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
|
||||
golang.org/x/net v0.0.0-20190522155817-f3200d17e092/go.mod h1:HSz+uSET+XFnRR8LxR5pz3Of3rY3CfYBVs4xY44aLks=
|
||||
golang.org/x/net v0.0.0-20190923162816-aa69164e4478/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
|
||||
golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
|
||||
golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sync v0.0.0-20181221193216-37e7f081c4d4/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
|
||||
golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180905080454-ebe1bf3edb33/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181107165924-66b7b1311ac8/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181116152217-5ac8a444bdc5/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20181205085412-a5c9d58dba9a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
|
||||
golang.org/x/sys v0.0.0-20191010194322-b09406accb47/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e h1:9vRrk9YW2BTzLP0VCB9ZDjU4cPqkg+IDWL7XgxA1yxQ=
|
||||
golang.org/x/sys v0.0.0-20191204072324-ce4227a45e2e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
|
||||
golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
|
||||
golang.org/x/text v0.3.2 h1:tW2bmiBqwgJj/UpqtC8EpXEZVYOwU0yG4iWbprSVAcs=
|
||||
golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
|
||||
golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
|
||||
golang.org/x/tools v0.0.0-20180221164845-07fd8470d635/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
|
||||
golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/tools v0.0.0-20190328211700-ab21143f2384/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
|
||||
golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
|
||||
google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
|
||||
google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
|
||||
google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
|
||||
google.golang.org/grpc v1.21.0/go.mod h1:oYelfM1adQP15Ek0mdvEgi9Df8B9CZIaU1084ijfRaM=
|
||||
gopkg.in/alecthomas/kingpin.v2 v2.2.6/go.mod h1:FMv+mEhP44yOT+4EoQTLFTRgOQ1FBLkstjWtayDeSgw=
|
||||
gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
|
||||
gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
|
||||
gopkg.in/ini.v1 v1.51.0 h1:AQvPpx3LzTDM0AjnIRlVFwFFGC+npRopjZxLJj6gdno=
|
||||
gopkg.in/ini.v1 v1.51.0/go.mod h1:pNLf8WUiyNEtQjuu5G5vTm06TEv9tsIgeAvK8hOrP4k=
|
||||
gopkg.in/resty.v1 v1.12.0/go.mod h1:mDo4pnntr5jdWRML875a/NmxYqAlA73dVijT2AXvQQo=
|
||||
gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
|
||||
gopkg.in/yaml.v2 v2.0.0-20170812160011-eb3733d160e7/go.mod h1:JAlM8MvJe8wmxCU4Bli9HhUf9+ttbYbLASfIpnQbh74=
|
||||
gopkg.in/yaml.v2 v2.2.1/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.2/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
gopkg.in/yaml.v2 v2.2.7 h1:VUgggvou5XRW9mHwD/yXxIYSMtY0zoKQf/v226p2nyo=
|
||||
gopkg.in/yaml.v2 v2.2.7/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
|
||||
honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
|
Reference in New Issue
Block a user