2
0
mirror of https://github.com/hibiken/asynq.git synced 2024-12-24 23:02:18 +08:00
golang基于redis的异步队列
Go to file
Ken Hibino 97316d6766 Fix flaky tests
Some tests were failing due to mismatch in Score in ZSetEntry.
Changed ZSetEntry Score to float64 type so that we can use
cmpopts.EquateApprox to allow for margin when comparing.
2020-01-11 10:09:15 -08:00
.github/ISSUE_TEMPLATE Update issue templates 2019-12-27 10:45:45 -08:00
internal Fix flaky tests 2020-01-11 10:09:15 -08:00
tools/asynqmon Allow filtering results of asynqmon ls enqueued by providing queue 2020-01-11 10:09:15 -08:00
.gitignore Change cmd dir to tools 2019-12-06 22:06:59 -08:00
.travis.yml Add .travis.yml 2019-11-30 09:08:14 -08:00
asynq_test.go Add license comment to all src files 2020-01-02 18:13:16 -08:00
asynq.go Make Task type immutable 2020-01-05 09:55:39 -08:00
background_test.go [ci skip] Normalize queue priority numbers 2020-01-07 21:55:18 -08:00
background.go [ci skip] Update changelog 2020-01-07 21:55:18 -08:00
benchmark_test.go Make Task type immutable 2020-01-05 09:55:39 -08:00
CHANGELOG.md Include each queue counts in stats command output 2020-01-11 10:09:15 -08:00
client_test.go Fix flaky tests 2020-01-11 10:09:15 -08:00
client.go Update comment 2020-01-07 21:55:18 -08:00
doc.go Make Task type immutable 2020-01-05 09:55:39 -08:00
go.mod [ci skip] Upgrade github.com/google/go-cmp to v0.4.0 2020-01-07 21:55:18 -08:00
go.sum [ci skip] Upgrade github.com/google/go-cmp to v0.4.0 2020-01-07 21:55:18 -08:00
LICENSE Add MIT License 2019-11-30 10:21:25 -08:00
payload_test.go Add test for payload key not exist 2020-01-05 09:55:39 -08:00
payload.go Make Task type immutable 2020-01-05 09:55:39 -08:00
processor_test.go Fix flaky tests 2020-01-11 10:09:15 -08:00
processor.go Remove stale field in processor struct 2020-01-11 10:09:15 -08:00
README.md [ci skip] Update readme 2020-01-05 09:55:39 -08:00
scheduler_test.go Fix flaky tests 2020-01-11 10:09:15 -08:00
scheduler.go Add license comment to all src files 2020-01-02 18:13:16 -08:00

Asynq

Build Status License: MIT

Simple, efficent asynchronous task processing library in Go.

Table of Contents

Overview

Asynq provides a simple interface to asynchronous task processing.

Asynq also ships with a CLI to monitor the queues and take manual actions if needed.

Asynq provides:

  • Clear separation of task producer and consumer
  • Ability to schedule task processing in the future
  • Automatic retry of failed tasks with exponential backoff
  • Ability to configure max retry count per task
  • Ability to configure max number of worker goroutines to process tasks
  • Unix signal handling to safely shutdown background processing
  • Enhanced reliability TODO(hibiken): link to wiki page describing this.
  • CLI to query and mutate queues state for mointoring and administrative purposes

Requirements

Dependency Version
Redis v2.6+
Go v1.12+
github.com/go-redis/redis v.7.0+

Installation

go get github.com/hibiken/asynq

Getting Started

  1. Import asynq and redis in your file.
import (
    "github.com/go-redis/redis/v7"
    "github.com/hibiken/asynq"
)
  1. Create a Client instance to create tasks.
func main() {
    r := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    }
    client := asynq.NewClient(r)

    // create a task with typename and payload.
    t1 := asynq.NewTask(
        "send_welcome_email",
        map[string]interface{}{"user_id": 42})

    t2 := asynq.NewTask(
        "send_reminder_email",
        map[string]interface{}{"user_id": 42})

    // process the task immediately.
    err := client.Schedule(t1, time.Now())

    // process the task 24 hours later.
    err = client.Schedule(t2, time.Now().Add(24 * time.Hour))

    // specify the max number of retry (default: 25)
    err = client.Schedule(t1, time.Now(), asynq.MaxRetry(1))
}
  1. Create a Background instance to process tasks.
func main() {
    r := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    }
    bg := asynq.NewBackground(r, &asynq.Config{
        Concurrency: 20,
    })

    // Blocks until signal TERM or INT is received.
    // For graceful shutdown, send signal TSTP to stop processing more tasks
    // before sending TERM or INT signal.
    bg.Run(handler)
}

The argument to (*asynq.Background).Run is an interface asynq.Handler which has one method ProcessTask.

// ProcessTask should return nil if the processing of a task
// is successful.
//
// If ProcessTask return a non-nil error or panics, the task
// will be retried.
type Handler interface {
    ProcessTask(*Task) error
}

The simplest way to implement a handler is to define a function with the same signature and use asynq.HandlerFunc adapter type when passing it to Run.

func handler(t *asynq.Task) error {
    switch t.Type {
    case "send_welcome_email":
        id, err := t.Payload.GetInt("user_id")
        if err != nil {
            return err
        }
        fmt.Printf("Send Welcome Email to %d\n", id)

    // ... handle other types ...

    default:
        return fmt.Errorf("unexpected task type: %s", t.Type)
    }
    return nil
}

func main() {
    r := redis.NewClient(&redis.Options{
        Addr: "localhost:6379",
    }
    bg := asynq.NewBackground(r, &asynq.Config{
        Concurrency: 20,
    })

    // Use asynq.HandlerFunc adapter for a handler function
    bg.Run(asynq.HandlerFunc(handler))
}

Acknowledgements

  • Sidekiq : Many of the design ideas are taken from sidekiq and its Web UI
  • Cobra : Asynqmon CLI is built with cobra

License

Asynq is released under the MIT license. See LICENSE.