go-pkg/asynq
2
0
Fork 0
mirror of https://github.com/hibiken/asynq.git synced 2024-09-20 11:05:58 +08:00
Code Issues Projects Releases Wiki Activity

[ci skip] Update readme

Browse Source
This commit is contained in:
Ken Hibino 2020-01-23 06:33:34 -08:00
parent 58d2ed94e7
commit 3ed155b45b
4 changed files with 139 additions and 46 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

103
README.md
View File

@ -6,9 +6,9 @@
[![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq)
[![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community)
Simple and efficent asynchronous task processing library in Go.
Simple and efficient asynchronous task processing library in Go.
**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before the release of verson 1.0.0.
**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before v1.0.0 release.
## Table of Contents
@ -16,7 +16,7 @@ Simple and efficent asynchronous task processing library in Go.
- [Requirements](#requirements)
- [Installation](#installation)
- [Getting Started](#getting-started)
- [Monitoring CLI](#monitoring-cli)
- [Command Line Tool](#command-line-tool)
- [Acknowledgements](#acknowledgements)
- [License](#license)
@ -24,21 +24,19 @@ Simple and efficent asynchronous task processing library in Go.
![Gif](/docs/assets/asynqmon_stats.gif)
Asynq provides a simple interface to asynchronous task processing.
It also ships with a tool to monitor the queues and take manual actions if needed.
Package asynq provides a framework for asynchronous task processing.
Asynq provides:
- Clear separation of task producer and consumer
- Ability to process multiple tasks concurrently
- Ability to schedule task processing in the future
- Automatic retry of failed tasks with exponential backoff
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) max retry count per task
- Ability to configure max number of worker goroutines to process tasks
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) task retry count and retry delay
- Support for [priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)
- [Unix signal handling](https://github.com/hibiken/asynq/wiki/Signals) to gracefully shutdown background processing
- [CLI tool](/tools/asynqmon/README.md) to query and mutate queues state for mointoring and administrative purposes
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
- [Command line tool](/tools/asynqmon/README.md) to query tasks for monitoring and troubleshooting purposes
## Requirements
@ -49,7 +47,7 @@ Asynq provides:
## Installation
To install both `asynq` library and `asynqmon` CLI tool, run the following command:
To install both `asynq` library and `asynqmon` command line tool, run the following command:
```
go get -u github.com/hibiken/asynq
@ -66,15 +64,22 @@ In this quick tour of `asynq`, we are going to create two programs.
**This guide assumes that you are running a Redis server at `localhost:6379`**.
Before we start, make sure you have Redis installed and running.
1. Import `asynq` in both files.
The first thing we need to do is create two main files.
```sh
mkdir producer consumer
touch producer/producer.go consumer/consumer.go
```
Import `asynq` in both files.
```go
import "github.com/hibiken/asynq"
```
2. Asynq uses Redis as a message broker.
Use one of `RedisConnOpt` types to specify how to connect to Redis.
We are going to use `RedisClientOpt` here.
Asynq uses Redis as a message broker.
Use one of `RedisConnOpt` types to specify how to connect to Redis.
We are going to use `RedisClientOpt` here.
```go
// both in producer.go and consumer.go
@ -88,7 +93,25 @@ var redis = &asynq.RedisClientOpt{
}
```
3. In `producer.go`, create a `Client` instance to create and schedule tasks.
In `producer.go`, we are going to create a `Client` instance to create and schedule tasks.
In `asynq`, a unit of work to be performed is encapsluated in a struct called `Task`.
Which has two fields: `Type` and `Payload`.
```go
// Task represents a task to be performed.
type Task struct {
// Type indicates the type of task to be performed.
Type string
// Payload holds data needed to perform the task.
Payload Payload
}
```
To create a task, use `NewTask` function and pass type and payload for the task.
You schedule a task by calling `Client.Schedule` passing in the task and the timethe task neeeds to be processed.
```go
// producer.go
@ -118,7 +141,13 @@ func main() {
}
```
4. In `consumer.go`, create a `Background` instance to process tasks.
In `consumer.go`, create a `Background` instance to process the tasks.
`NewBackground` function takes `RedisConnOpt` and `Config`.
You can take a look at documentation on `Config` to see the available options.
We are only going to specify the concurrency in this example.
```go
// consumer.go
@ -239,19 +268,11 @@ func sendReminderEmail(t *asynq.Task) error {
Now that we have both task producer and consumer, we can run both programs.
```sh
go run consumer.go
```
**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.
With our consumer running, also run
```sh
go run producer.go
```
This will create a task and the first task will get processed immediately by the consumer. The second task will be processed 24 hours later.
This will create two tasks: One that should processed immediately and another to be processed 24 hours later.
Let's use `asynqmon` tool to inspect the tasks.
@ -259,23 +280,37 @@ Let's use `asynqmon` tool to inspect the tasks.
asynqmon stats
```
This command will show the number of tasks in each state and stats for the current date as well as redis information.
You should able to see that there's one task in **Enqueued** state and another in **Scheduled** state.
To understand the meaning of each state, see [Life of a Task Wiki page](https://github.com/hibiken/asynq/wiki/Life-of-a-Task).
Note: To understand the meaning of each state, see [Life of a Task](https://github.com/hibiken/asynq/wiki/Life-of-a-Task) on our Wiki page.
For in-depth guide on `asynqmon` tool, see the [README](/tools/asynqmon/README.md) for the CLI.
Let's run `asynqmon` with `watch` command so that we can continuously run the command to see the changes.
This was a quick tour of `asynq` basics. To see all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see [the Wiki page](https://github.com/hibiken/asynq/wiki).
```sh
watch -n 3 asynqmon stats # Runs `asynqmon stats` every 3 seconds
```
## Monitoring CLI
And finally, let's start the consumer program to process scheduled tasks.
Asynq ships with a CLI tool to inspect the state of queues and tasks.
```sh
go run consumer.go
```
To install the CLI, run the following command:
**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.
You should be able to see text printed in your terminal indicating that the task was processed successfully.
This was a whirlwind tour of `asynq` basics. To learn more about all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see our [Wiki page](https://github.com/hibiken/asynq/wiki).
## Command Line Tool
Asynq ships with a command line tool to inspect the state of queues and tasks.
To install, run the following command:
go get github.com/hibiken/asynq/tools/asynqmon
For details on how to use the tool, see the [README](/tools/asynqmon/README.md) for the asynqmon CLI.
For details on how to use the tool, refer to the tool's [README](/tools/asynqmon/README.md).
## Acknowledgements

2
doc.go
View File

@ -3,7 +3,7 @@
// that can be found in the LICENSE file.
/*
Package asynq provides a framework for background task processing.
Package asynq provides a framework for asynchronous task processing.
Asynq uses Redis as a message broker. To connect to redis server,
specify the options using one of RedisConnOpt types.

BIN
docs/assets/asynqmon_history.gif Normal file
View File

Binary file not shown.
Side by Side

After

Width:  |  Height:  |  Size: 1.5 MiB

80
tools/asynqmon/README.md
View File

@ -1,6 +1,6 @@
# Asynqmon
Asynqmon is a CLI tool to monitor the queues managed by `asynq` package.
Asynqmon is a command line tool to monitor the tasks managed by `asynq` package.
## Table of Contents
@ -24,7 +24,7 @@ This will create the asynqmon executable under your `$GOPATH/bin` directory.
## Quick Start
Asynqmon tool has a few commands to inspect the state of tasks and queues.
The tool has a few commands to inspect the state of tasks and queues.
Run `asynqmon help` to see all the available commands.
@ -34,7 +34,7 @@ By default, Asynqmon will try to connect to a redis server running at `localhost
### Stats
Stats command gives the overview of the current state of tasks and queues. Run it in conjunction with `watch` command to repeatedly run `stats`.
Stats command gives the overview of the current state of tasks and queues. You can run it in conjunction with `watch` command to repeatedly run `stats`.
Example:
@ -46,35 +46,93 @@ This will run `asynqmon stats` command every 3 seconds.
### History
TODO: Add discription
History command shows the number of processed and failed tasks from the last x days.
By default, it shows the stats from the last 10 days. Use `--days` to specify the number of days.
Example:
asynqmon history --days=30
![Gif](/docs/assets/asynqmon_history.gif)
### List
TODO: Add discription
List command shows all tasks in the specified state in a table format
Example:
asynqmon ls retry
asynqmon ls scheduled
asynqmon ls dead
asynqmon ls enqueued
asynqmon ls inprogress
### Enqueue
TODO: Add discription
There are two commands to enqueue tasks.
Command `enq` takes a task ID and moves the task to **Enqueued** state. You can obtain the task ID by running `ls` command.
Example:
asynqmon enq d:1575732274:bnogo8gt6toe23vhef0g
Command `enqall` moves all tasks to **Enqueued** state from the specified state.
Example:
asynqmon enqall retry
Running the above command will move all **Retry** tasks to **Enqueued** state.
### Delete
TODO: Add discription
There are two commands for task deletion.
Command `del` takes a task ID and deletes the task. You can obtain the task ID by running `ls` command.
Example:
asynqmon del r:1575732274:bnogo8gt6toe23vhef0g
Command `delall` deletes all tasks which are in the specified state.
Example:
asynqmon delall retry
Running the above command will delete all **Retry** tasks.
### Kill
TODO: Add discription
There are two commands to kill (i.e. move to dead state) tasks.
Command `kill` takes a task ID and kills the task. You can obtain the task ID by running `ls` command.
Example:
asynqmon kill r:1575732274:bnogo8gt6toe23vhef0g
Command `killall` kills all tasks which are in the specified state.
Example:
asynqmon killall retry
Running the above command will move all **Retry** tasks to **Dead** state.
## Config File
You can use a config file to set default values for flags.
You can use a config file to set default values for the flags.
This is useful, for example when you have to connect to a remote redis server.
By default, `asynqmon` will try to read config file located in
`$HOME/.asynqmon.(yml|json)`. You can specify the file location via `--config` flag.
`$HOME/.asynqmon.(yaml|json)`. You can specify the file location via `--config` flag.
Config file example:
```yml
```yaml
uri: 127.0.0.1:6379
db: 2
password: mypassword