mirror of
https://github.com/hibiken/asynq.git
synced 2024-12-25 07:12:17 +08:00
[ci skip] Update readme
This commit is contained in:
parent
58d2ed94e7
commit
3ed155b45b
103
README.md
103
README.md
@ -6,9 +6,9 @@
|
|||||||
[![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq)
|
[![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq)
|
||||||
[![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community)
|
[![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community)
|
||||||
|
|
||||||
Simple and efficent asynchronous task processing library in Go.
|
Simple and efficient asynchronous task processing library in Go.
|
||||||
|
|
||||||
**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before the release of verson 1.0.0.
|
**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before v1.0.0 release.
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
||||||
@ -16,7 +16,7 @@ Simple and efficent asynchronous task processing library in Go.
|
|||||||
- [Requirements](#requirements)
|
- [Requirements](#requirements)
|
||||||
- [Installation](#installation)
|
- [Installation](#installation)
|
||||||
- [Getting Started](#getting-started)
|
- [Getting Started](#getting-started)
|
||||||
- [Monitoring CLI](#monitoring-cli)
|
- [Command Line Tool](#command-line-tool)
|
||||||
- [Acknowledgements](#acknowledgements)
|
- [Acknowledgements](#acknowledgements)
|
||||||
- [License](#license)
|
- [License](#license)
|
||||||
|
|
||||||
@ -24,21 +24,19 @@ Simple and efficent asynchronous task processing library in Go.
|
|||||||
|
|
||||||
![Gif](/docs/assets/asynqmon_stats.gif)
|
![Gif](/docs/assets/asynqmon_stats.gif)
|
||||||
|
|
||||||
Asynq provides a simple interface to asynchronous task processing.
|
Package asynq provides a framework for asynchronous task processing.
|
||||||
|
|
||||||
It also ships with a tool to monitor the queues and take manual actions if needed.
|
|
||||||
|
|
||||||
Asynq provides:
|
Asynq provides:
|
||||||
|
|
||||||
- Clear separation of task producer and consumer
|
- Clear separation of task producer and consumer
|
||||||
|
- Ability to process multiple tasks concurrently
|
||||||
- Ability to schedule task processing in the future
|
- Ability to schedule task processing in the future
|
||||||
- Automatic retry of failed tasks with exponential backoff
|
- Automatic retry of failed tasks with exponential backoff
|
||||||
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
|
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) task retry count and retry delay
|
||||||
- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) max retry count per task
|
|
||||||
- Ability to configure max number of worker goroutines to process tasks
|
|
||||||
- Support for [priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)
|
- Support for [priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)
|
||||||
- [Unix signal handling](https://github.com/hibiken/asynq/wiki/Signals) to gracefully shutdown background processing
|
- [Unix signal handling](https://github.com/hibiken/asynq/wiki/Signals) to gracefully shutdown background processing
|
||||||
- [CLI tool](/tools/asynqmon/README.md) to query and mutate queues state for mointoring and administrative purposes
|
- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels
|
||||||
|
- [Command line tool](/tools/asynqmon/README.md) to query tasks for monitoring and troubleshooting purposes
|
||||||
|
|
||||||
## Requirements
|
## Requirements
|
||||||
|
|
||||||
@ -49,7 +47,7 @@ Asynq provides:
|
|||||||
|
|
||||||
## Installation
|
## Installation
|
||||||
|
|
||||||
To install both `asynq` library and `asynqmon` CLI tool, run the following command:
|
To install both `asynq` library and `asynqmon` command line tool, run the following command:
|
||||||
|
|
||||||
```
|
```
|
||||||
go get -u github.com/hibiken/asynq
|
go get -u github.com/hibiken/asynq
|
||||||
@ -66,15 +64,22 @@ In this quick tour of `asynq`, we are going to create two programs.
|
|||||||
**This guide assumes that you are running a Redis server at `localhost:6379`**.
|
**This guide assumes that you are running a Redis server at `localhost:6379`**.
|
||||||
Before we start, make sure you have Redis installed and running.
|
Before we start, make sure you have Redis installed and running.
|
||||||
|
|
||||||
1. Import `asynq` in both files.
|
The first thing we need to do is create two main files.
|
||||||
|
|
||||||
|
```sh
|
||||||
|
mkdir producer consumer
|
||||||
|
touch producer/producer.go consumer/consumer.go
|
||||||
|
```
|
||||||
|
|
||||||
|
Import `asynq` in both files.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
import "github.com/hibiken/asynq"
|
import "github.com/hibiken/asynq"
|
||||||
```
|
```
|
||||||
|
|
||||||
2. Asynq uses Redis as a message broker.
|
Asynq uses Redis as a message broker.
|
||||||
Use one of `RedisConnOpt` types to specify how to connect to Redis.
|
Use one of `RedisConnOpt` types to specify how to connect to Redis.
|
||||||
We are going to use `RedisClientOpt` here.
|
We are going to use `RedisClientOpt` here.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// both in producer.go and consumer.go
|
// both in producer.go and consumer.go
|
||||||
@ -88,7 +93,25 @@ var redis = &asynq.RedisClientOpt{
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
3. In `producer.go`, create a `Client` instance to create and schedule tasks.
|
In `producer.go`, we are going to create a `Client` instance to create and schedule tasks.
|
||||||
|
|
||||||
|
In `asynq`, a unit of work to be performed is encapsluated in a struct called `Task`.
|
||||||
|
Which has two fields: `Type` and `Payload`.
|
||||||
|
|
||||||
|
```go
|
||||||
|
// Task represents a task to be performed.
|
||||||
|
type Task struct {
|
||||||
|
// Type indicates the type of task to be performed.
|
||||||
|
Type string
|
||||||
|
|
||||||
|
// Payload holds data needed to perform the task.
|
||||||
|
Payload Payload
|
||||||
|
}
|
||||||
|
```
|
||||||
|
|
||||||
|
To create a task, use `NewTask` function and pass type and payload for the task.
|
||||||
|
|
||||||
|
You schedule a task by calling `Client.Schedule` passing in the task and the timethe task neeeds to be processed.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// producer.go
|
// producer.go
|
||||||
@ -118,7 +141,13 @@ func main() {
|
|||||||
}
|
}
|
||||||
```
|
```
|
||||||
|
|
||||||
4. In `consumer.go`, create a `Background` instance to process tasks.
|
In `consumer.go`, create a `Background` instance to process the tasks.
|
||||||
|
|
||||||
|
`NewBackground` function takes `RedisConnOpt` and `Config`.
|
||||||
|
|
||||||
|
You can take a look at documentation on `Config` to see the available options.
|
||||||
|
|
||||||
|
We are only going to specify the concurrency in this example.
|
||||||
|
|
||||||
```go
|
```go
|
||||||
// consumer.go
|
// consumer.go
|
||||||
@ -239,19 +268,11 @@ func sendReminderEmail(t *asynq.Task) error {
|
|||||||
|
|
||||||
Now that we have both task producer and consumer, we can run both programs.
|
Now that we have both task producer and consumer, we can run both programs.
|
||||||
|
|
||||||
```sh
|
|
||||||
go run consumer.go
|
|
||||||
```
|
|
||||||
|
|
||||||
**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.
|
|
||||||
|
|
||||||
With our consumer running, also run
|
|
||||||
|
|
||||||
```sh
|
```sh
|
||||||
go run producer.go
|
go run producer.go
|
||||||
```
|
```
|
||||||
|
|
||||||
This will create a task and the first task will get processed immediately by the consumer. The second task will be processed 24 hours later.
|
This will create two tasks: One that should processed immediately and another to be processed 24 hours later.
|
||||||
|
|
||||||
Let's use `asynqmon` tool to inspect the tasks.
|
Let's use `asynqmon` tool to inspect the tasks.
|
||||||
|
|
||||||
@ -259,23 +280,37 @@ Let's use `asynqmon` tool to inspect the tasks.
|
|||||||
asynqmon stats
|
asynqmon stats
|
||||||
```
|
```
|
||||||
|
|
||||||
This command will show the number of tasks in each state and stats for the current date as well as redis information.
|
You should able to see that there's one task in **Enqueued** state and another in **Scheduled** state.
|
||||||
|
|
||||||
To understand the meaning of each state, see [Life of a Task Wiki page](https://github.com/hibiken/asynq/wiki/Life-of-a-Task).
|
Note: To understand the meaning of each state, see [Life of a Task](https://github.com/hibiken/asynq/wiki/Life-of-a-Task) on our Wiki page.
|
||||||
|
|
||||||
For in-depth guide on `asynqmon` tool, see the [README](/tools/asynqmon/README.md) for the CLI.
|
Let's run `asynqmon` with `watch` command so that we can continuously run the command to see the changes.
|
||||||
|
|
||||||
This was a quick tour of `asynq` basics. To see all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see [the Wiki page](https://github.com/hibiken/asynq/wiki).
|
```sh
|
||||||
|
watch -n 3 asynqmon stats # Runs `asynqmon stats` every 3 seconds
|
||||||
|
```
|
||||||
|
|
||||||
## Monitoring CLI
|
And finally, let's start the consumer program to process scheduled tasks.
|
||||||
|
|
||||||
Asynq ships with a CLI tool to inspect the state of queues and tasks.
|
```sh
|
||||||
|
go run consumer.go
|
||||||
|
```
|
||||||
|
|
||||||
To install the CLI, run the following command:
|
**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing.
|
||||||
|
|
||||||
|
You should be able to see text printed in your terminal indicating that the task was processed successfully.
|
||||||
|
|
||||||
|
This was a whirlwind tour of `asynq` basics. To learn more about all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see our [Wiki page](https://github.com/hibiken/asynq/wiki).
|
||||||
|
|
||||||
|
## Command Line Tool
|
||||||
|
|
||||||
|
Asynq ships with a command line tool to inspect the state of queues and tasks.
|
||||||
|
|
||||||
|
To install, run the following command:
|
||||||
|
|
||||||
go get github.com/hibiken/asynq/tools/asynqmon
|
go get github.com/hibiken/asynq/tools/asynqmon
|
||||||
|
|
||||||
For details on how to use the tool, see the [README](/tools/asynqmon/README.md) for the asynqmon CLI.
|
For details on how to use the tool, refer to the tool's [README](/tools/asynqmon/README.md).
|
||||||
|
|
||||||
## Acknowledgements
|
## Acknowledgements
|
||||||
|
|
||||||
|
2
doc.go
2
doc.go
@ -3,7 +3,7 @@
|
|||||||
// that can be found in the LICENSE file.
|
// that can be found in the LICENSE file.
|
||||||
|
|
||||||
/*
|
/*
|
||||||
Package asynq provides a framework for background task processing.
|
Package asynq provides a framework for asynchronous task processing.
|
||||||
|
|
||||||
Asynq uses Redis as a message broker. To connect to redis server,
|
Asynq uses Redis as a message broker. To connect to redis server,
|
||||||
specify the options using one of RedisConnOpt types.
|
specify the options using one of RedisConnOpt types.
|
||||||
|
BIN
docs/assets/asynqmon_history.gif
Normal file
BIN
docs/assets/asynqmon_history.gif
Normal file
Binary file not shown.
After Width: | Height: | Size: 1.5 MiB |
@ -1,6 +1,6 @@
|
|||||||
# Asynqmon
|
# Asynqmon
|
||||||
|
|
||||||
Asynqmon is a CLI tool to monitor the queues managed by `asynq` package.
|
Asynqmon is a command line tool to monitor the tasks managed by `asynq` package.
|
||||||
|
|
||||||
## Table of Contents
|
## Table of Contents
|
||||||
|
|
||||||
@ -24,7 +24,7 @@ This will create the asynqmon executable under your `$GOPATH/bin` directory.
|
|||||||
|
|
||||||
## Quick Start
|
## Quick Start
|
||||||
|
|
||||||
Asynqmon tool has a few commands to inspect the state of tasks and queues.
|
The tool has a few commands to inspect the state of tasks and queues.
|
||||||
|
|
||||||
Run `asynqmon help` to see all the available commands.
|
Run `asynqmon help` to see all the available commands.
|
||||||
|
|
||||||
@ -34,7 +34,7 @@ By default, Asynqmon will try to connect to a redis server running at `localhost
|
|||||||
|
|
||||||
### Stats
|
### Stats
|
||||||
|
|
||||||
Stats command gives the overview of the current state of tasks and queues. Run it in conjunction with `watch` command to repeatedly run `stats`.
|
Stats command gives the overview of the current state of tasks and queues. You can run it in conjunction with `watch` command to repeatedly run `stats`.
|
||||||
|
|
||||||
Example:
|
Example:
|
||||||
|
|
||||||
@ -46,35 +46,93 @@ This will run `asynqmon stats` command every 3 seconds.
|
|||||||
|
|
||||||
### History
|
### History
|
||||||
|
|
||||||
TODO: Add discription
|
History command shows the number of processed and failed tasks from the last x days.
|
||||||
|
|
||||||
|
By default, it shows the stats from the last 10 days. Use `--days` to specify the number of days.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon history --days=30
|
||||||
|
|
||||||
|
![Gif](/docs/assets/asynqmon_history.gif)
|
||||||
|
|
||||||
### List
|
### List
|
||||||
|
|
||||||
TODO: Add discription
|
List command shows all tasks in the specified state in a table format
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon ls retry
|
||||||
|
asynqmon ls scheduled
|
||||||
|
asynqmon ls dead
|
||||||
|
asynqmon ls enqueued
|
||||||
|
asynqmon ls inprogress
|
||||||
|
|
||||||
### Enqueue
|
### Enqueue
|
||||||
|
|
||||||
TODO: Add discription
|
There are two commands to enqueue tasks.
|
||||||
|
|
||||||
|
Command `enq` takes a task ID and moves the task to **Enqueued** state. You can obtain the task ID by running `ls` command.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon enq d:1575732274:bnogo8gt6toe23vhef0g
|
||||||
|
|
||||||
|
Command `enqall` moves all tasks to **Enqueued** state from the specified state.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon enqall retry
|
||||||
|
|
||||||
|
Running the above command will move all **Retry** tasks to **Enqueued** state.
|
||||||
|
|
||||||
### Delete
|
### Delete
|
||||||
|
|
||||||
TODO: Add discription
|
There are two commands for task deletion.
|
||||||
|
|
||||||
|
Command `del` takes a task ID and deletes the task. You can obtain the task ID by running `ls` command.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon del r:1575732274:bnogo8gt6toe23vhef0g
|
||||||
|
|
||||||
|
Command `delall` deletes all tasks which are in the specified state.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon delall retry
|
||||||
|
|
||||||
|
Running the above command will delete all **Retry** tasks.
|
||||||
|
|
||||||
### Kill
|
### Kill
|
||||||
|
|
||||||
TODO: Add discription
|
There are two commands to kill (i.e. move to dead state) tasks.
|
||||||
|
|
||||||
|
Command `kill` takes a task ID and kills the task. You can obtain the task ID by running `ls` command.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon kill r:1575732274:bnogo8gt6toe23vhef0g
|
||||||
|
|
||||||
|
Command `killall` kills all tasks which are in the specified state.
|
||||||
|
|
||||||
|
Example:
|
||||||
|
|
||||||
|
asynqmon killall retry
|
||||||
|
|
||||||
|
Running the above command will move all **Retry** tasks to **Dead** state.
|
||||||
|
|
||||||
## Config File
|
## Config File
|
||||||
|
|
||||||
You can use a config file to set default values for flags.
|
You can use a config file to set default values for the flags.
|
||||||
This is useful, for example when you have to connect to a remote redis server.
|
This is useful, for example when you have to connect to a remote redis server.
|
||||||
|
|
||||||
By default, `asynqmon` will try to read config file located in
|
By default, `asynqmon` will try to read config file located in
|
||||||
`$HOME/.asynqmon.(yml|json)`. You can specify the file location via `--config` flag.
|
`$HOME/.asynqmon.(yaml|json)`. You can specify the file location via `--config` flag.
|
||||||
|
|
||||||
Config file example:
|
Config file example:
|
||||||
|
|
||||||
```yml
|
```yaml
|
||||||
uri: 127.0.0.1:6379
|
uri: 127.0.0.1:6379
|
||||||
db: 2
|
db: 2
|
||||||
password: mypassword
|
password: mypassword
|
||||||
|
Loading…
Reference in New Issue
Block a user