diff --git a/README.md b/README.md index cb556b7..6878e8c 100644 --- a/README.md +++ b/README.md @@ -6,9 +6,9 @@ [![GoDoc](https://godoc.org/github.com/hibiken/asynq?status.svg)](https://godoc.org/github.com/hibiken/asynq) [![Gitter chat](https://badges.gitter.im/go-asynq/gitter.svg)](https://gitter.im/go-asynq/community) -Simple and efficent asynchronous task processing library in Go. +Simple and efficient asynchronous task processing library in Go. -**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before the release of verson 1.0.0. +**Important Note**: Current major version is zero (v0.x.x) to accomodate rapid development and fast iteration while getting early feedback from users. The public API could change without a major version update before v1.0.0 release. ## Table of Contents @@ -16,7 +16,7 @@ Simple and efficent asynchronous task processing library in Go. - [Requirements](#requirements) - [Installation](#installation) - [Getting Started](#getting-started) -- [Monitoring CLI](#monitoring-cli) +- [Command Line Tool](#command-line-tool) - [Acknowledgements](#acknowledgements) - [License](#license) @@ -24,21 +24,19 @@ Simple and efficent asynchronous task processing library in Go. ![Gif](/docs/assets/asynqmon_stats.gif) -Asynq provides a simple interface to asynchronous task processing. - -It also ships with a tool to monitor the queues and take manual actions if needed. +Package asynq provides a framework for asynchronous task processing. Asynq provides: - Clear separation of task producer and consumer +- Ability to process multiple tasks concurrently - Ability to schedule task processing in the future - Automatic retry of failed tasks with exponential backoff -- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels -- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) max retry count per task -- Ability to configure max number of worker goroutines to process tasks +- [Ability to configure](https://github.com/hibiken/asynq/wiki/Task-Retry) task retry count and retry delay - Support for [priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues) - [Unix signal handling](https://github.com/hibiken/asynq/wiki/Signals) to gracefully shutdown background processing -- [CLI tool](/tools/asynqmon/README.md) to query and mutate queues state for mointoring and administrative purposes +- [Automatic failover](https://github.com/hibiken/asynq/wiki/Automatic-Failover) using Redis sentinels +- [Command line tool](/tools/asynqmon/README.md) to query tasks for monitoring and troubleshooting purposes ## Requirements @@ -49,7 +47,7 @@ Asynq provides: ## Installation -To install both `asynq` library and `asynqmon` CLI tool, run the following command: +To install both `asynq` library and `asynqmon` command line tool, run the following command: ``` go get -u github.com/hibiken/asynq @@ -66,15 +64,22 @@ In this quick tour of `asynq`, we are going to create two programs. **This guide assumes that you are running a Redis server at `localhost:6379`**. Before we start, make sure you have Redis installed and running. -1. Import `asynq` in both files. +The first thing we need to do is create two main files. + +```sh +mkdir producer consumer +touch producer/producer.go consumer/consumer.go +``` + +Import `asynq` in both files. ```go import "github.com/hibiken/asynq" ``` -2. Asynq uses Redis as a message broker. - Use one of `RedisConnOpt` types to specify how to connect to Redis. - We are going to use `RedisClientOpt` here. +Asynq uses Redis as a message broker. +Use one of `RedisConnOpt` types to specify how to connect to Redis. +We are going to use `RedisClientOpt` here. ```go // both in producer.go and consumer.go @@ -88,7 +93,25 @@ var redis = &asynq.RedisClientOpt{ } ``` -3. In `producer.go`, create a `Client` instance to create and schedule tasks. +In `producer.go`, we are going to create a `Client` instance to create and schedule tasks. + +In `asynq`, a unit of work to be performed is encapsluated in a struct called `Task`. +Which has two fields: `Type` and `Payload`. + +```go +// Task represents a task to be performed. +type Task struct { + // Type indicates the type of task to be performed. + Type string + + // Payload holds data needed to perform the task. + Payload Payload +} +``` + +To create a task, use `NewTask` function and pass type and payload for the task. + +You schedule a task by calling `Client.Schedule` passing in the task and the timethe task neeeds to be processed. ```go // producer.go @@ -118,7 +141,13 @@ func main() { } ``` -4. In `consumer.go`, create a `Background` instance to process tasks. +In `consumer.go`, create a `Background` instance to process the tasks. + +`NewBackground` function takes `RedisConnOpt` and `Config`. + +You can take a look at documentation on `Config` to see the available options. + +We are only going to specify the concurrency in this example. ```go // consumer.go @@ -239,19 +268,11 @@ func sendReminderEmail(t *asynq.Task) error { Now that we have both task producer and consumer, we can run both programs. -```sh -go run consumer.go -``` - -**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing. - -With our consumer running, also run - ```sh go run producer.go ``` -This will create a task and the first task will get processed immediately by the consumer. The second task will be processed 24 hours later. +This will create two tasks: One that should processed immediately and another to be processed 24 hours later. Let's use `asynqmon` tool to inspect the tasks. @@ -259,23 +280,37 @@ Let's use `asynqmon` tool to inspect the tasks. asynqmon stats ``` -This command will show the number of tasks in each state and stats for the current date as well as redis information. +You should able to see that there's one task in **Enqueued** state and another in **Scheduled** state. -To understand the meaning of each state, see [Life of a Task Wiki page](https://github.com/hibiken/asynq/wiki/Life-of-a-Task). +Note: To understand the meaning of each state, see [Life of a Task](https://github.com/hibiken/asynq/wiki/Life-of-a-Task) on our Wiki page. -For in-depth guide on `asynqmon` tool, see the [README](/tools/asynqmon/README.md) for the CLI. +Let's run `asynqmon` with `watch` command so that we can continuously run the command to see the changes. -This was a quick tour of `asynq` basics. To see all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see [the Wiki page](https://github.com/hibiken/asynq/wiki). +```sh +watch -n 3 asynqmon stats # Runs `asynqmon stats` every 3 seconds +``` -## Monitoring CLI +And finally, let's start the consumer program to process scheduled tasks. -Asynq ships with a CLI tool to inspect the state of queues and tasks. +```sh +go run consumer.go +``` -To install the CLI, run the following command: +**Note**: This will not exit until you send a signal to terminate the program. See [Signal Wiki page](https://github.com/hibiken/asynq/wiki/Signals) for best practice on how to safely terminate background processing. + +You should be able to see text printed in your terminal indicating that the task was processed successfully. + +This was a whirlwind tour of `asynq` basics. To learn more about all of its features such as **[priority queues](https://github.com/hibiken/asynq/wiki/Priority-Queues)** and **[custom retry](https://github.com/hibiken/asynq/wiki/Task-Retry)**, see our [Wiki page](https://github.com/hibiken/asynq/wiki). + +## Command Line Tool + +Asynq ships with a command line tool to inspect the state of queues and tasks. + +To install, run the following command: go get github.com/hibiken/asynq/tools/asynqmon -For details on how to use the tool, see the [README](/tools/asynqmon/README.md) for the asynqmon CLI. +For details on how to use the tool, refer to the tool's [README](/tools/asynqmon/README.md). ## Acknowledgements diff --git a/doc.go b/doc.go index d80a63d..f56b5c4 100644 --- a/doc.go +++ b/doc.go @@ -3,7 +3,7 @@ // that can be found in the LICENSE file. /* -Package asynq provides a framework for background task processing. +Package asynq provides a framework for asynchronous task processing. Asynq uses Redis as a message broker. To connect to redis server, specify the options using one of RedisConnOpt types. diff --git a/docs/assets/asynqmon_history.gif b/docs/assets/asynqmon_history.gif new file mode 100644 index 0000000..0feab50 Binary files /dev/null and b/docs/assets/asynqmon_history.gif differ diff --git a/tools/asynqmon/README.md b/tools/asynqmon/README.md index 49f5964..2dc8007 100644 --- a/tools/asynqmon/README.md +++ b/tools/asynqmon/README.md @@ -1,6 +1,6 @@ # Asynqmon -Asynqmon is a CLI tool to monitor the queues managed by `asynq` package. +Asynqmon is a command line tool to monitor the tasks managed by `asynq` package. ## Table of Contents @@ -24,7 +24,7 @@ This will create the asynqmon executable under your `$GOPATH/bin` directory. ## Quick Start -Asynqmon tool has a few commands to inspect the state of tasks and queues. +The tool has a few commands to inspect the state of tasks and queues. Run `asynqmon help` to see all the available commands. @@ -34,7 +34,7 @@ By default, Asynqmon will try to connect to a redis server running at `localhost ### Stats -Stats command gives the overview of the current state of tasks and queues. Run it in conjunction with `watch` command to repeatedly run `stats`. +Stats command gives the overview of the current state of tasks and queues. You can run it in conjunction with `watch` command to repeatedly run `stats`. Example: @@ -46,35 +46,93 @@ This will run `asynqmon stats` command every 3 seconds. ### History -TODO: Add discription +History command shows the number of processed and failed tasks from the last x days. + +By default, it shows the stats from the last 10 days. Use `--days` to specify the number of days. + +Example: + + asynqmon history --days=30 + +![Gif](/docs/assets/asynqmon_history.gif) ### List -TODO: Add discription +List command shows all tasks in the specified state in a table format + +Example: + + asynqmon ls retry + asynqmon ls scheduled + asynqmon ls dead + asynqmon ls enqueued + asynqmon ls inprogress ### Enqueue -TODO: Add discription +There are two commands to enqueue tasks. + +Command `enq` takes a task ID and moves the task to **Enqueued** state. You can obtain the task ID by running `ls` command. + +Example: + + asynqmon enq d:1575732274:bnogo8gt6toe23vhef0g + +Command `enqall` moves all tasks to **Enqueued** state from the specified state. + +Example: + + asynqmon enqall retry + +Running the above command will move all **Retry** tasks to **Enqueued** state. ### Delete -TODO: Add discription +There are two commands for task deletion. + +Command `del` takes a task ID and deletes the task. You can obtain the task ID by running `ls` command. + +Example: + + asynqmon del r:1575732274:bnogo8gt6toe23vhef0g + +Command `delall` deletes all tasks which are in the specified state. + +Example: + + asynqmon delall retry + +Running the above command will delete all **Retry** tasks. ### Kill -TODO: Add discription +There are two commands to kill (i.e. move to dead state) tasks. + +Command `kill` takes a task ID and kills the task. You can obtain the task ID by running `ls` command. + +Example: + + asynqmon kill r:1575732274:bnogo8gt6toe23vhef0g + +Command `killall` kills all tasks which are in the specified state. + +Example: + + asynqmon killall retry + +Running the above command will move all **Retry** tasks to **Dead** state. ## Config File -You can use a config file to set default values for flags. +You can use a config file to set default values for the flags. This is useful, for example when you have to connect to a remote redis server. By default, `asynqmon` will try to read config file located in -`$HOME/.asynqmon.(yml|json)`. You can specify the file location via `--config` flag. +`$HOME/.asynqmon.(yaml|json)`. You can specify the file location via `--config` flag. Config file example: -```yml +```yaml uri: 127.0.0.1:6379 db: 2 password: mypassword