Update README about supported Go versions

* revert state to as it was before v9 updates for tools and x modules

.github/workflows/build.yml

@@ -7,7 +7,7 @@ jobs:
     strategy:
       matrix:
         os: [ubuntu-latest]
-        go-version: [1.14.x, 1.15.x, 1.16.x, 1.17.x]
+        go-version: [1.20.x, 1.21.x]
     runs-on: ${{ matrix.os }}
     services:
       redis:
@@ -28,9 +28,6 @@ jobs:
       - name: Build x module
         run: cd x && go build -v ./... && cd ..
 
-      - name: Build tools module
-        run: cd tools && go build -v ./... && cd ..
-
       - name: Test core module
         run: go test -race -v -coverprofile=coverage.txt -covermode=atomic ./...
 
@@ -42,3 +39,29 @@ jobs:
 
       - name: Upload coverage to Codecov
         uses: codecov/codecov-action@v1
+
+  build-tool:
+    strategy:
+      matrix:
+        os: [ubuntu-latest]
+        go-version: [1.18.x]
+    runs-on: ${{ matrix.os }}
+    services:
+      redis:
+        image: redis
+        ports:
+          - 6379:6379
+    steps:
+      - uses: actions/checkout@v2
+
+      - name: Set up Go
+        uses: actions/setup-go@v2
+        with:
+          go-version: ${{ matrix.go-version }}
+
+      - name: Build tools module
+        run: cd tools && go build -v ./... && cd ..
+
+      - name: Test tools module
+        run: cd tools && go test -race -v ./... && cd ..
+

CHANGELOG.md

@@ -7,7 +7,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
-## [0.21.0] - 2022-02-19
+## [0.24.1] - 2023-05-01
+
+### Changed
+- Updated package version dependency for go-redis 
+
+## [0.24.0] - 2023-01-02
+
+### Added
+- `PreEnqueueFunc`, `PostEnqueueFunc` is added in `Scheduler` and deprecated `EnqueueErrorHandler` (PR: https://github.com/hibiken/asynq/pull/476)
+
+### Changed
+- Removed error log when `Scheduler` failed to enqueue a task. Use `PostEnqueueFunc` to check for errors and task actions if needed.
+- Changed log level from ERROR to WARNINING when `Scheduler` failed to record `SchedulerEnqueueEvent`.
+
+## [0.23.0] - 2022-04-11
+
+### Added
+
+- `Group` option is introduced to enqueue task in a group.
+- `GroupAggregator` and related types are introduced for task aggregation feature.
+- `GroupGracePeriod`, `GroupMaxSize`, `GroupMaxDelay`, and `GroupAggregator` fields are added to `Config`.
+- `Inspector` has new methods related to "aggregating tasks".
+- `Group` field is added to `TaskInfo`.
+- (CLI): `group ls` command is added
+- (CLI): `task ls` supports listing aggregating tasks via `--state=aggregating --group=<GROUP>` flags
+- Enable rediss url parsing support
+
+### Fixed
+
+- Fixed overflow issue with 32-bit systems (For details, see https://github.com/hibiken/asynq/pull/426)
+
+## [0.22.1] - 2022-02-20
+
+### Fixed
+
+- Fixed Redis version compatibility: Keep support for redis v4.0+
+
+## [0.22.0] - 2022-02-19
 
 ### Added
 
@@ -258,9 +295,9 @@ Use `ProcessIn` or `ProcessAt` option to schedule a task instead of `EnqueueIn`
 
 #### `Inspector`
 
-All Inspector methods are scoped to a queue, and the methods take `qname (string)` as the first argument.  
-`EnqueuedTask` is renamed to `PendingTask` and its corresponding methods.  
-`InProgressTask` is renamed to `ActiveTask` and its corresponding methods.  
+All Inspector methods are scoped to a queue, and the methods take `qname (string)` as the first argument.
+`EnqueuedTask` is renamed to `PendingTask` and its corresponding methods.
+`InProgressTask` is renamed to `ActiveTask` and its corresponding methods.
 Command "Enqueue" is replaced by the verb "Run" (e.g. `EnqueueAllScheduledTasks` --> `RunAllScheduledTasks`)
 
 #### `CLI`

CONTRIBUTING.md

@@ -38,7 +38,7 @@ Thank you! We'll try to respond as quickly as possible.
 ## Contributing Code
 
 1. Fork this repo
-2. Download your fork `git clone https://github.com/your-username/asynq && cd asynq`
+2. Download your fork `git clone git@github.com:your-username/asynq.git && cd asynq`
 3. Create your branch `git checkout -b your-branch-name`
 4. Make and commit your changes
 5. Push the branch `git push origin your-branch-name`

README.md

@@ -33,6 +33,7 @@ Task queues are used as a mechanism to distribute work across multiple machines.
 - Low latency to add a task since writes are fast in Redis
 - De-duplication of tasks using [unique option](https://github.com/hibiken/asynq/wiki/Unique-Tasks)
 - Allow [timeout and deadline per task](https://github.com/hibiken/asynq/wiki/Task-Timeout-and-Cancelation)
+- Allow [aggregating group of tasks](https://github.com/hibiken/asynq/wiki/Task-aggregation) to batch multiple successive operations
 - [Flexible handler interface with support for middlewares](https://github.com/hibiken/asynq/wiki/Handler-Deep-Dive)
 - [Ability to pause queue](/tools/asynq/README.md#pause) to stop processing tasks from the queue
 - [Periodic Tasks](https://github.com/hibiken/asynq/wiki/Periodic-Tasks)
@@ -46,11 +47,11 @@ Task queues are used as a mechanism to distribute work across multiple machines.
 
 **Status**: The library is currently undergoing **heavy development** with frequent, breaking API changes.
 
-> ☝️ **Important Note**: Current major version is zero (`v0.x.x`) to accomodate rapid development and fast iteration while getting early feedback from users (_feedback on APIs are appreciated!_). The public API could change without a major version update before `v1.0.0` release.
+> ☝️ **Important Note**: Current major version is zero (`v0.x.x`) to accommodate rapid development and fast iteration while getting early feedback from users (_feedback on APIs are appreciated!_). The public API could change without a major version update before `v1.0.0` release.
 
 ## Quickstart
 
-Make sure you have Go installed ([download](https://golang.org/dl/)). Version `1.14` or higher is required.
+Make sure you have Go installed ([download](https://golang.org/dl/)). Latest two Go versions are supported (See https://go.dev/dl).
 
 Initialize your project by creating a folder and then running `go mod init github.com/your/repo` ([learn more](https://blog.golang.org/using-go-modules)) inside the folder. Then install Asynq library with the [`go get`](https://golang.org/cmd/go/#hdr-Add_dependencies_to_current_module_and_install_them) command:
 
@@ -291,12 +292,12 @@ Asynq ships with a command line tool to inspect the state of queues and tasks.
 To install the CLI tool, run the following command:
 
 ```sh
-go get -u github.com/hibiken/asynq/tools/asynq
+go install github.com/hibiken/asynq/tools/asynq
 ```
 
-Here's an example of running the `asynq stats` command:
+Here's an example of running the `asynq dash` command:
 
-![Gif](/docs/assets/demo.gif)
+![Gif](/docs/assets/dash.gif)
 
 For details on how to use the tool, refer to the tool's [README](/tools/asynq/README.md).
 

aggregator.go

@@ -0,0 +1,176 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package asynq
+
+import (
+	"context"
+	"sync"
+	"time"
+
+	"github.com/hibiken/asynq/internal/base"
+	"github.com/hibiken/asynq/internal/log"
+)
+
+// An aggregator is responsible for checking groups and aggregate into one task
+// if any of the grouping condition is met.
+type aggregator struct {
+	logger *log.Logger
+	broker base.Broker
+	client *Client
+
+	// channel to communicate back to the long running "aggregator" goroutine.
+	done chan struct{}
+
+	// list of queue names to check and aggregate.
+	queues []string
+
+	// Group configurations
+	gracePeriod time.Duration
+	maxDelay    time.Duration
+	maxSize     int
+
+	// User provided group aggregator.
+	ga GroupAggregator
+
+	// interval used to check for aggregation
+	interval time.Duration
+
+	// sema is a counting semaphore to ensure the number of active aggregating function
+	// does not exceed the limit.
+	sema chan struct{}
+}
+
+type aggregatorParams struct {
+	logger          *log.Logger
+	broker          base.Broker
+	queues          []string
+	gracePeriod     time.Duration
+	maxDelay        time.Duration
+	maxSize         int
+	groupAggregator GroupAggregator
+}
+
+const (
+	// Maximum number of aggregation checks in flight concurrently.
+	maxConcurrentAggregationChecks = 3
+
+	// Default interval used for aggregation checks. If the provided gracePeriod is less than
+	// the default, use the gracePeriod.
+	defaultAggregationCheckInterval = 7 * time.Second
+)
+
+func newAggregator(params aggregatorParams) *aggregator {
+	interval := defaultAggregationCheckInterval
+	if params.gracePeriod < interval {
+		interval = params.gracePeriod
+	}
+	return &aggregator{
+		logger:      params.logger,
+		broker:      params.broker,
+		client:      &Client{broker: params.broker},
+		done:        make(chan struct{}),
+		queues:      params.queues,
+		gracePeriod: params.gracePeriod,
+		maxDelay:    params.maxDelay,
+		maxSize:     params.maxSize,
+		ga:          params.groupAggregator,
+		sema:        make(chan struct{}, maxConcurrentAggregationChecks),
+		interval:    interval,
+	}
+}
+
+func (a *aggregator) shutdown() {
+	if a.ga == nil {
+		return
+	}
+	a.logger.Debug("Aggregator shutting down...")
+	// Signal the aggregator goroutine to stop.
+	a.done <- struct{}{}
+}
+
+func (a *aggregator) start(wg *sync.WaitGroup) {
+	if a.ga == nil {
+		return
+	}
+	wg.Add(1)
+	go func() {
+		defer wg.Done()
+		ticker := time.NewTicker(a.interval)
+		for {
+			select {
+			case <-a.done:
+				a.logger.Debug("Waiting for all aggregation checks to finish...")
+				// block until all aggregation checks released the token
+				for i := 0; i < cap(a.sema); i++ {
+					a.sema <- struct{}{}
+				}
+				a.logger.Debug("Aggregator done")
+				ticker.Stop()
+				return
+			case t := <-ticker.C:
+				a.exec(t)
+			}
+		}
+	}()
+}
+
+func (a *aggregator) exec(t time.Time) {
+	select {
+	case a.sema <- struct{}{}: // acquire token
+		go a.aggregate(t)
+	default:
+		// If the semaphore blocks, then we are currently running max number of
+		// aggregation checks. Skip this round and log warning.
+		a.logger.Warnf("Max number of aggregation checks in flight. Skipping")
+	}
+}
+
+func (a *aggregator) aggregate(t time.Time) {
+	defer func() { <-a.sema /* release token */ }()
+	for _, qname := range a.queues {
+		groups, err := a.broker.ListGroups(qname)
+		if err != nil {
+			a.logger.Errorf("Failed to list groups in queue: %q", qname)
+			continue
+		}
+		for _, gname := range groups {
+			aggregationSetID, err := a.broker.AggregationCheck(
+				qname, gname, t, a.gracePeriod, a.maxDelay, a.maxSize)
+			if err != nil {
+				a.logger.Errorf("Failed to run aggregation check: queue=%q group=%q", qname, gname)
+				continue
+			}
+			if aggregationSetID == "" {
+				a.logger.Debugf("No aggregation needed at this time: queue=%q group=%q", qname, gname)
+				continue
+			}
+
+			// Aggregate and enqueue.
+			msgs, deadline, err := a.broker.ReadAggregationSet(qname, gname, aggregationSetID)
+			if err != nil {
+				a.logger.Errorf("Failed to read aggregation set: queue=%q, group=%q, setID=%q",
+					qname, gname, aggregationSetID)
+				continue
+			}
+			tasks := make([]*Task, len(msgs))
+			for i, m := range msgs {
+				tasks[i] = NewTask(m.Type, m.Payload)
+			}
+			aggregatedTask := a.ga.Aggregate(gname, tasks)
+			ctx, cancel := context.WithDeadline(context.Background(), deadline)
+			if _, err := a.client.EnqueueContext(ctx, aggregatedTask, Queue(qname)); err != nil {
+				a.logger.Errorf("Failed to enqueue aggregated task (queue=%q, group=%q, setID=%q): %v",
+					qname, gname, aggregationSetID, err)
+				cancel()
+				continue
+			}
+			if err := a.broker.DeleteAggregationSet(ctx, qname, gname, aggregationSetID); err != nil {
+				a.logger.Warnf("Failed to delete aggregation set: queue=%q, group=%q, setID=%q",
+					qname, gname, aggregationSetID)
+			}
+			cancel()
+		}
+	}
+}

aggregator_test.go

@@ -0,0 +1,165 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package asynq
+
+import (
+	"sync"
+	"testing"
+	"time"
+
+	"github.com/google/go-cmp/cmp"
+	"github.com/hibiken/asynq/internal/base"
+	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
+)
+
+func TestAggregator(t *testing.T) {
+	r := setup(t)
+	defer r.Close()
+	rdbClient := rdb.NewRDB(r)
+	client := Client{broker: rdbClient}
+
+	tests := []struct {
+		desc             string
+		gracePeriod      time.Duration
+		maxDelay         time.Duration
+		maxSize          int
+		aggregateFunc    func(gname string, tasks []*Task) *Task
+		tasks            []*Task       // tasks to enqueue
+		enqueueFrequency time.Duration // time between one enqueue event to another
+		waitTime         time.Duration // time to wait
+		wantGroups       map[string]map[string][]base.Z
+		wantPending      map[string][]*base.TaskMessage
+	}{
+		{
+			desc:        "group older than the grace period should be aggregated",
+			gracePeriod: 1 * time.Second,
+			maxDelay:    0, // no maxdelay limit
+			maxSize:     0, // no maxsize limit
+			aggregateFunc: func(gname string, tasks []*Task) *Task {
+				return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
+			},
+			tasks: []*Task{
+				NewTask("task1", nil, Group("mygroup")),
+				NewTask("task2", nil, Group("mygroup")),
+				NewTask("task3", nil, Group("mygroup")),
+			},
+			enqueueFrequency: 300 * time.Millisecond,
+			waitTime:         3 * time.Second,
+			wantGroups: map[string]map[string][]base.Z{
+				"default": {
+					"mygroup": {},
+				},
+			},
+			wantPending: map[string][]*base.TaskMessage{
+				"default": {
+					h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(3).Build(),
+				},
+			},
+		},
+		{
+			desc:        "group older than the max-delay should be aggregated",
+			gracePeriod: 2 * time.Second,
+			maxDelay:    4 * time.Second,
+			maxSize:     0, // no maxsize limit
+			aggregateFunc: func(gname string, tasks []*Task) *Task {
+				return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
+			},
+			tasks: []*Task{
+				NewTask("task1", nil, Group("mygroup")), // time 0
+				NewTask("task2", nil, Group("mygroup")), // time 1s
+				NewTask("task3", nil, Group("mygroup")), // time 2s
+				NewTask("task4", nil, Group("mygroup")), // time 3s
+			},
+			enqueueFrequency: 1 * time.Second,
+			waitTime:         4 * time.Second,
+			wantGroups: map[string]map[string][]base.Z{
+				"default": {
+					"mygroup": {},
+				},
+			},
+			wantPending: map[string][]*base.TaskMessage{
+				"default": {
+					h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(4).Build(),
+				},
+			},
+		},
+		{
+			desc:        "group reached the max-size should be aggregated",
+			gracePeriod: 1 * time.Minute,
+			maxDelay:    0, // no maxdelay limit
+			maxSize:     5,
+			aggregateFunc: func(gname string, tasks []*Task) *Task {
+				return NewTask(gname, nil, MaxRetry(len(tasks))) // use max retry to see how many tasks were aggregated
+			},
+			tasks: []*Task{
+				NewTask("task1", nil, Group("mygroup")),
+				NewTask("task2", nil, Group("mygroup")),
+				NewTask("task3", nil, Group("mygroup")),
+				NewTask("task4", nil, Group("mygroup")),
+				NewTask("task5", nil, Group("mygroup")),
+			},
+			enqueueFrequency: 300 * time.Millisecond,
+			waitTime:         defaultAggregationCheckInterval * 2,
+			wantGroups: map[string]map[string][]base.Z{
+				"default": {
+					"mygroup": {},
+				},
+			},
+			wantPending: map[string][]*base.TaskMessage{
+				"default": {
+					h.NewTaskMessageBuilder().SetType("mygroup").SetRetry(5).Build(),
+				},
+			},
+		},
+	}
+
+	for _, tc := range tests {
+		h.FlushDB(t, r)
+
+		aggregator := newAggregator(aggregatorParams{
+			logger:          testLogger,
+			broker:          rdbClient,
+			queues:          []string{"default"},
+			gracePeriod:     tc.gracePeriod,
+			maxDelay:        tc.maxDelay,
+			maxSize:         tc.maxSize,
+			groupAggregator: GroupAggregatorFunc(tc.aggregateFunc),
+		})
+
+		var wg sync.WaitGroup
+		aggregator.start(&wg)
+
+		for _, task := range tc.tasks {
+			if _, err := client.Enqueue(task); err != nil {
+				t.Errorf("%s: Client Enqueue failed: %v", tc.desc, err)
+				aggregator.shutdown()
+				wg.Wait()
+				continue
+			}
+			time.Sleep(tc.enqueueFrequency)
+		}
+
+		time.Sleep(tc.waitTime)
+
+		for qname, groups := range tc.wantGroups {
+			for gname, want := range groups {
+				gotGroup := h.GetGroupEntries(t, r, qname, gname)
+				if diff := cmp.Diff(want, gotGroup, h.SortZSetEntryOpt); diff != "" {
+					t.Errorf("%s: mismatch found in %q; (-want,+got)\n%s", tc.desc, base.GroupKey(qname, gname), diff)
+				}
+			}
+		}
+
+		for qname, want := range tc.wantPending {
+			gotPending := h.GetPendingMessages(t, r, qname)
+			if diff := cmp.Diff(want, gotPending, h.SortMsgOpt, h.IgnoreIDOpt); diff != "" {
+				t.Errorf("%s: mismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
+			}
+		}
+		aggregator.shutdown()
+		wg.Wait()
+	}
+}

asynq.go

@@ -8,12 +8,13 @@ import (
 	"context"
 	"crypto/tls"
 	"fmt"
+	"net"
 	"net/url"
 	"strconv"
 	"strings"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 )
 
@@ -97,6 +98,14 @@ type TaskInfo struct {
 	// Deadline is the deadline for the task, zero value if not specified.
 	Deadline time.Time
 
+	// Group is the name of the group in which the task belongs.
+	//
+	// Tasks in the same queue can be grouped together by Group name and will be aggregated into one task
+	// by a Server processing the queue.
+	//
+	// Empty string (default) indicates task does not belong to any groups, and no aggregation will be applied to the task.
+	Group string
+
 	// NextProcessAt is the time the task is scheduled to be processed,
 	// zero if not applicable.
 	NextProcessAt time.Time
@@ -139,6 +148,7 @@ func newTaskInfo(msg *base.TaskMessage, state base.TaskState, nextProcessAt time
 		MaxRetry:      msg.Retry,
 		Retried:       msg.Retried,
 		LastErr:       msg.ErrorMsg,
+		Group:         msg.GroupKey,
 		Timeout:       time.Duration(msg.Timeout) * time.Second,
 		Deadline:      fromUnixTimeOrZero(msg.Deadline),
 		Retention:     time.Duration(msg.Retention) * time.Second,
@@ -161,6 +171,8 @@ func newTaskInfo(msg *base.TaskMessage, state base.TaskState, nextProcessAt time
 		info.State = TaskStateArchived
 	case base.TaskStateCompleted:
 		info.State = TaskStateCompleted
+	case base.TaskStateAggregating:
+		info.State = TaskStateAggregating
 	default:
 		panic(fmt.Sprintf("internal error: unknown state: %d", state))
 	}
@@ -188,6 +200,9 @@ const (
 
 	// Indicates that the task is processed successfully and retained until the retention TTL expires.
 	TaskStateCompleted
+
+	// Indicates that the task is waiting in a group to be aggregated into one task.
+	TaskStateAggregating
 )
 
 func (s TaskState) String() string {
@@ -204,6 +219,8 @@ func (s TaskState) String() string {
 		return "archived"
 	case TaskStateCompleted:
 		return "completed"
+	case TaskStateAggregating:
+		return "aggregating"
 	}
 	panic("asynq: unknown task state")
 }
@@ -419,9 +436,10 @@ func (opt RedisClusterClientOpt) MakeRedisClient() interface{} {
 // ParseRedisURI parses redis uri string and returns RedisConnOpt if uri is valid.
 // It returns a non-nil error if uri cannot be parsed.
 //
-// Three URI schemes are supported, which are redis:, redis-socket:, and redis-sentinel:.
+// Three URI schemes are supported, which are redis:, rediss:, redis-socket:, and redis-sentinel:.
 // Supported formats are:
 //     redis://[:password@]host[:port][/dbnumber]
+//     rediss://[:password@]host[:port][/dbnumber]
 //     redis-socket://[:password@]path[?db=dbnumber]
 //     redis-sentinel://[:password@]host1[:port][,host2:[:port]][,hostN:[:port]][?master=masterName]
 func ParseRedisURI(uri string) (RedisConnOpt, error) {
@@ -430,7 +448,7 @@ func ParseRedisURI(uri string) (RedisConnOpt, error) {
 		return nil, fmt.Errorf("asynq: could not parse redis uri: %v", err)
 	}
 	switch u.Scheme {
-	case "redis":
+	case "redis", "rediss":
 		return parseRedisURI(u)
 	case "redis-socket":
 		return parseRedisSocketURI(u)
@@ -444,6 +462,8 @@ func ParseRedisURI(uri string) (RedisConnOpt, error) {
 func parseRedisURI(u *url.URL) (RedisConnOpt, error) {
 	var db int
 	var err error
+	var redisConnOpt RedisClientOpt
+
 	if len(u.Path) > 0 {
 		xs := strings.Split(strings.Trim(u.Path, "/"), "/")
 		db, err = strconv.Atoi(xs[0])
@@ -455,7 +475,20 @@ func parseRedisURI(u *url.URL) (RedisConnOpt, error) {
 	if v, ok := u.User.Password(); ok {
 		password = v
 	}
-	return RedisClientOpt{Addr: u.Host, DB: db, Password: password}, nil
+
+	if u.Scheme == "rediss" {
+		h, _, err := net.SplitHostPort(u.Host)
+		if err != nil {
+			h = u.Host
+		}
+		redisConnOpt.TLSConfig = &tls.Config{ServerName: h}
+	}
+
+	redisConnOpt.Addr = u.Host
+	redisConnOpt.Password = password
+	redisConnOpt.DB = db
+
+	return redisConnOpt, nil
 }
 
 func parseRedisSocketURI(u *url.URL) (RedisConnOpt, error) {
@@ -486,7 +519,7 @@ func parseRedisSentinelURI(u *url.URL) (RedisConnOpt, error) {
 	if v, ok := u.User.Password(); ok {
 		password = v
 	}
-	return RedisFailoverClientOpt{MasterName: master, SentinelAddrs: addrs, Password: password}, nil
+	return RedisFailoverClientOpt{MasterName: master, SentinelAddrs: addrs, SentinelPassword: password}, nil
 }
 
 // ResultWriter is a client interface to write result data for a task.

asynq_test.go

@@ -5,15 +5,17 @@
 package asynq
 
 import (
+	"crypto/tls"
 	"flag"
 	"sort"
 	"strings"
 	"testing"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/google/go-cmp/cmp"
-	h "github.com/hibiken/asynq/internal/asynqtest"
+	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/hibiken/asynq/internal/log"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 //============================================================================
@@ -99,6 +101,10 @@ func TestParseRedisURI(t *testing.T) {
 			"redis://localhost:6379",
 			RedisClientOpt{Addr: "localhost:6379"},
 		},
+		{
+			"rediss://localhost:6379",
+			RedisClientOpt{Addr: "localhost:6379", TLSConfig: &tls.Config{ServerName: "localhost"}},
+		},
 		{
 			"redis://localhost:6379/3",
 			RedisClientOpt{Addr: "localhost:6379", DB: 3},
@@ -137,9 +143,9 @@ func TestParseRedisURI(t *testing.T) {
 		{
 			"redis-sentinel://:mypassword@localhost:5000,localhost:5001,localhost:5002?master=mymaster",
 			RedisFailoverClientOpt{
-				MasterName:    "mymaster",
-				SentinelAddrs: []string{"localhost:5000", "localhost:5001", "localhost:5002"},
-				Password:      "mypassword",
+				MasterName:       "mymaster",
+				SentinelAddrs:    []string{"localhost:5000", "localhost:5001", "localhost:5002"},
+				SentinelPassword: "mypassword",
 			},
 		},
 	}
@@ -151,7 +157,7 @@ func TestParseRedisURI(t *testing.T) {
 			continue
 		}
 
-		if diff := cmp.Diff(tc.want, got); diff != "" {
+		if diff := cmp.Diff(tc.want, got, cmpopts.IgnoreUnexported(tls.Config{})); diff != "" {
 			t.Errorf("ParseRedisURI(%q) = %+v, want %+v\n(-want,+got)\n%s", tc.uri, got, tc.want, diff)
 		}
 	}

benchmark_test.go

@@ -12,7 +12,7 @@ import (
 	"testing"
 	"time"
 
-	h "github.com/hibiken/asynq/internal/asynqtest"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 // Creates a new task of type "task<n>" with payload {"data": n}.

client.go

@@ -10,7 +10,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/google/uuid"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/errors"
@@ -24,7 +24,7 @@ import (
 //
 // Clients are safe for concurrent use by multiple goroutines.
 type Client struct {
-	rdb *rdb.RDB
+	broker base.Broker
 }
 
 // NewClient returns a new Client instance given a redis connection option.
@@ -33,7 +33,7 @@ func NewClient(r RedisConnOpt) *Client {
 	if !ok {
 		panic(fmt.Sprintf("asynq: unsupported RedisConnOpt type %T", r))
 	}
-	return &Client{rdb: rdb.NewRDB(c)}
+	return &Client{broker: rdb.NewRDB(c)}
 }
 
 type OptionType int
@@ -48,6 +48,7 @@ const (
 	ProcessInOpt
 	TaskIDOpt
 	RetentionOpt
+	GroupOpt
 )
 
 // Option specifies the task processing behavior.
@@ -73,6 +74,7 @@ type (
 	processAtOption time.Time
 	processInOption time.Duration
 	retentionOption time.Duration
+	groupOption     string
 )
 
 // MaxRetry returns an option to specify the max number of times
@@ -91,13 +93,13 @@ func (n retryOption) Type() OptionType   { return MaxRetryOpt }
 func (n retryOption) Value() interface{} { return int(n) }
 
 // Queue returns an option to specify the queue to enqueue the task into.
-func Queue(qname string) Option {
-	return queueOption(qname)
+func Queue(name string) Option {
+	return queueOption(name)
 }
 
-func (qname queueOption) String() string     { return fmt.Sprintf("Queue(%q)", string(qname)) }
-func (qname queueOption) Type() OptionType   { return QueueOpt }
-func (qname queueOption) Value() interface{} { return string(qname) }
+func (name queueOption) String() string     { return fmt.Sprintf("Queue(%q)", string(name)) }
+func (name queueOption) Type() OptionType   { return QueueOpt }
+func (name queueOption) Value() interface{} { return string(name) }
 
 // TaskID returns an option to specify the task ID.
 func TaskID(id string) Option {
@@ -142,7 +144,8 @@ func (t deadlineOption) Value() interface{} { return time.Time(t) }
 
 // Unique returns an option to enqueue a task only if the given task is unique.
 // Task enqueued with this option is guaranteed to be unique within the given ttl.
-// Once the task gets processed successfully or once the TTL has expired, another task with the same uniqueness may be enqueued.
+// Once the task gets processed successfully or once the TTL has expired,
+// another task with the same uniqueness may be enqueued.
 // ErrDuplicateTask error is returned when enqueueing a duplicate task.
 // TTL duration must be greater than or equal to 1 second.
 //
@@ -193,6 +196,16 @@ func (ttl retentionOption) String() string     { return fmt.Sprintf("Retention(%
 func (ttl retentionOption) Type() OptionType   { return RetentionOpt }
 func (ttl retentionOption) Value() interface{} { return time.Duration(ttl) }
 
+// Group returns an option to specify the group used for the task.
+// Tasks in a given queue with the same group will be aggregated into one task before passed to Handler.
+func Group(name string) Option {
+	return groupOption(name)
+}
+
+func (name groupOption) String() string     { return fmt.Sprintf("Group(%q)", string(name)) }
+func (name groupOption) Type() OptionType   { return GroupOpt }
+func (name groupOption) Value() interface{} { return string(name) }
+
 // ErrDuplicateTask indicates that the given task could not be enqueued since it's a duplicate of another task.
 //
 // ErrDuplicateTask error only applies to tasks enqueued with a Unique option.
@@ -212,6 +225,7 @@ type option struct {
 	uniqueTTL time.Duration
 	processAt time.Time
 	retention time.Duration
+	group     string
 }
 
 // composeOptions merges user provided options into the default options
@@ -223,7 +237,7 @@ func composeOptions(opts ...Option) (option, error) {
 		retry:     defaultMaxRetry,
 		queue:     base.DefaultQueueName,
 		taskID:    uuid.NewString(),
-		timeout:   0, // do not set to deafultTimeout here
+		timeout:   0, // do not set to defaultTimeout here
 		deadline:  time.Time{},
 		processAt: time.Now(),
 	}
@@ -239,8 +253,8 @@ func composeOptions(opts ...Option) (option, error) {
 			res.queue = qname
 		case taskIDOption:
 			id := string(opt)
-			if err := validateTaskID(id); err != nil {
-				return option{}, err
+			if isBlank(id) {
+				return option{}, errors.New("task ID cannot be empty")
 			}
 			res.taskID = id
 		case timeoutOption:
@@ -259,6 +273,12 @@ func composeOptions(opts ...Option) (option, error) {
 			res.processAt = time.Now().Add(time.Duration(opt))
 		case retentionOption:
 			res.retention = time.Duration(opt)
+		case groupOption:
+			key := string(opt)
+			if isBlank(key) {
+				return option{}, errors.New("group key cannot be empty")
+			}
+			res.group = key
 		default:
 			// ignore unexpected option
 		}
@@ -266,12 +286,9 @@ func composeOptions(opts ...Option) (option, error) {
 	return res, nil
 }
 
-// validates user provided task ID string.
-func validateTaskID(id string) error {
-	if strings.TrimSpace(id) == "" {
-		return errors.New("task ID cannot be empty")
-	}
-	return nil
+// isBlank returns true if the given s is empty or consist of all whitespaces.
+func isBlank(s string) bool {
+	return strings.TrimSpace(s) == ""
 }
 
 const (
@@ -290,7 +307,7 @@ var (
 
 // Close closes the connection with redis.
 func (c *Client) Close() error {
-	return c.rdb.Close()
+	return c.broker.Close()
 }
 
 // Enqueue enqueues the given task to a queue.
@@ -300,7 +317,7 @@ func (c *Client) Close() error {
 // The argument opts specifies the behavior of task processing.
 // If there are conflicting Option values the last one overrides others.
 // Any options provided to NewTask can be overridden by options passed to Enqueue.
-// By deafult, max retry is set to 25 and timeout is set to 30 minutes.
+// By default, max retry is set to 25 and timeout is set to 30 minutes.
 //
 // If no ProcessAt or ProcessIn options are provided, the task will be pending immediately.
 //
@@ -316,12 +333,15 @@ func (c *Client) Enqueue(task *Task, opts ...Option) (*TaskInfo, error) {
 // The argument opts specifies the behavior of task processing.
 // If there are conflicting Option values the last one overrides others.
 // Any options provided to NewTask can be overridden by options passed to Enqueue.
-// By deafult, max retry is set to 25 and timeout is set to 30 minutes.
+// By default, max retry is set to 25 and timeout is set to 30 minutes.
 //
 // If no ProcessAt or ProcessIn options are provided, the task will be pending immediately.
 //
 // The first argument context applies to the enqueue operation. To specify task timeout and deadline, use Timeout and Deadline option instead.
 func (c *Client) EnqueueContext(ctx context.Context, task *Task, opts ...Option) (*TaskInfo, error) {
+	if task == nil {
+		return nil, fmt.Errorf("task cannot be nil")
+	}
 	if strings.TrimSpace(task.Type()) == "" {
 		return nil, fmt.Errorf("task typename cannot be empty")
 	}
@@ -356,17 +376,23 @@ func (c *Client) EnqueueContext(ctx context.Context, task *Task, opts ...Option)
 		Deadline:  deadline.Unix(),
 		Timeout:   int64(timeout.Seconds()),
 		UniqueKey: uniqueKey,
+		GroupKey:  opt.group,
 		Retention: int64(opt.retention.Seconds()),
 	}
 	now := time.Now()
 	var state base.TaskState
-	if opt.processAt.Before(now) || opt.processAt.Equal(now) {
+	if opt.processAt.After(now) {
+		err = c.schedule(ctx, msg, opt.processAt, opt.uniqueTTL)
+		state = base.TaskStateScheduled
+	} else if opt.group != "" {
+		// Use zero value for processAt since we don't know when the task will be aggregated and processed.
+		opt.processAt = time.Time{}
+		err = c.addToGroup(ctx, msg, opt.group, opt.uniqueTTL)
+		state = base.TaskStateAggregating
+	} else {
 		opt.processAt = now
 		err = c.enqueue(ctx, msg, opt.uniqueTTL)
 		state = base.TaskStatePending
-	} else {
-		err = c.schedule(ctx, msg, opt.processAt, opt.uniqueTTL)
-		state = base.TaskStateScheduled
 	}
 	switch {
 	case errors.Is(err, errors.ErrDuplicateTask):
@@ -381,15 +407,22 @@ func (c *Client) EnqueueContext(ctx context.Context, task *Task, opts ...Option)
 
 func (c *Client) enqueue(ctx context.Context, msg *base.TaskMessage, uniqueTTL time.Duration) error {
 	if uniqueTTL > 0 {
-		return c.rdb.EnqueueUnique(ctx, msg, uniqueTTL)
+		return c.broker.EnqueueUnique(ctx, msg, uniqueTTL)
 	}
-	return c.rdb.Enqueue(ctx, msg)
+	return c.broker.Enqueue(ctx, msg)
 }
 
 func (c *Client) schedule(ctx context.Context, msg *base.TaskMessage, t time.Time, uniqueTTL time.Duration) error {
 	if uniqueTTL > 0 {
 		ttl := t.Add(uniqueTTL).Sub(time.Now())
-		return c.rdb.ScheduleUnique(ctx, msg, t, ttl)
+		return c.broker.ScheduleUnique(ctx, msg, t, ttl)
 	}
-	return c.rdb.Schedule(ctx, msg, t)
+	return c.broker.Schedule(ctx, msg, t)
+}
+
+func (c *Client) addToGroup(ctx context.Context, msg *base.TaskMessage, group string, uniqueTTL time.Duration) error {
+	if uniqueTTL > 0 {
+		return c.broker.AddToGroupUnique(ctx, msg, group, uniqueTTL)
+	}
+	return c.broker.AddToGroup(ctx, msg, group)
 }

client_test.go

@@ -12,8 +12,8 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 func TestClientEnqueueWithProcessAtOption(t *testing.T) {
@@ -478,6 +478,158 @@ func TestClientEnqueue(t *testing.T) {
 	}
 }
 
+func TestClientEnqueueWithGroupOption(t *testing.T) {
+	r := setup(t)
+	client := NewClient(getRedisConnOpt(t))
+	defer client.Close()
+
+	task := NewTask("mytask", []byte("foo"))
+	now := time.Now()
+
+	tests := []struct {
+		desc          string
+		task          *Task
+		opts          []Option
+		wantInfo      *TaskInfo
+		wantPending   map[string][]*base.TaskMessage
+		wantGroups    map[string]map[string][]base.Z // map queue name to a set of groups
+		wantScheduled map[string][]base.Z
+	}{
+		{
+			desc: "With only Group option",
+			task: task,
+			opts: []Option{
+				Group("mygroup"),
+			},
+			wantInfo: &TaskInfo{
+				Queue:         "default",
+				Group:         "mygroup",
+				Type:          task.Type(),
+				Payload:       task.Payload(),
+				State:         TaskStateAggregating,
+				MaxRetry:      defaultMaxRetry,
+				Retried:       0,
+				LastErr:       "",
+				LastFailedAt:  time.Time{},
+				Timeout:       defaultTimeout,
+				Deadline:      time.Time{},
+				NextProcessAt: time.Time{},
+			},
+			wantPending: map[string][]*base.TaskMessage{
+				"default": {}, // should not be pending
+			},
+			wantGroups: map[string]map[string][]base.Z{
+				"default": {
+					"mygroup": {
+						{
+							Message: &base.TaskMessage{
+								Type:     task.Type(),
+								Payload:  task.Payload(),
+								Retry:    defaultMaxRetry,
+								Queue:    "default",
+								Timeout:  int64(defaultTimeout.Seconds()),
+								Deadline: noDeadline.Unix(),
+								GroupKey: "mygroup",
+							},
+							Score: now.Unix(),
+						},
+					},
+				},
+			},
+			wantScheduled: map[string][]base.Z{
+				"default": {},
+			},
+		},
+		{
+			desc: "With Group and ProcessIn options",
+			task: task,
+			opts: []Option{
+				Group("mygroup"),
+				ProcessIn(30 * time.Minute),
+			},
+			wantInfo: &TaskInfo{
+				Queue:         "default",
+				Group:         "mygroup",
+				Type:          task.Type(),
+				Payload:       task.Payload(),
+				State:         TaskStateScheduled,
+				MaxRetry:      defaultMaxRetry,
+				Retried:       0,
+				LastErr:       "",
+				LastFailedAt:  time.Time{},
+				Timeout:       defaultTimeout,
+				Deadline:      time.Time{},
+				NextProcessAt: now.Add(30 * time.Minute),
+			},
+			wantPending: map[string][]*base.TaskMessage{
+				"default": {}, // should not be pending
+			},
+			wantGroups: map[string]map[string][]base.Z{
+				"default": {
+					"mygroup": {}, // should not be added to the group yet
+				},
+			},
+			wantScheduled: map[string][]base.Z{
+				"default": {
+					{
+						Message: &base.TaskMessage{
+							Type:     task.Type(),
+							Payload:  task.Payload(),
+							Retry:    defaultMaxRetry,
+							Queue:    "default",
+							Timeout:  int64(defaultTimeout.Seconds()),
+							Deadline: noDeadline.Unix(),
+							GroupKey: "mygroup",
+						},
+						Score: now.Add(30 * time.Minute).Unix(),
+					},
+				},
+			},
+		},
+	}
+
+	for _, tc := range tests {
+		h.FlushDB(t, r) // clean up db before each test case.
+
+		gotInfo, err := client.Enqueue(tc.task, tc.opts...)
+		if err != nil {
+			t.Error(err)
+			continue
+		}
+		cmpOptions := []cmp.Option{
+			cmpopts.IgnoreFields(TaskInfo{}, "ID"),
+			cmpopts.EquateApproxTime(500 * time.Millisecond),
+		}
+		if diff := cmp.Diff(tc.wantInfo, gotInfo, cmpOptions...); diff != "" {
+			t.Errorf("%s;\nEnqueue(task) returned %v, want %v; (-want,+got)\n%s",
+				tc.desc, gotInfo, tc.wantInfo, diff)
+		}
+
+		for qname, want := range tc.wantPending {
+			got := h.GetPendingMessages(t, r, qname)
+			if diff := cmp.Diff(want, got, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
+				t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.PendingKey(qname), diff)
+			}
+		}
+
+		for qname, groups := range tc.wantGroups {
+			for groupKey, want := range groups {
+				got := h.GetGroupEntries(t, r, qname, groupKey)
+				if diff := cmp.Diff(want, got, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
+					t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.GroupKey(qname, groupKey), diff)
+				}
+			}
+		}
+
+		for qname, want := range tc.wantScheduled {
+			gotScheduled := h.GetScheduledEntries(t, r, qname)
+			if diff := cmp.Diff(want, gotScheduled, h.IgnoreIDOpt, cmpopts.EquateEmpty()); diff != "" {
+				t.Errorf("%s;\nmismatch found in %q; (-want,+got)\n%s", tc.desc, base.ScheduledKey(qname), diff)
+			}
+		}
+	}
+}
+
 func TestClientEnqueueWithTaskIDOption(t *testing.T) {
 	r := setup(t)
 	client := NewClient(getRedisConnOpt(t))
@@ -707,6 +859,11 @@ func TestClientEnqueueError(t *testing.T) {
 		task *Task
 		opts []Option
 	}{
+		{
+			desc: "With nil task",
+			task: nil,
+			opts: []Option{},
+		},
 		{
 			desc: "With empty queue name",
 			task: task,

context.go

@@ -28,7 +28,7 @@ func GetRetryCount(ctx context.Context) (n int, ok bool) {
 
 // GetMaxRetry extracts maximum retry from a context, if any.
 //
-// Return value n indicates the maximum number of times the assoicated task
+// Return value n indicates the maximum number of times the associated task
 // can be retried if ProcessTask returns a non-nil error.
 func GetMaxRetry(ctx context.Context) (n int, ok bool) {
 	return asynqcontext.GetMaxRetry(ctx)
@@ -36,7 +36,7 @@ func GetMaxRetry(ctx context.Context) (n int, ok bool) {
 
 // GetQueueName extracts queue name from a context, if any.
 //
-// Return value qname indicates which queue the task was pulled from.
-func GetQueueName(ctx context.Context) (qname string, ok bool) {
+// Return value queue indicates which queue the task was pulled from.
+func GetQueueName(ctx context.Context) (queue string, ok bool) {
 	return asynqcontext.GetQueueName(ctx)
 }

forwarder.go

@@ -56,13 +56,15 @@ func (f *forwarder) start(wg *sync.WaitGroup) {
 	wg.Add(1)
 	go func() {
 		defer wg.Done()
+		timer := time.NewTimer(f.avgInterval)
 		for {
 			select {
 			case <-f.done:
 				f.logger.Debug("Forwarder done")
 				return
-			case <-time.After(f.avgInterval):
+			case <-timer.C:
 				f.exec()
+				timer.Reset(f.avgInterval)
 			}
 		}
 	}()

forwarder_test.go

@@ -10,9 +10,9 @@ import (
 	"time"
 
 	"github.com/google/go-cmp/cmp"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 func TestForwarder(t *testing.T) {

go.mod

@@ -3,17 +3,15 @@ module github.com/hibiken/asynq
 go 1.14
 
 require (
-	github.com/go-redis/redis/v8 v8.11.2
-	github.com/golang/protobuf v1.4.2
+	github.com/golang/protobuf v1.5.2
 	github.com/google/go-cmp v0.5.6
 	github.com/google/uuid v1.2.0
-	github.com/kr/pretty v0.1.0 // indirect
+	github.com/redis/go-redis/v9 v9.0.3
 	github.com/robfig/cron/v3 v3.0.1
 	github.com/spf13/cast v1.3.1
-	github.com/stretchr/testify v1.6.1 // indirect
-	go.uber.org/goleak v0.10.0
-	golang.org/x/sys v0.0.0-20210112080510-489259a85091
+	go.uber.org/goleak v1.1.12
+	golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e
 	golang.org/x/time v0.0.0-20190308202827-9d24e82272b4
-	google.golang.org/protobuf v1.25.0
+	google.golang.org/protobuf v1.26.0
 	gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 // indirect
 )

go.sum

@@ -1,177 +1,82 @@
-cloud.google.com/go v0.26.0 h1:e0WKqKTd5BnrG8aKH3J3h+QvEIQtSUcf2n5UZ5ZgLtQ=
-cloud.google.com/go v0.26.0/go.mod h1:aQUYkXzVsufM+DwF1aE+0xfcU+56JwCaLick0ClmMTw=
-github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
-github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
-github.com/census-instrumentation/opencensus-proto v0.2.1 h1:glEXhBS5PSLLv4IXzLA5yPRVX4bilULVyxxbrfOtDAk=
-github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
-github.com/cespare/xxhash/v2 v2.1.1 h1:6MnRN8NT7+YBpUIWxHtefFZOKTAPgGjpQSxqLNn0+qY=
-github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/client9/misspell v0.3.4 h1:ta993UF76GwbvJcIo3Y68y/M3WxlpEHPWIGDkJYwzJI=
-github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
+github.com/bsm/ginkgo/v2 v2.7.0 h1:ItPMPH90RbmZJt5GtkcNvIRuGEdwlBItdNVoyzaNQao=
+github.com/bsm/ginkgo/v2 v2.7.0/go.mod h1:AiKlXPm7ItEHNc/2+OkrNG4E0ITzojb9/xWzvQ9XZ9w=
+github.com/bsm/gomega v1.26.0 h1:LhQm+AFcgV2M0WyKroMASzAzCAJVpAxQXv4SaI9a69Y=
+github.com/bsm/gomega v1.26.0/go.mod h1:JyEr/xRbxbtgWNi8tIEVPUYZ5Dzef52k01W3YH0H+O0=
+github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
+github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/davecgh/go-spew v1.1.0/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/davecgh/go-spew v1.1.1 h1:vj9j/u1bqnvCEfJOwUhtlOARqs3+rkHYY13jYWTU97c=
 github.com/davecgh/go-spew v1.1.1/go.mod h1:J7Y8YcW2NihsgmVo/mv3lAwl/skON4iLHjSsI+c5H38=
 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f h1:lO4WD4F/rVNCu3HqELle0jiPLLBs70cWOduZpkS1E78=
 github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f/go.mod h1:cuUVRXasLTGF7a8hSLbxyZXjz+1KgoB3wDUb6vlszIc=
-github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473 h1:4cmBvAEBNJaGARUEs3/suWRyfyBfhf7I60WBZq+bv2w=
-github.com/envoyproxy/go-control-plane v0.9.1-0.20191026205805-5f8ba28d4473/go.mod h1:YTl/9mNaCwkRvm6d1a2C3ymFceY/DCBVvsKhRF0iEA4=
-github.com/envoyproxy/protoc-gen-validate v0.1.0 h1:EQciDnbrYxy13PgWoY8AqoxGiPrpgBZ1R8UNe3ddc+A=
-github.com/envoyproxy/protoc-gen-validate v0.1.0/go.mod h1:iSmxcyjqTsJpI2R4NaDN7+kN2VEUnK/pcBlmesArF7c=
-github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
-github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
-github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
-github.com/go-redis/redis/v8 v8.11.2 h1:WqlSpAwz8mxDSMCvbyz1Mkiqe0LE5OY4j3lgkvu1Ts0=
-github.com/go-redis/redis/v8 v8.11.2/go.mod h1:DLomh7y2e3ggQXQLd1YgmvIfecPJoFl7WU5SOQ/r06M=
-github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b h1:VKtxabqXZkF25pY9ekfRL6a582T4P37/31XEstQ5p58=
-github.com/golang/glog v0.0.0-20160126235308-23def4e6c14b/go.mod h1:SBH7ygxi8pfUlaOkMMuAQtPIUF8ecWP5IEl/CR7VP2Q=
-github.com/golang/mock v1.1.1 h1:G5FRp8JnTd7RQH5kemVNlMeyXQAztQ3mOWV95KxsXH8=
-github.com/golang/mock v1.1.1/go.mod h1:oTYuIxOrZwtPieC+H1uAHpcLFnEyAGVDL/k47Jfbm0A=
-github.com/golang/protobuf v1.2.0/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.3.2/go.mod h1:6lQm79b+lXiMfvg/cZm0SGofjICqVBUtrP5yJMmIC1U=
-github.com/golang/protobuf v1.4.0-rc.1/go.mod h1:ceaxUfeHdC40wWswd/P6IGgMaK3YpKi5j83Wpe3EHw8=
-github.com/golang/protobuf v1.4.0-rc.1.0.20200221234624-67d41d38c208/go.mod h1:xKAWHe0F5eneWXFV3EuXVDTCmh+JuBKY0li0aMyXATA=
-github.com/golang/protobuf v1.4.0-rc.2/go.mod h1:LlEzMj4AhA7rCAGe4KMBDvJI+AwstrUpVNzEA03Pprs=
-github.com/golang/protobuf v1.4.0-rc.4.0.20200313231945-b860323f09d0/go.mod h1:WU3c8KckQ9AFe+yFwt9sWVRKCVIyN9cPHBJSNnbL67w=
-github.com/golang/protobuf v1.4.0/go.mod h1:jodUvKwWbYaEsadDk5Fwe5c77LiNKVO9IDvqG2KuDX0=
-github.com/golang/protobuf v1.4.1/go.mod h1:U8fpvMrcmy5pZrNK1lt4xCsGvpyWQ/VVv6QDs8UjoX8=
-github.com/golang/protobuf v1.4.2 h1:+Z5KGCizgyZCbGh1KZqA0fcLLkwbsjIzS4aV2v7wJX0=
-github.com/golang/protobuf v1.4.2/go.mod h1:oDoupMAO8OvCJWAcko0GGGIgR6R6ocIYbsSw735rRwI=
-github.com/google/go-cmp v0.2.0/go.mod h1:oXzfMopK8JAjlY9xF4vHSVASa0yLyX7SntLO5aqRK0M=
-github.com/google/go-cmp v0.3.0/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/go-cmp v0.3.1/go.mod h1:8QqcDgzrUqlUb/G2PQTWiueGozuR1884gddMywk6iLU=
-github.com/google/go-cmp v0.4.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
-github.com/google/go-cmp v0.5.0/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
+github.com/golang/protobuf v1.5.0/go.mod h1:FsONVRAS9T7sI+LIUmWTfcYkHO4aIWwzhcaSAoJOfIk=
+github.com/golang/protobuf v1.5.2 h1:ROPKBNFfQgOUMifHyP+KYbvpjbdoFNs+aK7DXlji0Tw=
+github.com/golang/protobuf v1.5.2/go.mod h1:XVQd3VNwM+JqD3oG2Ue2ip4fOMUkwXdXDdiuN0vRsmY=
+github.com/google/go-cmp v0.5.5/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/go-cmp v0.5.6 h1:BKbKCqvP6I+rmFHt06ZmyQtvB8xAkWdhFyr0ZUNZcxQ=
 github.com/google/go-cmp v0.5.6/go.mod h1:v8dTdLbMG2kIc/vJvl+f65V22dbkXbowE6jgT/gNBxE=
 github.com/google/uuid v1.2.0 h1:qJYtXnJRWmpe7m/3XlyhrsLrEURqHRM2kxzoxXqyUDs=
 github.com/google/uuid v1.2.0/go.mod h1:TIyPZe4MgqvfeYDBFedMoGGpEw/LqOeaOT+nhxU+yHo=
-github.com/hpcloud/tail v1.0.0 h1:nfCOvKYfkgYP8hkirhJocXT2+zOD8yUNjXaWfTlyFKI=
-github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
 github.com/kr/pretty v0.1.0 h1:L/CwN0zerZDmRFUapSPitk6f+Q3+0za1rQkzVuMiMFI=
 github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORNo=
-github.com/kr/pty v1.1.1 h1:VkoXIwSboBpnk99O/KFauAEILuNHv5DVFKZMBN/gUgw=
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
-github.com/nxadm/tail v1.4.4 h1:DQuhQpB1tVlglWS2hLQ5OV6B5r8aGxSrPc5Qo6uTN78=
-github.com/nxadm/tail v1.4.4/go.mod h1:kenIhsEOeOJmVchQTgglprH7qJGnHDVpk1VPCcaMI8A=
-github.com/onsi/ginkgo v1.6.0/go.mod h1:lLunBs/Ym6LB5Z9jYTR76FiuTmxDTDusOGeTQH+WWjE=
-github.com/onsi/ginkgo v1.12.1/go.mod h1:zj2OWP4+oCPe1qIXoGWkgMRwljMUYCdkwsT2108oapk=
-github.com/onsi/ginkgo v1.15.0 h1:1V1NfVQR87RtWAgp1lv9JZJ5Jap+XFGKPi00andXGi4=
-github.com/onsi/ginkgo v1.15.0/go.mod h1:hF8qUzuuC8DJGygJH3726JnCZX4MYbRB8yFfISqnKUg=
-github.com/onsi/gomega v1.7.1/go.mod h1:XdKZgCCFLUoM/7CFJVPcG8C1xQ1AJ0vpAezJrB7JYyY=
-github.com/onsi/gomega v1.10.1/go.mod h1:iN09h71vgCQne3DLsj+A5owkum+a2tYe+TOCB1ybHNo=
-github.com/onsi/gomega v1.10.5 h1:7n6FEkpFmfCoo2t+YYqXH0evK+a9ICQz0xcAy9dYcaQ=
-github.com/onsi/gomega v1.10.5/go.mod h1:gza4q3jKQJijlu05nKWRCW/GavJumGt8aNRxWg7mt48=
 github.com/pmezard/go-difflib v1.0.0 h1:4DBwDE0NGyQoBHbLQYPwSUPoCMWR5BEzIk/f1lZbAQM=
 github.com/pmezard/go-difflib v1.0.0/go.mod h1:iKH77koFhYxTK1pcRnkKkqfTogsbg7gZNVY4sRDYZ/4=
-github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4 h1:gQz4mCbXsO+nc9n1hCxHcGA3Zx3Eo+UHZoInFGUIXNM=
-github.com/prometheus/client_model v0.0.0-20190812154241-14fe0d1b01d4/go.mod h1:xMI15A0UPsDsEKsMN9yxemIoYk6Tm2C1GtYGdfGttqA=
+github.com/redis/go-redis/v9 v9.0.3 h1:+7mmR26M0IvyLxGZUHxu4GiBkJkVDid0Un+j4ScYu4k=
+github.com/redis/go-redis/v9 v9.0.3/go.mod h1:WqMKv5vnQbRuZstUwxQI195wHy+t4PuXDOjzMvcuQHk=
 github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
 github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
 github.com/spf13/cast v1.3.1 h1:nFm6S0SMdyzrzcmThSipiEubIDy8WEXKNZ0UOgiRpng=
 github.com/spf13/cast v1.3.1/go.mod h1:Qx5cxh0v+4UWYiBimWS+eyWzqEqokIECu5etghLkUJE=
-github.com/stretchr/objx v0.1.0 h1:4G4v2dO3VZwixGIRoQ5Lfboy6nUhCyYzaqnIAPPhYs4=
 github.com/stretchr/objx v0.1.0/go.mod h1:HFkY916IF+rwdDfMAkV7OtwuqBVzrE8GR6GFx+wExME=
 github.com/stretchr/testify v1.2.2/go.mod h1:a8OnRcib4nhh0OaRAV+Yts87kKdq0PP7pXfy6kDkUVs=
-github.com/stretchr/testify v1.6.1 h1:hDPOHmpOpP40lSULcqw7IrRb/u7w6RpDC9399XyoNd0=
-github.com/stretchr/testify v1.6.1/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
-github.com/yuin/goldmark v1.2.1 h1:ruQGxdhGHe7FWOJPT0mKs5+pD2Xs1Bm/kdGlHO04FmM=
-github.com/yuin/goldmark v1.2.1/go.mod h1:3hX8gzYuyVAZsxl0MRgGTJEmQBFcNTphYh9decYSb74=
-go.uber.org/goleak v0.10.0 h1:G3eWbSNIskeRqtsN/1uI5B+eP73y3JUuBsv9AZjehb4=
-go.uber.org/goleak v0.10.0/go.mod h1:VCZuO8V8mFPlL0F5J5GK1rtHV3DrFcQ1R8ryq7FK0aI=
+github.com/stretchr/testify v1.7.0 h1:nwc3DEeHmmLAfoZucVR881uASk0Mfjw8xYJ99tb5CcY=
+github.com/stretchr/testify v1.7.0/go.mod h1:6Fq8oRcR53rry900zMqJjRRixrwX3KX962/h/Wwjteg=
+github.com/yuin/goldmark v1.3.5/go.mod h1:mwnBkeHKe2W/ZEtQ+71ViKU8L12m81fl3OWwC1Zlc8k=
+go.uber.org/goleak v1.1.12 h1:gZAh5/EyT/HQwlpkCy6wTpqfH9H8Lz8zbm3dZh+OyzA=
+go.uber.org/goleak v1.1.12/go.mod h1:cwTWslyiVhfpKIDGSZEM2HlOvcqm+tG4zioyIeLoqMQ=
 golang.org/x/crypto v0.0.0-20190308221718-c2843e01d9a2/go.mod h1:djNgcEr1/C05ACkg1iLfiJU5Ep61QUkGW8qpdssI0+w=
 golang.org/x/crypto v0.0.0-20191011191535-87dc89f01550/go.mod h1:yigFU9vqHzYiE8UmvKecakEJjdnWj3jj499lnFckfCI=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9 h1:psW17arqaxU48Z5kZ0CQnkZWQJsqcURM6tKiBApRjXI=
-golang.org/x/crypto v0.0.0-20200622213623-75b288015ac9/go.mod h1:LzIPMQfyMNhhGPhUkYOs5KpL4U8rLKemX1yGLhDgUto=
-golang.org/x/exp v0.0.0-20190121172915-509febef88a4 h1:c2HOrn5iMezYjSlGPncknSEr/8x5LELb/ilJbXi9DEA=
-golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
-golang.org/x/lint v0.0.0-20181026193005-c67002cb31c3/go.mod h1:UVdnD1Gm6xHRNCYTkRU2/jEulfH38KcIWyp/GAMgvoE=
-golang.org/x/lint v0.0.0-20190227174305-5b3e6a55c961/go.mod h1:wehouNa3lNwaWXcvxsM5YxQ5yQlVC4a0KAMCusXpPoU=
-golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3 h1:XQyxROzUlZH+WIQwySDgnISgOivlhjIEwaQaJEJrrN0=
-golang.org/x/lint v0.0.0-20190313153728-d0100b6bd8b3/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
-golang.org/x/mod v0.3.0 h1:RM4zey1++hCTbCVQfnWeKs9/IEsaBLA8vTkd0WVtmH4=
-golang.org/x/mod v0.3.0/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
-golang.org/x/net v0.0.0-20180724234803-3673e40ba225/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20180826012351-8a410e7b638d/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20180906233101-161cd47e91fd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
-golang.org/x/net v0.0.0-20190213061140-3a22650c66bd/go.mod h1:mL1N/T3taQHkDXs73rZJwtUhF3w3ftmwwsq0BUmARs4=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de h1:5hukYrvBGR8/eNkX5mdUezrA6JiaEZDtJb9Ei+1LlBs=
+golang.org/x/lint v0.0.0-20190930215403-16217165b5de/go.mod h1:6SW0HCj/g11FgYtHlgUYUwCkIfeOF89ocIRzGO/8vkc=
+golang.org/x/mod v0.4.2 h1:Gz96sIWK3OalVv/I/qNygP42zyoKp3xptRVCWRFEBvo=
+golang.org/x/mod v0.4.2/go.mod h1:s0Qsj1ACt9ePp/hMypM3fl4fZqREWJwdYDEqhRiZZUA=
 golang.org/x/net v0.0.0-20190311183353-d8887717615a/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190404232315-eb5bcb51f2a3/go.mod h1:t9HGtf8HONx5eT2rtn7q6eTqICYqUVnKs3thJo3Qplg=
 golang.org/x/net v0.0.0-20190620200207-3b0461eec859/go.mod h1:z5CRVTTTmAJ677TzLLGU+0bjPO0LkuOLi4/5GtJWs/s=
-golang.org/x/net v0.0.0-20200520004742-59133d7f0dd7/go.mod h1:qpuaurCH72eLCgpAm/N6yyVIVM9cpaDIP3A8BGJEC5A=
-golang.org/x/net v0.0.0-20201021035429-f5854403a974/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb h1:eBmm0M9fYhWpKZLjQUUKka/LtIxf46G4fxeEz5KJr9U=
-golang.org/x/net v0.0.0-20201202161906-c7110b5ffcbb/go.mod h1:sp8m0HH+o8qH0wwXwYZr8TS3Oi6o0r6Gce1SSxlDquU=
-golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be h1:vEDujvNQGv4jgYKudGeI/+DAX4Jffq6hpD55MmoEvKs=
-golang.org/x/oauth2 v0.0.0-20180821212333-d2e6202438be/go.mod h1:N/0e6XlmueqKjAGxoOufVs8QHGRruUQn6yWY3a++T0U=
-golang.org/x/sync v0.0.0-20180314180146-1d60e4601c6f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20181108010431-42b317875d0f/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
+golang.org/x/net v0.0.0-20210405180319-a5a99cb37ef4/go.mod h1:p54w0d4576C0XHj96bSt6lcn1PtDYWL6XObtHCRCNQM=
 golang.org/x/sync v0.0.0-20190423024810-112230192c58/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9 h1:SQFwaSi55rU7vdNs9Yr0Z324VNlrF+0wMqRXT4St8ck=
-golang.org/x/sync v0.0.0-20201020160332-67f06af15bc9/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
-golang.org/x/sys v0.0.0-20180830151530-49385e6e1522/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
-golang.org/x/sys v0.0.0-20180909124046-d0be0721c37e/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
+golang.org/x/sync v0.0.0-20210220032951-036812b2e83c/go.mod h1:RxMgew5VJxzue5/jJTE5uejpjVlOe/izrB70Jof72aM=
 golang.org/x/sys v0.0.0-20190215142949-d0b11bdaac8a/go.mod h1:STP8DvDyc/dI5b8T5hshtkjS+E42TnysNCUPdjciGhY=
 golang.org/x/sys v0.0.0-20190412213103-97732733099d/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20190904154756-749cb33beabd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20191005200804-aed5e4c7ecf9/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20191120155948-bd437916bb0e/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200323222414-85ca7c5b95cd/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20200930185726-fdedc70b468f/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210112080510-489259a85091 h1:DMyOG0U+gKfu8JZzg2UQe9MeaC1X+xQWlAKcRnjxjCw=
-golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210330210617-4fbd30eecc44/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
+golang.org/x/sys v0.0.0-20210510120138-977fb7262007/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e h1:fLOSk5Q00efkSvAm+4xcoXD+RRmLmmulPn5I3Y9F2EM=
+golang.org/x/sys v0.0.0-20211216021012-1d35b9e2eb4e/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
-golang.org/x/text v0.3.3 h1:cokOdA+Jmi5PJGXLlLllQSgYigAEfHXJAERHVMaCc2k=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/tools v0.0.0-20180917221912-90fa682c2a6e/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20190114222345-bf090417da8b/go.mod h1:n7NCudcB/nEzxVGmLbDWY5pfWTLqBcC2KZ6jyYvM4mQ=
-golang.org/x/tools v0.0.0-20190226205152-f727befe758c/go.mod h1:9Yl7xja0Znq3iFh3HoIrodX9oNMXvdceNzlUR8zjMvY=
 golang.org/x/tools v0.0.0-20190311212946-11955173bddd/go.mod h1:LCzVGOaR6xXOjkQ3onu1FJEFr0SW1gC7cKk1uF8kGRs=
-golang.org/x/tools v0.0.0-20190524140312-2c0ae7006135/go.mod h1:RgjU9mgBXZiqYHBnxXauZ1Gv1EHHAz9KjViQ78xBX0Q=
 golang.org/x/tools v0.0.0-20191119224855-298f0cb1881e/go.mod h1:b+2E5dAYhXwXZwtnZ6UAqBI28+e2cm9otk0dWdXHAEo=
-golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e h1:4nW4NLDYnU28ojHaHO8OVxFHk/aQ33U01a9cjED+pzE=
-golang.org/x/tools v0.0.0-20201224043029-2b0845dc783e/go.mod h1:emZCQorbCU4vsT4fOWvOPXz4eW1wZW4PmDk9uLelYpA=
+golang.org/x/tools v0.1.5 h1:ouewzE6p+/VEB31YYnTbEJdi8pFqKp4P4n85vwo3DHA=
+golang.org/x/tools v0.1.5/go.mod h1:o0xws9oXOQQZyjljx8fwUC0k7L1pTE6eaCbjGeHmOkk=
 golang.org/x/xerrors v0.0.0-20190717185122-a985d3407aa7/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191011141410-1b5146add898/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20191204190536-9bdfabe68543/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1 h1:go1bK/D/BFZV2I8cIQd1NKEZ+0owSTG1fDTci4IqFcE=
 golang.org/x/xerrors v0.0.0-20200804184101-5ec99f83aff1/go.mod h1:I/5z698sn9Ka8TeJc9MKroUUfqBBauWjQqLJ2OPfmY0=
-google.golang.org/appengine v1.1.0/go.mod h1:EbEs0AVv82hx2wNQdGPgUI5lhzA/G0D9YwlJXL52JkM=
-google.golang.org/appengine v1.4.0 h1:/wp5JvzpHIxhs/dumFmF7BXTf3Z+dd4uXta4kVyO508=
-google.golang.org/appengine v1.4.0/go.mod h1:xpcJRLb0r/rnEns0DIKYYv+WjYCduHsrkT7/EB5XEv4=
-google.golang.org/genproto v0.0.0-20180817151627-c66870c02cf8/go.mod h1:JiN7NxoALGmiZfu7CAH4rXhgtRTLTxftemlI0sWmxmc=
-google.golang.org/genproto v0.0.0-20190819201941-24fa4b261c55/go.mod h1:DMBHOl98Agz4BDEuKkezgsaosCRResVns1a3J2ZsMNc=
-google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013 h1:+kGHl1aib/qcwaRi1CbqBZ1rk19r85MNUf8HaBghugY=
-google.golang.org/genproto v0.0.0-20200526211855-cb27e3aa2013/go.mod h1:NbSheEEYHJ7i3ixzK3sjbqSGDJWnxyFXZblF3eUsNvo=
-google.golang.org/grpc v1.19.0/go.mod h1:mqu4LbDTu4XGKhr4mRzUsmM4RtVoemTSY81AxZiDr8c=
-google.golang.org/grpc v1.23.0/go.mod h1:Y5yQAOtifL1yxbo5wqy6BxZv8vAUGQwXBOALyacEbxg=
-google.golang.org/grpc v1.27.0 h1:rRYRFMVgRv6E0D70Skyfsr28tDXIuuPZyWGMPdMcnXg=
-google.golang.org/grpc v1.27.0/go.mod h1:qbnxyOmOxrQa7FizSgH+ReBfzJrCY1pSN7KXBS8abTk=
-google.golang.org/protobuf v0.0.0-20200109180630-ec00e32a8dfd/go.mod h1:DFci5gLYBciE7Vtevhsrf46CRTquxDuWsQurQQe4oz8=
-google.golang.org/protobuf v0.0.0-20200221191635-4d8936d0db64/go.mod h1:kwYJMbMJ01Woi6D6+Kah6886xMZcty6N08ah7+eCXa0=
-google.golang.org/protobuf v0.0.0-20200228230310-ab0ca4ff8a60/go.mod h1:cfTl7dwQJ+fmap5saPgwCLgHXTUD7jkjRqWcaiX5VyM=
-google.golang.org/protobuf v1.20.1-0.20200309200217-e05f789c0967/go.mod h1:A+miEFZTKqfCUM6K7xSMQL9OKL/b6hQv+e19PK+JZNE=
-google.golang.org/protobuf v1.21.0/go.mod h1:47Nbq4nVaFHyn7ilMalzfO3qCViNmqZ2kzikPIcrTAo=
-google.golang.org/protobuf v1.22.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
-google.golang.org/protobuf v1.23.0/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
-google.golang.org/protobuf v1.23.1-0.20200526195155-81db48ad09cc/go.mod h1:EGpADcykh3NcUnDUJcl1+ZksZNG86OlYog2l/sGQquU=
-google.golang.org/protobuf v1.25.0 h1:Ejskq+SyPohKW+1uil0JJMtmHCgJPJ/qWTxr8qp+R4c=
-google.golang.org/protobuf v1.25.0/go.mod h1:9JNX74DMeImyA3h4bdi1ymwjUzf21/xIlbajtzgsN7c=
+google.golang.org/protobuf v1.26.0-rc.1/go.mod h1:jlhhOSvTdKEhbULTjvd4ARK9grFBp09yW+WbY/TyQbw=
+google.golang.org/protobuf v1.26.0 h1:bxAC2xTBsZGibn2RTntX0oH50xLsqy1OxA9tTL3p/lk=
+google.golang.org/protobuf v1.26.0/go.mod h1:9q0QmTI4eRPtz6boOQmLYwt+qCgq0jsYwAQnmE0givc=
 gopkg.in/check.v1 v0.0.0-20161208181325-20d25e280405/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
+gopkg.in/check.v1 v1.0.0-20180628173108-788fd7840127/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15 h1:YR8cESwS4TdDjEe65xsg0ogRM/Nc3DYOhEAlW+xobZo=
 gopkg.in/check.v1 v1.0.0-20190902080502-41f04d3bba15/go.mod h1:Co6ibVJAznAaIkqp8huTwlJQCZ016jof/cbN4VW5Yz0=
-gopkg.in/fsnotify.v1 v1.4.7 h1:xOHLXZwVvI9hhs+cLKq5+I5onOuwQLhQwiu63xxlHs4=
-gopkg.in/fsnotify.v1 v1.4.7/go.mod h1:Tz8NjZHkW78fSQdbUxIjBTcgA1z1m8ZHf0WmKUhAMys=
-gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7 h1:uRGJdciOHaEIrze2W8Q3AKkepLTh2hOroT7a+7czfdQ=
-gopkg.in/tomb.v1 v1.0.0-20141024135613-dd632973f1e7/go.mod h1:dt/ZhP58zS4L8KSrWDmTeBkI65Dw0HsyUHuEVlX15mw=
-gopkg.in/yaml.v2 v2.2.4/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
-gopkg.in/yaml.v2 v2.3.0 h1:clyUAQHOM3G0M3f5vQj7LuJrETvjVot3Z5el9nffUtU=
-gopkg.in/yaml.v2 v2.3.0/go.mod h1:hI93XBmqTisBFMUTm0b8Fm+jr3Dg1NNxqwp+5A1VGuI=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c h1:dUUwHk2QECo/6vqA44rthZ8ie2QXMNeKRTHCNY2nXvo=
 gopkg.in/yaml.v3 v3.0.0-20200313102051-9f266ea9e77c/go.mod h1:K4uyk7z7BCEPqu6E+C64Yfv1cQ7kz7rIZviUmN+EgEM=
-honnef.co/go/tools v0.0.0-20190102054323-c2f93a96b099/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=
-honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc h1:/hemPrYIhOhy8zYrNj+069zDB68us2sMGsfkFJO0iZs=
-honnef.co/go/tools v0.0.0-20190523083050-ea95bdfd59fc/go.mod h1:rf3lG4BRIbNafJWhAfAdb/ePZxsR/4RtNHQocxwk9r4=

heartbeat_test.go

@@ -12,17 +12,17 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
 	"github.com/hibiken/asynq/internal/testbroker"
+	h "github.com/hibiken/asynq/internal/testutil"
 	"github.com/hibiken/asynq/internal/timeutil"
 )
 
 // Test goes through a few phases.
 //
 // Phase1: Simulate Server startup; Simulate starting tasks listed in startedWorkers
-// Phase2: Simluate finishing tasks listed in finishedTasks
+// Phase2: Simulate finishing tasks listed in finishedTasks
 // Phase3: Simulate Server shutdown;
 func TestHeartbeater(t *testing.T) {
 	r := setup(t)
@@ -41,7 +41,7 @@ func TestHeartbeater(t *testing.T) {
 	t5 := h.NewTaskMessageWithQueue("task5", nil, "custom")
 	t6 := h.NewTaskMessageWithQueue("task6", nil, "default")
 
-	// Note: intentionally set to time less than now.Add(rdb.LeaseDuration) to test lease extention is working.
+	// Note: intentionally set to time less than now.Add(rdb.LeaseDuration) to test lease extension is working.
 	lease1 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
 	lease2 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)
 	lease3 := h.NewLeaseWithClock(now.Add(10*time.Second), clock)

inspector.go

@@ -10,7 +10,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/errors"
 	"github.com/hibiken/asynq/internal/rdb"
@@ -43,7 +43,32 @@ func (i *Inspector) Queues() ([]string, error) {
 	return i.rdb.AllQueues()
 }
 
-// QueueInfo represents a state of queues at a certain time.
+// Groups returns a list of all groups within the given queue.
+func (i *Inspector) Groups(queue string) ([]*GroupInfo, error) {
+	stats, err := i.rdb.GroupStats(queue)
+	if err != nil {
+		return nil, err
+	}
+	var res []*GroupInfo
+	for _, s := range stats {
+		res = append(res, &GroupInfo{
+			Group: s.Group,
+			Size:  s.Size,
+		})
+	}
+	return res, nil
+}
+
+// GroupInfo represents a state of a group at a certain time.
+type GroupInfo struct {
+	// Name of the group.
+	Group string
+
+	// Size is the total number of tasks in the group.
+	Size int
+}
+
+// QueueInfo represents a state of a queue at a certain time.
 type QueueInfo struct {
 	// Name of the queue.
 	Queue string
@@ -56,9 +81,12 @@ type QueueInfo struct {
 	Latency time.Duration
 
 	// Size is the total number of tasks in the queue.
-	// The value is the sum of Pending, Active, Scheduled, Retry, and Archived.
+	// The value is the sum of Pending, Active, Scheduled, Retry, Aggregating and Archived.
 	Size int
 
+	// Groups is the total number of groups in the queue.
+	Groups int
+
 	// Number of pending tasks.
 	Pending int
 	// Number of active tasks.
@@ -71,6 +99,8 @@ type QueueInfo struct {
 	Archived int
 	// Number of stored completed tasks.
 	Completed int
+	// Number of aggregating tasks.
+	Aggregating int
 
 	// Total number of tasks being processed within the given date (counter resets daily).
 	// The number includes both succeeded and failed tasks.
@@ -92,11 +122,11 @@ type QueueInfo struct {
 }
 
 // GetQueueInfo returns current information of the given queue.
-func (i *Inspector) GetQueueInfo(qname string) (*QueueInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) GetQueueInfo(queue string) (*QueueInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, err
 	}
-	stats, err := i.rdb.CurrentStats(qname)
+	stats, err := i.rdb.CurrentStats(queue)
 	if err != nil {
 		return nil, err
 	}
@@ -105,12 +135,14 @@ func (i *Inspector) GetQueueInfo(qname string) (*QueueInfo, error) {
 		MemoryUsage:    stats.MemoryUsage,
 		Latency:        stats.Latency,
 		Size:           stats.Size,
+		Groups:         stats.Groups,
 		Pending:        stats.Pending,
 		Active:         stats.Active,
 		Scheduled:      stats.Scheduled,
 		Retry:          stats.Retry,
 		Archived:       stats.Archived,
 		Completed:      stats.Completed,
+		Aggregating:    stats.Aggregating,
 		Processed:      stats.Processed,
 		Failed:         stats.Failed,
 		ProcessedTotal: stats.ProcessedTotal,
@@ -134,11 +166,11 @@ type DailyStats struct {
 }
 
 // History returns a list of stats from the last n days.
-func (i *Inspector) History(qname string, n int) ([]*DailyStats, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) History(queue string, n int) ([]*DailyStats, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, err
 	}
-	stats, err := i.rdb.HistoricalStats(qname, n)
+	stats, err := i.rdb.HistoricalStats(queue, n)
 	if err != nil {
 		return nil, err
 	}
@@ -175,13 +207,13 @@ var (
 // If the specified queue does not exist, DeleteQueue returns ErrQueueNotFound.
 // If force is set to false and the specified queue is not empty, DeleteQueue
 // returns ErrQueueNotEmpty.
-func (i *Inspector) DeleteQueue(qname string, force bool) error {
-	err := i.rdb.RemoveQueue(qname, force)
+func (i *Inspector) DeleteQueue(queue string, force bool) error {
+	err := i.rdb.RemoveQueue(queue, force)
 	if errors.IsQueueNotFound(err) {
-		return fmt.Errorf("%w: queue=%q", ErrQueueNotFound, qname)
+		return fmt.Errorf("%w: queue=%q", ErrQueueNotFound, queue)
 	}
 	if errors.IsQueueNotEmpty(err) {
-		return fmt.Errorf("%w: queue=%q", ErrQueueNotEmpty, qname)
+		return fmt.Errorf("%w: queue=%q", ErrQueueNotEmpty, queue)
 	}
 	return err
 }
@@ -190,8 +222,8 @@ func (i *Inspector) DeleteQueue(qname string, force bool) error {
 //
 // Returns an error wrapping ErrQueueNotFound if a queue with the given name doesn't exist.
 // Returns an error wrapping ErrTaskNotFound if a task with the given id doesn't exist in the queue.
-func (i *Inspector) GetTaskInfo(qname, id string) (*TaskInfo, error) {
-	info, err := i.rdb.GetTaskInfo(qname, id)
+func (i *Inspector) GetTaskInfo(queue, id string) (*TaskInfo, error) {
+	info, err := i.rdb.GetTaskInfo(queue, id)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -267,13 +299,13 @@ func Page(n int) ListOption {
 // ListPendingTasks retrieves pending tasks from the specified queue.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListPendingTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListPendingTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListPending(qname, pgn)
+	infos, err := i.rdb.ListPending(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -295,20 +327,20 @@ func (i *Inspector) ListPendingTasks(qname string, opts ...ListOption) ([]*TaskI
 // ListActiveTasks retrieves active tasks from the specified queue.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListActiveTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListActiveTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListActive(qname, pgn)
+	infos, err := i.rdb.ListActive(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
 	case err != nil:
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
-	expired, err := i.rdb.ListLeaseExpired(time.Now(), qname)
+	expired, err := i.rdb.ListLeaseExpired(time.Now(), queue)
 	if err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
@@ -332,17 +364,45 @@ func (i *Inspector) ListActiveTasks(qname string, opts ...ListOption) ([]*TaskIn
 	return tasks, nil
 }
 
-// ListScheduledTasks retrieves scheduled tasks from the specified queue.
-// Tasks are sorted by NextProcessAt in ascending order.
+// ListAggregatingTasks retrieves scheduled tasks from the specified group.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListScheduledTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListAggregatingTasks(queue, group string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListScheduled(qname, pgn)
+	infos, err := i.rdb.ListAggregating(queue, group, pgn)
+	switch {
+	case errors.IsQueueNotFound(err):
+		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
+	case err != nil:
+		return nil, fmt.Errorf("asynq: %v", err)
+	}
+	var tasks []*TaskInfo
+	for _, i := range infos {
+		tasks = append(tasks, newTaskInfo(
+			i.Message,
+			i.State,
+			i.NextProcessAt,
+			i.Result,
+		))
+	}
+	return tasks, nil
+}
+
+// ListScheduledTasks retrieves scheduled tasks from the specified queue.
+// Tasks are sorted by NextProcessAt in ascending order.
+//
+// By default, it retrieves the first 30 tasks.
+func (i *Inspector) ListScheduledTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
+		return nil, fmt.Errorf("asynq: %v", err)
+	}
+	opt := composeListOptions(opts...)
+	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
+	infos, err := i.rdb.ListScheduled(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -365,13 +425,13 @@ func (i *Inspector) ListScheduledTasks(qname string, opts ...ListOption) ([]*Tas
 // Tasks are sorted by NextProcessAt in ascending order.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListRetryTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListRetryTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListRetry(qname, pgn)
+	infos, err := i.rdb.ListRetry(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -394,13 +454,13 @@ func (i *Inspector) ListRetryTasks(qname string, opts ...ListOption) ([]*TaskInf
 // Tasks are sorted by LastFailedAt in descending order.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListArchivedTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListArchivedTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListArchived(qname, pgn)
+	infos, err := i.rdb.ListArchived(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -423,13 +483,13 @@ func (i *Inspector) ListArchivedTasks(qname string, opts ...ListOption) ([]*Task
 // Tasks are sorted by expiration time (i.e. CompletedAt + Retention) in descending order.
 //
 // By default, it retrieves the first 30 tasks.
-func (i *Inspector) ListCompletedTasks(qname string, opts ...ListOption) ([]*TaskInfo, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ListCompletedTasks(queue string, opts ...ListOption) ([]*TaskInfo, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return nil, fmt.Errorf("asynq: %v", err)
 	}
 	opt := composeListOptions(opts...)
 	pgn := rdb.Pagination{Size: opt.pageSize, Page: opt.pageNum - 1}
-	infos, err := i.rdb.ListCompleted(qname, pgn)
+	infos, err := i.rdb.ListCompleted(queue, pgn)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return nil, fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -450,51 +510,61 @@ func (i *Inspector) ListCompletedTasks(qname string, opts ...ListOption) ([]*Tas
 
 // DeleteAllPendingTasks deletes all pending tasks from the specified queue,
 // and reports the number tasks deleted.
-func (i *Inspector) DeleteAllPendingTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteAllPendingTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.DeleteAllPendingTasks(qname)
+	n, err := i.rdb.DeleteAllPendingTasks(queue)
 	return int(n), err
 }
 
 // DeleteAllScheduledTasks deletes all scheduled tasks from the specified queue,
 // and reports the number tasks deleted.
-func (i *Inspector) DeleteAllScheduledTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteAllScheduledTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.DeleteAllScheduledTasks(qname)
+	n, err := i.rdb.DeleteAllScheduledTasks(queue)
 	return int(n), err
 }
 
 // DeleteAllRetryTasks deletes all retry tasks from the specified queue,
 // and reports the number tasks deleted.
-func (i *Inspector) DeleteAllRetryTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteAllRetryTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.DeleteAllRetryTasks(qname)
+	n, err := i.rdb.DeleteAllRetryTasks(queue)
 	return int(n), err
 }
 
 // DeleteAllArchivedTasks deletes all archived tasks from the specified queue,
 // and reports the number tasks deleted.
-func (i *Inspector) DeleteAllArchivedTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteAllArchivedTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.DeleteAllArchivedTasks(qname)
+	n, err := i.rdb.DeleteAllArchivedTasks(queue)
 	return int(n), err
 }
 
 // DeleteAllCompletedTasks deletes all completed tasks from the specified queue,
 // and reports the number tasks deleted.
-func (i *Inspector) DeleteAllCompletedTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteAllCompletedTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.DeleteAllCompletedTasks(qname)
+	n, err := i.rdb.DeleteAllCompletedTasks(queue)
+	return int(n), err
+}
+
+// DeleteAllAggregatingTasks deletes all tasks from the specified group,
+// and reports the number of tasks deleted.
+func (i *Inspector) DeleteAllAggregatingTasks(queue, group string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
+		return 0, err
+	}
+	n, err := i.rdb.DeleteAllAggregatingTasks(queue, group)
 	return int(n), err
 }
 
@@ -505,11 +575,11 @@ func (i *Inspector) DeleteAllCompletedTasks(qname string) (int, error) {
 // If a queue with the given name doesn't exist, it returns an error wrapping ErrQueueNotFound.
 // If a task with the given id doesn't exist in the queue, it returns an error wrapping ErrTaskNotFound.
 // If the task is in active state, it returns a non-nil error.
-func (i *Inspector) DeleteTask(qname, id string) error {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) DeleteTask(queue, id string) error {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return fmt.Errorf("asynq: %v", err)
 	}
-	err := i.rdb.DeleteTask(qname, id)
+	err := i.rdb.DeleteTask(queue, id)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -522,33 +592,43 @@ func (i *Inspector) DeleteTask(qname, id string) error {
 
 }
 
-// RunAllScheduledTasks transition all scheduled tasks to pending state from the given queue,
-// and reports the number of tasks transitioned.
-func (i *Inspector) RunAllScheduledTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+// RunAllScheduledTasks schedules all scheduled tasks from the given queue to run,
+// and reports the number of tasks scheduled to run.
+func (i *Inspector) RunAllScheduledTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.RunAllScheduledTasks(qname)
+	n, err := i.rdb.RunAllScheduledTasks(queue)
 	return int(n), err
 }
 
-// RunAllRetryTasks transition all retry tasks to pending state from the given queue,
-// and reports the number of tasks transitioned.
-func (i *Inspector) RunAllRetryTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+// RunAllRetryTasks schedules all retry tasks from the given queue to run,
+// and reports the number of tasks scheduled to run.
+func (i *Inspector) RunAllRetryTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.RunAllRetryTasks(qname)
+	n, err := i.rdb.RunAllRetryTasks(queue)
 	return int(n), err
 }
 
-// RunAllArchivedTasks transition all archived tasks to pending state from the given queue,
-// and reports the number of tasks transitioned.
-func (i *Inspector) RunAllArchivedTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+// RunAllArchivedTasks schedules all archived tasks from the given queue to run,
+// and reports the number of tasks scheduled to run.
+func (i *Inspector) RunAllArchivedTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.RunAllArchivedTasks(qname)
+	n, err := i.rdb.RunAllArchivedTasks(queue)
+	return int(n), err
+}
+
+// RunAllAggregatingTasks schedules all tasks from the given grou to run.
+// and reports the number of tasks scheduled to run.
+func (i *Inspector) RunAllAggregatingTasks(queue, group string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
+		return 0, err
+	}
+	n, err := i.rdb.RunAllAggregatingTasks(queue, group)
 	return int(n), err
 }
 
@@ -559,11 +639,11 @@ func (i *Inspector) RunAllArchivedTasks(qname string) (int, error) {
 // If a queue with the given name doesn't exist, it returns an error wrapping ErrQueueNotFound.
 // If a task with the given id doesn't exist in the queue, it returns an error wrapping ErrTaskNotFound.
 // If the task is in pending or active state, it returns a non-nil error.
-func (i *Inspector) RunTask(qname, id string) error {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) RunTask(queue, id string) error {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return fmt.Errorf("asynq: %v", err)
 	}
-	err := i.rdb.RunTask(qname, id)
+	err := i.rdb.RunTask(queue, id)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -577,31 +657,41 @@ func (i *Inspector) RunTask(qname, id string) error {
 
 // ArchiveAllPendingTasks archives all pending tasks from the given queue,
 // and reports the number of tasks archived.
-func (i *Inspector) ArchiveAllPendingTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ArchiveAllPendingTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.ArchiveAllPendingTasks(qname)
+	n, err := i.rdb.ArchiveAllPendingTasks(queue)
 	return int(n), err
 }
 
 // ArchiveAllScheduledTasks archives all scheduled tasks from the given queue,
 // and reports the number of tasks archiveed.
-func (i *Inspector) ArchiveAllScheduledTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ArchiveAllScheduledTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.ArchiveAllScheduledTasks(qname)
+	n, err := i.rdb.ArchiveAllScheduledTasks(queue)
 	return int(n), err
 }
 
 // ArchiveAllRetryTasks archives all retry tasks from the given queue,
 // and reports the number of tasks archiveed.
-func (i *Inspector) ArchiveAllRetryTasks(qname string) (int, error) {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ArchiveAllRetryTasks(queue string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return 0, err
 	}
-	n, err := i.rdb.ArchiveAllRetryTasks(qname)
+	n, err := i.rdb.ArchiveAllRetryTasks(queue)
+	return int(n), err
+}
+
+// ArchiveAllAggregatingTasks archives all tasks from the given group,
+// and reports the number of tasks archived.
+func (i *Inspector) ArchiveAllAggregatingTasks(queue, group string) (int, error) {
+	if err := base.ValidateQueueName(queue); err != nil {
+		return 0, err
+	}
+	n, err := i.rdb.ArchiveAllAggregatingTasks(queue, group)
 	return int(n), err
 }
 
@@ -612,11 +702,11 @@ func (i *Inspector) ArchiveAllRetryTasks(qname string) (int, error) {
 // If a queue with the given name doesn't exist, it returns an error wrapping ErrQueueNotFound.
 // If a task with the given id doesn't exist in the queue, it returns an error wrapping ErrTaskNotFound.
 // If the task is in already archived, it returns a non-nil error.
-func (i *Inspector) ArchiveTask(qname, id string) error {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) ArchiveTask(queue, id string) error {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return fmt.Errorf("asynq: err")
 	}
-	err := i.rdb.ArchiveTask(qname, id)
+	err := i.rdb.ArchiveTask(queue, id)
 	switch {
 	case errors.IsQueueNotFound(err):
 		return fmt.Errorf("asynq: %w", ErrQueueNotFound)
@@ -638,20 +728,20 @@ func (i *Inspector) CancelProcessing(id string) error {
 
 // PauseQueue pauses task processing on the specified queue.
 // If the queue is already paused, it will return a non-nil error.
-func (i *Inspector) PauseQueue(qname string) error {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) PauseQueue(queue string) error {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return err
 	}
-	return i.rdb.Pause(qname)
+	return i.rdb.Pause(queue)
 }
 
 // UnpauseQueue resumes task processing on the specified queue.
 // If the queue is not paused, it will return a non-nil error.
-func (i *Inspector) UnpauseQueue(qname string) error {
-	if err := base.ValidateQueueName(qname); err != nil {
+func (i *Inspector) UnpauseQueue(queue string) error {
+	if err := base.ValidateQueueName(queue); err != nil {
 		return err
 	}
-	return i.rdb.Unpause(qname)
+	return i.rdb.Unpause(queue)
 }
 
 // Servers return a list of running servers' information.
@@ -741,8 +831,8 @@ type WorkerInfo struct {
 }
 
 // ClusterKeySlot returns an integer identifying the hash slot the given queue hashes to.
-func (i *Inspector) ClusterKeySlot(qname string) (int64, error) {
-	return i.rdb.ClusterKeySlot(qname)
+func (i *Inspector) ClusterKeySlot(queue string) (int64, error) {
+	return i.rdb.ClusterKeySlot(queue)
 }
 
 // ClusterNode describes a node in redis cluster.
@@ -757,8 +847,8 @@ type ClusterNode struct {
 // ClusterNodes returns a list of nodes the given queue belongs to.
 //
 // Only relevant if task queues are stored in redis cluster.
-func (i *Inspector) ClusterNodes(qname string) ([]*ClusterNode, error) {
-	nodes, err := i.rdb.ClusterNodes(qname)
+func (i *Inspector) ClusterNodes(queue string) ([]*ClusterNode, error) {
+	nodes, err := i.rdb.ClusterNodes(queue)
 	if err != nil {
 		return nil, err
 	}
@@ -826,11 +916,11 @@ func parseOption(s string) (Option, error) {
 	fn, arg := parseOptionFunc(s), parseOptionArg(s)
 	switch fn {
 	case "Queue":
-		qname, err := strconv.Unquote(arg)
+		queue, err := strconv.Unquote(arg)
 		if err != nil {
 			return nil, err
 		}
-		return Queue(qname), nil
+		return Queue(queue), nil
 	case "MaxRetry":
 		n, err := strconv.Atoi(arg)
 		if err != nil {

inspector_test.go

@@ -15,10 +15,11 @@ import (
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/google/uuid"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 	"github.com/hibiken/asynq/internal/timeutil"
+	"github.com/redis/go-redis/v9"
 )
 
 func TestInspectorQueues(t *testing.T) {
@@ -1120,6 +1121,106 @@ func TestInspectorListCompletedTasks(t *testing.T) {
 	}
 }
 
+func TestInspectorListAggregatingTasks(t *testing.T) {
+	r := setup(t)
+	defer r.Close()
+	now := time.Now()
+
+	m1 := h.NewTaskMessageBuilder().SetType("task1").SetQueue("default").SetGroup("group1").Build()
+	m2 := h.NewTaskMessageBuilder().SetType("task2").SetQueue("default").SetGroup("group1").Build()
+	m3 := h.NewTaskMessageBuilder().SetType("task3").SetQueue("default").SetGroup("group1").Build()
+	m4 := h.NewTaskMessageBuilder().SetType("task4").SetQueue("default").SetGroup("group2").Build()
+	m5 := h.NewTaskMessageBuilder().SetType("task5").SetQueue("custom").SetGroup("group1").Build()
+
+	inspector := NewInspector(getRedisConnOpt(t))
+
+	fxt := struct {
+		tasks     []*h.TaskSeedData
+		allQueues []string
+		allGroups map[string][]string
+		groups    map[string][]redis.Z
+	}{
+		tasks: []*h.TaskSeedData{
+			{Msg: m1, State: base.TaskStateAggregating},
+			{Msg: m2, State: base.TaskStateAggregating},
+			{Msg: m3, State: base.TaskStateAggregating},
+			{Msg: m4, State: base.TaskStateAggregating},
+			{Msg: m5, State: base.TaskStateAggregating},
+		},
+		allQueues: []string{"default", "custom"},
+		allGroups: map[string][]string{
+			base.AllGroups("default"): {"group1", "group2"},
+			base.AllGroups("custom"):  {"group1"},
+		},
+		groups: map[string][]redis.Z{
+			base.GroupKey("default", "group1"): {
+				{Member: m1.ID, Score: float64(now.Add(-30 * time.Second).Unix())},
+				{Member: m2.ID, Score: float64(now.Add(-20 * time.Second).Unix())},
+				{Member: m3.ID, Score: float64(now.Add(-10 * time.Second).Unix())},
+			},
+			base.GroupKey("default", "group2"): {
+				{Member: m4.ID, Score: float64(now.Add(-30 * time.Second).Unix())},
+			},
+			base.GroupKey("custom", "group1"): {
+				{Member: m5.ID, Score: float64(now.Add(-30 * time.Second).Unix())},
+			},
+		},
+	}
+
+	tests := []struct {
+		desc  string
+		qname string
+		gname string
+		want  []*TaskInfo
+	}{
+		{
+			desc:  "default queue group1",
+			qname: "default",
+			gname: "group1",
+			want: []*TaskInfo{
+				createAggregatingTaskInfo(m1),
+				createAggregatingTaskInfo(m2),
+				createAggregatingTaskInfo(m3),
+			},
+		},
+		{
+			desc:  "custom queue group1",
+			qname: "custom",
+			gname: "group1",
+			want: []*TaskInfo{
+				createAggregatingTaskInfo(m5),
+			},
+		},
+	}
+
+	for _, tc := range tests {
+		h.FlushDB(t, r)
+		h.SeedTasks(t, r, fxt.tasks)
+		h.SeedRedisSet(t, r, base.AllQueues, fxt.allQueues)
+		h.SeedRedisSets(t, r, fxt.allGroups)
+		h.SeedRedisZSets(t, r, fxt.groups)
+
+		t.Run(tc.desc, func(t *testing.T) {
+			got, err := inspector.ListAggregatingTasks(tc.qname, tc.gname)
+			if err != nil {
+				t.Fatalf("ListAggregatingTasks returned error: %v", err)
+			}
+
+			cmpOpts := []cmp.Option{
+				cmpopts.EquateApproxTime(2 * time.Second),
+				cmp.AllowUnexported(TaskInfo{}),
+			}
+			if diff := cmp.Diff(tc.want, got, cmpOpts...); diff != "" {
+				t.Errorf("ListAggregatingTasks = %v, want = %v; (-want,+got)\n%s", got, tc.want, diff)
+			}
+		})
+	}
+}
+
+func createAggregatingTaskInfo(msg *base.TaskMessage) *TaskInfo {
+	return newTaskInfo(msg, base.TaskStateAggregating, time.Time{}, nil)
+}
+
 func TestInspectorListPagination(t *testing.T) {
 	// Create 100 tasks.
 	var msgs []*base.TaskMessage
@@ -1221,6 +1322,9 @@ func TestInspectorListTasksQueueNotFoundError(t *testing.T) {
 		if _, err := inspector.ListCompletedTasks(tc.qname); !errors.Is(err, tc.wantErr) {
 			t.Errorf("ListCompletedTasks(%q) returned error %v, want %v", tc.qname, err, tc.wantErr)
 		}
+		if _, err := inspector.ListAggregatingTasks(tc.qname, "mygroup"); !errors.Is(err, tc.wantErr) {
+			t.Errorf("ListAggregatingTasks(%q, \"mygroup\") returned error %v, want %v", tc.qname, err, tc.wantErr)
+		}
 	}
 }
 
@@ -3323,3 +3427,99 @@ func TestParseOption(t *testing.T) {
 		})
 	}
 }
+
+func TestInspectorGroups(t *testing.T) {
+	r := setup(t)
+	defer r.Close()
+	inspector := NewInspector(getRedisConnOpt(t))
+
+	m1 := h.NewTaskMessageBuilder().SetGroup("group1").Build()
+	m2 := h.NewTaskMessageBuilder().SetGroup("group1").Build()
+	m3 := h.NewTaskMessageBuilder().SetGroup("group1").Build()
+	m4 := h.NewTaskMessageBuilder().SetGroup("group2").Build()
+	m5 := h.NewTaskMessageBuilder().SetQueue("custom").SetGroup("group1").Build()
+	m6 := h.NewTaskMessageBuilder().SetQueue("custom").SetGroup("group1").Build()
+
+	now := time.Now()
+
+	fixtures := struct {
+		tasks     []*h.TaskSeedData
+		allGroups map[string][]string
+		groups    map[string][]redis.Z
+	}{
+		tasks: []*h.TaskSeedData{
+			{Msg: m1, State: base.TaskStateAggregating},
+			{Msg: m2, State: base.TaskStateAggregating},
+			{Msg: m3, State: base.TaskStateAggregating},
+			{Msg: m4, State: base.TaskStateAggregating},
+			{Msg: m5, State: base.TaskStateAggregating},
+		},
+		allGroups: map[string][]string{
+			base.AllGroups("default"): {"group1", "group2"},
+			base.AllGroups("custom"):  {"group1"},
+		},
+		groups: map[string][]redis.Z{
+			base.GroupKey("default", "group1"): {
+				{Member: m1.ID, Score: float64(now.Add(-10 * time.Second).Unix())},
+				{Member: m2.ID, Score: float64(now.Add(-20 * time.Second).Unix())},
+				{Member: m3.ID, Score: float64(now.Add(-30 * time.Second).Unix())},
+			},
+			base.GroupKey("default", "group2"): {
+				{Member: m4.ID, Score: float64(now.Add(-20 * time.Second).Unix())},
+			},
+			base.GroupKey("custom", "group1"): {
+				{Member: m5.ID, Score: float64(now.Add(-10 * time.Second).Unix())},
+				{Member: m6.ID, Score: float64(now.Add(-20 * time.Second).Unix())},
+			},
+		},
+	}
+
+	tests := []struct {
+		desc  string
+		qname string
+		want  []*GroupInfo
+	}{
+		{
+			desc:  "default queue groups",
+			qname: "default",
+			want: []*GroupInfo{
+				{Group: "group1", Size: 3},
+				{Group: "group2", Size: 1},
+			},
+		},
+		{
+			desc:  "custom queue groups",
+			qname: "custom",
+			want: []*GroupInfo{
+				{Group: "group1", Size: 2},
+			},
+		},
+	}
+
+	var sortGroupInfosOpt = cmp.Transformer(
+		"SortGroupInfos",
+		func(in []*GroupInfo) []*GroupInfo {
+			out := append([]*GroupInfo(nil), in...)
+			sort.Slice(out, func(i, j int) bool {
+				return out[i].Group < out[j].Group
+			})
+			return out
+		})
+
+	for _, tc := range tests {
+		h.FlushDB(t, r)
+		h.SeedTasks(t, r, fixtures.tasks)
+		h.SeedRedisSets(t, r, fixtures.allGroups)
+		h.SeedRedisZSets(t, r, fixtures.groups)
+
+		t.Run(tc.desc, func(t *testing.T) {
+			got, err := inspector.Groups(tc.qname)
+			if err != nil {
+				t.Fatalf("Groups returned error: %v", err)
+			}
+			if diff := cmp.Diff(tc.want, got, sortGroupInfosOpt); diff != "" {
+				t.Errorf("Groups = %v, want %v; (-want,+got)\n%s", got, tc.want, diff)
+			}
+		})
+	}
+}

internal/base/base.go

@@ -14,16 +14,16 @@ import (
 	"sync"
 	"time"
 
-	"github.com/go-redis/redis/v8"
 	"github.com/golang/protobuf/ptypes"
 	"github.com/hibiken/asynq/internal/errors"
 	pb "github.com/hibiken/asynq/internal/proto"
 	"github.com/hibiken/asynq/internal/timeutil"
+	"github.com/redis/go-redis/v9"
 	"google.golang.org/protobuf/proto"
 )
 
 // Version of asynq library and CLI.
-const Version = "0.22.0"
+const Version = "0.24.1"
 
 // DefaultQueueName is the queue name used if none are specified by user.
 const DefaultQueueName = "default"
@@ -50,6 +50,7 @@ const (
 	TaskStateRetry
 	TaskStateArchived
 	TaskStateCompleted
+	TaskStateAggregating // describes a state where task is waiting in a group to be aggregated
 )
 
 func (s TaskState) String() string {
@@ -66,6 +67,8 @@ func (s TaskState) String() string {
 		return "archived"
 	case TaskStateCompleted:
 		return "completed"
+	case TaskStateAggregating:
+		return "aggregating"
 	}
 	panic(fmt.Sprintf("internal error: unknown task state %d", s))
 }
@@ -84,6 +87,8 @@ func TaskStateFromString(s string) (TaskState, error) {
 		return TaskStateArchived, nil
 	case "completed":
 		return TaskStateCompleted, nil
+	case "aggregating":
+		return TaskStateAggregating, nil
 	}
 	return 0, errors.E(errors.FailedPrecondition, fmt.Sprintf("%q is not supported task state", s))
 }
@@ -200,6 +205,32 @@ func UniqueKey(qname, tasktype string, payload []byte) string {
 	return fmt.Sprintf("%sunique:%s:%s", QueueKeyPrefix(qname), tasktype, hex.EncodeToString(checksum[:]))
 }
 
+// GroupKeyPrefix returns a prefix for group key.
+func GroupKeyPrefix(qname string) string {
+	return fmt.Sprintf("%sg:", QueueKeyPrefix(qname))
+}
+
+// GroupKey returns a redis key used to group tasks belong in the same group.
+func GroupKey(qname, gkey string) string {
+	return fmt.Sprintf("%s%s", GroupKeyPrefix(qname), gkey)
+}
+
+// AggregationSetKey returns a redis key used for an aggregation set.
+func AggregationSetKey(qname, gname, setID string) string {
+	return fmt.Sprintf("%s:%s", GroupKey(qname, gname), setID)
+}
+
+// AllGroups return a redis key used to store all group keys used in a given queue.
+func AllGroups(qname string) string {
+	return fmt.Sprintf("%sgroups", QueueKeyPrefix(qname))
+}
+
+// AllAggregationSets returns a redis key used to store all aggregation sets (set of tasks staged to be aggregated)
+// in a given queue.
+func AllAggregationSets(qname string) string {
+	return fmt.Sprintf("%saggregation_sets", QueueKeyPrefix(qname))
+}
+
 // TaskMessage is the internal representation of a task with additional metadata fields.
 // Serialized data of this type gets written to redis.
 type TaskMessage struct {
@@ -250,6 +281,11 @@ type TaskMessage struct {
 	// Empty string indicates that no uniqueness lock was used.
 	UniqueKey string
 
+	// GroupKey holds the group key used for task aggregation.
+	//
+	// Empty string indicates no aggregation is used for this task.
+	GroupKey string
+
 	// Retention specifies the number of seconds the task should be retained after completion.
 	Retention int64
 
@@ -277,6 +313,7 @@ func EncodeMessage(msg *TaskMessage) ([]byte, error) {
 		Timeout:      msg.Timeout,
 		Deadline:     msg.Deadline,
 		UniqueKey:    msg.UniqueKey,
+		GroupKey:     msg.GroupKey,
 		Retention:    msg.Retention,
 		CompletedAt:  msg.CompletedAt,
 	})
@@ -300,6 +337,7 @@ func DecodeMessage(data []byte) (*TaskMessage, error) {
 		Timeout:      pbmsg.GetTimeout(),
 		Deadline:     pbmsg.GetDeadline(),
 		UniqueKey:    pbmsg.GetUniqueKey(),
+		GroupKey:     pbmsg.GetGroupKey(),
 		Retention:    pbmsg.GetRetention(),
 		CompletedAt:  pbmsg.GetCompletedAt(),
 	}, nil
@@ -569,7 +607,7 @@ func DecodeSchedulerEnqueueEvent(b []byte) (*SchedulerEnqueueEvent, error) {
 
 // Cancelations is a collection that holds cancel functions for all active tasks.
 //
-// Cancelations are safe for concurrent use by multipel goroutines.
+// Cancelations are safe for concurrent use by multiple goroutines.
 type Cancelations struct {
 	mu          sync.Mutex
 	cancelFuncs map[string]context.CancelFunc
@@ -624,7 +662,7 @@ func NewLease(expirationTime time.Time) *Lease {
 	}
 }
 
-// Reset chanegs the lease to expire at the given time.
+// Reset changes the lease to expire at the given time.
 // It returns true if the lease is still valid and reset operation was successful, false if the lease had been expired.
 func (l *Lease) Reset(expirationTime time.Time) bool {
 	if !l.IsValid() {
@@ -662,7 +700,7 @@ func (l *Lease) Deadline() time.Time {
 	return l.expireAt
 }
 
-// IsValid returns true if the lease's expieration time is in the future or equals to the current time,
+// IsValid returns true if the lease's expiration time is in the future or equals to the current time,
 // returns false otherwise.
 func (l *Lease) IsValid() bool {
 	now := l.Clock.Now()
@@ -676,6 +714,7 @@ func (l *Lease) IsValid() bool {
 // See rdb.RDB as a reference implementation.
 type Broker interface {
 	Ping() error
+	Close() error
 	Enqueue(ctx context.Context, msg *TaskMessage) error
 	EnqueueUnique(ctx context.Context, msg *TaskMessage, ttl time.Duration) error
 	Dequeue(qnames ...string) (*TaskMessage, time.Time, error)
@@ -687,13 +726,30 @@ type Broker interface {
 	Retry(ctx context.Context, msg *TaskMessage, processAt time.Time, errMsg string, isFailure bool) error
 	Archive(ctx context.Context, msg *TaskMessage, errMsg string) error
 	ForwardIfReady(qnames ...string) error
+
+	// Group aggregation related methods
+	AddToGroup(ctx context.Context, msg *TaskMessage, gname string) error
+	AddToGroupUnique(ctx context.Context, msg *TaskMessage, groupKey string, ttl time.Duration) error
+	ListGroups(qname string) ([]string, error)
+	AggregationCheck(qname, gname string, t time.Time, gracePeriod, maxDelay time.Duration, maxSize int) (aggregationSetID string, err error)
+	ReadAggregationSet(qname, gname, aggregationSetID string) ([]*TaskMessage, time.Time, error)
+	DeleteAggregationSet(ctx context.Context, qname, gname, aggregationSetID string) error
+	ReclaimStaleAggregationSets(qname string) error
+
+	// Task retention related method
 	DeleteExpiredCompletedTasks(qname string) error
+
+	// Lease related methods
 	ListLeaseExpired(cutoff time.Time, qnames ...string) ([]*TaskMessage, error)
 	ExtendLease(qname string, ids ...string) (time.Time, error)
+
+	// State snapshot related methods
 	WriteServerState(info *ServerInfo, workers []*WorkerInfo, ttl time.Duration) error
 	ClearServerState(host string, pid int, serverID string) error
+
+	// Cancelation related methods
 	CancelationPubSub() (*redis.PubSub, error) // TODO: Need to decouple from redis to support other brokers
 	PublishCancelation(id string) error
+
 	WriteResult(qname, id string, data []byte) (n int, err error)
-	Close() error
 }

internal/base/base_test.go

@@ -395,6 +395,107 @@ func TestUniqueKey(t *testing.T) {
 	}
 }
 
+func TestGroupKey(t *testing.T) {
+	tests := []struct {
+		qname string
+		gkey  string
+		want  string
+	}{
+		{
+			qname: "default",
+			gkey:  "mygroup",
+			want:  "asynq:{default}:g:mygroup",
+		},
+		{
+			qname: "custom",
+			gkey:  "foo",
+			want:  "asynq:{custom}:g:foo",
+		},
+	}
+
+	for _, tc := range tests {
+		got := GroupKey(tc.qname, tc.gkey)
+		if got != tc.want {
+			t.Errorf("GroupKey(%q, %q) = %q, want %q", tc.qname, tc.gkey, got, tc.want)
+		}
+	}
+}
+
+func TestAggregationSetKey(t *testing.T) {
+	tests := []struct {
+		qname string
+		gname string
+		setID string
+		want  string
+	}{
+		{
+			qname: "default",
+			gname: "mygroup",
+			setID: "12345",
+			want:  "asynq:{default}:g:mygroup:12345",
+		},
+		{
+			qname: "custom",
+			gname: "foo",
+			setID: "98765",
+			want:  "asynq:{custom}:g:foo:98765",
+		},
+	}
+
+	for _, tc := range tests {
+		got := AggregationSetKey(tc.qname, tc.gname, tc.setID)
+		if got != tc.want {
+			t.Errorf("AggregationSetKey(%q, %q, %q) = %q, want %q", tc.qname, tc.gname, tc.setID, got, tc.want)
+		}
+	}
+}
+
+func TestAllGroups(t *testing.T) {
+	tests := []struct {
+		qname string
+		want  string
+	}{
+		{
+			qname: "default",
+			want:  "asynq:{default}:groups",
+		},
+		{
+			qname: "custom",
+			want:  "asynq:{custom}:groups",
+		},
+	}
+
+	for _, tc := range tests {
+		got := AllGroups(tc.qname)
+		if got != tc.want {
+			t.Errorf("AllGroups(%q) = %q, want %q", tc.qname, got, tc.want)
+		}
+	}
+}
+
+func TestAllAggregationSets(t *testing.T) {
+	tests := []struct {
+		qname string
+		want  string
+	}{
+		{
+			qname: "default",
+			want:  "asynq:{default}:aggregation_sets",
+		},
+		{
+			qname: "custom",
+			want:  "asynq:{custom}:aggregation_sets",
+		},
+	}
+
+	for _, tc := range tests {
+		got := AllAggregationSets(tc.qname)
+		if got != tc.want {
+			t.Errorf("AllAggregationSets(%q) = %q, want %q", tc.qname, got, tc.want)
+		}
+	}
+}
+
 func TestMessageEncoding(t *testing.T) {
 	id := uuid.NewString()
 	tests := []struct {
@@ -407,6 +508,7 @@ func TestMessageEncoding(t *testing.T) {
 				Payload:   toBytes(map[string]interface{}{"a": 1, "b": "hello!", "c": true}),
 				ID:        id,
 				Queue:     "default",
+				GroupKey:  "mygroup",
 				Retry:     10,
 				Retried:   0,
 				Timeout:   1800,
@@ -418,6 +520,7 @@ func TestMessageEncoding(t *testing.T) {
 				Payload:   toBytes(map[string]interface{}{"a": json.Number("1"), "b": "hello!", "c": true}),
 				ID:        id,
 				Queue:     "default",
+				GroupKey:  "mygroup",
 				Retry:     10,
 				Retried:   0,
 				Timeout:   1800,

internal/context/context.go

@@ -65,7 +65,7 @@ func GetRetryCount(ctx context.Context) (n int, ok bool) {
 
 // GetMaxRetry extracts maximum retry from a context, if any.
 //
-// Return value n indicates the maximum number of times the assoicated task
+// Return value n indicates the maximum number of times the associated task
 // can be retried if ProcessTask returns a non-nil error.
 func GetMaxRetry(ctx context.Context) (n int, ok bool) {
 	metadata, ok := ctx.Value(metadataCtxKey).(taskMetadata)

internal/errors/errors.go

@@ -161,7 +161,7 @@ func CanonicalCode(err error) Code {
 }
 
 /******************************************
-    Domin Specific Error Types & Values
+    Domain Specific Error Types & Values
 *******************************************/
 
 var (
@@ -263,26 +263,26 @@ func IsRedisCommandError(err error) bool {
 // New returns an error that formats as the given text.
 // Each call to New returns a distinct error value even if the text is identical.
 //
-// This function is the errors.New function from the standard libarary (https://golang.org/pkg/errors/#New).
-// It is exported from this package for import convinience.
+// This function is the errors.New function from the standard library (https://golang.org/pkg/errors/#New).
+// It is exported from this package for import convenience.
 func New(text string) error { return errors.New(text) }
 
 // Is reports whether any error in err's chain matches target.
 //
-// This function is the errors.Is function from the standard libarary (https://golang.org/pkg/errors/#Is).
-// It is exported from this package for import convinience.
+// This function is the errors.Is function from the standard library (https://golang.org/pkg/errors/#Is).
+// It is exported from this package for import convenience.
 func Is(err, target error) bool { return errors.Is(err, target) }
 
 // As finds the first error in err's chain that matches target, and if so, sets target to that error value and returns true.
 // Otherwise, it returns false.
 //
-// This function is the errors.As function from the standard libarary (https://golang.org/pkg/errors/#As).
-// It is exported from this package for import convinience.
+// This function is the errors.As function from the standard library (https://golang.org/pkg/errors/#As).
+// It is exported from this package for import convenience.
 func As(err error, target interface{}) bool { return errors.As(err, target) }
 
 // Unwrap returns the result of calling the Unwrap method on err, if err's type contains an Unwrap method returning error.
 // Otherwise, Unwrap returns nil.
 //
-// This function is the errors.Unwrap function from the standard libarary (https://golang.org/pkg/errors/#Unwrap).
-// It is exported from this package for import convinience.
+// This function is the errors.Unwrap function from the standard library (https://golang.org/pkg/errors/#Unwrap).
+// It is exported from this package for import convenience.
 func Unwrap(err error) error { return errors.Unwrap(err) }

internal/proto/asynq.pb.go

@@ -32,6 +32,7 @@ const _ = proto.ProtoPackageIsVersion4
 
 // TaskMessage is the internal representation of a task with additional
 // metadata fields.
+// Next ID: 15
 type TaskMessage struct {
 	state         protoimpl.MessageState
 	sizeCache     protoimpl.SizeCache
@@ -65,6 +66,9 @@ type TaskMessage struct {
 	// UniqueKey holds the redis key used for uniqueness lock for this task.
 	// Empty string indicates that no uniqueness lock was used.
 	UniqueKey string `protobuf:"bytes,10,opt,name=unique_key,json=uniqueKey,proto3" json:"unique_key,omitempty"`
+	// GroupKey is a name of the group used for task aggregation.
+	// This field is optional and empty value means no aggregation for the task.
+	GroupKey string `protobuf:"bytes,14,opt,name=group_key,json=groupKey,proto3" json:"group_key,omitempty"`
 	// Retention period specified in a number of seconds.
 	// The task will be stored in redis as a completed task until the TTL
 	// expires.
@@ -184,6 +188,13 @@ func (x *TaskMessage) GetUniqueKey() string {
 	return ""
 }
 
+func (x *TaskMessage) GetGroupKey() string {
+	if x != nil {
+		return x.GroupKey
+	}
+	return ""
+}
+
 func (x *TaskMessage) GetRetention() int64 {
 	if x != nil {
 		return x.Retention
@@ -614,7 +625,7 @@ var file_asynq_proto_rawDesc = []byte{
 	0x0a, 0x0b, 0x61, 0x73, 0x79, 0x6e, 0x71, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x12, 0x05, 0x61,
 	0x73, 0x79, 0x6e, 0x71, 0x1a, 0x1f, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2f, 0x70, 0x72, 0x6f,
 	0x74, 0x6f, 0x62, 0x75, 0x66, 0x2f, 0x74, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x2e,
-	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0xea, 0x02, 0x0a, 0x0b, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x22, 0x87, 0x03, 0x0a, 0x0b, 0x54, 0x61, 0x73, 0x6b, 0x4d, 0x65,
 	0x73, 0x73, 0x61, 0x67, 0x65, 0x12, 0x12, 0x0a, 0x04, 0x74, 0x79, 0x70, 0x65, 0x18, 0x01, 0x20,
 	0x01, 0x28, 0x09, 0x52, 0x04, 0x74, 0x79, 0x70, 0x65, 0x12, 0x18, 0x0a, 0x07, 0x70, 0x61, 0x79,
 	0x6c, 0x6f, 0x61, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0c, 0x52, 0x07, 0x70, 0x61, 0x79, 0x6c,
@@ -633,84 +644,86 @@ var file_asynq_proto_rawDesc = []byte{
 	0x6e, 0x65, 0x18, 0x09, 0x20, 0x01, 0x28, 0x03, 0x52, 0x08, 0x64, 0x65, 0x61, 0x64, 0x6c, 0x69,
 	0x6e, 0x65, 0x12, 0x1d, 0x0a, 0x0a, 0x75, 0x6e, 0x69, 0x71, 0x75, 0x65, 0x5f, 0x6b, 0x65, 0x79,
 	0x18, 0x0a, 0x20, 0x01, 0x28, 0x09, 0x52, 0x09, 0x75, 0x6e, 0x69, 0x71, 0x75, 0x65, 0x4b, 0x65,
-	0x79, 0x12, 0x1c, 0x0a, 0x09, 0x72, 0x65, 0x74, 0x65, 0x6e, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x0c,
-	0x20, 0x01, 0x28, 0x03, 0x52, 0x09, 0x72, 0x65, 0x74, 0x65, 0x6e, 0x74, 0x69, 0x6f, 0x6e, 0x12,
-	0x21, 0x0a, 0x0c, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x18,
-	0x0d, 0x20, 0x01, 0x28, 0x03, 0x52, 0x0b, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0x64,
-	0x41, 0x74, 0x22, 0x8f, 0x03, 0x0a, 0x0a, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x49, 0x6e, 0x66,
-	0x6f, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
-	0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x70, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01,
-	0x28, 0x05, 0x52, 0x03, 0x70, 0x69, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x65, 0x72, 0x76, 0x65,
-	0x72, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x65, 0x72, 0x76,
-	0x65, 0x72, 0x49, 0x64, 0x12, 0x20, 0x0a, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, 0x72, 0x72, 0x65,
-	0x6e, 0x63, 0x79, 0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75,
-	0x72, 0x72, 0x65, 0x6e, 0x63, 0x79, 0x12, 0x35, 0x0a, 0x06, 0x71, 0x75, 0x65, 0x75, 0x65, 0x73,
-	0x18, 0x05, 0x20, 0x03, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x61, 0x73, 0x79, 0x6e, 0x71, 0x2e, 0x53,
-	0x65, 0x72, 0x76, 0x65, 0x72, 0x49, 0x6e, 0x66, 0x6f, 0x2e, 0x51, 0x75, 0x65, 0x75, 0x65, 0x73,
-	0x45, 0x6e, 0x74, 0x72, 0x79, 0x52, 0x06, 0x71, 0x75, 0x65, 0x75, 0x65, 0x73, 0x12, 0x27, 0x0a,
-	0x0f, 0x73, 0x74, 0x72, 0x69, 0x63, 0x74, 0x5f, 0x70, 0x72, 0x69, 0x6f, 0x72, 0x69, 0x74, 0x79,
-	0x18, 0x06, 0x20, 0x01, 0x28, 0x08, 0x52, 0x0e, 0x73, 0x74, 0x72, 0x69, 0x63, 0x74, 0x50, 0x72,
-	0x69, 0x6f, 0x72, 0x69, 0x74, 0x79, 0x12, 0x16, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73,
-	0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x39,
-	0x0a, 0x0a, 0x73, 0x74, 0x61, 0x72, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x08, 0x20, 0x01,
-	0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74,
-	0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09,
-	0x73, 0x74, 0x61, 0x72, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x2e, 0x0a, 0x13, 0x61, 0x63, 0x74,
-	0x69, 0x76, 0x65, 0x5f, 0x77, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74,
-	0x18, 0x09, 0x20, 0x01, 0x28, 0x05, 0x52, 0x11, 0x61, 0x63, 0x74, 0x69, 0x76, 0x65, 0x57, 0x6f,
-	0x72, 0x6b, 0x65, 0x72, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x1a, 0x39, 0x0a, 0x0b, 0x51, 0x75, 0x65,
-	0x75, 0x65, 0x73, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18,
-	0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61,
-	0x6c, 0x75, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65,
-	0x3a, 0x02, 0x38, 0x01, 0x22, 0xb1, 0x02, 0x0a, 0x0a, 0x57, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x49,
-	0x6e, 0x66, 0x6f, 0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28,
-	0x09, 0x52, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x70, 0x69, 0x64, 0x18, 0x02,
-	0x20, 0x01, 0x28, 0x05, 0x52, 0x03, 0x70, 0x69, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x65, 0x72,
-	0x76, 0x65, 0x72, 0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x65,
-	0x72, 0x76, 0x65, 0x72, 0x49, 0x64, 0x12, 0x17, 0x0a, 0x07, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x69,
-	0x64, 0x18, 0x04, 0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x74, 0x61, 0x73, 0x6b, 0x49, 0x64, 0x12,
-	0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01,
-	0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x21, 0x0a, 0x0c,
-	0x74, 0x61, 0x73, 0x6b, 0x5f, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x18, 0x06, 0x20, 0x01,
-	0x28, 0x0c, 0x52, 0x0b, 0x74, 0x61, 0x73, 0x6b, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x12,
-	0x14, 0x0a, 0x05, 0x71, 0x75, 0x65, 0x75, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05,
-	0x71, 0x75, 0x65, 0x75, 0x65, 0x12, 0x39, 0x0a, 0x0a, 0x73, 0x74, 0x61, 0x72, 0x74, 0x5f, 0x74,
-	0x69, 0x6d, 0x65, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67,
-	0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65,
-	0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09, 0x73, 0x74, 0x61, 0x72, 0x74, 0x54, 0x69, 0x6d, 0x65,
-	0x12, 0x36, 0x0a, 0x08, 0x64, 0x65, 0x61, 0x64, 0x6c, 0x69, 0x6e, 0x65, 0x18, 0x09, 0x20, 0x01,
-	0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74,
-	0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x08,
-	0x64, 0x65, 0x61, 0x64, 0x6c, 0x69, 0x6e, 0x65, 0x22, 0xad, 0x02, 0x0a, 0x0e, 0x53, 0x63, 0x68,
-	0x65, 0x64, 0x75, 0x6c, 0x65, 0x72, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x0e, 0x0a, 0x02, 0x69,
-	0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x12, 0x0a, 0x04, 0x73,
-	0x70, 0x65, 0x63, 0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x73, 0x70, 0x65, 0x63, 0x12,
-	0x1b, 0x0a, 0x09, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x03, 0x20, 0x01,
-	0x28, 0x09, 0x52, 0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x21, 0x0a, 0x0c,
-	0x74, 0x61, 0x73, 0x6b, 0x5f, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x18, 0x04, 0x20, 0x01,
-	0x28, 0x0c, 0x52, 0x0b, 0x74, 0x61, 0x73, 0x6b, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x12,
-	0x27, 0x0a, 0x0f, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x6f, 0x70, 0x74, 0x69, 0x6f,
-	0x6e, 0x73, 0x18, 0x05, 0x20, 0x03, 0x28, 0x09, 0x52, 0x0e, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75,
-	0x65, 0x4f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x46, 0x0a, 0x11, 0x6e, 0x65, 0x78, 0x74,
-	0x5f, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x06, 0x20,
-	0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f,
-	0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52,
-	0x0f, 0x6e, 0x65, 0x78, 0x74, 0x45, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x54, 0x69, 0x6d, 0x65,
-	0x12, 0x46, 0x0a, 0x11, 0x70, 0x72, 0x65, 0x76, 0x5f, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65,
-	0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f,
-	0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69,
-	0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0f, 0x70, 0x72, 0x65, 0x76, 0x45, 0x6e, 0x71,
-	0x75, 0x65, 0x75, 0x65, 0x54, 0x69, 0x6d, 0x65, 0x22, 0x6f, 0x0a, 0x15, 0x53, 0x63, 0x68, 0x65,
-	0x64, 0x75, 0x6c, 0x65, 0x72, 0x45, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x45, 0x76, 0x65, 0x6e,
-	0x74, 0x12, 0x17, 0x0a, 0x07, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01,
-	0x28, 0x09, 0x52, 0x06, 0x74, 0x61, 0x73, 0x6b, 0x49, 0x64, 0x12, 0x3d, 0x0a, 0x0c, 0x65, 0x6e,
-	0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b,
+	0x79, 0x12, 0x1b, 0x0a, 0x09, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x5f, 0x6b, 0x65, 0x79, 0x18, 0x0e,
+	0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x67, 0x72, 0x6f, 0x75, 0x70, 0x4b, 0x65, 0x79, 0x12, 0x1c,
+	0x0a, 0x09, 0x72, 0x65, 0x74, 0x65, 0x6e, 0x74, 0x69, 0x6f, 0x6e, 0x18, 0x0c, 0x20, 0x01, 0x28,
+	0x03, 0x52, 0x09, 0x72, 0x65, 0x74, 0x65, 0x6e, 0x74, 0x69, 0x6f, 0x6e, 0x12, 0x21, 0x0a, 0x0c,
+	0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0x64, 0x5f, 0x61, 0x74, 0x18, 0x0d, 0x20, 0x01,
+	0x28, 0x03, 0x52, 0x0b, 0x63, 0x6f, 0x6d, 0x70, 0x6c, 0x65, 0x74, 0x65, 0x64, 0x41, 0x74, 0x22,
+	0x8f, 0x03, 0x0a, 0x0a, 0x53, 0x65, 0x72, 0x76, 0x65, 0x72, 0x49, 0x6e, 0x66, 0x6f, 0x12, 0x12,
+	0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x68, 0x6f,
+	0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x70, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52,
+	0x03, 0x70, 0x69, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x5f, 0x69,
+	0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72, 0x49,
+	0x64, 0x12, 0x20, 0x0a, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, 0x72, 0x72, 0x65, 0x6e, 0x63, 0x79,
+	0x18, 0x04, 0x20, 0x01, 0x28, 0x05, 0x52, 0x0b, 0x63, 0x6f, 0x6e, 0x63, 0x75, 0x72, 0x72, 0x65,
+	0x6e, 0x63, 0x79, 0x12, 0x35, 0x0a, 0x06, 0x71, 0x75, 0x65, 0x75, 0x65, 0x73, 0x18, 0x05, 0x20,
+	0x03, 0x28, 0x0b, 0x32, 0x1d, 0x2e, 0x61, 0x73, 0x79, 0x6e, 0x71, 0x2e, 0x53, 0x65, 0x72, 0x76,
+	0x65, 0x72, 0x49, 0x6e, 0x66, 0x6f, 0x2e, 0x51, 0x75, 0x65, 0x75, 0x65, 0x73, 0x45, 0x6e, 0x74,
+	0x72, 0x79, 0x52, 0x06, 0x71, 0x75, 0x65, 0x75, 0x65, 0x73, 0x12, 0x27, 0x0a, 0x0f, 0x73, 0x74,
+	0x72, 0x69, 0x63, 0x74, 0x5f, 0x70, 0x72, 0x69, 0x6f, 0x72, 0x69, 0x74, 0x79, 0x18, 0x06, 0x20,
+	0x01, 0x28, 0x08, 0x52, 0x0e, 0x73, 0x74, 0x72, 0x69, 0x63, 0x74, 0x50, 0x72, 0x69, 0x6f, 0x72,
+	0x69, 0x74, 0x79, 0x12, 0x16, 0x0a, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x18, 0x07, 0x20,
+	0x01, 0x28, 0x09, 0x52, 0x06, 0x73, 0x74, 0x61, 0x74, 0x75, 0x73, 0x12, 0x39, 0x0a, 0x0a, 0x73,
+	0x74, 0x61, 0x72, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32,
+	0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75,
+	0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x09, 0x73, 0x74, 0x61,
+	0x72, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x2e, 0x0a, 0x13, 0x61, 0x63, 0x74, 0x69, 0x76, 0x65,
+	0x5f, 0x77, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x5f, 0x63, 0x6f, 0x75, 0x6e, 0x74, 0x18, 0x09, 0x20,
+	0x01, 0x28, 0x05, 0x52, 0x11, 0x61, 0x63, 0x74, 0x69, 0x76, 0x65, 0x57, 0x6f, 0x72, 0x6b, 0x65,
+	0x72, 0x43, 0x6f, 0x75, 0x6e, 0x74, 0x1a, 0x39, 0x0a, 0x0b, 0x51, 0x75, 0x65, 0x75, 0x65, 0x73,
+	0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x10, 0x0a, 0x03, 0x6b, 0x65, 0x79, 0x18, 0x01, 0x20, 0x01,
+	0x28, 0x09, 0x52, 0x03, 0x6b, 0x65, 0x79, 0x12, 0x14, 0x0a, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x05, 0x52, 0x05, 0x76, 0x61, 0x6c, 0x75, 0x65, 0x3a, 0x02, 0x38,
+	0x01, 0x22, 0xb1, 0x02, 0x0a, 0x0a, 0x57, 0x6f, 0x72, 0x6b, 0x65, 0x72, 0x49, 0x6e, 0x66, 0x6f,
+	0x12, 0x12, 0x0a, 0x04, 0x68, 0x6f, 0x73, 0x74, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04,
+	0x68, 0x6f, 0x73, 0x74, 0x12, 0x10, 0x0a, 0x03, 0x70, 0x69, 0x64, 0x18, 0x02, 0x20, 0x01, 0x28,
+	0x05, 0x52, 0x03, 0x70, 0x69, 0x64, 0x12, 0x1b, 0x0a, 0x09, 0x73, 0x65, 0x72, 0x76, 0x65, 0x72,
+	0x5f, 0x69, 0x64, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52, 0x08, 0x73, 0x65, 0x72, 0x76, 0x65,
+	0x72, 0x49, 0x64, 0x12, 0x17, 0x0a, 0x07, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x69, 0x64, 0x18, 0x04,
+	0x20, 0x01, 0x28, 0x09, 0x52, 0x06, 0x74, 0x61, 0x73, 0x6b, 0x49, 0x64, 0x12, 0x1b, 0x0a, 0x09,
+	0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x05, 0x20, 0x01, 0x28, 0x09, 0x52,
+	0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x21, 0x0a, 0x0c, 0x74, 0x61, 0x73,
+	0x6b, 0x5f, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0c, 0x52,
+	0x0b, 0x74, 0x61, 0x73, 0x6b, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x12, 0x14, 0x0a, 0x05,
+	0x71, 0x75, 0x65, 0x75, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, 0x09, 0x52, 0x05, 0x71, 0x75, 0x65,
+	0x75, 0x65, 0x12, 0x39, 0x0a, 0x0a, 0x73, 0x74, 0x61, 0x72, 0x74, 0x5f, 0x74, 0x69, 0x6d, 0x65,
+	0x18, 0x08, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e,
+	0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61,
+	0x6d, 0x70, 0x52, 0x09, 0x73, 0x74, 0x61, 0x72, 0x74, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x36, 0x0a,
+	0x08, 0x64, 0x65, 0x61, 0x64, 0x6c, 0x69, 0x6e, 0x65, 0x18, 0x09, 0x20, 0x01, 0x28, 0x0b, 0x32,
+	0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75,
+	0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x08, 0x64, 0x65, 0x61,
+	0x64, 0x6c, 0x69, 0x6e, 0x65, 0x22, 0xad, 0x02, 0x0a, 0x0e, 0x53, 0x63, 0x68, 0x65, 0x64, 0x75,
+	0x6c, 0x65, 0x72, 0x45, 0x6e, 0x74, 0x72, 0x79, 0x12, 0x0e, 0x0a, 0x02, 0x69, 0x64, 0x18, 0x01,
+	0x20, 0x01, 0x28, 0x09, 0x52, 0x02, 0x69, 0x64, 0x12, 0x12, 0x0a, 0x04, 0x73, 0x70, 0x65, 0x63,
+	0x18, 0x02, 0x20, 0x01, 0x28, 0x09, 0x52, 0x04, 0x73, 0x70, 0x65, 0x63, 0x12, 0x1b, 0x0a, 0x09,
+	0x74, 0x61, 0x73, 0x6b, 0x5f, 0x74, 0x79, 0x70, 0x65, 0x18, 0x03, 0x20, 0x01, 0x28, 0x09, 0x52,
+	0x08, 0x74, 0x61, 0x73, 0x6b, 0x54, 0x79, 0x70, 0x65, 0x12, 0x21, 0x0a, 0x0c, 0x74, 0x61, 0x73,
+	0x6b, 0x5f, 0x70, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x18, 0x04, 0x20, 0x01, 0x28, 0x0c, 0x52,
+	0x0b, 0x74, 0x61, 0x73, 0x6b, 0x50, 0x61, 0x79, 0x6c, 0x6f, 0x61, 0x64, 0x12, 0x27, 0x0a, 0x0f,
+	0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x6f, 0x70, 0x74, 0x69, 0x6f, 0x6e, 0x73, 0x18,
+	0x05, 0x20, 0x03, 0x28, 0x09, 0x52, 0x0e, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x4f, 0x70,
+	0x74, 0x69, 0x6f, 0x6e, 0x73, 0x12, 0x46, 0x0a, 0x11, 0x6e, 0x65, 0x78, 0x74, 0x5f, 0x65, 0x6e,
+	0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x06, 0x20, 0x01, 0x28, 0x0b,
 	0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62,
-	0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0b, 0x65, 0x6e,
-	0x71, 0x75, 0x65, 0x75, 0x65, 0x54, 0x69, 0x6d, 0x65, 0x42, 0x29, 0x5a, 0x27, 0x67, 0x69, 0x74,
-	0x68, 0x75, 0x62, 0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x68, 0x69, 0x62, 0x69, 0x6b, 0x65, 0x6e, 0x2f,
-	0x61, 0x73, 0x79, 0x6e, 0x71, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x70,
-	0x72, 0x6f, 0x74, 0x6f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
+	0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0f, 0x6e, 0x65,
+	0x78, 0x74, 0x45, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x54, 0x69, 0x6d, 0x65, 0x12, 0x46, 0x0a,
+	0x11, 0x70, 0x72, 0x65, 0x76, 0x5f, 0x65, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x5f, 0x74, 0x69,
+	0x6d, 0x65, 0x18, 0x07, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e, 0x67, 0x6f, 0x6f, 0x67, 0x6c,
+	0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e, 0x54, 0x69, 0x6d, 0x65, 0x73,
+	0x74, 0x61, 0x6d, 0x70, 0x52, 0x0f, 0x70, 0x72, 0x65, 0x76, 0x45, 0x6e, 0x71, 0x75, 0x65, 0x75,
+	0x65, 0x54, 0x69, 0x6d, 0x65, 0x22, 0x6f, 0x0a, 0x15, 0x53, 0x63, 0x68, 0x65, 0x64, 0x75, 0x6c,
+	0x65, 0x72, 0x45, 0x6e, 0x71, 0x75, 0x65, 0x75, 0x65, 0x45, 0x76, 0x65, 0x6e, 0x74, 0x12, 0x17,
+	0x0a, 0x07, 0x74, 0x61, 0x73, 0x6b, 0x5f, 0x69, 0x64, 0x18, 0x01, 0x20, 0x01, 0x28, 0x09, 0x52,
+	0x06, 0x74, 0x61, 0x73, 0x6b, 0x49, 0x64, 0x12, 0x3d, 0x0a, 0x0c, 0x65, 0x6e, 0x71, 0x75, 0x65,
+	0x75, 0x65, 0x5f, 0x74, 0x69, 0x6d, 0x65, 0x18, 0x02, 0x20, 0x01, 0x28, 0x0b, 0x32, 0x1a, 0x2e,
+	0x67, 0x6f, 0x6f, 0x67, 0x6c, 0x65, 0x2e, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x62, 0x75, 0x66, 0x2e,
+	0x54, 0x69, 0x6d, 0x65, 0x73, 0x74, 0x61, 0x6d, 0x70, 0x52, 0x0b, 0x65, 0x6e, 0x71, 0x75, 0x65,
+	0x75, 0x65, 0x54, 0x69, 0x6d, 0x65, 0x42, 0x29, 0x5a, 0x27, 0x67, 0x69, 0x74, 0x68, 0x75, 0x62,
+	0x2e, 0x63, 0x6f, 0x6d, 0x2f, 0x68, 0x69, 0x62, 0x69, 0x6b, 0x65, 0x6e, 0x2f, 0x61, 0x73, 0x79,
+	0x6e, 0x71, 0x2f, 0x69, 0x6e, 0x74, 0x65, 0x72, 0x6e, 0x61, 0x6c, 0x2f, 0x70, 0x72, 0x6f, 0x74,
+	0x6f, 0x62, 0x06, 0x70, 0x72, 0x6f, 0x74, 0x6f, 0x33,
 }
 
 var (

internal/proto/asynq.proto

@@ -11,6 +11,7 @@ option go_package = "github.com/hibiken/asynq/internal/proto";
 
 // TaskMessage is the internal representation of a task with additional
 // metadata fields.
+// Next ID: 15
 message TaskMessage {
 	// Type indicates the kind of the task to be performed.
   string type = 1;
@@ -51,6 +52,10 @@ message TaskMessage {
 	// Empty string indicates that no uniqueness lock was used.
   string unique_key = 10;
 
+  // GroupKey is a name of the group used for task aggregation.
+  // This field is optional and empty value means no aggregation for the task.
+  string group_key = 14;
+
   // Retention period specified in a number of seconds.
   // The task will be stored in redis as a completed task until the TTL
   // expires.

internal/rdb/benchmark_test.go

@@ -10,19 +10,19 @@ import (
 	"testing"
 	"time"
 
-	"github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
+	"github.com/hibiken/asynq/internal/testutil"
 )
 
 func BenchmarkEnqueue(b *testing.B) {
 	r := setup(b)
 	ctx := context.Background()
-	msg := asynqtest.NewTaskMessage("task1", nil)
+	msg := testutil.NewTaskMessage("task1", nil)
 	b.ResetTimer()
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		b.StartTimer()
 
 		if err := r.Enqueue(ctx, msg); err != nil {
@@ -45,7 +45,7 @@ func BenchmarkEnqueueUnique(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		b.StartTimer()
 
 		if err := r.EnqueueUnique(ctx, msg, uniqueTTL); err != nil {
@@ -57,13 +57,13 @@ func BenchmarkEnqueueUnique(b *testing.B) {
 func BenchmarkSchedule(b *testing.B) {
 	r := setup(b)
 	ctx := context.Background()
-	msg := asynqtest.NewTaskMessage("task1", nil)
+	msg := testutil.NewTaskMessage("task1", nil)
 	processAt := time.Now().Add(3 * time.Minute)
 	b.ResetTimer()
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		b.StartTimer()
 
 		if err := r.Schedule(ctx, msg, processAt); err != nil {
@@ -87,7 +87,7 @@ func BenchmarkScheduleUnique(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		b.StartTimer()
 
 		if err := r.ScheduleUnique(ctx, msg, processAt, uniqueTTL); err != nil {
@@ -103,9 +103,9 @@ func BenchmarkDequeueSingleQueue(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		for i := 0; i < 10; i++ {
-			m := asynqtest.NewTaskMessageWithQueue(
+			m := testutil.NewTaskMessageWithQueue(
 				fmt.Sprintf("task%d", i), nil, base.DefaultQueueName)
 			if err := r.Enqueue(ctx, m); err != nil {
 				b.Fatalf("Enqueue failed: %v", err)
@@ -127,10 +127,10 @@ func BenchmarkDequeueMultipleQueues(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
+		testutil.FlushDB(b, r.client)
 		for i := 0; i < 10; i++ {
 			for _, qname := range qnames {
-				m := asynqtest.NewTaskMessageWithQueue(
+				m := testutil.NewTaskMessageWithQueue(
 					fmt.Sprintf("%s_task%d", qname, i), nil, qname)
 				if err := r.Enqueue(ctx, m); err != nil {
 					b.Fatalf("Enqueue failed: %v", err)
@@ -147,9 +147,9 @@ func BenchmarkDequeueMultipleQueues(b *testing.B) {
 
 func BenchmarkDone(b *testing.B) {
 	r := setup(b)
-	m1 := asynqtest.NewTaskMessage("task1", nil)
-	m2 := asynqtest.NewTaskMessage("task2", nil)
-	m3 := asynqtest.NewTaskMessage("task3", nil)
+	m1 := testutil.NewTaskMessage("task1", nil)
+	m2 := testutil.NewTaskMessage("task2", nil)
+	m3 := testutil.NewTaskMessage("task3", nil)
 	msgs := []*base.TaskMessage{m1, m2, m3}
 	zs := []base.Z{
 		{Message: m1, Score: time.Now().Add(10 * time.Second).Unix()},
@@ -161,9 +161,9 @@ func BenchmarkDone(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
-		asynqtest.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
-		asynqtest.SeedLease(b, r.client, zs, base.DefaultQueueName)
+		testutil.FlushDB(b, r.client)
+		testutil.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
+		testutil.SeedLease(b, r.client, zs, base.DefaultQueueName)
 		b.StartTimer()
 
 		if err := r.Done(ctx, msgs[0]); err != nil {
@@ -174,9 +174,9 @@ func BenchmarkDone(b *testing.B) {
 
 func BenchmarkRetry(b *testing.B) {
 	r := setup(b)
-	m1 := asynqtest.NewTaskMessage("task1", nil)
-	m2 := asynqtest.NewTaskMessage("task2", nil)
-	m3 := asynqtest.NewTaskMessage("task3", nil)
+	m1 := testutil.NewTaskMessage("task1", nil)
+	m2 := testutil.NewTaskMessage("task2", nil)
+	m3 := testutil.NewTaskMessage("task3", nil)
 	msgs := []*base.TaskMessage{m1, m2, m3}
 	zs := []base.Z{
 		{Message: m1, Score: time.Now().Add(10 * time.Second).Unix()},
@@ -188,9 +188,9 @@ func BenchmarkRetry(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
-		asynqtest.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
-		asynqtest.SeedLease(b, r.client, zs, base.DefaultQueueName)
+		testutil.FlushDB(b, r.client)
+		testutil.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
+		testutil.SeedLease(b, r.client, zs, base.DefaultQueueName)
 		b.StartTimer()
 
 		if err := r.Retry(ctx, msgs[0], time.Now().Add(1*time.Minute), "error", true /*isFailure*/); err != nil {
@@ -201,9 +201,9 @@ func BenchmarkRetry(b *testing.B) {
 
 func BenchmarkArchive(b *testing.B) {
 	r := setup(b)
-	m1 := asynqtest.NewTaskMessage("task1", nil)
-	m2 := asynqtest.NewTaskMessage("task2", nil)
-	m3 := asynqtest.NewTaskMessage("task3", nil)
+	m1 := testutil.NewTaskMessage("task1", nil)
+	m2 := testutil.NewTaskMessage("task2", nil)
+	m3 := testutil.NewTaskMessage("task3", nil)
 	msgs := []*base.TaskMessage{m1, m2, m3}
 	zs := []base.Z{
 		{Message: m1, Score: time.Now().Add(10 * time.Second).Unix()},
@@ -215,9 +215,9 @@ func BenchmarkArchive(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
-		asynqtest.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
-		asynqtest.SeedLease(b, r.client, zs, base.DefaultQueueName)
+		testutil.FlushDB(b, r.client)
+		testutil.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
+		testutil.SeedLease(b, r.client, zs, base.DefaultQueueName)
 		b.StartTimer()
 
 		if err := r.Archive(ctx, msgs[0], "error"); err != nil {
@@ -228,9 +228,9 @@ func BenchmarkArchive(b *testing.B) {
 
 func BenchmarkRequeue(b *testing.B) {
 	r := setup(b)
-	m1 := asynqtest.NewTaskMessage("task1", nil)
-	m2 := asynqtest.NewTaskMessage("task2", nil)
-	m3 := asynqtest.NewTaskMessage("task3", nil)
+	m1 := testutil.NewTaskMessage("task1", nil)
+	m2 := testutil.NewTaskMessage("task2", nil)
+	m3 := testutil.NewTaskMessage("task3", nil)
 	msgs := []*base.TaskMessage{m1, m2, m3}
 	zs := []base.Z{
 		{Message: m1, Score: time.Now().Add(10 * time.Second).Unix()},
@@ -242,9 +242,9 @@ func BenchmarkRequeue(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
-		asynqtest.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
-		asynqtest.SeedLease(b, r.client, zs, base.DefaultQueueName)
+		testutil.FlushDB(b, r.client)
+		testutil.SeedActiveQueue(b, r.client, msgs, base.DefaultQueueName)
+		testutil.SeedLease(b, r.client, zs, base.DefaultQueueName)
 		b.StartTimer()
 
 		if err := r.Requeue(ctx, msgs[0]); err != nil {
@@ -258,7 +258,7 @@ func BenchmarkCheckAndEnqueue(b *testing.B) {
 	now := time.Now()
 	var zs []base.Z
 	for i := -100; i < 100; i++ {
-		msg := asynqtest.NewTaskMessage(fmt.Sprintf("task%d", i), nil)
+		msg := testutil.NewTaskMessage(fmt.Sprintf("task%d", i), nil)
 		score := now.Add(time.Duration(i) * time.Second).Unix()
 		zs = append(zs, base.Z{Message: msg, Score: score})
 	}
@@ -266,8 +266,8 @@ func BenchmarkCheckAndEnqueue(b *testing.B) {
 
 	for i := 0; i < b.N; i++ {
 		b.StopTimer()
-		asynqtest.FlushDB(b, r.client)
-		asynqtest.SeedScheduledQueue(b, r.client, zs, base.DefaultQueueName)
+		testutil.FlushDB(b, r.client)
+		testutil.SeedScheduledQueue(b, r.client, zs, base.DefaultQueueName)
 		b.StartTimer()
 
 		if err := r.ForwardIfReady(base.DefaultQueueName); err != nil {

internal/rdb/inspect.go

@@ -10,7 +10,7 @@ import (
 	"strings"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/errors"
 	"github.com/spf13/cast"
@@ -34,13 +34,18 @@ type Stats struct {
 	Paused bool
 	// Size is the total number of tasks in the queue.
 	Size int
+
+	// Groups is the total number of groups in the queue.
+	Groups int
+
 	// Number of tasks in each state.
-	Pending   int
-	Active    int
-	Scheduled int
-	Retry     int
-	Archived  int
-	Completed int
+	Pending     int
+	Active      int
+	Scheduled   int
+	Retry       int
+	Archived    int
+	Completed   int
+	Aggregating int
 
 	// Number of tasks processed within the current date.
 	// The number includes both succeeded and failed tasks.
@@ -83,8 +88,10 @@ type DailyStats struct {
 // KEYS[9] ->  asynq:<qname>:processed
 // KEYS[10] -> asynq:<qname>:failed
 // KEYS[11] -> asynq:<qname>:paused
-//
+// KEYS[12] -> asynq:<qname>:groups
+// --------
 // ARGV[1] -> task key prefix
+// ARGV[2] -> group key prefix
 var currentStatsCmd = redis.NewScript(`
 local res = {}
 local pendingTaskCount = redis.call("LLEN", KEYS[1])
@@ -102,7 +109,7 @@ table.insert(res, KEYS[6])
 table.insert(res, redis.call("ZCARD", KEYS[6]))
 for i=7,10 do
     local count = 0
-	local n = redis.call("GET", KEYS[i])	
+	local n = redis.call("GET", KEYS[i])
 	if n then
 	    count = tonumber(n)
 	end
@@ -118,6 +125,15 @@ if pendingTaskCount > 0 then
 else
 	table.insert(res, 0)
 end
+local group_names = redis.call("SMEMBERS", KEYS[12])
+table.insert(res, "group_size")
+table.insert(res, table.getn(group_names))
+local aggregating_count = 0
+for _, gname in ipairs(group_names) do
+	aggregating_count = aggregating_count + redis.call("ZCARD", ARGV[2] .. gname)
+end
+table.insert(res, "aggregating_count")
+table.insert(res, aggregating_count)
 return res`)
 
 // CurrentStats returns a current state of the queues.
@@ -131,7 +147,7 @@ func (r *RDB) CurrentStats(qname string) (*Stats, error) {
 		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	}
 	now := r.clock.Now()
-	res, err := currentStatsCmd.Run(context.Background(), r.client, []string{
+	keys := []string{
 		base.PendingKey(qname),
 		base.ActiveKey(qname),
 		base.ScheduledKey(qname),
@@ -143,7 +159,13 @@ func (r *RDB) CurrentStats(qname string) (*Stats, error) {
 		base.ProcessedTotalKey(qname),
 		base.FailedTotalKey(qname),
 		base.PausedKey(qname),
-	}, base.TaskKeyPrefix(qname)).Result()
+		base.AllGroups(qname),
+	}
+	argv := []interface{}{
+		base.TaskKeyPrefix(qname),
+		base.GroupKeyPrefix(qname),
+	}
+	res, err := currentStatsCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
 		return nil, errors.E(op, errors.Unknown, err)
 	}
@@ -198,6 +220,11 @@ func (r *RDB) CurrentStats(qname string) (*Stats, error) {
 			} else {
 				stats.Latency = r.clock.Now().Sub(time.Unix(0, int64(val)))
 			}
+		case "group_size":
+			stats.Groups = val
+		case "aggregating_count":
+			stats.Aggregating = val
+			size += val
 		}
 	}
 	stats.Size = size
@@ -219,9 +246,12 @@ func (r *RDB) CurrentStats(qname string) (*Stats, error) {
 // KEYS[4] -> asynq:{qname}:retry
 // KEYS[5] -> asynq:{qname}:archived
 // KEYS[6] -> asynq:{qname}:completed
-//
-// ARGV[1] -> asynq:{qname}:t:
-// ARGV[2] -> sample_size (e.g 20)
+// KEYS[7] -> asynq:{qname}:groups
+// -------
+// ARGV[1] -> asynq:{qname}:t: (task key prefix)
+// ARGV[2] -> task sample size per redis list/zset (e.g 20)
+// ARGV[3] -> group sample size
+// ARGV[4] -> asynq:{qname}:g: (group key prefix)
 var memoryUsageCmd = redis.NewScript(`
 local sample_size = tonumber(ARGV[2])
 if sample_size <= 0 then
@@ -262,12 +292,36 @@ for i=3,6 do
         memusg = memusg + m
     end
 end
+local groups = redis.call("SMEMBERS", KEYS[7])
+if table.getn(groups) > 0 then
+	local agg_task_count = 0
+	local agg_task_sample_total = 0
+	local agg_task_sample_size = 0
+	for i, gname in ipairs(groups) do
+		local group_key = ARGV[4] .. gname
+		agg_task_count = agg_task_count + redis.call("ZCARD", group_key)
+		if i <= tonumber(ARGV[3]) then
+			local ids = redis.call("ZRANGE", group_key, 0, sample_size - 1)
+			for _, id in ipairs(ids) do
+				local bytes = redis.call("MEMORY", "USAGE", ARGV[1] .. id)
+				agg_task_sample_total = agg_task_sample_total + bytes
+				agg_task_sample_size = agg_task_sample_size + 1
+			end
+		end
+	end
+	local avg = agg_task_sample_total / agg_task_sample_size
+	memusg = memusg + (avg * agg_task_count)
+end
 return memusg
 `)
 
 func (r *RDB) memoryUsage(qname string) (int64, error) {
 	var op errors.Op = "rdb.memoryUsage"
-	const sampleSize = 20
+	const (
+		taskSampleSize  = 20
+		groupSampleSize = 5
+	)
+
 	keys := []string{
 		base.ActiveKey(qname),
 		base.PendingKey(qname),
@@ -275,10 +329,13 @@ func (r *RDB) memoryUsage(qname string) (int64, error) {
 		base.RetryKey(qname),
 		base.ArchivedKey(qname),
 		base.CompletedKey(qname),
+		base.AllGroups(qname),
 	}
 	argv := []interface{}{
 		base.TaskKeyPrefix(qname),
-		sampleSize,
+		taskSampleSize,
+		groupSampleSize,
+		base.GroupKeyPrefix(qname),
 	}
 	res, err := memoryUsageCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
@@ -490,6 +547,56 @@ func (r *RDB) GetTaskInfo(qname, id string) (*base.TaskInfo, error) {
 	}, nil
 }
 
+type GroupStat struct {
+	// Name of the group.
+	Group string
+
+	// Size of the group.
+	Size int
+}
+
+// KEYS[1] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> group key prefix
+//
+// Output:
+// list of group name and size (e.g. group1 size1 group2 size2 ...)
+//
+// Time Complexity:
+// O(N) where N being the number of groups in the given queue.
+var groupStatsCmd = redis.NewScript(`
+local res = {}
+local group_names = redis.call("SMEMBERS", KEYS[1])
+for _, gname in ipairs(group_names) do
+	local size = redis.call("ZCARD", ARGV[1] .. gname)
+	table.insert(res, gname)
+	table.insert(res, size)
+end
+return res
+`)
+
+func (r *RDB) GroupStats(qname string) ([]*GroupStat, error) {
+	var op errors.Op = "RDB.GroupStats"
+	keys := []string{base.AllGroups(qname)}
+	argv := []interface{}{base.GroupKeyPrefix(qname)}
+	res, err := groupStatsCmd.Run(context.Background(), r.client, keys, argv...).Result()
+	if err != nil {
+		return nil, errors.E(op, errors.Unknown, err)
+	}
+	data, err := cast.ToSliceE(res)
+	if err != nil {
+		return nil, errors.E(op, errors.Internal, "cast error: unexpected return value from Lua script")
+	}
+	var stats []*GroupStat
+	for i := 0; i < len(data); i += 2 {
+		stats = append(stats, &GroupStat{
+			Group: data[i].(string),
+			Size:  int(data[i+1].(int64)),
+		})
+	}
+	return stats, nil
+}
+
 // Pagination specifies the page size and page number
 // for the list operation.
 type Pagination struct {
@@ -619,7 +726,7 @@ func (r *RDB) ListScheduled(qname string, pgn Pagination) ([]*base.TaskInfo, err
 	if !exists {
 		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	}
-	res, err := r.listZSetEntries(qname, base.TaskStateScheduled, pgn)
+	res, err := r.listZSetEntries(qname, base.TaskStateScheduled, base.ScheduledKey(qname), pgn)
 	if err != nil {
 		return nil, errors.E(op, errors.CanonicalCode(err), err)
 	}
@@ -637,7 +744,7 @@ func (r *RDB) ListRetry(qname string, pgn Pagination) ([]*base.TaskInfo, error)
 	if !exists {
 		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	}
-	res, err := r.listZSetEntries(qname, base.TaskStateRetry, pgn)
+	res, err := r.listZSetEntries(qname, base.TaskStateRetry, base.RetryKey(qname), pgn)
 	if err != nil {
 		return nil, errors.E(op, errors.CanonicalCode(err), err)
 	}
@@ -654,7 +761,7 @@ func (r *RDB) ListArchived(qname string, pgn Pagination) ([]*base.TaskInfo, erro
 	if !exists {
 		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	}
-	zs, err := r.listZSetEntries(qname, base.TaskStateArchived, pgn)
+	zs, err := r.listZSetEntries(qname, base.TaskStateArchived, base.ArchivedKey(qname), pgn)
 	if err != nil {
 		return nil, errors.E(op, errors.CanonicalCode(err), err)
 	}
@@ -671,7 +778,24 @@ func (r *RDB) ListCompleted(qname string, pgn Pagination) ([]*base.TaskInfo, err
 	if !exists {
 		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	}
-	zs, err := r.listZSetEntries(qname, base.TaskStateCompleted, pgn)
+	zs, err := r.listZSetEntries(qname, base.TaskStateCompleted, base.CompletedKey(qname), pgn)
+	if err != nil {
+		return nil, errors.E(op, errors.CanonicalCode(err), err)
+	}
+	return zs, nil
+}
+
+// ListAggregating returns all tasks from the given group.
+func (r *RDB) ListAggregating(qname, gname string, pgn Pagination) ([]*base.TaskInfo, error) {
+	var op errors.Op = "rdb.ListAggregating"
+	exists, err := r.queueExists(qname)
+	if err != nil {
+		return nil, errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "sismember", Err: err})
+	}
+	if !exists {
+		return nil, errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
+	}
+	zs, err := r.listZSetEntries(qname, base.TaskStateAggregating, base.GroupKey(qname, gname), pgn)
 	if err != nil {
 		return nil, errors.E(op, errors.CanonicalCode(err), err)
 	}
@@ -707,20 +831,7 @@ return data
 
 // listZSetEntries returns a list of message and score pairs in Redis sorted-set
 // with the given key.
-func (r *RDB) listZSetEntries(qname string, state base.TaskState, pgn Pagination) ([]*base.TaskInfo, error) {
-	var key string
-	switch state {
-	case base.TaskStateScheduled:
-		key = base.ScheduledKey(qname)
-	case base.TaskStateRetry:
-		key = base.RetryKey(qname)
-	case base.TaskStateArchived:
-		key = base.ArchivedKey(qname)
-	case base.TaskStateCompleted:
-		key = base.CompletedKey(qname)
-	default:
-		panic(fmt.Sprintf("unsupported task state: %v", state))
-	}
+func (r *RDB) listZSetEntries(qname string, state base.TaskState, key string, pgn Pagination) ([]*base.TaskInfo, error) {
 	res, err := listZSetEntriesCmd.Run(context.Background(), r.client, []string{key},
 		pgn.start(), pgn.stop(), base.TaskKeyPrefix(qname)).Result()
 	if err != nil {
@@ -811,14 +922,67 @@ func (r *RDB) RunAllArchivedTasks(qname string) (int64, error) {
 	return n, nil
 }
 
+// runAllAggregatingCmd schedules all tasks in the group to run individually.
+//
+// Input:
+// KEYS[1] -> asynq:{<qname>}:g:<gname>
+// KEYS[2] -> asynq:{<qname>}:pending
+// KEYS[3] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> task key prefix
+// ARGV[2] -> group name
+//
+// Output:
+// integer: number of tasks scheduled to run
+var runAllAggregatingCmd = redis.NewScript(`
+local ids = redis.call("ZRANGE", KEYS[1], 0, -1)
+for _, id in ipairs(ids) do
+	redis.call("LPUSH", KEYS[2], id)
+	redis.call("HSET", ARGV[1] .. id, "state", "pending")
+end
+redis.call("DEL", KEYS[1])
+redis.call("SREM", KEYS[3], ARGV[2])
+return table.getn(ids)
+`)
+
+// RunAllAggregatingTasks schedules all tasks from the given queue to run
+// and returns the number of tasks scheduled to run.
+// If a queue with the given name doesn't exist, it returns QueueNotFoundError.
+func (r *RDB) RunAllAggregatingTasks(qname, gname string) (int64, error) {
+	var op errors.Op = "rdb.RunAllAggregatingTasks"
+	if err := r.checkQueueExists(qname); err != nil {
+		return 0, errors.E(op, errors.CanonicalCode(err), err)
+	}
+	keys := []string{
+		base.GroupKey(qname, gname),
+		base.PendingKey(qname),
+		base.AllGroups(qname),
+	}
+	argv := []interface{}{
+		base.TaskKeyPrefix(qname),
+		gname,
+	}
+	res, err := runAllAggregatingCmd.Run(context.Background(), r.client, keys, argv...).Result()
+	if err != nil {
+		return 0, errors.E(op, errors.Internal, err)
+	}
+	n, ok := res.(int64)
+	if !ok {
+		return 0, errors.E(op, errors.Internal, fmt.Sprintf("unexpected return value from script %v", res))
+	}
+	return n, nil
+}
+
 // runTaskCmd is a Lua script that updates the given task to pending state.
 //
 // Input:
 // KEYS[1] -> asynq:{<qname>}:t:<task_id>
 // KEYS[2] -> asynq:{<qname>}:pending
+// KEYS[3] -> asynq:{<qname>}:groups
 // --
 // ARGV[1] -> task ID
 // ARGV[2] -> queue key prefix; asynq:{<qname>}:
+// ARGV[3] -> group key prefix
 //
 // Output:
 // Numeric code indicating the status:
@@ -831,15 +995,24 @@ var runTaskCmd = redis.NewScript(`
 if redis.call("EXISTS", KEYS[1]) == 0 then
 	return 0
 end
-local state = redis.call("HGET", KEYS[1], "state")
+local state, group = unpack(redis.call("HMGET", KEYS[1], "state", "group"))
 if state == "active" then
 	return -1
 elseif state == "pending" then
 	return -2
-end
-local n = redis.call("ZREM", ARGV[2] .. state, ARGV[1])
-if n == 0 then
-	return redis.error_reply("internal error: task id not found in zset " .. tostring(state))
+elseif state == "aggregating" then
+	local n = redis.call("ZREM", ARGV[3] .. group, ARGV[1])
+	if n == 0 then
+		return redis.error_reply("internal error: task id not found in zset " .. tostring(ARGV[3] .. group))
+	end
+	if redis.call("ZCARD", ARGV[3] .. group) == 0 then
+		redis.call("SREM", KEYS[3], group)
+	end
+else
+	local n = redis.call("ZREM", ARGV[2] .. state, ARGV[1])
+	if n == 0 then
+		return redis.error_reply("internal error: task id not found in zset " .. tostring(ARGV[2] .. state))
+	end
 end
 redis.call("LPUSH", KEYS[2], ARGV[1])
 redis.call("HSET", KEYS[1], "state", "pending")
@@ -860,10 +1033,12 @@ func (r *RDB) RunTask(qname, id string) error {
 	keys := []string{
 		base.TaskKey(qname, id),
 		base.PendingKey(qname),
+		base.AllGroups(qname),
 	}
 	argv := []interface{}{
 		id,
 		base.QueueKeyPrefix(qname),
+		base.GroupKeyPrefix(qname),
 	}
 	res, err := runTaskCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
@@ -962,6 +1137,66 @@ func (r *RDB) ArchiveAllScheduledTasks(qname string) (int64, error) {
 	return n, nil
 }
 
+// archiveAllAggregatingCmd archives all tasks in the given group.
+//
+// Input:
+// KEYS[1] -> asynq:{<qname>}:g:<gname>
+// KEYS[2] -> asynq:{<qname>}:archived
+// KEYS[3] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> current timestamp
+// ARGV[2] -> cutoff timestamp (e.g., 90 days ago)
+// ARGV[3] -> max number of tasks in archive (e.g., 100)
+// ARGV[4] -> task key prefix (asynq:{<qname>}:t:)
+// ARGV[5] -> group name
+//
+// Output:
+// integer: Number of tasks archived
+var archiveAllAggregatingCmd = redis.NewScript(`
+local ids = redis.call("ZRANGE", KEYS[1], 0, -1)
+for _, id in ipairs(ids) do
+	redis.call("ZADD", KEYS[2], ARGV[1], id)
+	redis.call("HSET", ARGV[4] .. id, "state", "archived")
+end
+redis.call("ZREMRANGEBYSCORE", KEYS[2], "-inf", ARGV[2])
+redis.call("ZREMRANGEBYRANK", KEYS[2], 0, -ARGV[3])
+redis.call("DEL", KEYS[1])
+redis.call("SREM", KEYS[3], ARGV[5])
+return table.getn(ids)
+`)
+
+// ArchiveAllAggregatingTasks archives all aggregating tasks from the given group
+// and returns the number of tasks archived.
+// If a queue with the given name doesn't exist, it returns QueueNotFoundError.
+func (r *RDB) ArchiveAllAggregatingTasks(qname, gname string) (int64, error) {
+	var op errors.Op = "rdb.ArchiveAllAggregatingTasks"
+	if err := r.checkQueueExists(qname); err != nil {
+		return 0, errors.E(op, errors.CanonicalCode(err), err)
+	}
+	keys := []string{
+		base.GroupKey(qname, gname),
+		base.ArchivedKey(qname),
+		base.AllGroups(qname),
+	}
+	now := r.clock.Now()
+	argv := []interface{}{
+		now.Unix(),
+		now.AddDate(0, 0, -archivedExpirationInDays).Unix(),
+		maxArchiveSize,
+		base.TaskKeyPrefix(qname),
+		gname,
+	}
+	res, err := archiveAllAggregatingCmd.Run(context.Background(), r.client, keys, argv...).Result()
+	if err != nil {
+		return 0, errors.E(op, errors.Internal, err)
+	}
+	n, ok := res.(int64)
+	if !ok {
+		return 0, errors.E(op, errors.Internal, fmt.Sprintf("unexpected return value from script %v", res))
+	}
+	return n, nil
+}
+
 // archiveAllPendingCmd is a Lua script that moves all pending tasks from
 // the given queue to archived state.
 //
@@ -1022,12 +1257,14 @@ func (r *RDB) ArchiveAllPendingTasks(qname string) (int64, error) {
 // Input:
 // KEYS[1] -> task key (asynq:{<qname>}:t:<task_id>)
 // KEYS[2] -> archived key (asynq:{<qname>}:archived)
+// KEYS[3] -> all groups key (asynq:{<qname>}:groups)
 // --
 // ARGV[1] -> id of the task to archive
 // ARGV[2] -> current timestamp
 // ARGV[3] -> cutoff timestamp (e.g., 90 days ago)
 // ARGV[4] -> max number of tasks in archived state (e.g., 100)
 // ARGV[5] -> queue key prefix (asynq:{<qname>}:)
+// ARGV[6] -> group key prefix (asynq:{<qname>}:g:)
 //
 // Output:
 // Numeric code indicating the status:
@@ -1040,7 +1277,7 @@ var archiveTaskCmd = redis.NewScript(`
 if redis.call("EXISTS", KEYS[1]) == 0 then
 	return 0
 end
-local state = redis.call("HGET", KEYS[1], "state")
+local state, group = unpack(redis.call("HMGET", KEYS[1], "state", "group"))
 if state == "active" then
 	return -2
 end
@@ -1049,11 +1286,18 @@ if state == "archived" then
 end
 if state == "pending" then
 	if redis.call("LREM", ARGV[5] .. state, 1, ARGV[1]) == 0 then
-		return redis.error_reply("task id not found in list " .. tostring(state))
+		return redis.error_reply("task id not found in list " .. tostring(ARGV[5] .. state))
 	end
-else 
+elseif state == "aggregating" then
+	if redis.call("ZREM", ARGV[6] .. group, ARGV[1]) == 0 then
+		return redis.error_reply("task id not found in zset " .. tostring(ARGV[6] .. group))
+	end
+	if redis.call("ZCARD", ARGV[6] .. group) == 0 then
+		redis.call("SREM", KEYS[3], group)
+	end
+else
 	if redis.call("ZREM", ARGV[5] .. state, ARGV[1]) == 0 then
-		return redis.error_reply("task id not found in zset " .. tostring(state))
+		return redis.error_reply("task id not found in zset " .. tostring(ARGV[5] .. state))
 	end
 end
 redis.call("ZADD", KEYS[2], ARGV[2], ARGV[1])
@@ -1078,6 +1322,7 @@ func (r *RDB) ArchiveTask(qname, id string) error {
 	keys := []string{
 		base.TaskKey(qname, id),
 		base.ArchivedKey(qname),
+		base.AllGroups(qname),
 	}
 	now := r.clock.Now()
 	argv := []interface{}{
@@ -1086,6 +1331,7 @@ func (r *RDB) ArchiveTask(qname, id string) error {
 		now.AddDate(0, 0, -archivedExpirationInDays).Unix(),
 		maxArchiveSize,
 		base.QueueKeyPrefix(qname),
+		base.GroupKeyPrefix(qname),
 	}
 	res, err := archiveTaskCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
@@ -1103,7 +1349,7 @@ func (r *RDB) ArchiveTask(qname, id string) error {
 	case -1:
 		return errors.E(op, errors.FailedPrecondition, &errors.TaskAlreadyArchivedError{Queue: qname, ID: id})
 	case -2:
-		return errors.E(op, errors.FailedPrecondition, "cannot archive task in active state. use CancelTask instead.")
+		return errors.E(op, errors.FailedPrecondition, "cannot archive task in active state. use CancelProcessing instead.")
 	case -3:
 		return errors.E(op, errors.NotFound, &errors.QueueNotFoundError{Queue: qname})
 	default:
@@ -1168,9 +1414,11 @@ func (r *RDB) archiveAll(src, dst, qname string) (int64, error) {
 
 // Input:
 // KEYS[1] -> asynq:{<qname>}:t:<task_id>
+// KEYS[2] -> asynq:{<qname>}:groups
 // --
 // ARGV[1] -> task ID
 // ARGV[2] -> queue key prefix
+// ARGV[3] -> group key prefix
 //
 // Output:
 // Numeric code indicating the status:
@@ -1181,17 +1429,24 @@ var deleteTaskCmd = redis.NewScript(`
 if redis.call("EXISTS", KEYS[1]) == 0 then
 	return 0
 end
-local state = redis.call("HGET", KEYS[1], "state")
+local state, group = unpack(redis.call("HMGET", KEYS[1], "state", "group"))
 if state == "active" then
 	return -1
 end
 if state == "pending" then
 	if redis.call("LREM", ARGV[2] .. state, 0, ARGV[1]) == 0 then
-		return redis.error_reply("task is not found in list: " .. tostring(state))
+		return redis.error_reply("task is not found in list: " .. tostring(ARGV[2] .. state))
+	end
+elseif state == "aggregating" then
+	if redis.call("ZREM", ARGV[3] .. group, ARGV[1]) == 0 then
+		return redis.error_reply("task is not found in zset: " .. tostring(ARGV[3] .. group))
+	end
+	if redis.call("ZCARD", ARGV[3] .. group) == 0 then
+		redis.call("SREM", KEYS[2], group)
 	end
 else
 	if redis.call("ZREM", ARGV[2] .. state, ARGV[1]) == 0 then
-		return redis.error_reply("task is not found in zset: " .. tostring(state))
+		return redis.error_reply("task is not found in zset: " .. tostring(ARGV[2] .. state))
 	end
 end
 local unique_key = redis.call("HGET", KEYS[1], "unique_key")
@@ -1214,10 +1469,12 @@ func (r *RDB) DeleteTask(qname, id string) error {
 	}
 	keys := []string{
 		base.TaskKey(qname, id),
+		base.AllGroups(qname),
 	}
 	argv := []interface{}{
 		id,
 		base.QueueKeyPrefix(qname),
+		base.GroupKeyPrefix(qname),
 	}
 	res, err := deleteTaskCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
@@ -1233,7 +1490,7 @@ func (r *RDB) DeleteTask(qname, id string) error {
 	case 0:
 		return errors.E(op, errors.NotFound, &errors.TaskNotFoundError{Queue: qname, ID: id})
 	case -1:
-		return errors.E(op, errors.FailedPrecondition, "cannot delete task in active state. use CancelTask instead.")
+		return errors.E(op, errors.FailedPrecondition, "cannot delete task in active state. use CancelProcessing instead.")
 	default:
 		return errors.E(op, errors.Internal, fmt.Sprintf("unexpected return value from deleteTaskCmd script: %d", n))
 	}
@@ -1336,6 +1593,50 @@ func (r *RDB) deleteAll(key, qname string) (int64, error) {
 	return n, nil
 }
 
+// deleteAllAggregatingCmd deletes all tasks from the given group.
+//
+// Input:
+// KEYS[1] -> asynq:{<qname>}:g:<gname>
+// KEYS[2] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> task key prefix
+// ARGV[2] -> group name
+var deleteAllAggregatingCmd = redis.NewScript(`
+local ids = redis.call("ZRANGE", KEYS[1], 0, -1)
+for _, id in ipairs(ids) do
+	redis.call("DEL", ARGV[1] .. id)
+end
+redis.call("SREM", KEYS[2], ARGV[2])
+redis.call("DEL", KEYS[1])
+return table.getn(ids)
+`)
+
+// DeleteAllAggregatingTasks deletes all aggregating tasks from the given group
+// and returns the number of tasks deleted.
+func (r *RDB) DeleteAllAggregatingTasks(qname, gname string) (int64, error) {
+	var op errors.Op = "rdb.DeleteAllAggregatingTasks"
+	if err := r.checkQueueExists(qname); err != nil {
+		return 0, errors.E(op, errors.CanonicalCode(err), err)
+	}
+	keys := []string{
+		base.GroupKey(qname, gname),
+		base.AllGroups(qname),
+	}
+	argv := []interface{}{
+		base.TaskKeyPrefix(qname),
+		gname,
+	}
+	res, err := deleteAllAggregatingCmd.Run(context.Background(), r.client, keys, argv...).Result()
+	if err != nil {
+		return 0, errors.E(op, errors.Unknown, err)
+	}
+	n, ok := res.(int64)
+	if !ok {
+		return 0, errors.E(op, errors.Internal, "command error: unexpected return value %v", res)
+	}
+	return n, nil
+}
+
 // deleteAllPendingCmd deletes all pending tasks from the given queue.
 //
 // Input:

internal/rdb/rdb.go

@@ -11,10 +11,11 @@ import (
 	"math"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/google/uuid"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/errors"
 	"github.com/hibiken/asynq/internal/timeutil"
+	"github.com/redis/go-redis/v9"
 	"github.com/spf13/cast"
 )
 
@@ -66,7 +67,7 @@ func (r *RDB) runScript(ctx context.Context, op errors.Op, script *redis.Script,
 	return nil
 }
 
-// Runs the given script with keys and args and retuns the script's return value as int64.
+// Runs the given script with keys and args and returns the script's return value as int64.
 func (r *RDB) runScriptWithErrorCode(ctx context.Context, op errors.Op, script *redis.Script, keys []string, args ...interface{}) (int64, error) {
 	res, err := script.Run(ctx, r.client, keys, args...).Result()
 	if err != nil {
@@ -151,7 +152,7 @@ func (r *RDB) Enqueue(ctx context.Context, msg *base.TaskMessage) error {
 var enqueueUniqueCmd = redis.NewScript(`
 local ok = redis.call("SET", KEYS[1], ARGV[1], "NX", "EX", ARGV[2])
 if not ok then
-  return -1 
+  return -1
 end
 if redis.call("EXISTS", KEYS[2]) == 1 then
   return 0
@@ -348,7 +349,7 @@ func (r *RDB) Done(ctx context.Context, msg *base.TaskMessage) error {
 	argv := []interface{}{
 		msg.ID,
 		expireAt.Unix(),
-		math.MaxInt64,
+		int64(math.MaxInt64),
 	}
 	// Note: We cannot pass empty unique key when running this script in redis-cluster.
 	if len(msg.UniqueKey) > 0 {
@@ -367,7 +368,7 @@ func (r *RDB) Done(ctx context.Context, msg *base.TaskMessage) error {
 //
 // ARGV[1] -> task ID
 // ARGV[2] -> stats expiration timestamp
-// ARGV[3] -> task exipration time in unix time
+// ARGV[3] -> task expiration time in unix time
 // ARGV[4] -> task message data
 // ARGV[5] -> max int64 value
 var markAsCompleteCmd = redis.NewScript(`
@@ -378,7 +379,7 @@ if redis.call("ZREM", KEYS[2], ARGV[1]) == 0 then
   return redis.error_reply("NOT FOUND")
 end
 if redis.call("ZADD", KEYS[3], ARGV[3], ARGV[1]) ~= 1 then
-  redis.redis.error_reply("INTERNAL")
+  return redis.error_reply("INTERNAL")
 end
 redis.call("HSET", KEYS[4], "msg", ARGV[4], "state", "completed")
 local n = redis.call("INCR", KEYS[5])
@@ -404,7 +405,7 @@ return redis.status_reply("OK")
 //
 // ARGV[1] -> task ID
 // ARGV[2] -> stats expiration timestamp
-// ARGV[3] -> task exipration time in unix time
+// ARGV[3] -> task expiration time in unix time
 // ARGV[4] -> task message data
 // ARGV[5] -> max int64 value
 var markAsCompleteUniqueCmd = redis.NewScript(`
@@ -415,7 +416,7 @@ if redis.call("ZREM", KEYS[2], ARGV[1]) == 0 then
   return redis.error_reply("NOT FOUND")
 end
 if redis.call("ZADD", KEYS[3], ARGV[3], ARGV[1]) ~= 1 then
-  redis.redis.error_reply("INTERNAL")
+  return redis.error_reply("INTERNAL")
 end
 redis.call("HSET", KEYS[4], "msg", ARGV[4], "state", "completed")
 local n = redis.call("INCR", KEYS[5])
@@ -458,7 +459,7 @@ func (r *RDB) MarkAsComplete(ctx context.Context, msg *base.TaskMessage) error {
 		statsExpireAt.Unix(),
 		now.Unix() + msg.Retention,
 		encoded,
-		math.MaxInt64,
+		int64(math.MaxInt64),
 	}
 	// Note: We cannot pass empty unique key when running this script in redis-cluster.
 	if len(msg.UniqueKey) > 0 {
@@ -497,6 +498,128 @@ func (r *RDB) Requeue(ctx context.Context, msg *base.TaskMessage) error {
 	return r.runScript(ctx, op, requeueCmd, keys, msg.ID)
 }
 
+// KEYS[1] -> asynq:{<qname>}:t:<task_id>
+// KEYS[2] -> asynq:{<qname>}:g:<group_key>
+// KEYS[3] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> task message data
+// ARGV[2] -> task ID
+// ARGV[3] -> current time in Unix time
+// ARGV[4] -> group key
+//
+// Output:
+// Returns 1 if successfully added
+// Returns 0 if task ID already exists
+var addToGroupCmd = redis.NewScript(`
+if redis.call("EXISTS", KEYS[1]) == 1 then
+	return 0
+end
+redis.call("HSET", KEYS[1],
+           "msg", ARGV[1],
+           "state", "aggregating",
+	       "group", ARGV[4])
+redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
+redis.call("SADD", KEYS[3], ARGV[4])
+return 1
+`)
+
+func (r *RDB) AddToGroup(ctx context.Context, msg *base.TaskMessage, groupKey string) error {
+	var op errors.Op = "rdb.AddToGroup"
+	encoded, err := base.EncodeMessage(msg)
+	if err != nil {
+		return errors.E(op, errors.Unknown, fmt.Sprintf("cannot encode message: %v", err))
+	}
+	if err := r.client.SAdd(ctx, base.AllQueues, msg.Queue).Err(); err != nil {
+		return errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "sadd", Err: err})
+	}
+	keys := []string{
+		base.TaskKey(msg.Queue, msg.ID),
+		base.GroupKey(msg.Queue, groupKey),
+		base.AllGroups(msg.Queue),
+	}
+	argv := []interface{}{
+		encoded,
+		msg.ID,
+		r.clock.Now().Unix(),
+		groupKey,
+	}
+	n, err := r.runScriptWithErrorCode(ctx, op, addToGroupCmd, keys, argv...)
+	if err != nil {
+		return err
+	}
+	if n == 0 {
+		return errors.E(op, errors.AlreadyExists, errors.ErrTaskIdConflict)
+	}
+	return nil
+}
+
+// KEYS[1] -> asynq:{<qname>}:t:<task_id>
+// KEYS[2] -> asynq:{<qname>}:g:<group_key>
+// KEYS[3] -> asynq:{<qname>}:groups
+// KEYS[4] -> unique key
+// -------
+// ARGV[1] -> task message data
+// ARGV[2] -> task ID
+// ARGV[3] -> current time in Unix time
+// ARGV[4] -> group key
+// ARGV[5] -> uniqueness lock TTL
+//
+// Output:
+// Returns 1 if successfully added
+// Returns 0 if task ID already exists
+// Returns -1 if task unique key already exists
+var addToGroupUniqueCmd = redis.NewScript(`
+local ok = redis.call("SET", KEYS[4], ARGV[2], "NX", "EX", ARGV[5])
+if not ok then
+  return -1
+end
+if redis.call("EXISTS", KEYS[1]) == 1 then
+	return 0
+end
+redis.call("HSET", KEYS[1],
+           "msg", ARGV[1],
+           "state", "aggregating",
+	       "group", ARGV[4])
+redis.call("ZADD", KEYS[2], ARGV[3], ARGV[2])
+redis.call("SADD", KEYS[3], ARGV[4])
+return 1
+`)
+
+func (r *RDB) AddToGroupUnique(ctx context.Context, msg *base.TaskMessage, groupKey string, ttl time.Duration) error {
+	var op errors.Op = "rdb.AddToGroupUnique"
+	encoded, err := base.EncodeMessage(msg)
+	if err != nil {
+		return errors.E(op, errors.Unknown, fmt.Sprintf("cannot encode message: %v", err))
+	}
+	if err := r.client.SAdd(ctx, base.AllQueues, msg.Queue).Err(); err != nil {
+		return errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "sadd", Err: err})
+	}
+	keys := []string{
+		base.TaskKey(msg.Queue, msg.ID),
+		base.GroupKey(msg.Queue, groupKey),
+		base.AllGroups(msg.Queue),
+		base.UniqueKey(msg.Queue, msg.Type, msg.Payload),
+	}
+	argv := []interface{}{
+		encoded,
+		msg.ID,
+		r.clock.Now().Unix(),
+		groupKey,
+		int(ttl.Seconds()),
+	}
+	n, err := r.runScriptWithErrorCode(ctx, op, addToGroupUniqueCmd, keys, argv...)
+	if err != nil {
+		return err
+	}
+	if n == -1 {
+		return errors.E(op, errors.AlreadyExists, errors.ErrDuplicateTask)
+	}
+	if n == 0 {
+		return errors.E(op, errors.AlreadyExists, errors.ErrTaskIdConflict)
+	}
+	return nil
+}
+
 // KEYS[1] -> asynq:{<qname>}:t:<task_id>
 // KEYS[2] -> asynq:{<qname>}:scheduled
 // -------
@@ -688,7 +811,7 @@ func (r *RDB) Retry(ctx context.Context, msg *base.TaskMessage, processAt time.T
 		processAt.Unix(),
 		expireAt.Unix(),
 		isFailure,
-		math.MaxInt64,
+		int64(math.MaxInt64),
 	}
 	return r.runScript(ctx, op, retryCmd, keys, argv...)
 }
@@ -774,7 +897,7 @@ func (r *RDB) Archive(ctx context.Context, msg *base.TaskMessage, errMsg string)
 		cutoff.Unix(),
 		maxArchiveSize,
 		expireAt.Unix(),
-		math.MaxInt64,
+		int64(math.MaxInt64),
 	}
 	return r.runScript(ctx, op, archiveCmd, keys, argv...)
 }
@@ -796,24 +919,41 @@ func (r *RDB) ForwardIfReady(qnames ...string) error {
 // ARGV[1] -> current unix time in seconds
 // ARGV[2] -> task key prefix
 // ARGV[3] -> current unix time in nsec
+// ARGV[4] -> group key prefix
 // Note: Script moves tasks up to 100 at a time to keep the runtime of script short.
 var forwardCmd = redis.NewScript(`
 local ids = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1], "LIMIT", 0, 100)
 for _, id in ipairs(ids) do
-	redis.call("LPUSH", KEYS[2], id)
-	redis.call("ZREM", KEYS[1], id)
-	redis.call("HSET", ARGV[2] .. id,
-               "state", "pending",
-               "pending_since", ARGV[3])
+	local taskKey = ARGV[2] .. id
+	local group = redis.call("HGET", taskKey, "group")
+	if group and group ~= '' then
+	    redis.call("ZADD", ARGV[4] .. group, ARGV[1], id)
+		redis.call("ZREM", KEYS[1], id)
+		redis.call("HSET", taskKey,
+				   "state", "aggregating")
+	else
+		redis.call("LPUSH", KEYS[2], id)
+		redis.call("ZREM", KEYS[1], id)
+		redis.call("HSET", taskKey,
+				   "state", "pending",
+				   "pending_since", ARGV[3])
+	end
 end
 return table.getn(ids)`)
 
-// forward moves tasks with a score less than the current unix time
-// from the src zset to the dst list. It returns the number of tasks moved.
-func (r *RDB) forward(src, dst, taskKeyPrefix string) (int, error) {
+// forward moves tasks with a score less than the current unix time from the delayed (i.e. scheduled | retry) zset
+// to the pending list or group set.
+// It returns the number of tasks moved.
+func (r *RDB) forward(delayedKey, pendingKey, taskKeyPrefix, groupKeyPrefix string) (int, error) {
 	now := r.clock.Now()
-	res, err := forwardCmd.Run(context.Background(), r.client,
-		[]string{src, dst}, now.Unix(), taskKeyPrefix, now.UnixNano()).Result()
+	keys := []string{delayedKey, pendingKey}
+	argv := []interface{}{
+		now.Unix(),
+		taskKeyPrefix,
+		now.UnixNano(),
+		groupKeyPrefix,
+	}
+	res, err := forwardCmd.Run(context.Background(), r.client, keys, argv...).Result()
 	if err != nil {
 		return 0, errors.E(errors.Internal, fmt.Sprintf("redis eval error: %v", err))
 	}
@@ -825,15 +965,16 @@ func (r *RDB) forward(src, dst, taskKeyPrefix string) (int, error) {
 }
 
 // forwardAll checks for tasks in scheduled/retry state that are ready to be run, and updates
-// their state to "pending".
+// their state to "pending" or "aggregating".
 func (r *RDB) forwardAll(qname string) (err error) {
-	sources := []string{base.ScheduledKey(qname), base.RetryKey(qname)}
-	dst := base.PendingKey(qname)
+	delayedKeys := []string{base.ScheduledKey(qname), base.RetryKey(qname)}
+	pendingKey := base.PendingKey(qname)
 	taskKeyPrefix := base.TaskKeyPrefix(qname)
-	for _, src := range sources {
+	groupKeyPrefix := base.GroupKeyPrefix(qname)
+	for _, delayedKey := range delayedKeys {
 		n := 1
 		for n != 0 {
-			n, err = r.forward(src, dst, taskKeyPrefix)
+			n, err = r.forward(delayedKey, pendingKey, taskKeyPrefix, groupKeyPrefix)
 			if err != nil {
 				return err
 			}
@@ -842,6 +983,248 @@ func (r *RDB) forwardAll(qname string) (err error) {
 	return nil
 }
 
+// ListGroups returns a list of all known groups in the given queue.
+func (r *RDB) ListGroups(qname string) ([]string, error) {
+	var op errors.Op = "RDB.ListGroups"
+	groups, err := r.client.SMembers(context.Background(), base.AllGroups(qname)).Result()
+	if err != nil {
+		return nil, errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "smembers", Err: err})
+	}
+	return groups, nil
+}
+
+// aggregationCheckCmd checks the given group for whether to create an aggregation set.
+// An aggregation set is created if one of the aggregation criteria is met:
+// 1) group has reached or exceeded its max size
+// 2) group's oldest task has reached or exceeded its max delay
+// 3) group's latest task has reached or exceeded its grace period
+// if aggreation criteria is met, the command moves those tasks from the group
+// and put them in an aggregation set. Additionally, if the creation of aggregation set
+// empties the group, it will clear the group name from the all groups set.
+//
+// KEYS[1] -> asynq:{<qname>}:g:<gname>
+// KEYS[2] -> asynq:{<qname>}:g:<gname>:<aggregation_set_id>
+// KEYS[3] -> asynq:{<qname>}:aggregation_sets
+// KEYS[4] -> asynq:{<qname>}:groups
+// -------
+// ARGV[1] -> max group size
+// ARGV[2] -> max group delay in unix time
+// ARGV[3] -> start time of the grace period
+// ARGV[4] -> aggregation set expire time
+// ARGV[5] -> current time in unix time
+// ARGV[6] -> group name
+//
+// Output:
+// Returns 0 if no aggregation set was created
+// Returns 1 if an aggregation set was created
+//
+// Time Complexity:
+// O(log(N) + M) with N being the number tasks in the group zset
+// and M being the max size.
+var aggregationCheckCmd = redis.NewScript(`
+local size = redis.call("ZCARD", KEYS[1])
+if size == 0 then
+	return 0
+end
+local maxSize = tonumber(ARGV[1])
+if maxSize ~= 0 and size >= maxSize then
+	local res = redis.call("ZRANGE", KEYS[1], 0, maxSize-1, "WITHSCORES")
+	for i=1, table.getn(res)-1, 2 do
+		redis.call("ZADD", KEYS[2], tonumber(res[i+1]), res[i])
+	end
+	redis.call("ZREMRANGEBYRANK", KEYS[1], 0, maxSize-1)
+	redis.call("ZADD", KEYS[3], ARGV[4], KEYS[2])
+	if size == maxSize then
+		redis.call("SREM", KEYS[4], ARGV[6])
+	end
+	return 1
+end
+local maxDelay = tonumber(ARGV[2])
+local currentTime = tonumber(ARGV[5])
+if maxDelay ~= 0 then
+	local oldestEntry = redis.call("ZRANGE", KEYS[1], 0, 0, "WITHSCORES")
+	local oldestEntryScore = tonumber(oldestEntry[2])
+	local maxDelayTime = currentTime - maxDelay
+	if oldestEntryScore <= maxDelayTime then
+		local res = redis.call("ZRANGE", KEYS[1], 0, maxSize-1, "WITHSCORES")
+		for i=1, table.getn(res)-1, 2 do
+			redis.call("ZADD", KEYS[2], tonumber(res[i+1]), res[i])
+		end
+		redis.call("ZREMRANGEBYRANK", KEYS[1], 0, maxSize-1)
+		redis.call("ZADD", KEYS[3], ARGV[4], KEYS[2])
+		if size <= maxSize or maxSize == 0 then
+			redis.call("SREM", KEYS[4], ARGV[6])
+		end
+		return 1
+	end
+end
+local latestEntry = redis.call("ZREVRANGE", KEYS[1], 0, 0, "WITHSCORES")
+local latestEntryScore = tonumber(latestEntry[2])
+local gracePeriodStartTime = currentTime - tonumber(ARGV[3])
+if latestEntryScore <= gracePeriodStartTime then
+	local res = redis.call("ZRANGE", KEYS[1], 0, maxSize-1, "WITHSCORES")
+	for i=1, table.getn(res)-1, 2 do
+		redis.call("ZADD", KEYS[2], tonumber(res[i+1]), res[i])
+	end
+	redis.call("ZREMRANGEBYRANK", KEYS[1], 0, maxSize-1)
+	redis.call("ZADD", KEYS[3], ARGV[4], KEYS[2])
+	if size <= maxSize or maxSize == 0 then
+		redis.call("SREM", KEYS[4], ARGV[6])
+	end
+	return 1
+end
+return 0
+`)
+
+// Task aggregation should finish within this timeout.
+// Otherwise an aggregation set should be reclaimed by the recoverer.
+const aggregationTimeout = 2 * time.Minute
+
+// AggregationCheck checks the group identified by the given queue and group name to see if the tasks in the
+// group are ready to be aggregated. If so, it moves the tasks to be aggregated to a aggregation set and returns
+// the set ID. If not, it returns an empty string for the set ID.
+// The time for gracePeriod and maxDelay is computed relative to the time t.
+//
+// Note: It assumes that this function is called at frequency less than or equal to the gracePeriod. In other words,
+// the function only checks the most recently added task against the given gracePeriod.
+func (r *RDB) AggregationCheck(qname, gname string, t time.Time, gracePeriod, maxDelay time.Duration, maxSize int) (string, error) {
+	var op errors.Op = "RDB.AggregationCheck"
+	aggregationSetID := uuid.NewString()
+	expireTime := r.clock.Now().Add(aggregationTimeout)
+	keys := []string{
+		base.GroupKey(qname, gname),
+		base.AggregationSetKey(qname, gname, aggregationSetID),
+		base.AllAggregationSets(qname),
+		base.AllGroups(qname),
+	}
+	argv := []interface{}{
+		maxSize,
+		int64(maxDelay.Seconds()),
+		int64(gracePeriod.Seconds()),
+		expireTime.Unix(),
+		t.Unix(),
+		gname,
+	}
+	n, err := r.runScriptWithErrorCode(context.Background(), op, aggregationCheckCmd, keys, argv...)
+	if err != nil {
+		return "", err
+	}
+	switch n {
+	case 0:
+		return "", nil
+	case 1:
+		return aggregationSetID, nil
+	default:
+		return "", errors.E(op, errors.Internal, fmt.Sprintf("unexpected return value from lua script: %d", n))
+	}
+}
+
+// KEYS[1] -> asynq:{<qname>}:g:<gname>:<aggregation_set_id>
+// ------
+// ARGV[1] -> task key prefix
+//
+// Output:
+// Array of encoded task messages
+//
+// Time Complexity:
+// O(N) with N being the number of tasks in the aggregation set.
+var readAggregationSetCmd = redis.NewScript(`
+local msgs = {}
+local ids = redis.call("ZRANGE", KEYS[1], 0, -1)
+for _, id in ipairs(ids) do
+	local key = ARGV[1] .. id
+	table.insert(msgs, redis.call("HGET", key, "msg"))
+end
+return msgs
+`)
+
+// ReadAggregationSet retrieves members of an aggregation set and returns a list of tasks in the set and
+// the deadline for aggregating those tasks.
+func (r *RDB) ReadAggregationSet(qname, gname, setID string) ([]*base.TaskMessage, time.Time, error) {
+	var op errors.Op = "RDB.ReadAggregationSet"
+	ctx := context.Background()
+	aggSetKey := base.AggregationSetKey(qname, gname, setID)
+	res, err := readAggregationSetCmd.Run(ctx, r.client,
+		[]string{aggSetKey}, base.TaskKeyPrefix(qname)).Result()
+	if err != nil {
+		return nil, time.Time{}, errors.E(op, errors.Unknown, fmt.Sprintf("redis eval error: %v", err))
+	}
+	data, err := cast.ToStringSliceE(res)
+	if err != nil {
+		return nil, time.Time{}, errors.E(op, errors.Internal, fmt.Sprintf("cast error: Lua script returned unexpected value: %v", res))
+	}
+	var msgs []*base.TaskMessage
+	for _, s := range data {
+		msg, err := base.DecodeMessage([]byte(s))
+		if err != nil {
+			return nil, time.Time{}, errors.E(op, errors.Internal, fmt.Sprintf("cannot decode message: %v", err))
+		}
+		msgs = append(msgs, msg)
+	}
+	deadlineUnix, err := r.client.ZScore(ctx, base.AllAggregationSets(qname), aggSetKey).Result()
+	if err != nil {
+		return nil, time.Time{}, errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "zscore", Err: err})
+	}
+	return msgs, time.Unix(int64(deadlineUnix), 0), nil
+}
+
+// KEYS[1] -> asynq:{<qname>}:g:<gname>:<aggregation_set_id>
+// KEYS[2] -> asynq:{<qname>}:aggregation_sets
+// -------
+// ARGV[1] -> task key prefix
+//
+// Output:
+// Redis status reply
+//
+// Time Complexity:
+// max(O(N), O(log(M))) with N being the number of tasks in the aggregation set
+// and M being the number of elements in the all-aggregation-sets list.
+var deleteAggregationSetCmd = redis.NewScript(`
+local ids = redis.call("ZRANGE", KEYS[1], 0, -1)
+for _, id in ipairs(ids)  do
+	redis.call("DEL", ARGV[1] .. id)
+end
+redis.call("DEL", KEYS[1])
+redis.call("ZREM", KEYS[2], KEYS[1])
+return redis.status_reply("OK")
+`)
+
+// DeleteAggregationSet deletes the aggregation set and its members identified by the parameters.
+func (r *RDB) DeleteAggregationSet(ctx context.Context, qname, gname, setID string) error {
+	var op errors.Op = "RDB.DeleteAggregationSet"
+	keys := []string{
+		base.AggregationSetKey(qname, gname, setID),
+		base.AllAggregationSets(qname),
+	}
+	return r.runScript(ctx, op, deleteAggregationSetCmd, keys, base.TaskKeyPrefix(qname))
+}
+
+// KEYS[1] -> asynq:{<qname>}:aggregation_sets
+// -------
+// ARGV[1] -> current time in unix time
+var reclaimStateAggregationSetsCmd = redis.NewScript(`
+local staleSetKeys = redis.call("ZRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
+for _, key in ipairs(staleSetKeys) do
+	local idx = string.find(key, ":[^:]*$")
+	local groupKey = string.sub(key, 1, idx-1)
+	local res = redis.call("ZRANGE", key, 0, -1, "WITHSCORES")
+	for i=1, table.getn(res)-1, 2 do
+		redis.call("ZADD", groupKey, tonumber(res[i+1]), res[i])
+	end
+	redis.call("DEL", key)
+end
+redis.call("ZREMRANGEBYSCORE", KEYS[1], "-inf", ARGV[1])
+return redis.status_reply("OK")
+`)
+
+// ReclaimStateAggregationSets checks for any stale aggregation sets in the given queue, and
+// reclaim tasks in the stale aggregation set by putting them back in the group.
+func (r *RDB) ReclaimStaleAggregationSets(qname string) error {
+	var op errors.Op = "RDB.ReclaimStaleAggregationSets"
+	return r.runScript(context.Background(), op, reclaimStateAggregationSetsCmd,
+		[]string{base.AllAggregationSets(qname)}, r.clock.Now().Unix())
+}
+
 // KEYS[1] -> asynq:{<qname>}:completed
 // ARGV[1] -> current time in unix time
 // ARGV[2] -> task key prefix
@@ -941,7 +1324,8 @@ func (r *RDB) ExtendLease(qname string, ids ...string) (expirationTime time.Time
 		zs = append(zs, redis.Z{Member: id, Score: float64(expireAt.Unix())})
 	}
 	// Use XX option to only update elements that already exist; Don't add new elements
-	err = r.client.ZAddArgs(context.Background(), base.LeaseKey(qname), redis.ZAddArgs{XX: true, GT: true, Members: zs}).Err()
+	// TODO: Consider adding GT option to ensure we only "extend" the lease. Ceveat is that GT is supported from redis v6.2.0 or above.
+	err = r.client.ZAddXX(context.Background(), base.LeaseKey(qname), zs...).Err()
 	if err != nil {
 		return time.Time{}, err
 	}
@@ -983,10 +1367,10 @@ func (r *RDB) WriteServerState(info *base.ServerInfo, workers []*base.WorkerInfo
 	}
 	skey := base.ServerInfoKey(info.Host, info.PID, info.ServerID)
 	wkey := base.WorkersKey(info.Host, info.PID, info.ServerID)
-	if err := r.client.ZAdd(ctx, base.AllServers, &redis.Z{Score: float64(exp.Unix()), Member: skey}).Err(); err != nil {
+	if err := r.client.ZAdd(ctx, base.AllServers, redis.Z{Score: float64(exp.Unix()), Member: skey}).Err(); err != nil {
 		return errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "sadd", Err: err})
 	}
-	if err := r.client.ZAdd(ctx, base.AllWorkers, &redis.Z{Score: float64(exp.Unix()), Member: wkey}).Err(); err != nil {
+	if err := r.client.ZAdd(ctx, base.AllWorkers, redis.Z{Score: float64(exp.Unix()), Member: wkey}).Err(); err != nil {
 		return errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "zadd", Err: err})
 	}
 	return r.runScript(ctx, op, writeServerStateCmd, []string{skey, wkey}, args...)
@@ -1039,7 +1423,7 @@ func (r *RDB) WriteSchedulerEntries(schedulerID string, entries []*base.Schedule
 	}
 	exp := r.clock.Now().Add(ttl).UTC()
 	key := base.SchedulerEntriesKey(schedulerID)
-	err := r.client.ZAdd(ctx, base.AllSchedulers, &redis.Z{Score: float64(exp.Unix()), Member: key}).Err()
+	err := r.client.ZAdd(ctx, base.AllSchedulers, redis.Z{Score: float64(exp.Unix()), Member: key}).Err()
 	if err != nil {
 		return errors.E(op, errors.Unknown, &errors.RedisCommandError{Command: "zadd", Err: err})
 	}

internal/testbroker/testbroker.go

@@ -11,11 +11,11 @@ import (
 	"sync"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 )
 
-var errRedisDown = errors.New("asynqtest: redis is down")
+var errRedisDown = errors.New("testutil: redis is down")
 
 // TestBroker is a broker implementation which enables
 // to simulate Redis failure in tests.
@@ -234,3 +234,66 @@ func (tb *TestBroker) Close() error {
 	}
 	return tb.real.Close()
 }
+
+func (tb *TestBroker) AddToGroup(ctx context.Context, msg *base.TaskMessage, gname string) error {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return errRedisDown
+	}
+	return tb.real.AddToGroup(ctx, msg, gname)
+}
+
+func (tb *TestBroker) AddToGroupUnique(ctx context.Context, msg *base.TaskMessage, gname string, ttl time.Duration) error {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return errRedisDown
+	}
+	return tb.real.AddToGroupUnique(ctx, msg, gname, ttl)
+}
+
+func (tb *TestBroker) ListGroups(qname string) ([]string, error) {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return nil, errRedisDown
+	}
+	return tb.real.ListGroups(qname)
+}
+
+func (tb *TestBroker) AggregationCheck(qname, gname string, t time.Time, gracePeriod, maxDelay time.Duration, maxSize int) (aggregationSetID string, err error) {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return "", errRedisDown
+	}
+	return tb.real.AggregationCheck(qname, gname, t, gracePeriod, maxDelay, maxSize)
+}
+
+func (tb *TestBroker) ReadAggregationSet(qname, gname, aggregationSetID string) ([]*base.TaskMessage, time.Time, error) {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return nil, time.Time{}, errRedisDown
+	}
+	return tb.real.ReadAggregationSet(qname, gname, aggregationSetID)
+}
+
+func (tb *TestBroker) DeleteAggregationSet(ctx context.Context, qname, gname, aggregationSetID string) error {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return errRedisDown
+	}
+	return tb.real.DeleteAggregationSet(ctx, qname, gname, aggregationSetID)
+}
+
+func (tb *TestBroker) ReclaimStaleAggregationSets(qname string) error {
+	tb.mu.Lock()
+	defer tb.mu.Unlock()
+	if tb.sleeping {
+		return errRedisDown
+	}
+	return tb.real.ReclaimStaleAggregationSets(qname)
+}

internal/testutil/builder.go

@@ -0,0 +1,84 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package testutil
+
+import (
+	"time"
+
+	"github.com/google/uuid"
+	"github.com/hibiken/asynq/internal/base"
+)
+
+func makeDefaultTaskMessage() *base.TaskMessage {
+	return &base.TaskMessage{
+		ID:       uuid.NewString(),
+		Type:     "default_task",
+		Queue:    "default",
+		Retry:    25,
+		Timeout:  1800, // default timeout of 30 mins
+		Deadline: 0,    // no deadline
+	}
+}
+
+type TaskMessageBuilder struct {
+	msg *base.TaskMessage
+}
+
+func NewTaskMessageBuilder() *TaskMessageBuilder {
+	return &TaskMessageBuilder{}
+}
+
+func (b *TaskMessageBuilder) lazyInit() {
+	if b.msg == nil {
+		b.msg = makeDefaultTaskMessage()
+	}
+}
+
+func (b *TaskMessageBuilder) Build() *base.TaskMessage {
+	b.lazyInit()
+	return b.msg
+}
+
+func (b *TaskMessageBuilder) SetType(typename string) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Type = typename
+	return b
+}
+
+func (b *TaskMessageBuilder) SetPayload(payload []byte) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Payload = payload
+	return b
+}
+
+func (b *TaskMessageBuilder) SetQueue(qname string) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Queue = qname
+	return b
+}
+
+func (b *TaskMessageBuilder) SetRetry(n int) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Retry = n
+	return b
+}
+
+func (b *TaskMessageBuilder) SetTimeout(timeout time.Duration) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Timeout = int64(timeout.Seconds())
+	return b
+}
+
+func (b *TaskMessageBuilder) SetDeadline(deadline time.Time) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.Deadline = deadline.Unix()
+	return b
+}
+
+func (b *TaskMessageBuilder) SetGroup(gname string) *TaskMessageBuilder {
+	b.lazyInit()
+	b.msg.GroupKey = gname
+	return b
+}

internal/testutil/builder_test.go

@@ -0,0 +1,94 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package testutil
+
+import (
+	"testing"
+	"time"
+
+	"github.com/google/go-cmp/cmp"
+	"github.com/google/go-cmp/cmp/cmpopts"
+	"github.com/hibiken/asynq/internal/base"
+)
+
+func TestTaskMessageBuilder(t *testing.T) {
+	tests := []struct {
+		desc string
+		ops  func(b *TaskMessageBuilder) // operations to perform on the builder
+		want *base.TaskMessage
+	}{
+		{
+			desc: "zero value and build",
+			ops:  nil,
+			want: &base.TaskMessage{
+				Type:     "default_task",
+				Queue:    "default",
+				Payload:  nil,
+				Retry:    25,
+				Timeout:  1800, // 30m
+				Deadline: 0,
+			},
+		},
+		{
+			desc: "with type, payload, and queue",
+			ops: func(b *TaskMessageBuilder) {
+				b.SetType("foo").SetPayload([]byte("hello")).SetQueue("myqueue")
+			},
+			want: &base.TaskMessage{
+				Type:     "foo",
+				Queue:    "myqueue",
+				Payload:  []byte("hello"),
+				Retry:    25,
+				Timeout:  1800, // 30m
+				Deadline: 0,
+			},
+		},
+		{
+			desc: "with retry, timeout, and deadline",
+			ops: func(b *TaskMessageBuilder) {
+				b.SetRetry(1).
+					SetTimeout(20 * time.Second).
+					SetDeadline(time.Date(2017, 3, 6, 0, 0, 0, 0, time.UTC))
+			},
+			want: &base.TaskMessage{
+				Type:     "default_task",
+				Queue:    "default",
+				Payload:  nil,
+				Retry:    1,
+				Timeout:  20,
+				Deadline: time.Date(2017, 3, 6, 0, 0, 0, 0, time.UTC).Unix(),
+			},
+		},
+		{
+			desc: "with group",
+			ops: func(b *TaskMessageBuilder) {
+				b.SetGroup("mygroup")
+			},
+			want: &base.TaskMessage{
+				Type:     "default_task",
+				Queue:    "default",
+				Payload:  nil,
+				Retry:    25,
+				Timeout:  1800,
+				Deadline: 0,
+				GroupKey: "mygroup",
+			},
+		},
+	}
+	cmpOpts := []cmp.Option{cmpopts.IgnoreFields(base.TaskMessage{}, "ID")}
+
+	for _, tc := range tests {
+		var b TaskMessageBuilder
+		if tc.ops != nil {
+			tc.ops(&b)
+		}
+
+		got := b.Build()
+		if diff := cmp.Diff(tc.want, got, cmpOpts...); diff != "" {
+			t.Errorf("%s: TaskMessageBuilder.Build() = %+v, want %+v;\n(-want,+got)\n%s",
+				tc.desc, got, tc.want, diff)
+		}
+	}
+}

internal/testutil/testutil.go

@@ -2,8 +2,8 @@
 // Use of this source code is governed by a MIT license
 // that can be found in the LICENSE file.
 
-// Package asynqtest defines test helpers for asynq and its internal packages.
-package asynqtest
+// Package testutil defines test helpers for asynq and its internal packages.
+package testutil
 
 import (
 	"context"
@@ -13,12 +13,12 @@ import (
 	"testing"
 	"time"
 
-	"github.com/go-redis/redis/v8"
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
 	"github.com/google/uuid"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/timeutil"
+	"github.com/redis/go-redis/v9"
 )
 
 // EquateInt64Approx returns a Comparer option that treats int64 values
@@ -93,6 +93,20 @@ var SortStringSliceOpt = cmp.Transformer("SortStringSlice", func(in []string) []
 	return out
 })
 
+var SortRedisZSetEntryOpt = cmp.Transformer("SortZSetEntries", func(in []redis.Z) []redis.Z {
+	out := append([]redis.Z(nil), in...) // Copy input to avoid mutating it
+	sort.Slice(out, func(i, j int) bool {
+		// TODO: If member is a comparable type (int, string, etc) compare by the member
+		// Use generic comparable type here once update to go1.18
+		if _, ok := out[i].Member.(string); ok {
+			// If member is a string, compare the member
+			return out[i].Member.(string) < out[j].Member.(string)
+		}
+		return out[i].Score < out[j].Score
+	})
+	return out
+})
+
 // IgnoreIDOpt is an cmp.Option to ignore ID field in task messages when comparing.
 var IgnoreIDOpt = cmpopts.IgnoreFields(base.TaskMessage{}, "ID")
 
@@ -238,13 +252,28 @@ func SeedLease(tb testing.TB, r redis.UniversalClient, entries []base.Z, qname s
 	seedRedisZSet(tb, r, base.LeaseKey(qname), entries, base.TaskStateActive)
 }
 
-// SeedCompletedQueue initializes the completed set witht the given entries.
+// SeedCompletedQueue initializes the completed set with the given entries.
 func SeedCompletedQueue(tb testing.TB, r redis.UniversalClient, entries []base.Z, qname string) {
 	tb.Helper()
 	r.SAdd(context.Background(), base.AllQueues, qname)
 	seedRedisZSet(tb, r, base.CompletedKey(qname), entries, base.TaskStateCompleted)
 }
 
+// SeedGroup initializes the group with the given entries.
+func SeedGroup(tb testing.TB, r redis.UniversalClient, entries []base.Z, qname, gname string) {
+	tb.Helper()
+	ctx := context.Background()
+	r.SAdd(ctx, base.AllQueues, qname)
+	r.SAdd(ctx, base.AllGroups(qname), gname)
+	seedRedisZSet(tb, r, base.GroupKey(qname, gname), entries, base.TaskStateAggregating)
+}
+
+func SeedAggregationSet(tb testing.TB, r redis.UniversalClient, entries []base.Z, qname, gname, setID string) {
+	tb.Helper()
+	r.SAdd(context.Background(), base.AllQueues, qname)
+	seedRedisZSet(tb, r, base.AggregationSetKey(qname, gname, setID), entries, base.TaskStateAggregating)
+}
+
 // SeedAllPendingQueues initializes all of the specified queues with the given messages.
 //
 // pending maps a queue name to a list of messages.
@@ -303,6 +332,18 @@ func SeedAllCompletedQueues(tb testing.TB, r redis.UniversalClient, completed ma
 	}
 }
 
+// SeedAllGroups initializes all groups in all queues.
+// The map maps queue names to group names which maps to a list of task messages and the time it was
+// added to the group.
+func SeedAllGroups(tb testing.TB, r redis.UniversalClient, groups map[string]map[string][]base.Z) {
+	tb.Helper()
+	for qname, g := range groups {
+		for gname, entries := range g {
+			SeedGroup(tb, r, entries, qname, gname)
+		}
+	}
+}
+
 func seedRedisList(tb testing.TB, c redis.UniversalClient, key string,
 	msgs []*base.TaskMessage, state base.TaskState) {
 	tb.Helper()
@@ -311,13 +352,14 @@ func seedRedisList(tb testing.TB, c redis.UniversalClient, key string,
 		if err := c.LPush(context.Background(), key, msg.ID).Err(); err != nil {
 			tb.Fatal(err)
 		}
-		key := base.TaskKey(msg.Queue, msg.ID)
+		taskKey := base.TaskKey(msg.Queue, msg.ID)
 		data := map[string]interface{}{
 			"msg":        encoded,
 			"state":      state.String(),
 			"unique_key": msg.UniqueKey,
+			"group":      msg.GroupKey,
 		}
-		if err := c.HSet(context.Background(), key, data).Err(); err != nil {
+		if err := c.HSet(context.Background(), taskKey, data).Err(); err != nil {
 			tb.Fatal(err)
 		}
 		if len(msg.UniqueKey) > 0 {
@@ -335,17 +377,18 @@ func seedRedisZSet(tb testing.TB, c redis.UniversalClient, key string,
 	for _, item := range items {
 		msg := item.Message
 		encoded := MustMarshal(tb, msg)
-		z := &redis.Z{Member: msg.ID, Score: float64(item.Score)}
+		z := redis.Z{Member: msg.ID, Score: float64(item.Score)}
 		if err := c.ZAdd(context.Background(), key, z).Err(); err != nil {
 			tb.Fatal(err)
 		}
-		key := base.TaskKey(msg.Queue, msg.ID)
+		taskKey := base.TaskKey(msg.Queue, msg.ID)
 		data := map[string]interface{}{
 			"msg":        encoded,
 			"state":      state.String(),
 			"unique_key": msg.UniqueKey,
+			"group":      msg.GroupKey,
 		}
-		if err := c.HSet(context.Background(), key, data).Err(); err != nil {
+		if err := c.HSet(context.Background(), taskKey, data).Err(); err != nil {
 			tb.Fatal(err)
 		}
 		if len(msg.UniqueKey) > 0 {
@@ -434,6 +477,14 @@ func GetCompletedEntries(tb testing.TB, r redis.UniversalClient, qname string) [
 	return getMessagesFromZSetWithScores(tb, r, qname, base.CompletedKey, base.TaskStateCompleted)
 }
 
+// GetGroupEntries returns all scheduled messages and its score in the given queue.
+// It also asserts the state field of the task.
+func GetGroupEntries(tb testing.TB, r redis.UniversalClient, qname, groupKey string) []base.Z {
+	tb.Helper()
+	return getMessagesFromZSetWithScores(tb, r, qname,
+		func(qname string) string { return base.GroupKey(qname, groupKey) }, base.TaskStateAggregating)
+}
+
 // Retrieves all messages stored under `keyFn(qname)` key in redis list.
 func getMessagesFromList(tb testing.TB, r redis.UniversalClient, qname string,
 	keyFn func(qname string) string, state base.TaskState) []*base.TaskMessage {
@@ -485,3 +536,107 @@ func getMessagesFromZSetWithScores(tb testing.TB, r redis.UniversalClient,
 	}
 	return res
 }
+
+// TaskSeedData holds the data required to seed tasks under the task key in test.
+type TaskSeedData struct {
+	Msg          *base.TaskMessage
+	State        base.TaskState
+	PendingSince time.Time
+}
+
+func SeedTasks(tb testing.TB, r redis.UniversalClient, taskData []*TaskSeedData) {
+	for _, data := range taskData {
+		msg := data.Msg
+		ctx := context.Background()
+		key := base.TaskKey(msg.Queue, msg.ID)
+		v := map[string]interface{}{
+			"msg":        MustMarshal(tb, msg),
+			"state":      data.State.String(),
+			"unique_key": msg.UniqueKey,
+			"group":      msg.GroupKey,
+		}
+		if !data.PendingSince.IsZero() {
+			v["pending_since"] = data.PendingSince.Unix()
+		}
+		if err := r.HSet(ctx, key, v).Err(); err != nil {
+			tb.Fatalf("Failed to write task data in redis: %v", err)
+		}
+		if len(msg.UniqueKey) > 0 {
+			err := r.SetNX(ctx, msg.UniqueKey, msg.ID, 1*time.Minute).Err()
+			if err != nil {
+				tb.Fatalf("Failed to set unique lock in redis: %v", err)
+			}
+		}
+	}
+}
+
+func SeedRedisZSets(tb testing.TB, r redis.UniversalClient, zsets map[string][]redis.Z) {
+	for key, zs := range zsets {
+		// FIXME: How come we can't simply do ZAdd(ctx, key, zs...) here?
+		for _, z := range zs {
+			if err := r.ZAdd(context.Background(), key, z).Err(); err != nil {
+				tb.Fatalf("Failed to seed zset (key=%q): %v", key, err)
+			}
+		}
+	}
+}
+
+func SeedRedisSets(tb testing.TB, r redis.UniversalClient, sets map[string][]string) {
+	for key, set := range sets {
+		SeedRedisSet(tb, r, key, set)
+	}
+}
+
+func SeedRedisSet(tb testing.TB, r redis.UniversalClient, key string, members []string) {
+	for _, mem := range members {
+		if err := r.SAdd(context.Background(), key, mem).Err(); err != nil {
+			tb.Fatalf("Failed to seed set (key=%q): %v", key, err)
+		}
+	}
+}
+
+func SeedRedisLists(tb testing.TB, r redis.UniversalClient, lists map[string][]string) {
+	for key, vals := range lists {
+		for _, v := range vals {
+			if err := r.LPush(context.Background(), key, v).Err(); err != nil {
+				tb.Fatalf("Failed to seed list (key=%q): %v", key, err)
+			}
+		}
+	}
+}
+
+func AssertRedisLists(t *testing.T, r redis.UniversalClient, wantLists map[string][]string) {
+	for key, want := range wantLists {
+		got, err := r.LRange(context.Background(), key, 0, -1).Result()
+		if err != nil {
+			t.Fatalf("Failed to read list (key=%q): %v", key, err)
+		}
+		if diff := cmp.Diff(want, got, SortStringSliceOpt); diff != "" {
+			t.Errorf("mismatch found in list (key=%q): (-want,+got)\n%s", key, diff)
+		}
+	}
+}
+
+func AssertRedisSets(t *testing.T, r redis.UniversalClient, wantSets map[string][]string) {
+	for key, want := range wantSets {
+		got, err := r.SMembers(context.Background(), key).Result()
+		if err != nil {
+			t.Fatalf("Failed to read set (key=%q): %v", key, err)
+		}
+		if diff := cmp.Diff(want, got, SortStringSliceOpt); diff != "" {
+			t.Errorf("mismatch found in set (key=%q): (-want,+got)\n%s", key, diff)
+		}
+	}
+}
+
+func AssertRedisZSets(t *testing.T, r redis.UniversalClient, wantZSets map[string][]redis.Z) {
+	for key, want := range wantZSets {
+		got, err := r.ZRangeWithScores(context.Background(), key, 0, -1).Result()
+		if err != nil {
+			t.Fatalf("Failed to read zset (key=%q): %v", key, err)
+		}
+		if diff := cmp.Diff(want, got, SortRedisZSetEntryOpt); diff != "" {
+			t.Errorf("mismatch found in zset (key=%q): (-want,+got)\n%s", key, diff)
+		}
+	}
+}

janitor_test.go

@@ -10,9 +10,9 @@ import (
 	"time"
 
 	"github.com/google/go-cmp/cmp"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 func newCompletedTask(qname, tasktype string, payload []byte, completedAt time.Time) *base.TaskMessage {

processor_test.go

@@ -15,11 +15,11 @@ import (
 
 	"github.com/google/go-cmp/cmp"
 	"github.com/google/go-cmp/cmp/cmpopts"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/errors"
 	"github.com/hibiken/asynq/internal/log"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 	"github.com/hibiken/asynq/internal/timeutil"
 )
 

recoverer.go

@@ -82,11 +82,16 @@ func (r *recoverer) start(wg *sync.WaitGroup) {
 var ErrLeaseExpired = errors.New("asynq: task lease expired")
 
 func (r *recoverer) recover() {
-	// Get all tasks which have expired 30 seconds ago or earlier to accomodate certain amount of clock skew.
+	r.recoverLeaseExpiredTasks()
+	r.recoverStaleAggregationSets()
+}
+
+func (r *recoverer) recoverLeaseExpiredTasks() {
+	// Get all tasks which have expired 30 seconds ago or earlier to accommodate certain amount of clock skew.
 	cutoff := time.Now().Add(-30 * time.Second)
 	msgs, err := r.broker.ListLeaseExpired(cutoff, r.queues...)
 	if err != nil {
-		r.logger.Warn("recoverer: could not list lease expired tasks")
+		r.logger.Warnf("recoverer: could not list lease expired tasks: %v", err)
 		return
 	}
 	for _, msg := range msgs {
@@ -98,6 +103,14 @@ func (r *recoverer) recover() {
 	}
 }
 
+func (r *recoverer) recoverStaleAggregationSets() {
+	for _, qname := range r.queues {
+		if err := r.broker.ReclaimStaleAggregationSets(qname); err != nil {
+			r.logger.Warnf("recoverer: could not reclaim stale aggregation sets in queue %q: %v", qname, err)
+		}
+	}
+}
+
 func (r *recoverer) retry(msg *base.TaskMessage, err error) {
 	delay := r.retryDelayFunc(msg.Retried, err, NewTask(msg.Type, msg.Payload))
 	retryAt := time.Now().Add(delay)

recoverer_test.go

@@ -10,9 +10,9 @@ import (
 	"time"
 
 	"github.com/google/go-cmp/cmp"
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 func TestRecoverer(t *testing.T) {

scheduler.go

@@ -10,7 +10,7 @@ import (
 	"sync"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/google/uuid"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/log"
@@ -26,14 +26,16 @@ type Scheduler struct {
 
 	state *serverState
 
-	logger     *log.Logger
-	client     *Client
-	rdb        *rdb.RDB
-	cron       *cron.Cron
-	location   *time.Location
-	done       chan struct{}
-	wg         sync.WaitGroup
-	errHandler func(task *Task, opts []Option, err error)
+	logger          *log.Logger
+	client          *Client
+	rdb             *rdb.RDB
+	cron            *cron.Cron
+	location        *time.Location
+	done            chan struct{}
+	wg              sync.WaitGroup
+	preEnqueueFunc  func(task *Task, opts []Option)
+	postEnqueueFunc func(info *TaskInfo, err error)
+	errHandler      func(task *Task, opts []Option, err error)
 
 	// guards idmap
 	mu sync.Mutex
@@ -67,16 +69,18 @@ func NewScheduler(r RedisConnOpt, opts *SchedulerOpts) *Scheduler {
 	}
 
 	return &Scheduler{
-		id:         generateSchedulerID(),
-		state:      &serverState{value: srvStateNew},
-		logger:     logger,
-		client:     NewClient(r),
-		rdb:        rdb.NewRDB(c),
-		cron:       cron.New(cron.WithLocation(loc)),
-		location:   loc,
-		done:       make(chan struct{}),
-		errHandler: opts.EnqueueErrorHandler,
-		idmap:      make(map[string]cron.EntryID),
+		id:              generateSchedulerID(),
+		state:           &serverState{value: srvStateNew},
+		logger:          logger,
+		client:          NewClient(r),
+		rdb:             rdb.NewRDB(c),
+		cron:            cron.New(cron.WithLocation(loc)),
+		location:        loc,
+		done:            make(chan struct{}),
+		preEnqueueFunc:  opts.PreEnqueueFunc,
+		postEnqueueFunc: opts.PostEnqueueFunc,
+		errHandler:      opts.EnqueueErrorHandler,
+		idmap:           make(map[string]cron.EntryID),
 	}
 }
 
@@ -105,28 +109,44 @@ type SchedulerOpts struct {
 	// If unset, the UTC time zone (time.UTC) is used.
 	Location *time.Location
 
+	// PreEnqueueFunc, if provided, is called before a task gets enqueued by Scheduler.
+	// The callback function should return quickly to not block the current thread.
+	PreEnqueueFunc func(task *Task, opts []Option)
+
+	// PostEnqueueFunc, if provided, is called after a task gets enqueued by Scheduler.
+	// The callback function should return quickly to not block the current thread.
+	PostEnqueueFunc func(info *TaskInfo, err error)
+
+	// Deprecated: Use PostEnqueueFunc instead
 	// EnqueueErrorHandler gets called when scheduler cannot enqueue a registered task
 	// due to an error.
 	EnqueueErrorHandler func(task *Task, opts []Option, err error)
 }
 
-// enqueueJob encapsulates the job of enqueing a task and recording the event.
+// enqueueJob encapsulates the job of enqueuing a task and recording the event.
 type enqueueJob struct {
-	id         uuid.UUID
-	cronspec   string
-	task       *Task
-	opts       []Option
-	location   *time.Location
-	logger     *log.Logger
-	client     *Client
-	rdb        *rdb.RDB
-	errHandler func(task *Task, opts []Option, err error)
+	id              uuid.UUID
+	cronspec        string
+	task            *Task
+	opts            []Option
+	location        *time.Location
+	logger          *log.Logger
+	client          *Client
+	rdb             *rdb.RDB
+	preEnqueueFunc  func(task *Task, opts []Option)
+	postEnqueueFunc func(info *TaskInfo, err error)
+	errHandler      func(task *Task, opts []Option, err error)
 }
 
 func (j *enqueueJob) Run() {
+	if j.preEnqueueFunc != nil {
+		j.preEnqueueFunc(j.task, j.opts)
+	}
 	info, err := j.client.Enqueue(j.task, j.opts...)
+	if j.postEnqueueFunc != nil {
+		j.postEnqueueFunc(info, err)
+	}
 	if err != nil {
-		j.logger.Errorf("scheduler could not enqueue a task %+v: %v", j.task, err)
 		if j.errHandler != nil {
 			j.errHandler(j.task, j.opts, err)
 		}
@@ -139,7 +159,7 @@ func (j *enqueueJob) Run() {
 	}
 	err = j.rdb.RecordSchedulerEnqueueEvent(j.id.String(), event)
 	if err != nil {
-		j.logger.Errorf("scheduler could not record enqueue event of enqueued task %+v: %v", j.task, err)
+		j.logger.Warnf("scheduler could not record enqueue event of enqueued task %s: %v", info.ID, err)
 	}
 }
 
@@ -147,15 +167,17 @@ func (j *enqueueJob) Run() {
 // It returns an ID of the newly registered entry.
 func (s *Scheduler) Register(cronspec string, task *Task, opts ...Option) (entryID string, err error) {
 	job := &enqueueJob{
-		id:         uuid.New(),
-		cronspec:   cronspec,
-		task:       task,
-		opts:       opts,
-		location:   s.location,
-		client:     s.client,
-		rdb:        s.rdb,
-		logger:     s.logger,
-		errHandler: s.errHandler,
+		id:              uuid.New(),
+		cronspec:        cronspec,
+		task:            task,
+		opts:            opts,
+		location:        s.location,
+		client:          s.client,
+		rdb:             s.rdb,
+		logger:          s.logger,
+		preEnqueueFunc:  s.preEnqueueFunc,
+		postEnqueueFunc: s.postEnqueueFunc,
+		errHandler:      s.errHandler,
 	}
 	cronID, err := s.cron.AddJob(cronspec, job)
 	if err != nil {
@@ -252,6 +274,7 @@ func (s *Scheduler) runHeartbeater() {
 		case <-s.done:
 			s.logger.Debugf("Scheduler heatbeater shutting down")
 			s.rdb.ClearSchedulerEntries(s.id)
+			ticker.Stop()
 			return
 		case <-ticker.C:
 			s.beat()

scheduler_test.go

@@ -10,8 +10,9 @@ import (
 	"time"
 
 	"github.com/google/go-cmp/cmp"
-	"github.com/hibiken/asynq/internal/asynqtest"
+
 	"github.com/hibiken/asynq/internal/base"
+	"github.com/hibiken/asynq/internal/testutil"
 )
 
 func TestSchedulerRegister(t *testing.T) {
@@ -69,8 +70,8 @@ func TestSchedulerRegister(t *testing.T) {
 		time.Sleep(tc.wait)
 		scheduler.Shutdown()
 
-		got := asynqtest.GetPendingMessages(t, r, tc.queue)
-		if diff := cmp.Diff(tc.want, got, asynqtest.IgnoreIDOpt); diff != "" {
+		got := testutil.GetPendingMessages(t, r, tc.queue)
+		if diff := cmp.Diff(tc.want, got, testutil.IgnoreIDOpt); diff != "" {
 			t.Errorf("mismatch found in queue %q: (-want,+got)\n%s", tc.queue, diff)
 		}
 	}
@@ -148,9 +149,62 @@ func TestSchedulerUnregister(t *testing.T) {
 		time.Sleep(tc.wait)
 		scheduler.Shutdown()
 
-		got := asynqtest.GetPendingMessages(t, r, tc.queue)
+		got := testutil.GetPendingMessages(t, r, tc.queue)
 		if len(got) != 0 {
 			t.Errorf("%d tasks were enqueued, want zero", len(got))
 		}
 	}
 }
+
+func TestSchedulerPostAndPreEnqueueHandler(t *testing.T) {
+	var (
+		preMu       sync.Mutex
+		preCounter  int
+		postMu      sync.Mutex
+		postCounter int
+	)
+	preHandler := func(task *Task, opts []Option) {
+		preMu.Lock()
+		preCounter++
+		preMu.Unlock()
+	}
+	postHandler := func(info *TaskInfo, err error) {
+		postMu.Lock()
+		postCounter++
+		postMu.Unlock()
+	}
+
+	// Connect to non-existent redis instance to simulate a redis server being down.
+	scheduler := NewScheduler(
+		getRedisConnOpt(t),
+		&SchedulerOpts{
+			PreEnqueueFunc:  preHandler,
+			PostEnqueueFunc: postHandler,
+		},
+	)
+
+	task := NewTask("test", nil)
+
+	if _, err := scheduler.Register("@every 3s", task); err != nil {
+		t.Fatal(err)
+	}
+
+	if err := scheduler.Start(); err != nil {
+		t.Fatal(err)
+	}
+	// Scheduler should attempt to enqueue the task three times (every 3s).
+	time.Sleep(10 * time.Second)
+	scheduler.Shutdown()
+
+	preMu.Lock()
+	if preCounter != 3 {
+		t.Errorf("PreEnqueueFunc was called %d times, want 3", preCounter)
+	}
+	preMu.Unlock()
+
+	postMu.Lock()
+	if postCounter != 3 {
+		t.Errorf("PostEnqueueFunc was called %d times, want 3", postCounter)
+	}
+	postMu.Unlock()
+}

server.go

@@ -15,7 +15,7 @@ import (
 	"sync"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/log"
 	"github.com/hibiken/asynq/internal/rdb"
@@ -50,6 +50,7 @@ type Server struct {
 	recoverer     *recoverer
 	healthchecker *healthchecker
 	janitor       *janitor
+	aggregator    *aggregator
 }
 
 type serverState struct {
@@ -193,9 +194,55 @@ type Config struct {
 	//
 	// If unset or zero, the interval is set to 5 seconds.
 	DelayedTaskCheckInterval time.Duration
+
+	// GroupGracePeriod specifies the amount of time the server will wait for an incoming task before aggregating
+	// the tasks in a group. If an incoming task is received within this period, the server will wait for another
+	// period of the same length, up to GroupMaxDelay if specified.
+	//
+	// If unset or zero, the grace period is set to 1 minute.
+	// Minimum duration for GroupGracePeriod is 1 second. If value specified is less than a second, the call to
+	// NewServer will panic.
+	GroupGracePeriod time.Duration
+
+	// GroupMaxDelay specifies the maximum amount of time the server will wait for incoming tasks before aggregating
+	// the tasks in a group.
+	//
+	// If unset or zero, no delay limit is used.
+	GroupMaxDelay time.Duration
+
+	// GroupMaxSize specifies the maximum number of tasks that can be aggregated into a single task within a group.
+	// If GroupMaxSize is reached, the server will aggregate the tasks into one immediately.
+	//
+	// If unset or zero, no size limit is used.
+	GroupMaxSize int
+
+	// GroupAggregator specifies the aggregation function used to aggregate multiple tasks in a group into one task.
+	//
+	// If unset or nil, the group aggregation feature will be disabled on the server.
+	GroupAggregator GroupAggregator
 }
 
-// An ErrorHandler handles an error occured during task processing.
+// GroupAggregator aggregates a group of tasks into one before the tasks are passed to the Handler.
+type GroupAggregator interface {
+	// Aggregate aggregates the given tasks in a group with the given group name,
+	// and returns a new task which is the aggregation of those tasks.
+	//
+	// Use NewTask(typename, payload, opts...) to set any options for the aggregated task.
+	// The Queue option, if provided, will be ignored and the aggregated task will always be enqueued
+	// to the same queue the group belonged.
+	Aggregate(group string, tasks []*Task) *Task
+}
+
+// The GroupAggregatorFunc type is an adapter to allow the use of  ordinary functions as a GroupAggregator.
+// If f is a function with the appropriate signature, GroupAggregatorFunc(f) is a GroupAggregator that calls f.
+type GroupAggregatorFunc func(group string, tasks []*Task) *Task
+
+// Aggregate calls fn(group, tasks)
+func (fn GroupAggregatorFunc) Aggregate(group string, tasks []*Task) *Task {
+	return fn(group, tasks)
+}
+
+// An ErrorHandler handles an error occurred during task processing.
 type ErrorHandler interface {
 	HandleError(ctx context.Context, task *Task, err error)
 }
@@ -338,6 +385,8 @@ const (
 	defaultHealthCheckInterval = 15 * time.Second
 
 	defaultDelayedTaskCheckInterval = 5 * time.Second
+
+	defaultGroupGracePeriod = 1 * time.Minute
 )
 
 // NewServer returns a new Server given a redis connection option
@@ -387,6 +436,14 @@ func NewServer(r RedisConnOpt, cfg Config) *Server {
 	if healthcheckInterval == 0 {
 		healthcheckInterval = defaultHealthCheckInterval
 	}
+	// TODO: Create a helper to check for zero value and fall back to default (e.g. getDurationOrDefault())
+	groupGracePeriod := cfg.GroupGracePeriod
+	if groupGracePeriod == 0 {
+		groupGracePeriod = defaultGroupGracePeriod
+	}
+	if groupGracePeriod < time.Second {
+		panic("GroupGracePeriod cannot be less than a second")
+	}
 	logger := log.NewLogger(cfg.Logger)
 	loglevel := cfg.LogLevel
 	if loglevel == level_unspecified {
@@ -468,6 +525,15 @@ func NewServer(r RedisConnOpt, cfg Config) *Server {
 		queues:   qnames,
 		interval: 8 * time.Second,
 	})
+	aggregator := newAggregator(aggregatorParams{
+		logger:          logger,
+		broker:          rdb,
+		queues:          qnames,
+		gracePeriod:     groupGracePeriod,
+		maxDelay:        cfg.GroupMaxDelay,
+		maxSize:         cfg.GroupMaxSize,
+		groupAggregator: cfg.GroupAggregator,
+	})
 	return &Server{
 		logger:        logger,
 		broker:        rdb,
@@ -480,6 +546,7 @@ func NewServer(r RedisConnOpt, cfg Config) *Server {
 		recoverer:     recoverer,
 		healthchecker: healthchecker,
 		janitor:       janitor,
+		aggregator:    aggregator,
 	}
 }
 
@@ -556,6 +623,7 @@ func (srv *Server) Start(handler Handler) error {
 	srv.forwarder.start(&srv.wg)
 	srv.processor.start(&srv.wg)
 	srv.janitor.start(&srv.wg)
+	srv.aggregator.start(&srv.wg)
 	return nil
 }
 
@@ -601,6 +669,7 @@ func (srv *Server) Shutdown() {
 	srv.syncer.shutdown()
 	srv.subscriber.shutdown()
 	srv.janitor.shutdown()
+	srv.aggregator.shutdown()
 	srv.healthchecker.shutdown()
 	srv.heartbeater.shutdown()
 	srv.wg.Wait()

server_test.go

@@ -11,16 +11,16 @@ import (
 	"testing"
 	"time"
 
-	"github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/rdb"
 	"github.com/hibiken/asynq/internal/testbroker"
+	"github.com/hibiken/asynq/internal/testutil"
 	"go.uber.org/goleak"
 )
 
 func TestServer(t *testing.T) {
 	// https://github.com/go-redis/redis/issues/1029
-	ignoreOpt := goleak.IgnoreTopFunction("github.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper")
-	defer goleak.VerifyNoLeaks(t, ignoreOpt)
+	ignoreOpt := goleak.IgnoreTopFunction("github.com/redis/go-redis/v9/internal/pool.(*ConnPool).reaper")
+	defer goleak.VerifyNone(t, ignoreOpt)
 
 	redisConnOpt := getRedisConnOpt(t)
 	c := NewClient(redisConnOpt)
@@ -40,12 +40,12 @@ func TestServer(t *testing.T) {
 		t.Fatal(err)
 	}
 
-	_, err = c.Enqueue(NewTask("send_email", asynqtest.JSON(map[string]interface{}{"recipient_id": 123})))
+	_, err = c.Enqueue(NewTask("send_email", testutil.JSON(map[string]interface{}{"recipient_id": 123})))
 	if err != nil {
 		t.Errorf("could not enqueue a task: %v", err)
 	}
 
-	_, err = c.Enqueue(NewTask("send_email", asynqtest.JSON(map[string]interface{}{"recipient_id": 456})), ProcessIn(1*time.Hour))
+	_, err = c.Enqueue(NewTask("send_email", testutil.JSON(map[string]interface{}{"recipient_id": 456})), ProcessIn(1*time.Hour))
 	if err != nil {
 		t.Errorf("could not enqueue a task: %v", err)
 	}
@@ -55,8 +55,8 @@ func TestServer(t *testing.T) {
 
 func TestServerRun(t *testing.T) {
 	// https://github.com/go-redis/redis/issues/1029
-	ignoreOpt := goleak.IgnoreTopFunction("github.com/go-redis/redis/v8/internal/pool.(*ConnPool).reaper")
-	defer goleak.VerifyNoLeaks(t, ignoreOpt)
+	ignoreOpt := goleak.IgnoreTopFunction("github.com/redis/go-redis/v9/internal/pool.(*ConnPool).reaper")
+	defer goleak.VerifyNone(t, ignoreOpt)
 
 	srv := NewServer(RedisClientOpt{Addr: ":6379"}, Config{LogLevel: testLogLevel})
 

subscriber.go

@@ -8,7 +8,7 @@ import (
 	"sync"
 	"time"
 
-	"github.com/go-redis/redis/v8"
+	"github.com/redis/go-redis/v9"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/log"
 )

syncer_test.go

@@ -11,9 +11,9 @@ import (
 	"testing"
 	"time"
 
-	h "github.com/hibiken/asynq/internal/asynqtest"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
+	h "github.com/hibiken/asynq/internal/testutil"
 )
 
 func TestSyncer(t *testing.T) {

tools/asynq/README.md

@@ -12,7 +12,7 @@ Asynq CLI is a command line tool to monitor the queues and tasks managed by `asy
 
 In order to use the tool, compile it using the following command:
 
-    go get github.com/hibiken/asynq/tools/asynq
+    go install github.com/hibiken/asynq/tools/asynq
 
 This will create the asynq executable under your `$GOPATH/bin` directory.
 
@@ -22,6 +22,7 @@ This will create the asynq executable under your `$GOPATH/bin` directory.
 
 To view details on any command, use `asynq help <command> <subcommand>`.
 
+- `asynq dash`
 - `asynq stats`
 - `asynq queue [ls inspect history rm pause unpause]`
 - `asynq task [ls cancel delete archive run delete-all archive-all run-all]`

tools/asynq/cmd/cron.go

@@ -11,6 +11,7 @@ import (
 	"sort"
 	"time"
 
+	"github.com/MakeNowJust/heredoc/v2"
 	"github.com/hibiken/asynq"
 	"github.com/spf13/cobra"
 )
@@ -24,21 +25,30 @@ func init() {
 }
 
 var cronCmd = &cobra.Command{
-	Use:   "cron",
+	Use:   "cron <command> [flags]",
 	Short: "Manage cron",
+	Example: heredoc.Doc(`
+		$ asynq cron ls
+		$ asynq cron history 7837f142-6337-4217-9276-8f27281b67d1`),
 }
 
 var cronListCmd = &cobra.Command{
-	Use:   "ls",
-	Short: "List cron entries",
-	Run:   cronList,
+	Use:     "list",
+	Aliases: []string{"ls"},
+	Short:   "List cron entries",
+	Run:     cronList,
 }
 
 var cronHistoryCmd = &cobra.Command{
-	Use:   "history [ENTRY_ID...]",
+	Use:   "history <entry_id> [<entry_id>...]",
 	Short: "Show history of each cron tasks",
 	Args:  cobra.MinimumNArgs(1),
 	Run:   cronHistory,
+	Example: heredoc.Doc(`
+		$ asynq cron history 7837f142-6337-4217-9276-8f27281b67d1
+		$ asynq cron history 7837f142-6337-4217-9276-8f27281b67d1 bf6a8594-cd03-4968-b36a-8572c5e160dd
+		$ asynq cron history 7837f142-6337-4217-9276-8f27281b67d1 --size=100
+		$ asynq cron history 7837f142-6337-4217-9276-8f27281b67d1 --page=2`),
 }
 
 func cronList(cmd *cobra.Command, args []string) {

tools/asynq/cmd/dash.go

@@ -0,0 +1,45 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package cmd
+
+import (
+	"fmt"
+	"os"
+	"time"
+
+	"github.com/MakeNowJust/heredoc/v2"
+	"github.com/hibiken/asynq/tools/asynq/cmd/dash"
+	"github.com/spf13/cobra"
+)
+
+var (
+	flagPollInterval = 8 * time.Second
+)
+
+func init() {
+	rootCmd.AddCommand(dashCmd)
+	dashCmd.Flags().DurationVar(&flagPollInterval, "refresh", 8*time.Second, "Interval between data refresh (default: 8s, min allowed: 1s)")
+}
+
+var dashCmd = &cobra.Command{
+	Use:   "dash",
+	Short: "View dashboard",
+	Long: heredoc.Doc(`
+		Display interactive dashboard.`),
+	Args: cobra.NoArgs,
+	Example: heredoc.Doc(`
+        $ asynq dash
+        $ asynq dash --refresh=3s`),
+	Run: func(cmd *cobra.Command, args []string) {
+		if flagPollInterval < 1*time.Second {
+			fmt.Println("error: --refresh cannot be less than 1s")
+			os.Exit(1)
+		}
+		dash.Run(dash.Options{
+			PollInterval: flagPollInterval,
+			RedisConnOpt: getRedisConnOpt(),
+		})
+	},
+}

tools/asynq/cmd/dash/dash.go

@@ -0,0 +1,220 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"errors"
+	"fmt"
+	"os"
+	"strings"
+	"time"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/hibiken/asynq"
+)
+
+// viewType is an enum for dashboard views.
+type viewType int
+
+const (
+	viewTypeQueues viewType = iota
+	viewTypeQueueDetails
+	viewTypeHelp
+)
+
+// State holds dashboard state.
+type State struct {
+	queues []*asynq.QueueInfo
+	tasks  []*asynq.TaskInfo
+	groups []*asynq.GroupInfo
+	err    error
+
+	// Note: index zero corresponds to the table header; index=1 correctponds to the first element
+	queueTableRowIdx int             // highlighted row in queue table
+	taskTableRowIdx  int             // highlighted row in task table
+	groupTableRowIdx int             // highlighted row in group table
+	taskState        asynq.TaskState // highlighted task state in queue details view
+	taskID           string          // selected task ID
+
+	selectedQueue *asynq.QueueInfo // queue shown on queue details view
+	selectedGroup *asynq.GroupInfo
+	selectedTask  *asynq.TaskInfo
+
+	pageNum int // pagination page number
+
+	view     viewType // current view type
+	prevView viewType // to support "go back"
+}
+
+func (s *State) DebugString() string {
+	var b strings.Builder
+	b.WriteString(fmt.Sprintf("len(queues)=%d ", len(s.queues)))
+	b.WriteString(fmt.Sprintf("len(tasks)=%d ", len(s.tasks)))
+	b.WriteString(fmt.Sprintf("len(groups)=%d ", len(s.groups)))
+	b.WriteString(fmt.Sprintf("err=%v ", s.err))
+
+	if s.taskState != 0 {
+		b.WriteString(fmt.Sprintf("taskState=%s ", s.taskState.String()))
+	} else {
+		b.WriteString(fmt.Sprintf("taskState=0"))
+	}
+	b.WriteString(fmt.Sprintf("taskID=%s ", s.taskID))
+
+	b.WriteString(fmt.Sprintf("queueTableRowIdx=%d ", s.queueTableRowIdx))
+	b.WriteString(fmt.Sprintf("taskTableRowIdx=%d ", s.taskTableRowIdx))
+	b.WriteString(fmt.Sprintf("groupTableRowIdx=%d ", s.groupTableRowIdx))
+
+	if s.selectedQueue != nil {
+		b.WriteString(fmt.Sprintf("selectedQueue={Queue:%s} ", s.selectedQueue.Queue))
+	} else {
+		b.WriteString("selectedQueue=nil ")
+	}
+
+	if s.selectedGroup != nil {
+		b.WriteString(fmt.Sprintf("selectedGroup={Group:%s} ", s.selectedGroup.Group))
+	} else {
+		b.WriteString("selectedGroup=nil ")
+	}
+
+	if s.selectedTask != nil {
+		b.WriteString(fmt.Sprintf("selectedTask={ID:%s} ", s.selectedTask.ID))
+	} else {
+		b.WriteString("selectedTask=nil ")
+	}
+
+	b.WriteString(fmt.Sprintf("pageNum=%d", s.pageNum))
+	return b.String()
+}
+
+type Options struct {
+	DebugMode    bool
+	PollInterval time.Duration
+	RedisConnOpt asynq.RedisConnOpt
+}
+
+func Run(opts Options) {
+	s, err := tcell.NewScreen()
+	if err != nil {
+		fmt.Printf("failed to create a screen: %v\n", err)
+		os.Exit(1)
+	}
+	if err := s.Init(); err != nil {
+		fmt.Printf("failed to initialize screen: %v\n", err)
+		os.Exit(1)
+	}
+	s.SetStyle(baseStyle) // set default text style
+
+	var (
+		state = State{} // confined in this goroutine only; DO NOT SHARE
+
+		inspector = asynq.NewInspector(opts.RedisConnOpt)
+		ticker    = time.NewTicker(opts.PollInterval)
+
+		eventCh = make(chan tcell.Event)
+		done    = make(chan struct{})
+
+		// channels to send/receive data fetched asynchronously
+		errorCh  = make(chan error)
+		queueCh  = make(chan *asynq.QueueInfo)
+		taskCh   = make(chan *asynq.TaskInfo)
+		queuesCh = make(chan []*asynq.QueueInfo)
+		groupsCh = make(chan []*asynq.GroupInfo)
+		tasksCh  = make(chan []*asynq.TaskInfo)
+	)
+	defer ticker.Stop()
+
+	f := dataFetcher{
+		inspector,
+		opts,
+		s,
+		errorCh,
+		queueCh,
+		taskCh,
+		queuesCh,
+		groupsCh,
+		tasksCh,
+	}
+
+	d := dashDrawer{
+		s,
+		opts,
+	}
+
+	h := keyEventHandler{
+		s:            s,
+		fetcher:      &f,
+		drawer:       &d,
+		state:        &state,
+		done:         done,
+		ticker:       ticker,
+		pollInterval: opts.PollInterval,
+	}
+
+	go fetchQueues(inspector, queuesCh, errorCh, opts)
+	go s.ChannelEvents(eventCh, done) // TODO: Double check that we are not leaking goroutine with this one.
+	d.Draw(&state)                    // draw initial screen
+
+	for {
+		// Update screen
+		s.Show()
+
+		select {
+		case ev := <-eventCh:
+			// Process event
+			switch ev := ev.(type) {
+			case *tcell.EventResize:
+				s.Sync()
+			case *tcell.EventKey:
+				h.HandleKeyEvent(ev)
+			}
+
+		case <-ticker.C:
+			f.Fetch(&state)
+
+		case queues := <-queuesCh:
+			state.queues = queues
+			state.err = nil
+			if len(queues) < state.queueTableRowIdx {
+				state.queueTableRowIdx = len(queues)
+			}
+			d.Draw(&state)
+
+		case q := <-queueCh:
+			state.selectedQueue = q
+			state.err = nil
+			d.Draw(&state)
+
+		case groups := <-groupsCh:
+			state.groups = groups
+			state.err = nil
+			if len(groups) < state.groupTableRowIdx {
+				state.groupTableRowIdx = len(groups)
+			}
+			d.Draw(&state)
+
+		case tasks := <-tasksCh:
+			state.tasks = tasks
+			state.err = nil
+			if len(tasks) < state.taskTableRowIdx {
+				state.taskTableRowIdx = len(tasks)
+			}
+			d.Draw(&state)
+
+		case t := <-taskCh:
+			state.selectedTask = t
+			state.err = nil
+			d.Draw(&state)
+
+		case err := <-errorCh:
+			if errors.Is(err, asynq.ErrTaskNotFound) {
+				state.selectedTask = nil
+			} else {
+				state.err = err
+			}
+			d.Draw(&state)
+		}
+	}
+
+}

tools/asynq/cmd/dash/draw.go

@@ -0,0 +1,724 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"fmt"
+	"math"
+	"strconv"
+	"strings"
+	"time"
+	"unicode"
+	"unicode/utf8"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/hibiken/asynq"
+	"github.com/mattn/go-runewidth"
+)
+
+var (
+	baseStyle  = tcell.StyleDefault.Background(tcell.ColorReset).Foreground(tcell.ColorReset)
+	labelStyle = baseStyle.Foreground(tcell.ColorLightGray)
+
+	// styles for bar graph
+	activeStyle      = baseStyle.Foreground(tcell.ColorBlue)
+	pendingStyle     = baseStyle.Foreground(tcell.ColorGreen)
+	aggregatingStyle = baseStyle.Foreground(tcell.ColorLightGreen)
+	scheduledStyle   = baseStyle.Foreground(tcell.ColorYellow)
+	retryStyle       = baseStyle.Foreground(tcell.ColorPink)
+	archivedStyle    = baseStyle.Foreground(tcell.ColorPurple)
+	completedStyle   = baseStyle.Foreground(tcell.ColorDarkGreen)
+)
+
+// drawer draws UI with the given state.
+type drawer interface {
+	Draw(state *State)
+}
+
+type dashDrawer struct {
+	s    tcell.Screen
+	opts Options
+}
+
+func (dd *dashDrawer) Draw(state *State) {
+	s, opts := dd.s, dd.opts
+	s.Clear()
+	// Simulate data update on every render
+	d := NewScreenDrawer(s)
+	switch state.view {
+	case viewTypeQueues:
+		d.Println("=== Queues ===", baseStyle.Bold(true))
+		d.NL()
+		drawQueueSizeGraphs(d, state)
+		d.NL()
+		drawQueueTable(d, baseStyle, state)
+	case viewTypeQueueDetails:
+		d.Println("=== Queue Summary ===", baseStyle.Bold(true))
+		d.NL()
+		drawQueueSummary(d, state)
+		d.NL()
+		d.NL()
+		d.Println("=== Tasks ===", baseStyle.Bold(true))
+		d.NL()
+		drawTaskStateBreakdown(d, baseStyle, state)
+		d.NL()
+		drawTaskTable(d, state)
+		drawTaskModal(d, state)
+	case viewTypeHelp:
+		drawHelp(d)
+	}
+	d.GoToBottom()
+	if opts.DebugMode {
+		drawDebugInfo(d, state)
+	} else {
+		drawFooter(d, state)
+	}
+}
+
+func drawQueueSizeGraphs(d *ScreenDrawer, state *State) {
+	var qnames []string
+	var qsizes []string // queue size in strings
+	maxSize := 1        // not zero to avoid division by zero
+	for _, q := range state.queues {
+		qnames = append(qnames, q.Queue)
+		qsizes = append(qsizes, strconv.Itoa(q.Size))
+		if q.Size > maxSize {
+			maxSize = q.Size
+		}
+	}
+	qnameWidth := maxwidth(qnames)
+	qsizeWidth := maxwidth(qsizes)
+
+	// Calculate the multipler to scale the graph
+	screenWidth, _ := d.Screen().Size()
+	graphMaxWidth := screenWidth - (qnameWidth + qsizeWidth + 3) // <qname> |<graph> <size>
+	multipiler := 1.0
+	if graphMaxWidth < maxSize {
+		multipiler = float64(graphMaxWidth) / float64(maxSize)
+	}
+
+	const tick = '▇'
+	for _, q := range state.queues {
+		d.Print(q.Queue, baseStyle)
+		d.Print(strings.Repeat(" ", qnameWidth-runewidth.StringWidth(q.Queue)+1), baseStyle) // padding between qname and graph
+		d.Print("|", baseStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Active)*multipiler))), activeStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Pending)*multipiler))), pendingStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Aggregating)*multipiler))), aggregatingStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Scheduled)*multipiler))), scheduledStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Retry)*multipiler))), retryStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Archived)*multipiler))), archivedStyle)
+		d.Print(strings.Repeat(string(tick), int(math.Floor(float64(q.Completed)*multipiler))), completedStyle)
+		d.Print(fmt.Sprintf(" %d", q.Size), baseStyle)
+		d.NL()
+	}
+	d.NL()
+	d.Print("active=", baseStyle)
+	d.Print(string(tick), activeStyle)
+	d.Print(" pending=", baseStyle)
+	d.Print(string(tick), pendingStyle)
+	d.Print(" aggregating=", baseStyle)
+	d.Print(string(tick), aggregatingStyle)
+	d.Print(" scheduled=", baseStyle)
+	d.Print(string(tick), scheduledStyle)
+	d.Print(" retry=", baseStyle)
+	d.Print(string(tick), retryStyle)
+	d.Print(" archived=", baseStyle)
+	d.Print(string(tick), archivedStyle)
+	d.Print(" completed=", baseStyle)
+	d.Print(string(tick), completedStyle)
+	d.NL()
+}
+
+func drawFooter(d *ScreenDrawer, state *State) {
+	if state.err != nil {
+		style := baseStyle.Background(tcell.ColorDarkRed)
+		d.Print(state.err.Error(), style)
+		d.FillLine(' ', style)
+		return
+	}
+	style := baseStyle.Background(tcell.ColorDarkSlateGray).Foreground(tcell.ColorWhite)
+	switch state.view {
+	case viewTypeHelp:
+		d.Print("<Esc>: GoBack", style)
+	default:
+		d.Print("<?>: Help    <Ctrl+C>: Exit ", style)
+	}
+	d.FillLine(' ', style)
+}
+
+// returns the maximum width from the given list of names
+func maxwidth(names []string) int {
+	max := 0
+	for _, s := range names {
+		if w := runewidth.StringWidth(s); w > max {
+			max = w
+		}
+	}
+	return max
+}
+
+// rpad adds padding to the right of a string.
+func rpad(s string, padding int) string {
+	tmpl := fmt.Sprintf("%%-%ds ", padding)
+	return fmt.Sprintf(tmpl, s)
+
+}
+
+// lpad adds padding to the left of a string.
+func lpad(s string, padding int) string {
+	tmpl := fmt.Sprintf("%%%ds ", padding)
+	return fmt.Sprintf(tmpl, s)
+}
+
+// byteCount converts the given bytes into human readable string
+func byteCount(b int64) string {
+	const unit = 1000
+	if b < unit {
+		return fmt.Sprintf("%d B", b)
+
+	}
+	div, exp := int64(unit), 0
+	for n := b / unit; n >= unit; n /= unit {
+		div *= unit
+		exp++
+
+	}
+	return fmt.Sprintf("%.1f %cB", float64(b)/float64(div), "kMGTPE"[exp])
+}
+
+var queueColumnConfigs = []*columnConfig[*asynq.QueueInfo]{
+	{"Queue", alignLeft, func(q *asynq.QueueInfo) string { return q.Queue }},
+	{"State", alignLeft, func(q *asynq.QueueInfo) string { return formatQueueState(q) }},
+	{"Size", alignRight, func(q *asynq.QueueInfo) string { return strconv.Itoa(q.Size) }},
+	{"Latency", alignRight, func(q *asynq.QueueInfo) string { return q.Latency.Round(time.Second).String() }},
+	{"MemoryUsage", alignRight, func(q *asynq.QueueInfo) string { return byteCount(q.MemoryUsage) }},
+	{"Processed", alignRight, func(q *asynq.QueueInfo) string { return strconv.Itoa(q.Processed) }},
+	{"Failed", alignRight, func(q *asynq.QueueInfo) string { return strconv.Itoa(q.Failed) }},
+	{"ErrorRate", alignRight, func(q *asynq.QueueInfo) string { return formatErrorRate(q.Processed, q.Failed) }},
+}
+
+func formatQueueState(q *asynq.QueueInfo) string {
+	if q.Paused {
+		return "PAUSED"
+	}
+	return "RUN"
+}
+
+func formatErrorRate(processed, failed int) string {
+	if processed == 0 {
+		return "-"
+	}
+	return fmt.Sprintf("%.2f", float64(failed)/float64(processed))
+}
+
+func formatNextProcessTime(t time.Time) string {
+	now := time.Now()
+	if t.Before(now) {
+		return "now"
+	}
+	return fmt.Sprintf("in %v", (t.Sub(now).Round(time.Second)))
+}
+
+func formatPastTime(t time.Time) string {
+	now := time.Now()
+	if t.After(now) || t.Equal(now) {
+		return "just now"
+	}
+	return fmt.Sprintf("%v ago", time.Since(t).Round(time.Second))
+}
+
+func drawQueueTable(d *ScreenDrawer, style tcell.Style, state *State) {
+	drawTable(d, style, queueColumnConfigs, state.queues, state.queueTableRowIdx-1)
+}
+
+func drawQueueSummary(d *ScreenDrawer, state *State) {
+	q := state.selectedQueue
+	if q == nil {
+		d.Println("ERROR: Press q to go back", baseStyle)
+		return
+	}
+	d.Print("Name      ", labelStyle)
+	d.Println(q.Queue, baseStyle)
+	d.Print("Size      ", labelStyle)
+	d.Println(strconv.Itoa(q.Size), baseStyle)
+	d.Print("Latency   ", labelStyle)
+	d.Println(q.Latency.Round(time.Second).String(), baseStyle)
+	d.Print("MemUsage  ", labelStyle)
+	d.Println(byteCount(q.MemoryUsage), baseStyle)
+}
+
+// Returns the max number of groups that can be displayed.
+func groupPageSize(s tcell.Screen) int {
+	_, h := s.Size()
+	return h - 16 // height - (# of rows used)
+}
+
+// Returns the number of tasks to fetch.
+func taskPageSize(s tcell.Screen) int {
+	_, h := s.Size()
+	return h - 15 // height - (# of rows used)
+}
+
+func shouldShowGroupTable(state *State) bool {
+	return state.taskState == asynq.TaskStateAggregating && state.selectedGroup == nil
+}
+
+func getTaskTableColumnConfig(taskState asynq.TaskState) []*columnConfig[*asynq.TaskInfo] {
+	switch taskState {
+	case asynq.TaskStateActive:
+		return activeTaskTableColumns
+	case asynq.TaskStatePending:
+		return pendingTaskTableColumns
+	case asynq.TaskStateAggregating:
+		return aggregatingTaskTableColumns
+	case asynq.TaskStateScheduled:
+		return scheduledTaskTableColumns
+	case asynq.TaskStateRetry:
+		return retryTaskTableColumns
+	case asynq.TaskStateArchived:
+		return archivedTaskTableColumns
+	case asynq.TaskStateCompleted:
+		return completedTaskTableColumns
+	}
+	panic("unknown task state")
+}
+
+var activeTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Retried", alignRight, func(t *asynq.TaskInfo) string { return strconv.Itoa(t.Retried) }},
+	{"Max Retry", alignRight, func(t *asynq.TaskInfo) string { return strconv.Itoa(t.MaxRetry) }},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+}
+
+var pendingTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Retried", alignRight, func(t *asynq.TaskInfo) string { return strconv.Itoa(t.Retried) }},
+	{"Max Retry", alignRight, func(t *asynq.TaskInfo) string { return strconv.Itoa(t.MaxRetry) }},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+}
+
+var aggregatingTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+	{"Group", alignLeft, func(t *asynq.TaskInfo) string { return t.Group }},
+}
+
+var scheduledTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Next Process Time", alignLeft, func(t *asynq.TaskInfo) string {
+		return formatNextProcessTime(t.NextProcessAt)
+	}},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+}
+
+var retryTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Retry", alignRight, func(t *asynq.TaskInfo) string { return fmt.Sprintf("%d/%d", t.Retried, t.MaxRetry) }},
+	{"Last Failure", alignLeft, func(t *asynq.TaskInfo) string { return t.LastErr }},
+	{"Last Failure Time", alignLeft, func(t *asynq.TaskInfo) string { return formatPastTime(t.LastFailedAt) }},
+	{"Next Process Time", alignLeft, func(t *asynq.TaskInfo) string {
+		return formatNextProcessTime(t.NextProcessAt)
+	}},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+}
+
+var archivedTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Retry", alignRight, func(t *asynq.TaskInfo) string { return fmt.Sprintf("%d/%d", t.Retried, t.MaxRetry) }},
+	{"Last Failure", alignLeft, func(t *asynq.TaskInfo) string { return t.LastErr }},
+	{"Last Failure Time", alignLeft, func(t *asynq.TaskInfo) string { return formatPastTime(t.LastFailedAt) }},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+}
+
+var completedTaskTableColumns = []*columnConfig[*asynq.TaskInfo]{
+	{"ID", alignLeft, func(t *asynq.TaskInfo) string { return t.ID }},
+	{"Type", alignLeft, func(t *asynq.TaskInfo) string { return t.Type }},
+	{"Completion Time", alignLeft, func(t *asynq.TaskInfo) string { return formatPastTime(t.CompletedAt) }},
+	{"Payload", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Payload) }},
+	{"Result", alignLeft, func(t *asynq.TaskInfo) string { return formatByteSlice(t.Result) }},
+}
+
+func drawTaskTable(d *ScreenDrawer, state *State) {
+	if shouldShowGroupTable(state) {
+		drawGroupTable(d, state)
+		return
+	}
+	if len(state.tasks) == 0 {
+		return // print nothing
+	}
+	drawTable(d, baseStyle, getTaskTableColumnConfig(state.taskState), state.tasks, state.taskTableRowIdx-1)
+
+	// Pagination
+	pageSize := taskPageSize(d.Screen())
+	totalCount := getTaskCount(state.selectedQueue, state.taskState)
+	if state.taskState == asynq.TaskStateAggregating {
+		// aggregating tasks are scoped to each group when shown in the table.
+		totalCount = state.selectedGroup.Size
+	}
+	if pageSize < totalCount {
+		start := (state.pageNum-1)*pageSize + 1
+		end := start + len(state.tasks) - 1
+		paginationStyle := baseStyle.Foreground(tcell.ColorLightGray)
+		d.Print(fmt.Sprintf("Showing %d-%d out of %d", start, end, totalCount), paginationStyle)
+		if isNextTaskPageAvailable(d.Screen(), state) {
+			d.Print("  n=NextPage", paginationStyle)
+		}
+		if state.pageNum > 1 {
+			d.Print("  p=PrevPage", paginationStyle)
+		}
+		d.FillLine(' ', paginationStyle)
+	}
+}
+
+func isNextTaskPageAvailable(s tcell.Screen, state *State) bool {
+	totalCount := getTaskCount(state.selectedQueue, state.taskState)
+	end := (state.pageNum-1)*taskPageSize(s) + len(state.tasks)
+	return end < totalCount
+}
+
+func drawGroupTable(d *ScreenDrawer, state *State) {
+	if len(state.groups) == 0 {
+		return // print nothing
+	}
+	d.Println("<<< Select group >>>", baseStyle)
+	colConfigs := []*columnConfig[*asynq.GroupInfo]{
+		{"Name", alignLeft, func(g *asynq.GroupInfo) string { return g.Group }},
+		{"Size", alignRight, func(g *asynq.GroupInfo) string { return strconv.Itoa(g.Size) }},
+	}
+	// pagination
+	pageSize := groupPageSize(d.Screen())
+	total := len(state.groups)
+	start := (state.pageNum - 1) * pageSize
+	end := min(start+pageSize, total)
+	drawTable(d, baseStyle, colConfigs, state.groups[start:end], state.groupTableRowIdx-1)
+
+	if pageSize < total {
+		d.Print(fmt.Sprintf("Showing %d-%d out of %d", start+1, end, total), labelStyle)
+		if end < total {
+			d.Print("  n=NextPage", labelStyle)
+		}
+		if start > 0 {
+			d.Print("  p=PrevPage", labelStyle)
+		}
+	}
+	d.FillLine(' ', labelStyle)
+}
+
+type number interface {
+	int | int64 | float64
+}
+
+// min returns the smaller of x and y. if x==y, returns x
+func min[V number](x, y V) V {
+	if x > y {
+		return y
+	}
+	return x
+}
+
+// Define the order of states to show
+var taskStates = []asynq.TaskState{
+	asynq.TaskStateActive,
+	asynq.TaskStatePending,
+	asynq.TaskStateAggregating,
+	asynq.TaskStateScheduled,
+	asynq.TaskStateRetry,
+	asynq.TaskStateArchived,
+	asynq.TaskStateCompleted,
+}
+
+func nextTaskState(current asynq.TaskState) asynq.TaskState {
+	for i, ts := range taskStates {
+		if current == ts {
+			if i == len(taskStates)-1 {
+				return taskStates[0]
+			} else {
+				return taskStates[i+1]
+			}
+		}
+	}
+	panic("unknown task state")
+}
+
+func prevTaskState(current asynq.TaskState) asynq.TaskState {
+	for i, ts := range taskStates {
+		if current == ts {
+			if i == 0 {
+				return taskStates[len(taskStates)-1]
+			} else {
+				return taskStates[i-1]
+			}
+		}
+	}
+	panic("unknown task state")
+}
+
+func getTaskCount(queue *asynq.QueueInfo, taskState asynq.TaskState) int {
+	switch taskState {
+	case asynq.TaskStateActive:
+		return queue.Active
+	case asynq.TaskStatePending:
+		return queue.Pending
+	case asynq.TaskStateAggregating:
+		return queue.Aggregating
+	case asynq.TaskStateScheduled:
+		return queue.Scheduled
+	case asynq.TaskStateRetry:
+		return queue.Retry
+	case asynq.TaskStateArchived:
+		return queue.Archived
+	case asynq.TaskStateCompleted:
+		return queue.Completed
+	}
+	panic("unkonwn task state")
+}
+
+func drawTaskStateBreakdown(d *ScreenDrawer, style tcell.Style, state *State) {
+	const pad = "    " // padding between states
+	for _, ts := range taskStates {
+		s := style
+		if state.taskState == ts {
+			s = s.Bold(true).Underline(true)
+		}
+		d.Print(fmt.Sprintf("%s:%d", strings.Title(ts.String()), getTaskCount(state.selectedQueue, ts)), s)
+		d.Print(pad, style)
+	}
+	d.NL()
+}
+
+func drawTaskModal(d *ScreenDrawer, state *State) {
+	if state.taskID == "" {
+		return
+	}
+	task := state.selectedTask
+	if task == nil {
+		// task no longer found
+		fns := []func(d *modalRowDrawer){
+			func(d *modalRowDrawer) { d.Print("=== Task Info ===", baseStyle.Bold(true)) },
+			func(d *modalRowDrawer) { d.Print("", baseStyle) },
+			func(d *modalRowDrawer) {
+				d.Print(fmt.Sprintf("Task %q no longer exists", state.taskID), baseStyle)
+			},
+		}
+		withModal(d, fns)
+		return
+	}
+	fns := []func(d *modalRowDrawer){
+		func(d *modalRowDrawer) { d.Print("=== Task Info ===", baseStyle.Bold(true)) },
+		func(d *modalRowDrawer) { d.Print("", baseStyle) },
+		func(d *modalRowDrawer) {
+			d.Print("ID: ", labelStyle)
+			d.Print(task.ID, baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("Type: ", labelStyle)
+			d.Print(task.Type, baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("State: ", labelStyle)
+			d.Print(task.State.String(), baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("Queue: ", labelStyle)
+			d.Print(task.Queue, baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("Retry: ", labelStyle)
+			d.Print(fmt.Sprintf("%d/%d", task.Retried, task.MaxRetry), baseStyle)
+		},
+	}
+	if task.LastErr != "" {
+		fns = append(fns, func(d *modalRowDrawer) {
+			d.Print("Last Failure: ", labelStyle)
+			d.Print(task.LastErr, baseStyle)
+		})
+		fns = append(fns, func(d *modalRowDrawer) {
+			d.Print("Last Failure Time: ", labelStyle)
+			d.Print(fmt.Sprintf("%v (%s)", task.LastFailedAt, formatPastTime(task.LastFailedAt)), baseStyle)
+		})
+	}
+	if !task.NextProcessAt.IsZero() {
+		fns = append(fns, func(d *modalRowDrawer) {
+			d.Print("Next Process Time: ", labelStyle)
+			d.Print(fmt.Sprintf("%v (%s)", task.NextProcessAt, formatNextProcessTime(task.NextProcessAt)), baseStyle)
+		})
+	}
+	if !task.CompletedAt.IsZero() {
+		fns = append(fns, func(d *modalRowDrawer) {
+			d.Print("Completion Time: ", labelStyle)
+			d.Print(fmt.Sprintf("%v (%s)", task.CompletedAt, formatPastTime(task.CompletedAt)), baseStyle)
+		})
+	}
+	fns = append(fns, func(d *modalRowDrawer) {
+		d.Print("Payload: ", labelStyle)
+		d.Print(formatByteSlice(task.Payload), baseStyle)
+	})
+	if task.Result != nil {
+		fns = append(fns, func(d *modalRowDrawer) {
+			d.Print("Result: ", labelStyle)
+			d.Print(formatByteSlice(task.Result), baseStyle)
+		})
+	}
+	withModal(d, fns)
+}
+
+// Reports whether the given byte slice is printable (i.e. human readable)
+func isPrintable(data []byte) bool {
+	if !utf8.Valid(data) {
+		return false
+	}
+	isAllSpace := true
+	for _, r := range string(data) {
+		if !unicode.IsGraphic(r) {
+			return false
+		}
+		if !unicode.IsSpace(r) {
+			isAllSpace = false
+		}
+	}
+	return !isAllSpace
+}
+
+func formatByteSlice(data []byte) string {
+	if data == nil {
+		return "<nil>"
+	}
+	if !isPrintable(data) {
+		return "<non-printable>"
+	}
+	return strings.ReplaceAll(string(data), "\n", "  ")
+}
+
+type modalRowDrawer struct {
+	d        *ScreenDrawer
+	width    int // current width occupied by content
+	maxWidth int
+}
+
+// Note: s should not include newline
+func (d *modalRowDrawer) Print(s string, style tcell.Style) {
+	if d.width >= d.maxWidth {
+		return // no longer write to this row
+	}
+	if d.width+runewidth.StringWidth(s) > d.maxWidth {
+		s = truncate(s, d.maxWidth-d.width)
+	}
+	d.d.Print(s, style)
+}
+
+// withModal draws a modal with the given functions row by row.
+func withModal(d *ScreenDrawer, rowPrintFns []func(d *modalRowDrawer)) {
+	w, h := d.Screen().Size()
+	var (
+		modalWidth  = int(math.Floor(float64(w) * 0.6))
+		modalHeight = int(math.Floor(float64(h) * 0.6))
+		rowOffset   = int(math.Floor(float64(h) * 0.2)) // 20% from the top
+		colOffset   = int(math.Floor(float64(w) * 0.2)) // 20% from the left
+	)
+	if modalHeight < 3 {
+		return // no content can be shown
+	}
+	d.Goto(colOffset, rowOffset)
+	d.Print(string(tcell.RuneULCorner), baseStyle)
+	d.Print(strings.Repeat(string(tcell.RuneHLine), modalWidth-2), baseStyle)
+	d.Print(string(tcell.RuneURCorner), baseStyle)
+	d.NL()
+	rowDrawer := modalRowDrawer{
+		d:        d,
+		width:    0,
+		maxWidth: modalWidth - 4, /* borders + paddings */
+	}
+	for i := 1; i < modalHeight-1; i++ {
+		d.Goto(colOffset, rowOffset+i)
+		d.Print(fmt.Sprintf("%c ", tcell.RuneVLine), baseStyle)
+		if i <= len(rowPrintFns) {
+			rowPrintFns[i-1](&rowDrawer)
+		}
+		d.FillUntil(' ', baseStyle, colOffset+modalWidth-2)
+		d.Print(fmt.Sprintf(" %c", tcell.RuneVLine), baseStyle)
+		d.NL()
+	}
+	d.Goto(colOffset, rowOffset+modalHeight-1)
+	d.Print(string(tcell.RuneLLCorner), baseStyle)
+	d.Print(strings.Repeat(string(tcell.RuneHLine), modalWidth-2), baseStyle)
+	d.Print(string(tcell.RuneLRCorner), baseStyle)
+	d.NL()
+}
+
+func adjustWidth(s string, width int) string {
+	sw := runewidth.StringWidth(s)
+	if sw > width {
+		return truncate(s, width)
+	}
+	var b strings.Builder
+	b.WriteString(s)
+	b.WriteString(strings.Repeat(" ", width-sw))
+	return b.String()
+}
+
+// truncates s if s exceeds max length.
+func truncate(s string, max int) string {
+	if runewidth.StringWidth(s) <= max {
+		return s
+	}
+	return string([]rune(s)[:max-1]) + "…"
+}
+
+func drawDebugInfo(d *ScreenDrawer, state *State) {
+	d.Println(state.DebugString(), baseStyle)
+}
+
+func drawHelp(d *ScreenDrawer) {
+	keyStyle := labelStyle.Bold(true)
+	withModal(d, []func(*modalRowDrawer){
+		func(d *modalRowDrawer) { d.Print("=== Help ===", baseStyle.Bold(true)) },
+		func(d *modalRowDrawer) { d.Print("", baseStyle) },
+		func(d *modalRowDrawer) {
+			d.Print("<Enter>", keyStyle)
+			d.Print("              to select", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<Esc>", keyStyle)
+			d.Print(" or ", baseStyle)
+			d.Print("<q>", keyStyle)
+			d.Print("         to go back", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<UpArrow>", keyStyle)
+			d.Print(" or ", baseStyle)
+			d.Print("<k>", keyStyle)
+			d.Print("     to move up", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<DownArrow>", keyStyle)
+			d.Print(" or ", baseStyle)
+			d.Print("<j>", keyStyle)
+			d.Print("   to move down", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<LeftArrow>", keyStyle)
+			d.Print(" or ", baseStyle)
+			d.Print("<h>", keyStyle)
+			d.Print("   to move left", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<RightArrow>", keyStyle)
+			d.Print(" or ", baseStyle)
+			d.Print("<l>", keyStyle)
+			d.Print("  to move right", baseStyle)
+		},
+		func(d *modalRowDrawer) {
+			d.Print("<Ctrl+C>", keyStyle)
+			d.Print("             to quit", baseStyle)
+		},
+	})
+}

tools/asynq/cmd/dash/draw_test.go

@@ -0,0 +1,33 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import "testing"
+
+func TestTruncate(t *testing.T) {
+	tests := []struct {
+		s    string
+		max  int
+		want string
+	}{
+		{
+			s:    "hello world!",
+			max:  15,
+			want: "hello world!",
+		},
+		{
+			s:    "hello world!",
+			max:  6,
+			want: "hello…",
+		},
+	}
+
+	for _, tc := range tests {
+		got := truncate(tc.s, tc.max)
+		if tc.want != got {
+			t.Errorf("truncate(%q, %d) = %q, want %q", tc.s, tc.max, got, tc.want)
+		}
+	}
+}

tools/asynq/cmd/dash/fetch.go

@@ -0,0 +1,185 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"sort"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/hibiken/asynq"
+)
+
+type fetcher interface {
+	// Fetch retries data required by the given state of the dashboard.
+	Fetch(state *State)
+}
+
+type dataFetcher struct {
+	inspector *asynq.Inspector
+	opts      Options
+	s         tcell.Screen
+
+	errorCh  chan<- error
+	queueCh  chan<- *asynq.QueueInfo
+	taskCh   chan<- *asynq.TaskInfo
+	queuesCh chan<- []*asynq.QueueInfo
+	groupsCh chan<- []*asynq.GroupInfo
+	tasksCh  chan<- []*asynq.TaskInfo
+}
+
+func (f *dataFetcher) Fetch(state *State) {
+	switch state.view {
+	case viewTypeQueues:
+		f.fetchQueues()
+	case viewTypeQueueDetails:
+		if shouldShowGroupTable(state) {
+			f.fetchGroups(state.selectedQueue.Queue)
+		} else if state.taskState == asynq.TaskStateAggregating {
+			f.fetchAggregatingTasks(state.selectedQueue.Queue, state.selectedGroup.Group, taskPageSize(f.s), state.pageNum)
+		} else {
+			f.fetchTasks(state.selectedQueue.Queue, state.taskState, taskPageSize(f.s), state.pageNum)
+		}
+		// if the task modal is open, additionally fetch the selected task's info
+		if state.taskID != "" {
+			f.fetchTaskInfo(state.selectedQueue.Queue, state.taskID)
+		}
+	}
+}
+
+func (f *dataFetcher) fetchQueues() {
+	var (
+		inspector = f.inspector
+		queuesCh  = f.queuesCh
+		errorCh   = f.errorCh
+		opts      = f.opts
+	)
+	go fetchQueues(inspector, queuesCh, errorCh, opts)
+}
+
+func fetchQueues(i *asynq.Inspector, queuesCh chan<- []*asynq.QueueInfo, errorCh chan<- error, opts Options) {
+	queues, err := i.Queues()
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	sort.Strings(queues)
+	var res []*asynq.QueueInfo
+	for _, q := range queues {
+		info, err := i.GetQueueInfo(q)
+		if err != nil {
+			errorCh <- err
+			return
+		}
+		res = append(res, info)
+	}
+	queuesCh <- res
+}
+
+func fetchQueueInfo(i *asynq.Inspector, qname string, queueCh chan<- *asynq.QueueInfo, errorCh chan<- error) {
+	q, err := i.GetQueueInfo(qname)
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	queueCh <- q
+}
+
+func (f *dataFetcher) fetchGroups(qname string) {
+	var (
+		i        = f.inspector
+		groupsCh = f.groupsCh
+		errorCh  = f.errorCh
+		queueCh  = f.queueCh
+	)
+	go fetchGroups(i, qname, groupsCh, errorCh)
+	go fetchQueueInfo(i, qname, queueCh, errorCh)
+}
+
+func fetchGroups(i *asynq.Inspector, qname string, groupsCh chan<- []*asynq.GroupInfo, errorCh chan<- error) {
+	groups, err := i.Groups(qname)
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	groupsCh <- groups
+}
+
+func (f *dataFetcher) fetchAggregatingTasks(qname, group string, pageSize, pageNum int) {
+	var (
+		i       = f.inspector
+		tasksCh = f.tasksCh
+		errorCh = f.errorCh
+		queueCh = f.queueCh
+	)
+	go fetchAggregatingTasks(i, qname, group, pageSize, pageNum, tasksCh, errorCh)
+	go fetchQueueInfo(i, qname, queueCh, errorCh)
+}
+
+func fetchAggregatingTasks(i *asynq.Inspector, qname, group string, pageSize, pageNum int,
+	tasksCh chan<- []*asynq.TaskInfo, errorCh chan<- error) {
+	tasks, err := i.ListAggregatingTasks(qname, group, asynq.PageSize(pageSize), asynq.Page(pageNum))
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	tasksCh <- tasks
+}
+
+func (f *dataFetcher) fetchTasks(qname string, taskState asynq.TaskState, pageSize, pageNum int) {
+	var (
+		i       = f.inspector
+		tasksCh = f.tasksCh
+		errorCh = f.errorCh
+		queueCh = f.queueCh
+	)
+	go fetchTasks(i, qname, taskState, pageSize, pageNum, tasksCh, errorCh)
+	go fetchQueueInfo(i, qname, queueCh, errorCh)
+}
+
+func fetchTasks(i *asynq.Inspector, qname string, taskState asynq.TaskState, pageSize, pageNum int,
+	tasksCh chan<- []*asynq.TaskInfo, errorCh chan<- error) {
+	var (
+		tasks []*asynq.TaskInfo
+		err   error
+	)
+	opts := []asynq.ListOption{asynq.PageSize(pageSize), asynq.Page(pageNum)}
+	switch taskState {
+	case asynq.TaskStateActive:
+		tasks, err = i.ListActiveTasks(qname, opts...)
+	case asynq.TaskStatePending:
+		tasks, err = i.ListPendingTasks(qname, opts...)
+	case asynq.TaskStateScheduled:
+		tasks, err = i.ListScheduledTasks(qname, opts...)
+	case asynq.TaskStateRetry:
+		tasks, err = i.ListRetryTasks(qname, opts...)
+	case asynq.TaskStateArchived:
+		tasks, err = i.ListArchivedTasks(qname, opts...)
+	case asynq.TaskStateCompleted:
+		tasks, err = i.ListCompletedTasks(qname, opts...)
+	}
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	tasksCh <- tasks
+}
+
+func (f *dataFetcher) fetchTaskInfo(qname, taskID string) {
+	var (
+		i       = f.inspector
+		taskCh  = f.taskCh
+		errorCh = f.errorCh
+	)
+	go fetchTaskInfo(i, qname, taskID, taskCh, errorCh)
+}
+
+func fetchTaskInfo(i *asynq.Inspector, qname, taskID string, taskCh chan<- *asynq.TaskInfo, errorCh chan<- error) {
+	info, err := i.GetTaskInfo(qname, taskID)
+	if err != nil {
+		errorCh <- err
+		return
+	}
+	taskCh <- info
+}

tools/asynq/cmd/dash/key_event.go

@@ -0,0 +1,317 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"os"
+	"time"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/hibiken/asynq"
+)
+
+// keyEventHandler handles keyboard events and updates the state.
+// It delegates data fetching to fetcher and UI rendering to drawer.
+type keyEventHandler struct {
+	s     tcell.Screen
+	state *State
+	done  chan struct{}
+
+	fetcher fetcher
+	drawer  drawer
+
+	ticker       *time.Ticker
+	pollInterval time.Duration
+}
+
+func (h *keyEventHandler) quit() {
+	h.s.Fini()
+	close(h.done)
+	os.Exit(0)
+}
+
+func (h *keyEventHandler) HandleKeyEvent(ev *tcell.EventKey) {
+	if ev.Key() == tcell.KeyEscape || ev.Rune() == 'q' {
+		h.goBack() // Esc and 'q' key have "go back" semantics
+	} else if ev.Key() == tcell.KeyCtrlC {
+		h.quit()
+	} else if ev.Key() == tcell.KeyCtrlL {
+		h.s.Sync()
+	} else if ev.Key() == tcell.KeyDown || ev.Rune() == 'j' {
+		h.handleDownKey()
+	} else if ev.Key() == tcell.KeyUp || ev.Rune() == 'k' {
+		h.handleUpKey()
+	} else if ev.Key() == tcell.KeyRight || ev.Rune() == 'l' {
+		h.handleRightKey()
+	} else if ev.Key() == tcell.KeyLeft || ev.Rune() == 'h' {
+		h.handleLeftKey()
+	} else if ev.Key() == tcell.KeyEnter {
+		h.handleEnterKey()
+	} else if ev.Rune() == '?' {
+		h.showHelp()
+	} else if ev.Rune() == 'n' {
+		h.nextPage()
+	} else if ev.Rune() == 'p' {
+		h.prevPage()
+	}
+}
+
+func (h *keyEventHandler) goBack() {
+	var (
+		state = h.state
+		d     = h.drawer
+		f     = h.fetcher
+	)
+	if state.view == viewTypeHelp {
+		state.view = state.prevView // exit help
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	} else if state.view == viewTypeQueueDetails {
+		// if task modal is open close it; otherwise go back to the previous view
+		if state.taskID != "" {
+			state.taskID = ""
+			state.selectedTask = nil
+			d.Draw(state)
+		} else {
+			state.view = viewTypeQueues
+			f.Fetch(state)
+			h.resetTicker()
+			d.Draw(state)
+		}
+	} else {
+		h.quit()
+	}
+}
+
+func (h *keyEventHandler) handleDownKey() {
+	switch h.state.view {
+	case viewTypeQueues:
+		h.downKeyQueues()
+	case viewTypeQueueDetails:
+		h.downKeyQueueDetails()
+	}
+}
+
+func (h *keyEventHandler) downKeyQueues() {
+	if h.state.queueTableRowIdx < len(h.state.queues) {
+		h.state.queueTableRowIdx++
+	} else {
+		h.state.queueTableRowIdx = 0 // loop back
+	}
+	h.drawer.Draw(h.state)
+}
+
+func (h *keyEventHandler) downKeyQueueDetails() {
+	s, state := h.s, h.state
+	if shouldShowGroupTable(state) {
+		if state.groupTableRowIdx < groupPageSize(s) {
+			state.groupTableRowIdx++
+		} else {
+			state.groupTableRowIdx = 0 // loop back
+		}
+	} else if state.taskID == "" {
+		if state.taskTableRowIdx < len(state.tasks) {
+			state.taskTableRowIdx++
+		} else {
+			state.taskTableRowIdx = 0 // loop back
+		}
+	}
+	h.drawer.Draw(state)
+}
+
+func (h *keyEventHandler) handleUpKey() {
+	switch h.state.view {
+	case viewTypeQueues:
+		h.upKeyQueues()
+	case viewTypeQueueDetails:
+		h.upKeyQueueDetails()
+	}
+}
+
+func (h *keyEventHandler) upKeyQueues() {
+	state := h.state
+	if state.queueTableRowIdx == 0 {
+		state.queueTableRowIdx = len(state.queues)
+	} else {
+		state.queueTableRowIdx--
+	}
+	h.drawer.Draw(state)
+}
+
+func (h *keyEventHandler) upKeyQueueDetails() {
+	s, state := h.s, h.state
+	if shouldShowGroupTable(state) {
+		if state.groupTableRowIdx == 0 {
+			state.groupTableRowIdx = groupPageSize(s)
+		} else {
+			state.groupTableRowIdx--
+		}
+	} else if state.taskID == "" {
+		if state.taskTableRowIdx == 0 {
+			state.taskTableRowIdx = len(state.tasks)
+		} else {
+			state.taskTableRowIdx--
+		}
+	}
+	h.drawer.Draw(state)
+}
+
+func (h *keyEventHandler) handleEnterKey() {
+	switch h.state.view {
+	case viewTypeQueues:
+		h.enterKeyQueues()
+	case viewTypeQueueDetails:
+		h.enterKeyQueueDetails()
+	}
+}
+
+func (h *keyEventHandler) resetTicker() {
+	h.ticker.Reset(h.pollInterval)
+}
+
+func (h *keyEventHandler) enterKeyQueues() {
+	var (
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if state.queueTableRowIdx != 0 {
+		state.selectedQueue = state.queues[state.queueTableRowIdx-1]
+		state.view = viewTypeQueueDetails
+		state.taskState = asynq.TaskStateActive
+		state.tasks = nil
+		state.pageNum = 1
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	}
+}
+
+func (h *keyEventHandler) enterKeyQueueDetails() {
+	var (
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if shouldShowGroupTable(state) && state.groupTableRowIdx != 0 {
+		state.selectedGroup = state.groups[state.groupTableRowIdx-1]
+		state.tasks = nil
+		state.pageNum = 1
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	} else if !shouldShowGroupTable(state) && state.taskTableRowIdx != 0 {
+		task := state.tasks[state.taskTableRowIdx-1]
+		state.selectedTask = task
+		state.taskID = task.ID
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	}
+
+}
+
+func (h *keyEventHandler) handleLeftKey() {
+	var (
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if state.view == viewTypeQueueDetails && state.taskID == "" {
+		state.taskState = prevTaskState(state.taskState)
+		state.pageNum = 1
+		state.taskTableRowIdx = 0
+		state.tasks = nil
+		state.selectedGroup = nil
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	}
+}
+
+func (h *keyEventHandler) handleRightKey() {
+	var (
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if state.view == viewTypeQueueDetails && state.taskID == "" {
+		state.taskState = nextTaskState(state.taskState)
+		state.pageNum = 1
+		state.taskTableRowIdx = 0
+		state.tasks = nil
+		state.selectedGroup = nil
+		f.Fetch(state)
+		h.resetTicker()
+		d.Draw(state)
+	}
+}
+
+func (h *keyEventHandler) nextPage() {
+	var (
+		s     = h.s
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if state.view == viewTypeQueueDetails {
+		if shouldShowGroupTable(state) {
+			pageSize := groupPageSize(s)
+			total := len(state.groups)
+			start := (state.pageNum - 1) * pageSize
+			end := start + pageSize
+			if end <= total {
+				state.pageNum++
+				d.Draw(state)
+			}
+		} else {
+			pageSize := taskPageSize(s)
+			totalCount := getTaskCount(state.selectedQueue, state.taskState)
+			if (state.pageNum-1)*pageSize+len(state.tasks) < totalCount {
+				state.pageNum++
+				f.Fetch(state)
+				h.resetTicker()
+			}
+		}
+	}
+}
+
+func (h *keyEventHandler) prevPage() {
+	var (
+		s     = h.s
+		state = h.state
+		f     = h.fetcher
+		d     = h.drawer
+	)
+	if state.view == viewTypeQueueDetails {
+		if shouldShowGroupTable(state) {
+			pageSize := groupPageSize(s)
+			start := (state.pageNum - 1) * pageSize
+			if start > 0 {
+				state.pageNum--
+				d.Draw(state)
+			}
+		} else {
+			if state.pageNum > 1 {
+				state.pageNum--
+				f.Fetch(state)
+				h.resetTicker()
+			}
+		}
+	}
+}
+
+func (h *keyEventHandler) showHelp() {
+	var (
+		state = h.state
+		d     = h.drawer
+	)
+	if state.view != viewTypeHelp {
+		state.prevView = state.view
+		state.view = viewTypeHelp
+		d.Draw(state)
+	}
+}

tools/asynq/cmd/dash/key_event_test.go

@@ -0,0 +1,234 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"testing"
+	"time"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/google/go-cmp/cmp"
+	"github.com/hibiken/asynq"
+)
+
+func makeKeyEventHandler(t *testing.T, state *State) *keyEventHandler {
+	ticker := time.NewTicker(time.Second)
+	t.Cleanup(func() { ticker.Stop() })
+	return &keyEventHandler{
+		s:            tcell.NewSimulationScreen("UTF-8"),
+		state:        state,
+		done:         make(chan struct{}),
+		fetcher:      &fakeFetcher{},
+		drawer:       &fakeDrawer{},
+		ticker:       ticker,
+		pollInterval: time.Second,
+	}
+}
+
+type keyEventHandlerTest struct {
+	desc      string            // test description
+	state     *State            // initial state, to be mutated by the handler
+	events    []*tcell.EventKey // keyboard events
+	wantState State             // expected state after the events
+}
+
+func TestKeyEventHandler(t *testing.T) {
+	tests := []*keyEventHandlerTest{
+		{
+			desc:      "navigates to help view",
+			state:     &State{view: viewTypeQueues},
+			events:    []*tcell.EventKey{tcell.NewEventKey(tcell.KeyRune, '?', tcell.ModNone)},
+			wantState: State{view: viewTypeHelp},
+		},
+		{
+			desc: "navigates to queue details view",
+			state: &State{
+				view: viewTypeQueues,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 100, Active: 10, Pending: 40, Scheduled: 40, Completed: 10},
+				},
+				queueTableRowIdx: 0,
+			},
+			events: []*tcell.EventKey{
+				tcell.NewEventKey(tcell.KeyRune, 'j', tcell.ModNone),   // down
+				tcell.NewEventKey(tcell.KeyEnter, '\n', tcell.ModNone), // Enter
+			},
+			wantState: State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 100, Active: 10, Pending: 40, Scheduled: 40, Completed: 10},
+				},
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 100, Active: 10, Pending: 40, Scheduled: 40, Completed: 10},
+				queueTableRowIdx: 1,
+				taskState:        asynq.TaskStateActive,
+				pageNum:          1,
+			},
+		},
+		{
+			desc: "does nothing if no queues are present",
+			state: &State{
+				view:             viewTypeQueues,
+				queues:           []*asynq.QueueInfo{}, // empty
+				queueTableRowIdx: 0,
+			},
+			events: []*tcell.EventKey{
+				tcell.NewEventKey(tcell.KeyRune, 'j', tcell.ModNone),   // down
+				tcell.NewEventKey(tcell.KeyEnter, '\n', tcell.ModNone), // Enter
+			},
+			wantState: State{
+				view:             viewTypeQueues,
+				queues:           []*asynq.QueueInfo{},
+				queueTableRowIdx: 0,
+			},
+		},
+		{
+			desc: "opens task info modal",
+			state: &State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+			},
+			events: []*tcell.EventKey{
+				tcell.NewEventKey(tcell.KeyEnter, '\n', tcell.ModNone), // Enter
+			},
+			wantState: State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+				// new states
+				taskID:       "yyyy",
+				selectedTask: &asynq.TaskInfo{ID: "yyyy", Type: "bar"},
+			},
+		},
+		{
+			desc: "Esc closes task info modal",
+			state: &State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+				taskID:          "yyyy", // presence of this field opens the modal
+			},
+			events: []*tcell.EventKey{
+				tcell.NewEventKey(tcell.KeyEscape, ' ', tcell.ModNone), // Esc
+			},
+			wantState: State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+				taskID:          "", // this field should be unset
+			},
+		},
+		{
+			desc: "Arrow keys are disabled while task info modal is open",
+			state: &State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+				taskID:          "yyyy", // presence of this field opens the modal
+			},
+			events: []*tcell.EventKey{
+				tcell.NewEventKey(tcell.KeyLeft, ' ', tcell.ModNone),
+			},
+
+			// no change
+			wantState: State{
+				view: viewTypeQueueDetails,
+				queues: []*asynq.QueueInfo{
+					{Queue: "default", Size: 500, Active: 10, Pending: 40},
+				},
+				queueTableRowIdx: 1,
+				selectedQueue:    &asynq.QueueInfo{Queue: "default", Size: 50, Active: 10, Pending: 40},
+				taskState:        asynq.TaskStatePending,
+				pageNum:          1,
+				tasks: []*asynq.TaskInfo{
+					{ID: "xxxx", Type: "foo"},
+					{ID: "yyyy", Type: "bar"},
+					{ID: "zzzz", Type: "baz"},
+				},
+				taskTableRowIdx: 2,
+				taskID:          "yyyy", // presence of this field opens the modal
+			},
+		},
+		// TODO: Add more tests
+	}
+
+	for _, tc := range tests {
+		t.Run(tc.desc, func(t *testing.T) {
+			h := makeKeyEventHandler(t, tc.state)
+			for _, e := range tc.events {
+				h.HandleKeyEvent(e)
+			}
+			if diff := cmp.Diff(tc.wantState, *tc.state, cmp.AllowUnexported(State{})); diff != "" {
+				t.Errorf("after state was %+v, want %+v: (-want,+got)\n%s", *tc.state, tc.wantState, diff)
+			}
+		})
+	}
+
+}
+
+/*** fake implementation for tests ***/
+
+type fakeFetcher struct{}
+
+func (f *fakeFetcher) Fetch(s *State) {}
+
+type fakeDrawer struct{}
+
+func (d *fakeDrawer) Draw(s *State) {}

tools/asynq/cmd/dash/screen_drawer.go

@@ -0,0 +1,100 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"strings"
+
+	"github.com/gdamore/tcell/v2"
+	"github.com/mattn/go-runewidth"
+)
+
+/*** Screen Drawer ***/
+
+// ScreenDrawer is used to draw contents on screen.
+//
+// Usage example:
+//    d := NewScreenDrawer(s)
+//    d.Println("Hello world", mystyle)
+//    d.NL() // adds newline
+//    d.Print("foo", mystyle.Bold(true))
+//    d.Print("bar", mystyle.Italic(true))
+type ScreenDrawer struct {
+	l *LineDrawer
+}
+
+func NewScreenDrawer(s tcell.Screen) *ScreenDrawer {
+	return &ScreenDrawer{l: NewLineDrawer(0, s)}
+}
+
+func (d *ScreenDrawer) Print(s string, style tcell.Style) {
+	d.l.Draw(s, style)
+}
+
+func (d *ScreenDrawer) Println(s string, style tcell.Style) {
+	d.Print(s, style)
+	d.NL()
+}
+
+// FillLine prints the given rune until the end of the current line
+// and adds a newline.
+func (d *ScreenDrawer) FillLine(r rune, style tcell.Style) {
+	w, _ := d.Screen().Size()
+	if w-d.l.col < 0 {
+		d.NL()
+		return
+	}
+	s := strings.Repeat(string(r), w-d.l.col)
+	d.Print(s, style)
+	d.NL()
+}
+
+func (d *ScreenDrawer) FillUntil(r rune, style tcell.Style, limit int) {
+	if d.l.col > limit {
+		return // already passed the limit
+	}
+	s := strings.Repeat(string(r), limit-d.l.col)
+	d.Print(s, style)
+}
+
+// NL adds a newline (i.e., moves to the next line).
+func (d *ScreenDrawer) NL() {
+	d.l.row++
+	d.l.col = 0
+}
+
+func (d *ScreenDrawer) Screen() tcell.Screen {
+	return d.l.s
+}
+
+// Goto moves the screendrawer to the specified cell.
+func (d *ScreenDrawer) Goto(x, y int) {
+	d.l.row = y
+	d.l.col = x
+}
+
+// Go to the bottom of the screen.
+func (d *ScreenDrawer) GoToBottom() {
+	_, h := d.Screen().Size()
+	d.l.row = h - 1
+	d.l.col = 0
+}
+
+type LineDrawer struct {
+	s   tcell.Screen
+	row int
+	col int
+}
+
+func NewLineDrawer(row int, s tcell.Screen) *LineDrawer {
+	return &LineDrawer{row: row, col: 0, s: s}
+}
+
+func (d *LineDrawer) Draw(s string, style tcell.Style) {
+	for _, r := range s {
+		d.s.SetContent(d.col, d.row, r, nil, style)
+		d.col += runewidth.RuneWidth(r)
+	}
+}

tools/asynq/cmd/dash/table.go

@@ -0,0 +1,70 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package dash
+
+import (
+	"github.com/gdamore/tcell/v2"
+	"github.com/mattn/go-runewidth"
+)
+
+type columnAlignment int
+
+const (
+	alignRight columnAlignment = iota
+	alignLeft
+)
+
+type columnConfig[V any] struct {
+	name      string
+	alignment columnAlignment
+	displayFn func(v V) string
+}
+
+type column[V any] struct {
+	*columnConfig[V]
+	width int
+}
+
+// Helper to draw a table.
+func drawTable[V any](d *ScreenDrawer, style tcell.Style, configs []*columnConfig[V], data []V, highlightRowIdx int) {
+	const colBuffer = "    " // extra buffer between columns
+	cols := make([]*column[V], len(configs))
+	for i, cfg := range configs {
+		cols[i] = &column[V]{cfg, runewidth.StringWidth(cfg.name)}
+	}
+	// adjust the column width to accommodate the widest value.
+	for _, v := range data {
+		for _, col := range cols {
+			if w := runewidth.StringWidth(col.displayFn(v)); col.width < w {
+				col.width = w
+			}
+		}
+	}
+	// print header
+	headerStyle := style.Background(tcell.ColorDimGray).Foreground(tcell.ColorWhite)
+	for _, col := range cols {
+		if col.alignment == alignLeft {
+			d.Print(rpad(col.name, col.width)+colBuffer, headerStyle)
+		} else {
+			d.Print(lpad(col.name, col.width)+colBuffer, headerStyle)
+		}
+	}
+	d.FillLine(' ', headerStyle)
+	// print body
+	for i, v := range data {
+		rowStyle := style
+		if highlightRowIdx == i {
+			rowStyle = style.Background(tcell.ColorDarkOliveGreen)
+		}
+		for _, col := range cols {
+			if col.alignment == alignLeft {
+				d.Print(rpad(col.displayFn(v), col.width)+colBuffer, rowStyle)
+			} else {
+				d.Print(lpad(col.displayFn(v), col.width)+colBuffer, rowStyle)
+			}
+		}
+		d.FillLine(' ', rowStyle)
+	}
+}

tools/asynq/cmd/group.go

@@ -0,0 +1,52 @@
+// Copyright 2022 Kentaro Hibino. All rights reserved.
+// Use of this source code is governed by a MIT license
+// that can be found in the LICENSE file.
+
+package cmd
+
+import (
+	"fmt"
+	"os"
+
+	"github.com/MakeNowJust/heredoc/v2"
+	"github.com/spf13/cobra"
+)
+
+func init() {
+	rootCmd.AddCommand(groupCmd)
+	groupCmd.AddCommand(groupListCmd)
+	groupListCmd.Flags().StringP("queue", "q", "", "queue to inspect")
+	groupListCmd.MarkFlagRequired("queue")
+}
+
+var groupCmd = &cobra.Command{
+	Use:   "group <command> [flags]",
+	Short: "Manage groups",
+	Example: heredoc.Doc(`
+		$ asynq group list --queue=myqueue`),
+}
+
+var groupListCmd = &cobra.Command{
+	Use:     "list",
+	Aliases: []string{"ls"},
+	Short:   "List groups",
+	Args:    cobra.NoArgs,
+	Run:     groupLists,
+}
+
+func groupLists(cmd *cobra.Command, args []string) {
+	qname, err := cmd.Flags().GetString("queue")
+	if err != nil {
+		fmt.Println(err)
+		os.Exit(1)
+	}
+	inspector := createInspector()
+	groups, err := inspector.Groups(qname)
+	if len(groups) == 0 {
+		fmt.Printf("No groups found in queue %q\n", qname)
+		return
+	}
+	for _, g := range groups {
+		fmt.Println(g.Group)
+	}
+}

tools/asynq/cmd/queue.go

@@ -9,6 +9,7 @@ import (
 	"io"
 	"os"
 
+	"github.com/MakeNowJust/heredoc/v2"
 	"github.com/fatih/color"
 	"github.com/hibiken/asynq"
 	"github.com/hibiken/asynq/internal/errors"
@@ -31,51 +32,75 @@ func init() {
 }
 
 var queueCmd = &cobra.Command{
-	Use:   "queue",
+	Use:   "queue <command> [flags]",
 	Short: "Manage queues",
+	Example: heredoc.Doc(`
+	  $ asynq queue ls
+	  $ asynq queue inspect myqueue
+	  $ asynq queue pause myqueue`),
 }
 
 var queueListCmd = &cobra.Command{
-	Use:   "ls",
-	Short: "List queues",
+	Use:     "list",
+	Short:   "List queues",
+	Aliases: []string{"ls"},
 	// TODO: Use RunE instead?
 	Run: queueList,
 }
 
 var queueInspectCmd = &cobra.Command{
-	Use:   "inspect QUEUE [QUEUE...]",
+	Use:   "inspect <queue> [<queue>...]",
 	Short: "Display detailed information on one or more queues",
 	Args:  cobra.MinimumNArgs(1),
 	// TODO: Use RunE instead?
 	Run: queueInspect,
+	Example: heredoc.Doc(`
+		$ asynq queue inspect myqueue
+		$ asynq queue inspect queue1 queue2 queue3`),
 }
 
 var queueHistoryCmd = &cobra.Command{
-	Use:   "history QUEUE [QUEUE...]",
+	Use:   "history <queue> [<queue>...]",
 	Short: "Display historical aggregate data from one or more queues",
 	Args:  cobra.MinimumNArgs(1),
 	Run:   queueHistory,
+	Example: heredoc.Doc(`
+		$ asynq queue history myqueue
+		$ asynq queue history queue1 queue2 queue3
+		$ asynq queue history myqueue --days=90`),
 }
 
 var queuePauseCmd = &cobra.Command{
-	Use:   "pause QUEUE [QUEUE...]",
+	Use:   "pause <queue> [<queue>...]",
 	Short: "Pause one or more queues",
 	Args:  cobra.MinimumNArgs(1),
 	Run:   queuePause,
+	Example: heredoc.Doc(`
+		$ asynq queue pause myqueue
+		$ asynq queue pause queue1 queue2 queue3`),
 }
 
 var queueUnpauseCmd = &cobra.Command{
-	Use:   "unpause QUEUE [QUEUE...]",
-	Short: "Unpause one or more queues",
-	Args:  cobra.MinimumNArgs(1),
-	Run:   queueUnpause,
+	Use:     "resume <queue> [<queue>...]",
+	Short:   "Resume (unpause) one or more queues",
+	Args:    cobra.MinimumNArgs(1),
+	Aliases: []string{"unpause"},
+	Run:     queueUnpause,
+	Example: heredoc.Doc(`
+		$ asynq queue resume myqueue
+		$ asynq queue resume queue1 queue2 queue3`),
 }
 
 var queueRemoveCmd = &cobra.Command{
-	Use:   "rm QUEUE [QUEUE...]",
-	Short: "Remove one or more queues",
-	Args:  cobra.MinimumNArgs(1),
-	Run:   queueRemove,
+	Use:     "remove <queue> [<queue>...]",
+	Short:   "Remove one or more queues",
+	Aliases: []string{"rm", "delete"},
+	Args:    cobra.MinimumNArgs(1),
+	Run:     queueRemove,
+	Example: heredoc.Doc(`
+		$ asynq queue rm myqueue
+		$ asynq queue rm queue1 queue2 queue3
+		$ asynq queue rm myqueue --force`),
 }
 
 func queueList(cmd *cobra.Command, args []string) {
@@ -145,12 +170,13 @@ func printQueueInfo(info *asynq.QueueInfo) {
 	bold.Println("Queue Info")
 	fmt.Printf("Name:   %s\n", info.Queue)
 	fmt.Printf("Size:   %d\n", info.Size)
+	fmt.Printf("Groups: %d\n", info.Groups)
 	fmt.Printf("Paused: %t\n\n", info.Paused)
 	bold.Println("Task Count by State")
 	printTable(
-		[]string{"active", "pending", "scheduled", "retry", "archived", "completed"},
+		[]string{"active", "pending", "aggregating", "scheduled", "retry", "archived", "completed"},
 		func(w io.Writer, tmpl string) {
-			fmt.Fprintf(w, tmpl, info.Active, info.Pending, info.Scheduled, info.Retry, info.Archived, info.Completed)
+			fmt.Fprintf(w, tmpl, info.Active, info.Pending, info.Aggregating, info.Scheduled, info.Retry, info.Archived, info.Completed)
 		},
 	)
 	fmt.Println()

tools/asynq/cmd/root.go

@@ -14,11 +14,15 @@ import (
 	"unicode"
 	"unicode/utf8"
 
+	"github.com/MakeNowJust/heredoc/v2"
+	"github.com/fatih/color"
 	"github.com/go-redis/redis/v8"
 	"github.com/hibiken/asynq"
 	"github.com/hibiken/asynq/internal/base"
 	"github.com/hibiken/asynq/internal/rdb"
 	"github.com/spf13/cobra"
+	"github.com/spf13/pflag"
+	"golang.org/x/exp/utf8string"
 
 	homedir "github.com/mitchellh/go-homedir"
 	"github.com/spf13/viper"
@@ -39,10 +43,22 @@ var (
 
 // rootCmd represents the base command when called without any subcommands
 var rootCmd = &cobra.Command{
-	Use:     "asynq",
-	Short:   "A monitoring tool for asynq queues",
-	Long:    `Asynq is a montoring CLI to inspect tasks and queues managed by asynq.`,
+	Use:     "asynq <command> <subcommand> [flags]",
+	Short:   "Asynq CLI",
+	Long:    `Command line tool to inspect tasks and queues managed by Asynq`,
 	Version: base.Version,
+
+	SilenceUsage:  true,
+	SilenceErrors: true,
+
+	Example: heredoc.Doc(`
+		$ asynq stats
+		$ asynq queue pause myqueue
+		$ asynq task list --queue=myqueue --state=archived`),
+	Annotations: map[string]string{
+		"help:feedback": heredoc.Doc(`
+			Open an issue at https://github.com/hibiken/asynq/issues/new/choose`),
+	},
 }
 
 var versionOutput = fmt.Sprintf("asynq version %s\n", base.Version)
@@ -64,22 +80,239 @@ func Execute() {
 	}
 }
 
+func isRootCmd(cmd *cobra.Command) bool {
+	return cmd != nil && !cmd.HasParent()
+}
+
+// displayLine represents a line displayed in the output as '<name> <desc>',
+// where pad is used to pad the name from desc.
+type displayLine struct {
+	name string
+	desc string
+	pad  int // number of rpad
+}
+
+func (l *displayLine) String() string {
+	return rpad(l.name, l.pad) + l.desc
+}
+
+type displayLines []*displayLine
+
+func (dls displayLines) String() string {
+	var lines []string
+	for _, dl := range dls {
+		lines = append(lines, dl.String())
+	}
+	return strings.Join(lines, "\n")
+}
+
+// Capitalize the first word in the given string.
+func capitalize(s string) string {
+	str := utf8string.NewString(s)
+	if str.RuneCount() == 0 {
+		return ""
+	}
+	var b strings.Builder
+	b.WriteString(strings.ToUpper(string(str.At(0))))
+	b.WriteString(str.Slice(1, str.RuneCount()))
+	return b.String()
+}
+
+func rootHelpFunc(cmd *cobra.Command, args []string) {
+	// Display helpful error message when user mistypes a subcommand (e.g. 'asynq queue lst').
+	if isRootCmd(cmd.Parent()) && len(args) >= 2 && args[1] != "--help" && args[1] != "-h" {
+		printSubcommandSuggestions(cmd, args[1])
+		return
+	}
+
+	var lines []*displayLine
+	var commands []*displayLine
+	for _, c := range cmd.Commands() {
+		if c.Hidden || c.Short == "" || c.Name() == "help" {
+			continue
+		}
+		l := &displayLine{name: c.Name() + ":", desc: capitalize(c.Short)}
+		commands = append(commands, l)
+		lines = append(lines, l)
+	}
+	var localFlags []*displayLine
+	cmd.LocalFlags().VisitAll(func(f *pflag.Flag) {
+		l := &displayLine{name: "--" + f.Name, desc: capitalize(f.Usage)}
+		localFlags = append(localFlags, l)
+		lines = append(lines, l)
+	})
+	var inheritedFlags []*displayLine
+	cmd.InheritedFlags().VisitAll(func(f *pflag.Flag) {
+		l := &displayLine{name: "--" + f.Name, desc: capitalize(f.Usage)}
+		inheritedFlags = append(inheritedFlags, l)
+		lines = append(lines, l)
+	})
+	adjustPadding(lines...)
+
+	type helpEntry struct {
+		Title string
+		Body  string
+	}
+	var helpEntries []*helpEntry
+	desc := cmd.Long
+	if desc == "" {
+		desc = cmd.Short
+	}
+	if desc != "" {
+		helpEntries = append(helpEntries, &helpEntry{"", desc})
+	}
+	helpEntries = append(helpEntries, &helpEntry{"USAGE", cmd.UseLine()})
+	if len(commands) > 0 {
+		helpEntries = append(helpEntries, &helpEntry{"COMMANDS", displayLines(commands).String()})
+	}
+	if cmd.LocalFlags().HasFlags() {
+		helpEntries = append(helpEntries, &helpEntry{"FLAGS", displayLines(localFlags).String()})
+	}
+	if cmd.InheritedFlags().HasFlags() {
+		helpEntries = append(helpEntries, &helpEntry{"INHERITED FLAGS", displayLines(inheritedFlags).String()})
+	}
+	if cmd.Example != "" {
+		helpEntries = append(helpEntries, &helpEntry{"EXAMPLES", cmd.Example})
+	}
+	helpEntries = append(helpEntries, &helpEntry{"LEARN MORE", heredoc.Doc(`
+		Use 'asynq <command> <subcommand> --help' for more information about a command.`)})
+	if s, ok := cmd.Annotations["help:feedback"]; ok {
+		helpEntries = append(helpEntries, &helpEntry{"FEEDBACK", s})
+	}
+
+	out := cmd.OutOrStdout()
+	bold := color.New(color.Bold)
+	for _, e := range helpEntries {
+		if e.Title != "" {
+			// If there is a title, add indentation to each line in the body
+			bold.Fprintln(out, e.Title)
+			fmt.Fprintln(out, indent(e.Body, 2 /* spaces */))
+		} else {
+			// If there is no title, print the body as is
+			fmt.Fprintln(out, e.Body)
+		}
+		fmt.Fprintln(out)
+	}
+}
+
+func rootUsageFunc(cmd *cobra.Command) error {
+	out := cmd.OutOrStdout()
+	fmt.Fprintf(out, "Usage: %s", cmd.UseLine())
+	if subcmds := cmd.Commands(); len(subcmds) > 0 {
+		fmt.Fprint(out, "\n\nAvailable commands:\n")
+		for _, c := range subcmds {
+			if c.Hidden {
+				continue
+			}
+			fmt.Fprintf(out, "  %s\n", c.Name())
+		}
+	}
+
+	var localFlags []*displayLine
+	cmd.LocalFlags().VisitAll(func(f *pflag.Flag) {
+		localFlags = append(localFlags, &displayLine{name: "--" + f.Name, desc: capitalize(f.Usage)})
+	})
+	adjustPadding(localFlags...)
+	if len(localFlags) > 0 {
+		fmt.Fprint(out, "\n\nFlags:\n")
+		for _, l := range localFlags {
+			fmt.Fprintf(out, "  %s\n", l.String())
+		}
+	}
+	return nil
+}
+
+func printSubcommandSuggestions(cmd *cobra.Command, arg string) {
+	out := cmd.OutOrStdout()
+	fmt.Fprintf(out, "unknown command %q for %q\n", arg, cmd.CommandPath())
+	if cmd.SuggestionsMinimumDistance <= 0 {
+		cmd.SuggestionsMinimumDistance = 2
+	}
+	candidates := cmd.SuggestionsFor(arg)
+	if len(candidates) > 0 {
+		fmt.Fprint(out, "\nDid you mean this?\n")
+		for _, c := range candidates {
+			fmt.Fprintf(out, "\t%s\n", c)
+		}
+	}
+	fmt.Fprintln(out)
+	rootUsageFunc(cmd)
+}
+
+func adjustPadding(lines ...*displayLine) {
+	// find the maximum width of the name
+	max := 0
+	for _, l := range lines {
+		if n := utf8.RuneCountInString(l.name); n > max {
+			max = n
+		}
+	}
+	for _, l := range lines {
+		l.pad = max
+	}
+}
+
+// rpad adds padding to the right of a string.
+func rpad(s string, padding int) string {
+	tmpl := fmt.Sprintf("%%-%ds ", padding)
+	return fmt.Sprintf(tmpl, s)
+
+}
+
+// lpad adds padding to the left of a string.
+func lpad(s string, padding int) string {
+	tmpl := fmt.Sprintf("%%%ds ", padding)
+	return fmt.Sprintf(tmpl, s)
+}
+
+// indent indents the given text by given spaces.
+func indent(text string, space int) string {
+	if len(text) == 0 {
+		return ""
+	}
+	var b strings.Builder
+	indentation := strings.Repeat(" ", space)
+	lastRune := '\n'
+	for _, r := range text {
+		if lastRune == '\n' {
+			b.WriteString(indentation)
+		}
+		b.WriteRune(r)
+		lastRune = r
+	}
+	return b.String()
+}
+
+// dedent removes any indentation from the given text.
+func dedent(text string) string {
+	lines := strings.Split(text, "\n")
+	var b strings.Builder
+	for _, l := range lines {
+		b.WriteString(strings.TrimLeftFunc(l, unicode.IsSpace))
+		b.WriteRune('\n')
+	}
+	return b.String()
+}
+
 func init() {
 	cobra.OnInitialize(initConfig)
 
+	rootCmd.SetHelpFunc(rootHelpFunc)
+	rootCmd.SetUsageFunc(rootUsageFunc)
+
 	rootCmd.AddCommand(versionCmd)
 	rootCmd.SetVersionTemplate(versionOutput)
 
-	rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "config file to set flag defaut values (default is $HOME/.asynq.yaml)")
-	rootCmd.PersistentFlags().StringVarP(&uri, "uri", "u", "127.0.0.1:6379", "redis server URI")
-	rootCmd.PersistentFlags().IntVarP(&db, "db", "n", 0, "redis database number (default is 0)")
-	rootCmd.PersistentFlags().StringVarP(&password, "password", "p", "", "password to use when connecting to redis server")
-	rootCmd.PersistentFlags().BoolVar(&useRedisCluster, "cluster", false, "connect to redis cluster")
+	rootCmd.PersistentFlags().StringVar(&cfgFile, "config", "", "Config file to set flag defaut values (default is $HOME/.asynq.yaml)")
+	rootCmd.PersistentFlags().StringVarP(&uri, "uri", "u", "127.0.0.1:6379", "Redis server URI")
+	rootCmd.PersistentFlags().IntVarP(&db, "db", "n", 0, "Redis database number (default is 0)")
+	rootCmd.PersistentFlags().StringVarP(&password, "password", "p", "", "Password to use when connecting to redis server")
+	rootCmd.PersistentFlags().BoolVar(&useRedisCluster, "cluster", false, "Connect to redis cluster")
 	rootCmd.PersistentFlags().StringVar(&clusterAddrs, "cluster_addrs",
 		"127.0.0.1:7000,127.0.0.1:7001,127.0.0.1:7002,127.0.0.1:7003,127.0.0.1:7004,127.0.0.1:7005",
-		"list of comma-separated redis server addresses")
+		"List of comma-separated redis server addresses")
 	rootCmd.PersistentFlags().StringVar(&tlsServerName, "tls_server",
-		"", "server name for TLS validation")
+		"", "Server name for TLS validation")
 	// Bind flags with config.
 	viper.BindPFlag("uri", rootCmd.PersistentFlags().Lookup("uri"))
 	viper.BindPFlag("db", rootCmd.PersistentFlags().Lookup("db"))
@@ -118,7 +351,7 @@ func initConfig() {
 // createRDB creates a RDB instance using flag values and returns it.
 func createRDB() *rdb.RDB {
 	var c redis.UniversalClient
-	if useRedisCluster {
+	if viper.GetBool("cluster") {
 		addrs := strings.Split(viper.GetString("cluster_addrs"), ",")
 		c = redis.NewClusterClient(&redis.ClusterOptions{
 			Addrs:     addrs,
@@ -142,7 +375,7 @@ func createInspector() *asynq.Inspector {
 }
 
 func getRedisConnOpt() asynq.RedisConnOpt {
-	if useRedisCluster {
+	if viper.GetBool("cluster") {
 		addrs := strings.Split(viper.GetString("cluster_addrs"), ",")
 		return asynq.RedisClusterClientOpt{
 			Addrs:     addrs,

tools/asynq/cmd/server.go

@@ -12,6 +12,7 @@ import (
 	"strings"
 	"time"
 
+	"github.com/MakeNowJust/heredoc/v2"
 	"github.com/spf13/cobra"
 )
 
@@ -21,13 +22,16 @@ func init() {
 }
 
 var serverCmd = &cobra.Command{
-	Use:   "server",
+	Use:   "server <command> [flags]",
 	Short: "Manage servers",
+	Example: heredoc.Doc(`
+		$ asynq server list`),
 }
 
 var serverListCmd = &cobra.Command{
-	Use:   "ls",
-	Short: "List servers",
+	Use:     "list",
+	Aliases: []string{"ls"},
+	Short:   "List servers",
 	Long: `Server list (asynq server ls) shows all running worker servers
 pulling tasks from the given redis instance.
 

tools/asynq/cmd/stats.go

@@ -16,6 +16,7 @@ import (
 	"time"
 	"unicode/utf8"
 
+	"github.com/MakeNowJust/heredoc/v2"
 	"github.com/fatih/color"
 	"github.com/hibiken/asynq/internal/rdb"
 	"github.com/spf13/cobra"
@@ -24,19 +25,15 @@ import (
 // statsCmd represents the stats command
 var statsCmd = &cobra.Command{
 	Use:   "stats",
-	Short: "Shows current state of the tasks and queues",
-	Long: `Stats (aysnq stats) will show the overview of tasks and queues at that instant.
+	Short: "View current state",
+	Long: heredoc.Doc(`
+	  Stats shows the overview of tasks and queues at that instant.
 
-Specifically, the command shows the following:
-* Number of tasks in each state
-* Number of tasks in each queue
-* Aggregate data for the current day
-* Basic information about the running redis instance
-
-To monitor the tasks continuously, it's recommended that you run this
-command in conjunction with the watch command.
-
-Example: watch -n 3 asynq stats -> Shows current state of tasks every three seconds`,
+	  The command shows the following:
+	    * Number of tasks in each state
+	    * Number of tasks in each queue
+	    * Aggregate data for the current day
+	    * Basic information about the running redis instance`),
 	Args: cobra.NoArgs,
 	Run:  stats,
 }
@@ -59,15 +56,16 @@ func init() {
 }
 
 type AggregateStats struct {
-	Active    int       `json:"active"`
-	Pending   int       `json:"pending"`
-	Scheduled int       `json:"scheduled"`
-	Retry     int       `json:"retry"`
-	Archived  int       `json:"archived"`
-	Completed int       `json:"completed"`
-	Processed int       `json:"processed"`
-	Failed    int       `json:"failed"`
-	Timestamp time.Time `json:"timestamp"`
+	Active      int       `json:"active"`
+	Pending     int       `json:"pending"`
+	Aggregating int       `json:"aggregating"`
+	Scheduled   int       `json:"scheduled"`
+	Retry       int       `json:"retry"`
+	Archived    int       `json:"archived"`
+	Completed   int       `json:"completed"`
+	Processed   int       `json:"processed"`
+	Failed      int       `json:"failed"`
+	Timestamp   time.Time `json:"timestamp"`
 }
 
 type FullStats struct {
@@ -95,6 +93,7 @@ func stats(cmd *cobra.Command, args []string) {
 		}
 		aggStats.Active += s.Active
 		aggStats.Pending += s.Pending
+		aggStats.Aggregating += s.Aggregating
 		aggStats.Scheduled += s.Scheduled
 		aggStats.Retry += s.Retry
 		aggStats.Archived += s.Archived
@@ -155,13 +154,13 @@ func stats(cmd *cobra.Command, args []string) {
 }
 
 func printStatsByState(s *AggregateStats) {
-	format := strings.Repeat("%v\t", 6) + "\n"
+	format := strings.Repeat("%v\t", 7) + "\n"
 	tw := new(tabwriter.Writer).Init(os.Stdout, 0, 8, 2, ' ', 0)
-	fmt.Fprintf(tw, format, "active", "pending", "scheduled", "retry", "archived", "completed")
-	width := maxInt(9 /* defaultWidth */, maxWidthOf(s.Active, s.Pending, s.Scheduled, s.Retry, s.Archived, s.Completed)) // length of widest column
+	fmt.Fprintf(tw, format, "active", "pending", "aggregating", "scheduled", "retry", "archived", "completed")
+	width := maxInt(9 /* defaultWidth */, maxWidthOf(s.Active, s.Pending, s.Aggregating, s.Scheduled, s.Retry, s.Archived, s.Completed)) // length of widest column
 	sep := strings.Repeat("-", width)
-	fmt.Fprintf(tw, format, sep, sep, sep, sep, sep, sep)
-	fmt.Fprintf(tw, format, s.Active, s.Pending, s.Scheduled, s.Retry, s.Archived, s.Completed)
+	fmt.Fprintf(tw, format, sep, sep, sep, sep, sep, sep, sep)
+	fmt.Fprintf(tw, format, s.Active, s.Pending, s.Aggregating, s.Scheduled, s.Retry, s.Archived, s.Completed)
 	tw.Flush()
 }
 

tools/asynq/cmd/task.go

@@ -10,6 +10,7 @@ import (
 	"os"
 	"time"
 
+	"github.com/MakeNowJust/heredoc/v2"
 	"github.com/fatih/color"
 	"github.com/hibiken/asynq"
 	"github.com/spf13/cobra"
@@ -18,143 +19,166 @@ import (
 func init() {
 	rootCmd.AddCommand(taskCmd)
 	taskCmd.AddCommand(taskListCmd)
-	taskListCmd.Flags().StringP("queue", "q", "", "queue to inspect")
-	taskListCmd.Flags().StringP("state", "s", "", "state of the tasks to inspect")
+	taskListCmd.Flags().StringP("queue", "q", "", "queue to inspect (required)")
+	taskListCmd.Flags().StringP("state", "s", "", "state of the tasks; one of { active | pending | aggregating | scheduled | retry | archived | completed } (required)")
 	taskListCmd.Flags().Int("page", 1, "page number")
 	taskListCmd.Flags().Int("size", 30, "page size")
+	taskListCmd.Flags().StringP("group", "g", "", "group to inspect (required for listing aggregating tasks)")
 	taskListCmd.MarkFlagRequired("queue")
 	taskListCmd.MarkFlagRequired("state")
 
 	taskCmd.AddCommand(taskCancelCmd)
 
 	taskCmd.AddCommand(taskInspectCmd)
-	taskInspectCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs")
-	taskInspectCmd.Flags().StringP("id", "i", "", "id of the task")
+	taskInspectCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs (required)")
+	taskInspectCmd.Flags().StringP("id", "i", "", "id of the task (required)")
 	taskInspectCmd.MarkFlagRequired("queue")
 	taskInspectCmd.MarkFlagRequired("id")
 
 	taskCmd.AddCommand(taskArchiveCmd)
-	taskArchiveCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs")
-	taskArchiveCmd.Flags().StringP("id", "i", "", "id of the task")
+	taskArchiveCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs (required)")
+	taskArchiveCmd.Flags().StringP("id", "i", "", "id of the task (required)")
 	taskArchiveCmd.MarkFlagRequired("queue")
 	taskArchiveCmd.MarkFlagRequired("id")
 
 	taskCmd.AddCommand(taskDeleteCmd)
-	taskDeleteCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs")
-	taskDeleteCmd.Flags().StringP("id", "i", "", "id of the task")
+	taskDeleteCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs (required)")
+	taskDeleteCmd.Flags().StringP("id", "i", "", "id of the task (required)")
 	taskDeleteCmd.MarkFlagRequired("queue")
 	taskDeleteCmd.MarkFlagRequired("id")
 
 	taskCmd.AddCommand(taskRunCmd)
-	taskRunCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs")
-	taskRunCmd.Flags().StringP("id", "i", "", "id of the task")
+	taskRunCmd.Flags().StringP("queue", "q", "", "queue to which the task belongs (required)")
+	taskRunCmd.Flags().StringP("id", "i", "", "id of the task (required)")
 	taskRunCmd.MarkFlagRequired("queue")
 	taskRunCmd.MarkFlagRequired("id")
 
 	taskCmd.AddCommand(taskArchiveAllCmd)
-	taskArchiveAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong")
-	taskArchiveAllCmd.Flags().StringP("state", "s", "", "state of the tasks")
+	taskArchiveAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong (required)")
+	taskArchiveAllCmd.Flags().StringP("state", "s", "", "state of the tasks; one of { pending | aggregating | scheduled | retry } (required)")
 	taskArchiveAllCmd.MarkFlagRequired("queue")
 	taskArchiveAllCmd.MarkFlagRequired("state")
+	taskArchiveAllCmd.Flags().StringP("group", "g", "", "group to which the tasks belong (required for archiving aggregating tasks)")
 
 	taskCmd.AddCommand(taskDeleteAllCmd)
-	taskDeleteAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong")
-	taskDeleteAllCmd.Flags().StringP("state", "s", "", "state of the tasks")
+	taskDeleteAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong (required)")
+	taskDeleteAllCmd.Flags().StringP("state", "s", "", "state of the tasks; one of { pending | aggregating | scheduled | retry | archived | completed } (required)")
 	taskDeleteAllCmd.MarkFlagRequired("queue")
 	taskDeleteAllCmd.MarkFlagRequired("state")
+	taskDeleteAllCmd.Flags().StringP("group", "g", "", "group to which the tasks belong (required for deleting aggregating tasks)")
 
 	taskCmd.AddCommand(taskRunAllCmd)
-	taskRunAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong")
-	taskRunAllCmd.Flags().StringP("state", "s", "", "state of the tasks")
+	taskRunAllCmd.Flags().StringP("queue", "q", "", "queue to which the tasks belong (required)")
+	taskRunAllCmd.Flags().StringP("state", "s", "", "state of the tasks; one of { scheduled | retry | archived } (required)")
 	taskRunAllCmd.MarkFlagRequired("queue")
 	taskRunAllCmd.MarkFlagRequired("state")
+	taskRunAllCmd.Flags().StringP("group", "g", "", "group to which the tasks belong (required for running aggregating tasks)")
 }
 
 var taskCmd = &cobra.Command{
-	Use:   "task",
+	Use:   "task <command> [flags]",
 	Short: "Manage tasks",
+	Example: heredoc.Doc(`
+		$ asynq task list --queue=myqueue --state=scheduled
+		$ asynq task inspect --queue=myqueue --id=7837f142-6337-4217-9276-8f27281b67d1
+		$ asynq task delete --queue=myqueue --id=7837f142-6337-4217-9276-8f27281b67d1
+		$ asynq task deleteall --queue=myqueue --state=archived`),
 }
 
 var taskListCmd = &cobra.Command{
-	Use:   "ls --queue=QUEUE --state=STATE",
-	Short: "List tasks",
-	Long: `List tasks of the given state from the specified queue.
+	Use:     "list --queue=<queue> --state=<state> [flags]",
+	Aliases: []string{"ls"},
+	Short:   "List tasks",
+	Long: heredoc.Doc(`
+	List tasks of the given state from the specified queue.
 
-The value for the state flag should be one of:
-- active
-- pending
-- scheduled
-- retry
-- archived
-- completed
+	The --queue and --state flags are required.
 
-List opeartion paginates the result set.
-By default, the command fetches the first 30 tasks.
-Use --page and --size flags to specify the page number and size.
+	Note: For aggregating tasks, additional --group flag is required.
 
-Example: 
-To list pending tasks from "default" queue, run
-  asynq task ls --queue=default --state=pending
-
-To list the tasks from the second page, run
-  asynq task ls --queue=default --state=pending --page=1`,
+	List opeartion paginates the result set. By default, the command fetches the first 30 tasks.
+	Use --page and --size flags to specify the page number and size.`),
+	Example: heredoc.Doc(`
+		$ asynq task list --queue=myqueue --state=pending
+		$ asynq task list --queue=myqueue --state=aggregating --group=mygroup
+		$ asynq task list --queue=myqueue --state=scheduled --page=2`),
 	Run: taskList,
 }
 
 var taskInspectCmd = &cobra.Command{
-	Use:   "inspect --queue=QUEUE --id=TASK_ID",
+	Use:   "inspect --queue=<queue> --id=<task_id>",
 	Short: "Display detailed information on the specified task",
 	Args:  cobra.NoArgs,
 	Run:   taskInspect,
+	Example: heredoc.Doc(`
+		$ asynq task inspect --queue=myqueue --id=f1720682-f5a6-4db1-8953-4f48ae541d0f`),
 }
 
 var taskCancelCmd = &cobra.Command{
-	Use:   "cancel TASK_ID [TASK_ID...]",
+	Use:   "cancel <task_id> [<task_id>...]",
 	Short: "Cancel one or more active tasks",
 	Args:  cobra.MinimumNArgs(1),
 	Run:   taskCancel,
+	Example: heredoc.Doc(`
+		$ asynq task cancel f1720682-f5a6-4db1-8953-4f48ae541d0f`),
 }
 
 var taskArchiveCmd = &cobra.Command{
-	Use:   "archive --queue=QUEUE --id=TASK_ID",
+	Use:   "archive --queue=<queue> --id=<task_id>",
 	Short: "Archive a task with the given id",
 	Args:  cobra.NoArgs,
 	Run:   taskArchive,
+	Example: heredoc.Doc(`
+		$ asynq task archive --queue=myqueue --id=f1720682-f5a6-4db1-8953-4f48ae541d0f`),
 }
 
 var taskDeleteCmd = &cobra.Command{
-	Use:   "delete --queue=QUEUE --id=TASK_ID",
-	Short: "Delete a task with the given id",
-	Args:  cobra.NoArgs,
-	Run:   taskDelete,
+	Use:     "delete --queue=<queue> --id=<task_id>",
+	Aliases: []string{"remove", "rm"},
+	Short:   "Delete a task with the given id",
+	Args:    cobra.NoArgs,
+	Run:     taskDelete,
+	Example: heredoc.Doc(`
+		$ asynq task delete --queue=myqueue --id=f1720682-f5a6-4db1-8953-4f48ae541d0f`),
 }
 
 var taskRunCmd = &cobra.Command{
-	Use:   "run --queue=QUEUE --id=TASK_ID",
+	Use:   "run --queue=<queue> --id=<task_id>",
 	Short: "Run a task with the given id",
 	Args:  cobra.NoArgs,
 	Run:   taskRun,
+	Example: heredoc.Doc(`
+		$ asynq task run --queue=myqueue --id=f1720682-f5a6-4db1-8953-4f48ae541d0f`),
 }
 
 var taskArchiveAllCmd = &cobra.Command{
-	Use:   "archiveall --queue=QUEUE --state=STATE",
+	Use:   "archiveall --queue=<queue> --state=<state>",
 	Short: "Archive all tasks in the given state",
 	Args:  cobra.NoArgs,
 	Run:   taskArchiveAll,
+	Example: heredoc.Doc(`
+		$ asynq task archiveall --queue=myqueue --state=retry
+		$ asynq task archiveall --queue=myqueue --state=aggregating --group=mygroup`),
 }
 
 var taskDeleteAllCmd = &cobra.Command{
-	Use:   "deleteall --queue=QUEUE --state=STATE",
+	Use:   "deleteall --queue=<queue> --state=<state>",
 	Short: "Delete all tasks in the given state",
 	Args:  cobra.NoArgs,
 	Run:   taskDeleteAll,
+	Example: heredoc.Doc(`
+		$ asynq task deleteall --queue=myqueue --state=archived
+		$ asynq task deleteall --queue=myqueue --state=aggregating --group=mygroup`),
 }
 
 var taskRunAllCmd = &cobra.Command{
-	Use:   "runall --queue=QUEUE --state=STATE",
+	Use:   "runall --queue=<queue> --state=<state>",
 	Short: "Run all tasks in the given state",
 	Args:  cobra.NoArgs,
 	Run:   taskRunAll,
+	Example: heredoc.Doc(`
+		$ asynq task runall --queue=myqueue --state=retry
+		$ asynq task runall --queue=myqueue --state=aggregating --group=mygroup`),
 }
 
 func taskList(cmd *cobra.Command, args []string) {
@@ -192,6 +216,17 @@ func taskList(cmd *cobra.Command, args []string) {
 		listArchivedTasks(qname, pageNum, pageSize)
 	case "completed":
 		listCompletedTasks(qname, pageNum, pageSize)
+	case "aggregating":
+		group, err := cmd.Flags().GetString("group")
+		if err != nil {
+			fmt.Println(err)
+			os.Exit(1)
+		}
+		if group == "" {
+			fmt.Println("Flag --group is required for listing aggregating tasks")
+			os.Exit(1)
+		}
+		listAggregatingTasks(qname, group, pageNum, pageSize)
 	default:
 		fmt.Printf("error: state=%q is not supported\n", state)
 		os.Exit(1)
@@ -334,6 +369,27 @@ func listCompletedTasks(qname string, pageNum, pageSize int) {
 		})
 }
 
+func listAggregatingTasks(qname, group string, pageNum, pageSize int) {
+	i := createInspector()
+	tasks, err := i.ListAggregatingTasks(qname, group, asynq.PageSize(pageSize), asynq.Page(pageNum))
+	if err != nil {
+		fmt.Println(err)
+		os.Exit(1)
+	}
+	if len(tasks) == 0 {
+		fmt.Printf("No aggregating tasks in group %q \n", group)
+		return
+	}
+	printTable(
+		[]string{"ID", "Type", "Payload", "Group"},
+		func(w io.Writer, tmpl string) {
+			for _, t := range tasks {
+				fmt.Fprintf(w, tmpl, t.ID, t.Type, sprintBytes(t.Payload), t.Group)
+			}
+		},
+	)
+}
+
 func taskCancel(cmd *cobra.Command, args []string) {
 	i := createInspector()
 	for _, id := range args {
@@ -486,6 +542,17 @@ func taskArchiveAll(cmd *cobra.Command, args []string) {
 		n, err = i.ArchiveAllScheduledTasks(qname)
 	case "retry":
 		n, err = i.ArchiveAllRetryTasks(qname)
+	case "aggregating":
+		group, err := cmd.Flags().GetString("group")
+		if err != nil {
+			fmt.Printf("error: %v\n", err)
+			os.Exit(1)
+		}
+		if group == "" {
+			fmt.Println("error: Flag --group is required for aggregating tasks")
+			os.Exit(1)
+		}
+		n, err = i.ArchiveAllAggregatingTasks(qname, group)
 	default:
 		fmt.Printf("error: unsupported state %q\n", state)
 		os.Exit(1)
@@ -522,6 +589,17 @@ func taskDeleteAll(cmd *cobra.Command, args []string) {
 		n, err = i.DeleteAllArchivedTasks(qname)
 	case "completed":
 		n, err = i.DeleteAllCompletedTasks(qname)
+	case "aggregating":
+		group, err := cmd.Flags().GetString("group")
+		if err != nil {
+			fmt.Printf("error: %v\n", err)
+			os.Exit(1)
+		}
+		if group == "" {
+			fmt.Println("error: Flag --group is required for aggregating tasks")
+			os.Exit(1)
+		}
+		n, err = i.DeleteAllAggregatingTasks(qname, group)
 	default:
 		fmt.Printf("error: unsupported state %q\n", state)
 		os.Exit(1)
@@ -554,6 +632,17 @@ func taskRunAll(cmd *cobra.Command, args []string) {
 		n, err = i.RunAllRetryTasks(qname)
 	case "archived":
 		n, err = i.RunAllArchivedTasks(qname)
+	case "aggregating":
+		group, err := cmd.Flags().GetString("group")
+		if err != nil {
+			fmt.Printf("error: %v\n", err)
+			os.Exit(1)
+		}
+		if group == "" {
+			fmt.Println("error: Flag --group is required for aggregating tasks")
+			os.Exit(1)
+		}
+		n, err = i.RunAllAggregatingTasks(qname, group)
 	default:
 		fmt.Printf("error: unsupported state %q\n", state)
 		os.Exit(1)

tools/go.mod

@@ -1,15 +1,55 @@
 module github.com/hibiken/asynq/tools
 
-go 1.13
+go 1.18
 
 require (
+	github.com/MakeNowJust/heredoc/v2 v2.0.1
 	github.com/fatih/color v1.9.0
+	github.com/gdamore/tcell/v2 v2.5.1
 	github.com/go-redis/redis/v8 v8.11.4
-	github.com/hibiken/asynq v0.21.0
+	github.com/google/go-cmp v0.5.6
+	github.com/hibiken/asynq v0.23.0
 	github.com/hibiken/asynq/x v0.0.0-20220131170841-349f4c50fb1d
+	github.com/mattn/go-runewidth v0.0.13
 	github.com/mitchellh/go-homedir v1.1.0
 	github.com/prometheus/client_golang v1.11.0
-	github.com/spf13/afero v1.1.2 // indirect
 	github.com/spf13/cobra v1.1.1
+	github.com/spf13/pflag v1.0.5
 	github.com/spf13/viper v1.7.0
+	golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136
+)
+
+require (
+	github.com/beorn7/perks v1.0.1 // indirect
+	github.com/cespare/xxhash/v2 v2.2.0 // indirect
+	github.com/dgryski/go-rendezvous v0.0.0-20200823014737-9f7001d12a5f // indirect
+	github.com/fsnotify/fsnotify v1.4.9 // indirect
+	github.com/gdamore/encoding v1.0.0 // indirect
+	github.com/golang/protobuf v1.5.2 // indirect
+	github.com/google/uuid v1.3.0 // indirect
+	github.com/hashicorp/hcl v1.0.0 // indirect
+	github.com/inconshreveable/mousetrap v1.0.0 // indirect
+	github.com/lucasb-eyer/go-colorful v1.2.0 // indirect
+	github.com/magiconair/properties v1.8.1 // indirect
+	github.com/mattn/go-colorable v0.1.4 // indirect
+	github.com/mattn/go-isatty v0.0.11 // indirect
+	github.com/matttproud/golang_protobuf_extensions v1.0.1 // indirect
+	github.com/mitchellh/mapstructure v1.1.2 // indirect
+	github.com/pelletier/go-toml v1.2.0 // indirect
+	github.com/prometheus/client_model v0.2.0 // indirect
+	github.com/prometheus/common v0.26.0 // indirect
+	github.com/prometheus/procfs v0.6.0 // indirect
+	github.com/rivo/uniseg v0.2.0 // indirect
+	github.com/robfig/cron/v3 v3.0.1 // indirect
+	github.com/spf13/afero v1.1.2 // indirect
+	github.com/spf13/cast v1.3.1 // indirect
+	github.com/spf13/jwalterweatherman v1.0.0 // indirect
+	github.com/subosito/gotenv v1.2.0 // indirect
+	golang.org/x/sys v0.0.0-20220318055525-2edf467146b5 // indirect
+	golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf // indirect
+	golang.org/x/text v0.3.7 // indirect
+	golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 // indirect
+	google.golang.org/protobuf v1.26.0 // indirect
+	gopkg.in/ini.v1 v1.51.0 // indirect
+	gopkg.in/yaml.v2 v2.4.0 // indirect
 )

tools/go.sum

@@ -14,6 +14,8 @@ dmitri.shuralyov.com/gpu/mtl v0.0.0-20190408044501-666a987793e9/go.mod h1:H6x//7
 github.com/BurntSushi/toml v0.3.1 h1:WXkYYl6Yr3qBf1K79EBnL4mak0OimBfB0XUf9Vl28OQ=
 github.com/BurntSushi/toml v0.3.1/go.mod h1:xHWCNGjB5oqiDr8zfno3MHue2Ht5sIBksp03qcyfWMU=
 github.com/BurntSushi/xgb v0.0.0-20160522181843-27f122750802/go.mod h1:IVnqGOEym/WlBOVXweHU+Q+/VP0lqqI8lqeDx9IjBqo=
+github.com/MakeNowJust/heredoc/v2 v2.0.1 h1:rlCHh70XXXv7toz95ajQWOWQnN4WNLt0TdpZYIR/J6A=
+github.com/MakeNowJust/heredoc/v2 v2.0.1/go.mod h1:6/2Abh5s+hc3g9nbWLe9ObDIOhaRrqsyY9MWy+4JdRM=
 github.com/OneOfOne/xxhash v1.2.2/go.mod h1:HSdplMjZKSmBqAxg5vPj2TmRDmfkzw+cTzAElWljhcU=
 github.com/alecthomas/template v0.0.0-20160405071501-a0175ee3bccc/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
 github.com/alecthomas/template v0.0.0-20190718012654-fb15b899a751/go.mod h1:LOuyumcjzFXgccqObfd/Ljyb9UuFJ6TxHnclSeseNhc=
@@ -30,11 +32,11 @@ github.com/beorn7/perks v1.0.1/go.mod h1:G2ZrVWU2WbWT9wwq4/hrbKbnv/1ERSJQ0ibhJ6r
 github.com/bgentry/speakeasy v0.1.0/go.mod h1:+zsyZBPWlz7T6j88CTgSN5bM796AkVf0kBD4zp0CCIs=
 github.com/bketelsen/crypt v0.0.3-0.20200106085610-5cbc8cc4026c/go.mod h1:MKsuJmJgSg28kpZDP6UIiPt0e0Oz0kqKNGyRaWEPv84=
 github.com/census-instrumentation/opencensus-proto v0.2.1/go.mod h1:f6KPmirojxKA12rnyqOA5BBL4O983OfeGPqjHWSTneU=
-github.com/cespare/xxhash v1.1.0 h1:a6HrQnmkObjyL+Gs60czilIUGqrzKutQD6XZog3p+ko=
 github.com/cespare/xxhash v1.1.0/go.mod h1:XrSqR1VqqWfGrhpAt58auRo0WTKS1nRRg3ghfAqPWnc=
 github.com/cespare/xxhash/v2 v2.1.1/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
-github.com/cespare/xxhash/v2 v2.1.2 h1:YRXhKfTDauu4ajMg1TPgFO5jnlC2HCbmLXMcTG5cbYE=
 github.com/cespare/xxhash/v2 v2.1.2/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
+github.com/cespare/xxhash/v2 v2.2.0 h1:DC2CZ1Ep5Y4k3ZQ899DldepgrayRUGE6BBZ/cd9Cj44=
+github.com/cespare/xxhash/v2 v2.2.0/go.mod h1:VGX0DQ3Q6kWi7AoAeZDth3/j3BFtOZR5XLFGgcrjCOs=
 github.com/client9/misspell v0.3.4/go.mod h1:qj6jICC3Q7zFZvVWo7KLAzC3yx5G7kyvSDkc90ppPyw=
 github.com/coreos/bbolt v1.3.2/go.mod h1:iRUV2dpdMOn7Bo10OQBFzIJO9kkE559Wcmn+qkEiiKk=
 github.com/coreos/etcd v3.3.13+incompatible/go.mod h1:uF7uidLiAD3TWHmW31ZFd/JWoc32PjwdhPthX9715RE=
@@ -57,6 +59,10 @@ github.com/fatih/color v1.9.0/go.mod h1:eQcE1qtQxscV5RaZvpXrrb8Drkc3/DdQ+uUYCNjL
 github.com/fsnotify/fsnotify v1.4.7/go.mod h1:jwhsz4b93w/PPRr/qN1Yymfu8t87LnFCMoQvtojpjFo=
 github.com/fsnotify/fsnotify v1.4.9 h1:hsms1Qyu0jgnwNXIxa+/V/PDsU6CfLf6CNO8H7IWoS4=
 github.com/fsnotify/fsnotify v1.4.9/go.mod h1:znqG4EE+3YCdAaPaxE2ZRY/06pZUdp0tY4IgpuI1SZQ=
+github.com/gdamore/encoding v1.0.0 h1:+7OoQ1Bc6eTm5niUzBa0Ctsh6JbMW6Ra+YNuAtDBdko=
+github.com/gdamore/encoding v1.0.0/go.mod h1:alR0ol34c49FCSBLjhosxzcPHQbf2trDkoo5dl+VrEg=
+github.com/gdamore/tcell/v2 v2.5.1 h1:zc3LPdpK184lBW7syF2a5C6MV827KmErk9jGVnmsl/I=
+github.com/gdamore/tcell/v2 v2.5.1/go.mod h1:wSkrPaXoiIWZqW/g7Px4xc79di6FTcpB8tvaKJ6uGBo=
 github.com/ghodss/yaml v1.0.0/go.mod h1:4dBDuWmgqj2HViK6kFavaiC9ZROes6MMH2rRYeMEF04=
 github.com/go-gl/glfw v0.0.0-20190409004039-e6da0acd62b1/go.mod h1:vR7hzQXu2zJy9AVAgeJqvqgH9Q5CA+iKCZ2gyEVpxRU=
 github.com/go-kit/kit v0.8.0/go.mod h1:xBxKIO96dXMWWy0MnWVtmwkA9/13aqxPnvrjFYMA2as=
@@ -140,8 +146,8 @@ github.com/hashicorp/mdns v1.0.0/go.mod h1:tL+uN++7HEJ6SQLQ2/p+z2pH24WQKWjBPkE0m
 github.com/hashicorp/memberlist v0.1.3/go.mod h1:ajVTdAv/9Im8oMAAj5G31PhhMCZJV2pPBoIllUwCN7I=
 github.com/hashicorp/serf v0.8.2/go.mod h1:6hOLApaqBFA1NXqRQAsxw9QxuDEvNxSQRwA/JwenrHc=
 github.com/hibiken/asynq v0.19.0/go.mod h1:tyc63ojaW8SJ5SBm8mvI4DDONsguP5HE85EEl4Qr5Ig=
-github.com/hibiken/asynq v0.21.0 h1:uH9XogJhjq/S39E0/DEPWLZQ6hHJ73UiblZTe4RzHwA=
-github.com/hibiken/asynq v0.21.0/go.mod h1:tyc63ojaW8SJ5SBm8mvI4DDONsguP5HE85EEl4Qr5Ig=
+github.com/hibiken/asynq v0.23.0 h1:kmKkNFgqiXBatC8oz94Mer6uvKoGn4STlIVDV5wnKyE=
+github.com/hibiken/asynq v0.23.0/go.mod h1:K70jPVx+CAmmQrXot7Dru0D52EO7ob4BIun3ri5z1Qw=
 github.com/hibiken/asynq/x v0.0.0-20220131170841-349f4c50fb1d h1:Er+U+9PmnyRHRDQjSjRQ24HoWvOY7w9Pk7bUPYM3Ags=
 github.com/hibiken/asynq/x v0.0.0-20220131170841-349f4c50fb1d/go.mod h1:VmxwMfMKyb6gyv8xG0oOBMXIhquWKPx+zPtbVBd2Q1s=
 github.com/hpcloud/tail v1.0.0/go.mod h1:ab1qPbhIpdTxEkNHXyeSf5vhxWSCs/tWer42PpOxQnU=
@@ -167,6 +173,8 @@ github.com/kr/pretty v0.1.0/go.mod h1:dAy3ld7l9f0ibDNOQOHHMYYIIbhfbHSm3C4ZsoJORN
 github.com/kr/pty v1.1.1/go.mod h1:pFQYn66WHrOpPYNljwOMqo10TkYh1fy3cYio2l3bCsQ=
 github.com/kr/text v0.1.0 h1:45sCR5RtlFHMR4UwH9sdQ5TC8v0qDQCHnXt+kaKSTVE=
 github.com/kr/text v0.1.0/go.mod h1:4Jbv+DJW3UT/LiOwJeYQe1efqtUx/iVham/4vfdArNI=
+github.com/lucasb-eyer/go-colorful v1.2.0 h1:1nnpGOrhyZZuNyfu1QjKiUICQ74+3FNCN69Aj6K7nkY=
+github.com/lucasb-eyer/go-colorful v1.2.0/go.mod h1:R4dSotOR9KMtayYi1e77YzuveK+i7ruzyGqttikkLy0=
 github.com/magiconair/properties v1.8.1 h1:ZC2Vc7/ZFkGmsVC9KvOjumD+G5lXy2RtTKyzRKO2BQ4=
 github.com/magiconair/properties v1.8.1/go.mod h1:PppfXfuXeibc/6YijjN8zIbojt8czPbwD3XqdrwzmxQ=
 github.com/mattn/go-colorable v0.0.9/go.mod h1:9vuHe8Xs5qXnSaW/c/ABM9alt+Vo+STaOChaDxuIBZU=
@@ -176,6 +184,8 @@ github.com/mattn/go-isatty v0.0.3/go.mod h1:M+lRXTBqGeGNdLjl/ufCoiOlB5xdOkqRJdNx
 github.com/mattn/go-isatty v0.0.8/go.mod h1:Iq45c/XA43vh69/j3iqttzPXn0bhXyGjM0Hdxcsrc5s=
 github.com/mattn/go-isatty v0.0.11 h1:FxPOTFNqGkuDUGi3H/qkUbQO4ZiBa2brKq5r0l8TGeM=
 github.com/mattn/go-isatty v0.0.11/go.mod h1:PhnuNfih5lzO57/f3n+odYbM4JtupLOxQOAqxQCu2WE=
+github.com/mattn/go-runewidth v0.0.13 h1:lTGmDsbAYt5DmK6OnoV7EuIF1wEIFAcxld6ypU4OSgU=
+github.com/mattn/go-runewidth v0.0.13/go.mod h1:Jdepj2loyihRzMpdS35Xk/zdY8IAYHsh153qUoGf23w=
 github.com/matttproud/golang_protobuf_extensions v1.0.1 h1:4hp9jkHxhMHkqkrB3Ix0jegS5sx/RkqARlsWZ6pIwiU=
 github.com/matttproud/golang_protobuf_extensions v1.0.1/go.mod h1:D8He9yQNgCq6Z5Ld7szi9bcBfOoFv/3dc6xSMkL2PC0=
 github.com/miekg/dns v1.0.14/go.mod h1:W1PPwlIAgtquWBMBEV9nkV9Cazfe8ScdGz/Lj7v3Nrg=
@@ -242,6 +252,8 @@ github.com/prometheus/procfs v0.1.3/go.mod h1:lV6e/gmhEcM9IjHGsFOCxxuZ+z1YqCvr4O
 github.com/prometheus/procfs v0.6.0 h1:mxy4L2jP6qMonqmq+aTtOx1ifVWUgG/TAmntgbh3xv4=
 github.com/prometheus/procfs v0.6.0/go.mod h1:cz+aTbrPOrUb4q7XlbU9ygM+/jj0fzG6c1xBZuNvfVA=
 github.com/prometheus/tsdb v0.7.1/go.mod h1:qhTCs0VvXwvX/y3TZrWD7rabWM+ijKTux40TwIPHuXU=
+github.com/rivo/uniseg v0.2.0 h1:S1pD9weZBuJdFmowNwbpi7BJ8TNftyUImj/0WQi72jY=
+github.com/rivo/uniseg v0.2.0/go.mod h1:J6wj4VEh+S6ZtnVlnTBMWIodfgj8LQOQFoIToxlJtxc=
 github.com/robfig/cron/v3 v3.0.1 h1:WdRxkvbJztn8LMz/QEvLN5sBU+xKpSqwwUO1Pjr4qDs=
 github.com/robfig/cron/v3 v3.0.1/go.mod h1:eQICP3HwyT7UooqI/z+Ov+PtYAWygg1TEWWzGIFLtro=
 github.com/rogpeppe/fastuuid v0.0.0-20150106093220-6724a57986af/go.mod h1:XWv6SoW27p1b0cqNHllgS5HIMJraePCO15w5zCzIWYg=
@@ -305,6 +317,7 @@ golang.org/x/exp v0.0.0-20190121172915-509febef88a4/go.mod h1:CJ0aWSM057203Lf6IL
 golang.org/x/exp v0.0.0-20190306152737-a1d7652674e8/go.mod h1:CJ0aWSM057203Lf6IL+f9T1iT9GByDxfZKAQTCR3kQA=
 golang.org/x/exp v0.0.0-20190510132918-efd6b22b2522/go.mod h1:ZjyILWgesfNpC6sMxTJOJm9Kp84zZh5NQWvqDGG3Qr8=
 golang.org/x/exp v0.0.0-20190829153037-c13cbed26979/go.mod h1:86+5VVa7VpoJ4kLfm080zCjGlMRFzhUhsZKEZO7MGek=
+golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136 h1:A1gGSx58LAGVHUUsOf7IiR0u8Xb6W51gRwfDBhkdcaw=
 golang.org/x/exp v0.0.0-20191030013958-a1ab85dbe136/go.mod h1:JXzH8nQsPlswgeRAPE3MuO9GYsAcnJvJ4vnMwN/5qkY=
 golang.org/x/image v0.0.0-20190227222117-0694c2d4d067/go.mod h1:kZ7UVZpmo3dzQBMxlp+ypCbDeSB+sBbTgSJuh5dn5js=
 golang.org/x/image v0.0.0-20190802002840-cff245a6509b/go.mod h1:FeLwcggjj3mMvU+oOTbSwawSJRM1uh48EjtB4UJZlP0=
@@ -382,15 +395,19 @@ golang.org/x/sys v0.0.0-20201119102817-f84b799fce68/go.mod h1:h1NjWce9XRLGQEsW7w
 golang.org/x/sys v0.0.0-20210112080510-489259a85091/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210124154548-22da62e12c0c/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
 golang.org/x/sys v0.0.0-20210423082822-04245dca01da/go.mod h1:h1NjWce9XRLGQEsW7wpKNCjG9DtNlClVuFLEZdDNbEs=
-golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40 h1:JWgyZ1qgdTaF3N3oxC+MdTV7qvEEgHo3otj+HB5CM7Q=
 golang.org/x/sys v0.0.0-20210603081109-ebe580a85c40/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
+golang.org/x/sys v0.0.0-20220318055525-2edf467146b5 h1:saXMvIOKvRFwbOMicHXr0B1uwoxq9dGmLe5ExMES6c4=
+golang.org/x/sys v0.0.0-20220318055525-2edf467146b5/go.mod h1:oPkhp1MJrh7nUepCBck5+mAzfO9JrbApNNgaTdGDITg=
 golang.org/x/term v0.0.0-20201126162022-7de9c90e9dd1/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
+golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf h1:MZ2shdL+ZM/XzY3ZGOnh4Nlpnxz5GSOhOmtHo3iPU6M=
+golang.org/x/term v0.0.0-20201210144234-2321bbc49cbf/go.mod h1:bj7SfCRtBDWHUb9snDiAeCFNEtKQo2Wmx5Cou7ajbmo=
 golang.org/x/text v0.3.0/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.1-0.20180807135948-17ff2d5776d2/go.mod h1:NqM8EUOU14njkJ3fqMW+pc6Ldnwhi/IjpwHt7yyuwOQ=
 golang.org/x/text v0.3.2/go.mod h1:bEr9sfX3Q8Zfm5fL9x+3itogRgK3+ptLWKqgva+5dAk=
 golang.org/x/text v0.3.3/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
-golang.org/x/text v0.3.6 h1:aRYxNxv6iGQlyVaZmk6ZgYEDa+Jg18DxebPSrd6bg1M=
 golang.org/x/text v0.3.6/go.mod h1:5Zoc/QRtKVWzQhOtBMvqHzDpF6irO9z98xDceosuGiQ=
+golang.org/x/text v0.3.7 h1:olpwvP2KacW1ZWvsR7uQhoyTYvKAupfQrRGBFM352Gk=
+golang.org/x/text v0.3.7/go.mod h1:u+2+/6zg+i71rQMx5EYifcz6MCKuco9NR6JIITiCfzQ=
 golang.org/x/time v0.0.0-20181108054448-85acf8d2951c/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4 h1:SvFZT6jyqRaOeXpc5h/JSfZenJ2O330aBsf7JfSUXmQ=
 golang.org/x/time v0.0.0-20190308202827-9d24e82272b4/go.mod h1:tRJNPiyCQ0inRvYxbN9jk5I+vvW/OXSQhTDSoE431IQ=