...
Tawesoft Logo

Package loader

import "tawesoft.co.uk/go/loader"
Overview
Index
Subdirectories

Overview ▾

Package loader implements the ability to define a graph of tasks and dependencies, classes of synchronous and concurrent workers, and limiting strategies, and solve the graph incrementally or totally.

For example, this could be used to implement a loading screen for a computer game with a progress bar that updates in real time, with images being decoded concurrently with files being loaded from disk, and synchronised with the main thread for safe OpenGL operations such as creating texture objects on the GPU.

While this package is generally suitable for use in real world applications, we are waiting to get some experience with how it works for us in an internal application before polishing or committing to a stable API.

TODO: doesn't yet free temporary results

TODO: refactor the load loop to always send/receive at the same time

TODO: clean up generally

TODO: not decided about the API for Loader.Result (but loader.MustResult is ok)

TODO: a step to simplify the DAG to remove passthrough loader.NamedTask steps

Examples

Configure the Loader with a Strategy to limit concurrent connections per host

https://www.tawesoft.co.uk/go/doc/loader/examples/limit-connections-per-host/

Package Information

License: MIT (see LICENSE.txt)

Stable: no

For more information, documentation, source code, examples, support, links, etc. please see https://www.tawesoft.co.uk/go and https://www.tawesoft.co.uk/go/loader

type ConsumerID

ConsumerID uniquely identifies a consumer in a given Loader.

See the NewConsumer() method on the Loader type.

type ConsumerID int

type Loader

Loader is used to manage a graph of Task items to be completed synchronously or concurrently with different types of work divided among a set of Consumer objects.

type Loader struct {
    // contains filtered or unexported fields
}

func New

func New() *Loader

Returns a new Loader

func (*Loader) Add

func (l *Loader) Add(tasks []Task) error

Add adds a graph of Tasks to the Loader to later process with its Load() or LoadAll() methods. Tasks may be added even during or after a Load() loop.

Because Task names, even at the top level of the array, are scoped to this function, Two Tasks across a Loader Add() boundary cannot refer to each other by name. If this behaviour is desired, append to a Task array and send the combined array in one call to Add.

An error is generated if a named Task requirement is not in scope. In this event, the state of the task dependency graph is undefined and no methods on the Loader, other than Close, may be called.

func (*Loader) Close

func (l *Loader) Close()

Close TODO

func (*Loader) Load

func (l *Loader) Load(budget time.Duration) (Progress, error)

Load completes as many loading tasks as possible within the time budget. If idle while waiting for concurrent results, it may return early.

See also the LoadAll() method.

func (*Loader) LoadAll

func (l *Loader) LoadAll() (Progress, error)

LoadAll completes all loading tasks and blocks until finished

func (*Loader) MustResult

func (l *Loader) MustResult(name string) interface{}

func (*Loader) NewConsumer

func (l *Loader) NewConsumer(concurrency int, strategy Strategy) ConsumerID

NewConsumer creates a task consumer that performs the (possibly asynchronous) completion of tasks at a given level of concurrency (e.g. number of goroutines) and returns an opaque ID that uniquely identifies that consumer with the active Loader.

A concurrency of zero means that the consumer's tasks will be performed sequentially on the same thread as the Loader's Load() or LoadAll() methods.

The strategy argument allows control over temporarily delaying a task. Strategy may be nil to always accept.

The special ConsumerID of zero corresponds to a default builtin consumer that has a concurrency of zero and a nil strategy.

func (*Loader) Result

func (l *Loader) Result(name string) (workResult, bool)

Returns a named result from a Task where Keep is true and the Name is unique across all Tasks where Keep is True.

type Progress

Progress holds the result of a Loader's Load or LoadAll methods and represents progress in completing all Task items.

type Progress struct {
    Completed int
    Remaining int
    Total     int
    Done      bool
}

type Strategy

Strategy allows temporarily delaying of a task based on other currently progressing tasks e.g. see the examples folder for an implementation that avoids multiple concurrent connections to a single host.

Note: do not share the same instance of a Strategy across two consumers without making it thread safe e.g. use of mutexes.

Note: the Strategy must give a constant answer for the same sequence of Start and End methods and arguments i.e. must not depend on an external factor such as time or user input but must only depend on some innate property of the currently accepted tasks.

Note: if not used carefully, a Strategy, or the interaction between two Strategies, may lead to deadlocks. This may be the case if there is any construction of a Task dependency graph (considering only those Tasks that may be delayed by a Strategy), or any subgraph thereof formidable by removing leaf nodes, where the Strategy's lower bound on the number of simultaneous tasks is less than or equal to the number of leaf nodes minus the number of vertex disjoint subgraphs of that graph.

type Strategy interface {
    // Start takes the result of a task Info() and returns true if the task is
    // accepted, or false if the task must be temporarily delayed.
    Start(info interface{}) bool

    // End takes the result of a task Info() and registers that the task has
    // completed processing.
    End(info interface{})
}

type Task

Task is a (possibly recursively nested) unit of work that is to be performed (possibly concurrently), subject to dependencies and limits.

type Task struct {
    // Optional name of the task. Used to reference a task as a dependency of
    // another and to retrieve its result, if kept. Does not have to be unique
    // (it is scoped to its subtasks and successor siblings in one call to a
    // Loader Add method), unless it is kept (see the Keep field, below).
    Name string

    // Keep indicates that the task's Load() result will be available from
    // the Loader Result() method by the task Name. If Keep is true, the
    // task's Name must be globally unique across all Tasks kept by the loader.
    // If Keep is false, the task's name does not have to be unique, even if a
    // kept Task has the same name.
    Keep bool

    // Load performs the (possibly asynchronous) completion of the task e.g.
    // reading a file from disk, a unit of computation, etc.
    //
    // The results argument is the ordered results of the tasks in
    // RequiresNamed (if any) followed by the ordered results of the tasks in
    // RequiresDirect (if any).
    Load func(results ...interface{}) (interface{}, error)

    // Free performs the (possibly asynchronous) removal of a task's Load()
    // result e.g. releasing memory. May be nil.
    Free func(i interface{})

    // RequiresNamed is an array of names of tasks required to complete first
    // as a dependency of this task. May be nil.
    //
    // Note that a required named task must be defined before a task can depend
    // on it (e.g. by appearing earlier in the array passed to Loader Add()).
    RequiresNamed []string

    // RequiresDirect is an array of tasks required to complete first as a
    // dependency of this task. May be nil. These "subtasks" are in a new scope
    // for naming purposes.
    RequiresDirect []Task

    // Consumer performs the asynchronous completion of tasks at a given level
    // of concurrency. Use an ID returned by a Loader Consumer() method. May be
    // zero, in which case the task is completed in the same thread as the
    // caller Loader's Load() or LoadAll() methods.
    Consumer ConsumerID

    // Info returns some value for a task understood by the given Strategy.
    // May be nil. Must be a constant function i.e. always return the same
    // value for a given Task.
    Info func() interface{}
}

func NamedTask

func NamedTask(name string) Task

NamedTask is used to reference another task by name as a subtask i.e. in a Task's RequiresDirect instead of RequiresNamed.

TODO NamedTasks should be simplified in the DAG to remove the node entirely.

Subdirectories

Name Synopsis
examples
dev Configure the loader to limit concurrent connections per host
limit-connections-per-host Configure the Loader with a Strategy to limit concurrent connections per host