Update all .py source files by
$ pyupgrade --py3-only $(git ls-files | grep ".py$")
to modernize the code according to Python 3 syntaxes.
pep8 errors are fixed by
$ autopep8 --select=E127,E128,E501 --max-line-length 79 -r \
--in-place taskflow
Also add the pyupgrade hook to pre-commit to avoid merging additional
Python 2 syntaxes.
Change-Id: Ifd0a0ade9789497482c7937bffd82c48acfb3d78
This library no longer supports Python 2, thus usage of six can be
removed. This also removes workaround about pickle library used in
Python 2 only.
Change-Id: I19d298cf0f402d65f0b142dea0bf35cf992332a9
The networkx 2.x series has been out for two years now and supports
python 3.6 and greater[1]. This patch updates TaskFlow to require
a minimum of networkx 2.1. It also updates the code to support
recent deprecation expiration introduced in the 2.4 release.
[1] https://networkx.github.io/documentation/stable/news.html
Change-Id: Ife31d353ba80824ebc63c8b21ee90943badc8da3
With the release of NetworkX 2.0 the reporting API was
moved to view/iterator model. Many methods were moved from
reporting lists or dicts to iterating over the information.
Methods that used to return containers now return views and
methods that returned iterators have been removed in networkx.
Because of this change in NetworkX 2.0 , taskflow code
have to be changed also to support networkx > 2.0
Change-Id: I23c226f37bd85c1e38039fbcb302a2d0de49f333
Closes-Bug: #1778115
This unifies all the patterns to be graph based so
that they are more uniform and there underlying constraints
are more easy to understand (taskflow basically processes
graphs).
Change-Id: Ib2ab07c1c87165cf40a06508128010887f658391
Link the existing exceptions mentioned to there sphinx doc about
them and also adds a section about when the 'DependencyFailure'
exception is raised.
Also tweaks the class docstring a little, to make it easier to
understand.
Change-Id: Ie4b989444c5ad73660cc61c0c3b2b1702b669087
The docstring is now quite important to know what the
params are (they are no longer as obvious due to the
decider addition) so add a useful docstring to that
method.
This also fixes how the targeted flow add was not passing
allow **kwargs as it should have been and removes the
duplicated docstrings that are in the child class (the
parent class docstrings are just fine).
Change-Id: Idacb7ee9f652fab4bdc762c1b49d7523e46e9a7b
To make it possible to alter the runtime flow via a simple
conditional like structure make it possible to have the graph
flow link function take a decider that is expected to be some
callable that will decide (via a boolean return) whether the
edge should actually be traversed when running. When a decider
returns false; the affected + successors will be set into the
IGNORE state and they will be exempt from future runtime and
scheduling decisions.
Part of blueprint taskflow-conditional-execution
Change-Id: Iab0ee46f86d6b8e747911174d54a7295b3fa404d
Since graphs are composed of 'nodes' it seems more
appropriate to use that terminology (which is known
to people with graph experience) instead of the non
descriptive and/or standard 'item' terminology.
Change-Id: I04a4521386d3cdf7e58fb9fa8cf26c00443c2cf6
To make it possible for users to subclass this flow and
replace this method (if they so choose to) make it a static
method that is directed to the current module level function
and call into the staticmethod when used (users can override
staticmethods and change them in subclasses).
Change-Id: I747c45636596ca5a8ad81dabcbba12ab55ce77d7
Instead of using always using a set/dict which do not retain
use a ordered set and a ordered dict for requires, optional,
and provides and rebind mappings types so that the ordering
of these containers is maintained later when they are used.
These ordering can be useful depending on the atom type (such
as in a map and reduce tasks).
Partial-Bug: 1357117
Change-Id: I365d11bbba4aa221bc36ca15441acecf199b4d56
Instead of using strings (which can be easy to
mistype and get wrong), provide a set of constants
that can be used to attach and use these keys in
flows and at compilation.
This also helps make it more clear what the keys
do and where they are used.
Change-Id: I5283b27617961136a4582bbcfce4617f05e8dd1d
In order to make it possible to have a symbol
tree we need to relax the constraints that are
being imposed by the graph flow.
Part of blueprint taskflow-improved-scoping
Change-Id: I2e14de2131c3ba4e3e4eb3108477583d0f02dae2
In order to make it possible to have a symbol
tree we need to relax and remove the constraints
that are being imposed by the unordered constraints
and later move those constraint checks and validations
into the engines compilation stage.
Part of blueprint taskflow-improved-scoping
Change-Id: I80718b4bc01fbf0dce6a95cd2fac7e6e2e1814d1
In order to make it possible to have a symbol
tree we need to relax and remove the constraints
that are being imposed by the linear constraints
and later move those constraint checks and validations
into the engines compilation stage.
Part of blueprint taskflow-improved-scoping
Change-Id: I6efdc821ff991e83572d89f56be5c678d007f9f8
To complement the future changes in patterns we also want
to allow the execution of patterns to be affected in a similar
manner so that symbol lookup is no longer as confined as it was.
This change adds in the following:
- Symbol lookup by walking through an atoms contained scope/s.
- Better error messaging when symbols are not found.
- Adjusted & new tests (existing ones work).
- Better logging of the symbol lookup mechanism (helpful
during debugging, although it is very verbose...)
Part of blueprint taskflow-improved-scoping
Change-Id: Id921a4abd9bf2b7b5c5a762337f8e90e8f1fe194
Instead of blindly assuming all the symbols that
are provided automatically work for all flows even
if the flow has ordering constraints we should set
the base flow class requires property to be abstract
and provide flow specific properties that can do the
appropriate analysis to determine what the flows
unsatisfied symbol requirements actually are.
Part of blueprint taskflow-improved-scoping
Change-Id: Ie149c05b3305c5bfff9d9f2c05e7e064c3a6d0c7
Update hacking to the new requirements version and
fix about half of the new reported issues. The other
hacking issues are for now ignored until fixed by
adjusting our tox.ini file.
This commit fixes the following new hacking errors:
H405 - multi line docstring summary not separated
with an empty line
E265 - block comment should start with '# '
F402 - import 'endpoint' from line 21 shadowed by
loop variable
Change-Id: I6bae61591fb988cc17fa79e21cb5f1508d22781c
Code that calculates provides and requires for flow is almost
identical for all patterns, so this change makes it completely
identical and puts it to the base class. Other patterns are
still allowed to override these properties for sake of customization
or optimization.
Change-Id: I6e875e863047b5287ec727fc9a491f252f144ecf
Most of the utility graph functions we have can
be connected to a directed graph class that itself
derives (and adds on to) the networkx base class.
Doing this allows for functionality that isn't exposed
in networkx to be exposed in our subclass (which is a
useful pattern to have).
It also makes it possible (if ever needed) to replace
the networkx usage in taskflow with something else if
this ever becomes a major request.
Change-Id: I0a825d5637236d7b5dbdbda0d426adb0183d5ba3
Group the exceptions into the following groups
* Storage
* Jobs
* Execution
* Other (wrapped failure here)
This grouping makes it easier to understand where
one type of exception should be used vs using another
type of exception.
Backwards incompatible changes:
* StorageError -> StorageFailure
* AlreadyExists -> Duplicate
* WrappedFailure now inherits from Exception
and not directly from TaskFlowException since it
wraps arbitrary other exceptions and is not
specific to taskflow.
Cleanups:
* JobNotFound -> NotFound
* EmptyFlow -> Empty
* JobAlreadyExists -> AlreadyExists
* InvariantViolation (X)
* ConnectionFailure (X)
Change-Id: I0e1e81b513fbbc7adb8bfaa1244993e345ab70d3
This commit adds unit tests that check graph flow methods without
executing or flattening it. Now-redundant tests from other test
suites are deleted.
Change-Id: I8dafe0f9b295428831eddb3f9fd48f042d2f1ffc
- add 'invariant': True to all links produced by linear flow;
- write new unit tests for basic functionality of linear flow
pattern;
- remove some simple now-redundant unit tests from
test_flow_dependencies.
Change-Id: I1f4fa8a1dfc61485555a10e8d0629a67aab1997f
It is not needed. Also, using it complicates transitioning
to another pattern (e.g. graph flow) and encourages bad
programming practices.
Change-Id: I9d0a4646a27df1a11ed436f26a0cb074b717647e
In addition to iteration over its children (atoms or subflows) each
pattern now provides iter_links() method that iterates over
dependency links. This allows engines to treat all patterns
in the same way instead knowing what structure each pattern expresses.
Change-Id: I52cb5b0b501eefc8eb56a9ef5303aeb318013e11
Implement dependency checks when adding items to flows:
- retry can't be depended on tasks from it's subflow.
- retry can't provide same values as tasks or other retries
Change-Id: Iffa8e673fd2de39407ae22cd38ad523d484cbba7
* Added missing period for doc strings
* Correct syntax errors
* Remove H402 from flake8 ignore list
Change-Id: Ia8592bf99378e3658d6cca2ceb148bf9eb0b5de8
Most TaskFlow exception don't have 'Exception' suffix in their names, so
we drop it from the few that have it, for sake of consistency and more
concise and expressive code.
MissingDependencies parent object is changed from InvalidState to
InvariantViolation, which is more appropriate.
Unused ClosedException is dropped.
Breaking change: any client code that used renamed exceptions,
ClosedException or relied on catching MissingDependencies as
InvariantViolationException is affected.
Change-Id: I649b7760d382ccb6df333bdffd667ba6b67a431e
Check is added to linear_flow and unordered_flow patterns that
there are no two child items provide same values.
Unit tests added.
Change-Id: I7a293d58a962de37ba2fd1dd989bf0a218417430
Closes-Bug: #1264995
To retain the functionality where we can chain
function calls to add, link we need to return the
object we are acting on (instead of none).
Change-Id: Iadf0f9d3a21b6d079c1edd29063196180f5222ed
This change adds graph flow like pattern that is able to ignore graph
nodes (not even add them to flow) if they are not needed to run certain
task.
Implements blueprint targeted-graph-flow
Change-Id: I57d5a1c0598bd032d77e2a262b2d9644418ce3f6
In a way our resumption works it may not correspond uuid from flow
details, and so it is hardly useful.
Change-Id: I090d017e2f0f8475594af94a2430a34e6ed1ce70
Reasons are useful for later analysis when running
as to why the edge between two nodes was created so
when linking items in the graph it would be nice to
assign a reason.
Change-Id: I2185cf5fb3c2c07c0f5536d3b210080c6f61d5dd
Since a linear flow is a linear set of tasks or subflows
we can easily add in an accessor that can fetch these
items by index which is useful for indexing for any
post-add operations.
Change-Id: Ib1690584a08e24fae29285225e6c519b06ce2ddd
Instead of allowing a direct graph return of the underlying
graph_flow graph we should return a frozen version instead so that
users of the returned value can not mutate the graph without going
through the graph_flow pattern (which could have undesired and
harmful side-effects if this occurs).
Change-Id: I38b35119d6e7bd7387b8ab467eba53aee5500629
