Merge branch 'master' of https://github.com/wfcommons/wfcommons

Author: tainagdcoleman

docs/source/analyzing_instances.rst

@@ -1,37 +1,39 @@
-.. _traces-label:
+.. _instances-label:
 
 Analyzing Instances
-================
+===================
 
 Workflow execution instances have been widely used to profile and characterize
 workflow executions, and to build distributions of workflow execution behaviors,
 which are used to evaluate methods and techniques in simulation or in real
 conditions.
 
-The first axis of the WfCommons project targets the analysis of actual workflow
-execution instances (i.e., the workflow execution profile data and characterizations)
-in order to build **recipes** of workflow applications. These recipes contain
-the necessary information for generating synthetic, yet realistic, workflow
-instances that resemble the structure and distribution of the original workflow
-executions.
+The WfCommons project targets the analysis of actual workflow execution instances
+(i.e., the workflow execution profile data and characterizations)
+in order to build :ref:`workflow-recipe-label` of workflow applications.
+These recipes contain the necessary information for generating synthetic, yet
+realistic, workflow instances that resemble the structure and distribution of
+the original workflow executions.
 
 A `list of workflow execution instances <https://wfcommons.org/instances>`_
 that are compatible with :ref:`json-format-label` is kept constantly updated
 in our project website.
 
-Workflow Execution Instances
--------------------------
+.. _wfinstances-label:
+
+WfInstances
+-----------
 
 A workflow execution instance represents an actual execution of a scientific
 workflow on a distributed platform (e.g., clouds, grids, HPC, etc.). In the
 WfCommons project, an instance is represented in a JSON file following the
-schema described in :ref:`json-format-label` section. This Python package
+schema described in :ref:`json-format-label`. This Python package
 provides an *instance loader* tool for importing workflow execution instances
 for analysis. For instance, the code snippet below shows how an instance can
 be loaded using the :class:`~wfcommons.trace.trace.Trace` class: ::
 
     from wfcommons import Trace
-    trace = Trace(input_trace='/path/to/trace/file.json')
+    trace = Trace(input_trace='/path/to/instance/file.json')
 
 The :class:`~wfcommons.trace.trace.Trace` class provides a number of
 methods for interacting with the workflow instance, including:
@@ -41,16 +43,21 @@ methods for interacting with the workflow instance, including:
 - :meth:`~wfcommons.trace.trace.Trace.roots`: gets the roots of the workflow (i.e., the tasks without any predecessors).
 - :meth:`~wfcommons.trace.trace.Trace.write_dot`: writes a dot file of the instance.
 
+.. note::
+    Although the analysis methods are inherently used by WfCommons (specifically
+    WfChef) for :ref:`generating-workflows-recipe-label`, they can also be used
+    in a standalone manner.
+
 The Instance Analyzer
-------------------
+---------------------
 
 The :class:`~wfcommons.trace.trace_analyzer.TraceAnalyzer` class provides
 a number of tools for analyzing collection of workflow execution instances. The
 goal of the :class:`~wfcommons.trace.trace_analyzer.TraceAnalyzer` is to
 perform analyzes of one or multiple workflow execution instances, and build
 summaries of the analyzes per workflow' task type prefix.
 
-.. note::
+.. warning::
 
     Although any workflow execution instance represented as a
     :class:`~wfcommons.trace.trace.Trace` object (i.e., compatible with
@@ -89,7 +96,7 @@ summary showing the best fit probability distribution for runtime of the
         ...
     }
 
-Workflow analysis summaries can then be used to develop :ref:`workflow-recipe-label`,
+Workflow analysis summaries are used by WfChef to develop :ref:`workflow-recipe-label`,
 in which themselves are used to :ref:`generate realistic synthetic workflow instances
 <generating-workflows-label>`.
 
@@ -113,7 +120,7 @@ plots (runtime, and input and output files) into the :code:`fits` folder using
     from os.path import isfile, join
 
     # obtaining list of instance files in the folder
-    INSTANCES_PATH = "/Path/to/some/instance/folder/"
+    INSTANCES_PATH = "/path/to/some/instance/folder/"
     instance_files = [f for f in listdir(INSTANCES_PATH) if isfile(join(INSTANCES_PATH, f))]
 
     # creating the instance analyzer object

docs/source/generating_workflows_recipe.rst

@@ -1,17 +1,18 @@
 .. _generating-workflows-recipe-label:
 
-Generating Workflows Recipe
+Generating Workflows Recipes
 ============================
 
 **WfChef** is the WfCommons component that automates the construction of
 synthetic workflow generators for any given workflow application. The input
 to this component is a set of real workflow instances described in the
-*WfFormat* (e.g., instances available in **WfInstances**).
-WfChef automatically analyzes the real workflow instances for
+:ref:`json-format-label` (e.g., instances available in **WfInstances**).
+WfChef automatically analyzes a set of real workflow instances for
 two purposes. First, it discovers workflow subgraphs that represent
 fundamental task dependency patterns. Second, it derives
-statistical models of the workflow tasks' performance characteristics (more details :ref:`.. _traces-label:`).
-WfChef then outputs a **recipe** that will be used by **WfGen** 
+statistical models of the workflow tasks' performance characteristics
+(see :ref:`instances-label`).
+WfChef then outputs a **recipe** that will be used by **WfGen**
 (see :ref:`generating-workflows-label`) to generate realistic synthetic  
 workflow instances with any arbitrary number of tasks.
 
@@ -20,12 +21,12 @@ workflow instances with any arbitrary number of tasks.
 Workflow Recipes
 ----------------
 
-A **workfflow recipe** is a data structure that encodes the discovered pattern occurrences 
+A **workflow recipe** is a data structure that encodes the discovered pattern occurrences
 as well as the statistical models of workflow task characteristics.
 The WfCommons package provides a number of *workflow recipes* for generating realistic 
 synthetic workflow instances. 
 
-All workflow recipes provide a common method, :code:`from_num_tasks`, that defines the upper 
+All workflow recipes provide a common method, :code:`from_num_tasks`, that defines the lower
 bound for the total number of tasks in the synthetic workflow.
 
 

docs/source/images/wfcommons.png

@@ -5,12 +5,8 @@
 
 `WfCommons <https://wfcommons.org>`__ is a community framework for
 enabling scientific workflow research and development. This Python package
-provides a collection of tools for:
-
-- Analyzing instances of actual workflow executions;
-- Producing recipes structures for creating workflow recipes for workflow
-  generation; and
-- Generating synthetic realistic workflow instances.
+provides methods for analyzing instances, deriving recipes, and generating
+representative synthetic workflow instances.
 
 .. figure::  images/wfcommons.png
    :scale:   90 %
@@ -46,6 +42,7 @@ support@wfcommons.org.
     introduction.rst
     parsing_logs.rst
     analyzing_instances.rst
+    generating_workflows_recipe.rst
     generating_workflows.rst
 
 .. toctree::

docs/source/index.rst

@@ -1,60 +1,67 @@
 The WfCommons Project
 =======================
 
-The `WfCommons project <https://wfcommons.org>`_ is a community framework
+The `WfCommons project <https://wfcommons.org>`_ is an open source framework
 for enabling scientific workflow research and development by providing foundational
 tools for analyzing workflow execution instances, and generating synthetic, yet
 realistic, workflow instances that can be used to develop new techniques, algorithms
 and systems that can overcome the challenges of efficient and robust execution of
-ever larger workflows on increasingly complex distributed infrastructures. The
-figure below shows an overview of the workflow research life cycle process that
-integrates the three axis of the WfCommons project:
+ever larger workflows on increasingly complex distributed infrastructures.
+
+The figure below shows an overview of the research and development life cycle that
+integrates the four major components WfCommons: (i) workflow execution instances
+(**WfInstances**), (ii) workflow recipes (**WfChef**), (iii) workflow generator
+(**WfGen**), and (iv) workflow simulator (**WfSim**).
 
 .. figure::  images/wfcommons.png
    :align:   center
 
    The WfCommons conceptual architecture.
 
-The *first axis* (**Workflow Instances**) of the WfCommons project targets the
-collection and curation of open access production workflow executions from
-various scientific applications shared in a common instance format (i.e.,
-:ref:`json-format-label`). We keep a `list of workflow execution instances
+**WfInstances.**
+The WfInstances component provides a collection and curation of open-access
+production workflow instances from various scientific applications, all made
+available using a common format (i.e., :ref:`json-format-label`).
+A workflow instance is built based on logs of an actual execution of a scientific
+workflow on a distributed platform (e.g., clouds, grids, clusters) using a
+workflow system. We keep a `list of workflow execution instances
 <https://wfcommons.org/instances>`_ in our project website.
 
-The *second axis* (**Workflow Generator**) of the WfCommons project targets
-the generation of realistic synthetic workflow instances based on workflow execution
-profiles extracted from execution instances. We are constantly seeking for additional
-workflow execution instances for refining or developing new workflow recipes for
-the WfCommons's workflow generator.
-
-The *third axis* (**Workflow Simulator**) of the WfCommons project fosters the
-use of simulation for the development, evaluation, and verification of scheduling
-and resource provisioning algorithms (e.g., multi-objective function optimization,
-etc.), evaluation of current and emerging computing platforms (e.g., clouds, IoT,
-extreme scale, etc.), among others. We keep a `list of open source workflow
-management systems simulators and simulation frameworks
-<https://wfcommons.org/simulation>`_ that provide support for the WfCommons
-JSON format in our project website.
+**WfChef.**
+The WfChef component automates the construction of synthetic workflow generators
+(recipes) for any given workflow application. The input to this component is a set
+of real workflow instances described in the :ref:`json-format-label` (e.g.,
+instances available in WfInstances).
 
-This Python package provides a collection of tools for:
+**WfGen.**
+The WfGen component targets the generation of realistic synthetic workflow instances.
+WfGen takes as input a workflow recipe produced by WfChef for a particular application
+and a desired number of tasks. WfGen then automatically generates synthetic, yet
+realistic, randomized workflow instances with (approximately) the desired number of
+tasks.
 
-- Analyzing instances of actual workflow executions;
-- Producing recipes structures for creating workflow recipes for workflow
-  generation; and
-- Generating synthetic realistic workflow instances.
+**WfSim.**
+The WfCommons project fosters the use of simulation for the development, evaluation,
+and verification of scheduling and resource provisioning algorithms (e.g.,
+multi-objective function optimization, etc.), evaluation of current and emerging
+computing platforms (e.g., clouds, IoT, extreme scale, etc.), among others.
+We do not develop simulators as part of the WfCommons project. Instead, the WfSim
+component catalogs open source WMS simulators that provide support for
+:ref:`json-format-label`. We keep a `list of open source workflow
+management systems simulators and simulation frameworks
+<https://wfcommons.org/simulation>`_ on our project website.
 
 .. _json-format-label:
 
-The WfCommons JSON Format
----------------------------
+WfFormat
+--------
 
 The WfCommons project uses a common format for representing workflow execution
-instances and generated synthetic workflows instances, so that workflow simulators and
-simulation frameworks (that provide support for WfCommons format) can use
-such instances interchangeably. This common format uses a JSON specification
-available in the
-`WfCommons JSON schema GitHub <https://github.com/wfcommons/workflow-schema>`_
+instances and generated synthetic workflows instances. Workflow simulators and
+simulation frameworks that support WfFormat can then use both types of instances
+interchangeably. WfFormat uses a JSON specification available in the
+`WfFormat Schema GitHub <https://github.com/wfcommons/workflow-schema>`_
 repository. The current version of the WfCommons Python package uses the schema
 version :code:`1.1`. The schema GitHub repository provides detailed explanation
-of the WfCommons JSON format (including required fields), and also a validator
-script for verifying the compatibility of instances.
+of WfFormat (including required fields), and also a validator script for verifying
+the compatibility of instances.

docs/source/introduction.rst

@@ -3,9 +3,11 @@
 Parsing Workflow Execution Logs
 ===============================
 
-The most common way for obtaining instances from actual workflow executions is to parse
-execution logs. As part of the WfCommons project, we are constantly developing
-parsers for commonly used workflow management systems.
+The most common way for obtaining **workflow instances** from actual workflow
+executions is to parse execution logs. As part of the WfCommons project, we
+are constantly developing parsers for commonly used workflow management systems.
+The parsers provided in this Python package automatically scans execution logs
+to produce instances using :ref:`json-format-label`.
 
 Each parser class is derived from the abstract
 :class:`~wfcommons.trace.logs.abstract_logs_parser.LogsParser` class. Thus, each
@@ -18,7 +20,7 @@ Makeflow
 
 `Makeflow <http://ccl.cse.nd.edu/software/makeflow/>`_ is a workflow system for
 executing large complex workflows on clusters, clouds, and grids. The Makeflow
-language is similar to traditional Make, so if you can write a Makefile, then you
+language is similar to traditional "Make", so if you can write a Makefile, then you
 can write a Makeflow. A workflow can be just a few commands chained together, or
 it can be a complex application consisting of thousands of tasks. It can have an
 arbitrary DAG structure and is not limited to specific patterns. Makeflow is used

docs/source/parsing_logs.rst

@@ -13,75 +13,3 @@ wfcommons.generator.generator
    :members:
    :undoc-members:
    :show-inheritance:
-
-wfcommons.generator.workflow.blast\_recipe
---------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.blast_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.bwa\_recipe
-------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.bwa_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.cycles\_recipe
----------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.cycles_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.epigenomics\_recipe
---------------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.epigenomics_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.genome\_recipe
----------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.genome_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.montage\_recipe
-----------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.montage_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.seismology\_recipe
--------------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.seismology_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.soykb\_recipe
---------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.soykb_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:
-
-wfcommons.generator.workflow.srasearch\_recipe
-------------------------------------------------
-
-.. automodule:: wfcommons.generator.workflow.srasearch_recipe
-   :members:
-   :undoc-members:
-   :show-inheritance:

docs/source/user_api_generator.rst

@@ -8,3 +8,5 @@ requests>=2.24.0
 scipy>=1.5.2
 setuptools>=49.3.1
 pyyaml>=5.3.1
+pandas>=1.2.4
+stringcase>=1.2.0