Merge pull request #651 from joonaskuisma/enhancement

Major release: Improve subprocess handling, artifact management, and ordering logic

Author: joonaskuisma

.github/workflows/update-status-label.yml

@@ -4,7 +4,7 @@ on:
   issues:
     types: [opened, closed]
   pull_request:
-    types: [opened, edited, synchronize, closed]
+    types: [closed]
 
 jobs:
   update-status:
@@ -76,24 +76,7 @@ jobs:
               const pr = context.payload.pull_request;
               const prNumber = pr.number;
               await removeOldStatusLabels(prNumber);
-
-              if (['opened', 'edited', 'synchronize'].includes(context.payload.action)) {
-                const body = pr.body || "";
-                const linkedIssue = body.match(/(?:fixes|closes|resolves)\s+#\d+/i);
-                if (linkedIssue) {
-                  await github.rest.issues.addLabels({
-                    ...context.repo,
-                    issue_number: prNumber,
-                    labels: [issueLabels.pr_submitted]
-                  });
-                } else {
-                  await github.rest.issues.addLabels({
-                    ...context.repo,
-                    issue_number: prNumber,
-                    labels: [issueLabels.needs_review]
-                  });
-                }
-              } else if (context.payload.action === 'closed' && pr.merged) {
+              if (context.payload.action === 'closed' && pr.merged) {
                 await github.rest.issues.addLabels({
                   ...context.repo,
                   issue_number: prNumber,

README.md

@@ -13,6 +13,22 @@ A parallel executor for [Robot Framework](http://www.robotframework.org) tests.
 
 [![Pabot presentation at robocon.io 2018](http://img.youtube.com/vi/i0RV6SJSIn8/0.jpg)](https://youtu.be/i0RV6SJSIn8 "Pabot presentation at robocon.io 2018")
 
+## Table of Contents
+
+- [Installation](#installation)
+- [Basic use](#basic-use)
+- [Contact](#contact)
+- [Contributing](#contributing-to-the-project)
+- [Command-line options](#command-line-options)
+- [PabotLib](#pabotlib)
+- [Controlling execution order](#controlling-execution-order-and-level-of-parallelism)
+- [Programmatic use](#programmatic-use)
+- [Global variables](#global-variables)
+- [Output Files Generated by Pabot](#output-files-generated-by-pabot)
+- [Artifacts Handling and Parallel Execution Notes](#artifacts-handling-and-parallel-execution-notes)
+
+----
+
 ## Installation:
 
 From PyPi:
@@ -245,11 +261,24 @@ Note: The `--ordering` file is intended only for defining the execution order of
 There different possibilities to influence the execution:
 
   * The order of suites can be changed.
-  * If a directory (or a directory structure) should be executed sequentially, add the directory suite name to a row as a ```--suite``` option.
-  * If the base suite name is changing with robot option [```--name / -N```](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#setting-the-name) you can also give partial suite name without the base suite.
+  * If a directory (or a directory structure) should be executed sequentially, add the directory suite name to a row as a ```--suite``` option. This usage is also supported when `--testlevelsplit` is enabled. As an alternative to using `--suite` options, you can also group tests into sequential batches using `{}` braces. (See below for details.) Note that if multiple `--suite` options are used, they must not reference the same test case. This means you cannot specify both parent and child suite names at the same time. For instance:
+
+```
+--suite Top Suite.Sub Suite
+--suite Top Suite
+```
+
+  * If the base suite name is changing with robot option [```--name / -N```](https://robotframework.org/robotframework/latest/RobotFrameworkUserGuide.html#setting-the-name) you can use either the new or old full test path. For example:
+
+```
+--test New Suite Name.Sub Suite.Test 1
+OR
+--test Old Suite Name.Sub Suite.Test 1
+```
+
   * You can add a line with text `#WAIT` to force executor to wait until all previous suites have been executed.
   * You can group suites and tests together to same executor process by adding line `{` before the group and `}` after. Note that `#WAIT` cannot be used inside a group.
-  * You can introduce dependencies using the word `#DEPENDS` after a test declaration. This keyword can be used several times if it is necessary to refer to several different tests. Please take care that in case of circular dependencies an exception will be thrown. Note that each `#WAIT` splits suites into separate execution blocks, and it's not possible to define dependencies for suites or tests that are inside another `#WAIT` block or inside another `{}` brackets.
+  * You can introduce dependencies using the word `#DEPENDS` after a test declaration. This keyword can be used several times if it is necessary to refer to several different tests. The ordering algorithm is designed to preserve the exact user-defined order as closely as possible. However, if a test's execution dependencies are not yet satisfied, the test is postponed and moved to the earliest possible stage where all its dependencies are fulfilled. Please take care that in case of circular dependencies an exception will be thrown. Note that each `#WAIT` splits suites into separate execution blocks, and it's not possible to define dependencies for suites or tests that are inside another `#WAIT` block or inside another `{}` braces.
   * Note: Within a group `{}`, neither execution order nor the `#DEPENDS` keyword currently works. This is due to limitations in Robot Framework, which is invoked within Pabot subprocesses. These limitations may be addressed in a future release of Robot Framework. For now, tests or suites within a group will be executed in the order Robot Framework discovers them — typically in alphabetical order.
   * An example could be:
 
@@ -343,4 +372,68 @@ Pabot will insert following global variables to Robot Framework namespace. These
       PABOTEXECUTIONPOOLID - this contains the pool id (an integer) for the current Robot Framework executor. This is helpful for example when visualizing the execution flow from your own listener.
       PABOTNUMBEROFPROCESSES - max number of concurrent processes that pabot may use in execution.
       CALLER_ID - a universally unique identifier for this execution.
- 
+
+
+### Output Files Generated by Pabot
+
+Pabot generates several output files and folders during execution, both for internal use and for analysis purposes.
+
+#### Internal File: `.pabotsuitenames`
+
+Pabot creates a `.pabotsuitenames` file in the working directory. This is an internal hash file used to speed up execution in certain scenarios.  
+This file can also be used as a base for the `--ordering` file as described earlier. Although technically it can be modified, it will be overwritten during the next execution.  
+Therefore, it is **recommended** to maintain a separate file for the `--ordering` option if needed.
+
+#### Output Directory Structure
+
+In addition to the standard `log.html`, `report.html`, and `output.xml` files, the specified `--outputdir` will contain:
+
+- A folder named `pabot_results`, and  
+- All defined artifacts (default: `.png` files)  
+- Optionally, artifacts from subfolders if `--artifactsinsubfolders` is used
+
+Artifacts are **copied** into the output directory and renamed with the following structure:
+
+```
+TIMESTAMP-ARGUMENT_INDEX-PABOTQUEUEINDEX
+```
+
+- **TIMESTAMP** = Time of `pabot` command invocation (not the screenshot's actual timestamp), format: `YYYYmmdd_HHMMSS`
+- **ARGUMENT_INDEX** = Optional index number, only used if `--argumentfileN` options are given
+- **PABOTQUEUEINDEX** = Process queue index (see section [Global Variables](#global-variables))
+
+#### `pabot_results` Folder Structure
+
+The structure of the `pabot_results` folder is as follows:
+
+```
+pabot_results/
+├── [N]/                     # Optional: N = argument file index (if --argumentfileN is used)
+│   └── PABOTQUEUEINDEX/     # One per subprocess
+│       ├── output.xml
+│       ├── robot_argfile.txt
+│       ├── robot_stdout.out
+│       ├── robot_stderr.out
+│       └── artifacts...
+```
+
+Each `PABOTQUEUEINDEX` folder contains as default:
+
+- `robot_argfile.txt` – Arguments used in that subprocess
+- `robot_stdout.out` and `robot_stderr.out` – Stdout and stderr of the subprocess
+- `output.xml` – The partial output file to be merged later
+- Artifacts – Screenshots or other files copied from subprocess folders
+
+> **Note:** The entire `pabot_results` folder is considered temporary and will be **deleted/overwritten** on the next `pabot` run using the same `--outputdir`.
+
+
+### Artifacts Handling and Parallel Execution Notes
+
+Due to parallel execution, artifacts like screenshots should ideally be:
+
+- Embedded directly into the XML using tools like [SeleniumLibrary](https://robotframework.org/SeleniumLibrary/SeleniumLibrary.html#Set%20Screenshot%20Directory) with the `EMBED` option  
+  _Example:_  
+  `Library  SeleniumLibrary  screenshot_root_directory=EMBED`
+- Or saved to the subprocess’s working directory (usually default behavior), ensuring separation across processes
+
+If you manually specify a shared screenshot directory in your test code, **all processes will write to it concurrently**, which may cause issues such as overwriting or missing files if screenshots are taken simultaneously.

src/pabot/__init__.py

@@ -7,4 +7,4 @@
 except ImportError:
     pass
 
-__version__ = "4.3.2"
+__version__ = "5.0.0"

src/pabot/execution_items.py

@@ -1,5 +1,5 @@
 from functools import total_ordering
-from typing import Dict, List, Optional, Tuple, Union
+from typing import Dict, List, Optional, Tuple, Union, Set
 
 from robot import __version__ as ROBOT_VERSION
 from robot.errors import DataError
@@ -8,36 +8,68 @@
 import re
 
 
-def create_dependency_tree(items):  
+def create_dependency_tree(items):
     # type: (List[ExecutionItem]) -> List[List[ExecutionItem]]
-    independent_tests = list(filter(lambda item: not item.depends, items))
-    dependency_tree = [independent_tests]
-    dependent_tests = list(filter(lambda item: item.depends, items))
-    unknown_dependent_tests = dependent_tests
-    while len(unknown_dependent_tests) > 0:
-        run_in_this_stage, run_later = [], []
-        for d in unknown_dependent_tests:
-            stage_indexes = []
-            for i, stage in enumerate(dependency_tree):
-                for test in stage:
-                    if test.name in d.depends:
-                        stage_indexes.append(i)
-            # All #DEPENDS test are already run:
-            if len(stage_indexes) == len(d.depends):
-                run_in_this_stage.append(d)
+    dependency_tree = []  # type: List[List[ExecutionItem]]
+    scheduled = set()  # type: Set[str]
+    name_to_item = {item.name: item for item in items}  # type: Dict[str, ExecutionItem]
+
+    while items:
+        stage = []  #type: List[ExecutionItem]
+        stage_names = set()  # type: Set[str]
+
+        for item in items:
+            if all(dep in scheduled for dep in item.depends):
+                stage.append(item)
+                stage_names.add(item.name)
             else:
-                run_later.append(d)
-        unknown_dependent_tests = run_later
-        if len(run_in_this_stage) == 0:
-            text = "There are circular or unmet dependencies using #DEPENDS. Check this/these test(s): " + str(run_later)
-            raise DataError(text)
-        else:
-            dependency_tree.append(run_in_this_stage)
-    flattened_dependency_tree = sum(dependency_tree, [])
-    if len(flattened_dependency_tree) != len(items):
-        raise DataError(
-            "Invalid test configuration: Circular or unmet dependencies detected between test suites. Please check your #DEPENDS definitions."
-        )
+                break  # Preserve input order
+
+        if not stage:
+            # Try to find any schedulable item even if it's out of order
+            for item in items:
+                if all(dep in scheduled for dep in item.depends):
+                    stage = [item]
+                    stage_names = {item.name}
+                    break
+
+        if not stage:
+            # Prepare a detailed error message
+            unscheduled_items = [item.name for item in items]
+            unsatisfied_deps = {
+                item.name: [d for d in item.depends if d not in scheduled and d not in name_to_item]
+                for item in items
+            }
+            potential_cycles = {
+                item.name: [d for d in item.depends if d in unscheduled_items]
+                for item in items if item.depends
+            }
+
+            message = ["Invalid test configuration:"]
+
+            message_unsatisfied = []
+            for item, deps in unsatisfied_deps.items():
+                if deps:
+                    message_unsatisfied.append(f"    - {item} depends on missing: {', '.join(deps)}")
+            if message_unsatisfied:
+                message.append("  Unsatisfied dependencies:")
+                message.extend(message_unsatisfied)
+                message.append("  For these tests, check that there is not #WAIT between them and that they are not inside different groups { }")
+
+            message_cycles = []
+            for item, deps in potential_cycles.items():
+                if deps:
+                    message_cycles.append(f"    - {item} <-> {', '.join(deps)}")
+            if message_cycles:
+                message.append("  Possible circular dependencies:")
+                message.extend(message_cycles)
+
+            raise DataError("\n".join(message))
+
+        dependency_tree.append(stage)
+        scheduled.update(stage_names)
+        items = [item for item in items if item.name not in stage_names]
+
     return dependency_tree
 
 
@@ -47,6 +79,7 @@ class ExecutionItem(object):
     type = None  # type: str
     name = None  # type: str
     sleep = 0  # type: int
+    depends = []  # type: List[str]  # Note that depends is used by RunnableItems.
 
     def top_name(self):
         # type: () -> str
@@ -156,7 +189,6 @@ def modify_options_for_executor(self, options):
 class RunnableItem(ExecutionItem):
     pass
 
-    depends = None  # type: List[str]
     depends_keyword = "#DEPENDS"
 
     def _split_dependencies(self, line_name, depends_indexes):
@@ -182,7 +214,7 @@ def set_name_and_depends(self, name):
         self.depends = (
             self._split_dependencies(line_name, depends_indexes)
             if len(depends_indexes) != 0
-            else None
+            else []
         )
 
     def line(self):
@@ -243,6 +275,10 @@ def tags(self):
         # TODO Make this happen
         return []
 
+    def modify_options_for_executor(self, options):
+        if not(options.get("runemptysuite") and options.get("suite")):
+            options[self.type] = self.name
+
 
 class TestItem(RunnableItem):
     type = "test"