Assembly Execution Progress

Warning

This feature is still in an experimental stage. Its exact behavior, its API, and its pricing might change while we are continuing to work on it. If you encounter any bugs or odd behavior, please contact us.

Assembly Execution Progress provides real-time estimates about how far the execution of an Assembly has progressed. A percentage value is calculated for the progress of the entire Assembly, as well as for each uploaded or imported file. This provides accurate progress information even if files are processed differently in the same Assembly, for example because of different file type.

To calculate these percentage values, our Assembly engine predicts what Steps are likely to be executed for the remaining Assembly. This prediction is then compared to the already finished Steps to obtain the progress estimate. We then enhance this estimation by incorporating progress information from currently executing Steps. For example, for /video/encode Steps we include information about how much of the video has been encoded at the moment.

Assembly Execution Progress is a great enhancement for your user interface, providing your users insight into what's happening to their files. Combined with regular upload progress, which shows the progress for transferring the users' files to our servers, Assembly Execution Progress helps to let your users know that their files are taken care of and that the things are happening behind the scenes.

Compatible Assemblies

We attempt to provide Execution Progress for every Assembly, but this is not reliably possible for all cases. If your Assembly includes a Step, with an unpredictable result count, no progress can be estimated until this Step is completed. However, once all of these Steps are finished, Execution Progress can be calculated.

Steps whose result count cannot be predicted are:

Importing directories via /s3/import, /azure/import etc. This does not include importing a fixed number of files.
/file/compress: Extracting any type of archive.
/document/thumbs: Generating thumbnails for a document without specifying the page number.

If your Assembly does not include such Steps, Execution Progress is available from the beginning. If it does include such Steps, progress is available after these Steps are completed.

Enabling Assembly Execution Progress

You must explicitly opt-in for each Assembly, for which you want to receive Execution Progress. This is performed when the Assembly is created by including the "emit_execution_progress": true property in the Assembly Instructions, for example:

{
  "steps": {
    "resized": {
      "robot": "/image/resize",
      "use": ":original",
      "width": 100,
      "height": 100,
      "resize_strategy": "fillcrop",
    },
    // ...
  },
  // ...

  // Enable execution progress
  "emit_execution_progress": true,
}

Receiving Execution Progress

Assembly Execution Progress can be retrieved with the Assembly update stream. In a nutshell, once an Assembly has been created, the Assembly Status contains a URL, where you can retrieve messages and events in real-time about the status of the Assembly. The exact details are described further in the documentation.

Once the update stream for an Assembly is opened, and Execution Progress is enabled, you will receive progress updates regularly every 2-3s. The update is sent via the assembly_execution_progress event. It includes a JSON payload describing the progress for the entire Assembly and each imported or uploaded file individually, for example:

{
  "progress_combined": 75,
  "progress_per_original_file": [
    {
      "original_id": "6bb16f6cd49a44b4ae431f576e016c6d",
      "progress": 50
    },

    {
      "original_id": "56921a2dd46246d3b4d3a98b9f8ed4e1",
      "progress": 100
    }
  ]
}

In the above example, the entire Assembly is processed to 75%. The file with the original_id of 56921a2dd46246d3b4d3a98b9f8ed4e1 has finished processing, meaning it has progressed through all of its Steps. The file with the original_id of 6bb16f6cd49a44b4ae431f576e016c6d is not yet done, with 50% progress so far.

If no progress estimate is available because the Assembly includes unfinished Steps, whose result count cannot be predicted, no event will be emitted until these Steps are finished.