Visibility

This guide provides a comprehensive overview of Temporal Visibility.

Support, stability, and dependency info

For Temporal Server v1.19 and earlier, all supported databases for Visibility provide standard Visibility features, and an Elasticsearch database is required for advanced Visibility features.
For Temporal Server v1.20 and later, advanced Visibility features are enabled on all supported SQL databases, in addition to Elasticsearch.
In Temporal Server v1.21 and later, standard Visibility is no longer in development, and we recommend migrating to a database that supports advanced Visibility features. Visibility configuration in Temporal Cluster is updated and Dual Visibility is enabled. For details, see Visibility store setup.

The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view, filter, and search for Workflow Executions that currently exist within a Cluster.

The Visibility store in your Temporal Cluster stores persisted Workflow Execution Event History data and is set up as a part of your Persistence store 🔗

to enable listing and filtering details about Workflow Executions that exist on your Temporal Cluster.

How to set up a Visibility store

With Temporal Server v1.21, you can set up Dual Visibility 🔗

to migrate your Visibility store from one database to another.

What is standard Visibility?

Standard Visibility, within the Temporal Platform, is the subsystem and APIs that list Workflow Executions by a predefined set of filters.

Open Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, or Run Id.

Closed Workflow Executions can be filtered by a time constraint and either a Workflow Type, Workflow Id, Run Id, or Execution Status (Completed, Failed, Timed Out, Terminated, Canceled, or Continued-As-New).

Support for standard Visibility is deprecated beginning with Temporal Server v1.21. For updates, check Supported databases.

What is advanced Visibility?

Visibility, within the Temporal Platform, is the subsystem and APIs that enable the listing, filtering, and sorting of Workflow Executions 🔗

through a custom SQL-like List Filter 🔗.

In Temporal Cluster version 1.20 and later, advanced Visibility is available on SQL databases like MySQL (version 8.0.17 and later) and PostgreSQL (version 12 and later), in addition to support for Elasticsearch.
For Temporal Server versions 1.19.1 and earlier, you must integrate with ElasticSearch 🔗
x
Link preview
How to integrate Elasticsearch into a Temporal Cluster
To integrate Elasticsearch with your Temporal Cluster, edit the persistence section of your development.yaml configuration file and run the index schema setup commands.
operation-guide filtered-lists visibility
to use advanced Visibility. Elasticsearch takes on the Visibility request load, relieving potential performance issues. We highly recommend operating a Temporal Cluster with Elasticsearch for any use case that spawns more than just a few Workflow Executions.
On Temporal Cloud, advanced Visibility is enabled by default for all users.

What is Dual Visibility?

Dual Visibility is a feature that lets you set a secondary Visibility store in addition to a primary store in your Temporal Cluster. Setting up Dual Visibility is optional and can be used to migrate your Visibility database or create a backup Visibility store.

For example, if you have Cassandra configured as your Visibility database, you can set up a supported SQL database as your secondary Visibility store and gradually migrate your data to the secondary store before deprecating your primary one.

A Dual Visibility setup requires two Visibility store configurations:

Primary Visibility: The primary Visibility store where Visibility data is written to and read from by default. The primary Visibility store is set with the visibilityStore configuration key in your Temporal Cluster.
Secondary Visibility: A secondary storage for your Visibility data. The secondary Visibility store is set with the secondaryVisibilityStore configuration key in your Temporal Cluster.

For configuration details, see Dual Visibility setup.

The following combinations are allowed in a Dual Visibility setting.

Primary	Secondary
Standard (Cassandra or SQL)	Advanced (SQL or Elasticsearch)
Advanced (SQL)	Advanced (SQL)
Advanced (Elasticsearch)	Advanced (Elasticsearch)

With Dual Visibility, you can read from only one Visibility store at a time, but can configure your Temporal Cluster to write to primary only, secondary only, or to both primary and secondary Visibility stores. When migrating from one Visibility store database to another, set up the database you want to migrate to as your secondary Visibility store.

You can plan your migration using specific dynamic configuration keys that help you transition your read and write operations from the primary to the secondary Visibility store. For details on migrating your Visibility store databases, see Dual Visibility 🔗

What is a List Filter?

A List Filter is the SQL-like string that is provided as the parameter to a Visibility 🔗

List API.

A List Filter contains Search Attribute 🔗

names, Search Attribute values, and operators to pull a filtered list of Workflow Executions from the Visibility store.

List Filter Search Attribute 🔗

names are case sensitive, and each List Filter is scoped by a single Namespace 🔗.

A List Filter that uses a time range has a resolution of 1 ns on Elasticsearch 🔗

and 1 µs for SQL databases 🔗.

Supported operators

A List Filter contains Search Attribute 🔗

names, Search Attribute values, and the following supported operators:

=, !=, >, >=, <, <=
AND, OR, ()
BETWEEN ... AND
IN

Partial string match

The = operator works like CONTAINS to find Workflows with Search Attributes that contain a specific word.

For example, if you have a custom Search Attribute named Description of Text type with the value of "The quick brown fox jumps over the lazy dog", searching for Description='quick' or Description='fox' will successfully return the Workflow. However, partial word searches such as Description='qui' or Description='laz' will not return the Workflow. This is because Elasticsearch's tokenizer is configured to return complete words as tokens.

Efficient API usage

An Advanced List Filter API may take longer to respond if it is retrieving a large number of Workflow Executions (over 10,000).

With Temporal Server v1.20 and later, you can use the CountWorkflow API to efficiently count the number of Workflow Executions 🔗

Paginate the results with the ListWorkflow API by using the page token to retrieve the next page; continue until the page token is null/nil.

List Filter examples

The following is a List Filter set with tctl 🔗

WorkflowType = "main.YourWorkflowDefinition" and ExecutionStatus != "Running" and (StartTime > "2021-06-07T16:46:34.236-08:00" or CloseTime > "2021-06-07T16:46:34-08:00")

When used, a list of Workflows that meet the following conditions are returned:

The Workflow Type is set to main.YourWorkflowDefinition.
The Workflow isn't running.
The Workflow either started after "2021-06-07T16:46:34.236-08:00" or closed after "2021-06-07T16:46:34-08:00".

More List Filter examples have been provided below.

WorkflowId = '<workflow-id>'

WorkflowId = '<workflow-id>' or WorkflowId = '<another-workflow-id>'

WorkflowId IN ('<workflow-id>', '<another-workflow-id>')

WorkflowId = '<workflow-id>' and ExecutionStatus = 'Running'

WorkflowId = '<workflow-id>' or ExecutionStatus = 'Running'

WorkflowId = '<workflow-id>' and StartTime > '2021-08-22T15:04:05+00:00'

ExecutionTime between '2021-08-22T15:04:05+00:00' and '2021-08-28T15:04:05+00:00'

ExecutionTime < '2021-08-28T15:04:05+00:00' or ExecutionTime > '2021-08-22T15:04:05+00:00'

What is a Search Attribute?

A Search Attribute is an indexed field used in a List Filter 🔗

to filter a list of Workflow Executions that have the Search Attribute in their metadata.

Each Search Attribute is a key-value pair metadata object included in a Workflow Execution's Visibility information. This information is available in the Visibility store.

note

Search Attribute values are not encrypted because the Temporal Server must be able to read these values from the Visibility store when retrieving Workflow Execution details.

Temporal provides some default Search Attributes, such as ExecutionStatus, the current state of your Workflow Executions. You can also create custom Search Attribute keys in your Visibility store and assign values when starting a Workflow Execution or in Workflow code.

When using Continue-As-New 🔗

or a Temporal Cron Job 🔗, Search Attribute keys are carried over to the new Workflow Run by default. Search Attribute values are only available for as long as the Workflow is.

Search Attributes are most effective for search purposes or tasks requiring collection-based result sets. For business logic in which you need to get information about a Workflow Execution, consider one of the following:

Storing state in a local variable and exposing it with a Query.
Storing state in an external datastore through Activities and fetching it directly from the store.

If your business logic requires high throughput or low latency, store and fetch the data through Activities. You might experience lag due to time passing between the Workflow's state change and the Activity updating the Visibility store.

Default Search Attributes

A Temporal Cluster has a set of default Search Attributes already available. Default Search Attributes are set globally in any Namespace. These Search Attributes are created when the initial index is created.

NAME	TYPE	DEFINITION
BatcherUser	Keyword	Used by internal batcher Workflow that runs in `TemporalBatcher` Namespace division to indicate the user who started the batch operation.
BinaryChecksums	Keyword List	List of binary Ids of Workers that run the Workflow Execution. Deprecated since server version 1.21 in favor of the `BuildIds` search attribute.
BuildIds	Keyword List	List of Worker Build Ids that have processed the Workflow Execution, formatted as `versioned:{BuildId}` or `unversioned:{BuildId}`, or the sentinel `unversioned` value. Available from server version 1.21.
CloseTime	Datetime	The time at which the Workflow Execution completed.
ExecutionDuration	Int	The time needed to run the Workflow Execution (in nanoseconds). Available only for closed Workflows.
ExecutionStatus	Keyword	The current state of the Workflow Execution.
ExecutionTime	Datetime	The time at which the Workflow Execution actually begins running; same as `StartTime` for most cases but different for Cron Workflows and retried Workflows.
HistoryLength	Int	The number of events in the history of Workflow Execution. Available only for closed Workflows.
HistorySizeBytes	Long	The size of the Event History.
RunId	Keyword	Identifies the current Workflow Execution Run.
StartTime	Datetime	The time at which the Workflow Execution started.
StateTransitionCount	Int	The number of times that Workflow Execution has persisted its state. Available only for closed Workflows.
TaskQueue	Keyword	Task Queue used by Workflow Execution.
TemporalChangeVersion	Keyword List	Stores change/version pairs if the GetVersion API is enabled.
TemporalScheduledStartTime	Datetime	The time that the Workflow is schedule to start according to the Schedule Spec. Can be manually triggered. Set on Schedules.
TemporalScheduledById	Keyword	The Id of the Schedule that started the Workflow.
TemporalSchedulePaused	Boolean	Indicates whether the Schedule has been paused. Set on Schedules.
WorkflowId	Keyword	Identifies the Workflow Execution.
WorkflowType	Keyword	The type of Workflow.

All default Search Attributes are reserved and read-only. You cannot create a custom one with the same name or alter the existing one.
ExecutionStatus values correspond to Workflow Execution statuses: Running, Completed, Failed, Canceled, Terminated, ContinuedAsNew, TimedOut.
StartTime, CloseTime, and ExecutionTime are stored as dates but are supported by queries that use either EpochTime in nanoseconds or a string in RFC3339Nano format (such as "2006-01-02T15:04:05.999999999Z07:00").
ExecutionDuration is stored in nanoseconds but is supported by queries that use integers in nanoseconds, Golang duration format, or "hh:mm:ss" format.
CloseTime, HistoryLength, StateTransitionCount, and ExecutionDuration are present only in a closed Workflow Execution.
ExecutionTime can differ from StartTime in retry and cron use cases.

You can use the default Search Attributes in a List Filter, such as in the Temporal Web UI or with the tctl workflow list commands, under the following conditions:

Without advanced Visibility, you can only use the = operator with a single default Search Attribute in your List Filter. For example: tctl workflow list -q "ExecutionStatus = 'Completed'" or tctl workflow list -q "WorkflowType = 'YourWorkflow'".
With advanced Visibility, you can combine default Search Attributes in a List Filter to get a list of specific Workflow Executions. For example: tctl workflow list -q "WorkflowType = 'main.YourWorkflowDefinition' and ExecutionStatus != 'Running' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"

Custom Search Attributes

You can create custom Search Attributes 🔗

with unique key names that are relevant to your business needs.

Use custom Search Attributes in a List Filter, such as in the Temporal Web UI or with the tctl workflow list commands, under the following conditions:

Without advanced Visibility, you cannot use a custom Search Attribute in your List Filter.
With advanced Visibility, you can create multiple custom Search Attributes and use them in combinations with List Filters to get a list of specific Workflow Executions. For example: tctl workflow list -q "WorkflowType = 'main.YourWorkflowDefinition' and YourCustomSA = 'YourCustomSAValue' and (StartTime > '2022-06-07T16:46:34.236-08:00' or CloseTime < '2022-06-08T16:46:34-08:00')"
- With Temporal Server v1.19 and earlier, you must integrate Elasticsearch 🔗
  x
  Link preview
  How to integrate Elasticsearch into a Temporal Cluster
  To integrate Elasticsearch with your Temporal Cluster, edit the persistence section of your development.yaml configuration file and run the index schema setup commands.
  operation-guide filtered-lists visibility
  to use custom Search Attributes with List Filters.
- With Temporal Server v1.20 and later, custom Search Attribute capabilities are available on MySQL (v8.0.17 or later), PostgreSQL (v12 and later), and SQLite (v3.31.0 and later), in addition to Elasticsearch.

If you use Elasticsearch as your Visibility store, your custom Search Attributes apply globally and can be used across Namespaces. However, if using any of the supported SQL databases with Temporal Server v1.20 and later, your custom Search Attributes are associated with a specific Namespace and can be used for Workflow Executions in that Namespace.

See custom Search Attributes limits for limits on the number and size of custom Search Attributes you can create.

Supported types

Custom Search Attributes must be one of the following types:

Bool
Datetime
Double
Int
Keyword
KeywordList
Text

Note:

Double is backed up by scaled_float Elasticsearch type with scale factor 10000 (4 decimal digits).
Datetime is backed up by date type with milliseconds precision in Elasticsearch 6 and date_nanos type with nanoseconds precision in Elasticsearch 7.
Int is 64-bit integer (long Elasticsearch type).
Keyword and Text types are concepts taken from Elasticsearch. Each word in a Text is considered a searchable keyword. For a UUID, that can be problematic because Elasticsearch indexes each portion of the UUID separately. To have the whole string considered as a searchable keyword, use the Keyword type. For example, if the key ProductId has the value of 2dd29ab7-2dd8-4668-83e0-89cae261cfb1:
- As a Keyword it would be matched only by ProductId = "2dd29ab7-2dd8-4668-83e0-89cae261cfb1.
- As a Text it would be matched by ProductId = 2dd8, which could cause unwanted matches.
With Temporal Server v1.19 and earlier, the Keyword type can store a list of values.
With Temporal Server v1.20 and later, the Keyword type supports only a single value. To store a list of values, use KeywordList.
The Text type cannot be used in the "Order By" clause.

Custom Search Attributes limits

The following table lists the maximum number of custom Search Attributes you can create per Namespace by supported Visibility database.

Search Attribute type	MySQL (v8.0.17 and later)	PostgreSQL (v12 and later)	SQLite (v3.31.0 and later)	Temporal Cloud
Bool	3	3	3	20
Datetime	3	3	3	20
Double	3	3	3	20
Int	3	3	3	20
Keyword	10	10	10	20
KeywordList	3	3	3	20
Text	3	3	3	5

Temporal does not impose a limit on the number of custom Search Attributes you can create with Elasticsearch. However, Elasticsearch sets a default mapping limit that may apply. Custom Search Attributes are an advanced Visibility feature and are not supported on Cassandra.

Size limits for a custom Search Attribute:

The default single Search Attribute value size limit is 2 KB.

The maximum total Search Attribute size is 40 KB.

The maximum total characters per Search Attribute value is 255.

Usage

Search Attributes available in your Visibility store can be used with Workflow Executions for that Cluster. To actually have results from the use of a List Filter 🔗

, Search Attributes must be added to a Workflow Execution as metadata.

To create custom Search Attributes in your Visibility store, see Create custom Search Attributes 🔗
x
Link preview
How to create custom Search Attributes
Add custom Search Attributes to your Visibility store using tctl for self-hosted Temporal Cluster, and tcld for Temporal Cloud.
operation-guide filtered-lists visibility
.
To remove a custom Search Attribute from the Visbility store, see Remove custom Search Attributes 🔗
x
Link preview
How to remove custom Search Attributes
Remove custom Search Attributes from your self-hosted Temporal Cluster Visibility store using tctl.
operation-guide filtered-lists visibility
. Removing custom Search Attributes is not supported on Temporal Cloud.
To rename a custom Search Attribute on Temporal Cloud, see tcld namespace search-attributes rename.

With Workflows you can do the following:

Set the value of Search Attributes in your Workflow
Update the value set for a Search Attribute from within the Workflow code
Remove the value set for a Search Attribute from within the Workflow code
How to manage Search Attributes using the Go SDK 🔗
x
Link preview
How to use Visibility APIs
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.
developer-guide-doc-type go sdk visibility api
How to manage Search Attributes using the Java SDK 🔗
x
Link preview
How to use Visibility APIs
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.
guide-context
How to manage Search Attributes using the PHP SDK 🔗
x
Link preview
How to use Visibility APIs
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.
guide-context
How to manage Search Attributes using the Python SDK 🔗
x
Link preview
How to use Visibility APIs
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.
guide-context
How to manage Search Attributes using the TypeScript SDK 🔗
x
Link preview
How to use Visibility APIs
The term Visibility, within the Temporal Platform, refers to the subsystems and APIs that enable an operator to view Workflow Executions that currently exist within a Cluster.
guide-context
To get a list of Search Attributes using tctl, see How to view Search Attributes using tctl.

After you add and set your Search Attributes, use your default or custom Search Attributes in a List Filter.

What is standard Visibility?​

What is advanced Visibility?​

What is Dual Visibility?​

What is a List Filter?​

Supported operators​

Partial string match​

Efficient API usage​

List Filter examples​

What is a Search Attribute?​

Default Search Attributes​

Custom Search Attributes​

Supported types​

Custom Search Attributes limits​

Usage​