The
Dataform folders and repositories
feature is now
generally available
(GA). This feature lets you organize code assets like notebooks and saved
queries into a hierarchical structure with IAM policy inheritance. This release
also introduces deleteTree API methods for deleting folders and
team folders.
You can organize your code assets into a hierarchical structure with folders and repositories using the Dataform API. This feature is in preview.
]]>Dataform workflows,
BigQuery notebooks,
pipelines,
and
data preparations
are enforcing strict act-as mode at the project level. To avoid failures and
maintain automatic releases, you must use custom service accounts instead of the
default Dataform service agent across all repositories. You must also grant the
Service Account User role (roles/iam.serviceAccountUser) to the default
Dataform service agent and relevant principals. For more information and to
verify act-as permissions, see
Use strict act-as mode.
Strict act-as mode
for Dataform is now
generally available
(GA). This feature enhances security by requiring users to have the
iam.serviceAccounts.actAs permission on the service account used to run
workflows, ensuring a more secure and predictable permissions model for your
Dataform projects.
You can verify and resolve iam.serviceAccounts.actAs permission issues in
Dataform by checking Cloud Logging, interpreting log entries, and granting the
necessary IAM roles. For more information, see
Verify act-as permissions for the effective service account.
This feature is in
preview.
Dataform offers enhanced IAM permissions, providing more granular control over resource creation and scheduling. Administrators can also enable private workspaces to restrict access to the workspace creator. These features are in preview.
]]>Dataform now lets you automate the creation of BigLake tables for Apache Iceberg in BigQuery. This feature is generally available (GA).
]]>You can use custom constraints with Organization Policy to provide more
granular control over specific fields for the CompilationResult,
ReleaseConfig, WorkflowConfig, WorkflowInvocation, and Workspace
resources. For more information, see
Create custom organization policy constraints.
This feature is
generally available
(GA).
You can use custom constraints with Organization Policy to provide more granular control over specific fields for some Dataform resources. For more information, see Create custom organization policy constraints. This feature is generally available (GA).
You can set the priority of BigQuery jobs in Dataform to run queries as interactive jobs that start running as quickly as possible or as batch jobs with lower priority. For more information, see Create a workflow configuration and InvocationConfig. This feature is generally available (GA).
]]>You can now update an incremental table schema without a full table refresh. This feature is in Preview.
]]>Dataform now automatically selects a processing location based on the datasets referenced in your SQL queries. This makes setting the default location optional in your workflow configurations. For more information, see About repository settings. This feature is generally available (GA).
]]>A security vulnerability was discovered in the Dataform API. This vulnerability could potentially allow unauthorized access to customer code repositories and data. For more information, see GCP-2025-045 security bulletin.
]]>Updates to the automatic cataloging of Dataform metadata in Dataplex improve the near real-time management and search capabilities for repository metadata. These features are generally available (GA).
]]>Dark theme is now available for BigQuery and Dataform. To turn on the dark theme, go to the Console, open the Settings and utilities menu next to your avatar, and select Preferences. On the User preferences page, select Appearance in the navigation, select your color theme, and save your selection.
]]>You can now use your Google Account user credentials to authorize the creation, scheduling, and running of pipelines, the scheduling of notebooks and data preparations, and the creation of workflow configurations. For more information, see Schedule runs. This feature is in preview.
]]>You can now use strict act-as mode to enable an additional security check for certain user actions in Dataform. For more information, see Use strict act-as mode. This feature is in preview.
]]>You can now search for and view the metadata of Dataform repositories in the Dataplex console. This feature is in preview.
You can now manage Dataform repositories in Dataplex. Metadata of Dataform repositories is automatically available in Dataplex, without additional configuration. For more information, see Manage Dataform assets with Dataplex. This feature is generally available (GA).
]]>Dataform is available in the following regions:
For more information, see Locations.
]]>The maximum size limit for workspaces encrypted with customer-managed encryption keys (CMEK) is 512 MB. For more information about Dataform quotas and limits, see Quotas and limits. For more information about encrypting Dataform repositories with CMEK, see Use customer-managed encryption keys.
]]>You can now set a default Dataform customer-managed encryption keys (CMEK) key for your project to encrypt multiple Dataform repositories with the same CMEK key. For more information, see Use Dataform default CMEK keys.
]]>Customer-Managed Encryption Keys (CMEK) in Dataform are generally available (GA). For more information, see Use customer-managed encryption keys.
You can now use Cloud External Key Manager (Cloud EKM) keys to protect Dataform data. Cloud EKM keys in Dataform are generally available (GA). For more information, see Using and managing external keys.
]]>Access Approval supports Dataform in the GA stage.
]]>The maximum size limit for repositories encrypted with customer-managed encryption keys (CMEK) is 512 MB. For more information about Dataform quotas and limits, see Quotas and limits. For more information about encrypting Dataform repositories with CMEK, see Use customer-managed encryption keys.
]]>As of Dataform Core 3.0.0.,
Dataform doesn't distribute a Docker image. You can build your own
Docker image of Dataform, which you can use to run the equivalent of Dataform CLI commands. To build your own Docker image, see Containerize an application
in the Docker documentation.
The 3.0.0 version of the open-source Dataform framework is available.
The workflow_settings.yaml file, which was introduced in Dataform Core 3.0.0-beta.0, replaces dataform.json.
You can specify the Dataform Core version directly in the workflow_settings.yaml file, which removes the need for package.json for most repositories. To have package dependencies other than @dataform/core, the package.json file is still required.
No immediate action to convert existing Dataform code is required. You can continue to use dataform.json and package.json in existing repositories.
You can convert your dataform.json file into workflow_settings.yaml by following the instructions in the 3.0.0 GitHub release.
New repositories use workflow_settings.yaml by default. You can replace the workflow_settings.yaml file with dataform.json to continue using the JSON format. If you remove workflow_settings.yaml, you need to add a package.json file to your repository to install @dataform/core.
For more information, see the 3.0.0 release on GitHub.
]]>You can now inspect past manual compilation results of a selected release configuration. For more information, see View details of a release configuration.
]]>Dataform Core includeDependentAssertions and dependOnDependencyAssertions parameters for adding assertions as dependencies are available.
You can set the includeDependentAssertions parameter in a selected action to automatically add assertions of a selected dependency action as dependencies of the edited action.
You can set dependOnDependencyAssertions the parameter in a selected action to automatically add assertions of all dependency actions as dependencies of the edited action.
For more information, see Set assertions as dependencies.
]]>Gemini, an AI-powered collaborator in Google Cloud, can help you generate code in Dataform. This feature is in preview. For more information, see Write queries with Gemini assistance.
]]>You can now use Customer-Managed Encryption Keys (CMEK) to protect repositories in Dataform. CMEK in Dataform is available in preview. For more information, see Use customer-managed encryption keys.
]]>The 3.0.0-beta.0 version of the open-source Dataform framework is available. This update introduces significant changes, including, but not limited to, the following:
dataform.json in favor of workflow_settings.yaml@dataform/cliYou don't need to take immediate action to update your Dataform code.
For more information, see the 3.0.0-beta.0 release on GitHub.
]]>