Skip to main content

Release - July 12th, 2021

· 7 min read
Roxanna Pourzand
Product Manager

✨ New ✨#

Materializations#

Details

Product Component

Metrics Framework

More Detail

We're excited to announce that Materializations is a mechanism in Transform that enables the user to produce denormalized tables in a user's data warehouse. These are pre-computed and never evicted from the cache, and can be surfaced in downstream tools directly from the data warehouse.

Users can define a set of metrics and corresponding dimensions that are most commonly used and need to be accessed frequently by their organizations, which Transform will compute ahead of time at a scheduled cadence and write them to the data warehouse.

Verification Steps

We've reached out to most of you about deploying the materialization branch as it depends on a few changes in the framework around time granularity..

  1. Configure a materialization: Here is an example of a materialization definition that reflects data from a booking service company.
materialization:   name: user_bookings_summary # name your materialization. this will be the table name written in your data warehouse.  description: #add an optional description.  metrics: # list all the metrics you want to materialize in your materialization. all metrics must be defined in your metrics.yaml.        - bookings        - booking_value        - customer_service_tickets    - guest_host_messages      dimensions: #list all corresponding dimensions you want to include for these metrics. these must all be defined in your data sources.    - ds    - guest__signup_channel
# Optional destinations sections # By default the location is the datawarehouse. # Optionally provide additional destinations for the materialization and rollup    additional_destinations:    - location: FAST_CACHE      format: WIDE    - location: FAST_CACHE      format: WIDE      rollups: [ds]     # provide the location, format, and dimensions for rollups # - locations: [DW, FAST_CACHE] # DW is the only supported option currently #   format: WIDE # WIDE is the only supported option currently #   dimension_sets: # rollups allow you to create separate tables that group by a subset of the dimensions in your materialization #   - [ds] #add a dimension you want to a rollup by. this should be a subset of the dimensions you list in your materialization. These rollups will be stored in separate tables than the core materializations
  1. Calling a Materialization:
mql materialize --name my_materialization --start-time <date> --end-time <date>
**Usage**Options:  --materialization-name TEXT  Name of materialization to materialize                               [required]
  --start-time TEXT            iso8601 timestamp to materialize from  --end-time TEXT              iso8601 timestamp to materialize to  -t, --timeout INTEGER        Sets the timeout to wait for query completion. Not applicable in --detach mode.
  --detach BOOLEAN             Returns the created query ID to allow for                               asynchronous querying.
  --help                       Show this message and exit.

Measure Based Constraints#

Details

Product Component

Metrics Framework

More Detail

You can now add constraints to your measures in case you want to bound your metric result set to particular values. This configuration occurs when you define your metric.

This feature is available with the new branch we're rolling out to you to support.

Verification Steps

Example metric config with a constraint:

name: messages_on_days_when_there_were_a_lot_of_messagesdescription: Messages sent of any type, but only for days where message count exceeds the threshold (2000)owners:  - messages-team@thecompany.comdisplay_name: Messages Senttype: measure_proxytype_params:    measure: messagestier: "2"**constraint: |         where messages > 20**

Markdown for Metric Definition#

Details

Product Component

Metrics Catalog

More Detail

We now support markdown in our metrics definition editing module in the user interface - on a metric page.

Verification Steps

  • Navigate to a metric page where you are a metric owner
  • Choose Edit in the top right-hand corner in the configuration box
  • Test your favorite markdown shortcuts!

Various UI aesthetic Improvements#

Details

Product Component

Metrics Catalog

More Detail

This description contains a few different small usability improvements in the UI:

  1. We've updated the bottom of our left nav menu to distinguish the logout button a little more clearly.
  2. We removed the "Metric:" introduction on our metric page because it's a little redundant.
  3. We now sort the list of users and teams owners by the current owners in the Owner selection menu so it's clear who owns the metric while you're selecting others

Verification Steps

  1. We've updated the bottom of our left nav menu to distinguish the logout button a little more clearly.
  2. We removed the bolded "Metric:" introduction on our metric page because it's a little redundant
  3. Navigate to a Metric and choose the "Add/Remove Owners" but and confirm that the owning teams and/or users are listed at the top. Previously, you had to search through them to identify who owned the metric

API Explorer#

Details

Product Components

Metrics Catalog

More Detail

We've added an API explorer into the user interface so you can explore GraphQL requests in the interface directly. This is useful for troubleshooting and exploring the capabilities of our APIs.

Verification Steps

  1. Navigate to Settings —> MQL Server API explorer
  2. Start writing GraphQL requests in the console

Ability to Delete Saved Query#

Details

Product Component

Metric Catalog

More Detail

We've added a menu option under saved queries that allow you to delete them if you are the creator. In the case of wanting to remove your test queries or if you made a mistake, fear not!

Verification Steps

  1. Open the Queries panel on the chart and navigate to a query you created
  2. Open the three-dot menu on the right-hand top corner
  3. Select 'delete query'
  4. Confirm your selection

Added Verbose Configurations to template repo#

Details

More Detail

In order to provide more details around how to build out data sources and metrics, we've added a verbose data source and metric template into our example configs in our template repository.

These files are commented out by default but include detailed instructions on what each field and area in the data source and metric YAML means.

This configuration template will be useful for everyone but especially new users of Transform who are just getting started, and the intent is that you can copy these and fill in your own information.

Verification Steps

When deploying Transform, there's a template repository at metrics-config-template/example/that includes a set of test data sources including rainfall and stations. In the same folder, we've added two files - one for data sources and one for metrics, that provide comments, instructions, and examples for how to configure these files.

The files are example_datasource.yaml and example_metric.yaml

Amazon Athena Support#

Details

Product Component

Metrics Framework

More Detail

We now support Amazon Athena as a source data warehouse to Transform, and among other things, we're excited to add support for our first database named after a Greek Goddess (of wisdom).

Verification Steps

Work with us to deploy Transform with your Athena instance.

🐛 Bug Fixes 🐛#

Install Page Hanging Issue#

Details

Product Component

Metrics Catalog

More Detail

If you tried to access: https://app.transformdata.io/install without being logged in first, the page would hang indefinitely. We hope you didn't think that we didn't want you to install Transform - we really do! So, we fixed it and now lead you to a page with more information and login links.

Verification Steps

Visit https://app.transformdata.io/install before logging into the app and you should see a page that prompts you to log in as opposed to this one:

Secondary Time Dimension Bug#

Details

Product Component

Metrics Framework

We identified and fixed an issue where configuring a time dimension that wasn't primary would cause models to break. We fixed this issue and you can now specify both primary and non-primary time dimensions, as originally intended.

Verification Steps Add secondary time dimensions to your model using is_primary:false