Product Component

Metrics Catalog

More Detail

Both individuals and teams often want to group together the most important metrics that apply to their job function. Transform provides the metrics in the UI that you can click into individually, which is helpful and all, but you have to click back and forth between different metrics if you want to look at multiple. Wouldn't it be nice to be able to see a handful of your key metrics in one place? Well, I'm so glad you're wondering... because now you can! We've introduced Collections! Collections allow you to group all your important metrics in one place.

You can associate collections to yourself, or to a team, which will show up on the Team Pages.

Verification Steps

1. Navigate to the Collections menu in the sidebar and choose "Make your Collection"
2. Fill in information about your collection
3. Create Collection by adding Metrics
4. View your Collection and add other metrics by using the "edit" button.
5. Be Excited!

Product Component

Metrics Catalog

More Detail

Have you ever wanted to revisit a dimensional slice and grouping of a metric that you previously created in Transform without having to select all the filters and groups again? Us too! This is why we introduced Saved Queries.

Not only will this feature allow you to save common queries and go back and analyze results without re-creating the filter, but it also allows other users to see common queries against metrics and learn more about what might be important to look at for a particular metric.

Verification Steps

1. To create a saved query, go to the main metric page chart and hit "edit chart". Select your filters and groupings that you desire.
2. Save the query by clicking on "More Actions". Title and Save the query associated with yourself or a Team. Note: anyone can see this query in either selection, but if you add it to a Team page, it will also show up on the respective Team Page under "Saved Queries"
3. Access your Queries associated with Metrics by Clicking on the "Queries" tab. A count is provided so you can see which metrics are the "cool kids" for queries.

Product Component

Metrics Catalog

More Detail

Have you ever been itching to view all your Team's important metrics, queries, and collections in one place? Collaboration with your teams is crucial to building a data culture at your company, and our mission at Transform is to empower you to do that, which is why we introduced the concept of Team Pages.

Team Pages allow you to create a team in Transform, add members to it, and all collaborate on a page that has your key metrics, collections, and saved queries.

Verification Steps

1. You can find team pages on the lefthand menu
2. To create a team, select "Create a Team" which will take you to the following page to ask you the team name, members, and description. You can always edit teams in the Settings menu as well under " Manage Teams"
3. Access your Team Page listed in the lefthand menu.

Product Component

Metrics Catalog

More Detail

We added a new chart menu that allows you to Export as CSV, Download full screen, or go into the query logs for that particular query.

I was trying to come up with a good CSV joke, but I struggled because they are Comedy-Limited 😏

Verification Steps

Go into any metric chart. This can be accessed through a metric page, or even on the homepage. Select the three dots on the top of the graph and select the desired option.

Product Component

Metrics Catalog

More Detail

We've added a few additional chart actions when editing a chart or query so that users know when the chart data properly reflects their changes.

We also added options to revert back to original filters during an editing process and actions around updating a saved query.

Verification Steps

1. Go into a metrics data chart
2. Choose Edit Chart and make some changes to the filters. You should notice that the Chart prompts you to update or revert (previously, the chart would show un-updated data and so it wasn't clear whether the chart was showing new or old information)
3. Scroll down and find the new action menu, which will have information about resetting filters, clearing filters, saving as a query, or updating a query if you're editing a Saved Query.

Product Component

Interfaces - MQL CLI

More Details

🎉 You can now export query results directly to a csv file using the Transform CLI.

This accompanies the feature in the UI to export a CSV.

Verification Steps

$ mql query --metrics bookings --dimensions ds --order ds --csv ~/Desktop/bookings.csv✔ Query initialized: 1622140544978_0000007✔ Success 🦄 - query completed after 2.24 secondsSuccessfully written query output to /Users/marco/Desktop/bookings.csv

Product Component

Metrics Framework

More Detail

We added the ability to create a metric directly when creating a measure. When you create a metric from a measure, there was no mechanism to configure a display name, so we added a new argument called create_metric_display_name

Verification Steps

If you create a metric from a measure, you can add the argument and confirm that it shows up with a "pretty" name in the UI.

For example:

name: api_requestsexpr: "1"agg: sumcreate_metric: truecreate_metric_display_name: API Requests

Product Component

Metrics Framework

More Detail

Previously, we would throw an error when looking at dimensions if you had a metric that had two measures that came from different data sources, one where the dimension was local and one where it was non-local.

Verification Steps

Create a metrics configuration that has two different data sources. The metric should contain two measures - one that is local to the data source containing the requested dimension and one that is non-local.

For Example:

Assume you have a metric called bookings that has information from two different data sources, X and Y.

You might have a dimension called country from data source X that is non-local to the primary identifier, so it needs to be joined onto another identifier, such as listing to obtain the values (e.g., listing__country)

The booking metrics may have another measure from Data Source Y that has the country dimension locally, e.g., simply country. The discrepancy was that our query builder assumes dimensions will be accessed by name, and so we were unable to retrieve data source X and data source Y to retrieve the dimension values for the country by the bookings metric because listing__country != country.

Product Component

Framework

More Detail

We've added a new aggregation type for measures: boolean. If you have a field thats TRUE/FALSE, you can now use this aggregation type as opposed to creating additional expression logic. This aggregation type will sum true values as 1, 0 otherwise.

Verification Steps

1. Create a measure from a boolean field and denote agg: boolean

- name: has_reply  description: Denotes if there was a reply to a question.  expr: col_name  agg: boolean

2. Tell a Boolean dad joke, such as "What did the 1 say to the 0?" — "Stop Boolean me!"

Product Component

Metrics Catalog

More Detail

The metric owner was previously not showing up on the main page that shows recently viewed and popular metrics. Among other things, we care a lot about ownership at Transform, so we want to make sure this field was very visible to you and your teams. The metric owner should now show up on the left-hand side of the graph card.

Note that you'll have to first republish your configs for this change to propagate into the UI. Also, in general, you should make sure that the metric owner you've specified in the YAML has an account in the Transform UI.

Verification Steps

1. Republish your configurations
2. Load the metrics dashboard and make sure you see the owners on each respective metric at the bottom of the chart. See the example below

Product Component

Metrics Framework

More Detail

In the header to define measures in a data source, you previously had to use the field measure if you had only one measure defined. Then, if you added multiple measures later, you were required to make measure plural to measures. We thought that was rather silly and also inconsistent with the other objects in the YAML config.

To match the semantic of other fields in the file (e.g., identifiers, dimensions ), measures is now the field to use regardless of how many measures you have.

Verification Steps

Use measures to define a single or multiple measures.

measures:    - name: messages      description: The total number of messages sent.      expr: "1"      agg: sum

Product Component

Metrics Framework

More Detail

Previously, if you capitalized the name of a data source, such as the below example, there would be errors when processing the YAML files. Oopsie Daisy. We've fixed this so that names are now case-insensitive.

**data_source:  name: Rainfall_fct  description: rainfall across all weather stations, starting in Oct 2020.  ...**

Verification Steps

Create a datasource with a capitalized name and confirm it works.

Product Component Metrics Framework

More Detail

We removed the version from the YAML file because, well frankly, it wasn't used for anything! This applies to the version you find in the metrics YAML as well as your data sources. You no longer need to specify a version in your data sources or metric YAML.

Verification Steps

1. Create and verify a config without the version field
2. Say "you're welcome" to your fingers because they can type less when creating configs.

Product Component

Metrics Catalog

More Detail

Our Metric Chart Filtering is a nifty feature enabling you to filter and split metrics by dimension. In v1 we displayed the dimensions exactly as they are shown in your database with an added dunder (__)separating the table name from the column name. For example, if you have a table called "users" with a column called "user_name" our UI would display "usersuser_name". This is an incredibly useful feature but according to our users, maybe isn't so nice to look at. We listened, so in v2 of Metric Chart Filtering we've updated the way we display dimensions as well as the way we let you select them. You will no longer see the hard to read "usersuser_name" but instead, get "Users > User Name" as well as a spiffy new dropdown to help you select.

Verification Steps

1. Visit any Metric Page and open up the Chart Filters.
2. Open up the Dimension Dropdown to select a dimension identifier (table in your database).
3. After selecting a dimension identifier you will be able to select a dimension type (row in the table in your database. Now you can select the dimension values (values the row in your database) that you would like to filter your chart by.

Product Component

Metrics Catalog

More Detail

The MQL Query Logs allow you to look at the history of activity in the Transform UI to determine what queries were run, when they were run, the identifier of the query, the query execution time, and whether it was successful or failed. In addition, you can dig into the query execution details by looking at the server log and generated SQL from the request — all through the Transform interface. These logs collect information so you can troubleshoot requests and investigate activity on your interface. We realized that, while our customers are awesome, it was unreasonable to assume that they are omnipotent and that they can keep a mental model of all their queries, and so we added this log.

Verification Steps

1. Go to Settings and Select MQL Query Logs
2. View queries - you can search and filter by execution time, metrics, and statuses
3. Click on a query id to dig into logs, SQL, and results. You can share the query log with Transform in case you need the team's help to debug an issue.

Product Component

Interfaces - MQL CLI

More Detail

To write a table in your data warehouse, previously you would use the materialize command. Now, you should use the mql query command with a new argument we added called as-table.

Verification Steps

New Command: mql query --metrics metric_name --dimensions dimension_name --as-table schema.table

Old Command: mql materialize --metrics metric_name --dimensions dimension_name --table schema.table

PS: Don't worry, we will be using the materialize command for an exciting upcoming new feature which you'll hear more about soon!

Product Component

Metrics Catalog

More Detail

We really like underscores, but we know they can be hard to read. We now clean the dimension names in the Filter and Group By dropdown. For example, if the dimension name is "created_at", we will clean it up to display it as "Created At" in the chart interface.

If you would like to choose a different name than what we've inferred, you can add

display_name: <desired_display_name> to a given dimension in the YAML files.

Verification Steps

Navigate to the user interface for any metric and open up the display panel to confirm your dimension names have been cleaned of underscores.

Product Component

Metrics Framework

More Detail

You can now specify in your metrics.yaml whether you want to create a metric from a measure. By default, we don't create a metric from measures unless you specify create_metric = true.

Verification Steps

Example YAML snippet. Add create_metric: true to create a metric from your measure.

measures:    - name: new_stations      expr: "1"      agg: sum      create_metric: true

Run list-metrics in the CLI or visit the Transform UI and search the measure to confirm your metric is created.

mql list-metrics

Product Component

Metrics Catalog

More Detail

If you have a metric that's of the type ratio or expr, the dimensions dropdown under the chart to filter would not populate, and it would throw an error. We hope that you didn't think that it was because we didn't want you to filter dimensions on these metrics — it's just that we had a bug (which we fixed)! You can now see and filter the dimensions associated with these metric types.

Verification Steps

Navigate to a metric that's not a measure_proxy and open up the filter panel to confirm your dimensions are populated.

Product Component

Interfaces - MQL CLI

More Detail

We've made some performance improvements to our CLI so that commands return results faster. In our testing, we noticed a significant performance boost after releasing these changes.

Verification Steps

This speedup will affect any mql command.

mql identify

Product Component

Metrics Catalog

More Detail

We recently released the MQL Server List page that displays all of your organizations active MQL Servers. That was super cool but it didn't yet provide any insight into the state of your MQL Servers.

This release adds a "Heartbeat" column to the MQL Server List that lets you know how your server is running. It will display the status of your MQL Server and the time that status was updated.

Verification Steps

1. Open the Settings Panel, use the "MQL Server List" link to view the MQL Server List Table.

2. Use the icon in the "Last Heartbeat" row to open up the status modal to see more information about your MQL Server Status.

Product Component

Interfaces - MQL CLI

More Detail

It's easier to type and what the heck even is a model-config anyways?

Verification Steps

mql validate-configs --config-dir path/to/dir

Product Component

Interfaces - MQL CLI

More Detail

We will no longer ask you every time you make a query when you have a pinned model. Instead we're providing a set of mql commands that allow you to pin and unpin models for local querying.

Verification Steps

Commit configs and pin them

1. Commit and pin configs at the same time mql commit-configs --config-dir --pin true
   1. OR mql commit-configs --config-dir
   2. Then, confirm you would like to pin the model
   3. If you don't you will be given a command to pin that model in the future

Product Component

Interfaces - MQL CLI

More Detail

We made it easier to choose specific files or folders and run validation. They no longer have to be in git repos, so you can develop or test new metrics and queries wherever you feel comfortable.

There is nothing that gives data people nightmares more than unstructured json displayed as text. So rather than giving you error messages in the most stressful way we could imagine, we made it so they print a nice message for you!

Verification Steps

Test validation in your Current Working Directory

1. cd to a place with configs or to a git repo
2. Run mql validate-configs

Test validation for a specific git repo or section of your git repo

Run mql validate-configs —config-dir path/to/the/project

Test a local File -This should make developing and testing new metrics easy!

1. Make a yaml file (with a data source and metric) in any folder on your laptop.
2. Run mql validate-configs —config-dir path/to/the/file

Add a typo and see an error message

1. nam: revenue
2. Run mql validate-configs
3. File names and line numbers should make it easy to identify and fix errors

Currently, we only show one error at a time. In the future, we will make these errors into a structured list so you can check more of them off faster.

Test changes with a query

There are two paths to using this feature during development of Transform configs.

Single Queries Against a Local Model

Use the --config-dir . query flag when issuing MQL queries to test the query against a local model.

➜  transform git:(marco_test_config_dirs) mql query --metrics question_replies --dimensions created_at --config-dir .✔ 🌱 Successfully parsed local configs✔ Query initialized: 1617746880310_0000005✔ Success 🦄 - query completed after 7.90 seconds|   question_replies | created_at                ||-------------------:|:--------------------------||                  7 | 2020-11-13 00:00:00+00:00 |

Push a model, pin the local CLI config to that model

Push a commit via the commit-model-configs command, and allow us to save the model id for future queries:

➜  transform git:(marco_test_config_dirs) mql commit-model-configs --config-dir .✔ 🦄 Successfully committed model id 6940 on branch marco_test_config_dirs with commit bf73e5a1302aa2c20e5c51d1d090dad3c988e23f-dirtyWould you like to pin this this model commit for future MQL queries? [y/N]: ySuccess! You can make future queries against the model id 6940.

Run queries as you regularly would, and we will prompt you asking whether you want to reuse your saved model id:

➜  transform git:(marco_test_config_dirs) mql query --metrics question_replies --dimensions created_at📌 We've found a pinned model ID from a previous request, uploaded at 2021-04-06T21:55:18.240188+00:00, which does not match the primary model for the Transform Data organization.Do you want to reuse this model for this request? If not, we will default you to the current model for Transform Data. [y/N]: y✔ Query initialized: 1617746880310_0000005✔ Success 🦄 - query completed after 2.55 seconds|   question_replies | created_at                ||-------------------:|:--------------------------||                  7 | 2020-11-13 00:00:00+00:00 |

Product Component

Interfaces - MQL CLI

More Detail

Instead of using python lists, you can now query MQL using strings!

Previous MQL query

mql query --metrics '["revenue"]' --dimensions '["ds"]' --order '["ds"]'

New and improved

mql query --metrics revenue --dimensions ds --order ds

Verification Steps

Try your favorite query without the list!

mql query --metrics revenue --dimensions ds --order ds

Product Component

Interfaces - MQL CLI

If you want to group by (often high cardinality) identifiers, you can now follow your heart and use these as dimensions in MQL.

Verification Steps

Run a query using identifiers as dimensions, such as (assuming user_id is an identifer) mql query --metrics messages --dimensions user_id

Product Component

Metrics Framework

More Detail

You can now reference various databases in the data source YAML.

Verification Steps

In the Data Source Config, if you want to reference the database, you can do so directly in the sql_table or sql_query:

sql_table: **demo_db**.demo_schema.transactions

Product Component

Interfaces - MQL CLI

More Detail

Infinite data sounds awesome. Infinite printing in the CLI is not awesome. The CLI now defaults to run queries with a limit of 100 rows so you don't fry your computer! Some people just like to have no limits(—limit inf) so you can do that too!

Verification Steps

Default limit of 100 in the CLI

mql query —metrics revenue —dimensions ds = mql query —metrics revenue —dimensions ds --limit 100

Return any number of rows

mql query —metrics revenue —dimensions ds --limit 1

Or return all the rows

mql query —metrics revenue —dimensions ds --limit inf

Product Component Interfaces - MQL CLI

More Detail

The mql setup command now makes it easier for beginners to get up and running and for power users to connect to multiple MQL Servers.

Verification Steps

Change API Keys

mql setup -k new_api_key

Override the MQL Server URL

mql setup -o mql.server.url

Then revert to your main MQL Server

mql setup --remove-override

Reset your mql setup

mql setup -r

Product Component

Interfaces - MQL CLI

More Detail

Knowing your MQL Servers are buzzing away can be a great relief. Once you have mql setup in place, mql health-report returns information about any available servers associated with your organization.

Verification Steps

1. Run mql health-report

> mql health-report🏥 Health Report for 2 MQL Servers at Transform Sample• redshift-staging: https://mql-redshift.sample.transformdata.io/ running commit 709e448c255c1e4eb4d967fe60a5f6f59b5b18cf  • ✅ redshiftdbclient_select_1: No Errors  • ✅ mysqldbclient_select_1: No Errors• databricks-staging: https://mql-databricks.sample.transformdata.io/ running commit 709e448c255c1e4eb4d967fe60a5f6f59b5b18cf  • ❌ databricksdbclient_select_1:  TEMPORARILY_UNAVAILABLE: Cluster 0312-014734-sabre186 is temporarily unavailable. The current cluster state is Terminated. Please retry your request after 30 seconds.  • ✅ mysqldbclient_select_1: No Errors

Product Component

Interfaces - MQL CLI

More Detail

Sometimes you just want to know the MQL Server is there!

Verification Steps

1. Run mql ping

> mql ping🏓 Received HTTP 200 code from MQL in 0.9650 seconds. Healthily running commit: 561ab8d6f8cb270568ee7c6e43b913fb962462dc

Product Component

Metrics Catalog

More Detail

Previously the User Management page was only available to "admins", which restricted access to Transform Employees. That wasn't very helpful. Now, the User Management page is available to Org Admins enabling you to add/remove/edit your Transform account members freely.

Verification Steps

Login as an Org Admin and open the settings panel.

You should now be able to access the User Management page through the "Manage Users" link.

Product Component

Metrics Framework

More Detail

We have been in a fierce debate between the beauty of identifier/metric and the pain of quoting in queries on downstream tables produced by Transform select "identifier/metric" from table. We heard great cases for both sides but ultimately we decided to move forward in the simplest approach, dunders.

We have all made columns using an entity and a dimension like user_type(sunders?) so we hope the dunder user__type produced in the framework and used to query Transform is similarly intuitive... but twice the fun.

This resolves a number of issues for our Snowflake users and greatly simplifies the queries the framework produces for all. Fun fact: Snowflake is case sensitive and defaults to uppercase. so when you type SELECT revenue FROM transactions Snowflake hears SELECT "REVENUE" FROM "TRANSACTIONS" and errors if you're not bossy enough... SELECT "revenue" FROM "transactions".

Verification Steps

mql query —metrics revenue —dimensions user/signup_channel is now

mql query —metrics revenue —dimensions user__signup_channel and returns a table like

revenue   | user__signup_channel100       | paid200       | organic

Product Component Metrics Catalog

More Detail

If you chose to pivot a chart in the UI based on a dimension, we wanted you to use mind-reading to figure out which line corresponded to each dimension. Now we have line labeling!

Verification Steps

On a metric page, add a pivot to your chart to view the line label in the tool tip:

Product Component

Metrics Catalog

More Detail

When displaying the y-axis for values less than 1, we would show the number 1 repeated for every line because it's the best (but unhelpful). Now we show much more useful decimals numbers.

For large numbers, we now round or show commas.

Verification Steps

Y-Axis for values less than 1

Product Component

Interfaces - MQL CLI

More Detail

There's a time and place for stack traces and losing your internet connection, entering a typo or having the wrong API key are not those times! Instead we've replaced them with useful tips, encouragement and emojis.

Verification Steps

Lost Connection

1. Turn off wifi
2. Enter mql ping

Typos

1. Try a typo in your mql query

Wrong API Key

Warning: Make sure your MQL API Key is saved in a secure location

1. Run mql setup -k wrong_key to change your API key to an invalid key
2. Run mql ping

and a few more here and there that you can discover!