Skip to main content

Release - April 9th, 2021

· 8 min read
Roxanna Pourzand
Product Manager

✨ New ✨#

Validate and Test Models Locally#

Details

Product Component

Interfaces - MQL CLI

More Detail

We made it easier to choose specific files or folders and run validation. They no longer have to be in git repos, so you can develop or test new metrics and queries wherever you feel comfortable.

There is nothing that gives data people nightmares more than unstructured json displayed as text. So rather than giving you error messages in the most stressful way we could imagine, we made it so they print a nice message for you!

Verification Steps

Test validation in your Current Working Directory

  1. cd to a place with configs or to a git repo
  2. Run mql validate-configs

Test validation for a specific git repo or section of your git repo

Run mql validate-configs —config-dir path/to/the/project

Test a local File -This should make developing and testing new metrics easy!

  1. Make a yaml file (with a data source and metric) in any folder on your laptop.
  2. Run mql validate-configs —config-dir path/to/the/file

Add a typo and see an error message

  1. nam: revenue
  2. Run mql validate-configs
  3. File names and line numbers should make it easy to identify and fix errors

Currently, we only show one error at a time. In the future, we will make these errors into a structured list so you can check more of them off faster.

Test changes with a query

There are two paths to using this feature during development of Transform configs.

Single Queries Against a Local Model

Use the --config-dir . query flag when issuing MQL queries to test the query against a local model.

➜  transform git:(marco_test_config_dirs) mql query --metrics question_replies --dimensions created_at --config-dir .✔ 🌱 Successfully parsed local configs✔ Query initialized: 1617746880310_0000005✔ Success 🦄 - query completed after 7.90 seconds|   question_replies | created_at                ||-------------------:|:--------------------------||                  7 | 2020-11-13 00:00:00+00:00 |

Push a model, pin the local CLI config to that model

Push a commit via the commit-model-configs command, and allow us to save the model id for future queries:

➜  transform git:(marco_test_config_dirs) mql commit-model-configs --config-dir .✔ 🦄 Successfully committed model id 6940 on branch marco_test_config_dirs with commit bf73e5a1302aa2c20e5c51d1d090dad3c988e23f-dirtyWould you like to pin this this model commit for future MQL queries? [y/N]: ySuccess! You can make future queries against the model id 6940.

Run queries as you regularly would, and we will prompt you asking whether you want to reuse your saved model id:

➜  transform git:(marco_test_config_dirs) mql query --metrics question_replies --dimensions created_at📌 We've found a pinned model ID from a previous request, uploaded at 2021-04-06T21:55:18.240188+00:00, which does not match the primary model for the Transform Data organization.Do you want to reuse this model for this request? If not, we will default you to the current model for Transform Data. [y/N]: y✔ Query initialized: 1617746880310_0000005✔ Success 🦄 - query completed after 2.55 seconds|   question_replies | created_at                ||-------------------:|:--------------------------||                  7 | 2020-11-13 00:00:00+00:00 |

Query MQL using Strings#

Details

Product Component

Interfaces - MQL CLI

More Detail

Instead of using python lists, you can now query MQL using strings!

Previous MQL query

mql query --metrics '["revenue"]' --dimensions '["ds"]' --order '["ds"]'

New and improved

mql query --metrics revenue --dimensions ds --order ds

Verification Steps

Try your favorite query without the list!

mql query --metrics revenue --dimensions ds --order ds

Query Identifiers as Dimensions#

Details

Product Component

Interfaces - MQL CLI

If you want to group by (often high cardinality) identifiers, you can now follow your heart and use these as dimensions in MQL.

Verification Steps

Run a query using identifiers as dimensions, such as (assuming user_id is an identifer) mql query --metrics messages --dimensions user_id

Database reference in data source#

Details

Product Component

Metrics Framework

More Detail

You can now reference various databases in the data source YAML.

Verification Steps

In the Data Source Config, if you want to reference the database, you can do so directly in the sql_table or sql_query:

sql_table: **demo_db**.demo_schema.transactions

Timeout Improvements#

Details

Product Component

Interfaces - MQL CLI

More Detail

Infinite data sounds awesome. Infinite printing in the CLI is not awesome. The CLI now defaults to run queries with a limit of 100 rows so you don't fry your computer! Some people just like to have no limits(—limit inf) so you can do that too!

Verification Steps

Default limit of 100 in the CLI

mql query —metrics revenue —dimensions ds = mql query —metrics revenue —dimensions ds --limit 100

Return any number of rows

mql query —metrics revenue —dimensions ds --limit 1

Or return all the rows

mql query —metrics revenue —dimensions ds --limit inf

MQL Setup Ease of Use#

Details

Product Component Interfaces - MQL CLI

More Detail

The mql setup command now makes it easier for beginners to get up and running and for power users to connect to multiple MQL Servers.

Verification Steps

Change API Keys

mql setup -k new_api_key

Override the MQL Server URL

mql setup -o mql.server.url

Then revert to your main MQL Server

mql setup --remove-override

Reset your mql setup

mql setup -r

MQL Health Report#

Details

Product Component

Interfaces - MQL CLI

More Detail

Knowing your MQL Servers are buzzing away can be a great relief. Once you have mql setup in place, mql health-report returns information about any available servers associated with your organization.

Verification Steps

  1. Run mql health-report
> mql health-report🏥 Health Report for 2 MQL Servers at Transform Sample• redshift-staging: https://mql-redshift.sample.transformdata.io/ running commit 709e448c255c1e4eb4d967fe60a5f6f59b5b18cf  • ✅ redshiftdbclient_select_1: No Errors  • ✅ mysqldbclient_select_1: No Errors• databricks-staging: https://mql-databricks.sample.transformdata.io/ running commit 709e448c255c1e4eb4d967fe60a5f6f59b5b18cf  • ❌ databricksdbclient_select_1:  TEMPORARILY_UNAVAILABLE: Cluster 0312-014734-sabre186 is temporarily unavailable. The current cluster state is Terminated. Please retry your request after 30 seconds.  • ✅ mysqldbclient_select_1: No Errors

MQL Ping#

Details

Product Component

Interfaces - MQL CLI

More Detail

Sometimes you just want to know the MQL Server is there!

Verification Steps

  1. Run mql ping
> mql ping🏓 Received HTTP 200 code from MQL in 0.9650 seconds. Healthily running commit: 561ab8d6f8cb270568ee7c6e43b913fb962462dc

Additional User Management#

Details

Product Component

Metrics Catalog

More Detail

Previously the User Management page was only available to "admins", which restricted access to Transform Employees. That wasn't very helpful. Now, the User Management page is available to Org Admins enabling you to add/remove/edit your Transform account members freely.

Verification Steps

Login as an Org Admin and open the settings panel.

You should now be able to access the User Management page through the "Manage Users" link.

🐛 Bug Fixes 🐛#

Foreign Identifiers and Dunders#

Details

Product Component

Metrics Framework

More Detail

We have been in a fierce debate between the beauty of identifier/metric and the pain of quoting in queries on downstream tables produced by Transform select "identifier/metric" from table. We heard great cases for both sides but ultimately we decided to move forward in the simplest approach, dunders.

We have all made columns using an entity and a dimension like user_type(sunders?) so we hope the dunder user__type produced in the framework and used to query Transform is similarly intuitive... but twice the fun.

This resolves a number of issues for our Snowflake users and greatly simplifies the queries the framework produces for all. Fun fact: Snowflake is case sensitive and defaults to uppercase. so when you type SELECT revenue FROM transactions Snowflake hears SELECT "REVENUE" FROM "TRANSACTIONS" and errors if you're not bossy enough... SELECT "revenue" FROM "transactions".

Verification Steps

mql query —metrics revenue —dimensions user/signup_channel is now

mql query —metrics revenue —dimensions user__signup_channel and returns a table like

revenue   | user__signup_channel100       | paid200       | organic

Added Chart Label#

Details

Product Component Metrics Catalog

More Detail

If you chose to pivot a chart in the UI based on a dimension, we wanted you to use mind-reading to figure out which line corresponded to each dimension. Now we have line labeling!

Verification Steps

On a metric page, add a pivot to your chart to view the line label in the tool tip:

Y-Axis Labeling Improvements#

Details

Product Component

Metrics Catalog

More Detail

When displaying the y-axis for values less than 1, we would show the number 1 repeated for every line because it's the best (but unhelpful). Now we show much more useful decimals numbers.

For large numbers, we now round or show commas.

Verification Steps

Y-Axis for values less than 1

Fewer Stack Errors in the CLI#

Details

Product Component

Interfaces - MQL CLI

More Detail

There's a time and place for stack traces and losing your internet connection, entering a typo or having the wrong API key are not those times! Instead we've replaced them with useful tips, encouragement and emojis.

Verification Steps

Lost Connection

  1. Turn off wifi
  2. Enter mql ping

Typos

  1. Try a typo in your mql query

Wrong API Key

Warning: Make sure your MQL API Key is saved in a secure location

  1. Run mql setup -k wrong_key to change your API key to an invalid key
  2. Run mql ping

and a few more here and there that you can discover!