Product Component

Metrics Catalog

More Detail

Transform has now released our next iteration of the Homepage. This new Homepage has been refocused around Annotations and Subscriptions, providing more rich context about what is happening in a company's data ecosystem, tailored to each individual person at a company. The Homepage is now more of an insights hub, capturing important details about "the why" and helping contextualize what is causing metric values to move.

This new Homepage features Annotations in the left pane. When you click the "see more details" within an Annotation, the card will open to show all metrics affected by the Annotation and how the affected metric values moved during the relevant time range.

The new Homepage also has a set of Subscriptions widgets on the right-hand side of the page that show metric values and their percentage change from the last period.

At the bottom, there is a "Metrics You May Care About" section, containing a set of widgets that enable data consumers to subscribe to new metrics directly inline.

We will continue building out more features into this Homepage and expanding the functionality in the coming weeks, to help bring rich context about what is happening in our customers' data ecosystems.

Verification Steps

Login to Transform and confirm the updates to the homepage.

Product Component

CLI

More Detail

We have fixed a bug where the order was not being respected when a time comparison was applied.

Verification Steps

Run a query on the CLI like the following:

mql query —metrics metric_name —dimensions ds__day —time-comparison dod —order -ds__day

Product Component

Metrics Framework

More Detail

Snowflake allows for session-level overrides for the start of the week, which means one user might be seeing weekly granularity metrics (or results from other week-based functions) returning with a week start of Monday while another sees a different week start (e.g. Wednesday).

This change brings Snowflake in line with our other engines and ensures consistency across user sessions within an organization by pinning all Snowflake sessions to have WEEK_START set to the ISO standard of Monday. We have pegged the setting to ‘1’ with no available override mechanism, although if such a feature is needed (e.g. customers who wish WEEK_START to be Sunday), we can expose an organization-level configuration setting for it.

For more information on the implications of this change, please refer to the Snowflake documentation on the WEEK_START parameter, how it affects calendar weeks, weekdays and the associated list of affected functions, and examples of how different WEEK_START values might affect output.

Please note that the only customers affected are:

1. Customers running Snowflake warehouse instances with account or session-level overrides for the WEEK_START parameter that fix it to a value other than 0 or 1. These customers will now see MONDAY as the week start in all cases.
2. Customers who are using the DAYOFWEEK function inside the expr parameter with the legacy Snowflake default WEEK_START = 0. For these customers, DAYOFWEEK will now return the ISO-standard values of 1 (Monday) through 7 (Sunday) instead of Snowflake’s legacy default values of 0 (Monday) through 6 (Sunday).

Verification Steps

Use a weekly time granularity in the CLI or Metrics Catalog and confirm it uses the ISO standard date.

Product Component

Metrics Catalog

More Detail

We’re excited to release Notifications and Subscriptions for Transform customers!

Subscriptions allow users to subscribe to metrics that they care about, so they can get updates and context about those metrics.

The notification types launching today include updates when new annotations are authored, new questions are asked, users are subscribed to metrics, and notifications for metric owners when their colleagues become subscribers to their metrics.

In the near future, we will be launching new notification types for tracking when metric values go up/down and we’ll also launch fine-grained controls to help users decide how they receive notifications: in Transform, over email, or via Slack.

Verification Steps

You’ll find the “Subscribe” button on all metric pages, and the new notification pane can be accessed by clicking the bell icon on the upper right of any Transform page.

Product Component

Metrics Catalog

More Detail

When downloading a CSV with limited groupings on the chart, a modal appears that allows you to choose if you want all groupings or only the ones on the chart (limit 25). After making this selection, the modal n't close by default, and so now made sure it does.

Verification Steps

Download a CSV with a grouping that's been limited (more than 25 dimension values). Notice that the modal disappears after selection.

Product Component

Metrics Catalog

More Detail

We have fixed an issue with time comparison where time periods with no data were being padded with the latest non-null data (the default setting for the pandas pct_change method - documentation here). When time comparison is added to a query now, no fill method will be applied, and therefore nulls will be respected.

Verification Steps

Navigate to a non-cumulative metric page that has gaps in data. Apply day over day time comparison (or the smallest available time comparison period) via chart filters. For any periods where there was no data, you should see either an infinite value (if the next period had data) or a null value (if the next date had no data).

Product Component

Metrics Catalog

More Detail

We are now trimming the current date from chart queries by default, since the current date is not complete. The current date is determined in UTC, but we are also working on a feature to override this timezone at the organization & data source levels.

Verification Steps

Visit a metric page. The latest date shown on the chart should be no later than yesterday (in UTC).

Product Component

Security

More Detail

Transform now supports single sign-on with Okta. This integration supports both IDP and SP initiated login.

Okta administrators can add Transform as an application from Okta’s integration network and add users that should have access to Transform. Once enabled in Okta, you can follow the instructions provided to enable this using both Okta and Transform. In Transform, we’ve added a new Settings Tab called Security, which you will use to configure Okta.

Verification Steps

If your organization uses Okta, find Transform in Okta’s integration network. See our documentation for step-by-step instructions.

Product Component

Metrics Framework

More Detail

We’ve introduced both improved model and data warehouse validation that will make developing and committing configs easier by catching the errors ahead of the model being committed and surfacing useful errors.

Some examples of improved validation included but are not limited to the following:

• Ensuring referenced tables exist in your fields and data warehouse
• Checking that SQL expressions are valid
• Making sure Inputs meet the expected constraints (e.g., name uniqueness, having a primary time dimension, ensuring you’ve completed all fields)
• Handling empty documents and Nonetypes.

Verification Steps

• Upgrade your MQL CLI via pip install --upgrade transform
• Run mql validate-configs with your model to see new validations. If there are no issues found, you will not observe many changes to the validation process.

Note that we are consistently adding more data warehouse-based validations, if you run into issues please reach out to us. Additionally, if you experience validation failures due to timeouts you can add --skip-dw to the validate-configs or commit-config commands.

Product Component

Metrics Catalog

More Detail

While mostly for new organizations using Transform, we wanted to call out that we’ve introduced a new organization creation flow in Transform. This feature allows you to fill out a form with information which when submitted will create your Transform organization, spin up your MQL server, and guide you to help connect Github with Transform. Previously, this work was manual.

For existing users of Transform with an active organization, you might notice the Installation page differs a bit and now shows each step in the setup flow.

Verification Steps

You should only follow this guide if you are creating a new Transform instance: visit https://app.transformdata.io/create_organization.

For existing users curious about the new setup flow, visit the new install page to see what this looks like.

Product Component

Metrics Catalog

More Detail

The same menu that contains the export CSV now has an option called "Copy Last Day's Values". This simply copies out the latest day of data. If there is a grouping applied, it will copy out each respective dimension and value in list form with the latest date. If there is no grouping applied, it will copy out a single day of data with the date. The copy is meant to allow for easy pasting into a tool like Google Sheets or Excel.

Verification Steps

Open up any chart menu and confirm you see this option.

Product Component

Metrics Catalog

More Detail

We fixed a bug where dimensions with a large number of values could sometimes cause the page to be unresponsive (in the chart filter menu).

Verification Steps

Find a metric that has a dimension associated that contains a large number of cardinal values. Determine when you filter and choose from those dimensions in the filter that everything works properly.

Product Component

Metrics Catalog

More Detail#

For measure proxy metrics using SUM, SUM_BOOLEAN, and COUNT_DISTINCT aggregations, nulls should be replaced with 0 when a granularity is applied. We’ve fixed a bug where charts were not filling 0 for these metrics.

Verification Steps#

Navigate to a metric page for a measure proxy metric using using SUM, SUM_BOOLEAN, or COUNT_DISTINCT aggregation. Use the chart filters to apply a granularity. When the chart reloads, you should see a continuous line with no gaps in data.

Product Component

Metric Ownership and Management

More Detail

• Adding new types of rich metadata for metrics (display value formatting, increase is good/bad, and business owners).
• Ability to lock metadata down in configs or leave editable in Transform UI.
• New metric modal experience in the Transform UI.

What’s changing?

There is a problem today where some metadata fields like description and owner fields can be edited both in the config as well as in the UI, and this creates potential mismatch between what appears in the config and the UI. Because we cannot sync things back to config files, and because some organizations will actually want to lock down metadata under version control, we needed a solution that could ensure no collisions.

To accomplish this aim, we are improving metric config files so that they have a clearly delineated locked-metadata section, which is optional to fill in through the config. If any specific locked-metadata field is authored in the config, then that field becomes locked in the UI and is no longer editable in the UI — but all other fields remain editable in the UI. The config will be the “source of truth” for any metadata fields.

• We are also making entirely new types of metric metadata fields to help customers create more rich context and understanding metrics they build. New fields include: Business Owner, Display Number Type Format, Metric Increase Good or Bad. In the future, we’ll continue adding more metadata fields so that the context around metrics becomes richer.
• The metric owner field in the config will continue being called “Owner” and in the UI we will refer to the metric config owner as the “Technical Owner”. This change will help us with adding a new “Business Owner” field that differentiates folks who may have built the technical ETL pipeline to create the metric, from the business people or business analysts who understand the context about why a metric value may have changed. Technical owner is not editable in the UI (since it is set in the config as we described above).
• We will do away with the separation of config authored description and UI authored description, just having one description in the future that can be set as locked-metadata in the config or authored in the UI. We have a preference for companies authoring descriptions in the UI because we have found that the metric descriptions authored in config files are generally of poor quality and do not take advantage of our markdown editing abilities.

Our intention, and what we believe is the best user experience, is that people choose to edit almost all their metadata in the UI and only choose to lock down metadata in config files when needed for specific business reasons.

Product Component

Metrics Framework

More Detail

We now support Google BigQuery as a source data warehouse to build metrics off of in Transform!

Verification steps

If you use BigQuery as your source data warehouse, you can verify the integration by building your Transform configurations off of data in BQ.

Product Component

Metrics Catalog

More Detail

We now provide users the option to sign up for Transform directly as opposed to needing to have their account created by someone. Users associated with an existing Transform organization (as determined by their email domain) can access Transform's login page at app.transformdata.io and hit "sign up" to create an account. Once the user signs up with their email and password, they will receive an email prompting them to verify their email. Once verified, they can log into Transform and celebrate!

Note: We will also be releasing updates to the User Management page shortly so that users can invite people (as opposed to the current behavior of admins needing to manually creating users), which will send the invited user an email to access the signup flow.

Verification steps

Visit app.transformdata.io and confirm you have an option to log in or sign up. New users with a domain tied to an existing organization should be able to signup in Transform without an administrator.

Product Component

Metrics Catalog

More Detail

On the user management page, we've added a search bar so you can search for users. The search filters by name and email.

Verification Steps

Navigate to Settings —> User Management and see the search bar. Type in a user to find one or narrow down results.

Product Component

More Detail

Our Granularity option in the chart by default trims incomplete time periods so there are no false trends displayed.

If you would prefer the whole window of data when selecting granularity, we added a toggle in the granularity menu that indicates whether you want to return untrimmed granularity. If you select untrim as opposed to trim (default), it will include the whole time period that's selected on the chart regardless of incomplete windows.

For example, if you are looking at a metric chart that shows data from March 1st to June 15th, selecting a granularity of monthly with the default behavior will show you March, April, and May's monthly aggregated value since June is only a partially complete month. Conversely, selecting untrim will show you March, April, May and June. Note that including incomplete windows may show false trends of spikes or dips as compared to complete months since the data is not complete.

Verification Steps

View Granularity menu in edit chart, confirm you have an option to select trim (default selection) or untrim, and validate with a time range that contains an incomplete window (e.g., incomplete month or week).

Product Component

Metrics Catalog

More Detail

We updated the header of Transform's application on every page, which includes some aesthetic updates as well as an avatar logo on the right hand indicating the person using the app's identity.

We've also updated the right-hand side menu on the metric page that includes Tiers, Owner, Approval and Changes. The information on the right-hand side is largely the same, but it's organized and displayed differently.

Verification Steps

Log in to the application and see the updates to the header. Navigate to any metric page to see the updates to the right hand side.

Product Component

Metrics Catalog

More Detail

There are certain cases in the Transform application where objects are created around metrics that have since been removed from the Transform model. In this case, users will now see an indication that the metric used to exist but no longer does (with a graph that has a ghost icon). We wanted to expose this information so users won't be confused about a metric they might have viewed in the past and that there's a clear reason why the metric is no longer present.

Verification Steps

This is not easily verifiable unless a model has metrics that have since been removed. If you experience this case, you can verify that Transform shows the ghost metric graph.

Product Component

Metrics Catalog

More Detail

By default, we limit the number of lines we show on the chart when a user selects a grouping (25 dimensions). This applies to additive metrics only -ones created off of sum and sum_booleanmeasures. As a result, we added a new menu next to the grouping which allows users to choose how many lines/dimension values they want to show.

Verification Steps

Open "Edit Chart" for a metric that's additive, select a dimension grouping, and confirm next to the grouping menu there is a new menu that allows you to select how many lines to display on the chart (corresponding to respective dimension values).

Product Component

Metrics Catalog

More detail

We recently made an update to our dimension grouping charts which limits the number of dimensions displayed on the chart to 25 (for additive metrics). The remaining metrics are bundled into an "other dimensions" category. As a result, we've updated our CSV export to open a modal that allows you to choose whether to export the full data set or the grouping as limited by the chart.

Verification Steps

Navigate to an additive metric and select a dimension grouping with more than 25 dimension values. Hit the Export CSV option and confirm you have options with how to export.

Product Component

Metrics Catalog

More Detail

Previously, homepage charts displayed the last 30 data points, regardless of the date. With this update, “Mini” charts, like on the homepage, will show the past 90 days of data, and “Medium” format charts, like on the collections pages, will display 180 days of data. “Large” charts, found on the metrics page, will not change. We are making this change because visualizations could appear discontinuous or broken if there were long gaps between data points.

You will now see more data by default for metrics aggregated at a daily grain. For monthly granularities, you may see fewer data points i.e 3 data points in a 90-day window. This change does not affect charts on the metrics page, as you can customize the date range and granularity.

Verification steps

Visit Transform's homepage to view these updates

Product Component

Metrics Catalog

More Detail

Transform's Model Page gives you a birds-eye view of all metrics and data sources. We added a link under each metric that takes you to the corresponding metric page.

Verification Steps

Navigate to any metric on the model page and find "Visit Metric" underneath the metric definition

Product Component

Metrics Catalog

More Detail

Transform's Metric Catalog previously showed a default daily time granularity for all metrics. We now automatically apply the granularity defined for the data source of the metric. You can specify a granularity of daily, weekly, monthly, quarterly, yearly in the YAML configuration, and the catalog will automatically apply that for all metrics build on top of the data source. Specifically, we will show the granularity that's applied in the menu and automatically hide any granularity options that don't make sense. This includes hiding options for the Date Selector that might not make sense given the metric's granularity

Verification Steps

Visit any metric that's built off of a data source that is not daily granularity and confirm the right granularity is applied in the Catalog.

Product Component

Metrics Catalog

More Detail

Transform's application previously had no indication that a user's authentication was expired in certain cases, which leads the user to think they are still logged in. This leads to users running into many authentication errors on the graphs. We added an overlay that will warn the user that they need to log in again so they don't try to slice and dice data and get an authentication error.

Verification Steps

Confirm you don't get an "Unauthorized Credential" error on the graphs in Transform and instead receive a modal prompting you to login again.

Product Component

Metrics Catalog

More Detail

We've done a rebrand on colors and fonts and a few other aesthetic pieces, so you'll likely notice a makeover on our User Interface. In other words, "new look, same great product!"

Verification Steps Login to the product and notice the new look and feel.

Product Component

Metrics Framework

More Detail

You can optionally specify grain to date in your cumulative metric configuration to accumulate the metric from the start of a grain (i.e. week, year, month). When using a window, like a month, Transform will go back one calendar month, whereas grain to date will always start at the beginning of the grain regardless of the latest date of data.

In this example, we are calculating cumulative transaction revenue month to date using grain_to_date.

metric:  name: "revenue_mtd"  description: "revenue mtd"  owners:    - bobby@transformdata.io  type: cumulative  type_params:    measures:      - txn_revenue    grain_to_date: month

We support grain to date at the day, week, month, and quarter level.

Verification Steps

Update or create a new cumulative metric with grain_to_date configuration

Product Component

Metrics Catalog

More Detail

For any metric chart, the ... menu's "View Query Logs" option has been replaced by a "View SQL" option. We made this change from feedback that users mostly use this menu to quickly access the SQL and don't need to see the full query logs. This new option opens a modal that shows the source SQL and generated SQL for the given metric. The "View Server Logs" button in the modal takes you to the same place the "View Query Logs" button did to view the full query logs.

Verification Steps

Open the menu at the top of any chart and confirm you can access the View SQL modal.

Product Component

Metrics Catalog

More Detail

For additive metrics that can be summed to produce a total from a grouping, we added a "Total" at the top of the tooltip. The total will show the sum of values for metrics based on measures that use sum and sum_boolean aggregation types (e.g., additive typse)

Verification Steps

Select a "Group" in Edit Chart for a metric based on a measure that uses the sum or sum_boolean aggregation. Hover over the tooltip for any day and confirm a total is present in the tooltip.

Product Component

Metrics Catalog

More Detail

We have a new legend, which color codes and labels dimension groupings on the metric chart!

Verification Steps

Create a grouping and ensure you see a legend that specifies which color applies to which dimension values.

Product Component

Metrics Catalog

More Detail

Questions previously were only editable/deletable by question authors. We now allow admins to edit/delete/resolve replies on questions.

Verification Steps

Edit/Delete or take actions on questions that you did not author.

**Product Component**

Metrics Catalog

More Detail

We've made a few aesthetic and functional changes to the User Management and Team Management pages. Specifically,

User Management: we've updated controls for updating user roles and other actions

Team Management: We've consolidated the form for adding team members to any team, updated controls for both updating team roles and removing team members.

Verification Steps

Visit User Management and Team Management and notice new aesthetics and controls.

Product Component

Metrics Catalog

More Detail

There was a bug where any "x to date" selection in the date selector (e.g., month, year, quarter, week) would not return values even if values are present. This has been resolved.

Verification Steps

Use any preset selection with "x to date" for a metric that has values for the time period and ensure data is returned.

Product Component

Metrics Catalog

More Detail

We fixed an intermittent error that reported there was a client side exception when interacting with Transform charts.

Verification Steps Confirm you no longer see occurrences of the "Application Error A client side exception has occurred"

Product Component

Interfaces - JDBC

More Detail

Using ORDER BY in JDBC would incorrectly throw an error. This has been resolved

Verification Steps

Use ORDER BY in a query in the JDBC driver. For example:

SELECT * FROM MQL_QUERY(<messages by user__country, date ORDER BY date>);

Product Component

Interfaces - SQL

More Detail

We've released the production version of our integration with Mode analytics. The integration will enable Mode users to have the flexibility of reporting and BI of Mode, with the assurance of accurate metrics from Transform.

Our integration allows you to express MQL requests through SQL using a JDBC interface.

Example:

SELECT * FROM MQL_QUERY(<    bookings   BY     user__country>);

Verification Steps

To verify this, you need a Mode analytics instance.

Documentation for setup with Mode can be found here .

Documentation for using our JDBC interface can be found here.

Product Component

General

More Detail

We are excited to announce that we are officially live with a documentation site at docs.transform.co which you can access from our product. Above the setting link on the left navigation, you'll find a new link called "Documentation".

Note some of our documentation is a work in progress. Please contact us if there is specific content you'd like to see!

Verification Steps

Use the new left nav "Documentation" link to access Transform's Documentation Site.

Product Component

Metrics Catalog

More Detail

We have now released Quarterly granularity as an option in our chart menu, which will aggregate values by the calendar quarter.

Verification Steps

Open up the granularity menu in Edit Chart and confirm that Quarterly is an option!

Product Component

Metrics Catalog

More Detail

Certain data sources do not have primary identifiers and so those dimensions cannot be joined onto other data sources. However, those dimensions can still be accessed locally to that data source. Previously, our UI hid these dimensions, but now we list these dimensions at the bottom of the menu under "Local Dimensions". You can optionally search "Local" to find all these dimensions

Verification Steps

For a metric that's based on a data source without a primary identifier and with dimensions, check that you can see all the local dimensions at the bottom.

Product Component

Metrics Framework

More Detail

We fixed a bug where cumulative metrics were returning incorrect results for some situations

The issue was a spurious groupby when Transform renders the time range join for cumulative metrics. Cumulative metrics are implemented by aggregating the metric's measure across the window specified in the metric. For example:

select  sum(bookings) as bookingsfrom bookings_table ajoin bookings_table bon b.ds <= a.ds and b.ds > date_sub(a.ds, 7, days)group by b.ds, bookings

In this case, the time range (b.ds) was non-varying, Transform would incorrectly group all the bookings values across those 7 days (removing duplicated values). Grouping by a.ds, bookings would work but would be a no-operative, so the fix removed the group by entirely.

Verification Steps

Create a cumulative metric such that the data being accumulated contains repeated values.

ie "weekly bookings"

metric:  name: weekly_bookings  description: weekly bookings  owners:    - bobby@transformdata.io  type: cumulative  type_params:    measures:      - bookings    window: 7 days

measure_bookings, ds1000 2020/01/012000 2020/01/021000 2020/01/033000 2020/01/04

Previously calculating weekly_bookings for 2020-01-05 would yield 6000 (omitting the repeated 1k value). It would now yield the correct value (7000)

Product Component

Metrics Catalog

More Detail

Previously, we would autoformat your dimension values by removing special characters and adding spaces. This could be confusing because what we displayed in the UI is different than the source database. For any net new Transform deployments, dimension values will not be formatted. Note: Dimension names will still be formatted.

If you would like, you can request to go back to the dimension value auto-formatting behavior. Please reach out to us at support@transformdata.io or over Slack.

Verification Steps

To verify, you can confirm your dimension values in the filter menu match the values in your database.