Product Component

Metrics Catalog

More Detail

We're excited to announce that annotations are now populated directly on the metric charts! If you create an annotation, it will show up on the graph and will be color-coded based on the annotation impact. You can click through the annotations and modals will pop up to give you details on them.

Verification Steps Confirm your existing annotations are on the graph and that any new annotations show up on the chart for the relevant time period

Product Component

Metrics Framework

More Details

Transform now supports cumulative metrics in our Metrics Framework. These metrics are useful for measuring values over a certain time window, typically used for WAU and DAU-style metrics (Weekly Active Users and Daily Active Users)

Cumulative metrics aggregate a measure over a given window. If no window is specified, the window is considered infinite and would aggregate all available data.

Verification Steps

Define a cumulative metric using the following syntax:

  # Define the measure and the window.  type: cumulative  type_params:    measures:      - distinct_users    # default window is infinity

Caveats

Cumulative metrics are calculated through a self-join with a date window in a similar manner as below

SELECT   COUNT(DISTINCT user_id) AS wau  , days.ds FROM (SELECT DISTINCT ds FROM user_daily) days JOIN user_dailyON user_daily.ds BETWEEN DATEADD('day', -7, days.ds) AND days.ds GROUP BY   days.ds  

If the data size is large, this query can be large and slow to execute. To speed up access, we recommend using materializations / fast cache to precompute these results in a daily pipeline.

Product Component

Metrics Framework

More Detail

Previously in the framework, we did not allow joins that would result in fan out. For example, joining a data source with a measure with a foreign key identifier along with a dimension data source with the same foreign key identifier, leading to possibly multiple results for each identifier. However, there are some use cases where this is useful, so the framework now allows these kinds of joins.

Verification Steps

Create a measure data source and a dimension data source with the same foreign identifier, and query for a metric using that measure and a dimension in the aforementioned data source.

Product Component

Metrics Catalog

More Detail

We've enhanced our metric search so that it includes a live typeahead. When you search, results will autocomplete. If you don't search, you can scroll through all the metrics in your organization.

Verification Steps

Confirm searching for metrics produces suggestions that match your search, and that you can scroll through all metrics.

Product Component

Metrics Catalog

More Detail

We updated the Metric Tier icons as well as the location. Tier icons are now gold, silver, and bronze to represent 1,2,3 respectively. Additionally, the tier icon on the metric page now exists at the top by the metric name.

Verification Steps

Confirm metric tier icons are now gold, silver, and bronze and that they exist at the top of the metric page.

Product Component

More Detail

We've updated the dimension filter/group menu in a few different ways:

1. The list exposes identifiers and dimensions together. Identifiers have a top heading (in grey), and the corresponding dimensions live underneath them
2. The Filter values are now in a different menu next to the dimension selection
3. The dimension menu has search! You can search on either identifiers or metrics.
4. You can scroll through the dimension menu to load additional dimensions

Verification Steps#

Click on Edit Chart and verify the dimension menu and group by menu is updated with the above changes.

Product Component

Metrics Catalog

More Details

We fixed a bug where the saved query date ranges was showing the 30 day window from when the query was saved rather than the last 30 days.

Verification Steps

Verify the save query range accurately reflects the chart presets.

Product Component

Interfaces - Python

More Detail

Our Python Interface is shipped with Transform MQL and allows users to express MQL requests in Python scripts and notebooks.

Verification Steps

Installation Instructions

Same as the CLI

PyPI link: https://pypi.org/project/transform/

pip install transform

Auth setup

# Follow the prompts to either enter api_key or perform MFAmql setup

Python Library

Before querying Transform using the Python Interface, you should authenticate using an API key by running the following:

mql setup -k <your api key>

NOTE: All configs including pinned models, api_key, bearer tokens will be stored under the default directory at ~/.transform/config.yml. If another location is desired, please set an ENV variable at $TFD_CONFIG_DIR with the desired path.

Examples of Querying Metrics

To pull this into a notebook or Python script, you can run the following, which shows an example of querying a metric called messages with dimensions ds, and a where constraint.

from transform import mql
df = mql.query(metrics=["messages"], dimensions=["ds"], where="is_thread")

You can optionally instantiate the object manually. This will allow you to pass an API key and authenticate, and you should use this mechanism if you are not already authenticated via mql setup.

from transform.mql import MQLClient
mql = MQLClient(api_key: Optional[str], mql_server_url: Optional[str])df = mql.query(metrics=["messages"], dimensions=["ds"], where="is_thread")

Python Interface Functions

The interface supports most commands that Tranform's MQL CLI supports.

Note if you instantiated the MQL object manually, you should use MQLClient.<function> instead of mql.function (as listed below).

mql.query(metrics, dimensions, model_key_id=None, where=None, time_constraint=None,        time_granularity=None, order=None, limit=None, cache_mode=None, as_table=None)
mql.create_query(metrics, dimensions, model_key_id=None, where=None, time_constraint=None,        time_granularity=None, order=None, limit=None, cache_mode=None, as_table=None)
mql.list_queries(active_only, limit=None)
mql.list_metrics(model_key_id=None)
mql.list_servers()
mql.create_materialization(materialization_name, start_time, end_time,        model_key_id=None, output_table=None)
mql.materialization(materialization_name, start_time, end_time,        model_key_id=None, output_table=None)
mql.drop_materialization(materialization_name, start_time, end_time,        model_key_id=None, output_table=None)
mql.commit_configs(config_dir)
mql.validate_configs(config_dir)
mql.health_report()
mql.drop_cache()
mql.identify()
mql.ping()

Product Component

Metrics Framework

More Detail

When defining a data source with a primary time dimension, it is now possible to use other granularities other than day, which include week, month quarter, and year). This will appropriately constrict the options possible when working with a metric (e.g. can't query at a DAY granularity if the metric is defined at a WEEK granularity), and also properly aggregate when querying metrics with different time granularities (e.g. querying two metrics, one with a DAY granularity and another with a MONTH granularity will return a result with a MONTH granularity by default). In addition to having appropriate metadata, the specified granularity will allow graphs in the UI to render correctly (e.g. a DAY granularity can't be specified for a MONTHLY metric).

Verification Steps

Create a data source with a primary time dimension at a granularity other than a day. Verify that the metric can't be queried using a finer time granularity, and also verify that when querying with a metric that is at a different time granularity, the results are returned with the smallest common time granularity.

Product Component

Metrics Catalog

More Detail

The y-axis on our charts previously started at a minimum value for the dataset instead of zero. This sometimes led to misleading visual trends because drops and increases look more drastic than they actually are. As a result, we updating both charts on the metric page and homepage to start with 0 by default on the y-axis. For the charts with negative y values, the axis will still start with the minimum value. At a later time, we'll add customizability options to y-axis bounds for cases where a limited range might be useful.

Verification Steps

Confirm all charts with all positive y values start at zero.

Confirm all charts with negative y values remains the same

Product Component

Interfaces - MQL CLI

More Detail

In MQL CLI, we have added two flags, --debug and --web for query, drop-materialization, and materialize commands.

--debug is used to show the mql server side log during the query execution. The log will be the same as what is printed when running mql stream_query_logs.

--web is used to open the MQL query log web page (https://app.transformdata.io/mql/query_logs) so that users can browse the log, see the result, etc, through web browsers.

Verification Steps

With the MQL Cli (version≥ v0.0.39), you can add the two flags in the three commands aforementioned, the log should be printed and a web page should be opened automatically.

Product Component

Metrics Framework

More Detail

We fixed a bug where when you had create_metric: true set for measures without having any metrics in your model, validation does fail.

Verification Steps

Confirm you can set create_metric: true without having metrics in your model.

Product Component

Metrics Catalog

More Detail

There has been a bug when creating or updating a Saved Query where the Saved Query form clears itself while the preview chart is loading and you are typing. This has been resolved.

Verification Steps

Navigate to a Metric Page. Filter the chart, click "More Actions", select "Save New Query". Fill in the Saved Query form and see it no longer clears while you are typing!

Product Component

Metrics Catalog

More Detail

We now show the Source SQL generated from the data warehouse to create the dataset (along with the generated SQL) on our query logs page in the Metrics Catalog.

Verification Steps

1. Navigate to the query logs through the settings page or any metric chart
2. Notice how the page shows two code snippets, one for Source SQL and one for the Generated SQL

Product Component

Metrics Framework

More Detail

Multi-hop joins allow you to retrieve dimensions from data sources that require more than one join.

This is currently supported in the framework & CLI level, and will also be supported in the Metrics Catalog

Example:

transaction table:

measures: transaction_count

identifiers: account_id

bridge table:

identifiers: account_id, customer_id

customer table:

identifiers: customer_id

dimensions: customer_id

The following query will join the bridge table to the transaction table, then the customer table to the bridge table. This allows us to slice the transaction count metric by customer_name even though it requires two joins to retrieve customer_name. Additionally, note the syntax; the customer must specify the identifiers to chart a path to the dimension (account_id, then customer_id)

mql query --metrics transaction_count --dimensions account_id__customer_id__customer_name

this yields a query like:

SELECT  ct.customer_name as account_id__customer_id__customer_name  , SUM(am.txn_count) as txn_countFROM transform_schema.bridge_table btJOIN transform_schema.customer_table ct  ON ct.customer_id = bt.customer_idJOIN transform_schema.account_month_txns am  ON bt.account_id = am.account_idGROUP BY  ct.customer_name

Verification Steps

Pick a dimension you'd like to join to your metric that requires more than one join to retrieve. Specify that dimension by joining the identifiers (in order) via dunders "__" and pass that dimension to a query like the above query. Validate that it slices the metrics by the requested dimension as you'd expect.

Product Component

Metrics Catalog

More Detail

We have a new 'Model' settings page that shows all data sources & metrics created in the organization and it allows you to dig into code snippets for each of them.

Verification Steps

Navigate to Settings —> Models (Under Organization)

Find a list of your metrics and data sources. Each entity will show a code snippet, which aligns with the line in the file. Click on the entity to open and collapse it.

Product Component

Interfaces - MQL CLI

More Detail

1. mql commit-configs now has the same default as validate-configs, for the current working directory (no need to specify the --config-dir)
2. Removed scientific notation when we format results for CLI output
3. We added an option to show all dimensions in mql list-metrics, even when the list is long - via the show-all-dims argument in mql list-metrics
4. Added an Explain (--explain) option for an MQL query, which show you the raw SQL generated by MQL to build the query
5. When the CLI times out waiting for a long query, we'll give you a helpful message alerting you of the timeout so it’s clear that the error resulted from a timeout
6. If a query fails for any reason, we'll do our best to pull out a useful error message to show you inline (rather than always requiring you to run stream-query-logs).

Verification Steps

1. Run mql commit-configs from the root directory of your Transform repo without the --config-dir flag.
2. Run query with result of greater than x(?) digits and ensure scientific notation is not returned
3. Run mql list-metrics —show-all-dims to confirm all associated dimensions are listed
4. Add --explain to your query and confirm you see the SQL output
5. Set the timeout parameter to an arbitrarily short limit using-timeout argument and confirm the query failure indicates it was due to a timeout
6. Try to create a query with a metric that doesn't exist:

mql query --metrics ghost_metric --dimensions ds

Before:

✗ mql query --metrics ghost_metric --dimensions ds --order ds              ✔ Query initialized: 1630370876675_0000116✖ Failure 🧐 - query failed after 4.05 seconds...💡 See MQL server logs for details

After:

✗ mql query --metrics ghost_metric --dimensions ds --order ds --timeout 100✔ Query initialized: 1630370876675_0000107✖ 'Unable to find metric `ghost_metric`. Perhaps it has not been registered?💡 See MQL server logs for more details:        mql stream-query-logs --query-id 1630370876675_0000107

Product Component

Metrics Catalog

More Detail

On the Metrics Page, Annotation dates were shown as MM/DD/YY but this led to some confusion. Now we're showing YYYY-MM-DD per the ISO standard.

Verification Steps

Create or view and existing annotation and confirm the format is updated.

Product Component

Metrics Framework

More Details

We added configuration validation for materializations, so our validation process when committing your configurations to Transform will check if you have the necessary fields and that they are valid. For example, the metrics and dimensions listed in the materialization configuration need to refer to ones defined in the model, and we will throw a validation error if they do not.

Verification Steps

Create materialization config and confirm it's validated by Transform in the Github actions.

Product Component

Metrics Catalog

More Details

Granularity now by default trims incomplete time periods so there are no false trends exhibited due to incomplete time ranges. This will also ensure that when time comparison is applied, it's only happening on complete ranges.

As an example, if you select weekly granularity and the latest day of data occurs in the middle of the week, we will trim the output to return only the latest complete week.

In a later release, we will also provide the option not to trim.

Verification Steps

1. Select a time range that includes an incomplete month or week
2. Select weekly or monthly granularity to confirm the time period is by default trimmed.

Product Component

Metrics Framework

More Detail

For a given foreign key we show null dimension values if the primary key does not exist. This functionality is visible when querying through the CLI and the UI. In the UI, it will show up as a NULL series when grouping by a particular dimension.

Example: In this case, the country_id of 3 is null and does not exist in the country table. Previously, when querying sales by country, we would have omitted the NULL value. It is now included in results.

---Table: fct_sales
sales  country_id1        11        11        21        3
---Table: dim_country
country_id  country1           us2           canada
---Query: sales by country
sales   country_id   country2       1            us1       2            ca1       3            NULL

Verification Steps

Query a metric in the CLI with a dimension that has a value for a foreign key identifier, but not a primary one.

Product Component

Metrics Catalog

More Detail

We now abbreviate our y-axis number display to show only 3 significant digits if there is no decimal and 5 significant digits if there is a decimal. We will abbreviate the number to include “K” for one thousand and “M” for a million,...etc.

Examples:

• 100
• 100%
• 100K
• 100K%
• 100.88
• 100.88%
• 100.88K
• 100.88K%

Verification Steps

Check your chart y-axis for metrics that have values greater than 3 significant digits. See below example where we've appended a k to each value to represent thousands.

Product Component

Metrics Catalog

More Detail

The Team Settings page only showed 5 team members, and we fixed a bug so that the full list of all team members is present.

Verification Steps

Visit the team settings page for a team with more than 5 members and confirm you can see all team members.

Product Component

Metrics Catalog

More Detail

We fixed a bug where the original timezone of the data point in the Metrics Catalog was erroneously overridden, causing certain metric values in the UI to be offset by a day. This only affected the front end, not the framework and CLI.

Verification Steps

Verify tooltip/data in the chart shows the correct date for a given value.

Product Component

Metrics Catalog

More Detail

We resolved an issue where sometimes when a single line was present on a chart for a grouping, the tooltip would not indicate what category/group was being displayed.

Verification Steps

Edit your metric chart to create a grouping that resolves to one line of values and confirm the tooltip indicates the grouping for the dimension.

Product Component

Metrics Framework

More Detail

Transform's framework now supports configuring data sources that contain composite keys. In database design, a composite key is a key that consists of two or more attributes (table columns) that together uniquely identify an entity occurrence or a table row, juxtaposed to a key that uses a single field.

Transform supports composite foreign and composite primary keys which can be defined in the identifiers section of the data source and used to do joins between your data sources like any other identifier.

Verification Steps

Imagine for the following example that a table has a composite key made up of two fields: user_id and message_id

To create

• Name the composite key using the same name field as when naming any other identifier. In this example's case, it's user_message
• ref may be specified instead to reference another identifier, in which case the name and expr are inherited from the referenced identifier
• Define the fields that make up your key ahead of defining the key itself.
• Specify the composite primary key via the identifier type primary .
• The fields that make up the key should be listed under identifiers.

data source: users    identifiers:        - name: message            expr: message_id      type: foreign    - name: user_message      type: primary      identifiers:        - ref: message        - name: user_id

Product Component

Metrics Catalog

More Detail

Metric owners will now get notified via email if a question is asked on a metric.

Verification Steps

1. Ask a question on a metric and confirm the owner gets notified via the email tied to their Transform account.
2. If the owner is on a team, everyone on the team should receive an email.

Product Component

Metrics Catalog

More Detail

1. Granularity is now available for ratio metrics as well as metrics based off measures that use an aggregation of count distinct.
2. When you choose a Weekly granularity in the catalog, it now defaults to your data warehouse's settings on the week start truncation. Previously, we defaulted to a Sunday start. To determine what your data warehouse settings are, you can check your data warehouse documentation, or run a function like DATE_TRUNC('week', <sample-date>) to confirm what the starting date is.
3. We no longer show a default daily granularity on the metric page. (Side Note: we may later support default granularity once we have more options supported at the model level - e.g. monthly)
4. If data is missing when granularity is selected, we fill in 0's only for metrics based on measures that use count sum aggregations. For other metric types such as an average, we will show missing values.

Verification Steps

1. Navigate to the user interface to a ratio metric or one that uses a measure of count distinct and confirm Granularity is present
2. Choose weekly granularity on any metric and confirm the week start range on the tooltip matches what your data warehouse returns for DATE_TRUNC('week', date).
3. Navigate to a metric with large chunks of missing data (a monthly metric is a good example), and confirm that 0's are not filled in for those missing dates.
4. Select granularity on data that's incomplete
   1. For a sum or count metric, confirm 0's are filled in for the missing time periods
   2. For all other measure aggregation types (e.g., avg), confirm data is not filled in with 0's and instead has gaps.

Product Component

Interfaces - MQL CLI

More Detail

mql list-metrics will by default list 5 dimensions associated with a metric and denote that there are x more depending on the remaining number.

We added a --show-all-dims argument in case you want to view the full list of dimensions associated with a metric.

Verification Steps

Run mql list-metrics --show-all-dims

Product Component

Interfaces - MQL CLI

More Detail

MQL now allows you to query metrics using the time dimensions grouped by specific granularities. For example, assuming ds is the name of your primary time dimension:

Typically, you might run something like:

mql query --metrics yourmetric --dimensions ds

You can now run something like the following to get your metric values grouped by a given granularity:

mql query --metrics yourmetric --dimensions ds__month

We support day, month, week, year, quarter.

Note you can also achieve this by using the --time-granularity argument; the gendered approach is a shortcut.

Verification Steps

Query your metrics by adding a double underscore and either day, month, week, year, quarter afterward mql query --metrics yourmetric --dimensions <your-primary-time-dimension-name>__<month|day|week|quarter|year

Product Component

Metrics Catalog

More Detail

We now surface ratio metrics as percents by default. We multiply the number by 100 to display and add a percent sign.

Verification Steps

Access a ratio metric and confirm the y-axis units is % and the values are percentages.

Product Component

Metrics Catalog

More Detail

We now provide the ability to delete questions on a metric.

Verification Steps

Hover over a question and notice a trash can appear on the right-hand side.

If you delete a question, all replies will automatically be deleted.

Product Component

Metrics Catalog

More Detail

Admin users can now edit any user's annotations.

Verification Steps

As an admin, go to an annotation that you didn't author and confirm when you scroll over it, the edit and delete button appears.

If you delete a question, all replies will automatically be deleted.

Product Component

Metrics Framework

More Detail

We introduced an aggregation of boolean into the Framework for a measure that could be expressed using an agg:boolean . We renamed this to sum_boolean since we believe this more accurately represents an aggregation of boolean columns in a data source.

Note that the current boolean type is still supported, but we do recommend that you change it to sum_boolean, as we may deprecate the original in the future.

Verification Steps

Create a measure with an aggregation type of sum_boolean.

Product Component

Metrics Catalog

More Detail

The number of values for a particular dimension could be very large and therefore expensive to surface, so I'd like to mension (😉) two improvements that we made that should help speed the requests up.

1. Caching dimension values in our results cache - Our results cache now saves these values after the first time you load them, so they'll be much quicker to load on the second attempt.
2. Materializing dimension values - If you list a dimension to be materialized with your metrics in Materializations, the materialization can be used to retrieve the dimension values quickly.

It is preferred that you materialize the dimensions that are most important to you which will also ensure that the initial load time of these dimensions in the UI is performant. However, if you aren't using materializations, you can still benefit from the speed from the results cache on the second run.

Verification Steps

Results cache

1. Choose a filter in the UI on any metric for the first time, load a set of dimension and corresponding dimension values and estimate the timing
2. Reload the page and confirm the load time is faster.

Materializations

1. Create a materializations config with metrics and dimensions
2. Filter by dimensions in the UI to ensure they are loading in a performant way.

Product Component

Metrics Catalog

More Detail

We fixed an issue where a user created in the Transform organization that had capital letters in the email was not showing up properly in the ownership section in the Metrics Catalog.

Verification Steps

Confirm that any users with capital letters in their email are properly be added as owners to metrics.

Product Component

Metrics Catalog

More Detail

The homepage will now show the last 30 or 90 days of available data for metrics. Specifically, for "Popular Metrics", we show the last 30 days available. For "Your most viewed metrics", we show 90. Previously, it showed the last 30/90 days respectively, so metrics that did not have a daily time series returned no data.

Verification Steps

View the homepage to confirm that all your metrics are populating regardless of the frequency of data.

Product Component

Metrics Catalog

More Detail

Annotations and Question text now support line breaks.

Verification Steps

Create annotations and questions that include spaces and ensure you have line breaks between them when you submit it.

Product Component

Metrics Framework

More Detail

We added the average aggregation type for measures to the framework.

Verification Steps

Specify agg as average to use this aggregation type for your measure.

measures:    - name: avg_revenue      description: Average Revenue      expr: revenue_value      agg: average

Product Component

Metrics Catalog

More Detail

We updated the tooltip on the chart to better portray the given granularity for both weekly and monthly granularities. For week, we now show a range. For month, we show the month and the year.

Verification Steps

1. Navigate to a metric that supports granularity and update granularity to weekly (as an example)
2. Hover over the chart to show the range

Product Component

Metrics Catalog

More Detail

Our collections page had a bug where if you attempted to add more than five metrics, it would not save the metric onto the collection.

Verification Steps

1. Create a collection
2. Add more than five metrics to a collection page and confirm it saves properly.

Product Component

Metrics Catalog

More Detail

Fixed a bug where ratio/percentage metric y-axes were showing integers, not decimals.

Verification Steps

Navigate to any ratio and percentage metric to confirm y-axes aren't showing integers.

Product Components

Metrics Catalog

More Detail

The link to View Code in when you are drilled into the metric lineage would not navigate to the right place, and we want to make sure you can view your code anytime from your metric page!

Verification Steps

This fix requires a new model commit so we can properly grab the path to the code. Once you've committed a new model, go to any metric lineage and click into View Code at the bottom of the panel. Confirm this takes you to the correct configs in Github.

Product Component

Metrics Framework

More Detail

We fixed a validation gap in our framework for time granularity. Currently, time_granularity in our framework only supports the option of day. We didn't throw an error if you input an unsupported option. We added the correct validation.

Verification Steps

Input an unsupported granularity and confirm that configurations do not validate.

Product Component

Metrics Catalog

More Detail

We're excited to announce three new charting features!

1. Date Range Selector: You can now toggle the date range of the chart (default 365 days) to presets or a custom range.
2. Granularity: You can now control the granularity of the X-axis through our new granularity menu. You can choose monthly, weekly, yearly in case you want to aggregate your data to different time levels. (Note that this granularity is currently independent of our framework granularity, which currently supports day)
3. Time Comparison: You can now calculate a Growth Rate between certain time periods. This uses a standard percent change calculation.

Verification Steps

Date Range

Navigate to Edit Chart on a metric page, find the "Date Range" menu, and TADA! (Also conveniently an anagram of data). Switch between Presets and Customs to find the desired range.

Granularity

Navigate to Edit Chart on the metric page and find granularity:

Change the granularity of the x-axis, which aggregates all values over a time period based on the aggregation of the measures.

Supported options: Daily, Weekly, Monthly, Yearly

Time Comparison

Ever wondered how your metrics compare to last week, last year, or last month? Us too! This is why we've introduced the ability to do growth rate calculations across various time ranges between metrics in a current and previous period.

Navigate to Edit Chart and find the Time Comparison menu. You can do Year over Year, Month over Month, Week over Week (assumes Sunday start), and Quarter over Quarter.

Also, the growth rate calculation is:

[((Values in Current Time Period) - (Values in Previous Time Period))/ Values in previous time period] * 100

Product Component

Metrics Framework

More Detail

You can now create a metric from measures from two data sources with different primary identifiers. This mostly applies to ratio metrics. Previously, to create a ratio type metric, both of your measures (numerator and denominator) had to come from the same data sources

Verification Steps

Create two data sources with a given measure and a metric based on them

Example: Measures Data Source 1

measures: - name: forecast_value    description: forecast value    expr: 1    agg: sum

Example: Measures Data Source 2

measures: - name: actual_value    description: forecast value    expr: 1    agg: sum

Metric

metric: - name: forecast_versus_actual  description: Forecast versus actual value  owners:    - forecast@transformdata.io  display_name: Forecast over Actual  type: ratio  type_params:    numerator: forecast_value        denominator: actual_value  tier: 2

Product Component

Metrics Framework

More Details

The materialization fast cache is a release candidate feature, and provides the ability to store your materialized data (from Materializations) in a DB-based cache (e.g., MySQL) for faster access. Materializations by default store data in your data warehouse, and the latency can be many seconds. This feature is most useful for organizations that need low latency access to metrics.

Verification Steps

Reach out to Transform to learn more about the materialization fast cache.

Product Component

Metrics Catalog

More Detail

The lineage dropdown for ratio metrics on the metric page would only show the final metric, not the derivations. We want to make sure you can see all the lineage regardless of metric type, so we added the data sources and measure that the ratio type metric is based on.

Verification Steps

Navigate to Lineage on the metric page for your ratio metric. Confirm that a ratio metric shows the measures & data sources that the metric is derived from. You may notice that the hierarchy is no longer a singular layer, rather multiple layers to show more than a single data source.

Product Component

Metrics Catalog

More Detail

Our annotation character limit was previously 500 characters. We don't want to limit your descriptions for your important annotations so we removed this limit.

Verification Steps

Create a 500+ character annotation.

Product Component

Metrics Catalog

More Detail

This was a transient bug that would sometimes lead to metric page links from various parts of our UI erroneously leading to a 404 error.

Verification Steps

This is difficult to verify given its unpredictability. However, please let us know if you see 404 errors when navigating to metrics that exist.

Product Component

Metrics Framework

More Detail

We're excited to announce that Materializations is a mechanism in Transform that enables the user to produce denormalized tables in a user's data warehouse. These are pre-computed and never evicted from the cache, and can be surfaced in downstream tools directly from the data warehouse.

Users can define a set of metrics and corresponding dimensions that are most commonly used and need to be accessed frequently by their organizations, which Transform will compute ahead of time at a scheduled cadence and write them to the data warehouse.

Verification Steps

We've reached out to most of you about deploying the materialization branch as it depends on a few changes in the framework around time granularity..

1. Configure a materialization: Here is an example of a materialization definition that reflects data from a booking service company.

materialization:   name: user_bookings_summary # name your materialization. this will be the table name written in your data warehouse.  description: #add an optional description.  metrics: # list all the metrics you want to materialize in your materialization. all metrics must be defined in your metrics.yaml.        - bookings        - booking_value        - customer_service_tickets    - guest_host_messages      dimensions: #list all corresponding dimensions you want to include for these metrics. these must all be defined in your data sources.    - ds    - guest__signup_channel
# Optional destinations sections # By default the location is the datawarehouse. # Optionally provide additional destinations for the materialization and rollup    additional_destinations:    - location: FAST_CACHE      format: WIDE    - location: FAST_CACHE      format: WIDE      rollups: [ds]     # provide the location, format, and dimensions for rollups # - locations: [DW, FAST_CACHE] # DW is the only supported option currently #   format: WIDE # WIDE is the only supported option currently #   dimension_sets: # rollups allow you to create separate tables that group by a subset of the dimensions in your materialization #   - [ds] #add a dimension you want to a rollup by. this should be a subset of the dimensions you list in your materialization. These rollups will be stored in separate tables than the core materializations

2. Calling a Materialization:

mql materialize --name my_materialization --start-time <date> --end-time <date>

**Usage**Options:  --materialization-name TEXT  Name of materialization to materialize                               [required]
  --start-time TEXT            iso8601 timestamp to materialize from  --end-time TEXT              iso8601 timestamp to materialize to  -t, --timeout INTEGER        Sets the timeout to wait for query completion. Not applicable in --detach mode.
  --detach BOOLEAN             Returns the created query ID to allow for                               asynchronous querying.
  --help                       Show this message and exit.

Product Component

Metrics Framework

More Detail

You can now add constraints to your measures in case you want to bound your metric result set to particular values. This configuration occurs when you define your metric.

This feature is available with the new branch we're rolling out to you to support.

Verification Steps

Example metric config with a constraint:

name: messages_on_days_when_there_were_a_lot_of_messagesdescription: Messages sent of any type, but only for days where message count exceeds the threshold (2000)owners:  - messages-team@thecompany.comdisplay_name: Messages Senttype: measure_proxytype_params:    measure: messagestier: "2"**constraint: |         where messages > 20**

Product Component

Metrics Catalog

More Detail

We now support markdown in our metrics definition editing module in the user interface - on a metric page.

Verification Steps

• Navigate to a metric page where you are a metric owner
• Choose Edit in the top right-hand corner in the configuration box
• Test your favorite markdown shortcuts!

Product Component

Metrics Catalog

More Detail

This description contains a few different small usability improvements in the UI:

1. We've updated the bottom of our left nav menu to distinguish the logout button a little more clearly.
2. We removed the "Metric:" introduction on our metric page because it's a little redundant.
3. We now sort the list of users and teams owners by the current owners in the Owner selection menu so it's clear who owns the metric while you're selecting others

Verification Steps

1. We've updated the bottom of our left nav menu to distinguish the logout button a little more clearly.
2. We removed the bolded "Metric:" introduction on our metric page because it's a little redundant
3. Navigate to a Metric and choose the "Add/Remove Owners" but and confirm that the owning teams and/or users are listed at the top. Previously, you had to search through them to identify who owned the metric

Product Components

Metrics Catalog

More Detail

We've added an API explorer into the user interface so you can explore GraphQL requests in the interface directly. This is useful for troubleshooting and exploring the capabilities of our APIs.

Verification Steps

1. Navigate to Settings —> MQL Server API explorer
2. Start writing GraphQL requests in the console

Product Component

Metric Catalog

More Detail

We've added a menu option under saved queries that allow you to delete them if you are the creator. In the case of wanting to remove your test queries or if you made a mistake, fear not!

Verification Steps

1. Open the Queries panel on the chart and navigate to a query you created
2. Open the three-dot menu on the right-hand top corner
3. Select 'delete query'
4. Confirm your selection

More Detail

In order to provide more details around how to build out data sources and metrics, we've added a verbose data source and metric template into our example configs in our template repository.

These files are commented out by default but include detailed instructions on what each field and area in the data source and metric YAML means.

This configuration template will be useful for everyone but especially new users of Transform who are just getting started, and the intent is that you can copy these and fill in your own information.

Verification Steps

When deploying Transform, there's a template repository at metrics-config-template/example/that includes a set of test data sources including rainfall and stations. In the same folder, we've added two files - one for data sources and one for metrics, that provide comments, instructions, and examples for how to configure these files.

The files are example_datasource.yaml and example_metric.yaml

Product Component

Metrics Framework

More Detail

We now support Amazon Athena as a source data warehouse to Transform, and among other things, we're excited to add support for our first database named after a Greek Goddess (of wisdom).

Verification Steps

Work with us to deploy Transform with your Athena instance.

Product Component

Metrics Catalog

More Detail

If you tried to access: https://app.transformdata.io/install without being logged in first, the page would hang indefinitely. We hope you didn't think that we didn't want you to install Transform - we really do! So, we fixed it and now lead you to a page with more information and login links.

Verification Steps

Visit https://app.transformdata.io/install before logging into the app and you should see a page that prompts you to log in as opposed to this one:

Product Component

Metrics Framework

We identified and fixed an issue where configuring a time dimension that wasn't primary would cause models to break. We fixed this issue and you can now specify both primary and non-primary time dimensions, as originally intended.

Verification Steps Add secondary time dimensions to your model using is_primary:false

Product Component

Metrics Framework

More Detail

In order to benefit from our upcoming Materializations feature as well as some of our new charting features (both slotted to ship shortly), we require some changes in your configurations in the framework around time dimensions. We will reach out to you about this and work closely with you to make sure this transition goes smoothly. See details below:

Benefits

• More flexibility in time components when querying MQL
• Materialization use - primary time dimension required for materializing sets of metrics
• Intelligently offer group by's and granularity options with the data
• Use front-end features that require granularity and format, like offset calculations (e.g., year over year)
• Paves path for future support of additional granularity and format options explicitly in the framework

Time Dimension Primary, Format, and Granularity

Today, you define your time dimension as follows with a name, type (and an optional expression)

dimensions:- name: ds    type: time

Now, you specify those components along with a few type parameters (under type_params).

dimensions:  - name: ds        type: time    type_params:      is_primary: True      time_format: YYYY-MM-DD            time_granularity: day

• The primary time dimension is mostly used for materializations and it's used as the common time dimension for all metrics you're materializing in a config.
• Granularity currently supports day but (in the future will support more and less granular options)
• Format currently supports YYYY-MM-DD (in the future we will support additional formats)
• NOTE: The name of your primary time dimension must be the same across all your data sources.

Product Component

Interfaces - JDBC

More Detail

We're starting our journey to build additional APIs beyond our command-line client that allows you to move data from Transform into your third-party tools of choice. To start, we've released a beta of our JDBC connector. I prefer tea, but I'm pretty energized by Java (heh).

The JDBC Connector allows you to express MQL requests directly in a SQL query for tools that support the common JDBC interface (examples below). With our connector, you can be confident that all the third-party tools that you're querying from are getting the same data from cleaned metrics that are defined and governed in Transform.

Verification Steps

Please reach out to Transform for the Beta and we will set you up with the connector as well as more documentation. See summarized details below on usage and documentation.

Usage

MQL parameters

SELECT ... 
FROM MQL(
        [metric,...]]
    [BY]
        [dimension,..]
    {WHERE expr}
    {ORDER BY {col_name | expr | position} [ASC | DESC], ...]}
    {LIMIT [rowcount]}
)...;

Examples

Consider the following metric, dimension, and measures list from a popular travel booking service. It contains a dimensional data source for listings (listing) as well as a fact data source (bookings) and values associated with them.

SELECT * FROM MQL_LIST_METRICS();

If we want to examine the bookings metric and query bookings by the dimension country, the API request in your tool would be the following.

SELECT * FROM MQL(bookings BY country);

Any supported SQL in your data warehouse is valid outside of the MQL request, so you can do more analysis outside of the request as well as join onto other tables in the data warehouse.

SELECT  AVG(booking_value) AS avg_booking_valueFROM MQL(        booking_value     BY         ds    WHERE         ds >= '2020-01-01')GROUP BY 1;

Product Component

Metrics Catalog

More Details We've revamped our annotation menu in a few ways. These features are very similar to the features that we added to our chart filtering.

Specifically:

1) We cleaned up dimension names so that underscores are not part of the name

2) Instead of showing all possible dimensions, which can get unwieldy dimensions, dimensions are organized by identifier.

2) Annotation forms now allow you to add dimension values affected as opposed to just dimensions themselves (which was the case previously) In many cases, an annotation might affect a particular value of the dimension, like a country, not the entire dimension set so we wanted to make sure you can annotate granularly.

Verification Steps

Create an annotation using the menu on the metric page and select metrics, dimensions, and values associated with those dimensions

Product Component

Metrics Catalog

More Detail

You can now edit and delete an annotation. Fear not! You will no longer be continuously taunted by a mistake or typo and not be able to delete or edit it.

Verification Steps

Hover over the annotation and see the new edit and delete buttons next to them on the right hand side.

Product Component

Metrics Catalog

More Detail

Our collections menu gave you the option to see you, your team's, and popular collections. We've also added a place for a user to browse all collections in case they are curious about what others are up to and how other folks might be looking at the data👀

Verification Steps

Click the Collections menu in the lefthand panel and click "View Collections"