Skip to main content

Transform Product Components

Transform consists of 3 key components - the framework, the catalog and the API. The most important is the framework, which is where an organization will define the key metrics that will be viewed, explored, and queried in the catalog and in downstream BI tools.

Framework#

The objective of Transform's Metrics Framework is to define and manage the logic used to build a company's metrics. The Metrics Framework is an opinionated set of abstractions, created in YAML, that enable the data consumer to retrieve metric datasets efficiently from a data warehouse. There are a few key principles:

  1. Flexible, but complete - Define logic in abstractions that are flexible enough to construct any metric on any data model
  2. Don't Repeat Yourself (DRY) - Whenever possible enable the definition of metrics
  3. Simple with progressive complexity - Rely on known concepts and structures in data modeling to ensure that the Metrics Framework is approachable. Create escape hatches that enable folks to pursue more advanced and unsupported features.
  4. Performant and Efficient- Enable performance optimizations of centralized data engineering while enabling distributed definition and ownership of logic.

From a technical perspective, there are two critical objects to the metric definition in the Metrics Framework: the Data Source and Metric. Understanding these will allow you to configure metrics and dimensions in a way that will minimize redundancy while maximizing the potential of the Metrics Framework on top of your data.

Metrics are built on top of Data Sources and in many cases (but not always, see FAQ) are defined as a function of Measures. Data Sources together produce a data model that can then be compiled and consumed through Transform's various APIs which then constructs SQL on top of the various data sources to resolve metrics at various granularities.

Catalog#

The metrics catalog is the UI by which Transform surfaces the metrics defined in the Metrics Framework to an organization.

Technical and non-technical users should access the catalog in order to glean insights around metric governance, metric definition (in code and human-readable formats), and the ability to do some lightweight exploratory analysis on how a metric behaves. Users can also discuss metric changes by asking questions of metric owners, and can provide context around a metric's behavior by adding annotations directly to the metric.

A user might consume metrics by visiting a Collection, which organizes several important or related metrics together in a single place for easier analysis. Users can either create their own Collections, or view their Team's Page with associated Collections, which might focus around a specific business function within their organization.

Interfaces/API's#

The Metrics Framework actually exposes a GraphQL API with a number of clients built on top of that including:

  • Command Line Interface (CLI)* to pull data locally and improve the dev workflow
  • SQL over JDBC to integrate with BI Tools and other SQL Clients
  • Python Library to pull metrics into Jupyter or other analytical interfaces
  • Airflow Operator to schedule API requests and pre-construction of metrics
  • GraphQL interface which underlies all of these and is also exposed for the end-user to build their own interfaces
  • React components to build embadded analytics