Skip to content

Writing and Running a Record Generator

Abstract

trendify turns a folder of raw run directories into plots and tables by mapping a single Python function, a generator function, over each one. This guide covers how to write that function and how to run it through the trendify CLI.


Generator Function

A generator function takes a directory path as an argument and returns a RecordList:

Example Generator Function Handle
Python
import trendify


def generate_records(workdir: Path) -> trendify.RecordList:
    """
    Reads this run's `results.csv` and returns one example of every plottable record
    type, plus a table.
    """
    records: trendify.RecordList = []
    df = pl.read_csv(workdir / "results.csv")

trendify calls this function once per run directory, passing that directory's Path. The function processes raw data, and defines and returns a list of Record objects (Point2D, Trace2D, TableEntry, HistogramEntry, AxLine, and so on), each carrying one or more tags.

A tag is a string, an int, or a tuple of strings and ints, for example "histogram" or ("an_xy_plot", "trace_plot"). Records that share a tag get aggregated onto the same plot or table when the data is rendered.

Because of trendify's unique approach to building plots, the order you define a record has no bearing on the generators execution or the plot's appearnce.

Example

The example functions shown here are small record generators. In the example, it reads the results.csv that trendify example-data writes into each run directory (a time column plus one column per wave) and builds one example of every plottable record type.

Bash
trendify example-data --workdir ./data --n-folders 20

Scatter2D and Format2D

A Scatter2D plots a bulk collection of (x, y) points sharing one Marker style, unconnected by a line. Use it for a large array of points from one generation call that all share a style, such as a raw scatter of measurements from one run.

Format2D is itself a record: written once per tag, it configures that tag's axes (scale, limits, legend, grid, labels) for every other record sharing it:

Example
Python
def add_scatter_records(records: trendify.RecordList, df: pl.DataFrame) -> None:
    """
    Maps the relationship between two channels for this run.

    `Scatter2D` plots every raw `(wave_1, wave_2)` coordinate pair from this run as an
    unconnected point, sharing one marker style.
    """
    trendify.Format2D(
        tags=["signature_correlation"], label_x="wave_1", label_y="wave_2"
    ).append_to_list(records)

    trendify.Scatter2D(
        tags=["signature_correlation"],
        x=df["wave_1"].to_numpy(),  # compatible with numpy arrays
        y=df["wave_2"].to_numpy(),
        marker=trendify.Marker(label="wave_1 vs. wave_2", symbol="x", size=10),
    ).append_to_list(records)
Scatter2D Result
A rendered result of Scatter2D and Format2D.

Trace2D

A Trace2D draws an xy line and a Pen controls how it looks:

Example
Python
def add_trace_records(records: trendify.RecordList, df: pl.DataFrame) -> None:
    """
    Plots the raw time-series data: one `Trace2D` per channel.

    Because they all share the "time_series" tag, every channel overlays on a single
    unified plot for easy visual comparison.
    """
    # Define line styles
    pens = {
        "wave_1": trendify.Pen(
            label="wave_1",
            color=trendify.Color.ROSE_500,
            zorder=10,
        ),
        "wave_2": trendify.Pen(
            label="wave_2",
            color=trendify.Color.SKY_500,
            zorder=20,
        ),
        "wave_3": trendify.Pen(
            label="wave_3",
            color=trendify.Color.AMBER_500,
            zorder=30,
        ),
    }

    # Loop over columns and add lines
    for column in ["wave_1", "wave_2", "wave_3"]:
        trendify.Trace2D(
            tags=["time_series"],
            x=df["time"].to_numpy(),
            y=df[column].to_numpy(),
            pen=pens[column],
        ).append_to_list(records)

AxLine

An AxLine draws a horizontal or vertical reference line. Giving it the same tag as the traces above ("time_series") puts it on the same plot:

Example
Python
def add_axline_records(records: trendify.RecordList) -> None:
    """
    Draws a zero-amplitude static baseline directly across the "time_series" plot.

    This showcases how static annotations like `AxLine` seamlessly layer on top of
    dynamically generated traces sharing the same plot target tag.
    """
    # Define a Format2D with a grid
    trendify.Format2D(
        tags=["time_series"],
        label_x="Time (s)",
        label_y="Signal Amplitude",
        grid=trendify.Grid.from_theme(trendify.GridTheme.MATLAB),
    ).append_to_list(records)

    trendify.AxLine(
        tags=["time_series"],
        value=-1.0,
        orientation=trendify.LineOrientation.HORIZONTAL,
        pen=trendify.Pen(color="gray", size=3, linestyle="--", label="baseline"),
    ).append_to_list(records)
Trace2D and AxLine Result
A rendered result of Trace2D and AxLine.

Point2D

A Point2D is a single point, and unlike Scatter2D it's its own taggable, hoverable entity. This is the right choice when each point summarizes one run, aggregated across many runs that share a tag:

Example
Python
def add_summary_point_records(
    records: trendify.RecordList, df: pl.DataFrame, run_label: str
) -> None:
    """
    Aggregates one scalar summary per run.

    Unlike raw scatter data, each `Point2D` registers as an independent, hoverable entity.
    As multiple runs are processed, this plot populates a timeline of each run's `wave_1`
    mean across your entire batch history.
    """
    trendify.Format2D(
        tags=["run_means_tracking"], label_x="Run", label_y="wave_1 mean"
    ).append_to_list(records)

    trendify.Point2D(
        tags=["run_means_tracking"],
        x=run_label,
        y=df["wave_1"].to_numpy().mean(),
        marker=trendify.Marker(label="wave_1 run mean", size=20),
    ).append_to_list(records)
Point2D Result
A rendered result of Point2D.

Histogram Entry

A HistogramEntry bins one value into a histogram. Every entry sharing a tag (here, one per run) is binned into the same histogram:

Example
Python
def add_histogram_records(records: trendify.RecordList, df: pl.DataFrame) -> None:
    """
    Bins one scalar summary per run into a histogram.

    This extracts the mean value of `wave_1` from the current run and bins it into a
    population histogram shared across all runs, letting you monitor its distribution
    across the batch.
    """
    trendify.Format2D(
        tags=["harmonic_mean_distribution"], label_x="wave_1 mean", label_y="Count"
    ).append_to_list(records)

    trendify.HistogramEntry(
        tags=["harmonic_mean_distribution"],
        value=df["wave_1"].to_numpy().mean(),
        style=trendify.HistogramStyle(label="wave_1 batch mean", bins=10),
    ).append_to_list(records)
HistogramEntry Result
A rendered result of HistogramEntry.

Table Entry

A TableEntry is a single (row, col, value) cell, collected into melted, pivot, and stats tables. row must be distinct per run, or entries from different runs collide on the same (row, col) pair:

Example
Python
def add_table_entries_records(
    records: trendify.RecordList, df: pl.DataFrame, run_label: str
) -> None:
    """
    Tabulates multi-channel performance parameters into structured comparative matrices.

    One `TableEntry` is constructed per channel column. Assigning a unique, distinct
    `row` per execution context (the run directory name) ensures cross-run values map
    into explicit comparative rows instead of overwriting a single `(row, col)` storage slot.
    """
    for column in ["wave_1", "wave_2", "wave_3"]:
        trendify.TableEntry(
            tags=["batch_means_table"],
            row=run_label,
            col=column,
            value=df[column].to_numpy().mean(),
            unit="m/s",
        ).append_to_list(records)
TableEntry Result
Name min mean max sigma3
wave_1 0.0280749 0.0447426 0.0729484 0.0381242
wave_2 0.306726 0.636685 0.988092 0.594019
wave_3 0.339161 0.484203 0.694916 0.305491
row wave_1 wave_2 wave_3
0 0.033884 0.4814 0.433685
1 0.0516166 0.608159 0.652808
2 0.0381756 0.435484 0.42154
3 0.0319141 0.548191 0.501546
4 0.0451015 0.94053 0.408942
5 0.0487682 0.847561 0.505141
6 0.0405259 0.614581 0.589901
7 0.0411004 0.712014 0.446269
8 0.0620432 0.40625 0.544996
9 0.0280749 0.663112 0.687029
10 0.0371765 0.988092 0.381971
11 0.0292106 0.97166 0.400953
12 0.0729484 0.470315 0.392392
13 0.039272 0.841134 0.471227
14 0.0398635 0.66098 0.694916
15 0.066676 0.40861 0.339161
16 0.0542576 0.593144 0.487257
17 0.0304435 0.67114 0.427966
18 0.047172 0.306726 0.424944
19 0.0566284 0.564627 0.471412
row col value unit
0 wave_1 0.033884 m/s
0 wave_2 0.4814 m/s
0 wave_3 0.433685 m/s
1 wave_1 0.0516166 m/s
1 wave_2 0.608159 m/s
1 wave_3 0.652808 m/s
2 wave_1 0.0381756 m/s
2 wave_2 0.435484 m/s
2 wave_3 0.42154 m/s
3 wave_1 0.0319141 m/s
3 wave_2 0.548191 m/s
3 wave_3 0.501546 m/s
4 wave_1 0.0451015 m/s
4 wave_2 0.94053 m/s
4 wave_3 0.408942 m/s
5 wave_1 0.0487682 m/s
5 wave_2 0.847561 m/s
5 wave_3 0.505141 m/s
6 wave_1 0.0405259 m/s
6 wave_2 0.614581 m/s
6 wave_3 0.589901 m/s
7 wave_1 0.0411004 m/s
7 wave_2 0.712014 m/s
7 wave_3 0.446269 m/s
8 wave_1 0.0620432 m/s
8 wave_2 0.40625 m/s
8 wave_3 0.544996 m/s
9 wave_1 0.0280749 m/s
9 wave_2 0.663112 m/s
9 wave_3 0.687029 m/s
10 wave_1 0.0371765 m/s
10 wave_2 0.988092 m/s
10 wave_3 0.381971 m/s
11 wave_1 0.0292106 m/s
11 wave_2 0.97166 m/s
11 wave_3 0.400953 m/s
12 wave_1 0.0729484 m/s
12 wave_2 0.470315 m/s
12 wave_3 0.392392 m/s
13 wave_1 0.039272 m/s
13 wave_2 0.841134 m/s
13 wave_3 0.471227 m/s
14 wave_1 0.0398635 m/s
14 wave_2 0.66098 m/s
14 wave_3 0.694916 m/s
15 wave_1 0.066676 m/s
15 wave_2 0.40861 m/s
15 wave_3 0.339161 m/s
16 wave_1 0.0542576 m/s
16 wave_2 0.593144 m/s
16 wave_3 0.487257 m/s
17 wave_1 0.0304435 m/s
17 wave_2 0.67114 m/s
17 wave_3 0.427966 m/s
18 wave_1 0.047172 m/s
18 wave_2 0.306726 m/s
18 wave_3 0.424944 m/s
19 wave_1 0.0566284 m/s
19 wave_2 0.564627 m/s
19 wave_3 0.471412 m/s

Putting it Together

The generator function itself just calls each of the helpers above and returns the combined list:

Python
import trendify


def generate_records(workdir: Path) -> trendify.RecordList:
    """
    Reads this run's `results.csv` and returns one example of every plottable record
    type, plus a table.
    """
    records: trendify.RecordList = []
    df = pl.read_csv(workdir / "results.csv")
    run_label = workdir.name

    add_scatter_records(records, df)
    add_trace_records(records, df)
    add_axline_records(records)
    add_summary_point_records(records, df, run_label)
    add_histogram_records(records, df)
    add_table_entries_records(records, df, run_label)

    for record in records:
        # render all as SVG instead of JPG
        if isinstance(record, trendify.Format2D):
            record.renderer = trendify.Vector()
    return records

A generator you write for real data follows the same shape: read the run directory's contents (CSV, JSON, or whatever format your raw data uses), build up a RecordList, tag each record according to how you want it grouped, and return the list.


Running the Generator

The trendify CLI accepts a generator in the form module:function, package.submodule.module:Class.method (@classmethod or @staticmethod), or /path/to/file.py:function.

Generating Records

trendify generate maps the generator over a set of input directories and writes the resulting records into a trendify.db under the given output directory:

Bash
trendify generate \
    --input-directories "./data/models/*" \
    --record-generator "docs/example_generator.py:generate_records" \
    --output-directory "./data/trendify"

--input-directories accepts glob patterns and can be repeated to combine several. Use --n-procs to fan the generator call out across multiple worker processes.

But Why?

This can be helpful if you are still debugging your generator, or you only care about viewing interactive assets (see the later section on Trendify Viewer)

Rendering Assets

trendify render reads the records already written to trendify.db and produces CSV tables and Matplotlib images, one set of assets per tag, in <output-directory>/assets:

Bash
trendify render --output-directory ./data/trendify

Running In One Step

trendify run combines the two steps above:

Bash
trendify run \
    --input-directories "./data/models/*" \
    --record-generator docs/example_generator.py:generate_records \
    --output-directory ./data/trendify

Trendify Viewer

trendify viewer launches a local web dashboard for browsing a trendify.db file's records interactively, by tag, table, or plot:

Bash
trendify viewer ./data/trendify/trendify.db

Embedding the Viewer in Your Own Site

The viewer is an ordinary FastAPI app, so an existing FastAPI site can serve it and show it in an <iframe>:

Python
from fastapi import FastAPI
from trendify.viewer.app import create_app

site = FastAPI()
site.mount("/viz", create_app("data/trendify/trendify.db", show_db_path=False))
HTML
<iframe src="/viz/" title="Trendify viewer" allow="fullscreen; clipboard-write"
        style="width: 100%; height: 80vh; border: 0"></iframe>
  • Where it lives: a path prefix (as above) or its own hostname (Host("viewer.example.com", app=...)) both work, and your site's startup needs no changes. show_db_path=False keeps the server's file path out of the page.
  • Access control: the viewer has no login of its own, so put it behind your site's. A small ASGI wrapper around the mounted app protects its pages and data in one place.
  • CORS: not needed for an iframe, because the viewer only talks to its own address. Only if scripts on your page fetch from the viewer's API, pass cors_origins=["https://your-site.example"].
  • Security headers: if your site sets X-Frame-Options or frame-ancestors, allow framing on the viewer's path. If it sets a Content Security Policy, the viewer's pages need one that allows inline scripts and styles and unsafe-eval, for example script-src 'self' 'unsafe-inline' 'unsafe-eval'; style-src 'self' 'unsafe-inline'; img-src 'self' data: blob:.
  • Other sites: browsers that block third-party storage can't save the viewer's theme and layout when it is framed from a different site. It still works; those settings just reset on reload.

Success

You can now write a RecordGenerator and run it end to end:

  • generate to write records
  • render for static CSV tables and plots
  • run to do both in one step
  • viewer to browse the results in a live dashboard.

Point --record-generator at your own function and --input-directories at your real raw data to automate record creation in your project.