Introducing fabric-utils

byStefan JohanssononNov 11, 2026read4 mininsoftware

• fabric
• pyspark
• delta-lake
• fabric-utils

Every Fabric workspace I have worked in grows the same thing: a utilities notebook that every other notebook starts with %run utilities. It begins as 50 lines of imports. A year later it is 1,200 lines, has three hash functions that disagree about NULLs, and nobody dares touch it because nothing tests it.

fabric-utils is what I reckon that notebook should have been: a Python package. It is built as a wheel, installed into a Fabric Environment item, and used from a notebook with two lines. Version 0.2.0 went out on 9 November 2026 and the code is on GitHub.

What it targets

Fabric Runtime 2.0 only: Apache Spark 4.1.1, Delta Lake 4.2, Python 3.13, Java 21. Runtime 1.3 is not supported and I am not going to add shims for it. One runtime means one set of Spark settings, one Delta feature set, and code that can use Spark 4 functions like try_to_timestamp without a fallback path.

Local development uses the same Python 3.13. PySpark and Delta are an optional extra, because the Fabric runtime already has them. The core dependencies are small: azure-identity, pandas, pyodbc, python-dateutil, requests, and sqlalchemy.

What is in it

inject() populates a notebook’s globals from a registry of named feature sets. Five are on by default, four are opt-in.

Feature	What you get	Default
common	F, T, W, DeltaTable, pd, ZoneInfo, datetime, and the rest of the usual imports	yes
metadata	MetadataConfig and DEFAULT_CONFIG, the single source of truth for column names and hashing	yes
print_helpers	formatted notebook output	yes
utils	dates, strings, validation, schema comparison, Delta probes	yes
contract	NotebookResultContract, COUNTS, and RunTimer for handing results back to a pipeline	yes
patterns	DataFrame transforms and canonical hashing	opt-in
lakehouse	lakehouse discovery and Delta maintenance	opt-in
direct_interface	DirectOrchestrator, a SQL client for the DIRECT orchestration database	opt-in
keyvault	Key Vault secret retrieval	opt-in

The two lines

from fabric_utils.autorun import init, inject

init(spark=spark)
manifest = inject(globals(), include={'patterns', 'lakehouse'})

init applies the central Spark settings once per session. inject binds the selected features into the notebook and returns a manifest of what it put there. Most workspaces keep those lines in one setup notebook and call %run setup.

Why a wheel

A %run helper notebook has no version, no tests, and no type checker. You find out it broke when a downstream notebook fails at 3am.

A wheel in an Environment fixes all three. Every notebook attached to the Environment gets the same version, and it is on sys.path before the first cell runs. There is no zip in OneLake and no addPyFile. The version comes from the git tag through setuptools-scm, so fabric_utils.__version__ tells you exactly what is running. And the package has about 430 unit tests, with the Spark ones running in CI against a real Spark 4.1 session.

The trade is that you cannot edit it in the portal. Fair enough. That is the point.

Rules the code lives by

A handful of rules shape every module. They are in the repo’s AGENTS.md, so the coding agents that help me follow them too.

• import fabric_utils never imports PySpark, Delta, pyodbc, or the Azure SDKs. Subpackages load lazily, so the Spark-free parts work in a Fabric Python notebook and in a plain unit test.
• There is one hashing implementation. Checksums, business keys, and surrogate keys all come from it.
• Anything that can fail returns a Result[T]. No alternative result types, no duck typing on success.
• No df.rdd, no _jdf, no sparkContext, no input_file_name(). The code has to work under Spark Connect.
• Delta 4.2 defaults are the baseline. The settings module lists only what changes behaviour relative to Runtime 2.0.

What is coming

One post a week, on Wednesdays. The bootstrap and the feature registry. The Spark settings and why the list is short. MetadataConfig. How the hashing canonicalises values so a checksum means something. The transforms. Result. The notebook result contract. Lakehouse maintenance with guard rails. The DIRECT client and its authentication. The rest of utils. Testing a PySpark library on a Windows laptop. And the release pipeline from git tag to Fabric Environment.

Have a crack at the reference notebooks in the meantime.