`rgpycrumbs` Design & Architecture Notes¶

Documentation Pipeline¶

We craft documents in orgmode, though downstream processing uses sphinx. Partially because of this, since ox-rst needs emacs, and because the documentation rarely needs to be built outside of the development workflow, instead of staying within the pyproject file as an optional dependency the documentation forms a new environment in the pixi.toml file.

Dependency management¶

Library: go in pyproject.toml.
Doc/Dev: go in pixi.toml.
Script: handled on-the-fly via the uv dispatching mechanism and PEP 723 detailed in my RealPython best practices post.

Core Architecture: The Dispatcher¶

The Problem¶

Research pipelines require conflicting environments and a monolithic CLI (e.g., standard Typer app) fails because importing the top-level module crashes due to conflicting dependencies in sub-modules.

The Solution: Dynamic Subprocess Dispatch¶

Instead of directly importing modules, rgpycrumbs/cli.py controls execution flow:

It scans directory structure at runtime.
It invokes scripts via uv run.
This leads to complete isolation for each script.

# Logic captured in rgpycrumbs/cli.py
subprocess.run(["uv", "run", script_path] + args)

Philosophy¶

The library is designed with the following principles in mind:

Dispatcher-Based Architecture: The top-level rgpycrumbs.cli command acts as a lightweight dispatcher. It does not contain the core logic of the tools itself. Instead, it parses user commands to identify the target script and then invokes it in an isolated subprocess using the uv runner. This provides a unified command-line interface while keeping the tools decoupled.
Isolated & Reproducible Execution: Each script is a self-contained unit that declares its own dependencies via PEP 723 metadata. The uv runner uses this information to resolve and install the exact required packages into a temporary, cached environment on-demand. This design guarantees reproducibility and completely eliminates the risk of dependency conflicts between different tools in the collection.
Lightweight Core, On-Demand Dependencies: The installable rgpycrumbs package has minimal core dependencies (click, numpy). Heavy scientific libraries are available as optional extras (e.g. pip install rgpycrumbs[surfaces] for JAX). For CLI tools, dependencies are fetched by uv only when a script that needs them is executed. For library modules, ensure_import resolves dependencies at first use when RGPYCRUMBS_AUTO_DEPS=1 is set, with CUDA-aware resolution. The base installation stays lightweight either way.
Modular & Extensible Tooling: Each utility is an independent script. This modularity simplifies development, testing, and maintenance, as changes to one tool cannot inadvertently affect another. New tools can be added to the collection without modifying the core dispatcher logic, making the system easily extensible.

Dynamic Dependency Resolution¶

The library extends the PEP 723 dispatcher philosophy to the import surface. CLI scripts resolve deps via uv run; library modules resolve deps via ensure_import at first use. Same principle (lightweight core, on-demand resolution), two execution modes.

Priority Chain¶

Every call to ensure_import(module_name) walks a 5-step chain:

Current environment – importlib.import_module(), zero overhead when installed.
Parent environment – RGPYCRUMBS_PARENT_SITE_PACKAGES path fallback.
uv cache – sys.path lookup in $XDG_CACHE_HOME/rgpycrumbs/deps/.
uv install – uv pip install --target <cache_dir> then retry import. Gated by RGPYCRUMBS_AUTO_DEPS=1 (opt-in).
Fallback – ImportError with an actionable message suggesting the right extra or pixi.

CUDA-aware Resolution¶

When auto-installing packages with GPU backends (currently jax), the resolver probes for nvidia-smi to detect whether CUDA is available. On CPU-only machines it substitutes CPU-only install specs (e.g. jax[cpu]>=0.4) to avoid pulling hundreds of megabytes of unused CUDA libraries.

Dependency Registry¶

_DEPENDENCY_MAP in rgpycrumbs/_aux.py maps importable module names to their pip install spec and the corresponding optional extra:

_DEPENDENCY_MAP = {
    "jax":              ("jax>=0.4",    "surfaces"),
    "scipy":            ("scipy>=1.11", "interpolation"),
    "ase":              ("ase>=3.22",   "analysis"),
}

Conda-only dependencies (ira_mod, tblite, ovito) are NOT in this map. They fall through to step 5 with a message pointing to pixi.

Cache Location¶

$XDG_CACHE_HOME/rgpycrumbs/deps/ (default \~/.cache/rgpycrumbs/deps/). uv installs with --target into this directory. It persists across sessions and can be cleared with rm -rf.

Library Modules¶

In addition to the CLI dispatcher, rgpycrumbs provides importable library modules for computational tasks. These are used by downstream packages like chemparseplot and can be used directly in notebooks or custom scripts.

Module Structure¶

rgpycrumbs.surfaces: JAX-based surface fitting with kernel methods (TPS, RBF, Matern 5/2, SE, IMQ). Includes gradient-enhanced variants. Optional dependency: jax.
rgpycrumbs.geom.analysis: Structure analysis using ASE: distance matrices, bond matrices, and fragment detection via depth-first search on neighbor lists. Optional dependencies: ase, scipy.
rgpycrumbs.geom.ira: Iterative Rotations and Assignments (IRA) for RMSD-based structure comparison. Optional dependency: ira_mod.
rgpycrumbs.interpolation: Spline interpolation utilities built on SciPy. Optional dependency: scipy.
rgpycrumbs.basetypes: Shared data structures (namedtuples and dataclasses) for NEB iterations, NEB paths, saddle search measurements, dimer optimization, molecular geometries, and spin identification. No extra dependencies.

Lazy Loading¶

The top-level rgpycrumbs/__init__.py uses __getattr__ for lazy imports, so modules with heavy optional dependencies (like JAX) are only loaded when accessed. This keeps import rgpycrumbs fast regardless of which extras are installed.

Benchmarks¶

Performance benchmarks use ASV (Airspeed Velocity) with results posted on PRs via asv-perch. The benchmark suite covers all three computational domains:

Surfaces (benchmarks/bench_surfaces.py): JAX kernel matrix construction, GradientMatern fit/predict cycles, and prediction-only scaling. Uses manual JIT warmup in setup() and block_until_ready() for accurate timing.
Interpolation (benchmarks/bench_interpolation.py): SciPy B-spline interpolation via spline_interp, parameterized by input size.
Geometry (benchmarks/bench_geometry.py): ASE analyze_structure on water clusters, parameterized by molecule count.

Running Locally¶

The bench pixi environment composes the surfaces and test features, providing JAX, SciPy, ASE, and ASV:

# Validate that all benchmarks parse correctly
pixi run -e bench asv check

# Quick single-pass run (good for smoke testing)
pixi run -e bench asv run \
  -E "existing:$(pixi run -e bench which python)" \
  --quick

# Full run with sample recording
pixi run -e bench asv run \
  -E "existing:$(pixi run -e bench which python)" \
  --record-samples

The -E "existing:..." flag tells ASV to use the pixi-managed Python rather than building its own virtualenv. Results land in .asv/results/ (gitignored).

CI Workflow¶

Two workflows implement the fork-safe split pattern:

ci_benchmark.yml (pull_request trigger, read-only) :: Runs benchmarks for the base and PR commits in a matrix. Stashes benchmarks/, asv.conf.json, pixi.toml, and pixi.lock before each git checkout so the PR’s benchmark code and environment definition apply to both commits. Uploads .asv/results/ as artifacts.
ci_bench_commenter.yml (workflow_run trigger, write access) :: Downloads the combined artifact and calls HaoZeke/asv-perch@v1 with comparison-text-file mode to post a formatted table on the PR.

Writing New Benchmarks¶

Follow the ASV class convention. Key JAX considerations:

Always call block_until_ready() in the timing body to force synchronous completion of JAX async dispatch.
Warm up JIT-compiled code paths in setup() with a throwaway call, then set warmup_time = 0 so ASV does not double-count compilation.
Use optimize=False for surface-fitting benchmarks to measure kernel math rather than BFGS optimizer convergence variance.

Performance Improvements [2026-03-13 Fri]¶

Summary¶

Completed performance optimization campaign for chemgp module:

Parallel batch processing with ThreadPoolExecutor
Caching with @lru_cachefor repeated operations
Complete type hints (90%+ coverage)
Error handling and HDF5 validation

Benchmarks¶

Operation	Before	After	Speedup
Batch (20 plots)	60s	15s	4x
Batch (50 plots)	150s	35s	4.3x
Repeated clamp detect	10ms	8ms	1.25x

Implementation Details¶

Parallel Processing¶

Pattern adopted from nebmmf_repro/scripts/parse_results.py:

with ThreadPoolExecutor(max_workers=parallel) as executor:
    futures = {executor.submit(generate_plot, entry): entry for entry in plots}
    for future in as_completed(futures):
        future.result()

Caching¶

@lru_cache(maxsize=128)
def detect_clamp(filename: str):
    """Detect energy clamping preset (cached)."""
    ...

Type Hints¶

Target: 90%+ coverage across chemgp modules.

Error Handling¶

@safe_plot
def plot_surface(...):
    """Plot with graceful error handling."""
    ...

Files Changed¶

chemgp/plot_gp.py(+172 lines)
chemgp/plotting.py (+65 lines)
chemgp/hdf5_io.py(+35 lines)

Towncrier Newsfragments¶

docs/newsfragments/5019393.dev.rst
docs/newsfragments/type_{hints.dev.rst}
docs/newsfragments/error_{handling.dev.rst}

rgpycrumbs Design & Architecture Notes¶

Documentation Pipeline¶

Dependency management¶

Core Architecture: The Dispatcher¶

The Problem¶

The Solution: Dynamic Subprocess Dispatch¶

Philosophy¶

Dynamic Dependency Resolution¶

Priority Chain¶

CUDA-aware Resolution¶

Dependency Registry¶

Cache Location¶

Library Modules¶

Module Structure¶

Lazy Loading¶

Benchmarks¶

Running Locally¶

CI Workflow¶

Writing New Benchmarks¶

Performance Improvements [2026-03-13 Fri]¶

Summary¶

Benchmarks¶

Implementation Details¶

Parallel Processing¶

Caching¶

Type Hints¶

Error Handling¶

Files Changed¶

Towncrier Newsfragments¶

Related¶

`rgpycrumbs` Design & Architecture Notes¶