Guide#

This page presents the complete nd2py user guide in one place. Use the right-hand table of contents or your browser’s page search to jump directly to a concept.

Installation#

nd2py requires Python 3.12 or newer.

Install from PyPI#

python -m pip install nd2py

Install optional neural-network or search dependencies when needed:

python -m pip install "nd2py[nn]"
python -m pip install "nd2py[search]"
python -m pip install "nd2py[all]"

Install for development#

git clone https://github.com/yuzhTHU/nd2py.git
cd nd2py
python -m pip install -e ".[dev]"
pytest

The core symbolic engine and NumPy evaluator do not require the optional PyTorch stack.

Quick start#

Build and evaluate an expression#

Symbols compose through ordinary Python arithmetic:

import numpy as np
import nd2py as nd

x, y = nd.variables("x y")
expression = nd.sin(2 * x) + y**2

data = {
    "x": np.array([0.0, 0.5, 1.0]),
    "y": np.array([1.0, 2.0, 3.0]),
}

values = expression.eval(data)
print(expression)
print(values)
# sin(2 * x) + y ** 2
# [1.         4.84147098 9.90929743]

An expression is a tree. Inspect it with:

print(expression.to_tree())
# Add (scalar)
# ├ Sin (scalar)
# ┆ └ Mul (scalar)
# ┆   ├ 2 (scalar)
# ┆   └ x (scalar)
# └ Pow2 (scalar)
#   └ y (scalar)

Fit numerical constants#

Numbers created through arithmetic are fitable by default. BFGSFit optimizes them against observations while leaving the symbolic structure unchanged.

x = nd.Variable("x")
model_expression = 0.5 * x + 0.5

X = {"x": np.linspace(-2.0, 2.0, 100)}
target = 3.0 * X["x"] + 2.0

fit = nd.BFGSFit(model_expression, fold_constant=True)
fit.fit(X, target)

print(fit.expression)
print(fit.loss_)
# 3.0000000371975344 * x + 2.0000000376510387
# 3.2997463432895325e-15

The last digits can vary slightly with SciPy versions; the fitted coefficient and intercept should be close to 3 and 2, and the loss should be near zero.

Use nd2py.core.symbols.number.Number for a fitable number and nd2py.core.symbols.functions.Constant() for a fixed numerical constant.

Parse an expression#

expression = nd.parse("sin(2 * x) + y ** 2")

Parsing uses the same Symbol classes as programmatic construction, so parsed expressions support evaluation, printing, tree operations, and transforms.

Where to go next#

Read Expressions and symbols for the object model.
Read Network types for graph-aware expressions.
Read Parameter fitting for BFGS.
Read Custom operators outside nd2py to extend nd2py outside the repository.

Expressions and symbols#

Every nd2py expression is a tree of Symbol objects. Leaves provide data or parameters, while internal nodes describe operations.

Core symbol kinds#

Symbol	Meaning	Depends on input data?	Fitable?
`Variable`	Named value read from the data mapping	Yes	No
`Number`	Numerical constant or parameter	No	Optional
`GroupedParameter`	Parameter selected by a categorical variable	Yes	Yes
Operators	Functions such as addition, sine, and aggregation	Through operands	No internal state

import nd2py as nd

x = nd.Variable("x", nettype="scalar")
a = nd.Number(2.0, fitable=True)
c = nd.Constant(1.0)
expression = a * x + c

Python numbers used inside an expression are automatically wrapped as Number nodes:

expression = 2.0 * x + 1.0

Operands and ownership#

Each Symbol declares n_operands. A unary operator has one operand, a binary operator has two, and leaves have none. Symbol validates arity and links each child to its parent during construction.

nd2py maintains a tree rather than a general directed acyclic graph. A non-variable subexpression already owned by a parent is copied when inserted elsewhere. This keeps replacement, type inference, and traversal semantics unambiguous.

String forms#

print(expression.to_str())
# 2 * x + Constant(1)

print(expression.to_str(raw=True))
# Number(2.0, "scalar", True) * Variable("x", "scalar") + Number(1.0, "scalar", False)

print(expression.to_str(latex=True))
# 2 \times x + Constant(1)

print(expression.to_tree())
# Add (scalar)
# ├ Mul (scalar)
# ┆ ├ 2.0 (scalar)
# ┆ └ x (scalar)
# └ Constant(1.0) (scalar)

The raw representation is intended for round trips through nd2py.parse().

Public operations#

The methods exposed by a Symbol are a facade over specialized visitors. For example, eval() invokes NumpyCalc, to_str() invokes StringPrinter, and copy() invokes GetCopy. This separation lets the same expression tree have multiple interpretations without putting every implementation inside Symbol.

Evaluation and visitors#

nd2py evaluates and transforms expressions with visitors. A visitor dispatches on the exact runtime class name:

Add              -> visit_Add
Variable         -> visit_Variable
GroupedParameter -> visit_GroupedParameter

If a matching method is unavailable, the visitor uses generic_visit.

NumPy evaluation#

result = expression.eval(
    vars={"x": x_values},
    edge_list=(sources, targets),
    num_nodes=number_of_nodes,
    use_eps=1e-8,
)

NumpyCalc reads variables from vars and applies NumPy operations node by node. Sample dimensions are ordinary NumPy dimensions; nettype describes graph scope rather than array dtype or a general shape system.

Set return_exceptions=True to locate subexpressions that produce NaN or positive or negative infinity. The normal return value is unchanged by default; when diagnostics are enabled, evaluation returns (result, exceptions):

x = nd.Variable("x")
expression = nd.log(x) + nd.inv(x)

result, exceptions = expression.eval(
    {"x": np.array([-1.0, 0.0, 1.0])},
    return_exceptions=True,
)

print(result)
print(*exceptions, sep="\n")
# [nan nan  1.]
# Subexpression log(x) produced non-finite values: 33.33% NaN, 0.00% Inf, 33.33% NegInf
# Subexpression 1 / x produced non-finite values: 0.00% NaN, 33.33% Inf, 0.00% NegInf
# Subexpression log(x) + 1 / x produced non-finite values: 66.67% NaN, 0.00% Inf, 0.00% NegInf

Each diagnostic reports the offending subexpression and the fraction of its values that are NaN, Inf, or NegInf.

Effective Information Criterion#

expression.eval_eic(data) estimates the number of decimal digits effectively lost through numerical sensitivity in the expression tree. Larger values indicate less stable structure; zero means no estimated loss at the sampled inputs.

x = nd.Variable("x")
unstable = x - x

eic, exceptions = unstable.eval_eic(
    {"x": np.array([1.0, 2.0])},
    return_exceptions=True,
)

assert np.isinf(eic)
assert "x - x" in exceptions[0]

print(eic)
print(*exceptions, sep="\n")
# inf
# The subexpression x - x exhibits elevated local numerical sensitivity,
# corresponding to an estimated loss of inf decimal digits of precision.

Common elementwise operators use analytic derivatives. Other supported NumPy operators fall back to central finite differences controlled by perturbation. exception_threshold controls which local digit-loss values are reported; it does not change the returned EIC.

PyTorch evaluation#

result = expression.eval_torch(vars=data, device="cuda")

TorchCalc follows the same expression tree with PyTorch operations. Not every specialized Symbol necessarily supports both backends. In particular, GroupedParameter currently supports NumPy only.

Visitor protocol#

Visitor methods are generators. They yield a child together with the arguments used to visit it, then return the current node’s result:

def visit_Square(self, node, *args, **kwargs):
    x = yield (node.operands[0], args, kwargs)
    return x**2

The explicit stack in the base Visitor avoids Python recursion for deep trees. See Custom operators outside nd2py for a complete extension example.

Network types#

NetType describes where a value lives in a network-dynamics expression:

NetType = Literal["scalar", "node", "edge"]

It is not a NumPy shape or dtype.

Nettype	Interpretation	Typical last dimension
`scalar`	One value shared across the graph	`1` or broadcastable
`node`	One value for each node	number of nodes
`edge`	One value for each edge	number of edges

Inference#

NetTypeMixin stores two related pieces of state:

_assigned_nettypes: hard constraints supplied by the user;
_possible_nettypes: candidates remaining after inference.

infer_nettype() performs bottom-up and top-down propagation over the whole expression tree. Ordinary arithmetic follows the non-scalar operand: combining a node value with a scalar produces a node value, while directly combining node and edge values is invalid.

import nd2py as nd

x = nd.Variable("x", nettype="node")
c = nd.Number(2.0, nettype="scalar")
expression = c * x

assert expression.nettype == "node"

Graph-aware operators#

Sour(x) and Targ(x) gather node values onto source or target edges.
Aggr(x) and Rgga(x) aggregate edge values onto nodes.
Readout(x) reduces a node or edge value to scalar scope.

These operators implement custom map_nettype rules. A custom operator can override the same class method when the default arithmetic mapping is not appropriate.

Hard constraints and ambiguity#

Passing nettype= anchors a node. If inference cannot find a compatible combination of child types, nd2py raises a type-inference conflict rather than silently broadcasting node and edge quantities.

Expression trees#

Symbol inherits tree operations from TreeMixin. These operations work for built-in and custom Symbols because they depend on operands and parent, not on a fixed operator list.

Traversal#

import nd2py as nd

x = nd.Variable("x")
expression = nd.sin(2 * x + 1)

for node in expression.iter_preorder():
    print(type(node).__name__)
# Sin
# Add
# Mul
# Number
# Variable
# Number

for node in expression.iter_postorder():
    print(type(node).__name__)
# Number
# Variable
# Mul
# Number
# Add
# Sin

Preorder visits a node before its children. Postorder visits children before their parent and is useful for bottom-up analysis.

Paths and replacement#

subexpression = next(
    node for node in expression.iter_preorder()
    if isinstance(node, nd.Variable)
)
path = expression.path_to(subexpression)
same_node = expression.get_path(path)

print(path)
print(same_node)
# (0, 0, 1)
# x

replacement = nd.Variable("y")
expression = expression.replace(subexpression, replacement)
print(expression)
# sin(2 * y + 1)

Replacement must begin at a root expression. When the root itself is replaced, use the returned value because Python references to the old root cannot be updated automatically.

Copying#

copied = expression.copy()

Copies have independent parent links and parameter values. Stateful Symbol types may need a specialized GetCopy.visit_<Type> implementation; stateless operators whose constructor accepts only operands and nettype use the generic implementation.

Structural matching#

Variables in a pattern act as placeholders:

a = nd.Variable("a")
x = nd.Variable("x")

target = nd.sin(2 * x + 1)
bindings = target.match(nd.sin(a))

print(bindings)
# {'a': 2 * x + 1}

Repeated occurrences of the same pattern variable must bind to the same subexpression.

Parameter fitting#

BFGSFit optimizes fitable numerical parameters for a fixed symbolic structure.

import numpy as np
import nd2py as nd

x = nd.Variable("x")
expression = 0.5 * x + 0.5

X = {"x": np.linspace(-1.0, 1.0, 100)}
y = 3.0 * X["x"] + 2.0

fit = nd.BFGSFit(expression, fold_constant=True)
fit.fit(X, y)

print(fit.expression)
print(fit.loss_, fit.success_)
# 3.000000175954602 * x + 1.9999999661591596
# 1.1673694793736501e-14 True

Exact final digits can vary with SciPy versions. The meaningful checks are that the parameters approach 3 and 2, the loss is near zero, and fitting reports success.

Fittable and fixed constants#

Number(..., fitable=True) participates in optimization. The default depends on the set_fitable context. Constant(...) explicitly creates a fixed Number.

Constant folding#

With fold_constant=True, BFGS first combines fitable constant-only subtrees, then evaluates data-only subtrees once. The resulting loss expression is a temporary tree. After optimization, BFGS explicitly writes every fitted value back to fit.expression; it does not rely on the temporary and retained trees sharing node identity.

Parameter arrays#

Parameters are flattened before SciPy optimization and restored to their original shapes afterward. This supports scalar Numbers, array-valued Numbers, and GroupedParameters in one expression.

Estimator attributes#

After fit, inspect:

expression: retained expression containing fitted values;
loss_: final mean squared error;
success_: SciPy optimizer success flag;
message_: optimizer termination message;
n_iter_: number of optimizer iterations.

Grouped parameters#

GroupedParameter associates one fitable scalar with each distinct label of a categorical variable.

Suppose the same linear structure applies to several groups but each group has its own slope:

import numpy as np
import nd2py as nd

data = {
    "s": np.array(["group1", "group1", "group2", "group2", "group3", "group3"]),
    "x": np.array([1.0, 2.0, 3.0, 4.0, 5.0, 6.0]),
    "y": np.array([1.0, 2.0, 6.0, 8.0, 15.0, 18.0]),
}

s = nd.Variable("s")
x = nd.Variable("x")
slope = nd.GroupedParameter(s, default=1.0)
expression = slope * x

fit = nd.BFGSFit(expression)
fit.fit(data, data["y"])

print(fit.expression.operands[0].value_dict)
# {'group1': 1.0000001749568208, 'group2': 1.9999999888181819,
#  'group3': 2.99999999086196}

The final digits can vary with SciPy versions; the meaningful fitted values are approximately 1, 2, and 3.

Category binding#

Labels are bound in first-seen order. Previously unseen labels are appended to group_labels, added to label_to_index, and initialized with default. Evaluation therefore has defaultdict-like semantics:

parameter = nd.GroupedParameter(s, value={"known": 2.0}, default=0.5)
parameter.eval({"s": np.array(["known", "new"])})

assert parameter.value_dict == {"known": 2.0, "new": 0.5}

New categories encountered after fitting use default; they have not been optimized from training observations.

Representation#

One grouped parameter prints as alpha[s]. Multiple grouped parameters in one expression are numbered by preorder occurrence:

alpha^(1)[s] * x + alpha^(2)[s]

The LaTeX form is \alpha^{(1)}_{s}. Use grouped_parameter_symbol="theta" with to_str to change the displayed symbol.

Backend support#

GroupedParameter currently supports NumPy evaluation and BFGS fitting. Torch evaluation raises a deliberate NotImplementedError because string categories must first be encoded into integer tensor indices.

Custom operators outside nd2py#

You can define an operator and integrate it with nd2py visitors from an application, notebook, or separate package. No modification of the nd2py repository is required.

This page builds a unary Cube operator, adds NumPy evaluation and string rendering, and makes it available to the parser.

Define the Symbol#

import nd2py as nd

class Cube(nd.Symbol):
    """Compute the elementwise cube of one operand."""

    n_operands = 1

The inherited Symbol.map_nettype rule is sufficient because Cube preserves the network scope of its operand. Override map_nettype when an operator maps between scalar, node, and edge scope.

The generic tree, length, and nettype visitors already understand this class through n_operands. Backend-specific semantics require registration.

Add NumPy evaluation#

import numpy as np
from nd2py.core.calc.numpy_calc import NumpyCalc


def visit_Cube_numpy(self, node, *args, **kwargs):
    x = yield (node.operands[0], args, kwargs)
    return np.power(x, 3)


NumpyCalc.visit_Cube = visit_Cube_numpy

Visitor methods are generators. The yielded tuple asks the visitor to evaluate the child with the current arguments. The value sent back into x is the child’s NumPy result.

Add string rendering#

from nd2py.core.converter.string_printer import StringPrinter


def visit_Cube_string(self, node, *args, **kwargs):
    x = yield (node.operands[0], args, kwargs)
    if kwargs.get("raw"):
        return f"Cube({x})"
    if kwargs.get("latex"):
        return rf"\left({x}\right)^3"
    return f"cube({x})"


StringPrinter.visit_Cube = visit_Cube_string

Because dispatch uses the exact class name, the method must be named visit_Cube.

Protect existing registrations#

Direct monkey patching is process-global. A small registration helper prevents accidental replacement:

def register_visitor(visitor_class, symbol_class, visit_method):
    method_name = f"visit_{symbol_class.__name__}"
    if hasattr(visitor_class, method_name):
        raise ValueError(f"{visitor_class.__name__}.{method_name} already exists")
    setattr(visitor_class, method_name, visit_method)


register_visitor(NumpyCalc, Cube, visit_Cube_numpy)
register_visitor(StringPrinter, Cube, visit_Cube_string)

For a reusable extension package, perform registration in an explicit setup function rather than silently at import time.

Build, evaluate, and parse#

x = nd.Variable("x")
expression = Cube(x) + 1

values = expression.eval({"x": np.array([1.0, 2.0, 3.0])})
# array([2., 9., 28.])

parsed = nd.parse(
    "cube(x) + 1",
    callables={"cube": Cube},
)

Passing callables makes the custom operator available without adding it to nd2py.core.symbols.

Which visitors need integration?#

Capability	Usually needs a custom method?	Reason
Tree traversal	No	Uses `operands` generically
Tree printing	No	Generic structural output is sufficient
Copy	No for stateless operators	Reconstructs from operands and nettype
Nettype inference	No for scope-preserving arithmetic	Inherited mapping works
NumPy evaluation	Yes	Numerical semantics are operator-specific
EIC evaluation	Usually	Register an analytic derivative or NumPy semantics compatible with finite differences
Torch evaluation	Yes	Backend semantics are operator-specific
String rendering	Recommended	Generic output works but is less readable
Simplification	Only for special identities	Algebraic rules are operator-specific
Constant folding	Usually no	Generic evaluation works after a calculator is registered

Stateful Symbols need more care than the stateless Cube: copying and any transform that reconstructs the node must preserve their additional state.

Complete runnable example#

The following file is executed by the documentation test workflow:

"""Define and register an nd2py operator without modifying nd2py itself."""

import numpy as np
import nd2py as nd

from nd2py.core.calc.numpy_calc import NumpyCalc
from nd2py.core.converter.string_printer import StringPrinter


class Cube(nd.Symbol):
    """Compute the elementwise cube of one operand."""

    n_operands = 1

    @classmethod
    def register(cls):

        def register_visitor(visitor_class, symbol_class, visit_method):
            """Register a visit method while protecting existing registrations."""
            method_name = f"visit_{symbol_class.__name__}"
            if not hasattr(visitor_class, method_name):
                setattr(visitor_class, method_name, visit_method)

        def visit_Cube_string(self, node, *args, **kwargs):
            """Render Cube in readable and LaTeX expressions."""
            x = yield (node.operands[0], args, kwargs)
            if kwargs.get("raw"):
                return f"Cube({x})"
            if kwargs.get("latex"):
                return rf"\left({x}\right)^3"
            return f"cube({x})"

        register_visitor(StringPrinter, cls, visit_Cube_string)

        def visit_Cube_numpy(self, node, *args, **kwargs):
            """Evaluate Cube with NumPy."""
            x = yield (node.operands[0], args, kwargs)
            return np.power(x, 3)

        register_visitor(NumpyCalc, cls, visit_Cube_numpy)


Cube.register()

x = nd.Variable("x")
expression = Cube(x) + 1

# StringPrinter
print(expression)

# NumpyCalc
data = {"x": np.array([1.0, 2.0, 3.0])}
print(expression.eval(data))

# Parse
parsed = nd.parse("cube(x) + 1", callables={"cube": Cube})
print(parsed.eval(data))