02 — Type & Operator System¶

This is the technical heart of the product: which Python/Pydantic types we support, which operators each exposes, how compound types behave, and how a user controls all of it.

Mental model¶

Two registries drive everything:

1. Type → operator profile. Each supported type has a default profile: the set of operators it exposes out of the box. Profiles are tiered (safe, full) so we can keep dangerous operators off by default.
2. Operator → semantics. Each operator defines its value arity (single / list / range), the value's relationship to the field type (same type / bool / int), and how each backend adapter compiles it.

Resolving a model field is then: field type → profile → operator set, with per-field overrides layered on top.

Scalar types¶

Notes:

• regex is full-only and additionally gated by a config flag, because it is a ReDoS and full-scan risk. See Safety below.
• contains/startswith/endswith values are always re.escape()d before being compiled to Mongo $regex (anchored for the *with variants). The user value is a literal substring, never a pattern — without this guarantee, contains would silently be the regex operator, ReDoS included. Pattern matching is exclusively the job of the explicit, gated regex operator.
• text_search (full; requires the backend capability) compiles to a real text query — Mongo $text over a text index, match in Elasticsearch — instead of a scanning regex. Adapters that lack it reject it at registration.
• Case-insensitive variants (i*) are separate operators rather than a flag, so they appear explicitly in the docs and compile to explicit backend forms.
• eq is implicit: a bare field= is treated as field__eq=. This keeps the 90% case (?status=active) clean.

Optionals and nullability¶

Optional[T] / T | None exposes everything T does plus:

• isnull → field__isnull=true|false (compiles to {field: None} / {field: {$ne: None}}, or IS NULL in SQL).
• exists (Mongo-flavored; full only) → {field: {$exists: bool}}. For SQL adapters exists aliases to isnull semantics or is rejected — adapters declare which operators they support (doc 03/04).

Compound types¶

This is where the design earns its keep. The user explicitly called out list[str]; here is the full treatment.

list[T] / set[T] (arrays of scalars)¶

An array field needs operators about membership and shape, which are distinct from scalar operators. We expose a curated set, tiered like everything else:

len__eq is safe because it compiles to a plain $size match, which can use an index. The other five len comparisons compile to $expr — no index support, same collection-scan reasoning that gates other full operators — so they are full-tier.

Query forms:

?tags__has=python
?tags__has_any=python,rust            # any-of
?tags__has_all=python,rust            # all-of
?tags__len__eq=2
?tags__len__gte=2                     # full tier
?tags__empty=false

len range compilation. Mongo's aggregation $size errors the whole query when the field is missing, null, or not an array. So the len__ne/ gt/gte/lt/lte operators compile to a guarded $expr:

{"$expr": {"$gte": [{"$size": {"$cond": [{"$isArray": "$tags"}, "$tags", []]}}, 2]}}

The $cond/$isArray guard falls back to [] for anything that isn't an array, so a missing/null/non-array value counts as length 0 instead of failing the query.

Precise empty semantics (empty-vs-missing is a classic Mongo trap, so we pin it down):

• ?tags__empty=true → the field exists and is the empty array: {tags: {"$eq": []}} (equivalent to $size: 0, but index-friendlier).
• ?tags__empty=false → the field exists and has at least one element: {"tags.0": {"$exists": true}}.
• A missing field matches neither. Use isnull/exists to reason about presence; empty reasons only about shape. This distinction is documented on the operator itself.

Design choice: we do not silently apply scalar string operators (contains) to list[str] — tags__contains would be ambiguous (substring of an element? membership?). Array fields get array operators. Element-level substring matching would be an explicit, named, full-tier operator (tags__has_substr) so the intent is unmistakable — designed here, not shipped in Phase 3a; the shipped array profile is exactly the table above.

By default, array fields are not sortable, even when filterable — Mongo sorts arrays by their min/max element, which is rarely what anyone means by "sort by tags." Opt in deliberately with Filterable(sortable=True) on the field or by naming it in FilterConfig(sortable=[...]).

Nested Pydantic models (embedded documents)¶

class Address(BaseModel):
    city: str
    zip: str

class User(BaseModel):
    address: Address

Nested fields are reachable by dotted path, which maps cleanly to Mongo's dot notation:

?address__city__contains=ams      # field path = address.city, op = contains

The embedding field itself (address, as opposed to its leaves) exposes no operators of its own — filtering happens on the leaves — except the nullability pair when it is Optional: address__isnull/address__exists. Other Filterable options placed on the embedding field are scoped to that field, not the subtree, with one exception:

• Filterable(ops=ops.NONE) on address excludes the entire subtree (the embedding field and every descendant leaf) from the filter surface.
• Filterable(source=...) / Filterable(param=...) on address rename that one path segment for the whole subtree — every descendant path is composed from the renamed segment.
• Any other Filterable option on address (e.g. sortable=True) affects only the embedding field's own spec, never its descendants.
• An explicit Filterable(ops=[...]) list on a non-nullable embedding field is a ConfigurationError at registration — there is no operator it could validly name (not even isnull, since the field can't be null).

Parameter matching (precise — there is no request-time parsing)¶

Because the entire parameter surface is generated from the model at registration time, the library never splits incoming parameter names. Each generated parameter carries its own (field_path, operator) pair; an incoming name is matched exactly against the generated set. address__city__contains works not because a parser split it correctly, but because registration emitted a parameter with that exact name bound to (("address", "city"), contains).

This makes otherwise-nasty cases non-issues by construction:

• Multi-token operators (len__gte, elem) — the full spelling is just part of the generated name. Their internal __ is hardcoded, independent of FilterConfig.separator: under separator="--", the generated name is tags--len__gte (segment separator --, operator spelling untouched). Harmless — the name is still generated exactly, never parsed — but pinned so it isn't "fixed" into an inconsistency later.
• A field literally named with __ in it, or a nested field that shares a name with an operator (address.in) — the generated name is whatever it is; collisions between two generated names are detected at registration and raised as a config error naming both sources.
• Unknown incoming parameters are never mis-parsed — they simply don't match, and are handled per the ignore/strict setting (doc 01).

Generation notes:

• Bare address__city (no operator suffix) is emitted as the implicit-eq parameter for address.city.
• Recursion depth is bounded (default 2 levels, FilterConfig(max_depth=N)) and configurable, to keep the generated parameter surface finite and the docs readable. The precise, shipped semantics: max_depth counts the embedded-model boundaries a field's path crosses — address__city is 1, address__geo__lat is 2. A field whose path crosses more boundaries than the bound allows is silently skipped. A nested model sitting exactly at the bound still gets its own spec (so a nullable embedding keeps its isnull/exists) but none of its children. max_depth=0 disables nested-model traversal entirely — only top-level scalar/array fields are generated.
• Cycles (self-referential and mutually-recursive models) terminate by construction: recursion depth strictly increases on every descent, so the depth bound above is what truncates them — no separate cycle detection is needed.

list[NestedModel] (arrays of embedded documents)¶

class User(BaseModel):
    orders: list[Order]

These need element-match semantics ("a user with an order over $100 that is also refunded"). Mongo expresses this with $elemMatch. We expose it explicitly:

?orders__elem__amount__gte=100&orders__elem__status=refunded

The elem token groups conditions that must hold for the same array element, compiling to a single $elemMatch. Without elem, conditions on different parameters are independent (Mongo's default array-matching semantics) — we document this difference loudly because it surprises people.

elem parameters are full-tier (shipped that way given the subtlety): under the default safe profile they generate nothing, and there are exactly three opt-in paths — FilterConfig(default_profile="full"), a Filterable(ops=...) on the element field, or a FilterConfig.operators entry naming the elem path. text_search never applies inside elements. Shape operators (len__*, empty) remain available on the array field itself and merge with $elemMatch on the same field. Elem paths are never sortable. The elem hop counts as one max_depth boundary, like an embedding.

Adapter contract: in the AST, element-relative fields carry a literal $elem source segment (orders.$elem.amount — $-prefixed, so it cannot collide with a real Mongo field name; exported as fast_pager.ast.ELEM_SOURCE_MARKER). Backends group all conditions sharing the prefix before .$elem. into one element-match construct with keys relative to the element. The conformance suite (doc 04) exercises this grouping rule.

dict[str, T] / free-form maps¶

Free-form maps clash with a core promise: every parameter is pre-generated, typed, and documented in OpenAPI — which is impossible when the key set is unknown. So support is deliberately narrow:

• ?metadata__has_key=region — key presence. This is generatable (one parameter, value typed str) and always available when the field is enabled.
• Value-at-key filtering is available only for keys enumerated in config:

metadata: Annotated[dict[str, str], Filterable(keys=["region", "tier"])]

generates metadata__region, metadata__tier (typed as T, implicit eq only in this release), and nothing else. No enumeration → no value filtering; we do not accept arbitrary metadata__<anything> at request time, because those params would be undocumented and untyped. Richer per-key operator curation is deferred to FilterSet.

Because a map key becomes part of a Mongo field path, keys are sanitized: empty keys and keys containing ., $, or NUL are rejected — at Filterable(keys=...) construction time (ConfigurationError), at request time for has_key values (standard 422), and again in the compiler (CompilationError, defense in depth for direct AST users).

Default: not filterable unless explicitly enabled.

Union[A, B] (non-optional unions)¶

Discouraged for filtering — the operator set is ambiguous. Default: not filterable; the library logs a one-time warning naming the field and how to make it explicit (annotate it, or pick a concrete type via FilterSet).

Field → DB-name mapping and aliases¶

• Pydantic aliases are respected: an alias (e.g. alias="userName") defaults both names at once — the query parameter is named after the alias and the compiled query targets the alias too. Filterable(param=...) and Filterable(source=...) each override one side independently, so the resolution order is: public name = param → alias → field name; source name = source → alias → field name.
• Nested paths compose segment by segment. Each level of a nested model resolves its own public/source name independently by the rule above, and the two paths are joined separately: the public path with separator (address__city), the source path with . (address.city). Renaming one segment (e.g. Filterable(source="addr") on the embedding field) only changes that segment; descendants still compose normally on top of it (addr.city).
• FilterConfig.exclude is path-based, not string-based. An entry matches a field's full public path prefix by segment (exclude=["address"] removes address and its whole subtree), never by literal string prefixing — so a top-level field that happens to be named "address__city" is unaffected by an exclude=["address"] entry naming the nested subtree. The same rule applies to FilterConfig.operators and FilterConfig.sortable: all three are keyed by the field's public, separator-joined dotted-param spelling ("address__city"), the same spelling a client would use in the URL.
• Explicit source override for when the Mongo field differs from the model:

age: Annotated[int, Filterable(source="ageYears")]

?age__gte=21 → {"ageYears": {"$gte": 21}}.

• Custom parameter name (decouple URL from field):

age: Annotated[int, Filterable(param="minimum_age", ops=["gte"])]

Per-field configurability — the core question you asked¶

Yes — it must be configurable which operators are exposed per field, and the design provides four layers, from coarse to fine, each overriding the previous:

1. Global default profile. FilterConfig(default_profile="safe") — applies to every field by type. This is the zero-config behavior.

2. Per-type override. "All strings in this app expose icontains but never regex."

FilterConfig(type_profiles={str: ["eq", "contains", "icontains"]})

1. Per-field inline (Annotated). Wins over type-level.

email: Annotated[str, Filterable(ops=["eq"])]           # exact-match only
bio:   Annotated[str, Filterable(ops=ops.NONE)]         # explicitly unfilterable
score: Annotated[int, Filterable(ops=ops.ALL)]          # everything int supports

1. Per-field on the route. Wins over everything, including the field's own Filterable(ops=...) — the route has the final say over its own surface. The shipped spelling is the route-level mapping, keyed by public parameter name:

FilterConfig(operators={"name": ["startswith"], "age": ["gte", "lte"]})

A FilterSet's fields mapping (Stage 3) occupies this same layer, and is also where custom/computed filters live:

class UserFilter(FilterSet):
    class Meta:
        model = User
        fields = {"name": ["contains"], "age": ["gte", "lte"]}
    # custom filter not derivable from a single field:
    active_since = DateFilter(field="last_login", op="gte")

Two rules pin down how the ladder interacts with the safety gates (settled by the implementation, now normative):

• ops.NONE and sortable=False are absolute. They sit outside the ladder: a config that tries to override them — naming an ops.NONE field in FilterConfig.operators, or a sortable=False field in FilterConfig.sortable — raises ConfigurationError at registration. Route-level config can never quietly resurrect a field the model opted out; a model-level opt-out is a security posture, not a default.
• Explicit operator lists bypass the allow_regex gate. Listing "regex" in FilterConfig.operators, in Filterable(ops=[...]), or in a type_profiles entry is the eyes-open opt-in and needs no extra flag. The tier profiles remain gated: ops.ALL and the safe/full default profile emit regex only with FilterConfig(allow_regex=True).

Allow-list vs deny-list semantics¶

• In Option C (zero-config) the model is an implicit allow-list by type: filterable types in, sensitive types (bytes, dict, bare Union) out by default, plus a global exclude=[...] for named fields.
• In FilterSet the fields mapping is a strict allow-list: if it's not listed, it's not filterable. This is the safe default for public APIs — you opt fields in, never accidentally leak a new field by adding it to the model.

This allow-list-by-default-in-FilterSet property is a deliberate security posture, not an accident.

Operator value parsing¶

• Single-value ops (eq, gte, contains): the value is coerced to the field's type by Pydantic (so age__gte=21 yields an int, and age__gte=xx yields a clean 422).
• List-value ops (in, nin, has_any, has_all): accept both
• repeated keys: ?status__in=a&status__in=b, and
• comma-joined: ?status__in=a,b

Each element is coerced to the field type. A max_list_length guard applies (default 100) to prevent giant $in clauses. - Range ops (between): two values, ?age__between=21,65 → {$gte:21,$lte:65}. Sugar over gte+lte; we keep both spellings, between is just nicer to read. - Bool-valued ops (isnull, empty, exists): parse true/false/1/0.

Safety & performance (defaults that protect users)¶

Filtering APIs are an unbounded attack/footgun surface. Defaults are conservative:

• regex off by default (ReDoS + collection-scan risk). Enable per-field or globally with eyes open; when enabled we anchor/length-cap patterns and document the risk.
• contains/icontains values are re.escape()d — always, not as an option — so user input is a literal substring, never a pattern. They still compile to unanchored regex in Mongo → collection scans on unindexed fields. We expose this honestly in docs and offer the text_search operator, which uses a real Mongo text index where one exists.
• max_list_length caps $in/$all blowups.
• max_filters caps the number of simultaneous filters per request.
• max_limit / default_limit on pagination; an unbounded limit is never allowed.
• Sortable-field allow-list (default = filterable set) prevents sorting on unindexed fields by surprise.
• Field allow-list in FilterSet prevents accidental field exposure.

All guards are config knobs with safe defaults — the library is safe out of the box and tunable when you know your indexes.

Continue to 03 — Architecture.