Command line interface

The pathling command line interface surfaces the functionality of the Pathling Python library through concise, scriptable commands. It is shipped as a console script within the pathling Python package, so its version always matches the library version.

Each invocation starts a fresh Spark session. The cold start (typically 10-30 seconds) is communicated through a progress indicator on standard error. Data is written to standard output, while progress, status, and errors are written to standard error, so that piped output stays clean.

Installation

The CLI is part of the pathling package and requires a supported Java runtime to be present (as already required by the underlying PySpark dependency). The CLI reports the absence of Java clearly but does not install it.

The simplest way to run it is with uv:

# Run without installing.
uvx pathling --version

# Or install the tool so that `pathling` is on your PATH.
uv tool install pathling
pathling --help

Global options

The following options are accepted before any command and may also be supplied through a configuration file.

Option	Config key	Default
--tx-server	tx-server	the library default terminology server
--tx-store	tx-store.path	none (remote mode)
-	[tx-store.dialect-aliases]	none
--tx-client-id, --tx-client-secret, --tx-token-endpoint, --tx-scope	[terminology-auth]	none
--fhir-version	fhir-version	R4
--spark-conf KEY=VALUE	[spark]	none
--config PATH	-	$XDG_CONFIG_HOME/pathling/config.toml
--verbose	-	off

Values are resolved with the precedence flag > config file > built-in default. The --verbose flag re-enables Spark and JVM logging and prints full stack traces on error.

Exit codes are 0 for success, 1 for a runtime failure, and 2 for a usage error.

Data source inputs

Commands that read FHIR data take a positional SOURCE path. The format is auto-detected from the path contents (ndjson, FHIR Bundles, Parquet, or Delta) and can be set explicitly with --from ndjson|bundles|parquet|delta. If the format cannot be determined, the error lists what was found and shows how to specify --from.

Output options

Tabular results (from view, fhirpath, and the terminology commands) render as a human-readable table by default.

Option	Behaviour
--format	table (default), csv, ndjson; with -o also parquet, delta.
-o PATH	Write to a file instead of stdout; the format, and for .tsv the delimiter, is inferred from the extension (.csv/.tsv, .ndjson/.jsonl, .parquet).
--limit N	Row cap for stdout table output (default 50).
--overwrite	Allow replacing an existing output path.
--departition/--no-departition	Write file output as a single file (default) or as a Spark directory of part files. No effect on Delta.
--delimiter CHAR	Field separator for CSV input and output (default: a tab for a .tsv path, otherwise ,). Accepts an escape such as \t.
--header/--no-header	Include a header row in CSV output (default enabled).

File output is produced by Spark's distributed writers, so results larger than driver memory can be written. By default the output is departitioned to a single file at the path given; pass --no-departition to keep Spark's native directory of part files. Delta output is always written as a table directory.

For scripted use, prefer --format csv or --format ndjson, which stream the full result.

The --delimiter and --header/--no-header options apply to CSV output for view, fhirpath, and the terminology commands, and are ignored for non-CSV formats. The delimiter is also used to read CSV input in the terminology commands.

--delimiter has no fixed default. When it is omitted, the input side and the output side each take their default from their own path: a .tsv extension (in any letter case) means a tab, and anything else means a comma. CSV written to standard output, where there is no path, is comma-separated. A tab-separated dataset therefore round-trips with no delimiter flag at all:

pathling member-of codes.tsv --code-column code \
  --system http://snomed.info/sct --value-set <uri> \
  -o out.tsv

Because the two sides resolve independently, codes.tsv -o out.csv reads with a tab and writes with a comma. A delimiter given explicitly applies to both sides and overrides both extensions, so --delimiter ',' on a .tsv path yields a comma. Whenever a tab is derived from an extension rather than typed, the CLI says so on standard error, naming the path.

Commands

convert

Convert FHIR data between formats.

pathling convert data/ --to parquet -o warehouse/
pathling convert bundles/ --from bundles --to ndjson -o out/
pathling convert bundles/ --from bundles --type Patient --type Condition --to ndjson -o out/

The --mode overwrite|error|append|merge option controls the save mode for Parquet and Delta output (merge is valid for Delta only). For a Bundles source, the repeatable --type option names the resource types to read; when given, those types are used directly and the driver-side discovery pass is skipped. A summary of the resource types written and the output location is printed at the end.

view

Run a SQL on FHIR ViewDefinition against a data source.

pathling view data/ --view patients.json
pathling view data/ --view patients.json --format csv
pathling view data/ --view-json '{"resource":"Patient", ...}' -o results.parquet

The view is supplied as a file (--view) or inline JSON (--view-json). A --filter FHIR search expression restricts the resources processed.

fhirpath

Evaluate a FHIRPath expression.

# Data source mode: one row per resource with its id and result.
pathling fhirpath data/ -t Patient -e 'name.family'

# Single resource mode: the typed result values.
pathling fhirpath patient.json -e 'name.given.first()'

In data source mode, -t/--type selects the subject resource type and --filter restricts the resources processed. In single resource mode (when SOURCE is a single FHIR resource JSON file), --context and repeatable --var name=value options are supported.

Both modes name the result column result: data source mode pairs it with each resource's id (one row per resource), while single resource mode pairs it with each result item's type (one row per result item).

export

Bulk export data from a FHIR server.

pathling export https://server/fhir -o out/ --type Patient
pathling export https://server/fhir -o out/ --group 123

A system-level export runs by default; --group ID and repeatable --patient REF select group-level and patient-level exports (the two are mutually exclusive). The export supports --type, --elements, --since, --type-filter, --include-associated-data, --timeout, and --max-downloads.

SMART backend services authentication is configured with --client-id, --token-endpoint, --scope, and exactly one of --private-key-jwk or --client-secret. Secret values accept a literal, a @/path/to/file reference, or fall back to the PATHLING_PRIVATE_KEY_JWK / PATHLING_CLIENT_SECRET environment variables so that they need not appear in shell history.

run

Execute Python code with the Pathling environment ready. The code runs with two variables already in scope: spark (the Spark session) and pc (the configured Pathling context), built with the same configuration resolution as every other command.

The Pathling public functions are also pre-imported, so no from pathling import ... line is needed. This covers the terminology and coding functions (to_coding, to_snomed_coding, to_loinc_coding, member_of, translate, subsumes, and so on), the argument helper types (Coding, PropertyType, Equivalence), and the API types (PathlingContext, DataSource, and the rest). The pre-imported names are exactly the public functions and types listed in the Python API reference. The terminology display function is bound under both its natural name display and the alias tx_display. The name pathling is bound to the package module itself, so a bare import pathling is a harmless no-op that leaves pc intact. Any pre-imported name is only a default: assigning or defining the same name in your own code takes precedence, and an explicit from pathling import ... continues to work unchanged.

# Run a script file.
pathling run my_script.py

# Run an inline one-liner.
pathling run -c "print(spark.version)"

# Pipe a script through stdin.
cat job.py | pathling run -

# Pass arguments through to the script.
pathling run etl.py --input data.ndjson out/

For example, a script that projects a tabular view of patients and then summarises it with Spark SQL:

patients = pc.read.ndjson("data").view(
    "Patient",
    select=[{"column": [{"path": "gender", "name": "gender"}]}],
)
patients.createOrReplaceTempView("patient")
spark.sql("SELECT gender, count(*) AS count FROM patient GROUP BY gender").show()

The code source is exactly one of a script path, - (standard input), or -c CODE; supplying both a script and -c, or neither, is a usage error (exit code 2), reported before the Spark session is started.

Execution follows Python interpreter semantics: the module runs as __main__; for file scripts __file__ is set and the script's directory is prepended to sys.path; trailing arguments arrive in sys.argv with sys.argv[0] being the script path, -c, or - as the interpreter would set it. Dash-prefixed arguments pass through to the script rather than being parsed by the CLI.

The exit code is 0 on success and 1 for an uncaught exception or syntax error, with a standard Python traceback (no CLI frames) printed to standard error. sys.exit(n) exits with n and no traceback. The script's stdout passes through unmodified.

console

Open an interactive IPython console with the same spark and pc variables in scope, and the Pathling public functions pre-imported just as in run.

pathling console

After the startup progress indicator, a banner identifies the Pathling version, the variables in scope, and the pre-imported functions. Errors evaluated at the prompt show normal tracebacks without ending the session; leave with exit or Ctrl-D (exit code 0).

The one difference from run is the terminology display function. IPython installs its own display at the prompt, so Pathling's terminology display is bound as tx_display only; display remains IPython's built-in. A snippet that uses tx_display therefore behaves the same in the console and in a script.

Terminology commands

The member-of, translate, subsumes, subsumed-by, display, property-of, and designation commands read a tabular dataset (CSV, Parquet, or Delta), build codings from a --code-column plus either a fixed --system URI or a --system-column, and append the operation's result column(s).

pathling member-of codes.csv --code-column code \
  --system http://snomed.info/sct \
  --value-set 'http://snomed.info/sct?fhir_vs=refset/...'

pathling translate codes.csv --code-column code \
  --system http://snomed.info/sct --concept-map '<uri>'

In local terminology mode, the forms accepted by --value-set and --concept-map - imported canonical URLs, the SNOMED CT implicit forms, ECL, and VCL - are listed under value set and concept map expressions. Against a terminology server, they are whatever that server supports.

The input format is set with --from csv|parquet|delta. When omitted, it is auto-detected from the dataset path: files ending in .csv or .parquet are read as CSV or Parquet; a directory containing a _delta_log entry is read as Delta; and any other directory containing at least one .parquet file is read as Parquet. CSV inputs are read with a header row by default and all columns as strings. Passing --from bypasses detection, which is useful for Delta tables or CSV files with an unconventional extension. An input whose format cannot be determined - an unrecognised suffix, or a directory with neither a _delta_log entry nor .parquet files - is reported as a usage error before any Spark session starts. This also lets you read the CLI's own Parquet or Delta output directory straight back into another terminology command.

# Explicit Delta input.
pathling display warehouse/codes --from delta \
  --code-column code --system http://snomed.info/sct

# Auto-detected Delta directory (contains _delta_log).
pathling member-of warehouse/codes --code-column code \
  --system http://snomed.info/sct \
  --value-set 'http://snomed.info/sct?fhir_vs=refset/...'

The default result column names (member_of, translated_system and translated_code, subsumes, subsumed_by, display, property, designation) can be overridden with --result-column.

CSV input is read with the shared --delimiter (so a semicolon-separated dataset is read correctly); a .tsv file is read as CSV and, with no --delimiter, as tab-separated. Pass --no-input-header to read a headerless CSV; its columns are then addressable by the positional names Spark assigns (_c0, _c1, ...), which you reference via --code-column, --system-column, and the other column options.

pathling member-of headerless.csv --no-input-header --code-column _c0 \
  --system http://snomed.info/sct --value-set '<uri>'

Subsumption against a fixed target coding

The subsumes and subsumed-by commands test each row's coding against a target coding. The target code can come from a second column (--other-code-column) or be a single fixed value applied to every row (--other-code); exactly one of the two must be given. This lets you test a column of codes against one known concept without first adding a constant column to the data. The target system is supplied with --other-system (a fixed URI) or --other-system-column (a per-row column), and --system-version applies to both codings.

# Fixed target coding: is each code subsumed by Diabetes mellitus?
pathling subsumed-by codes.csv --code-column code \
  --system http://snomed.info/sct \
  --other-code 73211009 --other-system http://snomed.info/sct

# Two-column comparison: does each code in column a subsume the code in column b?
pathling subsumes pairs.csv --code-column a \
  --system http://snomed.info/sct \
  --other-code-column b --other-system http://snomed.info/sct

Invalid combinations - supplying both or neither of --other-code / --other-code-column, or both or neither of --other-system / --other-system-column - are reported as usage errors before any Spark session starts.

Terminology import commands

The import-snomed and import-fhir-terminology commands import terminology content into a local terminology store for use with local terminology mode. Both take a SOURCE path and a STORAGE_PATH for the store, report progress, and print a completion summary.

pathling import-snomed /data/rf2.zip /data/tx-store
pathling import-fhir-terminology /data/hl7.terminology.tgz /data/tx-store

import-snomed accepts --edition-uri to override the detected SNOMED edition/version, and --dense-id-order pre-order to assign internal concept identifiers by a depth-first traversal of the is-a hierarchy, which makes the hierarchy index materially smaller at query time in exchange for identifiers that shift more between releases - see reducing the memory the hierarchy takes at query time.

It also accepts --default-dialect, which names the dialect whose preferred synonyms become each concept's stored display: a tag such as en-GB, or a language reference set identifier. When the flag is omitted, the tx-store.default-dialect config key applies; when neither is set, the dialect is chosen from the release. A release holding several language reference sets that is not the International edition fails the import, listing the candidates, so that one can be named - see dialects.

pathling import-snomed --default-dialect en-GB /data/rf2.zip /data/tx-store

import-fhir-terminology accepts a JSON file, a directory of JSON files, or a FHIR NPM package (.tgz), and imports CodeSystems of any size with bounded memory (for example, the multi-gigabyte OMOP vocabulary CodeSystem).

An RF2 source given to import-snomed must be self-contained: rows referencing concepts the source does not itself ship are dropped, which is the ordinary shape of a derived or extension package supplied without the release it depends on. The import reports, for each file, how many of its active rows resolved, which is how a shortfall is detected. The same section gives a recipe for combining a package with its dependency.

Large imports run for many minutes. With --verbose, the command streams a running count of parsed concepts and stage-transition messages so progress is visible; without it, the startup spinner covers the wait. If an import fails partway through writing a CodeSystem, it reports that the store may hold a partial version and advises re-running; because content is keyed by system version, re-running with a corrected source repairs the store.

Peak memory does not grow with the number of concepts, but the largest vocabularies still need more driver heap than the 1 GB default to hold the working set of the Spark joins that build the store. The OMOP vocabulary (around 6.6 million concepts), for example, imports comfortably with a 4 GB heap. Set the heap with the SPARK_DRIVER_MEMORY environment variable; in local mode the driver JVM starts before --spark-conf can size its heap, so that flag has no effect on driver memory.

SPARK_DRIVER_MEMORY=4g pathling import-fhir-terminology \
  /data/ohdsi.fhir.omop-0.1.0.tgz /data/tx-store

The STORAGE_PATH positional is optional: when it is omitted, the commands fall back to the configured tx-store.path (see below). An explicit positional wins over the configured path. Supplying neither is a usage error.

Local terminology mode

Once a store has been populated, point any terminology-evaluating command at it with --tx-store (or the tx-store.path config key) to evaluate terminology against the local store instead of a remote server, entirely offline. This applies to every command that creates a session, including view, fhirpath, convert, run, and console.

pathling --tx-store /data/tx-store member-of codes.csv \
  --code-column code --system 'http://snomed.info/sct' \
  --value-set 'http://snomed.info/sct?fhir_vs=ecl/<404684003'

The [tx-store] config table records the store once, along with optional tuning values:

[tx-store]
path = "/data/tx-store"                      # selects local mode
default-snomed-edition = "32506021000036107" # optional
expansion-cache-size = 200                   # optional, positive integer
default-dialect = "en-AU"                    # optional, used by import-snomed

[tx-store.dialect-aliases]                   # optional
en-NZ = "271000210107"

The [tx-store.dialect-aliases] table registers additional dialect tags, mapping a language tag to the identifier of the SNOMED CT language reference set that serves it. An entry for a tag that is already recognised replaces the built-in mapping for it. A value that is not a table of strings is reported and ignored rather than being fatal, since an unrecognised tag simply expresses no preference. See dialects.

The presence of a store selects local mode. When a store is configured, the store wins over any explicitly set --tx-server or terminology authentication: each is ignored and a warning is printed. The built-in default server URL never triggers this warning. A runtime failure in local mode names the store path and suggests the import commands rather than a terminology server URL.

Configuration file

Defaults for the global options can be set in a TOML file at ${XDG_CONFIG_HOME:-~/.config}/pathling/config.toml:

tx-server = "https://tx.example.org/fhir"
fhir-version = "R4"

[terminology-auth]
client-id = "my-client"
client-secret = "..."
token-endpoint = "https://auth.example.org/token"

[bulk-auth]
client-id = "bulk-client"
token-endpoint = "https://auth.example.org/token"

[spark]
"spark.sql.shuffle.partitions" = 16
"spark.executor.memory" = "4g"

[tx-store]
path = "/data/tx-store"
default-snomed-edition = "32506021000036107"
expansion-cache-size = 200
default-dialect = "en-AU"

[tx-store.dialect-aliases]
en-NZ = "271000210107"

Command-line flags always take precedence over the config file. Unknown keys produce a warning that names the key and lists the valid keys. The [tx-store] table selects local terminology mode; its tuning keys are config-file only, while the store path can also be set with --tx-store, and default-dialect with the import-snomed command's --default-dialect flag.

Spark configuration

The [spark] table sets arbitrary Apache Spark properties on the session that every data command builds. Use it to tune the engine - for example to raise executor memory, change the shuffle partition count, or add a cloud storage connector - or pass one or more --spark-conf KEY=VALUE flags for a single invocation. The flag is repeatable and overrides the [spark] value for the same key; when the same key is given more than once on the command line, the last occurrence wins. Because the value is split on the first = only, a value may itself contain = (for example --spark-conf spark.driver.extraJavaOptions=-Dfoo=bar).

[spark]
"spark.sql.shuffle.partitions" = 16
"spark.sql.adaptive.enabled" = true
"spark.executor.memory" = "4g"
"spark.jars.packages" = "org.apache.hadoop:hadoop-aws:3.4.1"
"spark.hadoop.fs.s3a.secret.key" = "@/run/secrets/s3-key"

The resolved settings are merged with Pathling's own required Spark defaults rather than replacing them, so Pathling keeps working while your tuning takes effect. The full precedence, highest first, is:

1. A --spark-conf flag value.
2. The [spark] table value in the chosen config file.
3. The CLI's quiet-mode logging settings (applied only when not --verbose, and only for keys you did not set).
4. Pathling's managed defaults (always present for the managed keys below).

Keys and values. Every key must begin with spark.; any other key is an error that names the offending key and aborts before a Spark session starts. Values may be strings, integers, floats, or booleans, and are coerced to the strings Spark expects (booleans become true/false). TOML arrays and tables are not accepted; list-valued properties use Spark's native comma-separated string. A string value may be a @/path/to/file secret reference, read exactly as authentication secrets are.

Managed keys. Three keys are owned by Pathling and merged item by item so the library is never broken:

• spark.jars.packages - your coordinates are unioned with Pathling's managed coordinates (the library runtime and Delta Lake) and deduplicated. Supplying a managed coordinate at a different version is allowed and applies your version, but prints a warning naming the coordinate, since a non-default version may not be supported.
• spark.sql.extensions - your extension class names are unioned with Pathling's, and the Delta extension always remains present.
• spark.sql.catalog.spark_catalog - this is locked to Pathling's Delta catalog. Setting it to that value is a no-op; setting it to anything else is an error.

Quiet-mode logging. By default the CLI suppresses Spark and JVM logging by setting spark.driver.extraJavaOptions. If you set that key yourself, your value replaces the CLI's, so Spark logging is no longer suppressed. To keep both, run with --verbose, or include the quiet log4j2 option in the value you supply.

Project-local configuration

The CLI also looks for a file named pathling.toml in the current working directory. When present, it is used in place of the user-level config file - not merged with it. This lets a project carry its own settings (for example a particular terminology server) without editing your personal config or passing --config on every command.

Exactly one config file is ever read, chosen by this precedence (highest first):

1. The path given to --config.
2. A pathling.toml in the current working directory.
3. The user-level config.toml.
4. None, in which case built-in defaults apply.

Because files are never merged, any key the chosen file omits falls back to its built-in default rather than to a value from another file. Discovery is limited to the current working directory; the CLI does not search parent directories.

When a pathling.toml is discovered and used, the CLI prints a one-line notice on standard error naming the file (and the user-level file it overrides, when one exists), so the active configuration is never a surprise:

Using project config /path/to/pathling.toml (overrides ~/.config/pathling/config.toml).

Passing an explicit --config skips project-local discovery, and no such notice is printed. An explicit --config path must exist; a missing path is an error rather than a silent fall-back to another config file.