how it works

the problem

Running anything on an HPC cluster usually means writing SLURM batch scripts, copying files with scp, loading the right modules, pulling containers, and staring at squeue. On top of that, you're constantly fighting storage: home directories have tight quotas, container images are multi-gigabyte, and scratch space disappears on reboot.

the approach

cluster tccp your laptop DTN file sync, dtach port forwarding login sbatch, squeue, scancel compute your code runs here GPU, large /tmp SSH ControlMaster SSH hop SSH hop everything multiplexed on one SSH ControlMaster connection (one Duo push)

tccp opens a single SSH ControlMaster connection to the data transfer node (DTN) — one Duo push per session. Everything is multiplexed on that connection. For SLURM commands (sbatch, squeue, scancel), tccp hops through the DTN to the login node. For running code and syncing files, it hops to the compute node.

Compute nodes don't accept external SSH keys, so all access goes through the DTN using cluster-internal authentication. Port forwarding is set up on both SSH hops during tccp shell.

storage strategy

The cluster has two relevant storage tiers. tccp puts things in the right place automatically:

locationpropertieswhat tccp puts there
NFS home (~/) Persistent, quota-limited, visible from all nodes Container images (when cache-containers: true), dtach binary, output directory, session state
Compute /tmp Fast local disk, large (hundreds of GB), wiped on reboot Synced project files, container images (default), singularity cache, dtach socket, env script

Container pulls are the biggest storage challenge. OCI blobs and SIF conversion can need ~12GB of temp space — more than most DTN /tmp partitions and way over home dir quotas. So tccp always pulls containers on the compute node (which has a large /tmp). By default, the final SIF stays on compute /tmp. With cache-containers: true, it's stored on NFS instead, so it persists across nodes and sessions.

what happens when you type tccp start

  1. Allocate — submits an sbatch job on the login node with requested resources. If no GPU is specified, auto-selects by querying sinfo.
  2. Wait for node — polls squeue until the job is RUNNING and a compute node is assigned.
  3. Ensure container — checks for cached SIF, pulls if missing. Pull runs on compute node (large /tmp for temp files).
  4. Verify runtime — tests singularity exec to catch namespace or permission issues early.
  5. Ensure dtach — checks for dtach binary on DTN, builds from source if missing.
  6. Sync files — tars up your local project files and pipes them through SSH to the scratch dir on the compute node.
  7. Setup environment — creates output dirs (NFS + scratch), writes .tccp-env.sh with PATH, PYTHONUSERBASE, etc.
  8. Run init — executes your init command inside the container (if configured).
  9. Start dtach — launches a persistent bash shell via dtach inside the container.
  10. Save state — writes session info to ~/.tccp/projects/{name}/session.yaml.

session lifecycle

tccp start allocate + setup tccp shell attach/detach freely Ctrl+S sync + reattach tccp stop pull output + cancel

The shell runs under dtach, so it survives disconnects, laptop sleep, and WiFi drops. tccp shell reattaches anytime. Ctrl+S is the fast edit-sync-test cycle: detach, push code changes, reattach. When you're done, tccp stop pulls output from NFS and cancels the SLURM job.

where things live

NFS home (~/.tccp/) persistent, quota-limited, all nodes config.yaml global config bin/dtach built on first use containers/ only when cache-containers: true {image}.sif projects/{name}/ session.yaml session state output/ bind-mounted into container Compute /tmp/{user}/ fast, large, wiped on reboot containers/ default SIF location singularity-cache/ OCI layer cache {project}/ scratch dir (your code) synced files output/ bind mount → NFS .tccp.sock dtach socket .tccp-env.sh environment script .local/ PYTHONUSERBASE persistent (NFS) ephemeral (local /tmp) directory/ file description