Running anything on an HPC cluster usually means writing SLURM batch scripts, copying files with scp, loading the right modules, pulling containers, and staring at squeue. On top of that, you're constantly fighting storage: home directories have tight quotas, container images are multi-gigabyte, and scratch space disappears on reboot.
tccp opens a single SSH ControlMaster connection to the data transfer node (DTN) — one Duo push per session. Everything is multiplexed on that connection. For SLURM commands (sbatch, squeue, scancel), tccp hops through the DTN to the login node. For running code and syncing files, it hops to the compute node.
Compute nodes don't accept external SSH keys, so all access goes through
the DTN using cluster-internal authentication. Port forwarding is set up
on both SSH hops during tccp shell.
The cluster has two relevant storage tiers. tccp puts things in the right place automatically:
| location | properties | what tccp puts there |
|---|---|---|
| NFS home (~/) | Persistent, quota-limited, visible from all nodes | Container images (when cache-containers: true), dtach binary, output directory, session state |
| Compute /tmp | Fast local disk, large (hundreds of GB), wiped on reboot | Synced project files, container images (default), singularity cache, dtach socket, env script |
Container pulls are the biggest storage challenge. OCI blobs and SIF
conversion can need ~12GB of temp space — more than most DTN /tmp
partitions and way over home dir quotas. So tccp always pulls containers on the
compute node (which has a large /tmp). By default, the final SIF stays on
compute /tmp. With cache-containers: true, it's stored on NFS
instead, so it persists across nodes and sessions.
tccp startsbatch job on the login node with requested resources. If no GPU is specified, auto-selects by querying sinfo.squeue until the job is RUNNING and a compute node is assigned.singularity exec to catch namespace or permission issues early..tccp-env.sh with PATH, PYTHONUSERBASE, etc.~/.tccp/projects/{name}/session.yaml.
The shell runs under dtach, so it survives disconnects, laptop sleep,
and WiFi drops. tccp shell reattaches anytime. Ctrl+S
is the fast edit-sync-test cycle: detach, push code changes, reattach.
When you're done, tccp stop pulls output from NFS and
cancels the SLURM job.