Nextflow is a workflow system built for data-intensive pipelines that must run the same way on a laptop, a university cluster or the cloud. Instead of matching file names, it moves data through channels between processes that each run in their own isolated work directory, usually inside a container. This tutorial builds a complete RNA-seq quantification pipeline (fastp, Salmon, MultiQC) step by step, then explains configuration, profiles, executors, resuming, the nf-core community pipelines, best practices and troubleshooting.
Nextflow is a workflow engine and domain-specific language (DSL) built on the Java virtual machine and Groovy. It lets you write a pipeline as a set of processes (steps) connected by channels (queues of data). The same pipeline script can run on different executors (local, SLURM, SGE, AWS Batch, Google Batch, Kubernetes) by changing only the configuration.
Main strengths
Dataflow model: a process starts as soon as its input data are available, and many tasks run in parallel automatically.
Portability: the pipeline logic is separated from the execution settings in nextflow.config.
Containers first: Docker, Singularity or Apptainer, Podman and conda are supported per process.
Resume: -resume reuses cached results of tasks that did not change.
Reproducibility: pipeline version, parameters and software are tracked in reports.
Community: nf-core provides curated, tested pipelines (rnaseq, sarek, ampliseq, taxprofiler, and many more) that you can run without writing code.
Nextflow or Snakemake?
Aspect
Nextflow
Snakemake
Language
Groovy-based DSL
Python-based
Logic
Dataflow: channels connect processes
Pull: target files drive rules
Execution folders
Every task in a hashed work directory
Outputs written to the paths you declare
Containers
Central, easy per process
Supported per rule
Cloud
Very strong native support
Supported through plugins
Ready-made pipelines
nf-core (large, curated)
Workflow Catalog
Strong point
Large-scale, shared, production pipelines
Flexible custom analyses in Python
There is no wrong choice. Many labs use nf-core pipelines for standard analyses and write their own workflows in the system the team knows best.
2. Installation
Nextflow needs Java (a recent LTS version; check the documentation for the version required by your release), and is available through conda and as a single launcher.
# Option 1: conda or mamba
mamba create -n nextflow -c conda-forge -c bioconda nextflow
mamba activate nextflow
# Option 2: the installer
curl -s https://get.nextflow.io | bash
chmod +x nextflow && mv nextflow ~/bin/
nextflow -version
nextflow run hello # test with the built-in example
Install Docker (laptops, cloud) or Apptainer/Singularity (most HPC clusters) to run containers.
3. Core concepts
Concept
Meaning
Process
A step of the pipeline: input, output, and a script that runs for each input
Task
One execution of a process with specific input values
Channel
An asynchronous queue that carries values or files between processes
Operator
A function that transforms channels (map, collect, join, groupTuple, combine, mix)
Workflow
The block that connects processes and channels
Work directory
The folder work/ where each task runs in a unique subfolder
Config
nextflow.config, settings for resources, executor, containers and parameters
Profile
A named group of config settings (-profile slurm,apptainer)
Module
A process stored in its own file and imported in the workflow
Dataflow in one sentence: a process runs once for each item that arrives in its input channel, and its output is emitted into a new channel that downstream processes consume.
4. Your first pipeline
Create hello.nf:
nextflow.enable.dsl = 2
process SAYHELLO {
input:
val name
output:
stdout
script:
"""
echo "Hello ${name} from Bioeduc"
"""
}
workflow {
names = channel.of('ctrl1', 'ctrl2', 'treat1')
SAYHELLO(names) | view
}
Run it:
nextflow run hello.nf
Nextflow starts three tasks (one per value) in parallel, each in its own folder under work/, and prints the results. Rerunning with -resume reuses the cached tasks.
Important details:
nextflow.enable.dsl = 2 selects the modern syntax (DSL2), which is the default in current releases.
script: contains a shell block. In a triple-quoted string, ${name} is a Nextflow variable; use \$ to write a literal shell variable.
val handles a value; path handles a file; tuple bundles several together.
This keeps all metadata attached to the data throughout the workflow.
Process directives worth knowing
Directive
Example
Use
cpus, memory, time
cpus 8, memory '16 GB', time '2h'
Resource requests
errorStrategy
'retry', 'ignore', 'terminate'
What to do on failure
maxRetries
maxRetries 3
Number of retries
maxForks
maxForks 4
Limit simultaneous tasks of a process
conda
conda 'bioconda::salmon=1.10.3'
Per-process conda environment
storeDir
storeDir 'resources/index'
Store results permanently and skip if present (for indexes)
scratch
scratch true
Run on node-local disk
when
when: params.run_qc
Conditional execution
cache
cache 'lenient'
Control the resume behavior
6. Channels and operators
Creating channels
channel.of(1, 2, 3) // values
channel.fromPath('data/*.fastq.gz') // files
channel.fromFilePairs('data/*_{R1,R2}.fastq.gz') // paired-end: [id, [r1, r2]]
channel.fromPath('samplesheet.csv').splitCsv(header: true) // rows of a CSV as maps
channel.fromSRA('SRP123456') // download from SRA
Queue channels and value channels
A queue channel emits several items and is consumed once.
A value channel holds a single value that can be read any number of times. Reference files (an index, a genome) should be value channels, so that every sample can use them. Single path or val arguments passed to a process become value channels, and .first() or channel.value(...) create one explicitly.
Frequently used operators
Operator
What it does
Example
map
Transform each item
.map { row -> [row.sample, file(row.fq1)] }
filter
Keep items matching a condition
.filter { it.condition == 'treated' }
collect
Gather all items into one list
.collect() (for MultiQC input)
groupTuple
Group items by a key
.groupTuple() (merge lanes by sample)
join
Join two channels by a key
a.join(b)
combine
Cartesian product
samples.combine(references)
mix
Merge channels into one
a.mix(b)
flatten
Emit list elements one by one
.flatten()
view
Print items for debugging
.view()
splitFastq
Split large FASTQ into chunks
.splitFastq(by: 100000)
branch
Split a channel into several by condition
For single-end versus paired-end
**Debug channels with .view()** whenever a process does not receive what you expect.
7. A complete RNA-seq quantification pipeline
A Nextflow pipeline: processes connected by channels, each task isolated in its own work directory (schematic made for Bioeduc).
Project layout
project/
main.nf
nextflow.config
modules/
fastp.nf
salmon.nf
multiqc.nf
assets/
samplesheet.csv
conf/
base.config
results/ # created by the pipeline
# Locally with Docker
nextflow run main.nf -profile docker --transcripts resources/transcripts.fa
# On a SLURM cluster with Apptainer
nextflow run main.nf -profile slurm,apptainer --transcripts resources/transcripts.fa
# After a failure or when adding samples
nextflow run main.nf -profile slurm,apptainer --transcripts resources/transcripts.fa -resume
The log shows each process with the number of completed tasks, and each task has a hash such as [3f/a1b2c3] that points to its work directory work/3f/a1b2c3.../.
8. Understanding the work directory and -resume
Every task runs in its own folder under work/. It contains:
File
Content
.command.sh
The exact script that was executed
.command.run
The wrapper that sets up the environment and container
.command.out, .command.err
Standard output and error
.command.log
Combined log
.exitcode
Exit status of the task
Staged inputs
Symbolic links to the input files
Outputs
Files created by the task
**-resume** computes a hash for each task from the inputs, script, container and parameters. If the hash exists in the cache and the task finished, the stored result is reused. A task reruns when something in its hash changes, and all downstream tasks rerun too.
Tips:
Keep the work directory on a fast, shared file system and clean it regularly (nextflow clean -f -before <run name>), because it can grow very large.
publishDir (or the newer workflow outputs feature in recent releases) copies only the final results out of work/; check the documentation of your release for the recommended approach.
Use nextflow log to list previous runs and their session IDs, and nextflow log <run> -f name,status,workdir to find a task folder.
9. Containers, conda and reproducibility
process.container = 'quay.io/biocontainers/salmon:1.10.3--h45fbf2d_4' // for all processes
Method
Config
Notes
Docker
docker.enabled = true
Common on laptops and in the cloud
Apptainer or Singularity
apptainer.enabled = true
Standard on HPC where Docker is not allowed
Podman
podman.enabled = true
Rootless alternative
Conda
conda.enabled = true and conda 'bioconda::tool=x.y'
Simple but slower and less isolated than containers
Wave
wave.enabled = true
Builds containers on demand, from conda packages for example
For publication: pin pipeline version (-r v1.2.0 for pipelines from Git), container tags (or digests), and save the run report and command line.
10. Using nf-core pipelines
nf-core is a community that develops standardized, well-tested pipelines. You can run a published pipeline without writing any code.
# Install the helper tools
pip install nf-core
# List the available pipelines
nf-core pipelines list
# Run the RNA-seq pipeline on a test dataset
nextflow run nf-core/rnaseq -profile test,docker --outdir results_test
# Run on your data
nextflow run nf-core/rnaseq -r 3.14.0 \
-profile apptainer,slurm \
--input samplesheet.csv \
--outdir results \
--genome GRCh38 \
-resume
Practical points:
Read the usage and parameters pages of the pipeline; each has its own sample sheet format.
Use the **-profile test** first to confirm that your installation works.
Put your institution's settings into a custom config with -c my_cluster.config, or use a published institutional profile from the nf-core configs repository.
Pin the version with -r, and keep -params-file params.yaml for all parameters, so the run is reproducible.
Start from the template to build your own nf-core style pipeline (nf-core pipelines create), and test modules with nf-test.
Pipelines produce a MultiQC report and execution reports that you should read before the results.
Run the Nextflow head job inside tmux or screen, or as a small batch job, because it supervises all tasks. Set the NXF_WORK or -w option to scratch space and use NXF_APPTAINER_CACHEDIR to share container images between users and avoid repeated downloads.
Cloud and platforms
AWS Batch, Google Cloud Batch and Azure Batch executors move work to elastic compute and use object storage (S3, GCS) for the work directory. Seqera Platform provides a web interface for launching and monitoring runs. These features need extra credentials and cost control, so start with the official guides.
12. Debugging: a practical routine
Read the error message at the end of the log; it shows the failing task, exit status, the command and the work directory.
Go to the work directory and read .command.sh, .command.err and .command.out.
Rerun the failing task by hand: bash .command.run (it recreates the container environment) or bash .command.sh for the plain script.
Use .view() on channels and -dump-channels to check what is flowing.
Test with one sample and a tiny dataset and the -stub-run option (using stub: blocks) to check the wiring of the workflow without running the tools.
Look at nextflow log and the trace file to see runtime, memory and exit codes.
13. Troubleshooting: symptoms, causes and fixes
Message or symptom
Likely cause
What to do
Process never starts
Input channel is empty (wrong glob or pattern)
Use .view() on the channel, check file paths and checkIfExists: true
Only the first sample is processed
Reference passed as a queue channel and consumed once
Use a value channel (.first(), channel.value()) or pass a single path
Process runs only once per pair of lists
Channel mismatch between channels with different numbers of items
Use join on a common key or combine deliberately
Exit status 137
Out of memory (killed)
Increase memory, use memory = { 8.GB * task.attempt } with retry
Exit status 127
Command not found
Wrong container, missing software, or the tool is not in the PATH
Exit status 140 or 143
Scheduler killed the job (time limit)
Raise time, use retry with larger limits
-resume reruns everything
Inputs, script, parameters or file timestamps changed, or work directory moved
Compare with nextflow log -t, keep the same work directory, use cache 'lenient' on shared file systems
Cannot find input file in the container
Path not mounted
Use autoMounts, or bind paths explicitly (apptainer.runOptions = '-B /data')
Variable substitution problems in scripts
Dollar signs interpreted by Groovy
Escape shell variables as \\$VAR and use \\ for line continuation in triple-quoted strings
Output file not found error
Declared output pattern does not match actual files
Inspect the work directory and fix the path pattern
Work directory fills the disk
Intermediate files are kept for resume
Clean old runs with nextflow clean, store work/ on scratch
Docker permission denied on cluster
Docker not allowed
Use Apptainer or Singularity
Image pull is slow or fails on compute nodes
No network on nodes, repeated pulls
Pre-pull images (apptainer pull) and set a shared cache directory
Syntax or deprecation warnings after update
Newer releases tighten the language
Check the release notes, run nextflow lint if available, and pin the Nextflow version (NXF_VER=...)
Groovy error in the .map closure
Wrong tuple structure
Print items with .view() before and after the operator
14. Best practices checklist
Use DSL2, one process per module file, and a clear main.nf.
Separate logic (main.nf) from settings (nextflow.config, profiles).
Describe resources with labels rather than hard-coding them in processes.
Keep metadata with data using the meta map and tuples.
Test with a small dataset, -profile test and -stub-run before a large run.
Use -resume and keep work directories until the analysis is final.
Save reports (execution report, timeline, trace) and the exact command line.
Prefer existing nf-core pipelines and modules for standard analyses.
Store the pipeline in Git and document the sample sheet and parameters.
15. Frequently asked questions
Do I need to know Groovy or Java? Only a little. Most pipelines use simple closures, string handling and maps, which you can learn from examples.
Where are my results? Final results are in the folder set by publishDir (or the outputs configuration). The work/ folder holds intermediate files and is not meant to be your results folder.
Can I run a pipeline without Docker? Yes, with Apptainer or Singularity, Podman, conda, or software installed on the machine, though containers give the best reproducibility.
**What does -resume do when I add a new sample?** Tasks for the existing samples are reused from the cache and only the new sample's tasks run (steps that depend on all samples, such as MultiQC, rerun).
Should I use nf-core or write my own? Use nf-core for standard analyses (RNA-seq, variant calling, metagenomics). Write your own for specialized methods, and follow the nf-core structure where possible.
16. References and resources
Di Tommaso P et al. (2017). Nextflow enables reproducible computational workflows. Nature Biotechnology 35:316-319.
Ewels PA et al. (2020). The nf-core framework for community-curated bioinformatics pipelines. Nature Biotechnology 38:276-278.
Official Nextflow documentation (nextflow.io/docs), including the training material at training.nextflow.io.
nf-core documentation and pipeline list (nf-co.re).
Kurtzer GM, Sochat V, Bauer MW (2017). Singularity: scientific containers for mobility of compute. PLoS ONE 12:e0177459.
Merkel D (2014). Docker: lightweight Linux containers for consistent development and deployment. Linux Journal 2014(239):2.
Chen S et al. (2018). fastp: an ultra-fast all-in-one FASTQ preprocessor. Bioinformatics 34:i884-i890.
Patro R et al. (2017). Salmon provides fast and bias-aware quantification of transcript expression. Nature Methods 14:417-419.
Ewels P et al. (2016). MultiQC: summarize analysis results for multiple tools and samples in a single report. Bioinformatics 32:3047-3048.
Mölder F et al. (2021). Sustainable data analysis with Snakemake. F1000Research 10:33.
💬 Commentaires