Repeat Annotation: RepeatModeler and RepeatMasker
Repeats and transposable elements (TEs) make up a large part of most eukaryotic genomes, from about 10 percent in compact genomes to more than 80 percent in some plants. They hide genes from gene predictors, confuse aligners, and carry a record of genome evolution. This guide explains what repeats are, why they must be annotated and masked, which tools exist and how they differ, how to run a complete RepeatModeler and RepeatMasker workflow, how to interpret the results (repeat landscapes, classification), how to use the masked genome in gene prediction, and how to troubleshoot problems, with particular attention to non-model organisms.
1. What are genomic repeats?
A repeat is a DNA sequence that occurs many times in a genome. Repeats fall into two broad groups.
1. Tandem repeats are copies that sit next to each other:
- Microsatellites (SSRs): units of 1 to 6 bases (for example
CACACACA), used as genetic markers. - Minisatellites: units of about 10 to 60 bases.
- Satellite DNA: long arrays of units of tens to thousands of bases, found at centromeres and telomeres.
2. Interspersed repeats are copies scattered through the genome, mostly transposable elements (TEs), which are sequences able to move or copy themselves. The standard classification (Wicker et al., 2007) is:
| Class | Order | Examples | Mechanism |
|---|---|---|---|
| Class I: retrotransposons | LTR | Gypsy, Copia, ERVs (endogenous retroviruses) | "Copy and paste" through an RNA intermediate, with long terminal repeats |
| Class I | LINE | L1, CR1, RTE | Autonomous elements encoding reverse transcriptase, no LTR |
| Class I | SINE | Alu, B1, tRNA-derived SINEs | Non-autonomous, depend on LINEs for mobilization |
| Class II: DNA transposons | TIR | hAT, Tc1/mariner (TcMar), PiggyBac, CACTA | "Cut and paste" transposase with terminal inverted repeats |
| Class II | Helitron (RC) | Helitrons | Rolling-circle replication |
| Class II | Others | Maverick, Crypton | Large or unusual DNA elements |
Other repeated features include segmental duplications, multi-copy gene families (for example rRNA genes, histone genes), pseudogenes, and low-complexity regions (stretches rich in one or two bases).
How much of a genome is repeats? It varies greatly: roughly half of the human genome, about 85 percent of maize, and in many invertebrates, including mollusks, a large and often poorly characterized fraction. Genome size differences between related species are often explained mainly by TE content.
2. Why annotate repeats? Purposes
- Gene prediction: unmasked TEs produce thousands of false gene models (TE proteins such as reverse transcriptases and transposases look like genes), and repeats also confuse evidence alignment. Masking repeats before gene prediction is a standard step.
- Genome analysis: understand genome size evolution, TE invasions and bursts, and how TEs shape gene regulation, structure and adaptation.
- Assembly quality assessment: the completeness of repeat assembly (for example with the LTR Assembly Index) is a quality indicator.
- Comparative genomics: compare repeat content between species and populations.
- Variant and read analysis: avoid false variants and ambiguous alignments in repeats.
- Epigenetics: TE silencing (DNA methylation, small RNAs) is studied genome-wide only when TEs are annotated.
3. How repeats are identified: three strategies
| Strategy | Principle | Tools | Strengths | Limits |
|---|---|---|---|---|
| Homology-based | Compare the genome with libraries of known repeats | RepeatMasker with Dfam (and Repbase, now licensed) | Accurate classification of known families | Misses new or lineage-specific repeats |
| De novo repeat discovery | Find sequences that repeat in the genome itself, build consensus sequences | RepeatModeler2, RepeatScout, Red, Earl Grey | Finds species-specific repeats, needed for non-model organisms | Classification is partly unknown, may include multi-copy genes |
| Structure-based | Detect structural signatures (LTRs, TIRs, target-site duplications) | LTRharvest, LTR_FINDER, TIR-Learner, HelitronScanner, MITE-Hunter | Very high precision for full-length elements | Specific to element types |
| Integrated pipelines | Combine several of the above and filter | EDTA, Earl Grey, REPET | Comprehensive curated libraries | Run time, dependencies |
| Read-based (without assembly) | Cluster raw reads to find abundant repeats | RepeatExplorer, dnaPipeTE | Works with low-coverage reads, no assembly needed | Quantifies abundance but gives no genomic positions |
For a new genome, the usual best practice is: build a de novo library with RepeatModeler2 (optionally add structural tools such as LTRharvest through -LTRStruct), combine it with known repeats for the taxon from Dfam, and annotate and mask the genome with RepeatMasker.
4. Before you start: data and design
Genome assembly quality
Repeat annotation is only as good as the assembly. Long-read assemblies (PacBio HiFi, Oxford Nanopore) capture repeats much better than short-read assemblies, where repeats collapse or break contigs. For the best result:
- Use the highest-contiguity assembly you have (HiFi plus Hi-C scaffolding is excellent).
- Remove contamination first (bacterial, human, organellar contigs) with tools such as BlobToolKit or FCS-GX. Contaminant sequences produce spurious "repeat" families.
- Keep sequence names short and simple: very long headers, spaces and special characters cause trouble in RepeatMasker.
- Note the assembly version and keep a copy of the unmasked genome.
Wet-lab inputs (if you generate the genome)
High-molecular-weight DNA extraction, long-read sequencing, optional Hi-C library for scaffolding, and RNA-seq from several tissues to support gene prediction later. RNA-seq data also help to distinguish expressed TEs from genes.
Computing resources
RepeatModeler2 can take many hours to days on a large genome and needs tens of gigabytes of disk for temporary files. Run it on a cluster, with enough threads and a working directory on fast storage.
5. Step 1: build a de novo repeat library with RepeatModeler2
# 1. Create a BLAST-type database (use RMBlast as the engine)
BuildDatabase -name mygenome -engine rmblast genome.fasta
# 2. Run RepeatModeler2; -LTRStruct adds structural LTR detection
RepeatModeler -database mygenome -threads 16 -LTRStruct > repeatmodeler.out 2>&1The main output is mygenome-families.fa (consensus sequences, with classification in the header, for example rnd-1_family-12#LTR/Gypsy) and mygenome-families.stk (seed alignments for curation). Many sequences are annotated as Unknown, which means the program found a repeat family but could not classify it.
Tips:
- Use the same environment, version and database name for all steps.
- Large genomes are sampled in rounds, so results can differ slightly between runs.
- If the run is interrupted, the
-recoverDiroption allows it to resume.
Improve and classify the library
The raw library is a good start, not a finished product.
- Remove false positives: multi-copy host genes (for example gene families, histones, rRNA) are sometimes reported as repeats. BLAST the consensus sequences against a protein database such as UniProt (excluding TE proteins) and remove those that match non-TE proteins, using a tool such as ProtExcluder or manual filtering.
- Classify unknown families: tools such as TEsorter (domain-based), DeepTE and RepeatClassifier assign classes using conserved TE protein domains (reverse transcriptase, integrase, transposase).
- Curate important families manually if the project is about TEs: extend consensus sequences, check boundaries and target-site duplications (see Goubert et al., 2022).
- Add known repeats: combine your library with families for your clade from Dfam (and Repbase if you have a license) for better classification.
# Combine a custom and a curated library
cat mygenome-families.fa dfam_clade_repeats.fa > combined_library.fa6. Step 2: annotate and mask the genome with RepeatMasker
RepeatMasker -lib combined_library.fa -xsmall -gff -pa 8 -dir rm_out genome.fastaKey options:
| Option | Function |
|---|---|
-lib | Custom library (FASTA) of repeat consensus sequences |
-species | Use a Dfam taxonomic group instead of a custom library (for example -species human) |
-xsmall | Soft-masking: repeats become lowercase |
-gff | Also write annotations in GFF format |
-pa | Parallel search processes; with RMBlast each uses about four cores, so -pa can be the number of cores divided by four |
-dir | Output directory |
-s, -q | Sensitivity settings: slow (more sensitive) and quick |
-nolow | Do not mask low-complexity and simple repeats (use when you annotate them separately) |
Outputs
| File | Content |
|---|---|
genome.fasta.masked | Masked genome (soft-masked with -xsmall, hard-masked with N otherwise) |
genome.fasta.out | Table with every repeat copy: position, family, class, divergence |
genome.fasta.tbl | Summary: proportion of the genome by repeat class |
genome.fasta.out.gff | Annotations in GFF format |
genome.fasta.align | Alignments of each hit to the consensus (with -a) |
Soft-masking versus hard-masking
- Soft-masking converts repeats to lowercase while keeping the sequence. Gene predictors such as BRAKER, AUGUSTUS and MAKER understand it and can still use repeat regions where evidence supports a gene (for example a gene inside a TE-rich region or genuine TE-derived genes).
- Hard-masking replaces repeats with N, which destroys information and is rarely needed.
- For read alignment and variant calling, use the unmasked reference and apply the repeat annotation as a BED or GFF filter afterwards. Aligning to a masked reference can produce misleading coverage and mapping.
7. Step 3: tandem repeats and satellites
RepeatMasker annotates simple repeats and low-complexity regions, but dedicated programs describe tandem repeats better:
- Tandem Repeats Finder (TRF) identifies tandem repeats of any period.
- TRASH and similar tools find satellite arrays and centromeric repeats in long-read assemblies.
- SSR markers can be extracted with MISA.
8. Step 4: structural annotation and quality of TE annotation
For high-quality TE annotation, especially in plants and other TE-rich genomes, EDTA (Extensive de novo TE Annotator) combines LTRharvest, LTR_FINDER, TIR-Learner, HelitronScanner and RepeatModeler, filters the results and annotates the whole genome:
EDTA.pl --genome genome.fasta --species others --step all --sensitive 1 --anno 1 --threads 16The LTR Assembly Index (LAI) estimates assembly quality from the completeness of LTR retrotransposons: higher values indicate better repeat assembly (below 10 is draft quality, 10 to 20 is reference quality, above 20 is gold quality).
9. Step 5: interpret the results
Repeat content table
genome.fasta.tbl summarizes how much of the genome is masked and by which class. Check:
- The total masked fraction is plausible for your lineage (compare with relatives).
- The proportion of unclassified repeats. A large share is common in non-model organisms.
- Unexpected classes (for example a huge fraction of rRNA or satellites) that may indicate contamination or misassembly.
Repeat landscapes
A repeat landscape plots, for each TE class, the percentage of the genome by the divergence of each copy from its family consensus (Kimura distance). Copies with low divergence are young (recent activity), whereas high divergence indicates older insertions that accumulated mutations. A peak at low divergence suggests a recent burst of transposition.
# Divergence landscape from the RepeatMasker alignment file
calcDivergenceFromAlign.pl -s genome.divsum genome.fasta.align
createRepeatLandscape.pl -div genome.divsum -g <genome_size_in_bases> > landscape.htmlCompare species
Annotate all genomes with the same pipeline and library strategy, because differences in method can mimic biological differences. Report the number of copies, the genomic proportion and the mean divergence per family.
10. Step 6: use the masked genome for gene prediction
# BRAKER3: soft-masked genome + protein evidence + RNA-seq alignments
braker.pl --genome=genome.fasta.masked --prot_seq=orthodb_proteins.fa \
--bam=rnaseq_sorted.bam --softmasking --workingdir=braker_out --threads 16
# Assess the gene set completeness with BUSCO
busco -i braker_out/braker.aa -l metazoa_odb10 -m proteins -c 8 -o busco_proteinsOther widely used annotation pipelines include MAKER, Funannotate and GALBA. After prediction, remove or flag gene models that are TE-derived (by matching them to TE protein domains or to the repeat annotation), then check the number of genes, mean gene length, exon counts and BUSCO scores against close relatives.
11. Special considerations for non-model organisms
- Public libraries (Dfam, Repbase) contain few families for many invertebrates, so a de novo library is essential, and a large fraction of repeats will remain unclassified.
- Use close relatives to estimate expectations, and validate a few abundant families by BLAST against TE proteins and by manual inspection of the alignments.
- Genomes with high heterozygosity or collapsed repeats require care: purge duplicated haplotypes (purge_dups) before repeat annotation.
- Annotate the mitochondrial genome separately and remove it from the nuclear repeat analysis.
- Expression data can show which TEs are active.
12. Troubleshooting: symptoms, causes and fixes
| Symptom | Likely cause | What to do |
|---|---|---|
| RepeatModeler fails at the start | Wrong database name or missing BuildDatabase, missing dependencies (RMBlast, RepeatScout, RECON, LTR tools) | Check the install, create the database first and read the log |
| RepeatModeler runs for days or runs out of disk | Very large genome, too few threads, temporary files on slow storage | Use more threads and fast scratch space, and use -recoverDir after interruptions |
| RepeatMasker reports no repeats | No -lib or -species specified, or Dfam partition missing | Provide a library, install the Dfam database partition needed for your taxon |
| Almost the whole genome is masked | Library contains multi-copy genes or contaminants, or satellite expansion | BLAST the library against proteins, remove non-TE families, check assembly for contamination |
| Very low masked fraction in a genome known to be repeat-rich | Library too small or too stringent, collapsed assembly | Add a de novo library, run with a more sensitive setting, check the assembly quality |
Large fraction labeled Unknown | Poor classification of divergent families | Use TEsorter or DeepTE and domain-based classification, and report the unknown fraction |
| Genes disappear after masking | Hard-masking, or TE-like gene families in the library | Use soft-masking, filter the library with protein databases, check BUSCO |
| Output has all N where lowercase was expected | -xsmall not used | Rerun with -xsmall |
| Errors about sequence names | Long or special characters in headers | Shorten names (for example to scaffold_001) before running |
-pa set high but the run is slow | Each RMBlast process uses several cores | Set -pa to about the number of cores divided by four |
| BRAKER warns about lowercase letters | Soft-masked sequence is used without --softmasking | Add --softmasking when soft-masked input is used |
| Different repeat content in related species | Different assembly quality, different libraries | Reannotate all with the same pipeline and compare with LAI |
| Mitochondrial or contaminant contigs appear as repeats | Contamination | Remove them before repeat analysis |
13. Reporting and reproducibility
- Report the genome version, tools and versions (RepeatModeler, RepeatMasker, RMBlast, TRF, Dfam and Repbase releases), library composition, parameters, and the percentage of the genome masked per class.
- Deposit the repeat library (FASTA) and the GFF annotation in a public repository with the genome.
- Keep the commands in a workflow file (Snakemake or Nextflow).
14. Best-practice checklist
- Start from the best assembly available and remove contaminants.
- Build a de novo library with RepeatModeler2 (include
-LTRStruct). - Filter false positives (multi-copy genes) and classify unknown families.
- Combine with known repeat families for the clade.
- Mask with RepeatMasker using soft-masking.
- Check the masked fraction, landscape and unknown fraction against related species.
- Use the soft-masked genome for gene prediction, and the unmasked genome for read mapping.
- Annotate all compared genomes with the same pipeline.
15. Frequently asked questions
Do I need to mask repeats before BLAST or alignment? For gene prediction, yes. For read mapping, no. Use the unmasked reference.
Can I use only Dfam instead of a de novo library? It works for well-studied groups (mammals, many model organisms). For poorly studied taxa a de novo library is necessary.
What percentage of repeats is normal? It depends strongly on the lineage and genome size, so compare with close relatives.
Is RepeatMasker free? The software is open source. The Dfam database is free, while Repbase requires a license for recent versions.
How do I detect active TEs? Use low-divergence copies, full-length copies with intact protein domains, and expression evidence from RNA-seq and small RNA data.
16. References
- Flynn JM et al. (2020). RepeatModeler2 for automated genomic discovery of transposable element families. Proceedings of the National Academy of Sciences 117:9451-9457.
- Storer J et al. (2021). The Dfam community resource of transposable element families, sequence models, and genome annotations. Mobile DNA 12:2.
- Wicker T et al. (2007). A unified classification system for eukaryotic transposable elements. Nature Reviews Genetics 8:973-982.
- Tarailo-Graovac M, Chen N (2009). Using RepeatMasker to identify repeats in the genome. Current Protocols in Bioinformatics 25:4.10.
- Bao W, Kojima KK, Kohany O (2015). Repbase Update, a database of repetitive elements in eukaryotic genomes. Mobile DNA 6:11.
- Ou S, Jiang N (2018). LTR_retriever: a highly accurate and sensitive program for identification of long terminal repeat retrotransposons. Plant Physiology 176:1410-1422.
- Ou S et al. (2018). Assessing genome assembly quality using the LTR Assembly Index (LAI). Nucleic Acids Research 46:e126.
- Ou S et al. (2019). Benchmarking transposable element annotation methods for creation of a streamlined, comprehensive pipeline. Genome Biology 20:275.
- Baril T, Galbraith J, Hayward A (2024). Earl Grey: a fully automated user-friendly transposable element annotation and analysis pipeline. Molecular Biology and Evolution 41:msae068.
- Benson G (1999). Tandem repeats finder: a program to analyze DNA sequences. Nucleic Acids Research 27:573-580.
- Goubert C et al. (2022). A beginner's guide to manual curation of transposable elements. Mobile DNA 13:7.
- Novak P et al. (2013). RepeatExplorer: a Galaxy-based web server for genome-wide characterization of eukaryotic repetitive elements from next-generation sequence reads. Bioinformatics 29:792-793.
- Goubert C et al. (2015). De novo assembly and annotation of the Asian tiger mosquito (Aedes albopictus) repeatome with dnaPipeTE from raw genomic reads and comparative analysis with the yellow fever mosquito (Aedes aegypti). Genome Biology and Evolution 7:1192-1205.
- Gabriel L et al. (2024). BRAKER3: fully automated genome annotation using RNA-seq and protein evidence with GeneMark-ETP, AUGUSTUS, and TSEBRA. Genome Research 34:769-777.
- Manni M et al. (2021). BUSCO update: novel and streamlined workflows along with broader and deeper phylogenetic coverage for scoring of eukaryotic, prokaryotic, and viral genomes. Molecular Biology and Evolution 38:4647-4654.
💬 Commentaires