Outgroup (cladistics)

In cladistics or phylogenetics, an outgroup^[1] is a more distantly related group of organisms that serves as a reference group when determining the evolutionary relationships of the ingroup, the set of organisms under study, and is distinct from sociological outgroups. The outgroup is used as a point of comparison for the ingroup and specifically allows for the phylogeny to be rooted. Because the polarity (direction) of character change can be determined only on a rooted phylogeny, the choice of outgroup is essential for understanding the evolution of traits along a phylogeny.^[2]

Although the concept of outgroups has been in use from the earliest days of cladistics, the term "outgroup" is thought to have been coined in the early 1970s at the American Museum of Natural History.^[3] Prior to the advent of the term, various other terms were used by evolutionary biologists, including "exgroup", "related group", and "outside groups".^[3]

The chosen outgroup is hypothesized to be less closely related to the ingroup than the ingroup is related to itself. The evolutionary conclusion from these relationships is that the outgroup species has a common ancestor with the ingroup that is older than the common ancestor of the ingroup. Choice of outgroup can change the topology of a phylogeny.^[4] Therefore, phylogeneticists typically use more than one outgroup in cladistic analysis. The use of multiple outgroups is preferable because it provides a more robust phylogeny, buffering against poor outgroup candidates and testing the ingroup's hypothesized monophyly.^[3][5][6]

To qualify as an outgroup, a taxon must satisfy the following two characteristics:

• It must not be a member of the ingroup.
• It must be related to the ingroup, closely enough for meaningful comparisons to the ingroup.

Therefore, an appropriate outgroup must be unambiguously outside the clade of interest in the phylogenetic study. An outgroup that is nested within the ingroup will, when used to root the phylogeny, result in incorrect conclusions about phylogenetic relationships and trait evolution.^[7] However, the optimal level of relatedness of the outgroup to the ingroup depends on the depth of phylogenetic analysis. Choosing a closely related outgroup relative to the ingroup is more useful when looking at subtle differences, while choosing an unduly distant outgroup can result in mistaking convergent evolution for a direct evolutionary relationship due to a common ancestor.^[8][9] For shallow phylogenetics—for example, resolving the evolutionary relationships of a clade within a genus—an appropriate outgroup would be a member of the sister clade.^[10] However, for deeper phylogenetic analysis, less closely related taxa can be used. For example, Jarvis et al. (2014) used humans and crocodiles as outgroups while resolving the early branches of the avian phylogeny.^[11] In molecular phylogenetics, satisfying the second requirement typically means that DNA or protein sequences from the outgroup can be successfully aligned to sequences from the ingroup. Although there are algorithmic approaches to identify the outgroups with maximum global parsimony, they are often limited by failing to reflect the continuous, quantitative nature of certain character states.^[12] Character states are traits, either ancestral or derived, that affect the construction of branching patterns in a phylogenetic tree.^[13]

In each example, a phylogeny of organisms in the ingroup may be rooted by scoring the same character states for one or more members of the outgroup.

• Apomorphy, a derived trait of an organism
• Sister group, a group that may be closely related to an ingroup
• Plesiomorphy, an ancestral trait of an organism
• Primitive (phylogenetics), a term for ancestral traits