Speciation and Reproductive Isolation

The Species Problem

Darwin called his book On the Origin of Species and spent almost none of it explaining how species originate. He explained natural selection, descent with modification, the pattern of diversification. He did not have a satisfying account of how one lineage becomes two. Ernst Mayr pointed this out in the 1940s and spent much of his career fixing the gap.

The difficulty starts with defining what a species is. The intuitive answer — a species is a group of organisms that look alike — fails immediately. Sibling species are morphologically indistinguishable but genetically distinct and unable to interbreed; lumping them violates everything meaningful about the category. Polymorphic species contain wildly divergent morphs that look like different species; splitting them produces absurdities. Different color morphs of the same bird, sexual dimorphism, developmental stages — none of these are separate species, but morphology alone can’t tell you that.

Mayr’s Biological Species Concept cut through this: a species is a group of actually or potentially interbreeding populations that is reproductively isolated from other such groups. The definition is relational — species status is defined by the presence or absence of barriers to gene flow, not by appearance. It handles the sibling species problem (different species because reproductively isolated) and the polymorphism problem (same species because gene flow continues within the population).

The Biological Species Concept has its own limitations — it doesn’t apply well to asexual organisms, to fossils, or to populations that are geographically separated and untested. Alternative species concepts (the Phylogenetic Species Concept, the Ecological Species Concept) address some limitations while introducing others. The species problem hasn’t been fully resolved. But the reproductive isolation framework remains the most useful for understanding how speciation works.

Allopatric Speciation — The Standard Model

The dominant model for how new species form is allopatric speciation: geographic separation followed by divergence. A population is split by a barrier — a mountain range, a river, a glacier, a sea channel — and the two daughter populations, no longer exchanging genes, diverge independently. Natural selection acts on local conditions in each daughter population, favoring different traits in different environments. Random genetic drift changes allele frequencies differently in each isolated population. Mutations that arise in one population don’t spread to the other. Over time, the two populations diverge genetically, morphologically, and behaviorally.

Whether divergence produces reproductive isolation — and whether that isolation is maintained when and if the barrier is removed — depends on how much divergence has accumulated and in which traits. If the two populations are re-contacted while still early in divergence, they may hybridize freely and merge back into a single gene pool. If they’ve diverged far enough that hybrids are less fit, or that mate recognition prevents mating in the first place, reproductive isolation will be maintained even when the geographic barrier is gone. The populations have become separate species.

Darwin’s finches on the Galapagos are the textbook allopatric radiation. Ancestral finches colonized the archipelago from the mainland, probably multiple times. Each island population evolved in response to the resources available there — different bill sizes and shapes for different seed sizes, insect types, and food sources. With limited inter-island gene flow, divergence accumulated. Populations that later expanded into overlapping ranges were divergent enough that hybridization was rare or produced less fit offspring. Thirteen species (by current taxonomy) from a single ancestor in roughly a million years.

Sympatric Speciation — The Difficult Case

If geographic separation is the easy explanation for how populations split, sympatric speciation is the hard case: species diverging within the same geographic range, without a physical barrier to gene flow. For decades, most evolutionary biologists doubted sympatric speciation was a significant source of new species. The logic: any mutation or behavioral variant that reduced gene flow with one subgroup would be immediately diluted by gene flow with the rest of the population. How could reproductive isolation evolve in a panmictic population?

The theoretical answer came from disruptive selection. If a population occupies a habitat with two distinct resource types — say, large seeds and small seeds — and individuals with extreme phenotypes (large bills, small bills) outperform intermediates, then intermediates are disadvantaged. Selection favors assortative mating between similar phenotypes — large-billed individuals should preferentially mate with other large-billed individuals, because this is the type that leaves more descendants in the large-seed niche. If assortative mating on the ecologically relevant trait evolves, gene flow between the two ecotypes is reduced without geographic separation. Over time, the two ecotypes can diverge into species.

The most convincing empirical case is the apple maggot fly Rhagoletis pomonella. Historically a hawthorn specialist, populations in North America shifted to introduced domestic apple trees in the nineteenth century. The apple and hawthorn host races are now genetically distinguishable, mate preferentially on their own host plant species, and are adapted to the different fruiting times of their hosts. This is an early-stage sympatric speciation event observed in real time.

Cichlid fish in African lakes — Lake Victoria, Lake Malawi, Lake Tanganyika — have produced hundreds of species within single water bodies over what is, geologically speaking, almost no time. Lake Victoria is approximately fifteen thousand years old by some estimates, dry or nearly dry during the last glaciation, and now contains at least five hundred cichlid species. The radiation cannot be primarily allopatric; the lake’s geography doesn’t support that many separate refugia. Sexual selection on male coloration — females choose mates by color, and color varies widely across cichlid species — appears to have driven rapid reproductive isolation in sympatry.

Reinforcement — Selection Against Hybridization

A third mechanism for building reproductive isolation operates when two partially diverged populations come back into secondary contact. If hybrids between the populations are less fit — they’re intermediate between two ecotypes, adapted to neither habitat fully, or produce offspring with developmental problems from genetic incompatibility — then selection favors individuals who don’t hybridize. Individuals that preferentially mate within their own population leave more fit offspring than those that hybridize. Mate discrimination, or assortative mating based on population of origin, is selected for. Reproductive isolation is reinforced by selection acting directly on it.

The signature of reinforcement is stronger mate discrimination in sympatric populations (where the two groups overlap and the selection pressure to avoid hybridization is present) than in allopatric populations (where the two groups don’t encounter each other and no such selection operates). This pattern has been documented in several systems, including Drosophila species that hybridize in some geographic zones and in plants with overlapping ranges.

Polyploidy — Instant Speciation

In plants and some animals, species can arise nearly instantaneously through polyploidy: the doubling or multiplication of the entire chromosome set. A hybrid between two species that would normally be sterile (because the chromosomes from the two parent species can’t pair correctly in meiosis) can become fertile if its chromosomes are doubled. The doubled hybrid — an allopolyploid — can mate with itself or with other allopolyploids of the same type, but it’s reproductively isolated from both parent species, because crosses with either parent produce aneuploid offspring.

Common wheat (Triticum aestivum) is a hexaploid — it has six sets of chromosomes, three from each of two ancestral species, one of which was itself a hybrid. Bread wheat arose through two hybridization and polyploidy events, the most recent of which happened within the last ten thousand years. It is reproductively isolated from its parent species and is, by the Biological Species Concept, a distinct species — despite having originated in what amounts to an evolutionary instant.

Roughly half of all flowering plant species are estimated to have polyploidy somewhere in their history. In plants, polyploidy is a major mechanism of speciation. In animals, it’s rarer but not absent — many fish and amphibian species are polyploid.

What Speciation Research Has Resolved

Darwin’s difficulty with speciation was partly definitional — without a clear species concept, the question of how species originate doesn’t have a well-formed answer. Mayr’s reproductive isolation framework resolved the conceptual problem. Subsequent decades of work filled in the mechanisms: allopatry as the default, sympatry as real but requiring specific conditions, reinforcement as selection that can complete speciation, polyploidy as an occasionally instantaneous route.

What remains unclear is the relative contribution of each mechanism in the overall history of biodiversity. How much of the roughly ten million species currently estimated to exist on Earth originated allopatrically versus sympatrically? What fraction of speciation events involved polyploidy? How often is speciation the gradual process the Modern Synthesis assumed versus a rapid event driven by chromosome rearrangements, polyploidy, or rapid sexual selection?

The answers are coming from phylogenomics — the analysis of whole-genome sequences across hundreds of species simultaneously, which can reconstruct speciation events with a precision that wasn’t possible when the Modern Synthesis was built. The picture is messier than the textbook account: more hybridization between “good species” than expected, more reticulate evolution (multiple lineages fusing), more rapid speciation in some groups. The core mechanisms are confirmed; the proportions are still being sorted out.