Data
Birdsong3D models are trained on open, community-created data.
Where data comes from
- iNaturalist Recordings
- Research-grade sound observations, limited to recordings licensed CC0, CC BY or CC BY-SA. Each recording's recordist and license are kept, and shown wherever it plays.
- inaturalist.org
- Xeno-canto Recordings
- CC BY-SA recordings only, as a smaller supplement. Most of the archive is licensed non-commercial or no-derivatives, which we excluded.
- xeno-canto.org
- AVONET Body measurements
- Species averages of eight traits measured on museum specimens. They set the size of each bird's 3D body.
- Tobias, J. A. et al. (2022). AVONET: morphological, ecological and geographical data for all birds. Ecology Letters 25(3), 581–597. doi:10.1111/ele.13898. CC BY 4.0.
- eBird/Clements Checklist Names
- English and scientific names of the world's birds, from the eBird taxonomy. Used to check imagined birds' names against real ones.
- Cornell Lab of Ornithology, The eBird/Clements Checklist of Birds of the World. birds.cornell.edu/clementschecklist
- Nathan Pieplow Sound vocabulary
- Terms for a sound's pattern, pitch and tone. We measure each phrase against his definitions, using thresholds we set ourselves. We don't use any text or audio from the guides.
- Pieplow, N. (2017, 2019). Peterson Field Guide to Bird Sounds of Eastern and Western North America. Houghton Mifflin Harcourt.
All the species
| Species | Recordings | Phrases |
|---|
From recordings to phrases
The first step to training a model is cleaning the data.
- Filter each recording to the bird's own pitch range, and reduce wind, traffic and other noise.
- Find the moments where the bird is louder than the background.
- Cut up to six phrases of seconds from those moments.
- Turn each phrase into a spectrogram: pitch bands from , across steps in time, with the bands spaced as people hear pitch (the mel scale; Stevens, Volkmann & Newman 1937). We draw it as 3D terrain, with time along one side, pitch along the other and loudness as height. This is what the model learns from.
How the model learns
The model is a variational autoencoder (VAE; Kingma & Welling 2014). It learns to compress each phrase into a few numbers, and to rebuild the phrase from those numbers alone.
In training, the model rebuilds each phrase and is adjusted according to how far the rebuilt phrase is from the original. This runs over the whole dataset many times. A second term keeps the numbers close together, so that points between real phrases also rebuild into bird-like sounds.
The models
We trained and tested several models, and settled on two. One compresses each phrase into 3 numbers, so its space can be shown exactly as the model learned it. The other uses 32 numbers. It rebuilds phrases about twice as accurately and keeps species further apart, so its imagined birds are more convincing.
| Model | Numbers | Rebuild error | Same-species neighbours |
|---|---|---|---|
| z3 | 3 | 0.022 | 8.4% (chance 2.3%) |
| z32 | 32 | 0.011 | 26.3% |
Both keep pitch well but blur the timing of notes. We're testing versions of each that keep timing too, with results to come.
What it generates
At the model's narrowest point, each phrase is a short list of numbers, z. The space of all possible z is the latent space.
To draw the space, we show it in three dimensions. z3 already has three. For z32 we use the three directions in which phrases vary most (principal component analysis; Pearson 1901, Hotelling 1933). Each phrase is a point, and phrases that sound alike sit close together.
The model can rebuild a sound from any point, not only from points where a recording sits. We call the sounds from empty points imagined, because no bird made them. The model outputs a spectrogram, which can't be played as it is. Griffin-Lim (Griffin & Lim 1984) can rebuild sound from it, but leaves a metallic ring, so we trained a second network, a vocoder, to turn spectrograms into sound (Siuzdak 2023). Griffin-Lim plays while the vocoder loads.
A space for imagination
The model is never told which species a phrase belongs to. In z32, a vote of a phrase's five nearest neighbours names its species 26% of the time, against 2.3% by chance.
The model is trained to only generate sounds. An imagined bird's body and name come from the way these sounds (real and imagined) sit in the latent space.
Bodies
Every bird's body is drawn from its species' measurements in AVONET:
- Mass sets the size of the body.
- Beak length and depth set the beak.
- Wing length sets the wing, and Kipp's distance how pointed it is.
- Tail length sets the tail.
- Tarsus length sets the legs.
Each family also has a body plan, set by hand: posture, head size, neck, beak hook, how stout the body is, and whether there's a crest or a forked tail. An imagined bird mixes the measurements and plans of the species it sits between, in the same proportions as its sound. You can see every species' body, and blend any two, on Bodies.
Names
Each imagined bird gets a common name and a Latin name. Every word comes from the species it blends or from a measurement. Nothing comes from plumage, which the model doesn't know.
- Genus. If one species makes up 60% or more of the blend, the bird keeps that species' genus. Otherwise it gets a new genus made from its parents' genera, such as Cardipoecile.
- Common name. The main parent's first word joins the other parent's group noun, as in the real Wrentit. A Chipping Sparrow and a Carolina Wren make a Chippingwren.
- Description. One word for the trait that stands out most. For now, the Latin name describes the body (minor for small), and the common name the sound (Trilling, Hooting).
The model places each sound among real birds, but it has no way to say what the sound is like to a listener, or what a person would call the bird. That is what we ask visitors to do when they name birds and describe sounds.