Data

Birdsong3D models are trained on open, community-created data.

Where data comes from

iNaturalist Recordings
Research-grade sound observations, limited to recordings licensed CC0, CC BY or CC BY-SA. Each recording's recordist and license are kept, and shown wherever it plays.
inaturalist.org
Xeno-canto Recordings
CC BY-SA recordings only, as a smaller supplement. Most of the archive is licensed non-commercial or no-derivatives, which we excluded.
xeno-canto.org
AVONET Body measurements
Species averages of eight traits measured on museum specimens. They set the size of each bird's 3D body.
Tobias, J. A. et al. (2022). AVONET: morphological, ecological and geographical data for all birds. Ecology Letters 25(3), 581–597. doi:10.1111/ele.13898. CC BY 4.0.
eBird/Clements Checklist Names
English and scientific names of the world's birds, from the eBird taxonomy. Used to check imagined birds' names against real ones.
Cornell Lab of Ornithology, The eBird/Clements Checklist of Birds of the World. birds.cornell.edu/clementschecklist
Nathan Pieplow Sound vocabulary
Terms for a sound's pattern, pitch and tone. We measure each phrase against his definitions, using thresholds we set ourselves. We don't use any text or audio from the guides.
Pieplow, N. (2017, 2019). Peterson Field Guide to Bird Sounds of Eastern and Western North America. Houghton Mifflin Harcourt.
All the species
SpeciesFamilyRegionRecordingsPhrases

From recordings to phrases

The first step to training a model is cleaning the data.

  1. Filter each recording to the bird's own pitch range, and reduce wind, traffic and other noise.
  2. Find the moments where the bird is louder than the background.
  3. Cut up to six phrases of seconds from those moments.
  4. Turn each phrase into a spectrogram: pitch bands from , across steps in time, with the bands spaced as people hear pitch (the mel scale; Stevens, Volkmann & Newman 1937). We draw it as 3D terrain, with time along one side, pitch along the other and loudness as height. This is what the model learns from.

How the model learns

The model is a variational autoencoder (VAE; Kingma & Welling 2014). It learns to compress each phrase into a few numbers, and to rebuild the phrase from those numbers alone.

How Birdsong3D works A recording becomes 3D terrain, which the model compresses into a handful of numbers that place it in the latent space. Any point there, including the gaps between recordings, can be rebuilt into a song. A recording Its sound as 3D terrain A handful of numbers The latent space A song from a gap compress place rebuild A recording As 3D terrain A few numbers The latent space A song from a gap

In training, the model rebuilds each phrase and is adjusted according to how far the rebuilt phrase is from the original. This runs over the whole dataset many times. A second term keeps the numbers close together, so that points between real phrases also rebuild into bird-like sounds.

a real phrase compress a few numbers rebuild its rebuilt phrase compare the two · adjust the model a little · again

The models

We trained and tested several models, and settled on two. One compresses each phrase into 3 numbers, so its space can be shown exactly as the model learned it. The other uses 32 numbers. It rebuilds phrases about twice as accurately and keeps species further apart, so its imagined birds are more convincing.

ModelNumbersRebuild errorSame-species neighbours
z330.0228.4% (chance 2.3%)
z32320.01126.3%

Both keep pitch well but blur the timing of notes. We're testing versions of each that keep timing too, with results to come.

What it generates

At the model's narrowest point, each phrase is a short list of numbers, z. The space of all possible z is the latent space.

To draw the space, we show it in three dimensions. z3 already has three. For z32 we use the three directions in which phrases vary most (principal component analysis; Pearson 1901, Hotelling 1933). Each phrase is a point, and phrases that sound alike sit close together.

The model can rebuild a sound from any point, not only from points where a recording sits. We call the sounds from empty points imagined, because no bird made them. The model outputs a spectrogram, which can't be played as it is. Griffin-Lim (Griffin & Lim 1984) can rebuild sound from it, but leaves a metallic ring, so we trained a second network, a vocoder, to turn spectrograms into sound (Siuzdak 2023). Griffin-Lim plays while the vocoder loads.

A space for imagination

The model is never told which species a phrase belongs to. In z32, a vote of a phrase's five nearest neighbours names its species 26% of the time, against 2.3% by chance.

The model is trained to only generate sounds. An imagined bird's body and name come from the way these sounds (real and imagined) sit in the latent space.

Bodies

Every bird's body is drawn from its species' measurements in AVONET:

Each family also has a body plan, set by hand: posture, head size, neck, beak hook, how stout the body is, and whether there's a crest or a forked tail. An imagined bird mixes the measurements and plans of the species it sits between, in the same proportions as its sound. You can see every species' body, and blend any two, on Bodies.

Names

Each imagined bird gets a common name and a Latin name. Every word comes from the species it blends or from a measurement. Nothing comes from plumage, which the model doesn't know.

The model places each sound among real birds, but it has no way to say what the sound is like to a listener, or what a person would call the bird. That is what we ask visitors to do when they name birds and describe sounds.