Birdsong3D
Birdsong3D explores what neural networks learn when they listen to birds.
We collected more than 6,000 freely licensed recordings of 118 North American and European bird species, shared by over a thousand contributors.1,2 We then trained a variational autoencoder (VAE),3 a kind of AI that squeezes each song into a handful of numbers and then rebuilds it from those numbers alone.
Those numbers place every song in a space, where songs that sound alike sit close together. Here you can fly through that space. Each song appears as a 3D shape you can see and hear, and in the gaps between real recordings, the model imagines songs no bird has ever sung.
The 3D model of each bird’s body is generated parametrically from its species’ measurements in AVONET.4 An imagined bird’s model is shaped by the distance of points in the latent space.
Explore
Move through what the model learned in 3D, arranged by bird or by song. On a computer, click to go somewhere and use W A S D to move. On a phone, tap to go somewhere and drag to move.
Research
The model can generate new sounds, but only people can say what they sound like in everyday words, like buzzy, whistled or a chirp. Participate in this research by adding your answers to our data.
Sources
- iNaturalist, research-grade sound observations, under CC0, CC BY and CC BY-SA. inaturalist.org. Each recording is credited where it plays.
- Xeno-canto, openly licensed bird recordings. xeno-canto.org.
- Kingma, D. P. and Welling, M. (2014). Auto-encoding variational Bayes. ICLR. arXiv:1312.6114.
- Tobias, J. A. et al. (2022). AVONET: morphological, ecological and geographical data for all birds. Ecology Letters 25(3), 581–597. doi:10.1111/ele.13898. The birds' bodies.
- Griffin, D. and Lim, J. (1984). Signal estimation from modified short-time Fourier transform. IEEE Transactions on Acoustics, Speech, and Signal Processing 32(2), 236–243. Rebuilds sound from a spectrogram, with a metallic ring; it plays while the vocoder loads.
- The eBird/Clements Checklist of Birds of the World, from the eBird taxonomy. Used to check imagined birds' names against real ones.
- Pieplow, N. (2017, 2019). Peterson Field Guide to Bird Sounds of Eastern and Western North America. The words for sounds.
- Siuzdak, H. (2023). Vocos: Closing the gap between time-domain and Fourier-based neural vocoders for high-quality audio synthesis. arXiv:2306.00814. Our vocoder follows its design, trained on these recordings to turn imagined sounds into audio.