Efficient pollen identification – Interdisciplinary research team combines image-based particle analysis with artificial intelligence
From pollen forecasting, honey analysis and climate-related changes in plant-pollinator interactions, analysing pollen plays an important role in many areas of research. Microscopy is still the gold standard, but it is very time consuming and requires considerable expertise. In cooperation with Technische Universität (TU) Ilmenau, scientists from the Helmholtz Centre for Environmental Research (UFZ) and the German Centre for Integrative Biodiversity Research (iDiv) have now developed a method that allows them to efficiently automate the process of pollen analysis. Their study has been published in the specialist journal New Phytologist.
Pollen is produced in a flower’s stamens and consists of a multitude of minute pollen grains, which contain the plant’s male genetic material necessary for its reproduction. The pollen grains get caught in the tiny hairs of nectar-feeding insects as they brush past and are thus transported from flower to flower. Once there, in the ideal scenario, a pollen grain will cling to the sticky stigma of the same plant species, which may then result in fertilisation. “Although pollinating insects perform this pollen delivery service entirely incidentally, its value is immeasurably high, both ecologically and economically,” says Dr. Susanne Dunker, head of the working group on imaging flow cytometry at the Department for Physiological Diversity at UFZ and iDiv. “Against the background of climate change and the accelerating loss of species, it is particularly important for us to gain a better understanding of these interactions between plants and pollinators.” Pollen analysis is a critical tool in this regard.
Each species of plant has pollen grains of a characteristic shape, surface structure and size. When it comes to identifying and counting pollen grains – measuring between 10 and 180 micrometres – in a sample, microscopy has long been considered the gold standard. However, working with a microscope requires a great deal of expertise and is very time-consuming. “Although various approaches have already been proposed for the automation of pollen analysis, these methods are either unable to differentiate between closely related species or do not deliver quantitative findings about the number of pollen grains contained in a sample,” continues UFZ biologist Dr. Dunker. Yet it is precisely this information that is critical to many research subjects, such as the interaction between plants and pollinators.
In their latest study, Susanne Dunker and her team of researchers have developed a novel method for the automation of pollen analysis. To this end they combined the high throughput of imaging flow cytometry – a technique used for particle analysis – with a form of artificial intelligence (AI) known as deep learning to design a highly efficient analysis tool, which makes it possible to both accurately identify the species and quantify the pollen grains contained in a sample. Imaging flow cytometry is a process that is primarily used in the medical field to analyse blood cells but is now also being repurposed for pollen analysis. “A pollen sample for examination is first added to a carrier liquid, which then flows through a channel that becomes increasingly narrow,” says Susanne Dunker, explaining the procedure. “The narrowing of the channel causes the pollen grains to separate and line up as if they are on a string of pearls, so that each one passes through the built-in microscope element on its own and images of up to 2,000 individual pollen grains can be captured per second.” Two normal microscopic images are taken plus ten fluorescence microscopic images per grain of pollen. When excited with light radiated at certain wavelengths by a laser, the pollen grains themselves emit light. “The area of the colour spectrum in which the pollen fluoresces – and at which precise location – is sometimes very specific. This information provides us with additional traits that can help identify the individual plant species,” reports Susanne Dunker. In the deep learning process, an algorithm works in successive steps to abstract the original pixels of an image to a greater and greater degree in order to finally extract the species-specific characteristics. “Microscopic images, fluorescence characteristics and high throughput have never been used in combination for pollen analysis before – this really is an absolute first.” Where the analysis of a relatively straightforward sample takes, for example, four hours under the microscope, the new process takes just 20 minutes. UFZ has therefore applied for a patent for the novel high-throughput analysis method, with its inventor, Susanne Dunker, receiving the UFZ Technology Transfer Award in 2019.
The pollen samples examined in the study came from 35 species of meadow plants, including yarrow, sage, thyme and various species of clover such as white, mountain and red clover. In total, the researchers prepared around 430,000 images, which formed the basis for a data set. In cooperation with TU Ilmenau, this data set was then transferred using deep learning into a highly efficient tool for pollen identification. In subsequent analyses, the researchers tested the accuracy of their new method, comparing unknown pollen samples from the 35 plant species against the data set. “The result was more than satisfactory – the level of accuracy was 96 per cent,” says Susanne Dunker. Even species that are difficult to distinguish from one another, and indeed present experts with a challenge under the microscope, could be reliably identified. The new method is therefore not only extremely fast but also highly precise.
In the future, the new process for automated pollen analysis will play a key role in answering critical research questions about interactions between plants and pollinators. How important are certain pollinators like bees, flies and bumblebees for particular plant species? What would be the consequences of losing a species of pollinating insect or a plant? “We are now able to evaluate pollen samples on a large scale, both qualitatively and- at the same time – quantitatively. We are constantly expanding our pollen data set of insect-pollinated plants for that purpose,” comments Susanne Dunker. She aims to expand the data set to include at least those 500 plant species whose pollen is significant as a food source for honeybees.
Publication:
Susanne Dunker, Elena Motivans, Demetra Rakosy, David Boho, Patrick Mäder, Thomas Hornick, Tiffany M. Knight: Pollen analysis using multispectral imaging flow cytometry and deep learning. New Phytologist https://doi.org/10.1111/nph.16882
„The only things you want to conserve are the things you know”
Jana Wäldchen and her team from the Max Planck Institute for Biogeochemistry have played a key role in developing the plant identification app, Flora Incognita. We discussed with her how being able to identify different plants contributes to species diversity, which plant species are particularly under threat and how non-native species are suppressing local plants.
What role do Citizen Science projects such as Flora Incognita play in protecting the variety of species?
Projects such as Flora Incognita that involve the general public play two important roles. On the one hand, they simplify the identification process. Anyone who is interested in plants can now easily, quickly and fairly precisely put a name to an unknown species. This means that more attention is paid to plant variety and that people become more aware of nature and the need to protect it.
Naturally, documenting the variety of plant species also makes an important contribution. As a result, scientists and nature conservation authorities also benefit from the app. Thanks to the identified species and their location, extremely valuable data records can be created that provide information that is of relevance for research into species protection and biodiversity. In the long term, the data from the Flora Incognita app will make it possible to find new answers to questions such as: When do certain species flower, and where? How widely to the properties of a single plant species vary? How are the composition and locations of the plants shifting in response to climate change and the type of land use?
The Flora Incognita app has been in use for two years. What has changed during that time?
The goal of the project is to make it easier for people to identify plants, and in this way, to increase their awareness of the wide variety of species around them. We have received many emails and comments from users that confirm that we’re taking the right approach towards achieving this goal. We’re not just getting feedback about how easy it is to identify plants. Many users also write that this easy identification has broadened their view of the variety of species.
Comments such as “At last, we’re not just ‘blindly’ walking through the forest!” or “This is a fun way of finding out more about the environment” show that the app is making an important contribution towards raising awareness of plant diversity. It is very satisfying to have achieved this goal. Plant identification plays an important part in species protection, since we only want to conserve the things that we know.
How does the app make plant identification easier?
It’s a situation we’re all familiar with. While out walking, you spot a plant that you wouldlike to know more about. What is the plant called, is it poisonous, or might it be a protected species? For laypersons, the standard identification books are difficult to understand; identifying the plants takes a lot of time and is usually difficult to do because so many specialist terms are involved. This presents a major hurdle for people who are interested in identifying plants and who want to find out more about them.
Picture guides have made it easier to identify the most common species, but either we don’t always take them with us, or they don’t cover the entire range of plant varieties. Our Flora Incognita app allows users to identify plants quickly, easily and fairly precisely. Using the camera in your smartphone, you take a snapshot of the flower, and possibly also the leaves, and in just a few seconds, the suggested name of the plant is shown. Users are also given additional information such as its local protection status, important plant features, areas where it is found and information about similar species with which it is easily confused.
How often has the app been downloaded? How many plants have already been identified using the app?
The app has been downloaded over a million times. It has already been used to identify more than eight million plants. To date, Flora Incognita has identified around 4,600 different species.
Can the app be used to draw conclusions about plant variety in specific ecosystems?
The plant observations collected using the app are not systematic and only reflect what the users see in nature, and what attracts their interest. This means that we only get information about which species are growing in specific areas. This does not enable us to draw conclusions as to whether certain species are not present. Those species that are common and striking in appearance are identified more often than the rarer, less spectacular ones.
Even so, after just two vegetation periods, we can determine similar patterns of spread for certain species to those identified in floristic, systematic mapping for the whole of Germany. This illustrates the potential for the app to supplement floristic mapping in the long term. We are very confident that this autumn, we will have collected sufficient observations to allow us to make more detailed statements about plant variety in different regions.
However, we can now already say that most identifications are made in urban areas, in the places very close to where people live. The species that are photographed most often are nitrogen indicators such as yarrow (Achillea millefolium agg.), dandelions (Taraxacum), ground ivy (Glechoma hederacea agg.) or garlic mustard (Alliaria petiolata), which mainly grow in anthropogenic habitats. At the same time, we have also noticed that people often use the app when they are away on holiday. For example, in 2019, observation figures were higher in relation to the size of the population on the North and Baltic Sea coasts and in the Alpine region than in other areas.
Which plant species are particularly under threat?
In Germany, nearly a third of native wild species are endangered. This is the figure given in the endangered species list of ferns and flowering plants, mosses and algae, which was published by the German Federal Agency for Nature Conservation in December 2018. Due to intensive farming, many species have declined significantly, such as the flame adonis (Adonis flammea), which grows in central Europe on chalky, shallow arable land. The round-leafed modesty (Bupleurum rotundifolium), which has similar soil requirements, has disappeared in many places.
Another group of species that is particularly under threat are plants that grow in non-cultivated, in particular low-nutrient, poor soil. They include species such as the small pasque flower (Pulsatilla pratensis) and catsfoot (Antennaria dioica). However, typical moorland plants such as types of sundew (Drosea spec.) or bogbean (Menyanthes trifoliata) are also classified as endangered species. The main cause of the decline in numbers is the increasing addition of nutrients to the soil. As well as those plants that are included on the list of endangered species, there are also others for which Germany is responsible internationally for protecting, either because they only grow in Germany, or because their populations here are large compared to other countries. They include species such as the sycamore (Acer pseudoplatanus) and copper beech (Fagus sylvatica).
Which ecosystems need particular protection?
A particularly large number of species groups that are under threat or in danger of becoming extinct grow on low-nutrient soil, such as heathland, grassland, and moorland. However, hay meadows rich in plant species, orchards, floodplains, alpine meadows and coastal dunes also need special protection.
Which neophytes grow particularly well in Germany?
Neophytes are plants that began to grow after 1492 in areas from which they do not naturally originate. In Germany, more than 400 neophytes have become firmly established. Of these, more than 50 species have been classified by the German Federal Agency for Nature Conservation as being invasive or potentially invasive. Many of us are familiar with common plants such as the Canadian goldenrod (Solidago canadensis), hill mustard (Bunias orientalis), Japanese knotweed (Fallopia japonica), giant hogweed (Heracleum mantegazzianum) and Himalayan balsam (Impatiens glandulifera).
These “invasive” non-native species generate problems for nature conservation and lead to economic damage by reducing crop yields, causing increased use of pesticides in agriculture and forestry, or making it more expensive to maintain roads, waterways and railway tracks, for example. Giant hogweed and common ragweed (Ambrosia artemisiifolia) also contain substances that cause burns or allergies among humans.
After habitat destruction, invasive species are regarded as posing the second-greatest threat to biodiversity throughout the world. Above all, invasive species compete with native species for habitat and resources. In doing so, they can suppress individual native species or even entire natural communities. One clear example for the area near where we work in Jena is hill mustard. It has spread like wildfire and is now pushing out native and rare plant species from the species-rich meadow and semi-dry grassland biotopes. Early detection and rapid response are critical processes to prevent the spread and establishment of invasive species.
How will the app be developed in the future?
While in recent years, we have mainly worked on the app and automatic identification, in the follow-up project, Flora Incognita++, our main aim is to evaluate the observation data. Here, studies are needed for automatic quality control. We also want to be able to draw conclusions about flowering periods, dissemination, and coexistence.
We have also not yet met all our quality standards when it comes to automatic identification. In the follow-up project, we will improve the detection algorithms specifically for species groups that are critical to identification, such as sweet and sedge grasses.
We are also working on our third app (Flora Key), which contains an interactive identification key and which enables manual identification on the basis of morphological properties of the plant. This identification key will also be integrated into the Flora Incognita app.
The Flora Incognita project is jointly funded by the German Federal Ministry of Education and Research (BMBF), the German Federal Agency for Nature Conservation and the Thuringian Ministry for the Environment, Energy and Nature Conservation, and will continue to take an application-oriented approach. An important goal of our research group will continue to be to forge links between science, the general public and the government authorities.
Many thanks for this interview!
Interview: Barbara Abrell (Max Planck Society)
Plant identification app “Flora Incognita” honored with Thuringian Research Prize
One third of the plant species in Germany is listed as endangered, tendency increasing. At the same time, the number of people with species knowledge is continuously decreasing. But how can we protect species that we don’t recognize? The Flora Incognita research project combines smartphones, artificial intelligence and citizen participation in an app that interactively and automatically identifies plants based on image recordings. With every successful application, the app learns and improves its recognition accuracy. At the same time, the records of the identified species and locations create valuable data sets to answer questions of species protection and biodiversity. More than 1 million people, from enthusiastic laypersons to biology professors, are already using the free app. The interdisciplinary project team from the Max Planck Institute for Biogeochemistry in Jena and the Technical University of Ilmenau was honored for its development with the Thuringian Research Prize in the category of applied research.
Read more
Flora Incognita honored with special prize at Thuringian Environmental Award 2019
The Thuringian Environmental Prize, awarded by the Ministry for the Environment, Energy and Nature Conservation, recognizes the commitment for a habitable environment and healthy nature. In Thuringia, the prize is awarded for outstanding achievements and dedication to environmental protection that contribute to ecological improvements in the context of sustainable development.
The high-ranking, 12-member jury selected 2 special prizes from among the applications received in addition to the prize winners from outside academic research. The Flora Incognita research group of the Max Planck Institute for Biogeochemistry (MPI-BGC) was honored with this prize for its app for plant identification.
Dr. Jana Wäldchen, head of the research group at MPI-BGC, together with Prof. (JP) Dr. Patrick Mäder, TU Ilmenau, accepted the award from Minister Anja Siegesmund in the Jena Climate Pavillon. „After almost 5 years of intensive development work, we have created a widely used and popularly accepted app for automatic plant identification,“ says Patrick Mäder, „but we want to continue to make continuous improvements for the users.“
„In the long run, we as researchers can use the data from the Flora Incognita App to make many other decipherings: When do which species bloom? How strongly do the traits of the individual plants vary? What is the relationship to climate change and land use patterns?“ emphasized Jana Wäldchen.
Flora Incognita vs. Flora Capture
Our project „Flora Incognita“ for many ( and more ) people represents the almost automatic species identification of wild plants. We are grateful for the positive feedback, but of course there are always some uncertainties. For example, the areas of use of the two apps: Flora Incognita and Flora Capture
Flora Incognita – Automatically identify species
With Flora Incognita, you can automatically identify a plant in just a few moments. Depending on the type and quality of your observation, the app needs 1-3 images. You can confirm that the species is indeed identified or select another one from a list of suggestions. An internet connection is required for a determination: Either via mobile data on the go or later at home, where you can also select your photos from the gallery.
Publication:
Mäder, P., Boho, D., Rzanny, M., Seeland, M., Wittich, H. C., Deggelmann, A., & Wäldchen, J. (2021). The flora incognita app–interactive plant species identification. Methods in Ecology and Evolution. https://doi.org/10.1111/2041-210X.13611
Flora Capture – Document plants and contribute to the project
The aim of this app is to collect plant images from given perspectives. These provide, among other things, the data basis of the identification function of the Flora Incognita App. More so: it offers you the opportunity to collect plants without extracting them from nature.
Its distinctive feature is that a capture observation is taken from one and the same individual. This creates a true understanding of the variety in size, shape and color of plant parts in different degrees of flowering and maturity. This data ensures that Flora Incognita can reliably identify even difficult species.
The combination of flowers and leaf images in our database enables a species-specific identification of plants even when an identification should not be possible. Here you see different types of buttercup. An identification solemnly based on the flower would hardly be possible:
So the next time you are happy that in many cases even plants without flowers are recognized correctly, this is mainly possible due to the Flora Capture observations that we have already collected.
Maybe you could make an active contribution to our project from time to time? Each Flora Capture observation is evaluated and, if possible, identified by a team of experts. This manual process may be fast if the key features are clearly visible and the species is one of the current species list (~4800 species). If not, it will take a little longer and sometimes be impossible: For species hybrids, ornamental plants or photographs taken far away from the Central European flora. Flora Capture images can be taken without an internet connection and then uploaded to our server via WLAN at home.
Publication:
Boho, D., Rzanny, M., Wäldchen, J., Nitsche, F., Deggelmann, A., Wittich, H. C., … & Mäder, P. (2020). Flora Capture: a citizen science application for collecting structured plant observations. BMC bioinformatics, 21(1), 1-11. https://doi.org/10.1186/s12859-020-03920-9
Latest News
How smartphones can help detect ecological change9. June 2021 - 13:13
Efficient pollen identification – Interdisciplinary research team combines image-based particle analysis with artificial intelligence9. November 2020 - 12:49
„The only things you want to conserve are the things you know”9. June 2020 - 12:29