Interest in “self-driving labs,” which use artificial intelligence (AI) and automated systems to accelerate research and discovery, is growing in the fields of chemistry and materials science. The researchers are now proposing a set of definitions and performance metrics that will allow both researchers, non-experts, and future users to better understand what these new technologies are doing and how each technology compares to other self-driving labs. How is it performing in the competition?

Autonomous labs hold great promise for accelerating the discovery of new molecules, materials and manufacturing processes, with applications ranging from electronic devices to pharmaceuticals. Although the technologies are still fairly new, some have been shown to reduce the time required to identify new material from months or even years to days.

“Self-driving labs are getting a lot of attention right now, but there are many outstanding questions regarding these technologies,” says Milad Abolhasani, corresponding author of a paper on the new matrix and an associate professor of chemical and biomolecular engineering. are present.” at North Carolina State University. “This technology is described as ‘autonomous,’ but different research teams are defining ‘autonomy’ differently. By the same token, different research teams report different elements of their work in different ways. This makes it difficult to compare these technologies with each other, and comparison is essential if we are to be able to learn from each other and advance the field.

“What does Self-Driving Lab A really do well? How can we use this to improve the performance of Self-Driving Lab B? We are proposing a set of common definitions and performance metrics, which we Hopefully everyone working will embrace it. The ultimate goal in this space will be to allow us all to learn from each other and advance these powerful research acceleration technologies.

“For example, we seem to be seeing some challenges with the efficiency, accuracy and robustness of some autonomous systems in self-driving labs,” says Abulhasani. “This raises the question of how useful these technologies can be. If we have standardized metrics and results reporting, we can identify these challenges and better understand how to address them. “

At the heart of the new proposal is a clear definition of self-driving labs and seven proposed performance metrics, which researchers will include in any published work related to their self-driving labs.

  • Degree of autonomy: How much guidance does a system need from users?
  • Operational Lifetime: How long can the system operate without user intervention?
  • Throughput: How long does the system take to run an experiment?
  • Experimental validity: How reproducible are the results of the system?
  • Content Usage: What is the total amount of content used by the system for each experiment?
  • Accessible parameter space: How well can the system account for all variables in each experiment?
  • Optimization performance.

“Optimization performance is one of the most important of these metrics, but it’s also one of the most complex — it doesn’t lend itself to a short definition,” says Abulhasani. “Basically, we want researchers to quantitatively analyze the performance of our self-driving lab and its experimental selection algorithm by benchmarking it against a baseline — for example, random sampling.

“Ultimately, we think that having a standardized approach to reporting from self-driving labs will help ensure that the field is producing reliable, reproducible results that will benefit more AI programs.” are those that take advantage of large, high-quality data sets generated by self-driving labs,” says Abulhasani.

This work was supported by the Dreyfus Program for Machine Learning in Chemical Sciences and Engineering, under award number ML-21-064. University of North Carolina Research Opportunities Initiative Program; and the National Science Foundation, under Grants 1940959 and 2208406.