A recent study by the Google Brain Team proposes a new way of programming automated machine learning (AutoML) based on symbolic programming.

Original article available here.

A recent study by the Google Brain Team proposes a new way of programming automated machine learning (AutoML) based on symbolic programming. The researchers have also introduced PyGlove, a Python library that demonstrates the new paradigm’s promising results.

Neural network architectures are becoming increasingly complex, and finding the right one for a task is time and resource consuming. Since Google coined the term AutoML for its neural architecture search (NAS) solution in May 2017, the approach has gained traction in the ML research community for automating network hyperparameter and architecture choices.

Current software limitations have however hindered AutoML advancement. Popular NAS algorithms, for example, require implementation for coupling the AutoML search space and search algorithm, and implementing complex search flows to handle the dynamic interactions between such AutoML components remains difficult.

In the paper PyGlove: Symbolic Programming for Automated Machine Learning, researchers identify a potential solution — reformulate the program logic for AutoML as an automated process for manipulating ML programs symbolically.

Another important AutoML component is the child program, which refers to a performant program that could be a neural network architecture or a data augmentation policy. AutoML works to identify performant child programs out of a large set of possibilities defined by the search space. The search algorithm needs to iteratively sample multiple child programs to select and evaluate the most promising ones. A numeric measure of quality and reward is then fed back to the search algorithm to improve future child program sampling.

The researchers’ symbolic programming approach for AutoML enables the manipulation of a symbolic tree-based representation that encodes a machine learning program’s critical elements. As a result, the ML programs became mutable objects and can be cloned and modified after their creation. In this way, modifying an ML program no longer requires ML practitioners to repeatedly create objects from scratch — they can instead clone existing objects and modify them into new programs. With symbolic programming, the interactions between the child program, search space and search algorithms are no longer fixed, making AutoML programming more flexible.

To test the potential of their novel reformulation, the researchers introduced a general symbolic programming library for Python dubbed PyGlove, which enables AutoML techniques to be easily dropped into preexisting ML pipelines. The modification of search spaces, search algorithms and search flows can then be performed with only a few lines of code.

The proposed approach promises to simplify the development of search-based programs, which the researchers suggest could benefit research areas such as medicine design and video surveillance. Google’s AutoML has already helped developers design efficient new neural networks for practical use, and the Google Brain researchers believe their proposed symbolic programming combined with PyGlove will make AutoML even more productive and accessible to ML practitioners.

The paper PyGlove: Symbolic Programming for Automated Machine Learning was accepted at NeurIPS 2020 and is available on arXiv.