🔭 Your guide to AI: July 2020

Welcome back to regular readers and hello to everyone who joined since last month!. Enclosed you’ll find Your Guide to AI: July 2020. I’ll cover key developments in AI tech, geopolitics, health/bio, startups, research and blogs.

News: Next month, we'll publish the State of AI Report 2020, which will include 150+ slides that dive into major AI research, industry, talent, and geopolitics. We can't wait to share new data and perspectives with you. Ask - if you're interested in critically reviewing the Report or helping to expand its reach in your network, drop me a line. If you missed our 6th RAAIS conference you can now catch all the talks on our YouTube channel.

Startups and researchers: If you’d like to chat about your company or a project you’re working on or are looking for a new role, just hit reply!

Help spread the word: If you enjoy this issue, consider forwarding it to a couple of friends. It makes for great banter material 😉

---

Technology markets

🏥 Healthcare and life science

A study of scientific publications, patents, company reports for 21 major pharmaceutical companies between 2014 and 2019 concluded that the industry is in an “early mature” phase of using AI in their businesses. Novartis came in first with 20 internal AI projects and 8 cooperations with startups (AstraZeneca came in #2), while Gilead came in last with 1 internal AI project and 1 startup collaboration. Overall, the authors conclude that “AI has not yet contributed to a sufficient extent to R&D efficiency, effectiveness, or productivity in big pharma.”

Even so, there are an increasing number of positive examples. This includes Reverie Labs, which signed a multi-target collaboration agreement focused on kinase inhibitors with Roche and Genentech. The startup generates hit candidates by factoring in drug potency, selectivity and chemical properties. In the UK, Exscientia had signed a €250M deal with Sanofi in May 2017 to evaluate over one thousand combinations of immunological drug targets for potential synergistic effects with bispecific small molecules. Just over two years later, Sanofi exercised their option over a first-in-class small molecule that Exscientia discovered. Win!

With more open datasets and competition, progress will only accelerate. For example, NYU Langone Medical Center and Facebook AI released the fastMRI dataset that includes thousands of brain and knee MRIs with the goal of speeding up this imaging modality.

For drug discovery, The Broad Institute and 12 other academic and industrial partners launched The Joint Undertaking in Morphological Profiling with Cell Painting dataset. The data is of more than 1 billion cells responding to over 140,000 small molecules and genetic perturbations. Another all-important ImageNet for Biology project. 

🌎 The (geo)politics of AI

The UK government pulled a 180-degree policy stance change on Huawei’s 5G equipment. Despite approving its limited use in January 2020, Johnson’s government has now banned Huawei as a supplier. Many point to mounting pressure from the Trump administration, which has significantly escalated its trade war with China over the summer. 

Trump also pulled a huge move to escalate US-China tensions by mandating that ByteDance sell its US operations for TikTok to a US technology company, most likely Microsoft. The President cited data privacy concerns over content generated and consumed by US users being sent to China. This is a huge blow for the wildly successful app that is well known for making use of AI for rapidly hooking users into their content. More to follow soon...

On the facial recognition policy front, Clearview AI, which brands its “search engine for faces”, has been pushed out of the Canadian market following an investigation. The company, which licenses its technology to 600 law enforcement agencies, is also subject to a joint probe from the UK and Australian information commissions over how it scraped enormous amounts of private photos of people on social networks. Large enterprises including Amazon, Microsoft, and IBM have either paused or stopped their facial recognition programs. 

🚗 Autonomous everything

AV companies are maturing their approaches to building reliable autonomous services. Their efforts range from teleoperations to new hardware sensors to better capture the environment and larger datasets. For example, Voyage launched their in-house teleoperations service, which serves to monitor their AV fleet rides, provide human support for deciding how to drive tricky scenarios and remote controlling the vehicle. Lyft Level 5 are kicking off a Kaggle competition focused on motion prediction from 1,000 hours of traffic agent data. Aurora debuted their FirstLight lidar solution that came out of their acquisition of Blackmore last year. They’re excited about this lidar because it uses frequency modulated continuous wave technology to simultaneously measure both the location and velocity of surrounding objects instead of location alone. The company also expanded testing into Texas.

On the topic of lidar, the ever popular special purpose acquisition company (SPAC) method of taking companies public has made its way to Velodyne. The company entered into a reverse merger agreement with the blank cheque acquisition company Graf Industries. The deal will result in a publicly-listed Velodyne worth approximately $1.6B. The business generated over 80% of its $106M in 2019 revenue from hardware sales and pitched a major shift towards generating revenue from software (autopilot and collision avoidance) and a yet to be released smaller Lidar. 

In China, DiDi’s independent self-driving company launched its robotaxi service in Shanghai along a 6 kilometer loop in what some call a world first. This comes a few weeks after DiDi closed a $500M investment from SoftBank’s Vision Fund 2. 

🍪 Hardware

The big news in chip land is the rumored acquisition of ARM by NVIDIA for over $32B. ARM has struggled to grow under SoftBank’s ownership despite throwing profitability out the window. Its revenues have grown from $1.2B in 2016 to $1.9B while NVIDIA’s revenue has tripled over the same period. With NVIDIA’s meteoric rise in enterprise value, paying in shares would prove rather inexpensive. The challenge with this deal is that ARM’s business model is predicated on selling chip designs to anyone in the industry, i.e. competitors of NVIDIA. It would also be another lost opportunity to return ARM as an independent UK company. 

Graphcore (an Air Street Capital portfolio company) released their second generation Intelligence Processing Unit. The chip packs 59.4 billion transistors and almost 1,500 cores on a single wafer. The system provides more processing power, more memory, and built-in scalability for handling extremely large parallel processing workloads. This Mk2 chip trains a BERT-Large model 9.3x faster than the Mk1 chip. More details here.

Finally, MLPerf results have been released. It’s a benchmarking competition where AI hardware vendors run a variety of models on a variety of tasks to see who is best. This time around, both Google and NVIDIA showed several-fold improvements with their latest respective hardware releases. 

Research papers

Here’s a selection of impactful work that caught my eye, grouped in categories:

📝 NLP

Recent Advances in Google Translate, Google AI. This post shows how the company’s popular translation system has averaged +5 BLEU score improvement over all 100+ languages in the last 12 months. The performance increase is thanks to improvements to “model architecture and training, improved treatment of noise in datasets, increased multilingual transfer learning through M4 modeling, and use of monolingual data.”

TextAttack: A Framework for Adversarial Attacks, Data Augmentation, and Adversarial Training in NLP, UVA and MIT. There have been lots of papers exploring different kinds of adversarial attacks and how to protect against them. This paper seeks to accelerate research on this topic by introducing a Python framework for adversarial attacks, data augmentation, and adversarial training in NLP called TexAttack. The framework lets you create attacks by combining four components: a goal function, a set of constraints, a transformation, and a search method. It also offers implementations of 16 adversarial attacks from the literature and supports a variety of models and datasets, including BERT and other transformers, and all GLUE tasks.

📷 Computer vision

World-Consistent Video-to-Video Synthesis, NVIDIA. This paper addresses the problem of generating synthetic street-view imagery such that the scenes remain consistent as the agent revisits a path it already navigated. Their GAN-based approach generates 2D world renderings that are consistent over time and viewpoints, which was not possible with prior approaches. The method involves coloring a 3D point cloud of the world such that new regions the camera processes are colored in a consistent manner. The GAN learns to generate images of the world based on 2D projections of the point cloud. 

High-performance self-supervised image classification with contrastive clustering, Facebook AI. This paper develops a new method for self-supervised learning (SSL) where a network can learn from unlabeled data. While SSL is compelling, it is typically much slower than supervised learning and can use up to 100x more computational resources. To improve on these issues, the authors’ SSL method applied to an image classification model leverages the information that makes two images visually different as a training signal to discover semantics present in the images. This “contrastive learning” learning approach to SSL is 6x more computationally efficient than other SSL methods to achieve the same model performance. 

Multiview Neural Surface Reconstruction with Implicit Lighting and Material, Weizmann Institute. This paper presents a method to infer scene geometry, light and reflectance properties of the scene, and unknown camera parameters given a set of input masked 2D images. The method simulates the rendering process of an implicit neural geometry inspired by the rendering equation. The results are impressive. 

Reconstructing journalistic scenes in 3D, New York Times. Not a research paper but an interesting experiment in visual storytelling that uses computer vision to generate indoor scenes. 

🤖 Reinforcement learning

Strong Generalization and Efficiency in Neural Programs, DeepMind. This paper considers how to improve algorithm design, specifically to enable better generalization to out of training sample data and computational efficiency. They use a combination of imitation learning and RL on the neural program induction framework to show that their approach can learn to outperform custom-written solutions for sorting, searching in ordered lists and several other problems. 

🕸️ Graph networks

Graph-based, Self-Supervised Program Repair from Diagnostic Feedback, Stanford. This paper focuses on the problem of learning to repair programs based on diagnostic feedback (compiler error messages). The authors present DrRepair, an approach that uses a program-feedback graph to jointly reason over a broken program and its diagnostic feedback with the goal of localizing an erroneous line in the program and generating a repaired line. The model uses self-supervised learning with unlabeled programs to remove the need for manual labeling. 

One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control, Berkeley, Google, CMU, FAIR. This paper seeks to develop a learned policy that can control lots of different kinds of agents (“global policy”) without being trained specifically for each one. They do this by breaking down the global policy into a collection of identical modular neural networks (“Shared Modular Policies”) that control each of the agent’s actuators (e.g. arm joints, leg joints). Each SMP senses information locally, controls one specific actuator, and can pass messages to other modules during training. Using this setup, the authors use compositional graph neural networks to train a single policy that transfers to many agent form factors. 

🌌 Systems and methods

LEEP: A New Measure to Evaluate Transferability of Learned Representations, Amazon Web Services. Transfer learning is a popular and powerful technique for training neural networks that involve taking advantage of easily available data before fine-tuning a model on a target task that has limited data. A key question is how easy it is to transfer knowledge from one classification task to another. This paper introduces a new metric called LEEP, which measures the transferability between a trained machine learning model and the labeled data set for a new task. The metric requires just one forward pass through a target data set. 

Anatomy of Catastrophic Forgetting: Hidden Representations and Task Semantics, Google. This paper investigates catastrophic forgetting, the process where a model that is trained on a sequence of different tasks progressively sees its performance decay on earlier tasks as new ones are added. The paper shows that catastrophic forgetting disproportionately affects deeper layers. They also find that forgetting is most severe for task representations of intermediate similarity, and that task representational similarity is a function of the underlying data and the optimization procedure. 

Monitoring and explainability of models in production, Seldon. This paper discusses the challenges that include data and model prediction monitoring, outlier detection, and drift, as well as explainability. More on why monitoring is important here.

SECure: A Social and Environmental Certificate for AI Systems, Montreal AI Ethics Institute, Microsoft, McGill University, and CMU. The authors use an environment, society and governance (ESG) framework to assess the impact of machine learning systems and what we can do to improve. They consider compute-efficient models, federated learning, data sovereignty, and certificates that help consumers choose environmentally sensible AI services.

Private matching for compute: New solutions to the problem of enabling compute on private set intersections, Facebook AI. This paper addresses a key data analysis operation, matching records, and how to run this operation in a privacy-preserving manner. The authors describe two new cryptographic protocols that enable private matching for compute.  

AutoML-Zero: Evolving Code that Learns, Google AI. Traditional AutoML approaches to neural architecture search use hand-designed network components (e.g. convolutions, dropout, batch-norm) to find optimal configurations to solve a specific task. AutoML-Zero is new work that shows it is possible to evolve ML algorithms from scratch. This means starting from empty programs and using basic mathematical operations as building blocks. Using evolutionary methods, AutoML-Zero automatically finds the code for complete ML algorithms. As the training experiment progresses, the system “discovers” more and more complex building blocks (e.g. ReLu and gradient normalization) to achieve better and better performance. 

🧪 Science (bio, health, etc.)

From desktop to benchtop with automated computational workflows for computer-aided design in asymmetric catalysis, McGill University. This paper presents a method that allows chemists to predict the outcomes of asymmetric chemical reactions, which are a method for generating chiral drugs or materials, before running them experimentally in the lab. The system is applied to the simulation of four catalyst discovery scenarios: discovery of catalysis through trial and error, screening of potential catalysts, catalyst optimization, and investigation of catalyst scope. A computational approach like this one could allow for a broader exploration of chemical search space in a shorter amount of time compared to real-world experiments. 

Signal Peptides Generated by Attention-Based Neural Networks, Caltech and BASF. Signal peptides are short amino acid sequences that are appended to proteins and mark them for translocation to the cell membrane for secretion. The efficiency with which a cell secretes a protein is determined by its signal peptide. As such, it would be useful to generate the optimal signal peptide for the desired protein. This paper applies the transformer model to signal peptide sequences to generate sequences with a high probability of being functional. Interestingly, the sequences show between 58% and 73% sequence similarity with known natural signal peptides. This shows that ML methods can indeed explore a biological design space to uncover novel, diverse and functional candidates.

Resources

📜 Blogs or "essays"

Tractable’s AI estimates the cost to repair Boris Johnson’s recent car accident. 

Interoperability is holding back machine learning infrastructure: a blog. More on this topic here.

A compilation of MLOps resources, ranging from open source libraries to best practices.

A reddit thread critiques several issues with the ML community, namely reproducibility, idolization of large lab papers, and the pains of peer review at large conferences. 

How Roblox, the blockbuster game creation platform, deployed BERT models in production to serve 1 billion+ requests per day. 

A blog post on practical data advice from Facebook and Airbnb. 

The user experience of AI: a practical guide. 

🎥 Videos, talks

A group of AI researchers and engineers from top organisations published a new online course called full-stack deep learning. It teaches you how to formulate your problem, estimate the cost of solving it, data management, choosing frameworks and compute infrastructure, how to troubleshoot training and succeed with your deployment.

Here’s a fun unboxing video for the new $75k Boston Dynamics Spot Mini.

The five important trends in data, and the one megatrend powering them all.

🧰 Open source and tooling

Snorkel AI released their programmatic data labeling product that allows domain experts to easily write labeling rules to bootstrap dataset labeling. 

OpenCV created a low cost, small and power-efficient hardware solution for spatial AI that raised $1.3M from over 6.5k backers on Kickstarter. The project was funded in 20 mins!

💰Startups

Here’s a highlight of the most intriguing financing rounds: 

Anduril, the US defense company building AI-based aerial systems and software, raised a $200M Series C led by a16z. This round values the business at $1.9B post-money. The company said they generate $100M in revenue, of which 60% is from US government defense contracts. 

UiPath, a leading RPA software company, raised a $225M Series E led by Alkeon Capital Management at a $10.2B post-money valuation. The business breached $400M ARR in mid-2020 and counts over 7,000 enterprise customers. 

MonkeyLearn, a no-code NLP service, raised a $2.2M Seed round led by Uncork and Bling Capital. This business is looking to capitalise on the proliferation of no-code tools and language being a universal data source across workstreams in SMEs and enterprises. 

Paige, a computer vision startup for analysing medical biopsies, added a further $15M from Goldman Sachs to grow its Series B to $70M. 

Ravelin, the fraud prevention solution for online businesses, raised a $20M Series C led by Draper. Since its Series B in 2018, their machine learning product has blocked 4 million fraudulent accounts which attempted to place 14.7 million orders worth $53Bn.

Caption Health, an AI-first cardiac ultrasound analysis software company, raised a $53M Series B led by DCVC. 

ComplyAdvantage, the AML data and technology company for fighting financial crime, raised a $50M Series C led by Ontario Teachers’ Pension Plan. 

SenseTime, the Chinese facial recognition startup, raised $1.5B at a $10B valuation before pursuing its listing on the Shanghai STAR market. 

Skydio, the makers of an autonomous drone system, raised a $100M Series C led by Levitate Capital and NTT Docomo Ventures. Although the company started selling direct to consumers, they have now refocused on industrial outdoor inspection workflows. 

Tempo, the at-home exercise display, raised a $60M Series B led by General Catalyst. This product category has been heating up, most recently since the Mirror acquisition by Lululemon.

Chorus.ai, the AI-first sales analytics and coaching software, raised $45M led by Georgian Partners. The company claims that its customers see 30-50% improvement in sales agent’s quota attainment using their software.

Exits in June 2020:

WDG Automation, a Brazilian RPA solution provider, was acquired by IBM to strengthen its RPA offering alongside its cloud AI solutions. Terms were not disclosed.

--

Signing off, 

Nathan Benaich, 16 August 2020

Air Street Capital | Twitter | LinkedIn | RAAIS | London.AI

Air Street Capital is a venture capital firm that invests in AI-first technology and life science companies. We’re a team of experienced investors, engineering leaders, entrepreneurs and AI researchers from the World’s most innovative technology companies and research institutions.