Laurence will keynote our conference with a 'State of the Union' around the TensorFlow ecosystem, talking about where Google have come from with open-source AI technologies, and where the industry is going. His specialty is in education, and he'll share how Google strategized growing the AI developer ecosystem exponentially, and what they're doing to help academics, employers and individuals benefit together. He's particularly passionate about widening access to AI, and making it easier for anybody to get involved, and will share some success stories of how software developers without deep academic backgrounds have built innovative computer vision systems.

This talk covers the two levels of real-time machine learning: online prediction (as opposed to batch prediction) and continual learning (as opposed to batch learning). Online prediction means making predictions as soon as requests arrive. Continual learning is when an ML system allows an ML system to automatically update with new data in production, speeding up the iteration cycle to combat model performance’s decay due to concept drift.

Real-time machine learning allows a system to be adaptive to changing users' behaviors and environments, which leads to better model performance and, in some cases, reduced compute cost. However, real-time ML comes with many infrastructural and theoretical challenges, and this talk discusses the key challenges.

“Power-efficient hardware for deep neural network (DNN) acceleration is a major obstacle for the successful deployment at scale of many edge AI applications such as autonomous driving, smart retail, smart cities and more. Quantization is exploited by modern DNN acceleration at the edge to significantly reduce power consumption. In the first part of the talk, we will give a brief overview of the hardware aspects of quantization, followed by a high-level review of the main approaches to quantization.  We show how these concepts are leveraged in the Hailo-8, a highly power-efficient DNN accelerator. In the second part, we discuss "real-world" challenges of quantization as well as suggest perspectives for future work to address current gaps.”

Supervising with boolean class labels all future scenarios a machine learning application may encounter is understood to be infeasible, leading to renewed interest in adaptive and self-supervised methods.  Yet several popular “unsupervised” methods implicitly presume knowledge of target domain invariances.  I’ll present new methods that do not require hand-selecting augmentation strategies and learn without supervision: multi-task contrastive learning, automatic selection of augmentation policies, and entropy-based adaptation at test time without access to source labels or data.  Fully unsupervised and adaptive learning can work together to dramatically reduce the level of supervision required for real-world tasks.

At Datagen, we maintain a large catalogue of artist-made 3D assets from many categories (i.e. tables, chairs,bottles, etc.).  Each asset consists of a 3D mesh and a texture map. We form an alternative, implicit volumetric representation of all assets, where each asset in a category is assigned a latent code. A single network coupled with a differential renderer is trained to render 2D images of each asset given its code, that are similar to the images rendered directly from the asset's mesh. The codes fully encapsulate the assets shape and appearance, and can be used to extract the visual attributes of each asset, without the need to re-render it.

An important capability of autonomous Unmanned Aerial Vehicles (UAVs) is autonomous landing while avoiding collision with obstacles in the process. Such capability requires real-time local trajectory planning. Although trajectory-planning methods have been introduced for cases such as emergency landing, they have not been evaluated in real-life scenarios where only the surface of obstacles can be sensed and detected. We propose a novel optimization framework using a pre-planned global path and a priority map of the landing area. Several trajectory planning algorithms were implemented and evaluated in a simulator that includes a 3D urban environment, LiDAR-based obstacle-surface sensing and UAV guidance and dynamics. We show that using our proposed optimization criterion can successfully improve the landing-mission success probability while avoiding collisions with obstacles in real-time. 

StyleGAN is able to generate highly realistic images in a variety of domains, and therefore much recent work has focused on understanding how to use the latent spaces of StyleGAN to manipulate generated and real images. In this talk, I will present our recent paper "StyleCLIP: Text-Driven Manipulation of StyleGAN Imagery". In this paper, we explore leveraging the power of recently introduced CLIP models in order to develop a text-based interface for StyleGAN image manipulation. This interface provides a great expressivity for image editing and allows to perform edits that were not possible with previous approaches.

The fields of signal and image processing have been deeply influenced by the introduction of deep neural networks. Despite their impressive success, the architectures used in these solutions come with no clear justification, being ‘‘black box’’ machines that lack interpretability. A constructive remedy to this drawback is a systematic design of networks by unfolding well-understood iterative algorithms. A popular representative of this approach is LISTA, evaluating sparse representations of processed signals. In this paper, we revisit this task and propose an unfolded version of a greedy pursuit algorithm for the same goal, this method is known as LGM.

In fluid dynamics, one of the most important research fields is hydrodynamic instabilities and their evolution in different flow regimes.  Currently, three main methods are used for understanding such phenomena -- namely analytical and statistical models, experiments, and simulations -- and all of them are primarily investigated and correlated using human expertise. This presentation/work demonstrates how a major portion of this research effort could and should be analyzed using recent breakthrough advancements in the field of Computer Vision with Deep Learning. Specifically, Image Retrieval, Template Matching, Parameters Regression, and Spatiotemporal Prediction -- for the quantitative and qualitative benefits they provide. In order to do so, this research focuses mainly on one of the most representative instabilities, the Rayleigh-Taylor instability. The techniques which were developed and proved in this work can serve as essential tools for physicists in the field of hydrodynamics for investigating a variety of physical systems. Some of them can be easily applied to already existing simulation results, while others could be used via Transfer Learning to other instabilities research. 

Pictures of everyday life are inherently multi-label in nature.
Hence, multi-label classification is commonly used to analyze
their content. In typical multi-label datasets, each picture contains only a few positive labels, and many negative ones. This
positive-negative imbalance can result in under-emphasizing
gradients from positive labels during training, leading to poor
accuracy.
In this lecture, we will introduce a novel asymmetric loss (”ASL”),
that operates differently on positive and negative samples.
The loss dynamically down-weights the importance of easy
negative samples, causing the optimization process to focus
more on the positive samples, and also enables to discard mislabeled negative samples.
We demonstrate how ASL leads to a more ”balanced” network, with increased average probabilities for positive samples, and show how this balanced network is translated to better mAP scores, compared to commonly used losses. Furthermore, we offer a method that can dynamically adjust the level
of asymmetry throughout the training.
With ASL, we reach new state-of-the-art results on three
common multi-label datasets, including achieving 86.6% on
MS-COCO. We also demonstrate ASL applicability for other
tasks such as fine-grain single-label classification and object
detection.
ASL is effective, easy to implement, and does not increase
the training time or complexity. 

Code is availabl: https://github.com/Alibaba-MIIL/ASL

 

We propose a probe for the analysis of deep learning architectures that is based on machine learning and approximation theoretical principles. Given a deep learning architecture and a training set, during or after training, the Sparsity Probe allows to analyze the performance of intermediate layers by quantifying the geometrical features of representations of the training set. We talk about the importance of representation learning, and geometrical features in the latent space. 

StyleGAN has recently been established as the state-of-the-art generator, synthesizing images of phenomenal realism across various domains. With its rich semantic space, many works have attempted to understand and control StyleGAN’s latent representations with the goal of performing image manipulations. To perform manipulations on real images, however, one must learn to encode the image into StyleGAN’s latent space, which remains an open challenge. In this talk, I will present the ReStyle encoder, where we introduce an iterative refinement technique for approaching the encoding task. We’ll then demonstrate the power of such encoding techniques for performing a wide range of tasks.

 

Recently, deep neural networks (DNN) are successfully employed to predict the best deformation field by minimizing a dissimilarity function between the moving and target images. Bayesian DNN models enable safer utilization in medical imaging, improves generalization and assesses the uncertainty of the predictions. In our study, we propose a non-parametric Bayesian approach to estimate the uncertainty in DNN-based algorithms for MRI deformable registration. We demonstrated the added-value of our Bayesian registration framework on the brain MRI dataset. Further, we quantified the uncertainty of the registration and assessed its correlation with the out-of-distribution data.

Realistic use of neural networks often requires adhering to multiple constraints on latency, energy and memory among others. In Alibaba, models deployed in production need to have a very low inference time to answer several cost constraints. In this talk we present a Neural Architecture Search (NAS) method that we have developed to construct these models by introducing Hard Constrained diffeRentiable NAS (HardCoRe-NAS), that is based on an accurate formulation of the expected resource requirement and a scalable search method that satisfies the hard constraint throughout the search. Our experiments show that HardCoRe-NAS generates state-of-the-art architectures, surpassing other NAS methods.

 

Many systems have sensor redundancy thus require real time assessment of the data quality and relevance from each source. The assessment is useful when allocating the computation resources and user interface. It is difficult to reliably make such comparison between different sensor types in real time as that usually requires data registration and/or deep understanding of the observed scene.

We present a method for assessing and comparing data quality and relevancy through 3D reconstruction, such as by stereo vision. This method allows data assessment to be performed in real-time before other complex algorithms. By using this approach, we can avoid confusing data from noisy channels, and concentrate our resources on the best channel.

The performance of the Artificial Intelligence (AI) that powers automotive systems is directly linked to the data used to train and test it. Our research tackles the issue of using synthetic data for two different Computer Vision tasks where relevant data is hard to collect or simply doesn’t exist. We show how to train object detectors, based only on synthetic data, that has generalized from real-world data. We then show how to use synthetic data to evaluate the performance of distance estimation algorithms. The success of these two tasks paves the way for future research taking advantage of synthetic data.

Absolute camera pose regressors (APRs) estimate the position and orientation of a camera from the captured image alone, offering a fast, lightweight and standalone alternative to localization pipelines. However, APRs are also less accurate and are typically trained per scene. In this work, we propose a transformer-based approach for improving the accuracy of APRs while learning multiple scenes in parallel. We use encoders to aggregate activation maps with self-attention and decoders for transforming latent features and scenes encoding into candidate pose predictions. Our proposed approach achieves a new state-of-the-art localization accuracy for pose regression methods across indoor and outdoor benchmarks.

"By 2024, 60% of the data used for the development of AI and analytics projects will be synthetically generated” - Erick Brethenoux, Gartner VP analyst.

As data collection and annotating is hard, tedious, and expensive, using rich, accurately annotated, synthetically generated data is gaining momentum.

In manufacturing, 3D simulators are used for generating synthetic data to train tasks like bin picking, robotic manipulation, quality inspection etc.

We will review the current state, possibilities, and limitations of synthetic data, as well as known synthetic data generation tools.

Finally, we will discuss advanced methods for reducing the gap between synthetic and real data.

The key aspect behind compositionality is that humans understand the world as a sum of its parts. For example, objects can be broken down into pieces and events are composed of atomic actions. Our understanding of the world is naturally hierarchical and structured, and when new concepts are introduced, humans tend to decompose the familiar parts. This leads to the hypothesis that intelligent machines would need to develop a compositional understanding that is robust and generalizable. In this talk, I will present our two recent projects towards achieving this goal, our ICML21 paper on synthesizing goal-oriented videos and hot-off-the-press work on bringing object interactions to video transformers, called Object-Region Video Transformers (ORViT).

 

WSC Sports’ main goal is developing sports events segmentation and recognition algorithms in order to automatically generate highlight videos from sports broadcasts.

A segment of an event usually includes a short time of the preparation for the play, a celebration, or any other expression of players at the end of the event. Each event can have few valid boundaries and the ideal ones are subjective.

In this presentation, we will show our method to solve this problem, how we leverage our massive raw data to train a weakly supervised model and how we evaluated this multiple solutions’ task.

Quantitative Diffusion-Weighted MRI (DW-MRI) analysis with the “Intra-voxel Incoherent motion” (IVIM) signal decay model shows potential to provide quantitative imaging biomarkers for various clinical applications. Nevertheless, reliable estimation of the IVIM parameters from clinical DW-MRI data remains a challenge. Recently, deep-neural-networks (DNN) based approaches for IVIM model parameters prediction demonstrated potential in obtaining more accurate and faster parameter estimates compared to classical model-fitting methods; however, these methods’ capability to generalize the IVIM model fitting to different clinical acquisition protocols is limited. To address this shortcoming, we propose a novel DNN model: “IVIM-MIAP: Model Learning based on Incorporation of the Acquisition Protocol”, which incorporates the acquisition protocol as part of the architecture. We demonstrate the added value of IVIM-MIAP compared to previously proposed DNN-based methods through simulation studies as well as in-vivo DW-MRI data.

The search for efficient neural network architectures has gained much focus in recent years, where modern architectures focus not only on accuracy but also on inference time and model size. We present FUN, a family of novel Frequency-domain Utilization Networks. These networks utilize the inherent efficiency of the frequency-domain by working directly in that domain, represented with the Discrete Cosine Transform. Using modern techniques and building blocks such as compound-scaling and inverted-residual layers we generate a set of such networks allowing one to balance between size, latency and accuracy while outperforming competing RGB-based models. 

Glioblastoma multiforme (GBM) is the most frequent and lethal malignant brain tumor in adults. Novocure’s Tumor Treating Fields (TTFields) therapy was recently introduced as a novel therapeutic modality that significantly extends GBM patients’ life. TTFields treatment planning requires the segmentation of GBM tissues on postoperative MR images for evaluating the distribution of TTFields in the tumor. We present a novel method for the segmentation of GBM in postoperative patients which incorporates noisy labels and models the uncertainty of the segmentation. This uncertainty can be further deployed for intuitive and fast editing of the segmentation result.

In this lecture, we will explore state-of-the-art methods for Transformer interpretability. We will present applications for Transformer interpretability in computer vision, NLP, and multi-modal tasks while showing examples with some of the most popular Transformer-based models such as the Vision Transformer, CLIP, and BERT.

DNN interpretability is an ever-growing field of research targeted at explaining neural networks by developing methods to diagnose what aspects of the input drive the decisions made by the model.

Recently, Transformers gained increasing popularity in several fields, such as NLP and computer vision. This wave of popularity substantiates the necessity of interpretability methods for Transformers.

CT Perfusion (CTP) is a 4 dimensional CT time-series dynamic scan, where intravenous contrast is injected while the patient’s head is scanned repeatedly as contrast enters and leaves the brain. The scan is useful for assessing a stroke patient suitability for treatment, as it enables differentiation of salvageable brain tissue from the irrevocably damaged brain tissue.
In this talk, we describe the complex algorithmic pipeline we have developed at Viz.ai for the automatic, scalable and accurate analysis of this high-variability, noisy data including Deep Learning based motion correction, signal processing methods for denoising, Deep Learning models for damaged tissue segmentation, and more, turning data into clinically actionable information for improving patients’ care.

Deep fake images and videos are ubiquitous e.g. in social networks, Youtube and movies. Recent studies have shown rapid progress in facial manipulation, enabling attackers to manipulate the facial image of an individual and generate a new identity.  As a remedy, many CNN based approaches suggest a deep fake detection classifier, capable of distinguishing between real and fake images/videos. In recent years we are witness to competitions organized and driven by large and diverse fake footage, created from different generators. Apparently, the generalization capability of the current detectors to unknown generators is still a debate in the community. In this talk I’ll present an overview of the problem focusing on the generalization task.

Complicated assembly processes can be described as a sequence of two main activities: grasping and insertion. While general grasping solutions are common in industry, insertion is still only applicable to small subsets of problems, mainly ones involving simple shapes in fixed locations and in which the variations are not taken into consideration. Recently, RL approaches with prior knowledge (e.g., LfD or residual policy) have been adopted. However, these approaches might be problematic in contact-rich tasks since interaction might endanger the robot and its equipment. In this paper, we tackled this challenge by formulating the problem as a regression problem. By combining visual and force inputs, we demonstrate that our method can scale to 16 different insertion tasks in less than 10 minutes. The resulting policies are robust to changes in the socket position, orientation or peg color, as well as to small differences in peg shape. Finally, we demonstrate an end-to-end solution for 2 complex assembly tasks with multi-insertion objectives when the assembly board is randomly placed on a table.

Video recordings made by multiple cameras facilitate mitigating occlusion and depth ambiguities in pose and motion reconstruction. Yet, multi-view algorithms strongly depend on camera parameters. Such a dependency becomes a hurdle once shifting to uncontrolled settings. We introduce FLEX (Free muLti-view rEconstruXion), an end-to-end parameter-free multi-view model, that does not require any camera parameters. Our key idea is that the 3D angles between skeletal parts, as well as bone lengths, are invariant to the camera position, in contrast to joint locations. We present our results on several datasets and show that in the absence of camera parameters, we outperform other algorithms by a large margin.

Automatic image segmentation is a well-studied and long-lasting problem in computer vision. Most of the vision community effort in this field over the years has focused on semantic segmentation of a predefined and usually small set of object or scene categories. However, much less focus has been given to the problem of foreground layer segmentation, which is a fundamental task in the photo editing and video creation community. In this talk, we overview the main challenges of this task, show why semantic segmentation fails here, and present a system which can handle the problem robustly.

In this presentation we will present the underlying algorithmic work that has been developed and launched in Amazon's Health and Wellness CVML team. We developed a novel set of biomarkers and CV algorithms to assess physical (Movement) Health. Movement Health is based on functional fitness, which is your body’s readiness to execute the everyday movements you do without thinking. To provide these biomarkers, we use the customer's phone camera to acquire videos of fitness. Once acquired and uploaded to the cloud, our algorithms evaluate the customer's body movement to identify limitations in their stability, mobility, and posture. Following this, we provide an overall Movement score out of 100, details about stability, mobility, and posture, and a breakdown across the body areas.

 

After a brief history of mobile photography, we show how Corephotonics uses latest AI advancements and a unique proprietary camera hardware to introduce world’s first autonomous mobile camera.

Quantitative evaluation of the placenta is important for fetal health evaluation. However, manual segmentation of the placenta is time-consuming and suffers from high observer variability. We present a method for bootstrapping automatic placenta segmentation by deep learning on different MRI sequences without requiring annotations for each scanning sequence. The method consists of automatic segmentation of one sequence followed by automatic adaptation using self-training to a new sequence with only additional unlabeled cases. It uses a novel combined contour and soft Dice loss function. The contour Dice loss and self-training approach achieve state-of-the art placenta segmentation results and sequence transfer bootstrapping.

Depth perception is a unique task in which there is no available real-life sensor coverage for applying the task of monocular depth perception estimation from a camera. The best source for that task is Synthetic data.

We present in this session a couple of data handling techniques that improve the performance of monocular depth perception Deep Neural Networks utilizing Generative Adversarial Networks (GAN) and Scalable Data generation techniques.

We have based our results set on the known KITTI dataset and present a benchmark that was done on that dataset.

 

Any practitioner of computer vision or machine learning in general knows - supervision is an asset, and in some cases quite an expensive one!

In this talk, we will focus on two of our recent works highlighting what can be achieved with limited supervision, be it learning to recognize new fine-grained classes with few examples after pre-training with only coarse class labels, or learning to ground (localize) arbitrary free-text phrases in images while learning from image and text pairs without any image location supervision.

Software tool for the AI accelerators era.

How to accelerate the AI development process from data preparation to deployment and serving using Intel SW tools. How to program multi-device (CPU/GPU/AI accelerator) heterogeneous systems efficiently with oneAPI. A quick overview and demos of Intel SW tools to preprocess, visualize, train and deploy ML/AI and analytic workloads.

Wix provides a cloud-based website development platform for over 200 million users worldwide today. At Wix, we’re actively building creativity tools powered by machine learning and computer vision to help users build professional websites, from automatic image editing to website design understanding and more. Building such tools at Wix’s scale requires not only adapting standard models to specific application use cases, but also dataset engineering practices and supporting platforms to deal with constantly evolving data. 

 

In this talk, we’ll present our computer vision pipelines and workflows, and our journey to move from a conventional 'model-centric' AI to a more ‘data-centric’ one. Throughout this journey, we significantly improved product velocity, created unique tools for MLOps, and established new roles. We learned that many challenges of evolving AI systems can be addressed by high quality data and systematic approaches to data labelling, curation and analytics.

 

Wix provides a cloud-based website development platform for over 200 million users worldwide today. At Wix, we’re actively building creativity tools powered by machine learning and computer vision to help users build professional websites, from automatic image editing to website design understanding and more. Building such tools at Wix’s scale requires not only adapting standard models to specific application use cases, but also dataset engineering practices and supporting platforms to deal with constantly evolving data. 

 

In this talk, we’ll present our computer vision pipelines and workflows, and our journey to move from a conventional 'model-centric' AI to a more ‘data-centric’ one. Throughout this journey, we significantly improved product velocity, created unique tools for MLOps, and established new roles. We learned that many challenges of evolving AI systems can be addressed by high quality data and systematic approaches to data labelling, curation and analytics.

 

Recently, attention-based neural networks have been shown to achieve state of the art results on image classification using pre training on huge datasets. We address the problem of change detection using transformers on small change detection dataset. Following the Compact Transformers introduced by Hassani et al [1] Our model uses both convolution and the Compact Convolutional Transformer (CCT). We show that compared to pure CNNs, the compact transformers have fewer parameters, while obtaining similar accuracies. This method is modular in terms of model configuration, and in this case, it have 4.1M parameters which is third the amount of parameters of pure CNN based model with similar accuracy. In this work we demonstrate for the, the potential of using state of the art change detection transformers-based algorithm on the CD2014 dataset

Understanding what happens on the road can be tricky. Autonomous driving and road insight platforms require the ability to untangle diverse and complex dynamic scenes. At Nexar, we use crowd-sourced data to train deep-learning models to reconstruct corner-cases from monocular views taken by our network of dashcams. In this talk, we’ll discuss how we leverage vision with sensor fusion to produce full temporal 3D reconstruction of corner-cases, and how we overcome obstacles originating from monocular views. We will also see how such systems can be used as stepping stones for robust scene reconstruction even when sensor fusion is not applicable.

 

Brain computer interfaces (BCIs) provide direct communication between the brain and the external world.  A widely used technique for measuring brain activity is electroencephalogram (EEG). EEG measurements from a number of electrodes (here 35) over time (here 900 msec) can be represented as 3-dimensional (3-d) images and analyzed using 3-d image processing techniques. In particular, convolutional neural networks (CNN) can be trained to classify spatial-temporal EEG patterns.  

We developed 3-d CNNs for EEG classification to: (1) determine whether the user wants to close / open or maintain the posture of the hands, and (2) determine whether the executed movement is the intended one. The latter is very important since BCIs are prone to mistakes. Detected errors can be corrected to improve overall accuracy. Our experiments demonstrate that CNN-EEG based error correction can improve the accuracy of CNN-EEG based BCI for hand control by more than 7%.   

Our research facilitates the development of comfortable and intuitive brain-controlled hand prosthesis that may help 3 million upper limb amputees worldwide. 

Accurate lane detection, a crucial enabler for autonomous driving, currently relies on obtaining a large and diverse labeled training dataset. In this work, we explore learning from abundant, randomly generated synthetic data, together with unlabeled or partially labeled target domain data, instead. Randomly generated synthetic data has the advantage of controlled variability in the lane geometry and lighting, but it is limited in terms of photo-realism. This poses the challenge of adapting models learned on the unrealistic synthetic domain to real images. To this end we develop a novel autoencoder-based approach that uses synthetic labels unaligned with particular images for adapting to target domain data. In addition, we explore existing domain adaptation approaches, such as image translation and self-supervision, and adjust them to the lane detection task. We test all approaches in the unsupervised domain adaptation setting in which no target domain labels are available and in the semi-supervised setting in which a small portion of the target images are labeled. In extensive experiments using three different datasets, we demonstrate the possibility to save costly target domain labeling efforts. For example, using our proposed autoencoder approach on the llamas and tuSimple lane datasets, we can almost recover the fully supervised accuracy with only 10% of the labeled data. In addition, our autoencoder approach outperforms all other methods in the semi-supervised domain adaptation scenario.

Defect detection plays a key role in preventing yield loss excursion events in semiconductor manufacturing. In this talk we present the industry-standard detection pipeline, performed with Applied Materials electron-beam and optical inspection tools. We focus on two challenging real-world scenarios: one in which non-defective reference images of the pattern are available, and an unsupervised scenario where they are not. In the “with reference” scenario we propose an efficient method based on the internal statistics of the images. For the completely unsupervised scenario, our research is based on the recent advances in the self-supervised contrastive learning.

This research is the joint work with Dr. Ran Yacoby, Bar Dubovsky, Ore Shtalrid, Dr. Nati Ofir and Omer Granovitser

The LiDAR Scanner on iPhone Pro and iPad Pro unlocks a new way for users to experience and capture the world around them. In this 15 minute talk, we shed light on the underlying deep learning technology that performs LiDAR densification which drives these new experiences and the ARKit Depth API, enabling developers to create new applications on iOS and iPadOS.