One of my favorite samples from the Progressive GANs paper is this one from the "cat" category. Apparently some of the cat training photos were memes with text. The GAN doesn't know what text is so it has made up new text-like imagery in the right place for a meme caption.
I never heard back from MIT. I got rejected from CMU. I was accepted to U of T but not to work with the PI I wanted there. I got "honorable mention" for NSF GRFP but not actual money. Don't let temporary failures discourage you.
ML researchers, reviewers, and press coverage of ML need to get a lot more serious about statistically robustness of results and the effect of hyperparameters. This study shows that many papers over the last year or so were just observing sampling error, not true improvement.
By looking at this image, you can see how sensitive your own eyes are to contrast at different frequencies (taller apparent peaks=more sensitivity at that frequency). It's like a graph that is made by perceiving the graph itself. h/t
@catherineols
Whoa! It turns out that famous examples of NLP systems succeeding and failing were very misleading. “Man is to king as woman is to queen” only works if the model is hardcoded not to be able to say “king” for the last word.
1/7 Do word embeddings really say that man is to doctor as woman is to nurse? Apparently not. Check out this thread for a description of a short paper I co-wrote with Malvina Nissim and Rob van der Goot, available here:
#NLProc
#bias
Thrilled to announce a new program designed to help experts in applied fields build ML-powered products and experiences. Introducing the AI/ML residency program:
GANs can be used to automatically design dental crowns, that are then actually manufactured and used in the physical world. Crowns need to be made specifically for each patient and need to fit correctly with the other teeth and support biting and chewing.
An exciting property of style-based generators is that they have learned to do 3D viewpoint rotations around objects like cars. These kinds of meaningful latent interpolations show that the model has learned about the structure of the world.
Yoshua, Aaron, and I have released the LaTeX template for the Deep Learning book: Useful if you want to follow the same math notation conventions as we do or if you want to put a notation page in your document
This new family of GAN loss functions looks promising! I'm especially excited about Fig 4-6, where we see that the new loss results in much faster learning during the first several iterations of training. I implemented the RSGAN loss on a toy problem and it worked well.
My new paper is out! " The relativistic discriminator: a key element missing from standard GAN" explains how most GANs are missing a key ingredient which makes them so much better and much more stable!
#Deeplearning
#AI
Thank you to the many people who reached out after my now-deleted tweet last week asking for help with an urgent problem. For everyone still concerned, things are under control now.
I've spent several years studying machine learning security with the goal of making ML reliable before it is used in more and more important contexts. Unfortunately, ML capabilities and adoption are growing much faster than ML robustness.
ML paper writing pro-tip: you can download the raw source of any arxiv paper. Click on the "Other formats" link, then click "Download source". This gets you a .tar.gz with all the .tex files, all the image files for the figures in their original resolution, etc.
While GANs have been great at generating realistic images from a single category (one GAN for faces, another GAN for buildings) they've always struggled to fit all 1,000 classes of ImageNet with a single GAN. This ICLR submission has done it: -
New preprint () by Han Zhang, with
@goodfellow_ian
and Dimitris Metaxas. Substantially improves the state-of-the-art on the conditional Imagenet synthesis task.
This is really cool. Some of my PhD labmates worked on ML for compression back in the pretraining era, and I remember it being really hard to get a compression advantage.
Check out our new work on face-vid2vid, a neural talking-head model for video conferencing that is 10x more bandwidth efficient than H264
arxiv
project
video
@tcwang0509
@arunmallya
#GAN
One of my main concerns about machine learning interpretability tools is that they will make people think they understand ML when they don't. People seem to think linear models are interpretable, but no one looks at them and has the intuition that they have adversarial examples
NVIDIA gave me a new T-Rex, signed by Jensen. They are not even for sale yet! Thanks NVIDIA! The pic is with GAN extraordinaire Ming-Yu at the NVIDIA reception last night.
To gain some idea of the far future of ML security, we studied a simple toy problem called "adversarial spheres," simulating a future where advanced ML models are extremely accurate. We find that even then, an adversary can still easily fool them.
TensorFuzz automates the process of finding inputs that cause some specific testable behavior, like disagreement between float16 and float32 implementations of a neural network
Neural networks are notoriously hard to debug.
@gstsdn
has developed a new debugging methodology by adapting traditional coverage guided fuzzing techniques to neural networks.
When I invented adversarial training as a defense against adversarial examples, I focused on making it as cheap and scalable as possible. Eric and collaborators have now upgraded the original cheap version to compete with newer, more expensive versions.
1/ New paper on an old topic: turns out, FGSM works as well as PGD for adversarial training!*
*Just avoid catastrophic overfitting, as seen in picture
Paper:
Code:
Joint work with
@_leslierice
and
@zicokolter
to be at
#ICLR2020
I suspect that peer review *actually causes* rather than mitigates many of the “troubling trends” recently identified by
@zacharylipton
and Jacob Steinhardt:
The term “deep learning” reminds me of “horseless carriage.” It made sense when introduced, but now that it is the dominant paradigm, it feels quaint to specify that there is no horse. The horse here is of course the shallow model / convex cost constraint.
The Self-Organizing Conference on Machine Learning is returning as a 100% online event for 2020. Nov 30-Dec 4. It will still be small to maintain the group discussion feel. Apply at
I think changing the name of NIPS is the right thing to do. The majority of women in the poll voted for it, and moral leadership shouldn’t be driven by polls anyway.
This paper shows how to make adversarial examples with GANs. No need for a norm ball constraint. They look unperturbed to a human observer but break a model trained to resist large perturbations.
The definition of "adversarial examples" I prefer these days is "Adversarial examples are inputs to machine learning models that an attacker has intentionally designed to cause the model to make a mistake"
I'll present a talk called "Defense Against the Dark Arts" summarizing the state of the art and key research challenges for defenses against adversarial examples. Room 259, 1:30 PM.
Deep learning for predicting aftershocks of large earthquakes. Besides offering better predictions, interpretations of the model suggest promising directions for new physical theories
It’s strange to see people defining deep learning as supervised learning via backprop, considering that the 2006 deep learning revolution was originally based on the idea that neither of those things work very well
I originally thought of GANs as an unsupervised learning algorithm, but so far, to create recognizable object categories, they've needed a supervision signal / labeled images. This new work shows how to get them to work well with few labels.
Neural networks are notoriously hard to debug.
@gstsdn
has developed a new debugging methodology by adapting traditional coverage guided fuzzing techniques to neural networks.
Colin was a senior research scientist in my team at Google. He's done great technical work, especially on attention models and semi-supervised / transfer learning, and has been an excellent mentor for many Brain residents / interns. Will definitely be a great PhD advisor.
I'm starting a professorship in the CS department at UNC in fall 2020 (!!) and am hiring students! If you're interested in doing a PhD
@unccs
please get in touch. More info here:
Updates about SOCML: 1) I have failed to run a SOCML 2019 2) I’m not quitting, just having a busy year. I intend to run SOCML 2020 and beyond 3) We’re experimenting with a distributed SOCMLx program. See link for details
StarGAN: learning one model that translates between *multiple* domains without supervision (previous works were about translating between two domains without supervision)
Interpretation of a machine learning model by a human involves both the model and the human. Human misconceptions can cause as much trouble as any property of the model.
Remarkable finding: people don't trust transparent models any more than opaque ones, and have more difficulty detecting large errors in transparent ones:
Interested in jump-starting your career in machine learning research? Consider the Google AI Residency Program! Applications are now open until January 28th, 2019! Check out for more information.
It’s interesting to see the pendulum swing back to representation learning. During my PhD, most of my collaborators and I were primarily interested in representation learning as a biproduct of sample generation, not sample generation itself.
BigBiGAN shows that "progress in image generation quality translates to substantially improved representation learning performance." Competitive w/self-supervised approaches on ImageNet.
The cycle from generative models to other methods and back again continues.
Check out
@fermatslibrary
's Librarian, a Chrome extension that automatically shows comments for ArXiv papers: I've asked for this feature for a long time!
The discriminator often knows something about the data distribution that the generator didn't manage to capture. By using rejection sampling, it's possible to knock out a lot of bad samples.
New preprint by
@smnh_azadi
,
@catherineols
, Trevor Darrell,
@goodfellow_ian
, and me: . We perform rejection sampling on a trained GAN generator using a GAN discriminator. This helps quite a lot for not-much effort.
In our previous collaboration's I've benefited a lot from using David's machine learning frameworks. If you're a researcher or student looking for both simplicity and customizability, definitely check out Objax.
I’d like to share my new project: Objax, a new high-level JAX API with a PyTorch-like interface! Objax pursues the quest for the simplest design and code that’s as easy as possible to extend without sacrificing performance.
1/7