Ivan Skorokhodov Profile Banner
Ivan Skorokhodov Profile
Ivan Skorokhodov

@isskoro

2,127
Followers
422
Following
30
Media
192
Statuses

Research Scientist @Snap , working on generative models

Dubai, United Arab Emirates
Joined May 2011
Don't wanna be here? Send us removal request.
@isskoro
Ivan Skorokhodov
2 years
Some writing tips for the upcoming NeurIPS deadline
Tweet media one
127
491
5K
@isskoro
Ivan Skorokhodov
2 years
Applying for ML research internships is crazy. Last October, I had 2 MS degrees, 3 first-author papers (iclr/cvpr/iccv) and 4 years of engineering experience, applied in ~15 FAANGs, and the only company I heard back from was DeepMind who rejected me in 1.5 days
54
123
1K
@isskoro
Ivan Skorokhodov
5 months
This paper has received significantly less attention than it deserves, so let me shed a bit more light on it and describe why it's so good: 1. It turns out that the classical U-Net image diffusion backbone, which the entire community has been happily building upon during the…
@_akhaliq
AK
5 months
Analyzing and Improving the Training Dynamics of Diffusion Models paper page: Diffusion models currently dominate the field of data-driven image synthesis with their unparalleled scaling to large datasets. In this paper, we identify and rectify several…
Tweet media one
2
108
532
15
244
1K
@isskoro
Ivan Skorokhodov
1 year
I've just defended my PhD, and will now get a dog and go live in a forest
Tweet media one
23
12
564
@isskoro
Ivan Skorokhodov
2 years
A very bitter "piece of wisdom" which I carried out from this year internships round is that it's meaningless to apply to top-tier labs without a referral: you simply won't be reviewed regardless of your profile
15
17
414
@isskoro
Ivan Skorokhodov
3 months
Back in the days when I was just starting in deep learning, I got wildly blown away by unsupervised machine translation (starting from ). Imagine you travel to an alien planet, sneak into a library, steal all the books, and bring them here on Earth. Now…
Tweet media one
@sp_monte_carlo
Sam Power
3 months
what paper (not your own, maybe not even in your own area) can you not stop telling people about?
88
44
451
8
37
329
@isskoro
Ivan Skorokhodov
8 months
The sad truth about modern ML research is that it crazily incentivizes meeting hopping instead of doing actual science. Someone who spends 40 h/week in meetings for 40 different projects will have a more successful career than someone who grinds 40 h/week on a single project
@_jasonwei
Jason Wei
8 months
Hot take: what if Google Scholar reported two new metrics: (1) median citations per paper and (2) *percent* of papers with 100+ citations? I computed these metrics for some ~200 senior AI researchers: see . The top researchers by median citations per paper…
35
62
477
10
19
298
@isskoro
Ivan Skorokhodov
2 years
What do I need to pass screening? Turing Award?
6
1
237
@isskoro
Ivan Skorokhodov
7 months
Our internship applications at Snap for 2024 are now open! . We cover a wide range of topics from multi-modal LLMs and efficient DL to image/video/3D generation and editing
8
35
198
@isskoro
Ivan Skorokhodov
2 years
What "saved" me in the end, is that in 2 of these companies there were people who were interested in my research outside of my application flow. They helped me to get through the screening and that led to 2/2 offers
2
2
145
@isskoro
Ivan Skorokhodov
1 year
Today should be my first day as a research intern at Meta. But guess what? I didn't get the visa and the internship got canceled.
19
6
140
@isskoro
Ivan Skorokhodov
1 year
If you've ever used an off-the-shelf monocular depth estimator to learn 3D geometry, you likely suffered from "skewed" 3D shapes — because the predicted depth was relative. But looks like we now finally have a general-purpose metric depth estimator
@_akhaliq
AK
1 year
ZoeDepth: Zero-shot Transfer by Combining Relative and Metric Depth abs:
Tweet media one
3
31
223
0
19
90
@isskoro
Ivan Skorokhodov
6 months
When someone asks you whether it became a bit harder to publish in ML nowadays — show this ICLR'14 paper (with 1.3k+ citations now) by Goodfellow et al. which tests whether non-linearities or dropout affect forgetting of an MLP classifier (tldr: not much)
Tweet media one
7
9
79
@isskoro
Ivan Skorokhodov
7 months
I hope everyone attending ICCV has a great time! For me, it's the 3rd conference (hello NeurIPS, hello CVPR) in the past 9 months that I'll have to skip due to visa problems
4
3
76
@isskoro
Ivan Skorokhodov
5 months
I hope everyone attending NeurIPS has a great time! For me, it's the 4th conference in the past year that I'll have to skip due to visa problems
@isskoro
Ivan Skorokhodov
7 months
I hope everyone attending ICCV has a great time! For me, it's the 3rd conference (hello NeurIPS, hello CVPR) in the past 9 months that I'll have to skip due to visa problems
4
3
76
8
1
54
@isskoro
Ivan Skorokhodov
6 months
How to tell if a conference is a dumping ground for rejected papers: > We emphasize that Eurographics is not a "dumping ground" for papers rejected from SIGGRAPH, SIGGRAPH ASIA, TOG, or any vision conferences like CVPR, ICCV, or ECCV
Tweet media one
1
2
42
@isskoro
Ivan Skorokhodov
4 months
There is a legend at KAUST which says that if you beat Jurgen in benchpress, you become the new Head of AI
@SchmidhuberAI
Jürgen Schmidhuber
4 months
The GOAT of tennis @DjokerNole said: "35 is the new 25.” I say: “60 is the new 35.” AI research has kept me strong and healthy. AI could work wonders for you, too!
Tweet media one
167
150
2K
0
0
27
@isskoro
Ivan Skorokhodov
5 months
@kusichan Mitigating SGD noise is my current working hypothesis for EMA as well, but after I saw quite large EMA widths (e.g., BigGAN/iDDPM are trained for 125-2000K steps and use ema beta = 0.9999 which is ~6.9K training steps for the halflife), I also started pondering about some…
1
2
24
@isskoro
Ivan Skorokhodov
1 year
Ah, and I also couldn't come to NeurIPS to present my first NeurIPS paper
3
0
24
@isskoro
Ivan Skorokhodov
2 years
Just wanted to share (better late than never) that StyleGAN-V was accepted to CVPR 2022. In this work, we built a very simple video generator that trains as fast as StyleGAN2 and achieves the same image quality. A very nice collaboration with @SergeyTulyakov and @moElhoseiny
@_akhaliq
AK
2 years
StyleGAN-V: A Continuous Video Generator with the Price, Image Quality and Perks of StyleGAN2 abs: project page: can generate arbitrarily long videos at arbitrary high frame rate
5
57
327
0
1
23
@isskoro
Ivan Skorokhodov
2 years
It's that time of the year again, so I want to remind you all about the existence of this wonderful "How to write a good CVPR submission" presentation by Bill Freeman:
Tweet media one
1
3
20
@isskoro
Ivan Skorokhodov
7 months
Snap expands rapidly in the Middle East, and Dubai is a city of opportunities. In just 4 months of my employment here at Snap, I went from being an Administrator to a Software Engineer, and then to an Executive Director. Wondering how?
Tweet media one
6
0
20
@isskoro
Ivan Skorokhodov
2 years
@_akhaliq It’s not human-level performance if the robot does not complain after each missed shot that it just didnt practice for a long time
1
1
20
@isskoro
Ivan Skorokhodov
3 months
That was a fun project to work on, more stuff is coming soon!
@_akhaliq
AK
3 months
Snap Video Scaled Spatiotemporal Transformers for Text-to-Video Synthesis Contemporary models for generating images show remarkable quality and versatility. Swayed by these advantages, the research community repurposes them to generate videos. Since video content is highly…
3
32
212
0
0
19
@isskoro
Ivan Skorokhodov
1 year
Ok, looks like I found the best way to describe NeRF in the related work section
Tweet media one
1
1
19
@isskoro
Ivan Skorokhodov
2 years
@EsmalHaj if p(y) = N(y | \mu(x), \sigma), then -log p(y) = (y - \mu(x))^2 * const + const (here, \sigma is assumed to be constant)
0
0
16
@isskoro
Ivan Skorokhodov
2 years
@TheNerdStation The most practically influential GAN paper is (imho) StyleGAN2-ADA (which my friends in the industry build a lot on top), it was published in (late) 2020. The current trend is 3D-aware GANs (see EG3D), it should open new cool applications, but they are not mature yet
2
0
15
@isskoro
Ivan Skorokhodov
1 year
Thanks to @peter_wonka and @moElhoseiny for guiding me along the way. Thanks to @BernardSGhanem , Wolfgang Heidrich, and @MattNiessner for serving on my committee. Thanks to all the collaborators, teammates, friends, and family for their help and support.
3
0
13
@isskoro
Ivan Skorokhodov
2 months
Quite a neat DreamFusion follow-up on 3D layout learning. The most surprising part for me here is that the authors managed to make direct optimization of a 3D layout actually work. In multiple previous projects (for both 2D and 3D), we'd been exploring a similar idea, but were…
@_akhaliq
AK
3 months
Disentangled 3D Scene Generation with Layout Learning introduce a method to generate 3D scenes that are disentangled into their component objects. This disentanglement is unsupervised, relying only on the knowledge of a large pretrained text-to-image model. Our key insight is…
Tweet media one
2
13
90
1
2
13
@isskoro
Ivan Skorokhodov
2 years
@ameet_rahane I was applying for ML research internships in ~15 different top-tier tech companies from Google/Facebook/Microsoft/Apple/Amazon/etc to Qualcomm/IBM/Intel/etc to Disney/ByteDance/etc
2
0
13
@isskoro
Ivan Skorokhodov
8 months
If you are tired of the new GPT-4 twitter hype frenzy, then here is the good old StyleGAN2 from 2019 bravely saving us all from the AI apocalypse
Tweet media one
0
1
13
@isskoro
Ivan Skorokhodov
2 years
That's crazy, @dakovalev1 has 5 first-author papers at #NeurIPS2022 . Is it the largest amount in the entire NeurIPS history?
@peter_richtarik
Peter Richtarik
2 years
I am very very proud of my team at KAUST who have done a wonderful job by authoring several super exciting papers that were just accepted to the #NeurIPS2022 conference. Check them out! Also, I am hiring interns, students, postdocs & research scientists!
Tweet media one
9
27
266
0
1
12
@isskoro
Ivan Skorokhodov
2 years
Modern ML is like:
Tweet media one
0
1
12
@isskoro
Ivan Skorokhodov
2 years
@jbhuang0604 @CVPR This happened while I was uploading my paper, apparently CMT preferred rather to die than to take my submission. I'm sorry for this guys..
0
0
11
@isskoro
Ivan Skorokhodov
3 years
@JosePotatoKing @deliprao @huggingface In Jax, it is easier to develop hypernetworks or other models which parameters are a function of smth (for example, in MAML, our parameters inside the inner loop depend on the GD updates). In Pytorch, nn.Module does not allow any external intervention into parameter updates
1
0
11
@isskoro
Ivan Skorokhodov
2 years
Frankly, it feels quite sad that the community has ridiculed that consciousness tweet. It was such a nice idea to ponder about: what sorts of computation consciousness involves and what sorts of "reasoning" happen inside big LMs. And now you gonna be laughed at for thinking of it
0
0
10
@isskoro
Ivan Skorokhodov
1 year
ChatGPT can write stand-ups
Tweet media one
0
0
10
@isskoro
Ivan Skorokhodov
8 months
when your PhD advisor is a bit pushy
Tweet media one
2
0
10
@isskoro
Ivan Skorokhodov
3 years
@deliprao @huggingface I think Jax will become popular in industry because 1) Google will push it like crazy and 2) too many good models by Google/DeepMind are now being released in Jax
0
2
10
@isskoro
Ivan Skorokhodov
5 months
Saudi Arabia lives in 2123
Tweet media one
0
0
9
@isskoro
Ivan Skorokhodov
3 months
CVPR reviews bingo challenge! If your paper hits 5 of these, then it gets an oral
Tweet media one
0
0
9
@isskoro
Ivan Skorokhodov
2 years
This is insane
@_akhaliq
AK
2 years
Instant Neural Graphics Primitives with a Multiresolution Hash Encoding paper: project page: github:
21
359
2K
0
1
8
@isskoro
Ivan Skorokhodov
6 months
@jon_barron @ylecun @geoffreyhinton Wow, I am frankly surprised by the results of this poll: my feed contains so many calls for AI “safety”/doomerism that I have not expected it to have so little support in practice
5
0
7
@isskoro
Ivan Skorokhodov
2 years
its so lame doing phd during the pandemic. i wanna chill out at conferences watching bengio with ma boyz, not sit in the lab all day
1
0
8
@isskoro
Ivan Skorokhodov
2 years
If you (just as me) wonder why your #CVPR2022 reviews are not going well, that's because *both* Mercury and Venus are going retrograde right now
0
0
8
@isskoro
Ivan Skorokhodov
7 months
Well, it’s just the quirks of visa paperwork here. Follow me for more career advice.
2
0
8
@isskoro
Ivan Skorokhodov
9 months
@pfau For single-scene reconstruction, NeRF stopped being SotA within months after its release. But the power of NNs comes from scale. Once the community will learn how to do massive pre-training/knowledge sharing between scenes, NN-based 3D representations will shine
0
0
7
@isskoro
Ivan Skorokhodov
2 months
Well, looks like GPT-4.5 is getting released soon
@AnthropicAI
Anthropic
2 months
Today, we're announcing Claude 3, our next generation of AI models. The three state-of-the-art models—Claude 3 Opus, Claude 3 Sonnet, and Claude 3 Haiku—set new industry benchmarks across reasoning, math, coding, multilingual understanding, and vision.
Tweet media one
559
2K
10K
1
1
6
@isskoro
Ivan Skorokhodov
2 years
I really love the research pace in NeRF-based generative models. GRAM (CVPR’22 oral work with cool geometry) is not fully released yet, but already outperformed by at least x1.5 in terms of quality and x3.5 in terms of training cost
@_akhaliq
AK
2 years
EpiGRAF: Rethinking training of 3D GANs abs: project page:
0
31
139
0
0
6
@isskoro
Ivan Skorokhodov
2 years
Today I learned that despite all the recent progress on NeRF-like surface reconstruction methods (IDR/UNISORF/VolSDF/NeuS/etc), good ol' COLMAP still remains to be SotA for some benchmarks
Tweet media one
1
1
6
@isskoro
Ivan Skorokhodov
2 years
@ducha_aiki Well, the original BatchNorm paper falls into this category. I dont think that having an explanation of an observed phenomena is a necessary property of a good paper
0
0
6
@isskoro
Ivan Skorokhodov
1 year
@hardmaru Why do you attribute NeRF to Google Research? As far as I understand, only 1 out of 6 authors was affiliated with it at that time (J. Barron), the majority were from Berkeley
0
1
6
@isskoro
Ivan Skorokhodov
7 months
You are free to work on either pure or applied research, targeting scientific venues, production, or both! Apply, tell your friends, and don't hesitate to reach out if you have any questions!
0
0
5
@isskoro
Ivan Skorokhodov
2 years
@sangwoomo You are very kind to saying this, but I need to admit that my current works are nowhere near "well-known and impactful" :) Being non-US indeed feels like an issue sometimes (though I do not really know whether it's a problem). Wish you luck with your applications!
0
0
6
@isskoro
Ivan Skorokhodov
1 year
Oh man
Tweet media one
0
0
5
@isskoro
Ivan Skorokhodov
6 months
@yaroslavvb @dpkingma Agreed, citation parsers are about to hit uint64 overflow
0
0
5
@isskoro
Ivan Skorokhodov
3 years
@roydanroy Oh, I am sorry, I was sincerely unaware that you co-authored this :)
0
0
5
@isskoro
Ivan Skorokhodov
3 months
before training, i shuffle the dataset 7 times so that the entropy is higher
1
0
5
@isskoro
Ivan Skorokhodov
2 years
@ilyasut Is it then the act of murder to delete their weights?
0
1
5
@isskoro
Ivan Skorokhodov
1 year
Tweet media one
0
1
5
@isskoro
Ivan Skorokhodov
3 years
@ak92501 I wish ICLR also asked to report the amount of compute spent on the experiments (like NeurIPS did)
0
0
5
@isskoro
Ivan Skorokhodov
3 years
@roydanroy Scaling problems of LTH were also noted by (ICLR'19), but then (LTH authors) showed that LTH *does* scale to ImageNet, but when a different pruning technique is used
1
0
4
@isskoro
Ivan Skorokhodov
5 months
@jatentaki That's a good question, sounds possible. But one argument in the paper is that for diffusion models, it is particularly problematic because their objective is more "chaotic" than in standard tasks
0
0
3
@isskoro
Ivan Skorokhodov
2 years
@pratik_ratadiya Glad that you managed to get the position! It’s not easy, I fully agree. Congratulations :)
1
0
4
@isskoro
Ivan Skorokhodov
2 years
@jxmnop I think that the best example here is GPNN () outperforming SinGAN (ICCV’19 best paper) by a large margin, despite SinGAN being a “field-opener” in some sense
0
0
4
@isskoro
Ivan Skorokhodov
3 years
Damn, StyleGAN3 dropped the usage of DistributedDataParallel and performs the reduction manually: . Is this the way now?
0
0
4
@isskoro
Ivan Skorokhodov
8 months
@jxmnop At Snap Research, we now have some LLM training projects and can host PhD interns throughout the whole year (i.e., not only in summer)
0
0
4
@isskoro
Ivan Skorokhodov
1 year
Tweet media one
0
0
4
@isskoro
Ivan Skorokhodov
2 years
@BartWronsk That was also shown in StyleGAN-V/DIGAN papers recently: one can generate more or less decent videos (even non-autoregressively) with very loose dependence between frames
0
0
3
@isskoro
Ivan Skorokhodov
2 years
@devpramod95 The best possible way which comes into my head is to talk to people about their research at a conference (this will show that you are well interested in what they do)
1
0
3
@isskoro
Ivan Skorokhodov
3 years
@nicvadivelu Also, a weird thing about pytorch default init for Linear/Conv2d is that they implicitly assume that you use LeakyReLU with negative_slope = sqrt(5) (sic!). This makes the signal vanish the deeper you go, which contradicts the whole Xavier/He's init idea..
Tweet media one
0
0
3
@isskoro
Ivan Skorokhodov
2 years
@deliprao I would use smth like “This result was found on a clay tablet excavated by an archeological team in the ruins of a Sumerian town”
0
0
3
@isskoro
Ivan Skorokhodov
3 years
So in total I had to try 3 different vaccines and observed that each one feels very differently after a shot: - pfizer: you feel dizzy and live in a wonderland - sputnikv: you are exhausted, have fever and start calling everyone "comrade" - astrazeneca: you are hit by a train
0
0
3
@isskoro
Ivan Skorokhodov
2 years
@AbdalRameen After thinking about it a bit more, I suspect that many potential NN properties might be lost due to the use of such learnable positional embeddings (like, conditioning, weight space interpolations, etc.)
1
0
3
@isskoro
Ivan Skorokhodov
3 years
@rdeazambuja78 @arankomatsuzaki @unixpickle @prafdhar The most popular one is the average distance to a perceptual nearest neighbour: for a batch of generated images you find their nearest (wrt LPIPS distance) neighbours in the training set and report the average distance. Authors actually showed a similar thing in Figure 7
0
0
3
@isskoro
Ivan Skorokhodov
3 years
@ChrisChoy208 @vadimkantorov Last year, when NeRF came out, I subscribed to it on google scholar thinking that it will not hurt to have a paper or two on neural representations in my inbox from time to time But now it’s 10+ followups per week... Guys, have mercy, I cant read that fast
0
0
3
@isskoro
Ivan Skorokhodov
3 years
@JFPuget @JosePotatoKing @deliprao @huggingface This would work similar to wrapping it up in nn.Parameter(...): it will detach the tensor from the graph. does not propagate the gradient further during a backward pass
Tweet media one
1
0
3
@isskoro
Ivan Skorokhodov
2 years
@unixpickle We did some shape interpolations in EpiGRAF (), but the dataset here (Megascans Food) is quite small (<150 objects), so the interpolations are quite abrupt (the generator was overfitting)
0
0
2
@isskoro
Ivan Skorokhodov
2 years
@EladRichardson @peerxiv People do a ton of reviewing for the main conferences. If you will somehow allow them to reuse their reviews for your platform, this might simplify the cold start issue a lot
0
0
2
@isskoro
Ivan Skorokhodov
2 years
@SharkclawV I was applying through companies' websites or their conferences referral links. This does not feel "cold" to be honest. "Cold applying" to me is when the other side does not expect you contacting them
1
1
2
@isskoro
Ivan Skorokhodov
2 years
@unixpickle @ak92501 While I agree with what you said (I was reading up on diffusion models recently), I’v just realized that the social media ban by CVPR does not leave the authors here any chance to defend/justify themselves. @CSProfKGD what should authors do in such cases?
1
0
2
@isskoro
Ivan Skorokhodov
2 years
I read quite some articles/papers/presentations on paper writing and I feel it is my favorite one. It might be obvious to experienced researchers, but it has influenced my attitude both towards the writing and project selection in general
1
0
2
@isskoro
Ivan Skorokhodov
3 years
@JosePotatoKing @JFPuget @deliprao @huggingface In this case, gradient will not propagate to hypernetwork.weight. You can do: ``` output.sum().backward() print("Gradient is", hypernetwork.weight.grad) ``` to see this
0
0
2
@isskoro
Ivan Skorokhodov
1 year
@Mishok2000 It might be, but really hard to tell, since there is too many randomness in the process. In March, I got the same J-1 visa to Snap after just 1 month of waiting
1
0
2
@isskoro
Ivan Skorokhodov
3 years
@Buntworthy This is super similar to Swapping AE (from NeurIPS'20):
Tweet media one
1
0
2
@isskoro
Ivan Skorokhodov
2 years
@roydanroy This series of tweets that you convey is extremely deep and enlightening. Reading one such tweet is equivalent to doing one year of PhD. Bengio bless the day I subscribed to you
0
0
2
@isskoro
Ivan Skorokhodov
2 years
First time reviewing and received just one paper to review. Not sure what “reviewing burden” people were talking about
2
0
2
@isskoro
Ivan Skorokhodov
3 years
@JFPuget This is what I appreciate a lot in the NVidia’s stylegan2-ada-pytorch repo: they do validation after the very first iteration. At first I thought it to be a weird idea but then it saved me from way too many bugs and i really liked it
0
1
2
@isskoro
Ivan Skorokhodov
2 years
@Buntworthy gpustat -i 1.0
0
0
2
@isskoro
Ivan Skorokhodov
3 years
@kkitani > there is no way you can fit the entire watermelon in your mouth at once every phd student in their first year: challenge accepted.
0
0
2
@isskoro
Ivan Skorokhodov
7 months
@pratham_grover_ Yes, you are not even required to be a student
0
0
1
@isskoro
Ivan Skorokhodov
1 year
@Mishok2000 I know for sure that they get suspicious when you apply from your non-residence country (which is not my case). I heard that applying the second time for a J-1 is also suspicious to them
1
0
2
@isskoro
Ivan Skorokhodov
2 years
@Michael_J_Black For NeurIPS'20, one reviewer said that our work lacks novelty because: "this paper does not introduce a new model architecture"
1
0
2
@isskoro
Ivan Skorokhodov
3 years
@JFPuget @JosePotatoKing @deliprao @huggingface I have not worked with Jax much and thus cannot see where else it can be better than Pytorch. One thing is that it's more numpy-like, so it should be easier for adjacent fields (optimization/stats/physics/etc) to start using GPUs/autograd
0
0
2
@isskoro
Ivan Skorokhodov
2 years
@docmilanfar I think that the main benefit of using NNs under the hood is that it brings all the NN goodies with it, i.e. easy opportunity to do pretraining/few-shot, building generative models over nerfs, all sorts of conditioning, etc
0
0
2
@isskoro
Ivan Skorokhodov
2 years
@aryehazan Find the greatest element in the Matrix and return it
0
0
1