Deepak Pathak @pathak2206 Twitter profile | Pikagi

Pikagi

Deepak Pathak

@pathak2206

15,554

Followers

316

Following

150

Media

583

Statuses

I study topics in AI (machine learning, robotics & computer vision).

Pittsburgh, PA

https://t.co/al0Tk2WQdI

Joined May 2013

Don't wanna be here? Send us removal request.

Pinned Tweet

@pathak2206

Deepak Pathak

7 months

Even after 4yrs of locomotion research, we keep getting surprised by how far we can push the limits of legged robots! We report a major update 🚀🤖 Extreme Parkour: extremely long & high jumps, ramp, handstand, etc. all with a single neural net! 🧵(1/n)

27

232

1K

Last Seen Profiles

@Janakir52529343

@BevHunt72

@BeswickBrothers

@DanganronpaSH

@TheGamingLab_

@LisaLov11240652

@surgabaya

@qPlfZ8P92ZmXFdN

@TswanaGirl_Mpho

@Tsuyoko2020

@__sereto_K_

@baitomist

@InvestAbuDhabi

@ZenZeroTTV

@WePulseOfficial

@The_FOH

@andreaguala6

@WinCathedral

@TotalKroos

@gngjn08627852

@Rotee672207

@Xavierkeonmac

@TheThroneHoops

@Helgelending1

@TrendX_official

@Zetsu_Bou_God

@jibby7777777

@TheJosTurner

@1chaingangg

@BobStoopspart2

@101_dmas

@RachelL75909245

@akiramuracco

@maviswancy22697

@YOKOTA_Ishin

@nobu86651069

@pathak2206

Deepak Pathak

11 months

🤖 Robotics often faces a chicken and egg problem: no web-scale robot data for training (unlike CV or NLP) b/c robots aren't deployed yet & vice-versa. Introducing VRB: Use large-scale human videos to train a *general-purpose* affordance model to jumpstart any robotics paradigm!

9

135

698

@pathak2206

Deepak Pathak

2 years

LLMs like GPT-3 and Codex contain rich world knowledge. In this fun study, we ask if GPT like models can plan actions for embodied agents. Turns out, with apt sanity checks, even vanilla LLMs without any finetuning can generate good high-level plans given a low-level controller.

11

164

1K

@pathak2206

Deepak Pathak

4 years

RL gets specific to the robot it is trained on. Can a policy be trained to control many agents? Turns out, training (shared) policy for each motor instead of whole robot not only achieves SOTA at train but also transfers to unseen agents w/o fine-tuning!

7

238

965

@pathak2206

Deepak Pathak

1 year

After 3yrs of locomotion research, we report a major update in our #CoRL2022 (Oral) paper: vision-based locomotion. Our small, safe, low-cost robot can walk almost any terrain: high stairs, stepping stones, gaps, rocks. Stair for this robot is like climbing walls for humans.

34

159

900

@pathak2206

Deepak Pathak

2 years

How can we enable robots to perform diverse tasks? Designing rewards or demos for each task is not scalable. We propose WHIRL which learns by watching a single human video followed by autonomous exploration *directly* in the real world (no simulation)!

15

152

754

@pathak2206

Deepak Pathak

8 months

Robotic hands are daunting -- costly yet super fragile. After yrs of frustration, we decided to make one that is better, stronger & anyone can build! Open sourcing LEAP Hand 🚀🤖 - low cost ($2K) - 3D printed. Easy to assemble (3hr) - sim2real code etc.

17

125

721

@pathak2206

Deepak Pathak

4 years

RL agents get specific to tasks they are trained on. What if we remove the task itself during training? Turns out, a self-supervised planning agent can both explore efficiently & achieve SOTA on test tasks w/ zero or few samples in DMControl from images!

10

168

681

@pathak2206

Deepak Pathak

5 years

I'm excited to share that I'll be joining CMU ( @CarnegieMellon ) as an Assistant Professor in the School of Computer Science ( @SCSatCMU ) in 2020! Grateful to my mentors and friends in making this possible. Looking forward to the exciting years ahead at @CMU_Robotics and @mldcmu !

Tweet media one

23

11

590

@pathak2206

Deepak Pathak

3 years

How could we enable an agent to perform many tasks? Supervising for every new task is impractical. We present Latent Explorer Achiever (LEXA) that explores by discovering goals far beyond the frontier and then achieves test tasks, specified via images, in a zero-shot manner.

4

66

412

@pathak2206

Deepak Pathak

1 year

ML datasets have grown from 1M to 5B images but are still tiny compared to Internet where billions are uploaded per day. Wish you could scale to entire web? 🌎Internet Explorer🌏✨: an online agent that, given a task, learns on the web, self-supervised!

8

90

367

@pathak2206

Deepak Pathak

2 years

Humans manipulate objects based on size & shape. Can such geometric understanding enable generalization in dexterous manipulation? Turns out, a simple geometry-ware multi-task policy can not only manipulate 100+ objects but also outperform single-task RL experts for each obj. 1/

1

69

370

@pathak2206

Deepak Pathak

1 year

While we have made progress towards replicating the agility of animal mobility in robots, legs aren't just for walking, they are extended arms! Our #ICRA 2023 paper enables legs to act as manipulators for agile tasks: climbing walls, pressing button etc.

3

73

357

@pathak2206

Deepak Pathak

2 years

An arm can increase the utility of legged robots. But due to high dimensionality, most prior methods decouple learning for legs & arm. In #CoRL '22 (Oral), we present an end-to-end approach for *whole-body control* to get dynamic behaviors on the robot.

7

47

299

@pathak2206

Deepak Pathak

1 year

After working on curiosity-driven learning for many years, we finally scale it to real robot control. In our #ICRA2023 paper, we present ALAN which explores w/o any rewards to collect its own real-world data & then repurposes its experience to achieve goals at deployment.

2

57

296

@pathak2206

Deepak Pathak

2 years

Excited to report our progress on agile locomotion! In CoRL'21 paper, we simplify RMA rewards with just an energy term motivated by biomechanics. Optimal gaits *emerge* across speeds w/o *any* priors like high-speed galloping with emergent flight phase!!

4

35

247

@pathak2206

Deepak Pathak

4 years

Thank you, @GoogleAI ! So excited to setup my lab at @SCSatCMU .

@GoogleAI

Google AI

4 years

Congratulations to the recipients of the 2019 Google Faculty Research Awards! We received over 900 proposals from universities worldwide, covering diverse research areas such as #MachineLearning , #HumanComputerInteraction , #DistributedComputing and more.

11

139

660

8

8

242

@pathak2206

Deepak Pathak

3 years

Excited to share Worldsheet, a method to synthesize novel views with large camera changes from a *single* image. Turns out, simply shrink-wrapping a mesh sheet onto the image captures 3D well enough to render photorealistic far-away views. w/ @RonghangHu

7

45

241

@pathak2206

Deepak Pathak

2 years

Attending first in-person conf since the pandemic at #CVPR2022 . We gave live demos of our robots during my talk at Open-World Vision workshop. The convention center mostly had dull flat ground, so we had to find scraps and be creative with them to build "difficult" terrains! 😅

2

17

215

@pathak2206

Deepak Pathak

2 years

Today at #CVPR22 , we present our paper on visual navigation with legged robots in the morning. Our robot learns to walk (low-level) and plans (low-level) the path to a goal avoiding dynamic as well as "invisible" obstacles thanks to the coupling of vision with proprioception.

2

20

205

@pathak2206

Deepak Pathak

6 years

We have now released the training source code for our paper on "Large-Scale Study of Curiosity-Driven Learning". Check it out! Github Link: w/ @OpenAI

Tweet card media

Large-Scale Study of Curiosity-Driven Learning

Burda*, Edwards*, Pathak* et.al. (* equal contribution, alphabetical order) Large-Scale Study of Curiosity-Driven Learning. 2018.

pathak22.github.io

1

58

201

@pathak2206

Deepak Pathak

2 years

How can we reconstruct high-fidelity 3D from a *single* 2D image? That too w/o any 3D supervision during training? Our new #CVPR2022 paper, TARS recovers 3D shape & correspondence from a single image. Trained on just single-view internet images w/ poses.

3

25

188

@pathak2206

Deepak Pathak

5 years

Influenced by everything from @RichardDawkins to Karl Sims, we wanted to look at generalization in the context of primitive agents (think single-cell organisms to multi-celled). What we have so far is simplistic, but hopefully, we can now push it further!

4

40

179

@pathak2206

Deepak Pathak

1 year

Wonderful week at @corl_conf . We carried 3 robots across the world from CMU to New Zealand for live demos of both papers. Congratulations to @anag004 & @ashishkr9311 for winning the Best Systems Paper Award and to @zipengfu & @xuxin_cheng for being one of the 4 finalists!! 😇

1

11

167

@pathak2206

Deepak Pathak

4 years

Most paradigms in imitation learning require humans to manually move the robot to collect demonstrations for each new task. In @NeurIPSConf '19 paper, we train a robot to handle new scenarios by just watching a video in third-person view! w/ @pratyusha_PS

Tweet media one

4

33

157

@pathak2206

Deepak Pathak

6 months

Live demos of our parkour robot in Atlanta during @CoRL2023 . 🤖🚀 Check out this uncut 2mins clip of our robot nonstop climbing, leaping across gaps, and jumping down from boxes! More videos: A fun story below how we overcame hardware issues onsite👇

2

12

156

@pathak2206

Deepak Pathak

10 months

Live demos of our new low-cost dexterous hand by @kenny__shaw & @anag004 at #RSS2023 . It can be built from scratch for < 2K$. Open source release, assembling tutorials & paper coming soon. Thanks @petrenko_ai for sharing the video. Turn the sound on to hear Aleks' reaction. :)

5

22

147

@pathak2206

Deepak Pathak

1 year

Thank you @DorsaSadigh for visiting CMU and giving a great talk. It was a full house both in-person and zoom! Thoroughly enjoyed all the discussions.

Tweet media one

Tweet media two

3

8

120

@pathak2206

Deepak Pathak

5 months

Classic generative vs discriminative debate has been long-standing in the ML community. Our #NeurIPS2023 paper shows how to get the best of both worlds -- use the generative model feedback to adapt a discriminative model at test time. Robust on OOD data!

2

26

114

@pathak2206

Deepak Pathak

3 years

Robot control from raw images needs fine-grained object pose/location in 3D while keeping the input low-dimensional. Keypoints offer a solution but how to find useful ones for control w/o supervision? Our #ICML2021 work discovers visual *3D* Keypoints *jointly* with control. 1/n

1

22

108

@pathak2206

Deepak Pathak

5 years

Curious Robots!! Inspired by the classic Query-by-Committee [Seung et al. '92], we reward the agent to explore where its own world models disagree. It can be optimized in a differentiable manner --w/o using RL-- hence scalable to real robots! In ICML'19: .

Tweet card media

Self-Supervised Exploration via Disagreement

Pathak*, Gandhi*, Gupta. Self-Supervised Exploration via Disagreement. ICML 2019.

pathak22.github.io

3

23

104

@pathak2206

Deepak Pathak

4 years

Autoregressive models generate images pixel-by-pixel in a fixed order. But chain rule is order agnostic! Can we use different orders at train+test keeping the efficiency of CNNs? Checkout our Locally Masked Convolution. Short summary below👇 Paper & code:

Tweet media one

@ajayj_

Ajay Jain

4 years

PixelCNNs generate images pixel-by-pixel in a fixed order. Can we choose the order? Yes! We propose Locally Masked Convolution: a simple, efficient operation for arbitrary order training+testing & more accurate likelihoods. Paper w @pathak2206 @pabbeel 1/8

2

48

192

0

13

104

@pathak2206

Deepak Pathak

8 months

Excited to share that we have open-sourced the models, code & examples for VRB! Thanks to @shikharbahl 🚀🚀 Jump-start your robotics/RL application by simply downloading the affordance model and running it *as is* on your image/video data:

Tweet card media

GitHub - shikharbahl/vrb

Contribute to shikharbahl/vrb development by creating an account on GitHub.

@pathak2206

Deepak Pathak

11 months

🤖 Robotics often faces a chicken and egg problem: no web-scale robot data for training (unlike CV or NLP) b/c robots aren't deployed yet & vice-versa. Introducing VRB: Use large-scale human videos to train a *general-purpose* affordance model to jumpstart any robotics paradigm!

9

135

698

1

23

99

@pathak2206

Deepak Pathak

6 years

We have released our paper on large-scale curiosity-driven learning. Our purely curious agents learn to play games, "walk", form rallies against each other across 54 benchmark environments without using any *external rewards*! In collaboration with @OpenAI

Tweet card media

Large-Scale Study of Curiosity-Driven Learning

Burda*, Edwards*, Pathak* et.al. (* equal contribution, alphabetical order) Large-Scale Study of Curiosity-Driven Learning. 2018.

pathak22.github.io

8

27

97

@pathak2206

Deepak Pathak

5 years

Computer Vision has seen rapid progress lately. Wondering where the field could be in next several years? Come find out. We bring together pioneers in the field to predict areas where most progress would be made & challenges that would remain open. Link:

Tweet media one

4

16

75

@pathak2206

Deepak Pathak

2 years

Pleasantly surprised and honored to receive this award from my alma mater! :) This is for all the work done with my advisors/mentors, students, and collaborators -- extremely fortunate to be in such company.

@karandi65

Abhay Karandikar

2 years

The Young Alumnus Award (YAA) is an award that aims to recognize our alumni under the age of 40 who have distinguished themselves in their chosen areas of expertise. The awards will be given on the Foundation Day (November 02) of @IITKanpur .

Tweet media one

2

4

77

3

0

74

@pathak2206

Deepak Pathak

7 months

Our #ICCV2023 work "Your Diffusion Model is Secretly a Zero-Shot Classifier", shows that repurposing generative model as discriminator performs well & is more robust on OOD data. Generative AI advances may be close to settling the classic generative vs. discriminative debate! :)

@alexlioralexli

Alex Li

@alexlioralexli

7 months

Diffusion models have amazing image creation abilities. But how well does their generative knowledge transfer to discriminative tasks? We present Diffusion Classifier: strong classification results with pretrained conditional diffusion models, *with no additional training*! 1/9

14

79

401

2

6

71

@pathak2206

Deepak Pathak

4 years

I talked to @brianchristian about curiosity 2.5yrs ago and his book is finally out. Congrats Brian! It intuitively explains a breadth of ideas with historical+future context. With over 100 interviews, it should be a great read, even for folks within AI. Looking fwd to reading it!

Tweet media one

@brianchristian

Brian Christian

@brianchristian

4 years

Breaking a 7-year Twitter silence, I'm thrilled to announce that my new book launches today: THE ALIGNMENT PROBLEM. The product of 4 years of research and 100 interviews, it tells the story of the ethics and safety movement in AI:

13

59

326

2

0

65

@pathak2206

Deepak Pathak

5 months

A generative framework for learning diverse robotic skills from undirected play data!

@lchen915

Lili

5 months

How can robots perform a wide range of skills? At #CoRL2023 , we presented PlayFusion – a language-conditioned discrete diffusion model capable of performing many different tasks! 🌐 1/n

1

33

154

0

11

62

@pathak2206

Deepak Pathak

8 months

Congratulations to former student @zipengfu for fun work and great results! proud as always... :-)

@zipengfu

Zipeng Fu

8 months

Introduce our #CoRL2023 (Oral) project: "Robot Parkour Learning" Using vision, our robots can climb over high obstacles, leap over large gaps, crawl beneath low barriers, squeeze through thin slits, and run. All done by one neural network running onboard. And it's open-source!

24

229

1K

2

2

60

@pathak2206

Deepak Pathak

2 years

A nice survey paper from @hardmaru and @yujin_tang on the emergent collective intelligence from multiagent complex systems!

@hardmaru

hardmaru

2 years

I’m super excited to see ideas from complex systems such as swarm intelligence, self-organization, and emergent behavior gain traction again in AI research. We wrote a survey of recent developments that combine ideas from deep learning and complex systems:

Tweet media one

Tweet media two

Tweet media three

Tweet media four

27

306

2K

0

3

60

@pathak2206

Deepak Pathak

3 months

Introducing a simple approach to perform complex tasks in *unseen* environments by allowing the robot to practice. The result is fully autonomous mobile manipulation for opening doors, drawers, cabinets, etc in the open world. PS: door-opening is still far from being "solved" :)

@Haoyu_Xiong_

Haoyu Xiong

3 months

Introducing Open-World Mobile Manipulation 🦾🌍 – A full-stack approach for operating articulated objects in open-ended unstructured environments: Unlocking doors with lever handles/ round knobs/ spring-loaded hinges 🔓🚪 Opening cabinets, drawers, and refrigerators 🗄️ 👇…

30

102

785

1

7

56

@pathak2206

Deepak Pathak

6 years

Released our @ICLR18 (oral) paper on "Zero-Shot Visual Imitation". Our agents learn skills by self-supervised exploration s.t. they can manipulate a rope, or navigate to a goal, by simply watching a sparse image sequence from expert once! w/ @pulkitology

Tweet card media

Zero-Shot Visual Imitation

Pathak et al. Zero-Shot Visual Imitation. In ICLR, 2018.

pathak22.github.io

1

24

57

@pathak2206

Deepak Pathak

3 years

Can we combine the long-horizon ability of classic search algorithms w/ flexibility of modern deep RL or IL methods? Checkout our NeurIPS'20 paper on Sparse Graphical Memory. Its 2-way consistency criterion allows robust long-horizon planning from images.

@MishaLaskin

Misha Laskin

3 years

New paper coming up at @NeurIPSConf - Sparse Graphical Memory for Robust Planning uses state abstractions to improve long-horizon navigation tasks from pixels! Paper: Site: Co-led by @emmons_scott , @ajayj_ , and myself. [1/N]

Tweet media one

1

16

108

2

6

57

@pathak2206

Deepak Pathak

4 months

Cool to see how many tasks even low-cost robots can perform IF controlled by intelligent "software" (that software here is a human teleoperator)... still recall feeling this deep realization when @pabbeel made this argument in his robot learning talk almost a decade ago!! :)

@zipengfu

Zipeng Fu

4 months

Mobile ALOHA's hardware is very capable. We brought it home yesterday and tried more tasks! It can: - do laundry👔👖 - self-charge⚡️ - use a vacuum - water plants🌳 - load and unload a dishwasher - use a coffee machine☕️ - obtain drinks from the fridge and open a beer🍺 - open…

407

2K

7K

1

2

55

@pathak2206

Deepak Pathak

4 years

If you're curious about the pixel generation order below in our LMConv model and how it preserves spatial neighbors across different resolutions, look at this very cool video on Pseudo-Hilbert Curve & history of space filling curves by @3blue1brown :

Tweet card media

Hilbert's Curve: Is infinite math useful?

Space-filling curves, and the connection between infinite and finite math.Help fund future projects: https://www.patreon.com/3blue1brownAn equally valuable f...

www.youtube.com

@ajayj_

Ajay Jain

4 years

Fun application: Our Locally Masked PixelCNN can generate images along a Hilbert space-filling curve. The Hilbert ordering is resolution agnostic and ensures that consecutively generated pixels are nearby! Video of unconditional generation along Hilbert curve: 2/8

1

4

31

0

13

56

@pathak2206

Deepak Pathak

3 years

Many natural phenomena, from the *motion* of planets to that of pendulums, are described by dynamical systems (e.g. ODE/PDE). Can we also use them as inductive bias to learn the *motion* of robots? Check out our NeurIPS'20 work on Neural Dynamic Policy!

Tweet card media

Neural Dynamic Policies for End-to-End Sensorimotor Learning

Bahl, Mukadam, Gupta, Pathak. Neural Dynamic Policies for End-to-End Sensorimotor Learning. NeurIPS 2020.

shikharbahl.github.io

1

10

54

@pathak2206

Deepak Pathak

1 year

Led by @anag004 & @ashishkr9311 with Jitendra Malik. Unlike past works that decouple vision and locomotion by first building a map, our robot takes ego-centric camera image directly as input to predict joint angles. No need to deal with noisy mapping, planning, or tracking. 2/

3

9

53

@pathak2206

Deepak Pathak

8 months

Training the hand to do tasks is easy! Train in simulation and deploy directly on the hand. We also open-source sim2real code to go from IsaacGym to the real world in < 1hr. Code: LEAP also comes with a Python and ROS API. 3/

1

10

54

@pathak2206

Deepak Pathak

1 month

Very cool results by @chenwang_j !! Pleased to see the LEAP hand being adopted for such diverse and dexterous manipulation tasks. You too can make one for yourself:

@chenwang_j

Chen Wang

2 months

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

21

130

620

1

5

47

@pathak2206

Deepak Pathak

4 years

We are presenting this at #NeurIPS2019 today. Drop by to know about dynamic graph networks and spooky modular agents! Talk: 4.30pm today @ West Ballroom A+B Poster # 197: 5.30pm Longer Talk: 2.15pm on Saturday (Skills workshop) Code, poster, slides etc.:

Tweet card media

Learning to Control Self-Assembling Morphologies: A Study of Genera...

Pathak, Lu, Darrell, Isola, Efros. Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity. 2019.

pathak22.github.io

@pathak2206

Deepak Pathak

5 years

Nice summary of virtual creatures in Nature Machine Intelligence by @Kriegmerica ! Article also highlights our upcoming NeurIPS 2019 paper on modular joint learning of control and morphology for zero-shot out of domain generalization. Paper:

0

10

49

0

5

53

@pathak2206

Deepak Pathak

5 years

Nice summary of virtual creatures in Nature Machine Intelligence by @Kriegmerica ! Article also highlights our upcoming NeurIPS 2019 paper on modular joint learning of control and morphology for zero-shot out of domain generalization. Paper:

@DoctorJosh

Josh Bongard

5 years

Tired of training #NeuralNetworks ? Try optimizing virtual creatures instead. @nature Machine Intelligence article by Sam Kriegman ( @Kriegmerica ).

8

40

154

0

10

49

@pathak2206

Deepak Pathak

7 months

Our robot can generalize to combinations of obstacles not seen during training and can even do a parkour course fully autonomously (video below): - Single Neural Net Policy - Vision Input - Unified Reward Function The code is *open-sourced*. Details: 6/6

3

6

50

@pathak2206

Deepak Pathak

4 years

Excited about this workshop happening today on "Self Supervised Learning: What is Next?". I will talk about why I believe learning for embodied agents is the answer. Looking forward to other talks from an amazing line of speakers!

@y_m_asano

Yuki

4 years

Looking forward to the Self-Supervised Learning Workshop we’ve organized with @chrirupp , A. Vedaldi and A. Joulin at #ECCV2020 . Join us tomorrow for our speakers: @avdnoord , P. Favaro, @CarlDoersch , A. Zisserman, I. Misra, S. Yu, A. Efros, @pathak2206 ! .

Tweet media one

2

33

108

2

3

50

@pathak2206

Deepak Pathak

7 months

Thanks to @ROBOTIS , one can now order all the LEAP Hand parts in a bundle (1-click) from robotis website: => LEAP parts: =>LEAP LITE parts: => LEAP Hand design, code, API, sim2real:

Tweet card media

LEAP Hand Bundle Lite (XL330)

ROBOTIS is a global robot solutions provider and one of the leading manufacturers of robotic hardware. ROBOTIS is the exclusive producer of the DYNAMIXEL brand of all-in-one smart servos. We specia...

@anag004

Ananye Agarwal

7 months

Great to run into Sam and others from Robotis at IROS! LEAP hand parts can be bought with a single click from their website (). None of this would be possible without @kenny__shaw who couldn't attend but was there with us in spirit :)

Tweet media one

2

2

39

1

6

48

@pathak2206

Deepak Pathak

3 years

Check out the #ICRA2021 talk by @d_yuqing & @OliviaGWatkins2 on how to alleviate the need for hand-engineering in sim2real transfer. Work w/ Trevor, @pabbeel . Delighted to find out that our paper was a finalist for Best Cognitive Robotics Paper Award! 😌

2

7

48

@pathak2206

Deepak Pathak

4 years

If you missed our Plan2Explore presentation at ICML’20, check out the latest blog post highlighting the key ideas and intuition behind the approach! CMU: BAIR:

Tweet card media

Plan2Explore: Active Model-Building for Self-Supervised Visual Reinforcement Learning

To operate successfully in unstructured open-world environments, autonomous intelligent agents need to solve many different tasks and learn new tasks quickly. Reinforcement learning has enabled...

blog.ml.cmu.edu

@_oleh

Oleg Rybkin

4 years

Can agents explore the environment and learn a world model for use in downstream tasks? Our blog post explains Plan2Explore, our agent that does just that! Paper w/ _ramanans @KostasPenn @pabbeel @danijarh @pathak2206 BAIR: CMU:

1

3

17

1

10

42

@pathak2206

Deepak Pathak

6 years

@Miles_Brundage Thanks for sharing our paper! Here is the link to the website and the video.

Tweet card media

Large-Scale Study of Curiosity-Driven Learning

Burda*, Edwards*, Pathak* et.al. (* equal contribution, alphabetical order) Large-Scale Study of Curiosity-Driven Learning. 2018.

pathak22.github.io

2

20

41

@pathak2206

Deepak Pathak

8 months

In #RSS2023 @RoboticsSciSys paper, we study how to train a *unified world model* across robot data and human videos. The idea is to use affordances as a common action space: - obtained for free in human videos - abstracts the robot's low-level actions - agnostic to embodiments

@mendonca_rl

Russell Mendonca

8 months

World models are promising for enabling general-purpose robots. But how do we train them since robot data is limited? Our solution: Shared world models on human videos and robots using affordances as the joint action space. Pre-train on human video, then fine-tune to robot tasks!

4

30

129

1

4

39

@pathak2206

Deepak Pathak

1 year

We tested our robot on stepping stones, forest hikes, trails, and abandoned hospital sites. We hope our method enables the wider application of robot dogs in homes, search and rescue, inspection, and more. Paper & Videos: 6/6

7

4

39

@pathak2206

Deepak Pathak

2 months

Wonderful to find out that our real-world robotic adaptation work got featured in Nature’s “Images of the Month” series across all areas of Science alongside case studies of plasma on the Sun’s surface, neon nerve cells, and marine life in the deep ocean:

Tweet card media

Giant plume of plasma on the Sun’s surface and more — February’s best science images

The month’s sharpest science shots, selected by Nature’s photo team.

1

3

37

@pathak2206

Deepak Pathak

7 months

This is wonderful -- reminds me of so many wonderful dinner discussions with Alyosha! Sad to miss ICCV Paris in person.

@CSProfKGD

Kosta Derpanis

7 months

Banter between Alyosha Efros and Antonio Torralba on nearest neighbours is all you need, circa 2010.

5

17

228

0

3

37

@pathak2206

Deepak Pathak

2 years

This work is being presented at #ICML2022 today by @wenlong_huang !

@pathak2206

Deepak Pathak

2 years

LLMs like GPT-3 and Codex contain rich world knowledge. In this fun study, we ask if GPT like models can plan actions for embodied agents. Turns out, with apt sanity checks, even vanilla LLMs without any finetuning can generate good high-level plans given a low-level controller.

11

164

1K

0

4

36

@pathak2206

Deepak Pathak

2 years

Join our workshop at CoRL with a live robotics demo track!

@pulkitology

Pulkit Agrawal

2 years

We are organizing the @corl_conf 2022 Workshop on “Sim-to-Real Robot Learning: Locomotion and Beyond”. Join us for an exciting set of talks and tutorials, and participate in our *live spotlight demo track*! 🌐: @pathak2206 @gabe_mrgl

Tweet media one

0

8

81

0

0

35

@pathak2206

Deepak Pathak

6 months

Check out LEAP Hand in action at @corl_conf : live demos + 2 papers on how to use it for sim2real dexterous functional grasping and real-world finetuning.

@kenny__shaw

Kenny Shaw

6 months

Visit us demoing at #CoRL2023 Poster 5 and afternoon demos today! @anag004 uses LEAP Hand for sim2real dexterous, functional grasping @aditya__kannan uses DASH Hand to learn from internet videos and real-world fine-tuning

0

12

67

2

1

35

@pathak2206

Deepak Pathak

7 months

Led by @xuxin_cheng , @teenhaci & @anag004 . We have shown end-to-end walking with vision, but extreme parkour is a different ballgame where even a single mistake can be fatal. Our sim2real training allows for precise foot adjustments to maximize mechanical advantage. 2/

1

2

34

@pathak2206

Deepak Pathak

3 years

Really wonderful to see this work on curiosity-driven learning for robotics from @DeepMind . It focuses on how to retain (and not override) the diverse skills that emerge from curious exploration.

@GoogleDeepMind

Google DeepMind

@GoogleDeepMind

3 years

Applying an off-policy curiosity method to robotic manipulation and locomotion yields impressive emerging behaviour. Can curiosity-based exploration coupled with a suitable behaviour retention mechanism be a general principle for continual learning? 1/

2

31

184

0

4

32

@pathak2206

Deepak Pathak

4 years

We note that most exploration methods retrospectively compute the novelty of observations after the agent has already reached them. We present Plan2Explore to efficiently seek out expected future novelty via planning. led by @_ramanans @_oleh w/ @KostasPenn @pabbeel @danijarh

Tweet media one

1

0

32

@pathak2206

Deepak Pathak

2 years

Since no training is required, the plans reflect the knowledge *already contained* in LLMs. It was a fun project and we hope it'll be insightful for high-level decision-making problems. Led by @wenlong_huang w/ @pabbeel and @IMordatch Paper+Code:

Tweet media one

2

3

32

@pathak2206

Deepak Pathak

3 years

Three main takeaways: - No hand-coded or heuristic control stack - Online & real-time adaptation to unseen terrains - Asynchronous design for low compute Paper & Videos: Oral Talk at #RSS2021 : 9/9

2

3

28

@pathak2206

Deepak Pathak

8 months

We have easy-to-follow assembly videos with step-by-step instructions on the website. All the parts are easily available off-the-shelf, and the CAD files are open-source. Our design is stronger and more robust than other hands. Takes 3 hours to assemble. 2/

2

3

31

@pathak2206

Deepak Pathak

4 years

Very excited to participate and looking forward to this workshop on "The Origins of Commonsense in Humans and Machines" today at CogSci 2020.

@TomerUllman

Tomer Ullman

4 years

Even though the others rejected my idea of having an AI and a 3-year-old on the panel, I'm really looking forward to our CogSci workshop on the Origins of Common Sense (tomorrow!) co-org w/ @AlisonGopnik @realkevinsmith @pathak2206 among others

Tweet media one

Tweet media two

3

12

97

0

1

29

@pathak2206

Deepak Pathak

2 years

We look at VirtualHome env from Antonio Torralba's & @FidlerSanja 's group where past attempts use human instructions for the kind of knowledge that should already be there in LLMs. However, just querying GPT results in meaningful but non-executable plans due to lack of grounding.

2

2

28

@pathak2206

Deepak Pathak

3 years

We are presenting Worldsheet at #ICCV2021 this week as Oral. Join QnA Wed & Fri. We updated the arXiv since v1: *Multi-layered* Worldsheets to autonomously handle sharp depth discontinuities/occlusions which a single sheet may fail to capture (Sec 3.5):

@pathak2206

Deepak Pathak

3 years

Excited to share Worldsheet, a method to synthesize novel views with large camera changes from a *single* image. Turns out, simply shrink-wrapping a mesh sheet onto the image captures 3D well enough to render photorealistic far-away views. w/ @RonghangHu

7

45

241

0

4

26

@pathak2206

Deepak Pathak

5 years

The video recordings of the talks in our Task-Agnostic RL Workshop at ICLR 2019 are now online!

@danijarh

Danijar Hafner

5 years

It's fantastic to see so many people being interested in task-agnostic RL and making it to our workshop yesterday. Feels well worth the effort to organize and like we actually did something good for the community :) Recordings (starts at 23:30):

Tweet media one

2

19

97

0

3

27

@pathak2206

Deepak Pathak

2 years

Intriguing study of curiosity-driven exploration in humans. Seems like humans are also prone to noisy TV problem... no wonder slot machines in casinos never go out of business! 😅

@modirshanechi

Alireza Modirshanechi

2 years

🚨🥳 Preprint alert! 🥳🚨 In our new preprint **The curse of optimism: a persistent distraction by novelty**, we show that human curiosity-driven exploration is prone to distraction by stochasticity and can be well described by novelty-driven RL! 🧵 1/7

2

25

78

0

5

27

@pathak2206

Deepak Pathak

7 months

Walking on two legs is much harder than walking on four and our robot can do both using the same basic approach. It can even go downstairs while doing a handstand! 3/

1

0

27

@pathak2206

Deepak Pathak

4 years

Excited to present this work at ICML 2020 next week! Check out our recorded long oral talk below for closer look. Long Talk : Code: Paper & videos: 8/8

Tweet card media

ICML 2020 Oral Talk: One Policy to Control Them All

ICML 2020 long oral talk for the paper titled "One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control". Source code and paper at ...

www.youtube.com

4

4

26

@pathak2206

Deepak Pathak

4 years

To estimate intrinsic novelty, we use ensemble disagreement in the learned latent space of images, which works better than alternative intrinsic rewards. More details: Code: Paper & videos: 5/5

0

4

25

@pathak2206

Deepak Pathak

5 years

Our work on "generalization via modularity in self-assembling agents" is one of the finalists at the Virtual Creatures Competition. paper: video: joint work with @luchris429 , @trevor_darrell_ , @phillip_isola , Alyosha Efros

Tweet card media

Learning to Generalize Self-Assembling Agents [with audio narration]

This video summarizes our paper on "Learning to Control Self-Assembling Morphologies: A Study of Generalization via Modularity". More details, source code, a...

www.youtube.com

@risi1979

Sebastian Risi

5 years

Congrats to the Virtual Creatures Competition finalists! @hardmaru @Kriegmerica @pathak2206 @_joelsimon If at #GECCO2019 stop by Sunday, 15:20-15:45, Club D (1F) to learn more.

1

16

57

1

3

26

@pathak2206

Deepak Pathak

5 years

This is joint work with Chris Lu, @trevor_darrell_ , @phillip_isola , and Alyosha Efros. Following is a snapshot of zero-shot generalizations, we test in these environments. More details on the webpage:

Tweet media one

1

8

26

@pathak2206

Deepak Pathak

2 years

Check out our #NeurIPS2021 Oral presentation this week on disentangling environment-centric exploration (e.g., pressing a red button should be interesting to press in every new environment) from agent-centric exploration (commonly modeled by algos like info-gain/curiosity, etc.).

@vdean314

Victoria Dean

2 years

How should agents explore and interact with the world in order to quickly learn tasks? Learn how in our @NeurIPSConf paper “Interesting Object, Curious Agent” - oral session 5, tomorrow at 7pm ET! With Simone Parisi, @pathak2206 , and Abhinav Gupta

1

11

79

2

2

27

@pathak2206

Deepak Pathak

4 years

Because the exploration is self-supervised, the resulting model is task-agnostic! Plan2Explore uses a single model to quickly solve multiple downstream tasks w/o any task-specific interaction in a zero-shot manner. SOTA performance wrt prior self-supervised methods.

Tweet media one

1

0

26

@pathak2206

Deepak Pathak

2 years

To improve executability, we translate each step to admissible action by calculating similarity b/t embeddings from RoBERTa. Translated action is appended back to prompt to generate next step. We also choose which example to use in the prompt depending on similarity. 3/4

1

1

26

@pathak2206

Deepak Pathak

4 years

We are excited to present this work at #ICML2020 today. ICML'20 full oral talk: Code: If you're registered, consider attending our virtual Q & A today (Thu, July 16) at either 8am PST or 8pm PST. Link at:

Tweet card media

GitHub - huangwl18/modular-rl: [ICML 2020] PyTorch Code for "One Policy to Control Them All: Shared...

[ICML 2020] PyTorch Code for "One Policy to Control Them All: Shared Modular Policies for Agent-Agnostic Control" - huangwl18/modular-rl

@pathak2206

Deepak Pathak

4 years

RL gets specific to the robot it is trained on. Can a policy be trained to control many agents? Turns out, training (shared) policy for each motor instead of whole robot not only achieves SOTA at train but also transfers to unseen agents w/o fine-tuning!

7

238

965

0

3

25

@pathak2206

Deepak Pathak

4 years

@MaCroPhilosophy Indeed, we later realized that we should have cited Dostoyevsky and Tolstoy there!! 😅

0

0

26

@pathak2206

Deepak Pathak

1 year

Due to no mapping & only a front camera, note how robot's rear legs recall the terrain from its memory to walk over them. Kind of how cats avoid obstacles via rear legs despite looking front. We did live demos at #CVPR2022 & will do more demos at NZ during @corl_conf in Dec. 5/

Tweet media one

Tweet media two

1

2

25

@pathak2206

Deepak Pathak

7 months

For parkour, the robot has to adjust its motions with precise eye-muscle coordination & change direction rapidly. We use a teacher-student framework. To learn direction, we use MTS (Mixed Teacher Student) where predicted direction is used only if it is close to ground truth. 4/

1

1

26

@pathak2206

Deepak Pathak

10 months

Enjoying #ICVSS2023 with great company in beautiful Sicily!

@FrancRagusa

Francesco Ragusa

10 months

Last talk of today at #icvss2023 by @pathak2206 !

Tweet media one

0

0

1

1

0

26

@pathak2206

Deepak Pathak

6 years

I am co-organizing a workshop on "Action, Perception, and Organization" at ECCV 2018. Extended abstracts of late-breaking results or in-submission/accepted papers about latest developments in perceptual organization and embodied agents are welcome. Web: .

Tweet card media

11th POCV Workshop: Action, Perception and Organization

pocv18.eecs.berkeley.edu

0

2

25

@pathak2206

Deepak Pathak

4 years

Plan2Explore is to appear at #ICML2020 today. ICML'20 full oral talk: For those who are registered, join virtual Q & A today (Wed, July 15) at either 8am PST or 7pm PST. Link at:

Tweet card media

ICML 2020 Oral Talk: Planning to Explore via Self-Supervised World...

ICML 2020 long oral talk for the paper titled "Planning to Explore via Self-Supervised World Models". Code and paper at https://ramanans1.github.io/plan2expl...

www.youtube.com

@pathak2206

Deepak Pathak

4 years

RL agents get specific to tasks they are trained on. What if we remove the task itself during training? Turns out, a self-supervised planning agent can both explore efficiently & achieve SOTA on test tasks w/ zero or few samples in DMControl from images!

10

168

681

1

8

25

@pathak2206

Deepak Pathak

2 years

We scale WHIRL to 20 different household tasks, each took 1-2 hours to train in the real world! We hope this direction could bring robots into homes someday. :) Paper: RSS2022 Talk: Led by @shikharbahl w/ Abhinav Gupta 7/7

0

3

25

@pathak2206

Deepak Pathak

7 months

Engineering rewards for complex behavior can be tricky. We use a simple inner product reward function for all tasks. This is able to overcome the exploration burden and learn emergent, optimized behavior for each task without the need for a complex curriculum. 5/

1

2

24

@pathak2206

Deepak Pathak

2 years

Shocking and extremely sad news! A huge loss to the community.

@YiMaTweets

Yi Ma

2 years

I was shocked to know that Dr. Jian Sun, my former colleague of the MSRA Visual Computing Group, has passed away. We will miss him dearly. May his soul rest in peace.

20

61

605

0

1

24

@pathak2206

Deepak Pathak

3 years

Really enjoyed giving this seminar talk at MIT and talking about our latest projects on legged robot and exploration -- which I am personally super excited about! Many thanks to @FerranAlet for hosting it.

@FerranAlet

Ferran Alet

3 years

Deepak Pathak's ( @pathak2206 ) @MIT_CSAIL EI Seminar talk is now on Youtube: There, he covered three interesting works on rapid adaptation for robot learning .

1

2

32

0

1

23

@pathak2206

Deepak Pathak

1 year

Thank you @shaneguML for sharing our work and the kind description! :)

@shaneguML

Shane Gu

1 year

ImageNet moment for robot locomotion. Congratulations @pathak2206 et al! "first, we train a policy using RL with a cheap-to-compute variant of depth image and then in phase 2 distill it into the final policy that uses depth using supervised learning."

8

83

512

1

2

23

@pathak2206

Deepak Pathak

1 year

We never specify how the leg should move and the optimal gait emerges via learning. This is important for a small robot as it does not have clearance to fold legs like humans or Spot. It learns an emergent hip abduction (extend leg sideways) behavior to climb such terrain. 3/

Tweet media one

1

2

22

@pathak2206

Deepak Pathak

2 years

We hope that this work provides insights for generalization in dexterous manipulation and that the released benchmark is useful for future research. Work led by @wenlong_huang w/ @IMordatch and @pabbeel Paper & Code & Videos: 5/5

0

0

22