Tony Z. Zhao @tonyzzhao Twitter profile | Pikagi

Pikagi

Tony Z. Zhao

@tonyzzhao

12,690

Followers

789

Following

38

Media

303

Statuses

CS PhD student @Stanford . Aspiring full-stack roboticist. Prev Deepmind, Tesla, GoogleX, Berkeley.

Stanford, CA

https://t.co/aJIMRLFwwf

Joined December 2018

Don't wanna be here? Send us removal request.

Pinned Tweet

@tonyzzhao

Tony Z. Zhao

5 months

Introducing 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Hardware! A low-cost, open-source, mobile manipulator. One of the most high-effort projects in my past 5yrs! Not possible without co-lead @zipengfu and @chelseabfinn . At the end, what's better than cooking yourself a meal with the 🤖🧑‍🍳

235

1K

5K

Last Seen Profiles

@DVC_Athletics

@IkikiLakev

@tudruj73437

@Matin27311

@marathonvet33

@jandakembangstw

@inharryscinema

@stw_pdg

@mimi_k23

@stw_pdg

@13_buuutterfly

@kuranaga15

@floydophone

@JerermyZyl1

@floydophone

@nikmat_sesaat69

@ceo_yy

@jaxonsc

@anderson_joch

@jessi_puk

@sturdyx200

@Pecintastw4050

@DillyDilworth

@dtchq_delhi

@nyon44397139453

@jandakembangstw

@crpnoubarris

@minigoogy

@AirSerBiaFTN

@hyun_huiwon_luv

@pengen_stw

@Imelda1049650

@uraharafeed

@jandakembangstw

@yooo_ky

@Aled74

@tonyzzhao

Tony Z. Zhao

1 year

Introducing ALOHA 🏖: 𝐀 𝐋ow-cost 𝐎pen-source 𝐇𝐀rdware System for Bimanual Teleoperation After 8 months iterating @stanford and 2 months working with beta users, we are finally ready to release it! Here is what ALOHA is capable of:

94

711

3K

@tonyzzhao

Tony Z. Zhao

11 months

China's progress in humanoid robots deserves more attention. The video below has <300 views on YouTube, while the robot appears to be - more agile than @Tesla 's Optimus - more dexterous than @agilityrobotics 's Digit - (likely) a lot cheaper than both

222

232

2K

@tonyzzhao

Tony Z. Zhao

2 months

Introducing 𝐀𝐋𝐎𝐇𝐀 𝐔𝐧𝐥𝐞𝐚𝐬𝐡𝐞𝐝 🌋 - Pushing the boundaries of dexterity with low-cost robots and AI. @GoogleDeepMind Finally got to share some videos after a few months. Robots are fully autonomous filmed in one continuous shot. Enjoy!

56

333

2K

@tonyzzhao

Tony Z. Zhao

5 months

Robots are not ready to take over the world yet! @zipengfu and I just compiled a video of the dumbest mistakes 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 made in the autonomous mode 🤣 We are also planning to organize some live demos after taking a break. Stay tuned!

66

219

1K

@tonyzzhao

Tony Z. Zhao

1 year

How can robots acquire fine-grained manipulation skills? Introducing ACT: Action Chunking with Transformers 🤖 Key idea: Imitation, but predict actions in chunks instead of one at a time. Here are results with only ~15min of demonstrations, running on low-cost arms:

28

215

1K

@tonyzzhao

Tony Z. Zhao

1 year

a short teaser of what we’ve been up to lately ⁦ @stanford ⁩: here is an end-to-end policy running on low-cost arms!

30

85

790

@tonyzzhao

Tony Z. Zhao

4 months

Led by @GoogleDeepMind , we present ALOHA 2 🤙: An Enhanced Low-Cost Hardware for Bimanual Teleoperation. ALOHA 2 🤙 significantly improves the durability of the original ALOHA 🏖️, enabling fleet-scale data collection on more complex tasks. As usual, everything is open-sourced!

17

152

630

@tonyzzhao

Tony Z. Zhao

5 months

Not just cooking! We made another video showing what 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 is capable of in a real home, inspired by the famous PR1 video. 2024 will be the year of robotics, and this is just the beginning!

@zipengfu

Zipeng Fu

5 months

Mobile ALOHA's hardware is very capable. We brought it home yesterday and tried more tasks! It can: - do laundry👔👖 - self-charge⚡️ - use a vacuum - water plants🌳 - load and unload a dishwasher - use a coffee machine☕️ - obtain drinks from the fridge and open a beer🍺 - open

407

2K

7K

23

82

513

@tonyzzhao

Tony Z. Zhao

5 months

We open-source all hardware and software of 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀: Tutorial: Github: Project Website:

Tweet card media

GitHub - MarkFzp/mobile-aloha: Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost...

Mobile ALOHA: Learning Bimanual Mobile Manipulation with Low-Cost Whole-Body Teleoperation - MarkFzp/mobile-aloha

7

49

315

@tonyzzhao

Tony Z. Zhao

1 year

With the advent of AGI, humans will soon be the weakest link in software industry. How can we have better coding buddies that *enhance* humans? Introducing 𝐁ug 𝐀nalysis and 𝐈dentification with enhanced 𝐓oads (BAIT), where we fit toads with contact lenses to better catch bugs

11

37

289

@tonyzzhao

Tony Z. Zhao

4 months

I wish we can come back to this tweet in a decade and be like "Hey here is when we finally cracked data collection". Low-cost, portable, hardware agnostic. I could not ask for more!

@chichengcc

Cheng Chi

4 months

Can we collect robot data without any robots? Introducing Universal Manipulation Interface (UMI) An open-source $400 system from @Stanford designed to democratize robot data collection 0 teleop -> autonomously wash dishes (precise), toss (dynamic), and fold clothes (bimanual)

42

343

2K

6

29

278

@tonyzzhao

Tony Z. Zhao

10 months

Just built another ALOHA🏖️ @Stanford ! Rumor said there are now more than 20 ALOHAs in the world 👀 Notice the new grippers? Folks at @GoogleDeepMind actually redesigned it and kind enough to open source. We will be announcing it shortly!

3

20

202

@tonyzzhao

Tony Z. Zhao

5 months

How does 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀 work? We seek to achieve a few more goals to augment the dexterity of the original 𝐀𝐋𝐎𝐇𝐀:

@tonyzzhao

Tony Z. Zhao

1 year

Introducing ALOHA 🏖: 𝐀 𝐋ow-cost 𝐎pen-source 𝐇𝐀rdware System for Bimanual Teleoperation After 8 months iterating @stanford and 2 months working with beta users, we are finally ready to release it! Here is what ALOHA is capable of:

94

711

3K

3

25

186

@tonyzzhao

Tony Z. Zhao

2 months

This is probably the most surprising result I had in 2023!

@ayzwah

Ayzaan Wahid

2 months

one time @tonyzzhao took off his sweater to try it with the model. The policy was never trained on an adult sized shirt or any type of sweaters, but we found it's able to generalize.

2

9

130

3

24

183

@tonyzzhao

Tony Z. Zhao

1 year

How to build ALOHA? We open-sourced everything about the setup, and prepared a detailed tutorial. In short: it's with off-the-shelf robots + 3D printed components. We also contacted @trossenrobotics , who agreed to manufacture and sell the whole ALOHA kit that you can buy now!

Tweet media one

Tweet media two

6

20

158

@tonyzzhao

Tony Z. Zhao

1 year

The paper is now on ArXiv. Thanks @_akhaliq !

@_akhaliq

AK

1 year

Learning Fine-Grained Bimanual Manipulation with Low-Cost Hardware abs: project page:

14

140

637

3

22

153

@tonyzzhao

Tony Z. Zhao

5 months

To achieve these goals, we mount ALOHA to a mobile base designed for warehouses: Tracer AGV It can carry 100kg, move up to 1.6m/s, while costing only $7k To allow simultaneous arms and base control, we simply tether the operator to the mobile base, i.e. backdriving the wheels.

Tweet media one

Tweet media two

9

17

150

@tonyzzhao

Tony Z. Zhao

5 months

We have so many cool results to share.. wrapping up the open-sourcing rn. Stay tuned! 🏄 🏄

@zipengfu

Zipeng Fu

5 months

Mobile ALOHA 🏄 is coming soon! Special thanks to @tonyzzhao for throwing random objects into the scene, and @chelseabfinn for the heavy pot (> 3 lbs) ! Stay tuned!

9

58

378

3

15

151

@tonyzzhao

Tony Z. Zhao

1 year

@Stanford We built ALOHA to be maximally user-friendly for researchers: it is simple, dependable and performant. The whole system costs <$20k, yet it is more capable than setups with 5-10x the price.

Tweet media one

3

8

146

@tonyzzhao

Tony Z. Zhao

5 months

So, what new skills does 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀 unlock when controlled by a neural network? Check out 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 - Learning from Co-lead @zipengfu !

@zipengfu

Zipeng Fu

5 months

Introduce 𝐌𝐨𝐛𝐢𝐥𝐞 𝐀𝐋𝐎𝐇𝐀🏄 -- Learning! With 50 demos, our robot can autonomously complete complex mobile manipulation tasks: - cook and serve shrimp🦐 - call and take elevator🛗 - store a 3Ibs pot to a two-door cabinet Open-sourced! Co-led @tonyzzhao , @chelseabfinn

187

887

4K

10

16

139

@tonyzzhao

Tony Z. Zhao

3 years

Thrilled to announce that I will be joining @StanfordAILab as a PhD student! Starting to code in my freshman year, it has been a wild ride: I'm fortunate to be part of both @svlevine 's lab and @BerkeleyNLP . For my PhD, I want to explore the synergy of Robotics, Language and ML!

2

3

139

@tonyzzhao

Tony Z. Zhao

5 months

1. Moves fast. Similar to human walking of 1.42m/s. 2. Stable. Manipulate heavy pots, a vacuum, etc. 3. Whole-body. All dofs teleoperated simultaneously. 4. Untethered. Onboard power and compute.

2

9

119

@tonyzzhao

Tony Z. Zhao

6 months

Seems to be a quite significant improvement over the original ALOHA 🏖️ ! Just from this video: - Smooth active gravity comp - Larger payload and gripper opening - The bottle throw and catch demo is 🔥 Excited to see ALOHA applied to another new robot!

@ARX_Zhang

ARX

6 months

方舟无限ARX5 x ALOHA 数据采集测试

3

53

294

0

12

111

@tonyzzhao

Tony Z. Zhao

1 year

It is so inspiring to see researchers outside of academia being able to replicate ALOHA🏖️ and ACT. This is really the best-case scenario I can hope for, to democratize access to robotics and AI research!

@MindFactoryAI

MindFactory

1 year

First task on the Aloha system is running autonomously.

1

1

19

4

5

106

@tonyzzhao

Tony Z. Zhao

10 months

I love this video (and the thesis) so much. It is actually part of the initial inspiration of ALOHA🏖️ when we started working on it 1.5yrs ago!

@kevin_zakka

Kevin Zakka

10 months

Ben Katz's thesis is full of golden nuggets. In particular, I discovered today he had a really cool bilateral teleoperation system using two Mini Cheetah legs.

7

71

524

3

5

105

@tonyzzhao

Tony Z. Zhao

1 year

How does it work? ALOHA has two leader & two follower arms, and syncs the joint positions from leaders to followers at 50Hz. The user teleops by simply moving the leader robots. This takes 10 lines to implement, yet intuitive and responsive anywhere within the joint limits.

Tweet media one

6

11

96

@tonyzzhao

Tony Z. Zhao

5 months

At test time when the robot is autonomous, the backdriving structure and the leader arms can be easily detached. This reduces the robot's footprint by 45% and shaves off 15kg in weight. The robot can reach 65cm to 200cm vertically, and 100cm away from its base.

Tweet media one

Tweet media two

2

7

96

@tonyzzhao

Tony Z. Zhao

1 year

Our open-source page: Trossen’s purchase page: (we receive no monetary benefits from this collaboration!)

Tweet card media

Aloha Kits | Trossen Robotics

Trossen Robotics makes THE official Aloha research kits for robotic machine learning. Available in stationary, mobile, and manipulator arm versions.

www.trossenrobotics.com

6

15

82

@tonyzzhao

Tony Z. Zhao

9 months

Curious about deploying robot learning solutions in the real world? 🤖 Join us and our amazing lineup of speakers at #CoRL2023 this year. We will be holding a debate on the future of robot learning, in addition to talks and poster sessions! CfP:

Tweet media one

2

17

83

@tonyzzhao

Tony Z. Zhao

10 months

Is Silicon Valley too obsessed with pure software businesses? Do we still have a chance to disrupt DJI? Will Unitree be the next DJI but with a much much larger scope? I have so many questions.

@UnitreeRobotics

Unitree

@UnitreeRobotics

10 months

Introducing Unitree H1: Its First General-purpose Humanoid Robot| Embodied AI, Price below $90k The preview of half-a-year achievement The highest-power-performance robot of its counterparts with similar specifications in the world, weigh ~47Kg, maximum joint torque of 360N.m

121

467

2K

11

7

75

@tonyzzhao

Tony Z. Zhao

1 year

This simple idea + proper mechanical design allows ALOHA to perform precise tasks like RAM insertion, dynamic tasks like juggling a ping pong ball, and contact-rich tasks like putting on a shoe. It is reliable: there were no motor failures throughout the 8 months testing.

Tweet media one

2

8

76

@tonyzzhao

Tony Z. Zhao

4 months

Before diving into the hardware, we also release a *proper* ALOHA sim model with SysID, thanks to @kevin_zakka @the_real_btaba @ayzwah . Even if you don’t have the hardware, there is now a way to perform complex tasks with ALOHA in Mujoco!

2

5

76

@tonyzzhao

Tony Z. Zhao

11 months

#RSS2023 I am unable to present in-person because of visa issues😢 But the amazing @siddkaramcheti is kind enough to help me present it, on Tue 11am! I will be at the poster session to answer any questions (through an iPad on tripod.) Thanks @du_maximilian for setting it up!

@tonyzzhao

Tony Z. Zhao

1 year

Introducing ALOHA 🏖: 𝐀 𝐋ow-cost 𝐎pen-source 𝐇𝐀rdware System for Bimanual Teleoperation After 8 months iterating @stanford and 2 months working with beta users, we are finally ready to release it! Here is what ALOHA is capable of:

94

711

3K

3

7

76

@tonyzzhao

Tony Z. Zhao

11 months

It is worth noting, however, that this robot has not been publicly demoed like Optimus or Digit. Additionally, its payload capacity is likely smaller. Nevertheless, it deserves more than 300 views 🙂 Original video: Product page:

Tweet card media

Fourier Intelligence's GR-1: Igniting a New Wave of General-Purpose...

Discover the future of robotics with Fourier Intelligence's GR-1, the revolutionary general-purpose humanoid robot. Witness a new era of technological advanc...

www.youtube.com

5

8

74

@tonyzzhao

Tony Z. Zhao

10 months

Check out our new waypoint extraction method led by @lucy_x_shi @archit_sharma97 ! It’s a plug-and-play module that boosts imitation learning performance 🤖 Very impressed by Lucy’s execution in this project. She would also be applying for PhD this cycle!

@chelseabfinn

Chelsea Finn

10 months

Our robot can now make you coffee 🤖☕ A short 🧵 on how it works ⬇️

31

132

911

4

4

74

@tonyzzhao

Tony Z. Zhao

3 months

Unitree feels like… the elephant in the room?

@UnitreeRobotics

Unitree

@UnitreeRobotics

3 months

Unitree H1 Breaking humanoid robot speed world record [full-size humanoid] Evolution V3.0 🥰 The humanoid robot driven by the robot AI world model unlocks many new skills! Strong power is waiting for you to develop! #Unitree #AI #subject3 #BlackTech

40

226

911

6

3

72

@tonyzzhao

Tony Z. Zhao

1 month

The world of hardware is accelerating fast. Attention to the whole system, not just software/AI, will be necessary for real embodied AI.

@danfei_xu

Danfei Xu

1 month

Super neat system! It seems that Chinese robotics startups have everything they need to quickly iterate on capable & low-cost hardware. Will US startups be able to compete? Chaining together dynamixals/off-the-shelf motors likely won’t cut it…

6

11

96

0

4

68

@tonyzzhao

Tony Z. Zhao

8 months

Kids love ALOHA🏖️, robot dog, and more at @chelseabfinn 's lab! cc their robotics teacher 🧑‍🏫 @lucy_x_shi @zipengfu

Tweet media one

Tweet media two

Tweet media three

5

3

68

@tonyzzhao

Tony Z. Zhao

8 months

Maybe joint space teleop is all you need? 👀 Amazing project from @philippswu making teleoperation more accessible on a series of cobots. Its also awesome to see more hardware advances optimized for robot learning use cases!

@philippswu

Philipp Wu

8 months

🎉Excited to share a fun little hardware project we’ve been working on. GELLO is an intuitive and low cost teleoperation device for robot arms that costs less than $300. We've seen the importance of data quality in imitation learning. Our goal is to make this more accessible 1/n

26

109

685

0

3

64

@tonyzzhao

Tony Z. Zhao

1 year

@heskelbalas @Stanford Thanks for pointing it out Heskel: it is indeed my video. There has been some misinformation that associates it with OpenAI's investment in @1x__tech

1

2

59

@tonyzzhao

Tony Z. Zhao

9 months

It takes a lot of effort to not only build something that "works", but also document the process and make it available to the community. Kudos to @kenny__shaw and the team!

@pathak2206

Deepak Pathak

9 months

We have easy-to-follow assembly videos with step-by-step instructions on the website. All the parts are easily available off-the-shelf, and the CAD files are open-source. Our design is stronger and more robust than other hands. Takes 3 hours to assemble. 2/

2

3

31

2

8

60

@tonyzzhao

Tony Z. Zhao

2 months

Check out the tweet from @ayzwah for more details and closeup videos!

@ayzwah

Ayzaan Wahid

2 months

For the past year we've been working on ALOHA Unleashed 🌋 @GoogleDeepmind - pushing the scale and dexterity of tasks on our ALOHA 2 fleet. Here is a thread with some of the coolest videos! The first task is hanging a shirt on a hanger (autonomous 1x)

32

114

547

2

2

56

@tonyzzhao

Tony Z. Zhao

1 year

Here is the thread!

@tonyzzhao

Tony Z. Zhao

1 year

How can robots acquire fine-grained manipulation skills? Introducing ACT: Action Chunking with Transformers 🤖 Key idea: Imitation, but predict actions in chunks instead of one at a time. Here are results with only ~15min of demonstrations, running on low-cost arms:

28

215

1K

3

8

50

@tonyzzhao

Tony Z. Zhao

1 year

This project is not possible without the support from my advisor @chelseabfinn and @svlevine @Vikashplus . But so far, we’ve only covered *half* of the project! In a second thread, I will show how ALOHA can *autonomously* perform these tasks!

1

5

51

@tonyzzhao

Tony Z. Zhao

10 months

What’s better than presenting ALOHA 🏖️ at ICML this year! Come to Frontiers4LCD workshop (Ballroom B) at 12pm and 4pm today!

@tonyzzhao

Tony Z. Zhao

1 year

Introducing ALOHA 🏖: 𝐀 𝐋ow-cost 𝐎pen-source 𝐇𝐀rdware System for Bimanual Teleoperation After 8 months iterating @stanford and 2 months working with beta users, we are finally ready to release it! Here is what ALOHA is capable of:

94

711

3K

4

9

48

@tonyzzhao

Tony Z. Zhao

8 months

A very cool low-cost exoskeleton for joint-space teleop. Also works for multiple robots similar to Gello. Interesting learning results as well!

@haoshu_fang

Hao-Shu Fang

8 months

🤖Joint-level control + portability = robot data in the wild! We present AirExo, a low-cost hardware, and showcase how in-the-wild data enhances robot learning, even in contact-rich tasks. A promising tool for large-scale robot learning & TeleOP, now at !

6

37

206

1

8

48

@tonyzzhao

Tony Z. Zhao

7 months

Very nice arms! Consider making an ALOHA out of it? 👀

@ARX_Zhang

ARX

8 months

方舟无限ARX5 超轻型力控机械臂

2

13

102

1

1

46

@tonyzzhao

Tony Z. Zhao

8 months

Super proud to be contributing ALOHA 🏖️ data to this effort! This is one of the most forward-looking, bold, and open science project I’ve encountered!

@QuanVng

Quan Vuong

8 months

RT-X: generalist AI models lead to 50% improvement over RT-1 and 3x improvement over RT-2, our previous best models. 🔥🥳🧵 Project website:

7

143

619

0

4

43

@tonyzzhao

Tony Z. Zhao

5 months

@zipengfu tbh this might be my favourite video so far, less fun when it fails in front of you 🤣

2

2

44

@tonyzzhao

Tony Z. Zhao

1 year

@AiBreakfast We @Stanford will be releasing the research next week. Silver lining: *Everything* you saw it that video will be open-sourced to everyone. Stay tuned!

2

4

43

@tonyzzhao

Tony Z. Zhao

3 years

"Insert anything into anything!" New paper applying offline RL to industrial insertion. Test it with 12 new tasks, 100/100 success rate on all of them, with only 6 minutes of finetuning time on average! 📝 🌎

ArXiv: https://arxiv.org/abs/2110.04276

sites.google.com

@svlevine

Sergey Levine

3 years

Offline RL + meta-learning enables industrial robots to learn new insertion tasks with near-perfect success rates with AWAC + PEARL + finetuning! w/ @tonyzzhao , @jianlanluo , @DeepMind , Intrinsic Short summary below:

1

19

99

0

3

42

@tonyzzhao

Tony Z. Zhao

4 months

We start by improving the grippers: to make them grasp better and more robust. We use a low-friction rail design that transmits 2x more force to the gripper tips. We also change the grip tape layout to improve grasping of small objects. Led by @SpencerGoodric6 and Thinh Nguyen

Tweet media one

Tweet media two

Tweet media three

1

3

40

@tonyzzhao

Tony Z. Zhao

3 months

It is refreshing to see highly creative, open-source works like DexCap building on top of another highly creative, open-source work (LEAP hand by @kenny__shaw .) This is the best way forward for the community. Congratulations @chenwang_j !

@chenwang_j

Chen Wang

3 months

Can we use wearable devices to collect robot data without actual robots? Yes! With a pair of gloves🧤! Introducing DexCap, a portable hand motion capture system that collects 3D data (point cloud + finger motion) for training robots with dexterous hands Everything open-sourced

21

132

620

1

3

37

@tonyzzhao

Tony Z. Zhao

1 year

In case you missed ALOHA 🏖, the hardware we use for all these experiments, here is the thread!

@tonyzzhao

Tony Z. Zhao

1 year

Introducing ALOHA 🏖: 𝐀 𝐋ow-cost 𝐎pen-source 𝐇𝐀rdware System for Bimanual Teleoperation After 8 months iterating @stanford and 2 months working with beta users, we are finally ready to release it! Here is what ALOHA is capable of:

94

711

3K

1

2

37

@tonyzzhao

Tony Z. Zhao

10 months

Very cool work from @Meta with full dataset open-sourced! Also excited to see ACT scale to multi-task setting and tackle visual generalization.

@Vikashplus

Vikash Kumar

10 months

#𝗥𝗼𝗯𝗼𝗔𝗴𝗲𝗻𝘁 -- A universal multi-task agent on a data-budget 💪 with 12 non-trivial skills 💪 can generalize them across 38 tasks 💪& 100s of novel scenarios! 🌐 w/ @mangahomanga @jdvakil @m0hitsharma , Abhinav Gupta, @shubhtuls

4

67

249

0

4

37

@tonyzzhao

Tony Z. Zhao

1 year

Similar to ALOHA, we open source ACT together with 2 simulated environments for reproducibility. You can find it in the project website: We hope ALOHA+ACT would be a helpful resource towards advancing fine-grained manipulation!

1

2

35

@tonyzzhao

Tony Z. Zhao

3 months

End-to-end low-level skills with language corrections on the fly. Huge commitment from @lucy_x_shi to pull this off. Congratulations!

@lucy_x_shi

Lucy Shi

3 months

Introducing Yell At Your Robot (YAY Robot!) 🗣️- a fun collaboration b/w @Stanford and @UCBerkeley 🤖 We enable robots to improve on-the-fly from language corrections: robots rapidly adapt in real-time and continuously improve from human verbal feedback. YAY Robot enables

19

79

461

1

1

36

@tonyzzhao

Tony Z. Zhao

2 years

Parallel jaw grippers are more capable than you might think. We ( @stanford ) also has some very exciting work coming soon!

@Wenxuan_Zhou

Wenxuan Zhou

2 years

Are simple grippers limited to simple motions such as pick-and-place? Our work to be presented at #CoRL2022 demonstrates that RL can be used to enable a parallel gripper to find interesting strategies to exploit the environment to enhance its “dexterity”. A thread:

2

35

205

3

1

33

@tonyzzhao

Tony Z. Zhao

9 months

One of the most impressive dexterous hand policy I’ve seen. Love the in hand reorientation!

@chenwang_j

Chen Wang

9 months

How to chain multiple dexterous skills to tackle complex long-horizon manipulation tasks? Imagine retrieving a LEGO block from a pile, rotating it in-hand, and inserting it at the desired location to build a structure. Introducing our new work - Sequential Dexterity 🧵👇

27

92

473

1

6

35

@tonyzzhao

Tony Z. Zhao

4 months

To learn more, please visit our website: Paper: Tutorial: Designs: Sim:

Tweet card media

Project page for ALOHA 2

aloha-2.github.io

1

4

33

@tonyzzhao

Tony Z. Zhao

1 year

maybe our next lab social should be a teleop competition 👀👀

@chelseabfinn

Chelsea Finn

1 year

Can you peel an egg without tactile feedback? 🥚 Turns out yes! We had some fun pushing the limits of our robot hardware last night. 😄🤖

9

73

659

3

0

32

@tonyzzhao

Tony Z. Zhao

11 months

Always love seeing people's first reaction using ALOHA. This is Koko's first try closing the lid of that small cup, pretty much no learning curve! Learn more about ALOHA here 👉

@kokoxsu

Koko Xsu

11 months

Super impressed by ALOHA’s precision, latency, and intuitiveness! (cc @tonyzzhao )

2

0

64

0

7

32

@tonyzzhao

Tony Z. Zhao

4 months

Very cool results from @HaoyuXiong1 ! Proper real-world evaluations are always the most time consuming but important part of robotics research!

@Haoyu_Xiong_

Haoyu Xiong

4 months

Introducing Open-World Mobile Manipulation 🦾🌍 – A full-stack approach for operating articulated objects in open-ended unstructured environments: Unlocking doors with lever handles/ round knobs/ spring-loaded hinges 🔓🚪 Opening cabinets, drawers, and refrigerators 🗄️ 👇

30

103

778

1

6

31

@tonyzzhao

Tony Z. Zhao

1 year

You should definitely chat with Phil & folks from Tesla if you are excited about large scale vision, robotics and more! They got a ton of data and compute to test your newest algorithm 👀 (Also wonderful people! Had a great time there last summer)

@philduan

Phil Duan

1 year

@Tesla AI team is at @CVPR in Vancouver this week! If you are also here, stop by and check out what we have been working on for Autopilot, Optimus, and dojo! #CVPR2023

Tweet media one

142

712

2K

0

4

30

@tonyzzhao

Tony Z. Zhao

1 year

It is also robust to a certain level of distractor objects:

2

4

30

@tonyzzhao

Tony Z. Zhao

1 year

With all above, ACT obtains 64%, 96%, 84%, 92% success for 4 tasks shown, with objects randomized along the 15 cm line. It does not just memorize the training data, and is able to react to external disturbances:

1

1

30

@tonyzzhao

Tony Z. Zhao

1 year

(1) Predict action sequence Standard BC predicts one action at a time, while a fine manipulation task can have >1000 steps easily. Predicting action in chunks slows down compounding error, and can better model non-stationary human behavior.

Tweet media one

2

2

28

@tonyzzhao

Tony Z. Zhao

5 months

Salute to the failure compilations of DARPA Robotics Challenge back in 2015 and of course the Boston Dynamics Atlas I am secretly hoping to see @Tesla_Optimus fall 🙈

Tweet card media

Watch: Boston Dynamic's Atlas robot fail compilation

Boston Dynamics has shared an amusing compilation of its robot attempting to pick up a new set of skills.Atlas, the bot that went viral with its dance moves ...

www.youtube.com

0

7

30

@tonyzzhao

Tony Z. Zhao

7 months

Is scaling all we need to deploy general purpose robot? We have an exciting lineup of speakers and a debate session tomorrow at @corl_conf . Look forward to seeing everyone in Atlanta! 🏙️

@fangchenliu_

Fangchen Liu

9 months

We are organizing @corl_conf 2023 workshop on Reliable and Deployable Learning-Based Robotic Systems with an exciting list of invited speakers, looking towards the future of robot learning systems: . Please don't hesitate to submit your work here!

Tweet media one

2

15

81

0

3

29

@tonyzzhao

Tony Z. Zhao

1 year

Fine manipulation is difficult: either from RL, Sim2Real, or Imitation. - Hard exploration and sparse reward - Large Sim2Real gap - Compounding error for BC - No large dataset We introduce three important design choices behind ACT, an efficient imitation learning method:

Tweet media one

1

2

26

@tonyzzhao

Tony Z. Zhao

1 year

(3) Transformer We modernize the VAE by using a BERT-like encoder and a DETR-like decoder, training end-to-end from scratch. This transformer architecture benefits more from chunking than ConvNets and non-parametric methods.

Tweet media one

Tweet media two

2

2

26

@tonyzzhao

Tony Z. Zhao

11 months

Reproducibility FTW! 🏖️🦾

@MindFactoryAI

MindFactory

11 months

Test of slotting battery, first run.

1

1

12

1

1

28

@tonyzzhao

Tony Z. Zhao

1 year

Personally, this is a challenging project to work on, spanning from hardware to ML. It would certainly not be possible without my amazing advisor @chelseabfinn and collaboration from @svlevine @Vikashplus !

2

3

25

@tonyzzhao

Tony Z. Zhao

1 year

Glad it works ; )

@MeRTcooking

Masato Kobayashi @るっと🐺

1 year

@tonyzzhao Great job! I just tried your ACT and it was wonderful for me! Thank you so much!!

1

1

9

1

1

25

@tonyzzhao

Tony Z. Zhao

4 months

Next, we improve the gravity compensation of the leader arm. With a constant-force retractor and a spring-pulley system, the arm can "float" in most places. It is also much more durable than the original rubberbands!

1

2

24

@tonyzzhao

Tony Z. Zhao

2 months

@ayzwah team: @ayzwah , @JonathanTompson , @DannyDriess , @peteflorence , @coolboi95 , @chelseabfinn @SpencerGoodric6 🚀

1

1

24

@tonyzzhao

Tony Z. Zhao

4 months

sleek! Open hardware ftw

@clayhaight

clayton

4 months

some friends working in robotics expressed interest in a gripper I was designing for my senior capstone project, so I've decided to make it open source! it's extremely simple and cheap, but doesn't sacrifice on performance 🦾 check it out:

Tweet media one

10

16

212

0

0

24

@tonyzzhao

Tony Z. Zhao

4 months

Looks so natural! Fascinating progress from Boston dynamics.

@BostonDynamics

Boston Dynamics

@BostonDynamics

4 months

Can't trip Atlas up! Our humanoid robot gets ready for real work combining strength, perception, and mobility.

227

1K

5K

0

2

24

@tonyzzhao

Tony Z. Zhao

1 year

(2) Generative model policy The policy is trained as the decoder of a VAE, reconstructing action chunks from latent z, 4 RGB images, and proprioception. Intuitively, z extracts the “style” of the action chunk. This is crucial when learning from human demos.

Tweet media one

1

4

21

@tonyzzhao

Tony Z. Zhao

1 year

@arthurallshire @Stanford It is RGB to joint position, imitating from just 50 demos. Stay tuned for more!

2

1

21

@tonyzzhao

Tony Z. Zhao

4 months

We use the same rail design on the leader side. To further improve ergonomics, we replace the original servo with a lower gear ratio one that is easier to backdrive. This results in a 10x reduction in friction that the operator needs to overcome when opening grippers!

Tweet media one

Tweet media two

1

1

21

@tonyzzhao

Tony Z. Zhao

4 months

Last but not least: we simplify the frame surrounding the workcell while maintaining the rigidity of the camera mounting points. This opens up the space for both human-robot collaborators and props for the robot to interact with.

Tweet media one

Tweet media two

1

1

20

@tonyzzhao

Tony Z. Zhao

1 year

Would love to come back to this in 5 years, and hopefully this is not true! I am still optimistic ; )

@shaneguML

Shane Gu

1 year

[2/3] Robotics is hard, and I focus my time mostly on generative AI these days. Because in 5 years, we likely still can't match motor control of a 3-year old baby (generalization, adaptability, smoothness, dexterity)... Last remaining AI challenge would be mastering dexterity.

1

7

32

1

0

20

@tonyzzhao

Tony Z. Zhao

5 months

@PTrubey @zipengfu Thank you! It is indeed a mix and we really hope people could go to the project website and read the paper/code!

0

0

19

@tonyzzhao

Tony Z. Zhao

1 year

ALOHA Sabera!

@SaberaTalukder

Sabera Talukder

@SaberaTalukder

1 year

Aloha means both hello and goodbye, so what a perfect platform to test out on my last day at Stanford 🙃🌺👋 Thank you to @RishiBommasani @CharlieTMarx @megha_byte @lxuechen @tonyzzhao @archit_sharma97 ++ for making my visit so welcoming, fun & productive🤓🤖

1

3

50

1

0

19

@tonyzzhao

Tony Z. Zhao

2 years

Accepted #ICRA2022 ! Appreciate all the reviewer feedback and super excited about the in-person event in Philadelphia 🏙️🌆 Project website:

ArXiv: https://arxiv.org/abs/2110.04276

sites.google.com

@svlevine

Sergey Levine

3 years

Offline RL + meta-learning enables industrial robots to learn new insertion tasks with near-perfect success rates with AWAC + PEARL + finetuning! w/ @tonyzzhao , @jianlanluo , @DeepMind , Intrinsic Short summary below:

1

19

99

0

1

19

@tonyzzhao

Tony Z. Zhao

4 months

Thanks to the core ALOHA 2 Team: @RandomRobotics @chelseabfinn @peteflorence @SpencerGoodric6 Thinh Nguyen @JonathanTompson @ayzwah @tonyzzhao and those who helped with hardware, software, data, simulation, and user studies: Jorge Aldaco, Robert Baruch, Jeff Bingham, Sanky Chan,

0

1

17

@tonyzzhao

Tony Z. Zhao

5 months

@DrJimFan Thanks for the post Jim!! 2024 is the year for robotics 🦾

0

1

16

@tonyzzhao

Tony Z. Zhao

7 months

The deployable workshop @corl_conf is starting in 30 min! We are located at the second floor (follow the sign of "robot demo"), and will be hosting a debate, a panel and invited talks. Streaming:

Tweet media one

Tweet media two

1

1

16

@tonyzzhao

Tony Z. Zhao

1 year

Thank you @Ken_Goldberg ! We were also surprised by how well it works. Hoping this is an Aloha moment for accessible fine manipulation!

@Ken_Goldberg

Ken Goldberg

1 year

Just played with it in @SvLevine 's lab. ALOHA is FAR more intuitive and responsive than I expected, esp. at that price. Thanks for the demo @jianlanluo and hats off to @tonyzzhao for outstanding engineering!

0

2

11

0

0

15

@tonyzzhao

Tony Z. Zhao

1 year

@timshi_ai @Stanford We fit it with custom grippers. This whole setup will be open-sourced soon!

0

1

15

@tonyzzhao

Tony Z. Zhao

1 year

Diffusion policy from @chichengcc : also uses a generative model for policy. Great for fitting multi-modal data and made large progress on the RoboMimic benchmark. Also very impressive real-world experiments!

@chichengcc

Cheng Chi

1 year

What if the form of visuomotor policy has been the bottleneck for robotic manipulation all along? Diffusion Policy achieves 46.9% improvement vs prior StoA on 11 tasks from 4 benchmarks + 4 real world tasks! (1/7) website : paper:

9

100

536

0

2

15

@tonyzzhao

Tony Z. Zhao

5 months

@chenwang_j Thanks Chen!! We actually released all commit history at the moment (might remove it later haha) but first commit is from @zipengfu Oct 16, when we just received all the hardware and start putting things together!

Tweet media one

1

2

15

@tonyzzhao

Tony Z. Zhao

1 year

Here are some really cool related works you should also know about! Chopstick-holding cherry-picking robot from @xkelym , trained with RL in the real world. The motion is very reactive and precise!

@xkelym

Kay - Liyiming Ke

1 year

Let’s do 🍒 Cherry Picking with Reinforcement Learning - 🥢 Dynamic fine manipulation with chopsticks - 🤖 Only 30 minutes of real world interactions - ⛔️ Too lazy for parameter tuning = off-the-shelf RL algo + default params + 3 seeds in real world

6

29

202

1

1

14

@tonyzzhao

Tony Z. Zhao

11 months

@ShikharMurty I've seen something similar for our transformer (the ACT robot policy ). While validation loss seems largely plateaued, real-world performance keeps improving. Not sure if it is for the same reason though!

0

0

11

@tonyzzhao

Tony Z. Zhao

5 months

@zipengfu 🚀🚀🚀

0

0

12

@tonyzzhao

Tony Z. Zhao

2 years

It’s lit! #AIDay2022

Tweet media one

Tweet media two

1

0

11

@tonyzzhao

Tony Z. Zhao

4 months

@HaoyuXiong1 Another super impressive work from NYU. It's really fun to see the diversity of ideas in mobile manipulation!

@LerrelPinto

Lerrel Pinto

4 months

Excited to release OK-Robot, an open-vocabulary mobile-manipulator for homes. Simply tell the robot what to pick and where to drop it in natural language, and it will do it. Like: Me: "OK Robot, move the Takis from the desk to the nightstand" Robot: ⬇️

21

76

440

0

2

11

@tonyzzhao

Tony Z. Zhao

11 months

@siddkaramcheti @du_maximilian #RSS2023 I will be presenting virtually through this iPad in ~an hour! Will be doing some live demo with our own ALOHA. Come and say hi!

Tweet media one

1

1

11