Hidenori Tanaka @Hidenori8Tanaka Twitter profile | Pikagi

Pikagi

Hidenori Tanaka

@Hidenori8Tanaka

3,162

Followers

995

Following

64

Media

495

Statuses

Group Leader, CBS-NTT Program in "Physics of Intelligence" at Harvard

Cambridge, MA

https://t.co/KXCpdTrber

Joined September 2018

Don't wanna be here? Send us removal request.

Pinned Tweet

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

More excited than ever to announce $1.7M: "CBS-NTT Program in Physics of Intelligence at Harvard"! 🧠 With new technology comes new science. The time is ripe to build a better future with "Physics of Intelligence for Trustworthy and Green AI"! 🧵👇

Tweet card media

Gift establishes new program in physics of intelligence at Center for Brain Science

Harvard’s Center for Brain Science has received a gift from the NTT Research Foundation to establish the Center for Brain Science-NTT Fellowship Program. The two-year gift, renewable for up to…

news.harvard.edu

6

22

129

Last Seen Profiles

@sjo_79a

@trizzo1s

@NFTHawks

@lindacarte69116

@krtmn116475

@cindy53mccoy

@callmesteveo7

@1brad_ass

@KotasBJP

@deborah38154098

@marianswimdive

@TBurton_2

@AthleticsMCHS

@DitiroTumi39574

@roger_p_levy

@postschweiz

@Humedia1

@LLinwynne

@helenallen21030

@ksithu_

@JianchenChen

@BuckleScript

@annabella313

@fanspicy1

@maxime_maurer

@fanspicy1

@BobPalm8

@catrionawallace

@FrontierClimate

@pibedale

@KylieBurrett

@WorldAnimalW

@uastronomer

@susanta36392967

@Flashpoint

@mingisonc

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Wow!! Emperor Akihito of Japan -who is now 85 years old- has published last scientific paper in ichthyology before abdication. Great respect for pure scientific curiosity.

Tweet media one

2

974

2K

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Q. Can we solve learning dynamics of modern deep learning models trained on large datasets? A. Yes, by combining symmetry and modified equation analysis! co-led with @KuninDaniel (now on twitter) & @jvrsgsty @SuryaGanguli @dyamins Neural Mechanics 1/8

4

180

784

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Q. What does Noether’s theorem tell us about the “geometry of deep learning dynamics”? A. We derive Noether’s Learning Dynamics and show: ”SGD+momentum+BatchNorm+weight decay” = “RMSProp" due to symmetry breaking! w/ @KuninDaniel #NeurIPS2021 Paper: 1/

9

132

600

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Q. Can we find winning lottery tickets, or sparse trainable deep networks at initialization without ever looking at data? A. Yes, by conserving "Synaptic Flow" via our new SynFlow algorithm. co-led with Daniel Kunin & @dyamins , @SuryaGanguli paper: 1/

6

158

561

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

ハーバード大学に、"Physics of Intelligence" (知性の物理学) プログラムを新設しました！「知性とは何か？」という問いに、脳だけでなく生成AI等も実験系として用いることで理論構築に挑みます。得られた知見を、信頼性が高く環境に優しいAIの開発に応用します。

Tweet card media

Gift establishes new program in physics of intelligence at Center for Brain Science

Harvard’s Center for Brain Science has received a gift from the NTT Research Foundation to establish the Center for Brain Science-NTT Fellowship Program. The two-year gift, renewable for up to…

news.harvard.edu

2

71

367

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

New paper out on #NeurIPS2019 : “From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction” with fantastic collaborators @aran_nayebi , @niru_m , @lmcintosh , Stephen Baccus, @SuryaGanguli .

Tweet media one

0

51

179

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

[delayed personal update] Excited to share that I’ve recently relocated and joined Center for Brain Science at Harvard as an Associate for an industry-academia collaboration!🧠 We’ll continue to work on problems at the interface of physics, neuroscience, and machine learning. 1/

4

6

162

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

🧠 Internship openings: NTT Research at Harvard! 🤖 Want to solve cutting-edge problems in deep learning by theory-guided algorithm design? Want to apply tools and ideas in ML to understand the brain? Come join us this summer at the Center for Brain Science at Harvard! 1/

Tweet media one

6

38

153

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

BatchNorm, LayerNorm, WeightNorm… Too many normalization layers to choose from? Which one to use, when, and how? Theory can guide you! led by @EkdeepL , who amazingly bridges theory & practice of deep learning!

@EkdeepL

Ekdeep Singh

3 years

A multitude of normalization layers have been proposed recently, but are we ready to replace BatchNorm yet? In our new preprint, we address this question by developing a unified understanding of normalization layers in deep learning. arXiv link:

6

45

231

1

26

134

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

6 months

📰 #NeurIPS2023 Paper Alert!🥳 Q. How far can generative models generalize? Through the lens of compositionality, we introduce a "Concept Graph" to propose (i) the distance of generalization, and (ii) the "Multiplicative Emergence" hypothesis. 1/n co-led

2

24

108

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

Want to train RNNs on the large-scale, observed activity of tens of thousands of neurons in real-time? We're introducing CORNN! 🎉 Led by @fatihdin4en , A. Shai with M. Schnitzer from Stanford. Paper link: Meet us now at #NeurIPS2023 : 10:45 AM, Tue, Dec

Tweet media one

0

16

99

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

At #NeurIPS2023 to share a series of works on "Physics of AI" and "AI for Neuroscience" from NTT Research at Harvard CBS team! 🎉 Find me at the posters and/or DM me to chat & explore collaborations! Can't thank the Summer 2023 team enough♥️: @EkdeepL @fatihdin4en @MayaOkawa

Tweet media one

0

20

71

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 months

"Interested in the 'science of modern AI' to make it more trustworthy and efficient? Seeking fundamental principles of learning and computation in biological and artificial intelligence? Let's explore them together this summer at Harvard's Center for Brain Science!" 1/

Tweet media one

2

13

57

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 months

Very excited to be back at Stanford tomorrow to present at the Mind, Brain, Computation and Technology Seminar! 🧠🤖 Please come and join us if you are in the area and interested in physics, neuroscience, and AI.

Tweet media one

1

3

53

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

The best textbook on the theory and phenomenology of neural learning dynamics, and I highly recommend it to anyone fascinated by the subject! Contributing to the content of this textbook has been a major goal of my research. It has directly inspired a number of past projects,

@RogerGrosse

Roger Grosse

5 months

Two years ago, I taught a topics course on neural net training dynamics. While this isn't about safety/alignment per se, I recommend working through it if you're interested in safety/alignment of LLMs.

5

33

228

0

7

56

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

@KuninDaniel Geometry of data & representations has been central in the design of modern deepnets. e.g., #GeometricDeepLearning by @mmbronstein , @joanbruna , @TacoCohen , @PetarV_93 What are the geometric design principles for “learning dynamics in parameter space”? 2/

2

7

55

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

9 months

Very excited that this work is finally out 🎉 It was really an amazing interdisciplinary collaboration!

@SuryaGanguli

Surya Ganguli

9 months

1/Our paper @NeuroCellPress "Interpreting the retinal code for natural scenes" develops explainable AI ( #XAI ) to derive a SOTA deep network model of the retina and *understand* how this net captures natural scenes plus 8 seminal experiments over >2 decades

Tweet media one

13

258

931

1

1

53

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

In 1992, Emperor Akihito has written an article published in Science titled "Early Cultivators of Science in Japan". And yes, Emperor has an official publication list (in Japanese) if you are interested.

Tweet media one

1

27

49

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Overall, our work provides a first step towards understanding the mechanics of learning in neural networks without unrealistic simplifying assumptions Check out the paper for more details: 8/8

Tweet card media

Neural Mechanics: Symmetry and Broken Conservation Laws in Deep...

Understanding the dynamics of neural network parameters during training is one of the key challenges in building a theoretical foundation for deep learning. A central obstacle is that the motion...

1

5

47

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

NTT Research社の寄付により、ハーバード大学に新設される "Physics of Intelligence" (知性の物理学) プログラムがアナウンスされました。「知性とは何か？」という問いに、脳科学だけでなく生成AI等も実験系として用いることで、”知性の物理学”という新しい学問分野の創出を目指します！

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

More excited than ever to announce $1.7M: "CBS-NTT Program in Physics of Intelligence at Harvard"! 🧠 With new technology comes new science. The time is ripe to build a better future with "Physics of Intelligence for Trustworthy and Green AI"! 🧵👇

6

22

129

1

7

42

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

2 #NeurIPS2021 papers with the amazing first interns at PHI Lab @NttResearch ! (i) unified understanding of normalization layers w/ @EkdeepL , (ii) Noether’s theorem and the geometry of learning dynamics w/ @KuninDaniel , both advancing the practice of deep learning through theory

2

4

41

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Join us for an exciting summer of research! Apply here now: “PHI Lab: Research Intern 2023” If you are interested in working with me, please mention it in the “Accompanying Message” section.

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

🧠 Internship openings: NTT Research at Harvard! 🤖 Want to solve cutting-edge problems in deep learning by theory-guided algorithm design? Want to apply tools and ideas in ML to understand the brain? Come join us this summer at the Center for Brain Science at Harvard! 1/

Tweet media one

6

38

153

1

6

37

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

8 months

Very excited to share our research on the "Physics of Intelligence for Trustable AI" at IAIFI's monthly colloquium at MIT this Friday!

@iaifi_news

IAIFI

8 months

Our monthly Colloquium series kicks off again this Friday (Sept 15)! Check out the exciting lineup we have for Fall 2023 and join us at 2 pm ET on YouTube: @AIVOInfo @Hidenori8Tanaka

Tweet media one

0

1

9

0

3

36

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

🧠 Two exciting news 🧠 We @NttResearch are thrilled to collaborate with @NeuroVenki and the @Harvard Center for Brain Science & to welcome amazing @Gautam_Reddy_N to better understand the brain together! link:

0

2

31

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Every symmetry of a network has a corresponding conserved quantity through training under gradient flow (Noether's theorem for neural networks!) For translation, scale, and rescale symmetry the flow is constrained to a hyperplane, sphere, and hyperbola respectively 4/8

Tweet media one

1

2

31

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Overall, our data-agnostic pruning algorithm challenges the existing paradigm that data must be used to quantify which synapses are important. Please check out the paper for more details 7/

Tweet card media

Pruning neural networks without any data by iteratively conserving...

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have...

3

3

30

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Overall, understanding not only when symmetries exist, but how they are broken is essential to discover geometric design principles in neural networks. For more details see “Noether’s Learning Dynamics: Role of Symmetry Breaking in Neural Networks": 10/

2

2

29

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Looking forward to attending NeurIPS and seeing everyone next week! I'd love to chat about any combination of deep learning and physics, dynamical systems, symmetry, neuroscience, mechanistic understanding + internship/collaboration opportunities! Please feel free to DM me ;-)

1

0

28

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Q. How can we fix the decision mechanism of a pre-trained model efficiently? A. Mechanistic fine-tuning by driving the model over a barrier on the landscape! led by @EkdeepL , interning at @NttResearch at Harvard w/ @bigel_o R. Dick @DavidSKrueger

1

6

28

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 months

Really enjoyed presenting and attending the @iaifi_news symposium at MIT today! It was great to learn about the two-way interaction between physics and AI.

@iaifi_news

IAIFI

2 months

Wrapping up our talks for the day with @Hidenori8Tanaka sharing insights on the natural science of artificial intelligence.

Tweet media one

Tweet media two

1

0

7

0

3

27

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Interested in fundamental research at the interface of Physics and Informatics? Join us at NTT Physics & Informatics Lab! () Applications for the internship program is now open.

Tweet media one

3

6

25

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Abstract is here:

Tweet card media

京都御苑の仙洞御所の池に生息するビワヨシノボリRhinogobius biwaensisとシマヒレヨシノボリRhinogobius sp. BFの野外交雑個体

Hybrids of Rhinogobius biwaensis and Rhinogobius sp. BF were identified in the ponds of Sento Imperial Palace, Kyoto City, Japan, from microsatellite …

www.jstage.jst.go.jp

1

21

23

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Excited to co-present this work with @KuninDaniel at Physics ∩ ML seminar tomorrow! (Wed. Feb 24 12:00 EDT) Please join us at

Neural Mechanics: Symmetry and Broken Conservation Laws in Deep Learning Dynamics

Daniel Kunin and Hidenori Tanaka, 12:00 EDT

physicsmeetsml.org

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Q. Can we solve learning dynamics of modern deep learning models trained on large datasets? A. Yes, by combining symmetry and modified equation analysis! co-led with @KuninDaniel (now on twitter) & @jvrsgsty @SuryaGanguli @dyamins Neural Mechanics 1/8

4

180

784

1

3

23

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 months

🧠Final Call for Internships: NTT Research at Harvard!🤖 Interested in the science of modern AI? Want to explore the principles of natural and artificial intelligence? Join us for a unique industry internship for academic research! Recent works: 1/

Tweet media one

1

6

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Is there a theoretical analogy between ring attractor neural networks and Anderson localization in quantum systems? With David Nelson, we discovered a new class of random matrices whose eigenvectors are quasi-localized even with fully dense connections.

@PhysRevE

Physical Review E

5 years

Editors' Suggestion: Non-Hermitian quasilocalization and ring attractor neural networks

Tweet media one

0

4

8

1

10

22

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

Please join us soon at our poster at #NeurIPS2023 to discuss compositionally, learning dynamics, and emergence in multimodal diffusion models! Time: Thu 14, Dec 10:45 am CST - 12:45 pm CST Location: Great Hall & Hall B1+B2 (Level 1) #2021

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

6 months

📰 #NeurIPS2023 Paper Alert!🥳 Q. How far can generative models generalize? Through the lens of compositionality, we introduce a "Concept Graph" to propose (i) the distance of generalization, and (ii) the "Multiplicative Emergence" hypothesis. 1/n co-led

2

24

108

0

4

22

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

As a result, gradient descent becomes Lagrangian dynamics with a finite learning rate, where the learning rule corresponds to the kinetic energy and the loss function corresponds to the potential energy. 5/

1

1

22

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

21 days

今月サンフランシスコで開催された @NttResearch の #Upgrade2024 イベントでの講演動画が公開されました！自然界の複雑な現象を解明してきた科学的アプローチを、AIという複雑な人工物の理解と信頼性向上にどう活かせるのか？ぜひご覧ください！

Tweet card media

Physics of Intelligence for Trustworthy and Green AI

With the advent of modern artificial intelligence (AI) technologies like ChatGPT, "intelligence" is no longer unique to humans. Despite the rapid pace at whi...

www.youtube.com

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

ハーバード大学に、"Physics of Intelligence" (知性の物理学) プログラムを新設しました！「知性とは何か？」という問いに、脳だけでなく生成AI等も実験系として用いることで理論構築に挑みます。得られた知見を、信頼性が高く環境に優しいAIの開発に応用します。

2

71

367

0

5

22

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

@boazbaraktcs @RylanSchaeffer @BrandoHablando @sanmikoyejo Really enjoyed this blog post! Sharing our NeurIPS paper on the "Multiplicative Emergence of Compositional Abilities", where we designed an "interpretable task" and showed that "abilities that require composition of atomic abilities show emergent curves".😀 paper:

Tweet media one

1

0

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

To study complex learning dynamics of neural networks, existing works made major assumptions (i.e. single hidden layer, linear networks, infinite width) Instead, we uncover combinations of parameters with simplified dynamics that we solved exactly without a single assumption 2/8

Tweet media one

1

1

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

10 months

At #ICML2023 to present our mechanistic take on mode connectivity at Poster Session 4, Exhibit Hall 1 from 2--3:30 PM HST this Wednesday (tomorrow)! DM me if you want to meet & chat!

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Q. How can we fix the decision mechanism of a pre-trained model efficiently? A. Mechanistic fine-tuning by driving the model over a barrier on the landscape! led by @EkdeepL , interning at @NttResearch at Harvard w/ @bigel_o R. Dick @DavidSKrueger

1

6

28

0

4

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

The realistic model for SGD breaks the conservation laws of gradient flow, resulting in simple first and second order ODEs We can solve these ODEs exactly leading to theoretical solutions we empirically verify on VGG-16 training on Tiny ImageNet 6/8

Tweet media one

1

0

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

These solutions confirm existing phenomenon, such as the spherical motion of parameters before batch normalization, while highlighting new phenomenon, such as the harmonic motion of parameters before the softmax function 7/8

Tweet media one

1

0

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

We develop Lagrangian mechanics of learning by modeling it as the motion of a particle in high-dimensional parameter space. Just like physical dynamics, we can model the trajectory of discrete learning dynamics by continuous-time differential equations. 3/

3

1

20

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Gradients and Hessians, at all points in training, obey geometric constraints due to symmetry A network has a symmetry if the loss doesn’t change under a transformation of the parameters (i.e. translation, scale, rescale for parameters preceding softmax, batchnorm, ReLU) 3/8

Tweet media one

1

1

21

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Excited to be speaking at the Center of Mathematical Science and Applications colloquium at Harvard today! Please drop by, if you are in the area ;-)

Tweet media one

0

0

20

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

Gradient flow is too simple for realistic SGD training. We construct a more realistic model considering weight decay, momentum, mini-batches, and a finite learning rate We use modified equation analysis to model the effect of discretization (as also done in recent works) 5/8

Tweet media one

1

0

20

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

However, there is still a gap between Newton’s EOM and gradient flow. Thus, we model the effects of finite learning rate as “implicit acceleration”, a complementary route to the "implicit gradient regularization" by @dgtbarrett , Benoit Dherin, @SamuelMLSmith , @sohamde_ . 4/

1

2

19

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Interesting topic. On the interaction between biological and artificial intelligence by Prof. Kunihiko Fukushima, father of Neocognitron (the inspiration for CNN) recorded in 2002.

Tweet card media

Kunihiko Fukushima

Interview with prof. Kunihiko Fukushima, June 2002, who has answered to these questions:What is your opinion about machine intelligence and it's application ...

www.youtube.com

@demishassabis

Demis Hassabis

5 years

@ylecun @IRudyak yes agreed, I think it is a classic analogy, and has been around for a while. and yes also agree that many people are trying to copy how the brain works at too low a level, but we have always believed that systems-level neuroscience has an important part to play in developing AI

3

3

69

1

5

17

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

We can potentially reduce the cost of training if we can prune neural networks at initialization. The key challenge is "layer-collapse," the premature pruning of an entire layer making a network untrainable. 2/

Tweet media one

1

0

17

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

産業革命の歴史は、複雑なシステムの創発的な能力を理解し、活用してきた歴史でもあります。蒸気機関、電力、トランジスタ、液晶などはすべて、��れらの力を制御するための新しい物理学の分野を生み出してきました。

1

1

17

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

AIが社会に急速に浸透する中、偏見のない、信頼できる、環境に優しいAIシステムを構築することは喫緊の課題です。また、AIは知能の本質を探求する科学的な機会も提供しています。AIは複雑でありながらその実態は数学的に定義されており、知能の基本原理を解明する新たな道を開いています。

Tweet media one

1

2

17

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

12月22日 (土) に東京大学で大学院留学説明会が開催されます。米国大学院のシステムの概要や個人的な体験談なども共有させていただければと思います。進路の選択肢の一つとして、興味のある方はぜひ！

0

4

17

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

At Harvard, I’m blessed to be immersed in the rich intellectual community of @boazbaraktcs , @NeuroVenki , @jefrankle , @CPehlevan , @HSompolinsky , @gershbrain , @naoshigeuchida , and look forward to learning from everyone across the campus. 3/

1

0

15

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

NTT has just founded the "Institute for Fundamental Mathematics" to work on, yes, mathematics! I like how this press release (in Japanese) quotes Eugene Wigner's "The Unreasonable Effectiveness of Mathematics in the Natural Sciences" as the motivation.

0

5

16

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

We are presenting "Pruning neural networks without any data by iteratively conserving synaptic flow" at #NeurIPS2020 poster session (B2) today. Please visit us! Updated manuscript is here:

Tweet card media

Pruning neural networks without any data by iteratively conserving...

Pruning the parameters of deep neural networks has generated intense interest due to potential savings in time, memory and energy both during training and at test time. Recent works have...

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Q. Can we find winning lottery tickets, or sparse trainable deep networks at initialization without ever looking at data? A. Yes, by conserving "Synaptic Flow" via our new SynFlow algorithm. co-led with Daniel Kunin & @dyamins , @SuryaGanguli paper: 1/

6

158

561

0

2

16

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

21 days

今月サンフランシスコで開催された @NttResearch の #Upgrade2024 イベントでの講演動画が公開されました！自然界の複雑な現象を解明してきた科学的アプローチを、AIという複雑な人工物の理解と信頼性向上にどう活かせるのか？ぜひご覧ください！

Tweet card media

Physics of Intelligence for Trustworthy and Green AI

With the advent of modern artificial intelligence (AI) technologies like ChatGPT, "intelligence" is no longer unique to humans. Despite the rapid pace at whi...

www.youtube.com

0

1

15

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

We derive Noether’s Learning Dynamics (NLD), unified equality that holds for any combination of symmetry and learning rules. NLD accounts for damping, the unique symmetries of the loss, and the non-Euclidean metric used in learning rules. 8/

1

1

14

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

We establish an exact analogy between two seemingly unrelated components of modern deep learning: normalization and adaptive optimization. Benefits of this broken-symmetry-induced “implicit adaptive optimization” are all empirically confirmed! 9/

1

0

14

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

We prove that layer-collapse can be entirely avoided by designing an algorithm with iterative, positive, conservative scoring. We design SynFlow satisfying the key requirements and show that it reaches the theoretical limit of max compression without collapsing a network. 5/

Tweet media one

1

0

14

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

By studying the symmetry properties of the kinetic energy, we define “kinetic symmetry breaking”, where the kinetic energy corresponding to the learning rule explicitly breaks the symmetry of the potential energy corresponding to the loss function. 7/

1

0

14

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Symmetry properties of this Lagrangian govern the geometry of learning dynamics. Indeed, modern deep learning architectures introduce an array of symmetries to the loss function as we previously studied in . 6/

1

1

14

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

@SamuelAinsworth @stanislavfort @siddhss5 Very interesting discussions! a thought: could the lack of NormLayers be the reason why ResNet with NormLayers can use SGD with momentum while MLP can only use Adam? hope a theory below showing “NormLayer+SGD+momentum+WD = RMSProp” may be helpful here!

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Q. What does Noether’s theorem tell us about the “geometry of deep learning dynamics”? A. We derive Noether’s Learning Dynamics and show: ”SGD+momentum+BatchNorm+weight decay” = “RMSProp" due to symmetry breaking! w/ @KuninDaniel #NeurIPS2021 Paper: 1/

9

132

600

1

0

13

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Don't miss out on @EkdeepL ’s poster on "Mechanistic Mode Connectivity" today at the #NeurIPS2022 Workshop on Distribution Shifts! Room 388 - 390 (Poster Session: 11:00-12:30)

@EkdeepL

Ekdeep Singh

1 year

Preprint time! 🧵 DNNs can use entirely distinct prediction mechanisms to solve a task (e.g., background vs. shape). Q1: Are such models mode-connected in the landscape? Q2: Can we change a model’s mechanisms by exploiting such connectivity? Link: 1/12

5

32

139

0

1

11

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 months

Very excited to speak at the IAIFI Symposium on the fascinating topic of Generative AI & Physics!

@iaifi_news

IAIFI

2 months

We are excited to be organizing a Symposium on the Impact of Generative AI in the Physical Sciences next Thursday, March 14 and Friday, March 15! Join us on the 8th Floor of @MIT_SCC for a great lineup of speakers and panelists. Zoom link available soon.

Tweet media one

1

18

70

0

1

12

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

To better understand the phenomena, we first mathematically formulate and experimentally verify a conservation law. This conservation law explains why existing gradient-based pruning algorithms at initialization suffer from layer-collapse. 3/

Tweet media one

1

0

12

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Will be presenting our work this morning: Please drop by if you are at #NeurIPS2019 10:45 AM—12:45 PM poster #152

Tweet media one

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

New paper out on #NeurIPS2019 : “From deep learning to mechanistic understanding in neuroscience: the structure of retinal prediction” with fantastic collaborators @aran_nayebi , @niru_m , @lmcintosh , Stephen Baccus, @SuryaGanguli .

Tweet media one

0

51

179

0

1

12

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Notably, SynFlow makes no reference to the training data and consistently outperforms existing state-of-the-art pruning algorithms at initialization on 12 distinct combinations of models and datasets. 6/

Tweet media one

1

1

12

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

I regret leaving the Bay Area during the pandemic and I'd like to thank my mentors - @SuryaGanguli , without whom I wouldn’t be working in this exciting new frontier of science, Daniel Fisher and Stephen Baccus for their guidance and support, and all the friends! 2/

1

0

11

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

The energy and momentum at Harvard for science of deep learning with an interdisciplinary approach that combines physics, neuroscience, and cognitive science is truly amazing. In coming years, I hope to contribute in some small way to the vibrant community in the Boston area. 4/

1

0

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

This summer we've been very fortunate to welcome amazing intern students: @EkdeepL @fatihdin4en @m_aukana Ziyin Liu, and @WL_Tong with @Gautam_Reddy_N . Please reach out to us if you are in the Boston area (or not), and share interests! 5/

1

0

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

21 days

It was a pleasure to speak at @NttResearch 's #Upgrade2024 event in San Francisco this month about how a scientific approach can make AI more trustworthy and energy efficient! 🧠💡

Tweet card media

Physics of Intelligence for Trustworthy and Green AI

With the advent of modern artificial intelligence (AI) technologies like ChatGPT, "intelligence" is no longer unique to humans. Despite the rapid pace at whi...

www.youtube.com

0

3

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

The history of industrial revolutions has been a story of understanding and harnessing the emergent abilities of complex systems. Steam engines, electricity, transistors, & liquid crystals all sparked new fields of physics to control their powers. 🔥💡🌈

1

0

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

25 days

Nice article from the Harvard Crimson about our "CBS-NTT Program in Physics of Intelligence"! 🧠 “This is new for all of us. How do you explain intelligent behavior in equations or in physics terms?”

@thecrimson

The Harvard Crimson

1 month

Harvard University’s Center for Brain Science received a gift of more than $300,000 per year for up to five years from the NTT Research Foundation, the foundation announced Thursday. Eunice S. Chae, Patil Djerdjerian, and Rachel M. Fields report.

0

1

11

0

2

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

28 days

これから、 @NttResearch と @Harvard が一丸となって、コンピュータサイエンス、脳科学、物理学を結集し、知能の基盤となる数学的原理をより深く理解するために、研究を進めていきます。続報をお待ちください!

2

0

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

If interested, email me with your CV and research interests. Below are examples of past projects with our talented interns! 3/

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Q. How can we fix the decision mechanism of a pre-trained model efficiently? A. Mechanistic fine-tuning by driving the model over a barrier on the landscape! led by @EkdeepL , interning at @NttResearch at Harvard w/ @bigel_o R. Dick @DavidSKrueger

1

6

28

1

1

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

We then hypothesize that the conservative scoring combined with "iterative" re-evaluation can avoid layer collapse. This insight also explains how iterative magnitude pruning avoids layer-collapse to identify "winning-lottery ticket "subnetworks at initialization. 4/

Tweet media one

1

0

10

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

25 days

Nice detailed coverage of our "CBS-NTT Program in Physics of Intelligence"🧠 "As history teaches us, inventions can lead to new fields in physics. Today, AI is playing that role ... to explore fundamental questions involving the science of intelligence"

Tweet card media

NTT Research announces gift to Harvard Center for Brain Science to support research in Physics of...

The field of Artificial Intelligence (AI) is built using neural networks that mimic the neural pathways in our own brains. Yet despite the powerful capabilities of current AI technology, its software...

southhillslaundry.com

1

3

9

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

21 days

応援ありがとうございます、シェインさん！ぜひ池袋サンシャインの誓いの通り、日米、産学、異分野のAIコミュニティを一緒に繋げていきましょう😉

@shanegJP

シェイン・グウ

21 days

ヒデノリさんはスタンフォードで客員研究員時に知り合った日本人のAI研究者です。私が尊敬しているAI研究者の中には物理出身が多く、彼のプログラムも非常に面白そう（彼の指導教授のSurya Ganguliがいい例、2015年にDALLEなどで有名な拡散モデルDiffusion

1

8

66

0

0

9

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

This is just beautiful. "The development of a single-celled zygote into the hatched larva of an alpine newt" in six minute of time-lapse.

Tweet card media

Watch a single cell become a complete organism in six pulsing minutes of timelapse | Aeon Videos

Watch a single cell become a complete organism in six pulsing minutes of timelapse

0

7

9

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

4/

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

Q. What does Noether’s theorem tell us about the “geometry of deep learning dynamics”? A. We derive Noether’s Learning Dynamics and show: ”SGD+momentum+BatchNorm+weight decay” = “RMSProp" due to symmetry breaking! w/ @KuninDaniel #NeurIPS2021 Paper: 1/

9

132

600

2

0

8

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Our young group funded by NTT Physics and Informatics Lab () uniquely bridges industry and academia, focusing on the intersection of physics, neuroscience, and machine learning. View our recent works here: 2/

1

0

8

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

@RogerGrosse How about "multiplicative emergence" or "compositional abilities"? This emphasizes how tasks like CoT, achieved through a composition of N atomic abilities, exhibit a "multiplicative" effect on their metrics. We introduced the term in our NeurIPS paper:

Tweet media one

Tweet media two

0

0

7

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

Shaneさん、楽しみです！

0

1

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

“What shapes the loss landscape of self-supervised learning?” led by Ziyin Liu and @EkdeepL 6/

Tweet media one

0

0

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

@AlexGDimakis Please also check our NeurIPS paper on "Multiplicative Emergence of Compositional Abilities" (). We provide a concrete example of this by training text-conditioned diffusion models on interpretable compositional generalization tasks!

Tweet media one

Tweet media two

2

1

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

@EkdeepL @NttResearch @bigel_o @DavidSKrueger For more, see this tweetprint by the amazing @EkdeepL who led this work!

@EkdeepL

Ekdeep Singh

1 year

Preprint time! 🧵 DNNs can use entirely distinct prediction mechanisms to solve a task (e.g., background vs. shape). Q1: Are such models mode-connected in the landscape? Q2: Can we change a model’s mechanisms by exploiting such connectivity? Link: 1/12

5

32

139

1

0

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

An inspiring thread about the beautiful work led by wonderful colleagues @LoganGWright1 & @tatsuhiro_onod at NTT PHI Lab + @peterlmcmahon group. Looking forward to the future of this general approach that fundamentally integrates physics and neural computation!

@LoganGWright1

Logan G Wright

2 years

Our physical neural networks paper published in @nature last week. The main message: Everything can be a neural network (and can be trained efficiently with backpropagation). In light of common reactions/questions, a commentary 🧵 on key ideas, limitations, and futures. 1/n

Tweet media one

8

110

615

0

0

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Growing a healthy miniature brain (cerebral organoid) at the air-liquid interface. Very interesting thread.

@mad_lancaster

Madeline Lancaster

5 years

ICYMI We've got a new paper out in @NatureNeuro and I want to tell you about it. Here's the paper, and here's a short explainer - a thread. Cerebral organoids at the air–liquid interface generate diverse nerve tracts with functional output 1/11

Tweet media one

Tweet media two

4

99

254

0

0

6

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

2 years

A thought-provoking article about how science and technology interact and evolve together in a nurturing research environment. Much to learn from history in exploring the science of deep learning!

@quantumcascade

Federico Capasso

@quantumcascade

2 years

Inspiring article: Venky Narayanamurti our former Dean had a distinguished career as scientist and director at Bell Labs where I was fortunate to have him as a mentor. Research is fragile and needs to be nurtured in a special way in order to thrive!

1

19

65

0

0

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

1 year

5/

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

BatchNorm, LayerNorm, WeightNorm… Too many normalization layers to choose from? Which one to use, when, and how? Theory can guide you! led by @EkdeepL , who amazingly bridges theory & practice of deep learning!

1

26

134

1

0

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

29 days

We take a highly interdisciplinary approach, bringing computer science, brain science, and physics to better understand the mathematical principles that underlie intelligence. Excited for the future of this program – stay tuned for more to come! 🚀 @NttResearch @Harvard

1

0

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 years

Excited to find a thorough & timely review video on our theory of neural network pruning and SynFlow algorithm. Thank you for the thoughtful summary and feedback. @ykilcher

Tweet card media

SynFlow: Pruning neural networks without any data by iteratively...

The Lottery Ticket Hypothesis has shown that it's theoretically possible to prune a neural network at the beginning of training and still achieve good perfor...

www.youtube.com

@ykilcher

Yannic Kilcher 🇸🇨

4 years

Pruning is hard 🙂 Pruning before training is harder 😲 Pruning before training WITHOUT looking at data seems impossible 😱😱 Watch the video to find out how SynFlow achieves this. @Hidenori8Tanaka @dyamins @SuryaGanguli

Tweet media one

3

29

150

1

2

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 months

Our group funded by NTT Physics and Informatics Lab () uniquely bridges industry and academia, focusing on the intersection of physics, machine learning, psychology, and neuroscience. View our recent works here: … 2/

Tweet card media

Hidenori Tanaka

Hidenori Tanaka Group Leader, NTT Research at Harvard University Google Scholar email: hidenori_tanaka [at] fas.harvard.edu, twitter "Physics of Intelligence" for Trustworthy and Green AI News:...

sites.google.com

1

0

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

3 years

@NttResearch @EkdeepL @KuninDaniel Thanks to my collaborators & more details soon! We, Neural Network group at NTT Physics and Informatics Lab, are young and growing. Please feel free to reach out for discussions, collaborations, and internship opportunities.

0

0

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

Join us today and tomorrow for workshop presentations and discussions at #NeurIPS2023 !

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 months

At #NeurIPS2023 to share a series of works on "Physics of AI" and "AI for Neuroscience" from NTT Research at Harvard CBS team! 🎉 Find me at the posters and/or DM me to chat & explore collaborations! Can't thank the Summer 2023 team enough♥️: @EkdeepL @fatihdin4en @MayaOkawa

Tweet media one

0

20

71

0

2

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Kyogo, @kyogok , is one of the very best biophysicists I've met (+lived with!). He does both theoretical physics & experimental biology in the new exciting lab — a great opportunity at RIKEN, Kobe, Japan.

@kyogok

kyogo kawaguchi

5 years

Young Investigator Award @HFSP (3yr grant) with @NRivron and Shantanu (MIT)! Will be doing something fun related to organoid image analysis!

1

1

14

0

2

5

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

5 years

Commencement at Kyoto University (my home!) is famous for students wearing cosplay. Seems like this year, physics students dressed as the legendary “Course of Theoretical Physics” by Landau and Lifshitz! Miss the absolute freedom. ⛩️

0

0

4

@Hidenori8Tanaka

Hidenori Tanaka

@Hidenori8Tanaka

4 months

If interested, email me with your CV and research interests submit your application materials to “PHI Lab Research Intern 2024” () 3/

0

1

4