Jim Fan

NVIDIA Director of AI & Distinguished Scientist, GR00T & Foundation Agent Research

Profile

FieldDetails
Current PositionNVIDIA Director of AI & Distinguished Scientist
TeamGEAR Lab Co-Lead (Generalist Embodied Agent Research)
PhDStanford University (2016-2021)
AdvisorFei-Fei Li
UndergraduateColumbia University (Valedictorian 2016, Illig Medal)
Social MediaActive AI communicator (340K+ followers)

Key Contributions

  • GR00T: NVIDIA’s humanoid robot foundation model (Co-Lead)
  • Voyager: LLM-based autonomous Minecraft agent
  • MineDojo: Minecraft-based AI benchmark (NeurIPS 2022 Outstanding Paper)
  • Eureka: Automatic reward function generation with LLMs
  • Foundation Agent Vision: Pioneered direction for general-purpose agent research
  • OpenAI’s First Intern: 2016 OpenAI Universe/World of Bits project

Research Timeline

Columbia University (2012-2016)

YearWorkImpact
2016Undergraduate GraduationValedictorian, Illig Medal

Internships (2015-2020)

YearOrganizationWork
2015Baidu AI LabWorked with Andrew Ng, Dario Amodei
2015-2016MILAResearch with Yoshua Bengio
2016-2017OpenAIFirst intern, World of Bits (with Ilya Sutskever, Andrej Karpathy)
2018Google Cloud AIResearch Intern
2020NVIDIAResearch Intern

Stanford PhD (2016-2021)

Advised by Fei-Fei Li | Thesis: “Training and Deploying Visual Agents at Scale”

YearWorkImpact
2018Video UnderstandingVideo comprehension research
2021PhD GraduationStanford Vision Lab

NVIDIA (2021-present)

GEAR Lab Co-Lead (Co-founded with Yuke Zhu, February 2024)

YearWorkImpact
2021.12Joined NVIDIASenior Research Scientist
2022MineDojoMinecraft AI benchmark, NeurIPS 2022 Outstanding Paper
2023VoyagerLLM + Minecraft autonomous exploration
2023EurekaLLM-generated reward functions
2024.02GEAR Lab FoundedCo-founded with Yuke Zhu
2024GR00THumanoid foundation model announced (GTC 2024)
2025.03GR00T N1Open humanoid VLA (GTC 2025)
2025Promotion to DirectorDirector of AI & Distinguished Scientist

Major Publications

Foundation Agent

  • Voyager (2023) - LLM-based autonomous Minecraft agent
  • MineDojo (NeurIPS 2022 Outstanding Paper) - Minecraft AI benchmark
  • Eureka (2023) - Automatic reward function generation with LLMs
  • VIMA - Multimodal foundation model for robot manipulation
  • SURREAL - Distributed reinforcement learning framework

Robotics

  • GR00T (GTC 2024) - Humanoid foundation model
  • GR00T N1 (GTC 2025) - Open humanoid VLA (2.2B parameters)

Early Career

  • World of Bits (ICML 2017) - OpenAI Universe, web browser AI platform

Key Ideas

Voyager (2023)

Core: LLM writes code to autonomously explore Minecraft

Components:
1. Automatic Curriculum - LLM proposes next goals
2. Skill Library - Store/reuse discovered skills
3. Iterative Prompting - LLM fixes code on failure

Results:
- Crafted diamond tools without human intervention
- 3.3x faster skill acquisition than previous methods

Impact:

  • Pioneered LLM + game agent research
  • Set direction for LLM utilization in Embodied AI

GR00T & GR00T N1 (2024-2025)

Core: General-purpose foundation model for humanoid robots

Architecture (Dual-system):
- System 1: Fast-thinking action model (reflexes/intuition)
- System 2: Slow-thinking VLM (deliberate reasoning)
- Diffusion Transformer for continuous motion generation

GR00T N1 Specs:
- 2.2B parameters (VLM 1.34B)
- 16 action chunks generated in 63.9ms (L40 GPU)
- Trained on real robot data, human videos, synthetic data
- Up to 1024 GPUs used for training (~50,000 H100 GPU hours)

Features:
- Natural language understanding
- Human motion imitation
- Support for various humanoids (1X, Boston Dynamics, Agility, etc.)

Impact:

  • World’s first open humanoid foundation model (N1)
  • Core of NVIDIA’s robotics ecosystem
  • Newton physics engine (collaboration with Google DeepMind, Disney Research)

GEAR Lab Vision

GEAR Lab (Generalist Embodied Agent Research) - Co-founded by Jim Fan and Yuke Zhu in February 2024

Foundation Agent

Goal: One agent performing various tasks in various environments

Games (Minecraft) -> Simulation (Isaac) -> Real Robots

Research Areas:
- LLM-based planning & reasoning
- Vision-language models
- World models trained on internet-scale data
- Robot locomotion & dexterous manipulation
- Large action models (autonomous exploration in games/simulations)
- Simulation & synthetic data pipelines for large-scale learning

Integration with NVIDIA Strategy

  • Isaac Sim: Simulation environment
  • Omniverse: Synthetic data generation
  • Jetson: Edge computing
  • GR00T: Foundation model
  • Newton: Open-source physics engine

Philosophy & Direction

Research Philosophy

“Agents that succeed in games can succeed in the real world. Generality is the key.”

Research Direction

  1. 2016-2021: Video understanding (Stanford PhD)
  2. 2021-2023: MineDojo, LLM + games (Voyager, Eureka)
  3. 2024-present: Humanoid robotics (GR00T, Physical AGI)

Communication & Influence

Active Social Media Presence

  • Twitter/X: 340K+ followers (@DrJimFan)
  • AI research commentary, vision sharing
  • Industry news curation
  • Coined “Physical Turing Test” concept

Public Outreach

  • Explains complex AI research accessibly
  • Discusses AI research directions
  • Model for researcher-public communication

Media Coverage

  • Featured in New York Times, Forbes, MIT Technology Review, TechCrunch, WIRED, and more

Awards & Recognition

  • NeurIPS 2022 Outstanding Paper (MineDojo)
  • Columbia University Valedictorian & Illig Medal (2016)
  • Stanford AI Lab alumnus (advised by Fei-Fei Li)
  • OpenAI’s First Intern (2016)
  • Google Scholar citations: 12,000+

See Also