Hohoo AI Journey
Three tracks, one practice path
From software engineering to AI and embodied intelligence. Learn, validate and build in a virtual world around one long-term project.
M1 · Not started
Next: start with one concrete task
Start with validated robot tasks and test the agent with fake tools, then connect ROS2 and a virtual robot.
Natural language → robot task →Roadmap
Planned effort, not completion
01 / 25%
AI Application Engineering
- LLM API / Prompt
- Structured Output / JSON Schema
- Tool Calling / Agent Loop
- RAG / Memory
- Evaluation / Observability
02 / 15%
LLM & Multimodal
- Tokenizer / Token
- Embedding
- Transformer / Self Attention
- Context Window / Inference
- Multimodal / Vision Encoder
- VLM / VLA
03 / 60%
Embodied AI & Robotics
- Python / Linux / C++17
- ROS2
- MuJoCo / Gazebo
- Robot Data
- ManiSkill / ACT
- VLA / Sim2Real
Application skills and visual understanding converge in robot perception, decisions and actions.
M10 →Simulation first, starting in a virtual world
No robot or dedicated hardware purchases to get started. Choose environments by learning stage rather than installing every simulator. Check hardware and compute requirements before each experiment.
VL01
MuJoCo →PlannedVL02
Gazebo + ROS2 →PlannedVL03
ManiSkill →PlannedVL04
Isaac Sim / Isaac Lab →Future
Target project architecture · hohoo-embodied-agent
Target task: put the red cube on the table into the blue box.
- User
- Chat UI
- LLM
- AI Agent
- Task planner
- VLM
- Robot tools
- ROS2
- Policy
- Simulation engine
- Virtual robot
- Observation
- Robot data platform
- Dataset
- LeRobot Dataset
- Training
Robot data platform
- Virtual robot
- RGB / Depth / Robot state / Joint state / Action
- Data collector
- Time synchronizer
- Episode builder
- MP4 + Parquet
- Dataset
- Data quality check
- LeRobot Dataset
Two loops: acting and learning
Robot execution loop
- Environment
- Observation
- Perception
- Reasoning & planning
- Policy
- Action
- Environment
Data and policy iteration
- Episode
- Dataset
- Training
- New policy
Later, towards the real world · Sim2Real
The reality gap includes lighting, textures, friction, camera pose, sensor noise, object positions and robot dynamics. Study domain randomization and real-world evaluation later.
- Simulation engine
- Training
- Evaluation
- Domain randomization
- Robust policy
- Real robot
Phase 2: advanced simulation & transfer
FuturePhase 3: real robot validation
FuturePhysical hardware belongs to later validation; it is not a prerequisite for starting.
Milestones
- Not started
- Learning
- Building
- Completed
M1Natural language → robot taskNot started
Validation goal
Convert “put the red cube into the blue box” to a structured task, validate its schema and reject invalid actions or parameters.
Prerequisites
Start hereKnowledge & tools
Existing foundation · Calling an LLM with Java 8: from a first request to multi-turn chat →Next stage
M2Robot agent & fake toolsNot started
Validation goal
Validate the agent loop with fake tools, recording calls, failures and stop conditions; no physical robot control.
Prerequisites
Knowledge & tools
Next stage
M3Agent → ROS2Not started
Validation goal
Replace Python fake tools incrementally with ROS2 tools or a robot bridge; verify calls, feedback and cancellation.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M4ROS2 → virtual robotNot started
Validation goal
Connect MuJoCo first, adding Gazebo later as needed; verify control and observation from agent through ROS2 to a virtual robot.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M5Virtual pick & placeNot started
Validation goal
Use a simulated arm in MuJoCo to place the red cube in the blue box, recording trajectories, controls and failures.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M6Robot data platformNot started
Validation goal
Collect RGB, robot and joint states, actions, timestamps, tasks and success labels; build episodes and validate synchronization, encoding, storage and quality.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M7Robot CopilotNot started
Validation goal
Retrieve evidence from robot manuals, ROS2 docs, error codes and operation logs for grounded diagnosis; cite sources and test unsupported questions.
Prerequisites
Knowledge & tools
Next stage
M8Virtual vision & VLMNot started
Validation goal
Move from manually supplied scene JSON to virtual camera RGB/depth input; compare visual judgments against scene configuration and record limits.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M9Robot learningNot started
Validation goal
Train or reproduce a policy using robot data, LeRobot, PyTorch and ACT; record action-chunk execution and evaluation in a virtual robot.
Prerequisites
Knowledge & tools
Virtual labs
Next stage
M10VLA virtual robotNot started
Validation goal
When resources permit, select SmolVLA or a suitable lightweight VLA to map vision, language and robot state to actions; validate in simulation first.
Prerequisites
Knowledge & tools
Virtual labs
Suggested pace
An adjustable eight-week plan, not a completion promise. No start date is set.
- Week 1
- Week 2
- Week 3
- Week 4
- Week 5
- Week 6
- Week 7
- Week 8
Existing foundation
Calling an LLM with Java 8: from a first request to multi-turn chat →Review requests, response parsing and chat history in the published tutorial. It does not establish completion of a robot milestone.
Explore later
Build an end-to-end path first, then expand around concrete problems.