Perception Engineer

• Eyes: cameras, depth sensors, stereo, LiDAR, IMUs, sensor calibration, image pipelines, object detection, segmentation, tracking, 3D vision, SLAM, localization, and scene understanding.
• Brain: world models, visual memory, semantic maps, multimodal representations, perception confidence, prediction, and interfaces to planning and robot policies.
• Hands: object pose estimation, hand-eye calibration, grasp affordances, visual servoing, tactile fusion, clutter understanding, and manipulation success detection.
• Legs: localization, traversability, obstacle detection, human-aware motion, terrain understanding, footstep-relevant scene geometry, and dynamic environment tracking.
• Simulation layer: synthetic image generation, sensor simulation, domain randomization, dataset generation, sim-to-real validation, and perception regression tests.
• Fleet and factory layers: log replay, data quality checks, sensor bring-up, calibration procedures, perception telemetry, failure labeling, validation datasets, and monitoring.

A perception engineer develops the algorithms, models, software pipelines, calibration workflows, datasets, and test systems that help a humanoid robot understand its surroundings.

In a humanoid company, the work often includes:

• Building real-time perception pipelines from cameras, stereo cameras, depth sensors, LiDAR, IMUs, joint encoders, force/torque sensors, tactile sensors, microphones, and sometimes event cameras.
• Developing computer vision models for object detection, semantic segmentation, instance segmentation, keypoint detection, human detection, pose estimation, optical flow, tracking, and scene understanding.
• Building 3D perception systems that estimate depth, point clouds, surfaces, occupancy, object pose, free space, collision geometry, grasp regions, and traversable areas.
• Working on SLAM, localization, visual-inertial odometry, mapping, loop closure, drift detection, and robot pose relative to the environment.
• Designing sensor fusion systems that combine visual data, depth data, LiDAR, IMU, proprioception, tactile data, and robot kinematics into a more reliable world state.
• Performing intrinsic and extrinsic calibration for cameras and other sensors, including hand-eye calibration and multi-camera alignment.
• Optimizing perception software for real-time operation on robot hardware, including CPU/GPU profiling, memory bandwidth, latency, batching, quantization, TensorRT or ONNX deployment, and edge-device constraints.
• Creating datasets from robot logs, teleoperation sessions, simulation, synthetic images, customer-site runs, and failure cases.
• Building labeling, curation, training, evaluation, and regression-test pipelines for perception models.
• Defining perception-to-action interfaces: what the planner, manipulation policy, locomotion stack, safety system, or operator interface needs to receive from perception.
• Debugging failures where the robot saw the wrong object, estimated the wrong pose, lost localization, misread depth, failed in glare, missed a human, or produced a result too late to be useful.
• Supporting robot bring-up, sensor selection, sensor placement, camera cable validation, lens choice, compute budgeting, and field deployment analysis.

The job is cross-functional. You may work with robotics software engineers, embedded engineers, camera hardware engineers, manipulation engineers, locomotion engineers, controls engineers, simulation engineers, data engineers, safety engineers, field engineers, robot operators, and product managers.

What the work feels like day to day

A normal week might include:

• Reviewing robot logs to understand why an object detector worked in the lab but failed in a warehouse with reflective packaging.
• Calibrating a stereo camera rig and checking whether depth errors are coming from lens distortion, poor texture, bad exposure, timestamp skew, or an incorrect transform.
• Building a ROS 2 node that takes camera images and publishes object poses, segmentation masks, or obstacle geometry with clear timestamps and confidence values.
• Training or fine-tuning a segmentation model on robot-collected data, then testing whether it still works under occlusion, motion blur, new lighting, and unseen objects.
• Profiling a perception pipeline because a downstream manipulation behaviour needs object pose updates at a higher rate.
• Comparing classical geometry, learned depth, and foundation-model outputs to decide which is reliable enough for the current task.
• Working with a manipulation engineer to decide whether the robot needs exact 6D object pose, a grasp affordance map, a point cloud, a segmentation mask, or just a task-level success signal.
• Working with a locomotion engineer to decide which obstacles matter for walking, which can be ignored, and how stale perception data can become before it is unsafe.
• Building an evaluation set from failures: glare, darkness, transparent objects, deformable bags, human occlusion, crowded scenes, cable clutter, reflective floors, and moving carts.
• Turning a messy perception failure into a reproducible test case that can be run before the next robot software release.

The best perception engineers are not only model builders. They are systems-minded engineers who can connect geometry, machine learning, sensors, calibration, real-time software, robot logs, and physical task requirements.

Humanoid robots cannot safely or usefully act in human environments unless they can understand enough of the world around them. A robot that does not know where it is, what it is looking at, where objects are, where people are, or how reliable its sensor data is cannot make good decisions.

Perception matters because humanoids need:

1. Spatial understanding
   The robot must know where things are in 3D, not only what category appears in an image. Manipulation, walking, collision avoidance, hand placement, and human-aware motion all depend on geometry.

2. Reliable perception-to-action
   Seeing an object is not the same as using it. The robot needs outputs that downstream systems can act on: object pose, grasp affordance, obstacle geometry, traversable space, human location, task state, or uncertainty.

3. Robustness outside the lab
   Lighting changes. Objects move. Cameras vibrate. Lenses get dirty. People occlude the scene. Reflective and transparent objects break simple assumptions. Perception engineers make systems degrade gracefully instead of failing silently.

4. Real-time performance
   A perception output that arrives too late can be worse than no perception output. Humanoid robots need perception that is accurate enough, fast enough, timestamped correctly, and aligned with the robot's current physical state.

5. Localization and mapping
   Humanoids need to understand where they are in factories, warehouses, homes, labs, and customer sites. SLAM, visual-inertial odometry, mapping, and place recognition help the robot navigate and operate repeatedly in the same environment.

6. Manipulation success
   Hands need eyes. Object segmentation, pose estimation, depth, tracking, visual servoing, and tactile fusion are often the difference between a polished demo and a repeatable task.

7. Safety and human awareness
   Humanoids are designed to work around people. Perception helps detect humans, estimate motion, maintain safe zones, identify hazards, and trigger fallback behaviour when confidence is low.

8. Data for physical AI
   Robot learning depends on good visual data, labels, embeddings, failure cases, and evaluation sets. Perception engineers often build the measurement layer that makes embodied AI trainable and testable.

A simple rule: perception is the robot's bridge from sensor noise to physical action. If that bridge is weak, the rest of the autonomy stack becomes fragile.

This role is a strong fit for people who enjoy computer vision, geometry, robotics systems, real-world debugging, and model evaluation. It can be research-heavy at some companies and systems-heavy at others. The most hireable candidates show they can make perception work in a robot context, not only in offline notebooks.

Computer vision engineers moving into robotics

You already have useful skills: image processing, object detection, segmentation, model training, evaluation, camera geometry, datasets, and Python or C++ vision tooling.

You are probably missing: ROS 2, transforms, robot state, latency constraints, real-time deployment, sensor synchronization, calibration procedures, point clouds, SLAM, and downstream interfaces to planning, controls, manipulation, and safety.

Best entry angle: computer vision engineer - robotics, perception ML engineer, 3D vision engineer, object pose estimation engineer, scene understanding engineer, or perception evaluation engineer.

AI/ML engineers moving into embodied perception

You already understand training loops, model architectures, datasets, distributed training, experiment tracking, loss functions, embeddings, and deployment constraints.

You are probably missing: robot sensor data, coordinate frames, camera calibration, multi-modal sensor fusion, physical failure modes, low-latency inference, robot log replay, and how perception outputs are consumed by robot behaviours.

Best entry angle: perception ML engineer, vision foundation models engineer, robot learning perception engineer, video pretraining engineer, multimodal perception engineer, or embodied AI evaluation engineer.

Robotics students and graduates

You may already understand ROS, SLAM, navigation, perception, controls, simulation, and robot projects at a course or lab level.

You are probably missing: production-quality software, large-scale data pipelines, real hardware failure analysis, evaluation discipline, calibration depth, and performance optimization.

Best entry angle: junior perception engineer, robotics software engineer on a perception team, SLAM engineer intern, autonomy engineer intern, or robot test engineer supporting perception validation.

Robotics software engineers

You already have useful skills: C++, Python, ROS 2, Linux, middleware, debugging, logging, integration, testing, and deployment workflows.

You are probably missing: deeper computer vision, camera models, 3D geometry, model training, segmentation, SLAM, sensor fusion, and perception metrics.

Best entry angle: perception systems engineer, sensor data pipeline engineer, robotics software engineer - perception, camera integration engineer, calibration tooling engineer, or perception infrastructure engineer.

Embedded, camera, and sensor engineers

You already understand hardware interfaces, camera buses, timing, drivers, memory, synchronization, sensor bring-up, bandwidth, power, and reliability.

You are probably missing: high-level perception models, robot world-state requirements, ROS 2 perception data flow, SLAM, dataset evaluation, and autonomy interfaces.

Best entry angle: perception systems engineer, camera software engineer, sensor integration engineer, compute and sensing engineer, real-time perception pipeline engineer, or edge AI deployment engineer.

Simulation and synthetic data engineers

You already understand 3D environments, rendering, physics engines, assets, lighting, cameras, procedural generation, and automated data generation.

You are probably missing: real sensor noise, calibration drift, robot logs, sim-to-real perception failures, model evaluation, ROS 2 data formats, and task-level requirements for manipulation or locomotion.

Best entry angle: synthetic data engineer for robot perception, simulation perception engineer, perception evaluation engineer, dataset generation engineer, or sim-to-real validation engineer.

Manipulation, locomotion, and controls engineers

You already understand what the robot needs in order to act: object pose, contact geometry, obstacles, state estimates, task constraints, timing, and safety margins.

You are probably missing: model training, camera calibration, image pipelines, 3D vision, labeling pipelines, and perception-specific debugging tools.

Best entry angle: perception-to-action engineer, visual servoing engineer, grasp perception engineer, traversability perception engineer, or state-estimation/perception interface engineer.

Data and teleoperation engineers

You already understand robot data collection, labeling workflows, operator interfaces, quality checks, logs, and failure taxonomies.

You are probably missing: perception metrics, model training, calibration, 3D geometry, sensor synchronization, and how labels map to robot action.

Best entry angle: perception data engineer, annotation pipeline engineer, robot vision dataset engineer, perception QA engineer, or data quality engineer for embodied AI.

Think of perception in layers. First learn camera geometry and robot transforms. Then learn image and 3D perception. Then learn real-time robot pipelines. Then add learning systems, evaluation, and deployment.

Robotics and geometry foundations

These are the base skills for robot perception.

• Coordinate frames and transforms.
• Camera projection, the pinhole camera model, intrinsics, extrinsics, distortion, and rectification.
• Stereo geometry and epipolar geometry.
• Depth cameras, disparity, point clouds, voxel grids, occupancy, and surface representations.
• 2D and 3D transformations, quaternions, homogeneous transforms, and pose composition.
• Time synchronization, timestamps, clock drift, sensor latency, and message alignment.
• Robot kinematics at a practical level: where cameras are mounted, how the robot moves, and how robot state changes the interpretation of sensor data.
• ROS 2, tf2, image topics, camera info, point clouds, bags, launch files, QoS, and visualization.
• Basic state estimation: filtering, odometry, visual-inertial fusion, and uncertainty.

Computer vision fundamentals

These skills make perception more than model prompting.

• Image formation, exposure, lighting, blur, noise, color spaces, lens effects, and camera artifacts.
• Feature detection, matching, optical flow, tracking, and correspondence.
• Object detection and multi-object tracking.
• Semantic segmentation and instance segmentation.
• Keypoint detection and pose estimation.
• 6D object pose estimation and model-based tracking.
• Depth estimation, stereo matching, and monocular depth limits.
• 3D reconstruction, point-cloud processing, plane fitting, clustering, registration, and ICP.
• SLAM, visual odometry, visual-inertial odometry, mapping, relocalization, and loop closure.
• Scene understanding, affordance detection, semantic maps, and open-vocabulary perception.

Machine learning for perception

Useful perception engineers know how models are trained, evaluated, and broken.

• PyTorch or TensorFlow for model training and inference.
• TorchVision or equivalent model libraries for detection, segmentation, keypoints, and video models.
• Dataset curation, train/validation/test splits, hard-negative mining, label quality, and data versioning.
• Data augmentation and domain randomization.
• Model evaluation using precision, recall, AP, IoU, tracking metrics, depth error, pose error, latency, and task-level success.
• Confidence estimation, uncertainty, out-of-distribution detection, and failure thresholds.
• Transfer learning, fine-tuning, representation learning, self-supervised learning, and video pretraining.
• Foundation models, vision-language models, segmentation models, and the limits of zero-shot systems.
• Inference optimization, quantization, pruning, batching, TensorRT, ONNX, and edge deployment.

Perception systems engineering

Humanoid perception has to run as part of a robot system.

• C++ for performance-sensitive runtime code.
• Python for data tools, experiments, notebooks, evaluation, and automation.
• CUDA basics, GPU profiling, and memory-transfer awareness where needed.
• Multi-camera data pipelines.
• Camera and sensor driver integration.
• ROS 2 image transport, point cloud messages, transforms, bags, and diagnostics.
• Real-time or near-real-time processing.
• CPU/GPU profiling, memory bandwidth, frame drops, dropped messages, and bottleneck tracing.
• Hardware abstraction and sensor health checks.
• Log replay and deterministic evaluation.
• CI/regression tests for perception models and pipelines.

Humanoid-specific perception skills

These become more important when the robot has a human-like body and works in human spaces.

• Head-mounted, torso-mounted, wrist-mounted, and hand-mounted camera trade-offs.
• Self-occlusion from the robot's arms, hands, torso, and carried objects.
• Hand-eye calibration for manipulation.
• Human detection and human motion awareness.
• Scene understanding in dynamic, human-occupied environments.
• Object state tracking during manipulation.
• Grasp affordance perception and visual servoing.
• Terrain, stairs, floor transitions, obstacles, and traversability for legged systems.
• Perception confidence as an input to safety and fallback behaviour.
• Tactile and force/torque fusion for contact-rich tasks.
• Data capture from teleoperation, fleet logs, and customer deployments.

Evaluation and deployment skills

This is where candidates become useful to real robot teams.

• Building evaluation datasets from real robot failures.
• Separating model metrics from task-level robot metrics.
• Measuring end-to-end latency from sensor capture to downstream consumer.
• Creating failure taxonomies: glare, occlusion, transparent objects, motion blur, calibration drift, low light, clutter, reflective surfaces, moving people, deformable objects, dropped frames, and timestamp errors.
• Regression testing perception across software releases.
• Comparing synthetic data and real-world data.
• Monitoring perception performance in fleet deployments.
• Documenting assumptions and safe operating bounds.

Do not present this list as a syllabus where every tool is required. Different companies use different stacks. These are the common clusters to recognize.

Robot middleware and sensor data flow

• ROS 2: common framework for robot communication, transforms, image streams, point clouds, logs, launch files, diagnostics, and integration.
• tf2: coordinate transforms between robot links, cameras, sensors, world frames, and task frames.
• image_pipeline / image_proc: ROS packages for camera image processing, rectification, camera info, and image handling.
• image_transport: ROS tooling for moving image streams efficiently.
• sensor_msgs / vision_msgs / geometry_msgs: common message families for images, camera info, point clouds, detections, poses, and transforms.
• rosbag / MCAP: recording and replaying multimodal robot data.
• RViz and Foxglove: visualization and debugging of images, point clouds, transforms, robot state, and logs.

Languages

• C++: real-time perception nodes, sensor integration, point cloud processing, high-performance inference, and production robot runtime.
• Python: experiments, datasets, annotation tooling, model training, evaluation, automation, and log analysis.
• CUDA: GPU kernels, custom acceleration, memory transfers, and performance-critical perception workloads.
• TypeScript: useful for internal labeling tools, dashboards, dataset browsers, and operator interfaces.

Computer vision and 3D perception

• OpenCV: image processing, camera calibration, geometry, feature tracking, and classical computer vision.
• Open3D: point clouds, 3D geometry, visualization, registration, and reconstruction.
• PCL: point cloud processing in C++ robotics stacks.
• COLMAP: structure-from-motion, 3D reconstruction, and camera pose pipelines.
• AprilTag / ArUco: fiducial markers for calibration, localization, debugging, and lab test setups.
• Kornia: differentiable computer vision operations in PyTorch workflows.

Machine learning and model development

• PyTorch / TorchVision: model training, fine-tuning, computer vision models, experiments, and research-to-production workflows.
• TensorFlow / JAX: used in some research and production ML stacks.
• YOLO-family models: object detection, segmentation, tracking, and pose estimation use cases where real-time performance matters.
• DETR / Mask R-CNN / SegFormer / DeepLab-style models: common detection and segmentation families.
• Segment Anything / open-vocabulary segmentation models: useful for annotation, bootstrapping, and some open-world perception workflows, but not a complete robot perception stack by themselves.
• Vision-language models and vision-language-action models: useful for semantic grounding, task context, instruction-conditioned perception, and embodied AI research.

SLAM, localization, and mapping

• Isaac ROS Visual SLAM / cuVSLAM: GPU-accelerated visual SLAM options in the NVIDIA Isaac ROS ecosystem.
• ORB-SLAM / RTAB-Map / VINS-Fusion / OKVIS-style systems: common references for visual SLAM, RGB-D SLAM, and visual-inertial odometry.
• Cartographer / GMapping-style systems: more relevant to mobile robot mapping, but useful context for localization and mapping concepts.
• Factor graph and optimization libraries: useful for SLAM, calibration, and sensor fusion.

Simulation and synthetic data

• NVIDIA Isaac Sim / Isaac Lab: robot simulation, synthetic data, sensor simulation, robot learning, and perception validation workflows.
• Gazebo: robot simulation with sensors, physics, and ROS integration.
• Unity / Unreal Engine: sometimes used for synthetic data, teleoperation interfaces, simulation, or visualization.
• Blender / BlenderProc: synthetic image generation, object assets, lighting variation, and domain randomization.
• Domain randomization pipelines: lighting, camera noise, object textures, clutter, backgrounds, occlusions, and sensor artifacts.

Inference optimization and deployment

• ONNX / ONNX Runtime: model export and cross-runtime deployment.
• TensorRT: optimized inference on NVIDIA GPUs and edge devices.
• Triton Inference Server: sometimes used for inference serving in training, evaluation, or off-robot workflows.
• NVIDIA Jetson / embedded GPUs: common robot compute targets.
• Docker: reproducible environments for perception training, evaluation, and deployment.
• CMake / colcon / Bazel: build systems that show up in production robotics.

Data, labeling, and evaluation

• CVAT / Label Studio: labeling images, videos, masks, boxes, keypoints, and dataset review workflows.
• Weights & Biases / MLflow: experiment tracking and model comparison.
• DVC or dataset versioning systems: keeping perception datasets reproducible.
• Spark / Ray: useful for large-scale data processing and training pipelines.
• Custom dataset browsers: common inside robotics companies because robot data is multimodal, time-aligned, and tied to hardware context.

A good perception portfolio should show working robot perception pipelines, not only image-classification notebooks. The strongest projects include sensor calibration, robot coordinates, latency measurements, real or simulated data, evaluation, failure analysis, and clear README documentation.

Project 1: Calibrated camera-to-robot perception pipeline

Build: a ROS 2 package that takes a camera stream, calibrates the camera, detects a fiducial marker or known object, estimates its pose, transforms that pose into a robot/world frame, and visualizes the result.

This can be done with a webcam, depth camera, AprilTags, ArUco markers, a printed calibration board, and a simple robot model in simulation. The key is to show the full chain from raw image to spatial output.

What it proves:

• You understand camera intrinsics, extrinsics, distortion, and rectification.
• You can manage coordinate frames and transforms.
• You can publish perception output in a robot-usable format.
• You know how calibration errors affect physical tasks.
• You can build a reproducible perception demo without hiding behind a model notebook.

Evidence to include:

• GitHub repo with launch instructions.
• Calibration images and calibration result.
• TF tree or frame diagram.
• Video or screenshots in RViz/Foxglove.
• Pose accuracy notes and failure cases.
• Explanation of how the result would be consumed by manipulation, locomotion, or planning.

Project 2: Real-time object detection, segmentation, and tracking node

Build: a ROS 2 perception node that runs object detection or segmentation on a live or recorded camera stream, tracks objects over time, publishes detections with timestamps and confidence values, and records the output to a bag or MCAP file.

Use a model family that makes sense for your hardware. The point is not to chase the highest benchmark score. The point is to show a robot-ready perception pipeline with latency, robustness, and debugging.

What it proves:

• You can connect ML inference to robot middleware.
• You understand timestamps, dropped frames, throughput, and confidence thresholds.
• You can evaluate performance beyond a single screenshot.
• You can create a pipeline that downstream systems can use.

Evidence to include:

• Live demo video and recorded log.
• Latency measurements: sensor capture to published output.
• CPU/GPU usage and frame-rate notes.
• Evaluation on several scene conditions.
• Failure examples and next improvements.
• A clear interface definition for downstream consumers.

Project 3: 3D perception and scene geometry demo

Build: a system that uses stereo, RGB-D, or simulated depth to produce point clouds, segment objects or surfaces, estimate free space, and visualize 3D geometry in a robot/world frame.

A good version can detect table surfaces, obstacles, boxes, handles, or graspable regions. For a locomotion-adjacent version, detect traversable floor and obstacle geometry. For a manipulation-adjacent version, estimate object pose and grasp-relevant geometry.

What it proves:

• You understand depth, point clouds, and 3D geometry.
• You can transform sensor data into robot/world coordinates.
• You can reason about occlusion, missing depth, and noisy surfaces.
• You can produce outputs that are useful for physical action.

Evidence to include:

• Point cloud screenshots.
• Method explanation.
• Example successful and failed scenes.
• Accuracy or stability measurements.
• Notes on how the system handles missing/invalid depth.
• Clear README with dataset or recorded sample data.

Project 4: Visual SLAM or visual-inertial odometry evaluation

Build: a SLAM or VIO evaluation project using a public dataset, a recorded robot log, or a simulated environment. Run a SLAM/VIO pipeline, visualize trajectory, compare against ground truth or a reference, and analyze failure modes.

You do not need to invent a new SLAM algorithm. A strong project can be an honest engineering evaluation of when the pipeline works, when it drifts, and what sensor assumptions matter.

What it proves:

• You understand localization, mapping, odometry, drift, relocalization, and loop closure.
• You can evaluate perception over time, not just frame by frame.
• You can reason about sensor quality, motion blur, lighting, and dynamic objects.
• You can communicate uncertainty and limitations clearly.

Evidence to include:

• Dataset description.
• Trajectory plots.
• Error metrics.
• Failure analysis.
• Reproducible commands.
• Notes on how SLAM output would feed navigation, locomotion, or mapping.

Project 5: Perception-for-manipulation mini stack

Build: a small pipeline where a camera or simulated sensor detects an object, estimates its pose or grasp region, and triggers a simple robot action such as moving a simulated arm, aligning a gripper, or visualizing a grasp target.

This is one of the best portfolio projects for humanoid robotics because it links perception to physical action.

What it proves:

• You understand perception-to-action interfaces.
• You can handle object pose, transforms, and uncertainty.
• You can work with manipulation requirements rather than only vision benchmarks.
• You can explain what perception output is actually needed for a robot skill.

Evidence to include:

• Video showing the full pipeline.
• Transform diagram.
• Detection/pose confidence logic.
• Notes on calibration and failure recovery.
• Comparison of at least two failure cases.
• Clear explanation of what remains scripted, learned, or manually specified.

Project 6: Perception dataset and regression test harness

Build: a small dataset curation and evaluation tool for robot perception logs. Include images or videos, labels, metadata, failure categories, model outputs, metrics, and a regression test that runs before a model or pipeline change is accepted.

This project is especially strong for people entering from data, ML ops, simulation, or robotics software backgrounds.

What it proves:

• You understand that robot perception improves through data discipline.
• You can evaluate models against real failure cases.
• You can build tooling that helps a team ship safer perception updates.
• You can separate offline model quality from robot-useful quality.

Evidence to include:

• Dataset schema.
• Labeling guide.
• Failure taxonomy.
• Evaluation script.
• Before/after model comparison.
• Example dashboard or report.
• Notes on how this would scale to fleet data.

Project 7: Edge inference optimization for robot perception

Build: take a detection or segmentation model, export it to ONNX or TensorRT, run it on a GPU or edge device, measure latency/throughput, and compare accuracy and performance before and after optimization.

This does not require a full humanoid robot. It proves an important production skill: making perception fast enough to run on constrained hardware.

What it proves:

• You understand inference deployment, not only training.
• You can measure latency, throughput, memory, and accuracy trade-offs.
• You can make practical choices for robot runtime constraints.
• You can explain why a model that is accurate offline may not be deployable on the robot.

Evidence to include:

• Benchmark table.
• Hardware description.
• Model export steps.
• Accuracy comparison.
• Profiling screenshots.
• Discussion of trade-offs and remaining bottlenecks.

Perception roles often appear under several names. Use these titles and keywords when building the jobs taxonomy.

Direct titles

• Perception Engineer
• Robotics Perception Engineer
• Computer Vision Engineer, Robotics
• 3D Vision Engineer
• Perception Systems Engineer
• Perception Software Engineer
• Perception ML Engineer
• Robot Perception Engineer
• AI Engineer, Perception
• Vision & Perception Engineer
• Senior Perception Learning Engineer
• Perception Research Engineer

Specialist titles

• SLAM Engineer
• Visual SLAM Engineer
• Visual-Inertial Odometry Engineer
• Localization and Mapping Engineer
• Sensor Fusion Engineer
• Camera Software Engineer
• Object Pose Estimation Engineer
• Scene Understanding Engineer
• Semantic Mapping Engineer
• World Model Engineer
• Vision Foundation Models Engineer
• Video Pretraining Engineer
• Human Detection / Human-Robot Interaction Perception Engineer
• Grasp Perception Engineer
• Perception Infrastructure Engineer
• Synthetic Data Engineer, Perception
• Calibration Engineer, Robotics

Adjacent titles

• Robotics Software Engineer, Perception Systems
• Autonomy Software Engineer
• Robotics AI Engineer
• Robot Learning Engineer
• Manipulation Software Engineer
• Locomotion Software Engineer
• State Estimation Engineer
• Embedded Vision Engineer
• Sensor Integration Engineer
• Machine Learning Engineer, Computer Vision
• Data Engineer, Robotics Perception
• Robot Test Engineer, Perception Validation

Search keywords

Use these as job-board filters:

• perception engineer
• robotics perception
• robot perception
• computer vision robotics
• 3D vision robotics
• SLAM
• visual SLAM
• VIO
• visual-inertial odometry
• sensor fusion
• object detection robotics
• semantic segmentation robotics
• instance segmentation robotics
• object pose estimation
• scene understanding
• camera calibration
• hand-eye calibration
• depth perception
• stereo vision
• point cloud
• LiDAR perception
• ROS 2 image pipeline
• TensorRT perception
• humanoid perception
• embodied AI perception
• vision foundation models
• video pretraining robotics

Job openings change quickly. Treat this as a live company map, not a permanent list. These are strong examples to seed the Companies and Jobs sections.

Figure

Figure has current Helix AI roles that explicitly mention perception, onboard computer vision, localization, visuomotor policies, offline data annotation, 3D vision, VIO / SLAM, VLMs, diffusion models, geometric computer vision, camera models, and epipolar geometry.

Why it matters for this role: Figure is a strong example of perception sitting inside the core humanoid autonomy stack. The perception role connects computer vision, localization, ML systems, geometric vision, real-world deployment, and cross-functional autonomy work.

Useful internal links to create:

• /careers/companies/figure
• /careers/jobs?company=figure&role_family=perception
• /careers/role-atlas/robotics-ai-engineer
• /careers/role-atlas/manipulation-engineer
• /careers/role-atlas/data-teleoperation-engineer

Apptronik

Apptronik has perception-related roles around Apollo, including perception learning, SLAM, visual-inertial odometry, world modeling, object detection, multi-sensor fusion, sensor interface layers, real-time perception data pipelines, camera drivers, calibration, CPU/GPU profiling, memory bandwidth, and sensor hardware integration.

Why it matters for this role: Apptronik is a clean example of the two sides of perception: advanced learning-based scene understanding and hard systems work around sensors, drivers, calibration, throughput, and real-time deployment.

Useful internal links to create:

• /careers/companies/apptronik
• /careers/jobs?company=apptronik&role_family=perception
• /careers/role-atlas/embedded-systems-engineer
• /careers/role-atlas/simulation-engineer
• /careers/role-atlas/robot-test-validation-engineer

Tesla Optimus

Tesla has Optimus-related AI and vision roles, including vision and foundation models for humanoid robotics, plus camera software and AI systems work connected to real-time autonomy deployment.

Why it matters for this role: Tesla is a useful example for candidates interested in large-scale vision models, camera systems, edge inference, model deployment, and perception infrastructure that may span vehicles, robots, and simulated robot systems.

Useful internal links to create:

• /careers/companies/tesla-optimus
• /careers/jobs?company=tesla-optimus&role_family=perception
• /careers/role-atlas/robotics-ai-engineer
• /careers/role-atlas/embedded-systems-engineer
• /careers/role-atlas/robot-test-validation-engineer

NEURA Robotics

NEURA Robotics has humanoid vision and perception roles that describe building a complete perception stack from low-level sensor input to high-level scene understanding, including cameras, LiDAR, IMUs, force/torque sensors, joint sensors, SLAM, object detection, tracking, semantic segmentation, 3D reconstruction, ROS 2 data flow, and unified world models.

Why it matters for this role: NEURA is a strong example of perception as a full-stack robotics role: software architecture, 3D computer vision, deep learning, sensor fusion, high-performance C++, ROS 2, and downstream support for navigation, manipulation, and human-robot interaction.

Useful internal links to create:

• /careers/companies/neura-robotics
• /careers/jobs?company=neura-robotics&role_family=perception
• /careers/role-atlas/robotics-software-engineer
• /careers/role-atlas/manipulation-engineer
• /careers/role-atlas/robotics-ai-engineer

Boston Dynamics

Boston Dynamics should be tracked for perception, autonomy, Atlas, Spot, and robot software roles. Its hiring changes over time and some pages are difficult to scrape, so treat it as a company profile and jobs source to refresh regularly.

Why it matters for this role: Boston Dynamics is relevant for candidates interested in perception for advanced mobile robots, legged systems, manipulation, operator applications, autonomy, and robust real-world testing.

Useful internal links to create:

• /careers/companies/boston-dynamics
• /careers/jobs?company=boston-dynamics&role_family=perception
• /careers/role-atlas/locomotion-engineer
• /careers/role-atlas/manipulation-engineer
• /careers/role-atlas/robot-test-validation-engineer

Agility Robotics

Agility Robotics builds Digit for industrial automation and hires across robotics, AI, autonomy, manufacturing, robot software, field support, triage, and root-cause-analysis roles depending on the hiring cycle.

Why it matters for this role: Agility is relevant for perception engineers interested in bipedal robots in logistics and industrial environments, where perception has to support navigation, obstacle awareness, safety, task execution, and operations reliability.

Useful internal links to create:

• /careers/companies/agility-robotics
• /careers/jobs?company=agility-robotics&role_family=perception
• /careers/role-atlas/locomotion-engineer
• /careers/role-atlas/field-robotics-engineer
• /careers/role-atlas/robot-operations-fleet-operator

Sanctuary AI

Sanctuary AI hires AI/ML researchers, software engineers, simulation experts, mechanical engineers, and controls engineers for Physical AI and humanoid systems. Its work around dexterous robots makes perception relevant even when an open role is listed under ML, simulation, manipulation, or robotics software rather than the exact title “Perception Engineer.”

Why it matters for this role: Sanctuary is useful for candidates interested in dexterity-driven Physical AI, where perception connects closely to manipulation, tactile sensing, robot learning, simulation, and task-level intelligence.

Useful internal links to create:

• /careers/companies/sanctuary-ai
• /careers/jobs?company=sanctuary-ai&role_family=perception
• /careers/role-atlas/manipulation-engineer
• /careers/role-atlas/robotics-ai-engineer
• /careers/role-atlas/simulation-engineer

1X Technologies

1X hires across AI, world-model work, simulation, software engineering, fleet operations, hardware, manufacturing, and robot services depending on the cycle.

Why it matters for this role: 1X is relevant for candidates interested in home humanoids, world models, video data, simulation, robot learning, fleet data, and perception systems that have to operate in less structured environments.

Useful internal links to create:

• /careers/companies/1x-technologies
• /careers/jobs?company=1x-technologies&role_family=perception
• /careers/role-atlas/robotics-ai-engineer
• /careers/role-atlas/simulation-engineer
• /careers/role-atlas/data-teleoperation-engineer

A candidate becomes credible for perception roles when they can show evidence in geometry, sensor systems, model evaluation, robot integration, and deployment trade-offs.

Strong positive signals

• Can explain how raw camera data becomes a robot-usable output.
• Understands camera intrinsics, extrinsics, distortion, rectification, and hand-eye calibration.
• Can debug coordinate frame mistakes without hand-waving.
• Has built something with ROS 2 image streams, point clouds, transforms, and logs.
• Can explain the difference between object classification, detection, segmentation, tracking, pose estimation, and scene understanding.
• Understands 3D geometry, depth, point clouds, and uncertainty.
• Can measure latency from sensor capture to perception output.
• Has used logs to reproduce a perception failure.
• Understands dataset curation, label quality, evaluation metrics, and failure slices.
• Can explain when a foundation model helps and when it is not reliable enough for robot action.
• Has deployed or optimized inference on constrained hardware.
• Can define the perception output needed by manipulation, locomotion, planning, safety, or operations teams.
• Shows failures, not only polished demos.

Weak signals

• Only shows image classification notebooks with no robot integration.
• Uses “AI perception” language but cannot explain camera geometry.
• Cannot describe coordinate frames or transform chains.
• Has no calibration story.
• Does not measure latency, dropped frames, or throughput.
• Evaluates only on hand-picked demo images.
• Treats model confidence as ground truth.
• Cannot explain how perception output is consumed by a robot system.
• Ignores lighting, occlusion, sensor noise, motion blur, calibration drift, and field failures.
• Talks about SLAM without understanding drift, relocalization, loop closure, or dynamic scenes.
• Has no reproducible dataset, logs, metrics, or test procedure.

Interview questions to prepare for

• Walk me through a perception pipeline you built from sensor input to robot output.
• How would you calibrate a camera mounted on a humanoid robot's wrist or head?
• How do you debug an object pose estimate that looks correct in the image but is wrong in the robot frame?
• How would you measure end-to-end perception latency?
• What are the trade-offs between RGB, stereo, depth, LiDAR, and IMU data?
• How would you evaluate a segmentation model for a manipulation task?
• How would you handle transparent, reflective, deformable, or occluded objects?
• How do you know when a perception model is confident enough for robot action?
• What logs would you collect from a humanoid perception failure at a customer site?
• How would you design a dataset from robot failures?
• What is the difference between visual odometry, VIO, SLAM, localization, and mapping?
• How would you optimize a perception model that is accurate but too slow for the robot?
• How would you use simulation or synthetic data without fooling yourself about sim-to-real transfer?
• What perception outputs would a manipulation engineer need for grasping?
• What perception outputs would a locomotion engineer need for safe walking?

• Thinking perception is just object detection. Humanoid perception also includes calibration, geometry, depth, tracking, SLAM, sensor fusion, uncertainty, latency, and action interfaces.
• Ignoring coordinate frames. Many robot perception failures are transform failures disguised as model failures.
• Only building offline notebooks. Robot teams need perception that runs in a robot pipeline, not only in a dataset notebook.
• No latency measurements. A model that is accurate but late may be unusable.
• No calibration evidence. Calibration is not a footnote. It determines whether pixels become useful 3D information.
• Overtrusting foundation models. Open-vocabulary models are useful, but robot action needs predictable outputs, failure handling, and safety boundaries.
• Ignoring depth and 3D geometry. Humanoid robots act in physical space. 2D labels are often not enough.
• Using synthetic data without real validation. Synthetic data can help, but real sensor noise and field conditions still matter.
• Not defining downstream consumers. Perception output should be designed for the robot system that uses it.
• Hiding failures. Strong candidates show failure cases and explain how they would debug or improve them.
• Confusing model metrics with task success. A better IoU score does not automatically mean the robot completes the task more reliably.

This section is optional on Role Atlas pages, but useful for readers who are ready to act.

First 30 days: build the visual geometry base

• Learn camera intrinsics, extrinsics, distortion, rectification, stereo basics, and the pinhole camera model.
• Learn ROS 2 image topics, camera info, tf2, point clouds, launch files, bags, and visualization.
• Build a small camera calibration project.
• Use OpenCV to detect a marker or simple object and estimate pose.
• Publish the output in a robot/world frame.
• Write a clear README with diagrams.

Output: a calibrated camera-to-robot pipeline that publishes a marker or object pose and visualizes it in RViz or Foxglove.

Days 31–60: add real-time perception and evaluation

• Build a detection or segmentation node that runs on a live or recorded camera stream.
• Publish detections with timestamps, confidence values, and transform-aware outputs.
• Record and replay logs.
• Measure latency, frame rate, dropped frames, CPU/GPU usage, and failure cases.
• Create a small evaluation dataset with varied lighting, clutter, occlusion, and object poses.
• Add basic tests or repeatable evaluation scripts.

Output: a real-time ROS 2 perception demo with recorded logs, metrics, and documented failure cases.

Days 61–90: make it robot-relevant

• Add depth, stereo, RGB-D, or point-cloud processing.
• Connect perception output to a simulated robot action, manipulation target, navigation target, or safety zone.
• Add a regression test harness using recorded logs.
• Optimize inference or pipeline throughput.
• Add a failure taxonomy and evaluation report.
• Map your project to real job descriptions.

Output: a portfolio project that shows perception as part of a robot system, not just a computer vision demo.

Is a Perception Engineer the same as a Computer Vision Engineer?

Not exactly. Computer vision is a major part of the role, but robot perception also includes sensors, calibration, coordinate frames, depth, point clouds, SLAM, sensor fusion, latency, hardware constraints, robot logs, safety interfaces, and downstream action requirements.

Do I need a PhD?

Not always. A PhD can help for research-heavy roles in SLAM, 3D vision, foundation models, or learning-based perception. Many systems and production perception roles are open to strong engineers with good C++, Python, ROS 2, computer vision, calibration, evaluation, and robot integration evidence.

Should I learn ROS 2 or deep learning first?

If you already know computer vision and ML, learn ROS 2, transforms, calibration, and robot data flow next. If you already know robotics software, learn camera geometry, OpenCV, 3D perception, and model evaluation. The strongest candidates can bridge both sides.

Is SLAM required for every perception role?

No. Some perception roles focus on object detection, manipulation perception, segmentation, data pipelines, camera systems, or inference deployment. SLAM is important enough to understand, but not every perception engineer is a SLAM specialist.

Are foundation models replacing perception engineering?

No. Foundation models can help with segmentation, semantic understanding, annotation, and open-vocabulary perception. They do not remove the need for calibration, depth, timing, uncertainty, robot-specific evaluation, real-time deployment, safety, and reliable perception-to-action interfaces.

What is the fastest credible project?

A calibrated ROS 2 camera pipeline that detects an object or marker, publishes its pose in a robot/world frame, records logs, measures latency, and documents failure cases is more credible than a polished computer vision demo with no robot context.

What should I build if I want humanoid manipulation roles?

Build perception-for-manipulation evidence: object pose estimation, segmentation, depth, grasp affordance visualization, hand-eye calibration, visual servoing, and a clear explanation of how perception output feeds a reaching or grasping behaviour.

What should I build if I want humanoid locomotion or navigation roles?

Build 3D scene geometry evidence: localization, obstacle detection, floor/traversability estimation, dynamic object tracking, and perception outputs that can support footstep planning, obstacle avoidance, or safe navigation.