什么样的眼睛

Many existing methods utilize estimated depth information to project 2D image features into 3D voxel space?[16, 17, 18, 19, 20]. LSS?[21] explicitly predicts depth distributions to lift 2D image features into 3D space. More recent approaches?[3, 22, 23] have highlighted the importance of accurate depth estimation in the view transformation process. For instance, BEVDepth?[3] encodes both intrinsic and extrinsic camera parameters into its depth refinement module for improved 3D object detection. BEVStereo?[22] introduces an efficient temporal stereo mechanism to enhance depth estimation quality. In addition, some studies?[24, 25, 26, 27] focus on optimizing the projection stage itself. However existing approaches have largely overlooked the potential of leveraging height prediction as a direct supervisory signal for view transformation. Inspired by DHD?[5], our method introduces a novel approach by leveraging height scores from 2D images to guide the transformation into 3D voxel space. This enables more accurate height estimation in the voxel representation and lays a solid foundation for subsequent BEV feature reconstruction.

3D occupancy prediction reconstructs the 3D scene by predicting a dense voxel grid that encodes spatial occupancy?[12]. A straightforward approach is to extend the BEV representation used in 3D object detection into voxel space by lifting BEV features into 3D?[14] and applying a segmentation head?[11, 4, 8, 13]. However, this strategy disrupts the geometric integrity of the voxel representation due to the lack of vertical structure awareness. Directly processing full 3D voxel features, on the other hand, incurs substantial computational overhead. To address this, COTR?[9] introduces a compact voxel representation via downsampling, while PanoOcc?[28] proposes a novel panoramic occupancy segmentation task and leverages sparse 3D convolutions to reduce computation. Nonetheless, these methods still suffer from limited inference speed, making them less suitable for real-time deployment. To tackle this challenge, we propose a 2D-based approach that preserves the geometric structure of the voxel space while significantly improving inference efficiency, offering a more deployable and scalable solution for 3D occupancy prediction.

In 3D occupancy prediction tasks, the surrounding environment is discretized into a voxel-based volumetric grid?[21]. Assuming the ego vehicle is located at the origin of the global coordinate system, the 3D perception range is defined by the bounds $[X_{s},Y_{s},Z_{s},X_{e},W_{e},Z_{e}]$, where $[X_{s},Y_{s},Z_{s}]$ and $[X_{e},Y_{e},Z_{e}]$ denote the lower and upper bounds along the height, width, and depth axes, respectively. The scene volume is partitioned into a grid of shape $[X,Y,Z]$ (e.g. $[200,200,1]$ as in?[14]). Accordingly, the physical size of a single voxel along each axis is computed as $[\frac{X_{e}-X_{s}}{X},\frac{Y_{e}-Y_{s}}{Y},\frac{Z_{e}-Z_{s}}{Z}]$. At inference time, each voxel is assigned a binary occupancy state—occupied or empty along with a semantic label from a predefined set of categories or an unknown class. The final voxel-wise predictions are derived from visual features extracted from multi-view images, denoted as $\mathbf{X}\in\mathbb{R}^{N\times C\times H\times W}$, where $N$ is the number of cameras, $[C,H,W]$ is the feature size. Each camera is associated with both intrinsic and extrinsic parameters ($K$ and $[R|t]$), which are utilized to transform 2D image features into the unified 3D coordinate space. These parameters enable precise projection and aggregation of multi-view information into the voxel grid for occupancy and semantic prediction.

The overall architecture of DA-Occ is depicted in detail in Fig.?3. It comprises the following key components: (1)Image Backbone: A convolutional neural network (e.g., ResNet?[29]) is employed to extract high-level visual features from multi-view input images. (2) Geometry-Aware Feature Encoder: This integrates a depth estimation network (DepthNet) and a height attention network (HeightNet) to enrich the geometric understanding and improve the structural fidelity of subsequent 3D representations. (3) 2D-to-3D View Transformation: Image-plane features are lifted into the 3D voxel space using calibrated projection matrices, enabling spatial alignment across multiple views. (4) Geometry-Aware Feature Decoder: The final decoding stage incorporates geometric priors to enhance 3D structural consistency and improve the granularity of semantic predictions within the voxel grid.

In order to preserve the consistency of 3D geometric structures, particularly along the vertical axis, which is often underrepresented in 2D BEV-based representations, we introduce the Directional Attention Mechanism. This mechanism is utilized in both the Geometry-Aware Feature Encoder and the Geometry-Aware Feature Decoder, but at different stages of the model pipeline. In the Encoder, it is applied before the 2D-to-3D View Transformation to dynamically acquire height scores, which help enhance the vertical geometric features. After the transformation, in the Decoder, the Directional Height Attention (DHA) and Directional BEV Attention (DBA) modules are employed to refine and reconstruct the vertical and horizontal geometric structures, ensuring a comprehensive and accurate 3D representation. Inspired by ParC-Net?[30], which captures long-range dependencies in 2D image space, we propose a novel extension of this mechanism to the 3D voxel domain. Our extension dynamically extracts 3D geometric features, enabling the model to better capture long-range spatial dependencies and enhance the representation of 3D structures. By introducing direction-aware attention that model dependencies along both the vertical and horizontal axes, our approach preserves fine-grained 3D structure within a purely 2D framework. Specifically, DHA is designed to capture detailed height-dependent patterns by processing features along the $Z$-axis (vertical), while DBA captures geometric dependencies along the horizontal plane within the BEV representation. The directional attention mechanism can be expressed as $\mathbf{F}{out}=\mathrm{DA}(\mathbf{F}{in},\text{dir}=i)$, where $i$ refers to either the horizontal or vertical direction in feature maps. Figure?4 illustrates the structure of this mechanism. The computation proceeds as:

where, $W$ denotes the convolutional weights dynamically generated by the MLP. The function $\mathrm{Cp}(\cdot)$ performs average pooling along direction $i$ to compress spatial information. $\mathbf{F}_{ch}$ and $\mathbf{F}_{cv}$ represent horizontally and vertically concatenated features, respectively. The position encoding $p_{i}$ is a learnable parameter with a shape of $H\times 1$ or $1\times W$, and the final convolution $\mathrm{Conv}$ integrates both structure and learned weights to generate $\mathbf{F}_{out}$.

HeightNet first introduced in?[5], shares a similar structure with the DepthNet module in BEVDepth?[3]. It is used to generate a predicted height map $H_{pred}$, which is supervised by a ground-truth height map $H_{gt}$ to facilitate accurate height estimation. The generation of $H_{gt}$ proceeds as follows: a LiDAR point $p_{l}=[x_{l},y_{l},z_{l},1]^{T}$ in the LiDAR coordinate frame is transformed to the ego-vehicle coordinate system as $p_{e}=[x_{e},y_{e},z_{e},1]^{T}$. The corresponding 2D projection onto the image plane is computed using:

where $d$ is the depth value in the camera coordinate system, $[u,v,1]$ denotes the pixel coordinates in homogeneous form, $K$ is the intrinsic matrix, and $R\in\mathbb{R}^{3\times 3}$, $t\in\mathbb{R}^{3}$ are the camera extrinsic parameters.

Following?[5], we construct a set of tuples $[u,v,d,z_{e}]^{T}$, where each pixel location $[u,v]$ is assigned both a projected depth $d$ and a corresponding height $z_{e}$ in the ego coordinate system. For each pixel, only the point with the smallest depth is retained to eliminate occluded redundancies, resulting in a ground-truth height map $H_{gt}$ that represents the visible uppermost surface in the scene.

Directional Height Attention Network is designed to predict $H_{pred}$ by leveraging structural priors along the vertical ($Z$-axis) direction. It incorporates both the geometric shape prior of objects in height and the contextual information from adjacent horizontal locations. To efficiently capture vertical geometry while maintaining lightweight computation, DHA replaces the last two residual blocks in the original DepthNet with directional attention modules. This architectural modification enables more effective extraction of height-sensitive features with minimal additional cost.

The entire Geometry-Aware Feature Encoder process is described as follows: $\mathbf{F}_{n}$ is processed by DepthNet, generating the depth score $D_{pred}$ and the feature map $\mathbf{F}_{feat}$. $\mathbf{F}_{n}$ is then passed through HeightNet, producing the height score $H_{pred}$.

To preserve the integrity of 3D geometric structures while maintaining computational efficiency, we adopt the Lift-Splat-Shoot (LSS) framework?[21] for view transformation. Unlike existing approaches such as FlashOcc?[14], which rely on a purely 2D transformation grid (e.g., $1\times 200\times 200$) and often compress the vertical dimension, resulting in significant geometric distortion, our method takes a different approach. We adopt a lightweight 3D voxel grid (e.g., $16\times 32\times 32$) to enrich the BEV feature map with enhanced vertical geometric information. This enables BEV feature map to retain vertical spatial cues more effectively and recover structural geometry with higher fidelity under similar computational constraints.

Specifically, we construct a 3D voxel grid that enables finer-grained depth and height modeling during the lifting process. The generated 3D voxel features are then sliced and fused along the depth axis ($Y$-axis), effectively capturing vertical structure while still allowing the final output to remain in a 2D feature map format for efficient processing. In the Splat stage of the LSS, we enhance the lifted 3D voxel representation by the predicted height confidence scores. Specifically, the resulting voxel projection is formulated as:

where $H_{pred}$ denote the scores along the height directions, and $\mathbf{F}_{feat,i}$ represents $i$-th pixel feature. This design enhances the model’s perception of vertical geometric structures while maintaining real-time inference efficiency, leading to improved spatial feature representation. After constructing the 3D voxel feature volume $\mathbf{F}_{3D}\in\mathbb{R}^{B\times C\times Z\times Y\times X}$, we perform slicing along the depth axis (i.e., the $Y$-axis) and concatenate the resulting slices along the horizontal axis ($X$-axis). This transforms the 3D voxel structure into a 2D feature map with enhanced perception of vertical (height-wise) geometric structures, facilitating more efficient extraction of height-aware spatial representations.This process is described as:

where the concatenation is applied along the spatial $X$-axis, resulting in a feature map $\mathbf{F}_{height}\in\mathbb{R}^{B\times C\times Z\times(Y\times X)}$

The BEV feature map $\mathbf{F}_{bev}$, which primarily encodes the target’s depth-related information, and the sliced height-aware feature map feature map $\mathbf{F}_{height}$, which captures the target’s vertical structural cues, are obtained following the view transformation stage. To enhance the geometric fidelity of $\mathbf{F}_{bev}$ and $\mathbf{F}_{height}$, we introduce DBA and DHA, which specialize in refining horizontal and vertical geometrical features through direction-aware attention.

Directional-Aware Attention. The $\mathbf{F}_{\text{bev}}$ captures the relative distance between the ego vehicle and surrounding targets. This spatial relationship can be further decomposed into front-back and left-right directional components within the BEV plane. To better model depth-related geometric structures, we apply a directional attention operator $\text{DA}(\cdot,\text{dir})$ along both the $X$ and $Y$ axes of the BEV feature map. The refined BEV representation is computed as:

where the output $\mathbf{F}_{\text{BEV}}$ has the same shape as the input $\mathbf{F}_{\text{bev}}$, and the fusion enhances geometric sensitivity in both planar directions.

Due to the previous processing of sliced $\mathbf{F}_{\text{height}}$, it preserves the geometric structure in the vertical direction. The DHA performs direction-aware processing that better aligns with the vertical structure . This directional sensitivity allows it to more effectively capture and refine height-dependent geometric features. The refinement process is defined as:

where $\mathbf{F}_{h3D}\in\mathbb{R}^{B\times C\times Z\times Y\times X}$ is the reshaped 3D height-aware feature volume, and $\mathbf{F}_{Height}\in\mathbb{R}^{B\times(C\cdot Z)\times Y\times X}$ is the final height-enhanced feature map obtained by concatenating the $Z$ axes slices along the channel dimension.

This formulation enables fine-grained recovery of vertical geometry and improves the spatial expressiveness of the height feature representation. The final feature representation $\mathbf{F}_{d\_h}$ is obtained by fusing $\mathbf{F}_{Height}$ and $\mathbf{F}_{BEV}$, effectively combining fine-grained height-aware and depth-aware information to enhance 3D spatial understanding. The whole process looks like Figure?5.

We train and evaluate our model on the Occ3D-nuScenes benchmark?[2], which is built upon the nuScenes dataset?[31]. The dataset comprises 1,000 video sequences, split into 700 for training, 150 for validation, and 150 for testing. Each keyframe includes a 32-beam LiDAR point cloud, six RGB images from surround-view cameras, and dense voxel-level semantic occupancy annotations. The perception range is defined as $[-40\,\text{m},-40\,\text{m},-1\,\text{m},40\,\text{m},40\,\text{m},5.4\,\text{m}]$, and the voxel resolution is set to $[0.4\,\text{m},0.4\,\text{m},0.4\,\text{m}]$. Each voxel is assigned one of 18 semantic classes, including 16 predefined object categories, one other class for unknown objects, and one empty class representing free space. To assess the performance of our approach, we adopt evaluation metrics widely used in 3D occupancy prediction: the mean Intersection-over-Union (mIoU)?[32], computed over all semantic categories.

Following common practice?[9, 4, 32], we train our models using 4 NVIDIA H800 GPUs with a batch size of 4 per GPU. Unless otherwise specified, all models are optimized using the AdamW optimizer with a learning rate of $1\times 10^{-4}$ and a weight decay of 0.05. During the LSS stage, the resolution of the grid for the depth score is set to $[1\times 200\times 200]$, and the resolution of the grid for the height score is set to $[16\times 32\times 32]$. Regarding the loss function, we design it based on the characteristics of geometric structure recovery in DA-Occ. Specifically, we follow the formulation proposed in MonoScene?[1] and define the overall loss as: $\mathcal{L}=\lambda_{1}\mathcal{L}_{\text{bce}}^{\text{depth}}+\lambda_{2}\mathcal{L}_{\text{bce}}^{\text{height}}+\lambda_{3}\mathcal{L}_{\text{ce}}+\lambda_{4}\mathcal{L}_{\text{scal}}^{\text{sem}}+\lambda_{5}\mathcal{L}_{\text{scal}}^{\text{geo}}$, where $\lambda_{1},\lambda_{2},\lambda_{3},\lambda_{4},\lambda_{5}$ are set as $1.0,1.0,10.0,1.0,1.0$. For inference, we use a single NVIDIA A100 GPU with a batch size of 1. The frame-per-second (FPS) metric is measured using the official mmdetection3d codebase.

Table?1 presents a comparison between DA-Occ and state-of-the-art (SOTA) methods on the Occ3D-nuScenes benchmark. The results are grouped by the number of frames used in the temporal sequence. DA-Occ demonstrates outstanding efficiency while maintaining high accuracy. Notably, even when using only a single frame for data fusion, DA-Occ* achieves accuracy comparable to that of many multi-frame models. This is largely attributed to the use of the Stereo4D strategy in its temporal fusion module?[11]. When using a single frame, the proposed method improves accuracy by 2.43% compared to FlashOcc. By employing the frame fusion technique at the BEV level?[32], DA-Occ achieves FPS of 27.7 and accuracy of 39.3%. These results highlight DA-Occ’s ability to strike an effective balance between efficiency and accuracy The way we make comparisons is all based on innovations within the model itself, for example MonoScene?[1], TPVForme?[33], CTF-Occ?[9], OccFormer?[34], BEVDetOcc?[11], FlashOcc?[14], FB-OCC?[4], GSD-Occ?[12], EOPIA?[13](Since there was no name for the model, the abbreviation of the article title was used instead.), and COTR?[9]. These results underscore the effectiveness of DA-Occ in preserving geometric structures using a purely 2D-based approach. To further validate this capability, we conduct both ablation studies and qualitative visualizations, which consistently confirm the robustness of DA-Occ in maintaining geometric structural integrity.

To accelerate experimental evaluation, we conducted ablation experiments on the model without DepthNet and without temporal fusion. To investigate the impact of height information on occupancy prediction, we varied the resolution of the 3D voxel grid and analyzed how vertical geometric granularity influences performance. The impact of the $X$ and $Y$ grid size on accuracy is negligible, as it primarily encodes depth information, which is predominantly obtained from the BEV feature map. The results, summarized in Table?2, highlight the crucial role of height-aware geometry in achieving accurate 3D occupancy predictions.

Next, to assess the effectiveness of the DHA and DBA modules in preserving geometric structures, we performed additional ablation studies on the nuScenes dataset, as shown in Table?3. The results confirm that the direction-specific dynamic convolutions in DHA and DBA are highly effective at capturing structural features, contributing to both improved performance and model efficiency.

To further evaluate the robustness of DA-Occ, we conducted experiments without using the Visible Mask. This setting simulates real-world scenarios where the model may not have access to complete or perfect information, enabling us to evaluate the model’s ability to generalize and maintain accuracy under challenging conditions. The results, presented in Table?4, demonstrate that DA-Occ achieves an mIoU of 31.31% without the Visible Mask, indicating its strong resilience to missing data and its robustness in less-than-ideal environments.

Finally, we simulate the inference speed of DA-Occ on edge devices by imposing several hardware constraints to mimic real-world deployment environments. Specifically, we limit the available GPU memory to 4GB (with other memory being occupied to simulate concurrent tasks on the edge device) and restrict the GPU’s maximum power consumption to 90W. As shown in Table?5, even under these simulated deployment constraints, DA-Occ maintains its effectiveness. This further validates the applicability of DA-Occ for real-time inference on edge devices.

In visualization experiments, we adopt FlashOcc?[14] as the benchmark model, including both non-temporal variants and deeply supervised versions for comparison. A variety of scenarios from the val set are employed to ensure comprehensive evaluation. All visualizations are generated using the official FlashOcc codebase. As shown in Figure?6, the red box highlights a significant semantic loss along the vertical axis in FlashOcc. In contrast, DA-Occ maintains geometric structures in both horizontal and vertical directions, even within a purely 2D framework. This structural consistency greatly contributes to its outstanding mIoU performance.