Images captured by cameras have become ubiquitous. Being able to reconstruct 3D scenes only using these images would be a highly desirable capability. However, this is very challenging, as images are 2D snapshots of the world, therefore generating ambiguities when lifting them from 2D back to 3D. In this thesis, we focus on two methods for 3D scene reconstruction: single-image depth estimation, and primitive decomposition.Single-image depth estimation (SIDE) refers to the ability to reconstruct the visible 3D surface of a scene given only a single image as input. This is an ill-posed problem, since many 3D scenes can explain the observed image. In order to solve this problem, modern works rely on data-driven methods and are mainly deep-learning based. These methods therefore use large training datasets of RGB-D pairs, i.e. aligned color and depth images, along with deep neural networks, in order to learn a good prior to predict depth from a single image.Primitive decompositions can be used to represent a scene as an arrangement of elementary shapes. In the 1960s, Lawrence G. Roberts proposed to represent the 3D world as an arrangement of cuboids. This representation is particularly useful for its simplicity and compactness, which downstream applications such as robotics could leverage.The first contribution of this thesis introduces a solution to a notorious problem in single-image depth estimation; most methods suffer from smooth edges around occlusion boundaries, while they are supposed to be sharp. Our method, called SharpNet, introduces geometric constraints with synthetic data during training in order to predict sharp depth maps, guided by occlusion boundaries and surface normals.Our second contribution extends our pursuit of sharper depth edges in SIDE. Neural networks are notoriously biased towards low frequencies, implying that sharp edges, which correspond to high frequency details, will often be overlooked by deep-learning based methods. In this work we introduce a new depth refinement method that sharpens predicted depth maps, and that is able to estimate crisp high frequency details. This method predicts displacement fields, which are used to sharpen depth edges by moving pixels in 2D space. When put on top of baseline single-image depth estimation methods, our method consistently improves the sharpness of depth maps without sacrificing their accuracy.Our third contribution also aims to improve existing SIDE methods with a simple extension. Most of these previous work rely on U-shaped Encoder-Decoder architectures, often referred to as UNets. While our two first contributions focus on accuracy along occlusion boundaries, this third contribution focuses on efficiency. These occlusion boundaries are indeed usually sparse in natural scenes, which creates an imbalance that results in smooth predicted depth edges. In WaveletMonoDepth, we instead take advantage of this sparsity. Because depth edges are sparse, we can compute convolutions only in places with large depth variations i.e. occlusion boundaries. Using wavelet decomposition as an intermediate representation for depth maps, we obtain large gains in efficiency while suffering only a minimal loss in accuracy.Our final contribution explores primitive decompositions as a representation for 3D scenes. RGB-D cameras can be used to scan scenes and store them as 3D point clouds. However, this process is often noisy, and 3D point clouds are expensive to store. With MonteBoxFinder, we propose to represent noisy 3D point clouds with 3D cuboids, by first detecting many 3D cuboids candidates, then finding an arrangement that best fits the scene. This search problem is highly combinatorial, and an exhaustive search is often prohibitive. MonteBoxFinder therefore draws inspiration from Monte Carlo Tree Search methods, in order to efficiently find a good set of cuboids. These cuboids can then be used as ground truth to train single-image cuboid decomposition method.