Abstract: This paper developed an efficient hybrid algorithm based on the spectral element-boundary integral (SEBI) method and the Levenberg-Marquardt (LM) method for solving both the electromagnetic (EM) forward and inverse scattering from isotropic and anisotropic objects straddling multiple planar layers. In the forward scattering computation, a one-dimensional smooth boundary is used to enclose the scatterers straddling multiple layers. The spectral element method (SEM) is employed to solve for the 2D EM field distribution within the internal domain. Concurrently, a boundary integral equation (BIE) is applied on the boundary, serving as an exact radiation condition to truncate the SEM domain. This hybrid SEBI method is used to compute the equivalent electric/magnetic currents on the boundary for both transverse electric (TE) and transverse magnetic (TM) modes. Finally, they are combined with 2D layered-medium dyadic Green's functions to compute the scattered fields at the receiver array. In the full-wave inversion (FWI), to reconstruct the model parameters of isotropic scatterers in the TE mode and anisotropic scatterers in the TM mode, respectively, we computes the first-order derivatives of the system's mass matrix with respect to the isotropic parameters and the first-order derivatives of the system's stiffness matrix with respect to the anisotropic parameters. These derivatives are then used to construct the sensitivity matrix. Subsequently, the LM optimization algorithm iteratively calls the SEBI forward solver to complete the inversion process. Finally, the computation accuracy and efficiency of the proposed SEBI-LM algorithm in both forward modeling and FWI are validated through several numerical experiments.