Abstract: In recent years, there has been stunning progress in the development of end-to-end autonomous driving technology, which constitutes a major research topic in both academia and industry. Vision-based approaches to fully autonomous driving have, from an end-to-end perspective, overcome most of the drawbacks presented by traditional modular designs. These focus mainly on sequential execution, where perception, prediction, and planning algorithms run as separate software modules. This paper discusses a multimodal end-to-end autonomous driving model based on trajectory optimization. It builds upon an already existing FusionAD framework for end-to-end autonomous driving and extends a three-level structure of planning modules for feature interaction and trajectory generation, safety optimization modeling, and trajectory optimization solver, further developing it by an enhanced L-BFGS optimization algorithm based on the quasi-Newton method to replace the original IPOPT methodology based on the Newton method. It is also benchmarked and validated on the nuScenes dataset for various elaborated models, compared with various mainstream algorithms. Results show that L-BFGS showed the best overall performance; it reduces the average displacement error by 2% at comparable collision rates. The model was then applied to the nighttime scenarios in nuScenes, where the integration of LiDAR resulted in an 18.5% reduction in collisions. The trajectory collision rate (CR_traj) in nighttime scenarios is reduced by 5.6%. The research, therefore, proves that incorporation supports LiDAR's significant role in autonomous driving and further ensures the safety of autonomous driving planning.