The assessment and quantification of human and animal behavior is essential for various applications in Neuroscience. Experts can accurately annotate complex behaviors, but doing so is a tedious and time-consuming task. Here we describe a Python toolbox, called DLC2action, that enables the automatic annotation of complex behaviors directly from videos, extracted poses and/or additional features such as deep video features, built on state-of-the-art deep-learning based action segmentation algorithms. We illustrate the capabilities of this open-source toolbox on a broad range of datasets for human and animal behavior. The toolbox offers many features, including project management, feature engineering options, automatic hyperparameter tuning, model training, evaluation functionalities, and plotting functions to monitor training. Additionally, it provides various metrics and can visualize results in the form of ethograms. Furthermore, we developed a graphical annotation interface designed to be used in combination with the action segmentation pipeline. DLC2action supports manual annotations of actions with single-frame precision, multi-camera views, pose skeleton display in 2D and 3D, segmentation masks, and search features which can guide the user towards specific label classes. The tool supports active learning workflows when used in combination with DLC2action, leading to a significant decrease in annotation time for equivalent performance. We believe that DLC2action will help neuroscientists to provide large-scale complex behavior annotations and by extension, the Neuroscience community in many aspects. The code is available at https://github.com/amathislab/DLC2action.