Quantifying and analyzing animal behaviors involves many pre-defined subtasks such as annotating data (pose or behavior), training machine learning models (DeepLabCut, CEBRA etc.), writing post-hoc analysis code, and/or running visualization tools. Those pre-defined sub-tasks require domain expertise in coding, machine learning, and often heavy manual annotation. Yet, there has not been one system that can help abstract away all the sub-tasks and focus on interaction-based programming to meet researchers’ high-level language queries. Here, inspired by the design of Software-3.0, we introduce a Large language Models (LLMs) based system called AmadeusGPT. This system interacts with researchers via natural language, dynamically writes code that either selects the machine learning models to use (DeepLabCut, CEBRA, SAM), and/or analyzes user data and then executes code in a sandbox. Results from the code are then used to generate visualization and text answers. We benchmark our system in a range of common behavior tasks including mice performing open field assays and elevated plus maze in the lab and for gait analysis (in horses, highlighting it is not species specific). Qualitatively, we showcase a series of spatio-temporal reasoning tasks that couldn’t be done before. Quantitatively, we show that by leveraging SuperAnimals (pretrained DeepLabCut models), our system can outperform supervised methods in benchmarks such as MABe 2022 without any pose or behavior annotations. Finally, we provide AmadeusGPT with a local web server as well as step-by-step guides in Jupyter Notebooks.