Intelligent Inference Endpoint Scheduling for Heterogeneous Large Language Model Deployments: A Six-Dimensional Routing Framework

Abstract

Large Language Model (LLM) serving infrastructure has evolved from single-model deployments into heterogeneous fleets combining general-purpose models, domain-specialised variants, multimodal models, and quantised derivatives with widely differing context capacities. In such environments, the decision of which endpoint should serve a given request materially affects latency, throughput, monetary cost, and output quality. Despite growing industrial adoption of inference routers, the literature lacks a consolidated account of the decision dimensions such systems must reason over, or of how those dimensions interact when their objectives conflict. This paper addresses that gap. Through an analysis of inference-serving mechanisms and model-specialisation results, we identify and characterise six dimensions governing endpoint selection: load distribution, context-length requirement, input modality, task category, domain specialisation, and prefix-cache locality. For each dimension we formalise the decision criterion and describe the signals available at request-admission time, distinguishing hard feasibility constraints from soft optimisation preferences. We further show that these dimensions are not independent - notably, cache-locality routing and load-balancing objectives are structurally opposed, since the former concentrates traffic while the latter disperses it - and we propose a bounded-affinity policy and a priority-ordered evaluation sequence to resolve such conflicts. Finally, we specify an evaluation protocol comprising workload definitions, baselines, and metrics, by which implementations of the framework may be empirically validated and compared. The framework is intended as a design reference for practitioners building inference gateways and as a structuring basis for future empirical work on multi-objective LLM request scheduling.