Many scientific and high-performance computing applications consist of
multiple processes running on different processors that communicate
frequently. These applications can suffer severe performance penalties
if their processes are not all coscheduled to run together, due to
their synchronization needs.  Two common approaches to coschedule jobs
are batch scheduling, wherein nodes are dedicated for the duration of
the run, and gang scheduling, wherein time slicing is coordinated
across processors.  Both work well when jobs are load-balanced and
make use of the entire parallel machine. However, this is rarely the
case, and most realistic workloads consequently suffer from both
internal and external fragmentation, where resources and processors
are left idle because jobs cannot be packed with perfect efficiency.
This leads to reduced utilization and sub-optimal
performance. Flexible CoScheduling (FCS) addresses this problem by
monitoring each job's computation granularity and communication
pattern, and scheduling them based on their synchronization and load-
balancing requirements. In particular, jobs that do not require
stringent synchronization are identified, and are not coscheduled;
instead, these processes are used to reduce fragmentation.  FCS has
been fully implemented on top of the STORM resource manager on a
256-processor Alpha cluster, and compared to batch, gang, and implicit
coscheduling algorithms.  This paper describes in detail the
implementation of FCS and its performance evaluation with a variety of
workloads, including large-scale benchmarks, scientific applications,
and dynamic workloads.  The experimental results show that FCS
saturates at higher loads than other algorithms (up to 54% higher in
some cases), and displays lower response times and slowdown than the
other algorithms in nearly all