Shape models (SMs), capturing the common featuresof a set of training shapes, represent a new incoming object basedon its projection onto the corresponding model. Given a set oflearned SMs representing different objects classes, and an imagewith a new shape, this work introduces a joint classification-seg-mentation framework with a twofold goal. First, to automaticallyselect the SM that best represents the object, and second, toaccurately segment the image taking into account both the imageinformation and the features and variations learned from theonline selected model. A new energy functional is introducedthat simultaneously accomplishes both goals. Model selection isperformed based on a shape similarity measure, online deter-mining which model to use at each iteration of the steepest descentminimization, allowing for model switching and adaptation to thedata. High-order SMs are used in order to deal with very similarobject classes and natural variability within them. Position andtransformation invariance is included as part of the modelingas well. The presentation of the framework is complementedwith examples for the difficult task of simultaneously classifyingand segmenting closely related shapes, such as stages of humanactivities, in images with severe occlusions