A three-dimensional representation of the Roche potential in a binary star with a mass ratio of 2, in the co-rotating frame. The droplet-shaped figures in the equipotential plot at the bottom of the figure are called the Roche lobes of each star. L1, L2 and L3 are the points of Lagrange where forces cancel out. Mass can flow through the saddle point L1 from one star to its companion, if the star fills its Roche lobe. Source.
The Roche lobe is the region of space around a star in a binary system within which orbiting material is gravitationally bound to that star. If the star expands past its Roche lobe, then the material outside of the lobe will fall into the other star. It is an approximately tear-drop shaped region bounded by a critical gravitational equipotential, with the apex of the tear-drop pointing towards the other star (and the apex is at the Lagrange L1 point of the system). It is different from the Roche limit which is the distance at which an object held together only by gravity begins to break up due to tidal forces. It is different from the Roche sphere which approximates the gravitational sphere of influence of one astronomical body in the face of perturbations from another heavier body around which it orbits. The Roche lobe, Roche limit and Roche sphere are named after the French astronomer Édouard Roche.
Close to each star, surfaces of equal gravitational potential are approximately spherical and concentric with the nearer star. Far from the stellar system, the equipotentials are approximately ellipsoidal and elongated parallel to the axis joining the stellar centers. A critical equipotential intersects itself at the Lagrange L1 point of the system, forming a two-lobed figure-of-eight with one of the two stars at the center of each lobe. This critical equipotential defines the Roche lobes. If you were to hypothetically fill these lobes completely with water, a ship could sail from one star to the other, passing through the point between the two stellar oceans where the tips of their sharp peaks touch.
As such, the Roche lobe is one of two volumes of space in the system. These volumes are bounded by a particular surface of equal potential energy. The potential energy is calculated in a frame of reference that co-rotates with the binary system. Because this frame of reference is a non-inertial frame, the gravitational potentials due to the masses of each of the two stellar nuclei (which vary inversely with distance from the center of each star) must be supplemented by a pseudo-potential corresponding to centrifugal force. This pseudo-potential is proportional to the square of the perpendicular distance from the axis of rotation of the system.
Where matter moves relative to the co-rotating frame it will seem to be acted upon by a coriolis force. This is not derivable from the Roche lobe model as the coriolis force is a non-conservative force (i.e. not representable by a scalar potential).
When an object "exceeds its Roche lobe", its surface extends out beyond its Roche lobe and the material which lies outside the Roche lobe can "fall off" into the other object's Roche lobe. This can lead to the total disintegration of the object, since a reduction of the object's mass causes its Roche lobe to shrink. Overflow from the Roche lobe is responsible for a number of astronomical phenomena, including recurring novae (binary stars consisting of a red giant and a white dwarf that are sufficiently close enough together that material from the red giant dribbles down onto the white dwarf), X-ray binaries and millisecond pulsars.