The striking illusions produced by simultaneous brightness contrast generally are attributed to the center-surround receptive field organization of lower order neurons in the primary visual pathway. Here we show that the apparent brightness of test objects can be either increased or decreased in a predictable manner depending on how light and shadow are portrayed in the scene. This evidence suggests that perceptions of brightness are generated empirically by experience with luminance relationships, an idea whose implications we pursue in the accompanying paper.