Ultrafast manipulation of vibrational coherence provides a route to control the structure of solids. However, this strategy can only induce long-range correlations and cannot modify atomic structure locally, which is a requirement for many technological applications such as non-volatile electronics. Here we demonstrate that ultrafast lasers can generate incoherent structural fluctuations that are more efficient for material control than coherent vibrations, extending optical control to a wide range of materials. We observe that local non-equilibrium lattice distortions generated by a weak laser pulse reduce the energy barrier to switch between insulating and metallic states in vanadium dioxide. Seeding inhomogeneous structural fluctuations presents an alternative, more energy-efficient, route for controlling materials that may be applicable to all solids, including those used in data- and energy-storage devices.