Cosmos 1 is an ambitious experiment to harness the power of the Sun to drive a spacecraft.
Scientists believe "sun-sailing" craft may in the future be able to reach sufficiently high speeds to make travel around the Solar System viable.
The probe launched at 1946 GMT on Tuesday evening from a Russian Delta III class submarine in a Volna rocket, a converted former nuclear missile. The privately funded mission is being co-ordinated by The Planetary Society.
The two-stage rocket, fitted with a third-stage kick motor, was expected to propel the craft to a height of 840km (520 miles) above the Earth, where the experiment would begin.
However, the Russian space agency said that there were indications that the Volna rocket may have experienced a problem in its first or second firing stages.
As a result Cosmos-1 may have failed to reach its target orbit.
Once in orbit, Cosmos was designed to unfold its eight, triangular sails. These cover some 600 sq m (720 sq yds), about the size of a basketball court.
The sails can be rotated to different angles, and scientists hope this will allow them to steer and control the probe in its solar powered flight.
Cosmos works on the principle of photons - particles of light from the Sun - hitting the sail and propelling it forwards.
Acceleration is very slow, but constant; and in theory allows the craft to reach very high speeds.
Scientists predict a sun-sailing craft will accelerate at approximately five 10-thousandths of a metre per second, per second, depending on its weight and the size of the sail.
Over one day, its speed would reach 45 m/s (100mph); in 100 days its speed would be 10,000 mph, in three years 100,000 mph.
At that speed, a craft would reach Pluto, the most distant planet in the Solar System, in less than five years.