For decades now we’ve been waiting for the robot butler or smart home that knows what we need and when, but so far we haven’t come much further than digital reminders to leave for the airport and maybe follow up on an email. 

But researchers at Germany’s University of Bonn say they’ve taken an important first step in creating a self-learning computer program that can anticipate our next actions — in the kitchen, at least.

A team had two different neural networks “watch” four hours’ worth of videos in which humans are shown preparing different salads. This data was used to train the system when each of about 20 actions required to prepare the salad started, as well as the duration of each.

“Then we tested how successful the learning process was,” Jürgen Gall, co-author of a new paper detailing the research, said in a statement Wednesday. “For this we confronted the software with videos that it had not seen before.” 

The program was shown more salad-making videos and told what actions it was watching for the first 20 to 30 percent of one of the videos. It then had to predict what would happen next for the rest of the clip.

The result? The algorithm was able to predict both which actions were coming up with 40 percent accuracy. That clearly leaves some room for improvement, but it’s also a major first step.

“Our methods are the first to predict video content of up to several minutes length,” the paper reads. It’ll be presented at the Conference on Computer Vision and Pattern Recognition (CVPR), which is June 19-21 in Salt Lake City.

The system’s predictive accuracy falls the further out an action is, but still gets it right 15 percent of the time even for activities that were more than three minutes in the future. 

Still, the researchers emphasize their work is a key start to the brand new field of activity prediction. In addition to salads, the system was also trained on videos of people making a wide variety of breakfast dishes with similar success. 

8

8 sci-fi ideas that might become science fact pretty soon (pictures)

Once the technology is more dialed-in and refined it could lead to whole new ways for interacting with computer assistants and robots that anticipate when to lend just the right kind of helping hand, or whisk. 

“We want to predict the timing and duration of activities — minutes or even hours before they happen,” Gall explains.  

If that becomes reality, we could move beyond the era of smart homes that manage the thermostat and lights to helping homes that prep your ingredients and preheat the oven moments before you arrive home hungry after a long day. Decades later, Rosey the Robot is finally becoming feasible.

Crowd Control: A crowdsourced science fiction novel written by CNET readers.

Solving for XX: The tech industry seeks to overcome outdated ideas about “women in tech.”