If your site isn't optimized for different devices that people are using, it's not going to look good for Google when they point users to your site. This is something to keep in mind, when you're considering optimizing for mobile search and searches from tablets.
Google wants to provide a good experience to users, and users will not benefit from a site that doesn't cater to the device they are using, even if the content is there. Google recognizes this, and it seems fairly likely that they will take measures to keep your site from showing up if it's not optimized. Maybe not your site specifically, but this seems like they kind of thing they'd want to improve upon algorithmically.
Google suggests using responsive design as a way to ensure that your site looks good across devices. They don't say it will actually help you in search rankings, but considering Googe's emphasis on user experience, and the fact that they're even suggesting it, seems to indicate that this is something they're paying attention to.
Google has talked about this a number of times in the past. Here are some steps they provided earlier this year, for example. The next month, they shared more advice. Now, they're talking about making sure you give tablet users the full-sized web, emphasizing that you should not be showing these users a mobile-specific site.
"When considering your site’s visitors using tablets, it’s important to think about both the devices and what users expect, say Google's Pierre Far and Scott Main in a joint blog post. "Compared to smartphones, tablets have larger touch screens and are typically used on Wi-Fi connections. Tablets offer a browsing experience that can be as rich as any desktop or laptop machine, in a more mobile, lightweight, and generally more convenient package. This means that, unless you offer tablet-optimized content, users expect to see your desktop site rather than your site’s smartphone site."
"Our recommendation for smartphone-optimized sites is to use responsive web design, which means you have one site to serve all devices," they write. "If your website uses responsive web design as recommended, be sure to test your website on a variety of tablets to make sure it serves them well too. Remember, just like for smartphones, there are a variety of device sizes and screen resolutions to test."
They also note that another approach is to have separate sites for desktops and smartphones, and just to redirect users to the relevant version. Just make sure you're sending tablet users to the desktop version.
Still, tablets are coming in a variety of sizes these days. Some of them are getting quite small. Responsive design might be the best bet.
The two note that they "do not have specific recommendations for building search engine friendly tablet-optimized websites."