Post by Erik Bergstrom, SEO Analyst and Paul Williams, Group Account Director
Apple recently confirmed the existence of its web crawler, Applebot. This web crawler, similar in nature to Googlebot and Bingbot, crawls the web to index and organize content for its two main search properties, Siri (across iOS and Apple Watch) and Spotlight Suggestions (on Mac OS X).
Similar to other search engine bots, it respects standard robots.txt rules and robots meta tags when crawling the web. When not mentioned in the robots.txt file, it will default to follow Googlebot settings. Apple has posted a brief overview of Applebot with basic information on its support site.
Historically, Apple has chosen partnerships for binding content (via APIs) into its search products. For instance, it has teamed with Wolfram Alpha to calculate various mathematical inputs requested from Siri on iOS. Movie information and show times are pulled from Rotten Tomatoes and IMDb hook-ins. The expansion of Apple’s involvement in the search industry with Applebot is an indication that it’s not yet satisfied with pushing users to “search via Google” if there’s not an immediate, semantic answer that it is able to provide for more nuanced queries.
At this time, we presume the deployment of Applebot is an indication that Apple is exploring ways to better understand web content to supply more direct, semantic answers to a variety of queries that don’t hook into current partnerships.
UNDERSTANDING APPLEBOT: IMPLICATIONS FOR BRANDS
There’s no indication of what factors may play into a theoretical ranking algorithm for Siri and Spotlight results, but based on Apple’s past efforts, it’s likely it will list a minimal amount of “top results” (similar to what it does for restaurant queries via Siri) or attempt to identify exactly what the user has requested by relying on web standards microformatting for the semantic web.
Since current search engines have banded together to form a collective organization and best practices around microformatting (in the form of schema.org), we can make the assumption that Apple is relying on this methodology for classifying the various web “objects” and “data types” when crawled by Applebot.
It’s important for brands to properly markup pertinent information points on websites, such as products, locations and review information, to accommodate not only Applebot, but also Google and Bing’s crawlability of that content.
IS APPLE LOOKING TO BE THE FUTURE OF SEARCH?
It has been reported that after purchasing Topsy, a social media analytics firm in late 2013, Apple has used its search expertise to set the stage to potentially employ its own search engine. With a focus on mobile search, it’s believed that Apple could be planning to build on Spotlight’s web search capabilities to create a general-purpose search option for iOS and OS X users. Yet this is still speculation, as there’s no firm evidence to support a future search engine to compete with the likes of Google, Bing or Yahoo Search.
For more information about Applebot, please contact your performance account team today.