Understanding Mobile-First Indexing (3/3): Google’s Mobile-First Architecture Options for SEO - Mobile Moxie Blog
- Updated on: 2017-05-15
- Read original article here
Over the past year or more, Google has been testing different indexing and presentation models in mobile search results, as part of a transition to Mobile-First Indexing. About a month ago, Google made an announcement that indicated potential delays in launching Mobile-First Indexing; and if casual comments at conferencesare any indication of what is going on, it appears (somewhat ironically) that Google might bestruggling to integrate AMP results in their Mobile-First Index. Nevertheless, they have assured everyone that theMobile-First Index should still launch later this year, or in early 2018. Now, with Google I/O in a couple of days, those of us who work in ‘mobile SEO’ hope to get more clarity on the future of Mobile-First Indexing.
This is the third and final article in a series about Mobile-First Indexing. The first article in the series gave somebackground information about mobile indexing, and how Google has historically determined rankings for mobile content. The second article outlinedour theory that Mobile-First Indexingis less about limiting Google’s index to mobile-friendly content, and more about cross-device content being indexed in the cloud. We proposed that Mobile-First Indexing is about decreasing Google’s reliance on URLs as unique identifiers in their index; our theory also placed a heavy emphasis on the benefit that Google sees when webmasters host content in their cloud. This article expands on the concept of Mobile-First Indexing’s decreased reliance on URLs, beginning with a discussion of what will replace the URL as a unique identifier in Google’s index, and how the future Mobile-First architecture will rely on content API’s as a more efficient means of finding and indexing content. It will finish by outlining the various app and web development methods that Google has been pushing. These technologies tend to place minimal reliance on URLs and maximal reliance on feeds & APIs, which Google has recently been supporting and advancing in their developer communities.
Why URLs Are Not Ideal For Indexing
To understand Google’s vision of the future, you have to understand that Google believes the future isnotmobile-first — it is artificial intelligence/ AI-first. In his2016 Founders Letter, the CEO of Google, Sundar Pichai, said, “We will move from mobile first to an AI first world.” He reiterated that sentiment in subsequent speeches and statements throughout the year.
This emphasis on AI is important to remember in the context of SEO, because AI depends on fast processing of content and relationships. Effective AI can’t be hindered by slow crawling of websites and URLs. AI is what enables ChatBots to have meaningful conversations with people, to help them get what they need, and none of those things actually require website or a browser, though they do require “the web” in a more general sense. AI elements from the web can be plugged into any existing website, app or search result. With voice capabilities and the right hardware, they can be plugged into elements of the Internet of Things (IoT) too.
For Google to develop the best Machine Learning AI system possible, they need access to lots of information, not just static information; to be on top of everything in real time, they need access to huge feeds of information that they can process in real time. This kind of information comes from databases and APIs – not URLs.Google wants to cut out the middlemen, which it turns out, are URLs.
Google’s process of crawling and caching URLs page by page is slowed down by the need to access and ignore all of the site-design and action-oriented code in the page template again and again. Before the web was so enormous — when web content was static and being generated more slowly — , Google favored crawling around, looking for new content. But now, websites are more sophisticated and everything is moving away from static content, towards individualized experiences and unique, on-demand mixes of content. This makes reliance on individual URLs evenlessscalable.Google’s goal to ‘organize the world’s data’ becomes completely untenable when you realize that 90% of the world’s data was created in the past two years. Since the web continues to grow at an ever-increasing rate, crawling as a means to index content will only become more inefficient over time.
At the same time, Google’s ability to leverage cloud-storage and cloud-computing has improved, which flips the balance of efficiency: replicating and hosting entire databases is now potentially more efficient than crawling the pages that the databases generate.It has become more cost-beneficial for Google to simply to cache the presentation layer once and index the database or tap into it with an API directly – And that is where indexation is headed. This process is what will allow Google to rely less on URLs for indexing content, but it will also force them to become more dependent on things like APIs, XML feeds, JSON-LD feeds & ServiceWorkers.
The move away from indexing web content based only on URLs is quite significant for Google. They have been trying for years to minimize their reliance on links as a signal for quality, but this change takes that goal much further. Beyond just de-emphasizing the algorithmic value that one URL can share with another URL through linking, it opens up indexing to a huge amount of web content that historically could not be surfaced because it relied on a browser to display a URL. More and more people are accessing web content from devices like Google Home and Amazon Echo, where the limitation of only being able to surface content with a URL becomes apparent. Google knows that to really embrace cross device interactivity, they have to include devices without browsers, or sometimes even without screens. With this in mind, it is still important to note that this shift is called Mobile-First Indexing, and not Mobile-Only Indexing. Existing websites that meet a basic mobile-friendly standard are expected to remain in the index. They will just have to compete with additional content, and there is a chance that the new, mobile-first oriented content may be given an algorithmic or similar type of ranking advantage.The point is not that URLs will be eliminated from indexing, but that URLs will be eliminated as prerequisites of indexing.
With all these changes, it is hard to know how to focus development efforts with SEO in mind. Over a year ago, Google began introducing and advocating new mobile site development options that seem particularly well oriented for Mobile-First Indexing because of their reliance on feeds, APIs and Google-hosting. Many of these development options already seem to get priority in mobile search results, despite Google’s insistence that they are not algorithmically preferred.
Until Google provides more clarity, SEOs can consider the following development strategies as the defacto Mobile-First options: Progressive Web Apps (PWAs), Progressive Web AMP pages (PWAMPs), Google My Business landing pages and Google Assistant Actions, all detailed below:
Progressive Web Apps:Often called simply PWAs, these are HTML5, JavaScript-heavy web apps that behave like native apps. They are available on web URLs, but also include an option to be downloaded and added to a mobile home-screen like native apps. When users ‘download’ the app, what they are really downloading is a file for app metadata called an ‘App Manifest’ and a file called a ServiceWorker. In practice, the ServiceWorker caches and configures content for speedier presentation and smooth, app-like UX. In reality though, the ServiceWorker is divided into two parts: An app shell that controls templates and behaviors of the PWA display, which is set to pre-cache using the CacheAPI, and another part that controls which elements of the database content should be cached in the phone’s local storage, called theIndexedDB API.
Only unique text, images and structured data relationships are stored in the offline database cache, but the app shell and action scripts are stored in the other part of the ServiceWorker. This ensures that any content that has been viewed on a device before can be viewed again, even when the phone is offline. (Google actually has an internal initiative calledOffline-First Design, that focuses on stuff like this.) This is great for users, but also great for Google, because all the unique web content is now easily accessible in a JSON-LD formatted API that Google can, as the name suggests, ‘index.’ The ServiceWorker essentially becomes an API for your site, where the content is already separated from the design, and marked up with appropriate schema, so Google knows what it is.
PWAmp Mobile Apps:Many developers who build great PWAs are also leveraging AMP code to further improve load time for an even better user experience. Remember, webmasters can use AMP HTML, CSS and JavaScript code on any page to get most of the speed benefits, even if the pages are never intended to validate. This code upgrade allows developers to leverage theAMP-ServiceWorker,which allows the PWA to work within the confines of the Google-hosted AMP landing pages, leveraging theAMP URL API, which maps Google-hosted AMP urls that Google serves to their original AMP pages on the primary domain. This allows an AMP-valid PWAmp page to act as a doorway to the rest of the web app, allowing for better tracking and control of the interaction after the initial page load.
Google My Business Landing Pages: The business side of Google+, called Google My Business, is already great for Mobile-First Indexing. It is instrumental for business who want to control their results in Google Maps and Google Local queries, a place for businesses to collect star rankings and reviews, and a primary way that businesses promote local rankings, especially if they don’t have a website. Since the content that you submit to Google My Business is already hosted in a database in Google’s cloud, they don’t need an API. We anticipate that Google may soon begin to renew their efforts to promote Google My Business interactions, because it fits with their goals of enabling more people to leverage web content without needing developers.
The important thing that all of these development techniques have in common is that they almost completely segregate content from design. In all cases, this segregation makes it much more efficient for Google to crawl and index the content without eliminating the presentation layer. Significantly though, it also makes it easier for Google to become the presentation layer, like in the case of Answers, Maps, Images and similar results, which often float to the top, whenever they believe it is advantageous. With this in mind, it is no coincidence that the most recent Chrome update focused on improvements forcaching content for offline reading, and the most recent Headless Chrome updatefocused on not needing a visible UI shell. It is also not a coincidence thatChrome OS and Android OS are becoming interoperable. The differences between app and web are going away, and Google want’s to leverage their assets on all platforms possible.
Firebase is a cloud-hosted database that has been part of the Google suite of services for about four years. It was a major focus in many of the 2016 Google I/O sessions, and it seems to be central to Google’s plans moving forward, especially as they relate to Mobile-First Indexing. Last year, after making it a main focus at Google I/O, Google migrated most of their deep linking and app indexing documentation into new locations inhttps://firebase.google.com/. While the documentation and marketing materials still feel disjointed and sometimes obtuse, the overall goal of the platform appears to be enabling native apps and web apps to work seamlessly across many different connected devices, especially Android Auto, Android TV, wearables, Google Home and other elements of IoT.
Firebase is being positioned as an app management platform that allows cloud-hosting for your app and all its content. It works for PWAs since they are considered ‘apps,’ but it does not work for regular websites. In its best implementation, it helps webmasters to associate Android app screens with their corollary iOS app screens and/or corollary web content in a PWA, so that the three versions of the same content can be indexed and ranked together, and user interactions across the different platforms can be tracked and attributed more accurately. It is important to note that Firebase does not support regular mobile-friendly websites –only PWAs. This is another strong signal that Google is very bullish about the future of PWAs.
A useful byproduct of Firebase integration is Dynamic Links, which allow webmasters to use one link to share a piece of content on any device, regardless of whether the app is installed or not. Dynamic Links act like a disambiguation engine, to direct users to the right content on the fly, without knowing before the request if it will need to work in an Android app, iOS app or just on the web. You may recall the predecessor to Dynamic Links, Google’s Goo.gl URL Shortner, which first launched Now on Tap but is now being given new life again in Firebase.
While app assets don’t have to be hosted in Firebase for Dynamic Links to work, it helps. There arefour ways to create Dynamic Links, including a manual option and an API, which can work with iOS Dynamic Links, which could presumably help with iOS app indexing, though this is not made clear in the documentation. Android documentation is conspicuously missing from the explanation, so we think this may indicate that Google will be automatically transitioning Android deep links to Dynamic Links in the near future, or possibly when Mobile-First Indexing officially launches.
Dynamic Links have significant potential to change how we interact with apps on a variety of connected devices, especially if Apple removes the existing impediments to iOS integration that de-valued iOS app indexing last year (specifically the prohibition of the SafariViewController integration in Google’s App Indexing SDK for any apps submitted to the App Store). If Apple creates new impediments (like they did with the App Indexing SDK and the SafariViewController), iOS developers may have to be persuaded to develop two versions of their iOS app – one for submission to the App Store, and one for integration with Firebase. Alternately, Google may be able to persuade iOS developers to build their apps for integration with Firebase by making it a requirement to continue ranking in Google App Packs, since those App Packs now send a significant amount of traffic to the App Store.
Firebase also appears to be the enabling force behind Android Instant Apps, which may provide an alternate option for app developers, especially if iOS continues to limit app developers from integrating with Google. With Android Instant apps, Google enables native Android apps to be indexed, and run directly from the cloud in a browser, without a download. In the future, Google may try to use ‘Instant Apps’ technology to render the native app experience completely from the web, even if no corresponding website or PWA is included in the Dynamic Link. They may also try to run Instant Apps on iOS devices, to try to make the App Store irrelevant.
The focus and growth of Firebase is important because native apps have historically created a significant indexing hurdle for Google. They are not built in HTML and don’t live entirely on the web, so they can’t be accessed by regular crawlers which simply simulate web browsers. Apps natively don’t have URLs, but instead rely on ‘app schemes’ which generate app URI’s to locate experiences or ‘screens’ within the app. These function much like URLs, but can only access content in the app code, rather than on the web. Many apps now also pull in web-content, but this is done either with APIs into the content database, or with ‘web-viewer’ code, which allows the app to behave like a browser.
Like on the web, the new strategy for app indexing seems to focus on hosting the apps with Google and separating content from design, to make it easier to access the content as a feed or in an API. The following should be considered the best Mobile-First app development options:
Android Apps: Deep-linked Android apps can be indexed, but Google still relies heavily on their association with corresponding web content to determine the relevance of a deep link in a search result. In the past year, Google actively encouraged Android developers to update their app schemes to be HTTP or HTTPS, and to update app schemes URIs to match the web URL formatting of the corresponding website. This allows app URIs and web URLs to become interchangeable, making it easier for Google to match app content to its corresponding web content. Android apps that have been set up with this type of deep links can be crawled and indexed natively by Google.
Apps that use HTTP or HTTPS schemes should also incorporate Google’s App Indexing API and a “Digital Asset Links” mapping file that associates existing web URLs with app content and is hosted at the root of the domain. This mapping file allows Google to understand which web URLs should visually indicate and link to a deep link in a Google search result, when a user has the app installed. As mentioned above, the lack of documentation about automating the creation of Android Dynamic Links inGoogle’s recently updated documentation(5/5/17), is suspicious. Google may have figured out a way to automatically create Dynamic Links for Android apps, and that may become apparent soon, if they announce a merger between Google Play Console and Firebase Console at Google I/O next week. Android apps that still rely on custom schemes (not HTTP, HTTPS, or not mirroring web URLs) must also markup corresponding web pages or sitemaps with rel=alternate tags that refer to the custom URIs.
iOS Apps: From the middle of 2015 to the middle of 2016, Google was able to index iOS apps using their app indexing API. If that was still the case, iOS apps would be a great option for Mobile-First Indexing. Unfortunately, Google and Apple were only able to cooperate on App Indexing for a short while, until the mechanism for indexing apps with Google actually became grounds for an app being blocked or rejected from the iTunes App Store. iOS apps are included here provisionally because, as described above, it appears that Firebase may offer a new mechanism for iOS app indexing, through a new API, which generates Dynamic Links for custom link schemes in iOS. If this works and can be maintained, it will be incredibly useful for enabling the full expression of the cross-device nature of Mobile-First Indexing, but based on the rough history that iOS and Google cooperation on this issue, and the potential market-share that Apple stands to lose by allowing their loyal users to be exposed to non-iOS cross-device integrations and experiences, iOS apps are only included in this list provisionally.
Private-Index App Screens: Both Android and iOS allow app developers to set up some app screens for ‘private’ indexing, so that a user’s search can surface the screens with the user’s private information for their viewing only. In both cases, the process involves setting up the apps with deep links, and integrating them with an API – for Android it is called the App Indexing API and for iOS it is called CoreSpotlight. CoreSpotlight only works for searches in Siri, Safari or Spotlight search. On iOS, Google can currently surface Private Index content only from the Google family of apps, and only when users are logged into their Google account, but it seem unlikely that they will ever be able to surface other deep-linked app content, since that is currently only accessible through the native Apple OS. Android users are able to surface properly marked-up Private Index content in Google Now and in SERPS for logged-on Google searches.
Android Instant Apps: Google has also actively been testing native Android apps that live in the cloud and don’t need to be installed to function. They generally have no web-counterpart, but are indexable via deep links in the app code that can launch small parts of the app directly in the browser, and probably leverage Dynamic Links for indexing. While the process that enables this has not been publicly described, it appears to be made possible by cloud-hosting and dynamic-linking in Firebase. The important part here is that the process allows apps without website counterparts to be indexed. It is not clear how rankings for the content of these instant apps will be determined, but it seems likely that it will be a combination of signals like schema, engagement, and visible text, similar to web, but without web links. If Google can make Instant Apps a success, the move will go a long way to making app stores a thing of the past.
Google Assistant Actions: Google Actions are essentially an interactive presentation layer that can be accessed visually or as a series of verbal questions. The two presentation layers are designed to solicit the same answers, just in different ways. The utility linked into a database API, and capable of executing simple commands inside the database. Because of the requirement for both presentation layers – visual and voice, they work with or without a browser, and can add value to Google Home products. Google Actions seem to create interactions that are very similar to those that were temporarily available with the Google Now API, which disappeared from public discourse about two years ago, shortly after they were first announced. Google Actions might also be considered mini-Android Instant Apps that are juiced up with voice compatibility. ChatBots are an integral part of Google Actions and can be developed specifically for use in Google Actions, or can be deployed in parallel in Facebook Messenger, Twitter and Slack with the API.AI utility, as shown below.
While Action Schema and Google Actions do not yet seem inextricably tied together, they could be at some point in the future. Action schema has been around for a while, but the markup was often embedded in the HTML of pages or emails, rather than formatted as JSON-LD in the head of the document or in a separate file altogether, which is where schema seems to be headed now. What is important to note here is that the OpenTable Action is the first result, in position zero. Much like other position zero content such as Knowledge Graph, Answers, Maps, Images, Videos and Shopping results, Actions are hosted by Google, and use API’s to interact with the databases that control their content.Our guess is that Google will begin to incentivise webmasters to transition their websites into PWAs by enticing them with better cross-device tracking, free database or asset hosting and the improved ability to track and manage crashes. They will also focus their messaging on the vast improvements in user-engagement and conversion that PWAs statistically give, as well as the decreased reliance on data, increased speed and their ability to work offline. Once web content is set up as a PWA, it will likely be much easier for webmasters to package and repackage content into different Google Actions or pull elements of the content into other app experiences, like Android Auto with minimal effort or developer involvement.
How Might Mobile-First Indexing Benefit Us All in the Future
The transition to web-app experiences that engage the server in an on-going way, based on individual pieces of content rather than one full page/ URL at a time is important, and may be vital in Google’s transition to Mobile-First Indexing. Google aspires to accommodate this new model of development because it allows for a more customized user experience with a faster load time. It also makes things like Big Data-style machine learning, and AI responses much easier.
People love app experiences, but they are hard to manage and develop, and development issues often limit the marketing and promotion that companies can execute. However, once content is fully separated from the presentation layer, those concerns are minimized. Google sees Firebase as a way forward, and it may soon become become critical to managing mobile-first rankings and marketing. It forces more of the app control and configurations into the cloud, and in return, provides the potential for real-time reporting, and configuration of the apps. This system also hosts the content, making on-going crawling unnecessary.
There are also subtle signals that Firebase may also be used to manage new Home devices or other elements of the connected home, which will give the platform even more marketing and analytics power. But the Internet of Things (IoT) has created the need for more voice-enabled products, so assets like ChatBots that leverage AI or simply make AI data easier to collect will be important to many companies. Google sees the future, and believes this transition to Mobile-First Indexing is a critical part of it, and we tend to agree.
