Why would an SEO limit themselves to some keywords, when all keywords are available?
Historically, it’s been a matter of resources and capacity. The more data points you introduce, multiplied by the number of keywords you’re tracking, the faster it becomes unmanageable. When using a typical SEO platform to track keyword data, it requires prioritization and selection of a more limited keyword list that leaves off low-traffic queries and details about things like rich results.
That’s where big data, in the raw output form that can be parsed by BI tools, comes in to expand a keyword universe.
We’ll walk through the process of what raw, big keyword data looks like, how it’s given the structure that allows it to be manipulated, and the qualities your business should look for when choosing a BI tool that will allow you to turn data into insights.
Table of Contents
Why use Big Data for SEO
SEOs have a finite amount of keywords for their brand that they have the capacity to track. Typically, this is determined by the keywords that are known to drive traffic to a site, with the threshold being a significant number of clicks.
What happens when you have a very niche longtail keyword that only generates a nominal amount of clicks? It’s probably not going to move the needle on your bottom line, so it gets left off the list of priority keywords to track. Then you have a few more very niche keywords that generate just another handful of clicks.
For many businesses, this list of “insignificant” keywords can grow into the thousands, where the aggregate of all those nominal clicks suddenly has a lot more power over your business than you originally expected. For example, listen to Wil Reynolds of Seer Interactive recount his experience with discovering that 300,000 previously ignored keywords with ~1 click each accounted for 92% of his client’s traffic.
The problem then becomes the ability to scale the keyword research process to track exponentially more keywords, as well as more SERP data points like various rich results.
When the capacity of an SEO tool is limited, it makes sense to prioritize and limit keyword lists or the search features you track. However, moving to big data where the data is in its rawest form—and able to be pumped into any analytic environment—allows businesses to expand their keyword universe to all keywords, not just a “representative” sample size.
The next step is deciding what this analytic environment will be, and how you will process and manipulate it without a typical SEO platform. With Traject Data, it can go directly into your BI tool. Next we’ll dig into the process the data undergoes so it can be parsed into useful data sets, and ready to be connected to a BI tool that your company most likely already uses to make other data-driven decisions.
How Traject Data can be used
With more SERP data points now available for tracking, use cases for leveraging the data increase as well. Here’s a few things your business can do with Traject Data:
- Inform organic SEO and content creation strategy.
- Track keyword rankings across millions of keywords.
- Identify industry trends and evolving SERPs.
- Blend with other data sources to identify revenue-generating keywords.
- Optimize advertising campaigns based on evolving search engine algorithms and search intent.
- Monitor local performance across geo-location specifications.
- Track Google eCommerce performance.
- Develop a YouTube content strategy.
Take a look at our case study highlighting how Seer Interactive uses Traject Data to help their clients by saving them significant money on ad spend.
What does “raw SEO data” look like?
Raw data from the Traject Data SERP API comes in the form of JSON files that describe any given SERP, including the search features, ranked websites, and anything else that appears for that particular query.
Imagine you query “teriyaki and wok” and you’re shown this SERP. You’ll notice that result #1 is a standard blue link, not a local pack, a video, or any other type of search feature.
The output from the Traject Data SERP API for this query would look like this:
Underneath “SERP” in the JSON file, each ranking position will be labeled by number and list out the attributes of that ranking position. In the “teriyaki and wok” case, you can see the data for the link, the description, and which types of search features do and do not take up position #1 for that query.
If you were to keep scrolling, the same data would be listed for position #2, #3, and so on for that query’s SERP.
This more comprehensive set of SERP data points allows SEOs to increase their control over all the possible rich results at their disposal. Tracking this data will show rich results they’ve won and lost to competitors, search features that are new to a keyword after Google makes an algorithm update, and other data points that will help them hone in on all of their high-return traffic opportunities.
These are some of the most common rich results that we provide data on. You can click through on each to learn more about how you can leverage them.
How is this data converted into something I can manipulate and draw insights from?
The information in the JSON files is pretty extensive to look through for just one SERP. This raw output would be almost impossible to manipulate without building logic, or using a tool that can sort through them to access what you need and create more relevant datasets.
The Traject Data SERP API JSON files are stored in Amazon S3. In S3, the data points from the JSON files are structured and tagged into individual fields so that they are now able to be queried, sorted, and more easily analyzed. After the JSON files have been giving this structure, the Amazon product Athena will then give you the ability to query the database and pull the selection that’s interesting to you.
This staging ground and additional ability to organize the data creates a way for previously unmanageable amounts of data to be pulled into subsets that are relevant for a certain report or analysis.
Leveraging a BI tool
This dataset is now ready to be connected to your BI tool. Once you begin piping in the dataset, you’ll be able to visualize it for easy understanding. From there, you can build reports and combine it with your business’s other data, like sales and marketing data.
BI tools are an important cornerstone to this process because the visualizations allow users to discover new insights with the time, expertise, (or bias), that would typically go into creating pivot tables to investigate a hypothesis.
With the various tools we’ll discuss that are compatible with Traject Data, users can toggle between different types of visualizations much more deftly than building out a new analysis manually each time. Expert level BI users can manipulate the data into more complex visualizations, but beginner level users can still glean a multitude of insights from the visualizations the tool suggests once the data finishes populating.
Traject Data keyword data is able to connect to all of the most common BI tools on the market. To make a decision on which BI tool you’ll use, consider the required skill level, pricing, and tool capabilities below.
Compare and contrast BI tools
What it is
Tableau’s innovation was completely centered around enabling visual analysis.
Tableau connects and extracts the data stored in various places, and is touted as being able to pull data from just about any platform imaginable. It can extract “simple” databases such as excel or pdf files, to more complex databases like Oracle or AWS.
After ensuring thorough data access, Tableau emphasizes its ease of content discovery by allowing users to organize resources by project, recommending relevant data, allowing ‘go back’ with revision history, and enabling search.
Here’s an example of a Tableau dashboard built for Healthcare data. You’ll notice that showing the insights is highly visual, and that analytics can be performed almost instantaneously as users change any setting on the dashboard.
“Explore patient demographics and trends by department in this visualization (viz) by Bridget Cogley. This viz uses Tableau’s clustering feature to uncover insights in hospital data, identifying frequent short-stay patients compared to extended-stay patients.”
Gartner Magic Quadrant Designation: Leader
Plans and Pricing
Tableau divides their product tiers by implementation and user needs. First, your business will need to decide whether you’d like to host Tableau on-premise, in the public cloud, or have Tableau host your server. Then decide how many of each of the following user type you need:
- Creator: This is a data analyst role who will load and standardize data. Every Tableau account is mandated to have at least one Creator
- Explorer: general business user role, can make and edit visualizations
- Viewer: can view and interact with visualizations Explorers and Creators have made
Tableau’s pricing differs between users of on-premise/public cloud and Tableau hosted, and bills annually. For on prem, it costs the following:
Select the number of licenses you need:
- Creator: $70 USD/user/month
- Explorer: $42 USD/user/month
- Viewer: $15 USD/user/month
- Quick and easy representation of big data which helps in resolving the big data issues.
- Can be used with no coding or development knowledge.
- Easy to understand, drag and drop interface.
- You can use pretty much any data source imaginable—Tableau integrates with over 250 applications.
- Tableau has an additional platform for cleansing and prepping data before it’s loaded and analyzed.
- There’s a mobile app with comparable functionality.
- Tableau has a heavy community presence for learning in forums. Tableau Public is the only place Free users can share reports, so it provides a good resource for learning new ways to build reports.
- Some users criticize its collaboration capabilities. The notification functionality is simple, and only an admin, not end-users, can configure scheduled email subscription. Users can use custom Python to create trigger-based notifications, but that practice isn’t baked into Tableau.
- Tableau can be a fairly expensive tool based on your business and needs.
- While Tableau can suit novices with it’s easy to understand interface, to leverage the full capabilities of Tableau, users report a much steeper learning curve that that of other BI tools.
- Tableau operates with a multitude of different products that all handle a different aspect of the data analytics process. Some companies may enjoy the customized experience, but if you’re looking for one tool that gives all users a transferable experience, use a different tool.
The bottom line
Tableau is the top of the food chain for BI tools in terms of quality of analytics and visualization. However, for many users, the cumbersome process of piecing together multiple Tableau products and deploying on prem may be too complex—and too expensive.
What it is
Looker is the data visualization tool owned by Google. It’s main differentiator is its 100% browser-based experience, which eliminates the need for installation and maintenance at the same level as a tool like Tableau. Looker also emphasizes sharability in many of their opening statements, referring to the fact that the browser based experience allows for link sharing, as opposed to sharing with files.
Like most other BI tools, Looker asserts that any novice can use it and that in-depth SQL knowledge can create more intricate reports, but isn’t strictly necessary. However there does seem to be a significant investment in publishing lectures, documentation, and learning puzzles to appeal to Looker users across disciplines and learning styles.
With Looker, data analysts can use a language called LookML to create mini-applications that “add efficiency and power to data exploration.” Looker explains that the LookML syntax simplifies the development of powerful models and enhances the capabilities of SQL. According to them, “this language-based approach leads to faster query execution, and optimizes performance along the way.”
Below is an example of a Looker dashboard from Airbnb that visualizes different data points from their San Francisco listings.
Gartner Magic Quadrant Designation: Challenger
Plans and Pricing
Looker is unique from some other BI tools in that it doesn’t publicly share it’s pricing, reasoning that it’s customizable for small, medium and large businesses. SoftwareConnect estimates the following prices:
- Generally $300-$500/user/month, up to 10 users
- $50/user/month after 10 users
- It emphasizes pre-built templates or blocks to speed up set up time and move more quickly to custom reports.
- The 100% browser-based experience eliminates the need for desktop client software installation and maintenance and allows for link-based sharing of content, which they say makes collaboration “frictionless.”
- All users have a consistent platform experience. For instance, they can complete tasks without shifting from one desktop tool (such as Tableau Desktop), to another (such as Tableau Prep Builder), and then to the web to complete specific tasks.
- Users can leverage the ML code (that most say is fairly easy to learn) to help produce applications and optimize your queries.
- Looker still works when you have no SQL knowledge, as they have a rich database of videos and learning materials, the same as live recordings and screen cast lectures. Documentation also includes interactive puzzles which would please creative teams looking to convert analytics into an enjoyable activity.
- Some users report that the web-based experience caused problems with load time, even calling the wait for a visualization “a run-in with the spinning wheel of death.”
- Users report lessened functionality and viewability when using the mobile app.
- A major negative review of Tableau is price, but if third-party estimates are correct, Looker can actually be the more expensive option in some cases.
- While the out of the box dashboard templates can be incredibly useful to users, some report that using these as a foundation and then attempting to customize them can be very difficult.
- Since it’s completely hosted, there’s no options for fundamentally changing the tool besides waiting for an update to be released.
The bottom line
If you don’t want to install and maintain an on-prem solution, Looker’s hosted platform has data visualization capabilities that rival or exceed other BI tools. You’ll be able to enjoy a report sharing experience that isn’t cumbersome, but watch out for load times and the inability to change the functions of the tool itself.
What it is
Domo’s value proposition of choice is speed and relevancy, with their home page laden with phrases like “real time” “on the fly” “as it happens” and “in record time.” According to them, “With Domo, everyone in your organization, at any level, can access the same data sets and gain trusted insights to help them make faster decisions. And data is never stale; real-time metrics are instantly available on any device.”
“On a typical business day, our customers in the aggregate query between 100 trillion to 200 trillion rows of data. Even with this volume of data, we maintain a sub-second average query response time,” CEO and founder Josh James adds.
Like other top data visualization tools, Domo allows users to combine data from disparate sources, touting their data sources available as “in the thousands” and without needing duplication or pre-transformation to use.
Domo also includes collaboration tools that let teams communicate directly on visualizations with messages and annotations, in addition to scheduled updates and benchmarking alerts that are similar to those of other BI tools.
Similarly to Looker, there isn’t an on-prem deployment option, but Domo touts the cloud-based solution as a baked-in method to help your business scale indefinitely.
Below is an example of a Domo dashboard that uses Traject Data and shows the breakdown of all the different types of search features that appear in the chosen dataset.
Gartner Magic Quadrant Designation: Challenger (2022)
Plans and Pricing
Domo’s editions are primarily divided by number of users, support level, and administrative options. The enterprise edition offers unlimited storage, data sets, personalized data permissions, and advanced admin controls.
They previously offered two plans which, as of 2019, were priced as: Professional ($175/user/month) and Enterprise ($250/user/month). Domo has since removed pricing from their website.
- It’s regarded as one of the fastest operating BI platforms
- Domo provides templates based on business function. This variety of prebuilt pages that automatically assemble based on data input
- Many users enjoy the UI more than that of other BI tools, citing its “vibrance.” If you frequently use reports to woo stakeholders, this might be meaningful
- Allows for team communication and collaboration directly on the platform
- Seamlessly blends data from hundreds of sources. In fact, the amount of data sources available to connect is cited as the reason Domo has been able to gain traction on Tableau in the BI market
- 1-Click Apps are pre-built connectors that allow users to upload data without relying on IT
- As a cloud-based solution, Domo’s implementation tends to be shorter than on-prem solutions like the ones some Tableau users have
- Domo offers a full mobile app
- Domo users report some difficulty in configuring connectors from different sources at once
- There is no on-prem deployment option
- If you plan to create many similar reports, some users complain that you can’t copy a report and then make slight changes to the copy, as it will update the original
The bottom line
If your business favors cloud-based over on-prem and will be using the tool for extensive communication and collaboration while building visualizations, Domo is a great option.
What it is
Power BI is the data visualization tool owned by Microsoft, so you can expect that it will play nicely with other Microsoft products like Azure and Office. In fact, this is one of the first points that the Power BI website confirms.
These other Microsoft products, such as Power Apps, Power Automate, and Power Virtual Agents that create an “end to end” experience will most likely give users a feeling similar to picking and choosing certain Tableau products for a more customized experience, not the singular consistent experience of Looker and Domo.
Power BI connects to most types of on-premise databases, and they have a large and growing list of cloud-based connection options as well.
It also has around 16 different chart types, which is a good, middle-of-the-road amount if you want to represent your data in a variety of formats and visualizations.
Users report that it also works well for the companies that don’t have a data warehouse solution, because Power BI acts as a facilitator that enables data sets processing.
Below is an example of a Sales and Marketing Dashboard created with Power BI.
Gartner Magic Quadrant Designation: Leader
Plans and Pricing
Power BI has 2 plans. The Pro plan is the most inexpensive BI solution out of comparable tools in the market, but the Premium plan can get pricey.
Power BI Pro
- Self-service and modern BI in the cloud
- Collaboration, publishing, sharing, and ad-hoc analysis
- Fully managed by Microsoft
Power BI Premium
- Enterprise BI, big data analytics, cloud and on-premises reporting
- Advanced administration and deployment controls
- Dedicated cloud compute and storage resources
- Allows any user to consume Power BI content
- $20/user/month or $4,995/capacity/month for dedicated cloud compute and storage resource with annual subscription
- Power BI is the most inexpensive tool of its caliber on the market.
- Just like the other tools, Power BI connects to hundreds of data sources, and can read data from Microsoft Excel and text files like XML and JSON.
- It boasts a mobile app with comparable functionality.
- Anyone with moderately advanced Excel skills is likely to have an intuitive experience on Power BI, because users report the functionality as being similar.
- Microsoft releases updates to Power BI monthly, and listens to the user community. If you submit suggestions for improvements, other users can rank the suggestions as well.
- It plays well with Excel and is easy to export.
- Power BI limits the size of datasets that you can pipe in. The limit is 1GB, but if you want to import and analyze larger datasets, you can try creating multiple queries to process the entire data set or shift to Power BI Premium.
- The on-premises reporting is only included in the Power BI Premium package, which can add additional costs.
- Some users report that the design is not intuitive and that the UI/UX isn’t as pleasant as other tools, calling it “clunky.”
- Custom visuals don’t seem to be configurable. If you want to optimize a visual, there are limitations to what can actually be changed.
- There’s no solution for data scrubbing, so your data needs to be cleaned before using Power BI.
The bottom line
If you don’t have a lot of budget and you won’t be using datasets that exceed the size limit, Power BI will give you top-tier analytical capabilities—and may even listen to you on the updates you’d like to see.
Final Thoughts on Choosing a BI Tool for Traject Data
Because Traject Data works with all major BI tools currently on the market, the decision on which you’ll use to analyze Traject Data should be made on the factors above that fit your business best.
Once you’ve chosen a BI tool, you’re prepared to follow these 3 steps to begin visualizing SEO data the additional data points our SERP API provides:
- Pull the JSON files for each keyword you want to track with the Traject Data SERP API.
- Use S3 and Athena to make a database from your JSON files that you can query and filter to create a more relevant dataset.
- Connect your dataset to the BI tool that works best for you and your organization.
Complete these steps, and you’ll have visualizations of endless keyword data to inform your SEO strategy and business decisions as a whole.