Introduction to Terraform

Just imagine you are a customer of a cloud provider and you want to spin up some machines, you can go into some web console, fill in some forms, click some buttons and then launch an instance. But you can also use Terraform.

Terraform allows you to do the same thing but in code, i.e., Infrastructure as Code. It is the automation of your infrastructure and keep your infrastructure in a certain state. For example, you want to spin up 5 small instances and whenever you run Terraform, it will ensure that those run on a cloud platform. When we change something manually, Terraform will try to match the code with the actual infrastructure.

It will also make your infrastructure auditable. Just look at .tf files, we can see what the infrastructure is made of. And even better, we can keep changes in Version Control System, e.g., git.

Download and Install

Terraform can be downloaded and installed in different operational systems, pls check it out on official website.

After downloading and installing Terraform, lets verify it.

Use “terraform -v” in terminal, if the result shows the version, then it can verify if terraform is installed and everything works properly. However, if the result is “Terraform is not a command” or anything like this, there is an issue happen.

The last step in setting is to use an IDE to manage, e.g., Visual Studio.

What Language Terraform supports?

Terraform language is called Hashicorp configuration language in a file that has a .tf extension, so all of our Terraform codes will be stored in a file with .tf extension.

Let’s start!

Now we can create a new project in our own IDE and name it. Then we create a file called main.tf.

At this point, what providers Terraform can support can be checked out on their website.

For example, if we click into AWS Provider, we can see there is an example how to configure the AWS Provider.

terraform {
  required_providers {
    aws = {
      source  = "hashicorp/aws"
      version = "~> 3.0"
    }
  }
}

# Configure the AWS Provider
provider "aws" {
  region = "us-east-1"
}

# Create a VPC
resource "aws_vpc" "example" {
  cidr_block = "10.0.0.0/16"
}

We can copy and paste the part of “Configure the AWS Provider” codes in our main.tf file. And we can check what region we are in AWS and modify it.

Now we get the provider set up and if we go back to Terraform tutorial, it continues to teach us how to set things up. And the next thing to set up the Authentication.

We can take the hard-coding way to have a look: create static creditials. It is not recommended because if we publish the .tf file into Github or something else, the credentials will be stored there, which will cause a security vulnerability problem.

Now using this way just keeps things simper but later we will use a securer way.

From the tutorial, it can be seen that we need three parameter values: region, “access_key” and “secret_key”. Now we can go to AWS console and check those information in Identity and Access Management(IAM) service.

We can create an access key in IAM service. And then we click “show access key” and store those 2 values: “access_key” and “secret_key”.

Then let’s try to create and provision resources within AWS. The syntax is quite simple and same whatever you configure Azure, AWS or GCP in Terraform.

The resources syntax is shown as below:

resource "<provider>_<resource_type>""name"{
    config options....
    key = "value"
    key2= "another value"
}

So this is how to create a resources within a provider. Let us to try to create and deploy a EC2 instance in Terraform.

We need to refer back to the documentation now because we need to put value as “resource_type”:

resource "aws_instance" "web" {
  ami           = data.aws_ami.ubuntu.id
  instance_type = "t3.micro"

  tags = {
    Name = "HelloWorld"
  }
}

We can find such an example and it is barely what the minimum we need. We can copy and paste these codes in our main.tf file.

The “resource_type” in the example is “aws_instance”. We can give it a name and call it what we want to replace “web” in the example. “ami” value can be got if we launch an instance in EC2 service on AWS, then we can put the value into the example. And the “instance_type” is what we select in the AWS console as well.

As in the documentation, the “tags” is optional so we can delete it.

The next thing is go to terminal of your IDE, which is suggested because it will navigate to your project directory automatically. The first Terraform command we need to learn is “Terraform init”. This command will look at the config, which is all .tf files in our project. Because we just have one provider which is AWS, so it will download all necessary plugins to interact with AWS api.

After we run it in terminal, we can see it is initializing the backend and initializing provider plugins and so on. If we add another provider for Azure, then it will also download another plugin for Azure.

After it is successful, the second Terraform command we need to know is “Terraform plan”. Even it is completely optional, but it is a quick sanity check to make sure we won’t break anything. If we have a look at what is happening, it will color code things depending on the action.

  • “+” means creating a resource
  • “-“means deleting a resource
  • “~”means modifying an existing resource

The final command is “Terraform apply” and it will ask you to hit “yes” then it will create our server. Then it will show “Apply complete! Resources 1 added, 0 changed, 0 destroyed”.

Then we can verify it in our AWS console. At this point, we can see the first EC2 instance is deployed through Terraform.

There are more things we can learn in Terraform, maybe talk about it in next blog.

If you are interested in or have any problems with Terraform, feel free to contact me.

Or you can connect with me through my LinkedIn.

CloudFormation 101

What is CloudFormation?

CloudFormation is a tool which can spin up resources on AWS. If someone wants to be an AWS expert, CloudFormation is an essential service to master.

Before we jump into writing a CloudFormation template, let’s have a brief history about how to manage AWS infrastructure before CloudFormation.

Without CloudFormation, automating a process is time-consuming because of building tools to assist with automation, e.g., log in AWS console and manually provision servers.

Maybe it is a fast way to write some scripts to get job done and it will generally work fine at a small scale, however if we need to manage more systems in more environments, it will become tedious.

So that is why configuration management tools exist.

There are some popular tools like Chef, Puppet, Salt etc. They can be used to maintain consistency and track changes. However, their disadvantages are it is not necessatily needed for containerized application deployments.

So there is another way: Infrastructure as Code

Infrastructure as Code is all about having a single souce of truth that serves as a blueprint for what we want the infrastructure look like.

Templates are written using a DSL that describes resources and relationships, e.g., CloudFormation and Terraform.

Their pros:

  • Consistency
  • Auditing
  • Complicance
  • Rollbacks

What languages CloudFormation supports?

The answer is JSON /YAML

Let’s take a look at a JSON template (a template respresents a stack, the components with the stack are known as resources):

{
  "AWSTemplateFormatVersion":"2010-09-09",
  "Description":"Create a S3 bucket",
  "Resources":{
  "datalake":{
    "Type":"AWS::S3::Bucket"
             }
}
}

Inside Resources, uniquely named keys are mapped to specific AWS resources. In our example, this CloudFormation template is used to create a S3 bucket in AWS.

Functions:

CloudFormation also supports some functions, and the most useful is Ref, it is used to pass value.

Next blog we will continue our journey with AWS.

If you are interested in or have any problems with CloudFormation, feel free to contact me .

Or you can connect with me through my LinkedIn.

Medium-Hard Data Analyst SQL questions(Part 1)

As we know, the first 70% of SQL is pretty straightforward but the remaining 30% can be pretty tricky.

So, in this blog, some popular hard SQL interview questions will be covered for people to sharpen their skills.

Self-Join Practice Problems

Part 1: How much a key metric, e.g., monthly active users, changes between months, e.g., a table named ‘logins’ is shown as below.

userz-iddate
12020-07-01
2342020-07-02
32020-07-02
12020-07-02
2342020-10-04
table:logins

Q: find the month-over-month percentage change for monthly active users

Solutions:

WITH mau AS
(
  SELECT
  DATE_TRUNC('month', date) AS month_timestamp,
  COUNT(DISTINCT user_id) AS mau
  FROM logins
  GROUP BY
  DATE_TRUNC('month', date)
)

SELECT
a.month_timestamp AS previous_month,
a.mau AS previous_mau,
b.month_timestamp AS current_month,
b.mau AS current_mau,
ROUND(100.0*(b.mau - a.mau)/a.mau,2) AS percent_change
FROM mau a
JOIN mau b
  ON a.month_timestamp = b.month_timestamp - interval '1 month'

Next blog we will continue our journey with SQL medium-hard questions.

If you are interested in or have any problems with SQL, feel free to contact me .

Or you can connect with me through my LinkedIn.

Send Google Analytics payload length as a Custom Dimension

The maximum length of a Google Analytics payload is 8192 bytes. It is useful to check if you are approaching this value with some of your hits because if the payload length exceeds this, the hit is never sent to GA.

How can we know the payload size with each hit?

Today i will show you how to send the payload size as a custom dimension to GA with each hit. The tool is Google Tag Manager.

Before starting, creating a new hit-scoped custom dimension in GA is essential, named ‘Hit Payload Length’ and check its index.

Then, create a custom task in GTM, the custom task is to let users modify the request sent to GA before it is sent. We can take Client ID as an example.

Custom task will work with a custom javascript variable. Here is the javacript code which we create in GTM.

function () {
  // clientIdIndex: The Custom Dimension index to where you want to send the visitor's Client ID, my example is 7.
  var clientIdIndex = 7;

  // payloadLengthIndex: The Custom Dimension index to where you want to send the length of the payload of the request,  my example is 18.

  var payloadLengthIndex = 18;

  // DO NOT EDIT ANYTHING BELOW THIS LINE
  var readFromStorage = function (key) {
    if (!window.Storage) {
      // From: https://stackoverflow.com/a/15724300/2367037
      var value = '; ' + document.cookie;
      var parts = value.split('; ' + key + '=');
      if (parts.length === 2) {
        return parts.pop().split(';').shift();
      }
    } else {
      return window.localStorage.getItem(key);
    }
  };

  var writeToStorage = function (key, value, expireDays) {
    if (!window.Storage) {
      var expiresDate = new Date();
      expiresDate.setDate(expiresDate.getDate() + expireDays);
      document.cookie = key + '=' + value + ';expires=' + expiresDate.toUTCString();
    } else {
      window.localStorage.setItem(key, value);
    }
  };

  var globalSendHitTaskName   = '_ga_originalSendHitTask';

  return function (customTaskModel) {

    window[globalSendHitTaskName] = window[globalSendHitTaskName] || customTaskModel.get('sendHitTask');

    // clientIdIndex
    if (typeof clientIdIndex === 'number') {
      customTaskModel.set('dimension' + clientIdIndex, customTaskModel.get('clientId'));
    }
    // /clientIdIndex

    customTaskModel.set('sendHitTask', function (sendHitTaskModel) {

      var originalSendHitTaskModel = sendHitTaskModel,
          originalSendHitTask      = window[globalSendHitTaskName],
          canSendHit               = true;

      try {

        // payloadLengthIndex
        if (typeof payloadLengthIndex === 'number') {
          var _pl_hitPayload = sendHitTaskModel.get('hitPayload');
          _pl_hitPayload += '&cd' + payloadLengthIndex + '=';
          _pl_hitPayload += (_pl_hitPayload.length + _pl_hitPayload.length.toString().length);
          sendHitTaskModel.set('hitPayload', _pl_hitPayload, true);
        }
        // /payloadLengthIndex

        if (canSendHit) {
          originalSendHitTask(sendHitTaskModel);
        }

      } catch(err) {
        originalSendHitTask(originalSendHitTaskModel);
      }

    });

  };
}

The last step is the add the custom task into your tags, we can scroll down to a tag and add a new field, whose value will be the custom task we just created.

After this step, any tags which has this custom task will add the hit payload length as a custom dimension.

How to debug your work?

Verify it through the developer tools on your browser.

  • Open the Network on the developer tool on your browser
  • Click the request to collect
  • Check if there is a payload parameter

With this custom dimension, we can monitor if the payload maximum size will be hit.

If you are interested in or have any problems with GTM, feel free to contact me.

Or you can connect with me through my LinkedIn.

Why we need data integration and what can we do?

The first question is why we need data integration?

Let me give you an example here to answer this question.

Every company has many departments, and different departments use different tools to store their data. For example, marketing team may use hubspot tool.

Now, we have different departments which store different types of data in a company.

However, insightful information is needed to make business decisions through those large amount of data.

What can we do?

Maybe we can connect all the databases everytime to generate reports. However, it will cost us large amount of time, then the term of data integration is raised.

What is data integration?

Data integration is a process in which heterogeneous data is retrieved and combined as an incorporated form and structure.

There are some advantages of data integration:

  1. Reduce data complexity
  2. Data integrity
  3. Easy data collaboration
  4. Smarter business decisions

There are also some well-known tools can do data integration, e.g., Microsoft SQL Server Integration Services (SSIS), Informatica, Oracle Data Integrator, AWS Glue etc.

This blog we will talk about SSIS, which is one of the most popular ETL tool in New Zealand.

Why SSIS?

  1. Data can be loaded in parallel to many varied destinations
  2. SSIS removes the need of hard core programmers
  3. Tight integration with other products of Microsoft
  4. SSIS is cheaper than most other ETL tools

SSIS stands for SQL Server Integration Services, which is a component of the Microsoft SQL Server database software that can be used to perform a broad range of data integration and data transformation tasks.

What can SSIS do?

It combines the data residing in different sources and provides users with an unified view of these data.

Also, it can also be used to automate maintenance of SQL Server databases and update to multidimensional analytical data.

How is works?

These tasks of data transformation and workflow creation is carried out using SSIS package:

Operational Data–>ETL–>Data Warehouse

In the first place, operational data store (ODS) is a database designed to integrate data from multiple sources for additional operations on the data. This is the place where most of the data used in current operation is housed before it’s tranferred to the data warehouse for longer tem storage or achiiving.

Next step is ETL(Extract, Transform and Load), the process of extracting the data from various sources, tranforming this data to meet your requirement and then loading into a target data warehouse.

The third step is data warehousing, which is a large set of data accumulated which is used for assembling and managing data from various sources of answering business questions. Hence, helps in making decisions.

What is SSIS package?

It is an object that implements integration services functionality to extract, transform and load data. A package is composed of:

  1. connection
  2. control flow elements(handle workflow)
  3. data flow elements(handle data transform)

If you want to investigate more about SSIS, check it out on Microsoft official documents.

If you are interested in or have any problems with SSIS, feel free to contact me.

Or you can connect with me through my LinkedIn.

Some Cloud Computing Fundamentals

What is cloud computing?

The practice of using a network of remote servers hosted on the Internet to store, manage, and process data, rather than a local server or a personal computer.

On-premise:

  1. You own the servers
  2. You hire the IT people
  3. You pay or rent the real-estate
  4. You take all the risk

Cloud providers:

  1. Someone else owns the servers
  2. Someone else hires the IT people
  3. Someone else pays or rents the real-estate
  4. You are responsible for your figuring cloud services and code, someone else takes care of the rest.

Different kinds of hosting:

  1. Dedicated Server: One physical machine dedicated to single a business. Runs a single web-app/site. (Very expensive, high maintenance, high security)
  2. Virtual Private Server: One physical machine dedicated to a single business. The physical machine is virtualized into sub-machines runs multiple web-apps/sites.
  3. Shared Hosting: One physical machine, shared by hundreds of businesses. Relies on most tenants under-utilizing their resources. (Very cheap, Very limited)
  4. Cloud Hosting: Multiple physical machines that act as one system. The system is abstracted into multiple cloud services. (Flexible, Scalable, Secure, Cost-Effective, High Configurability)

Common Cloud Services

A cloud provider can have hundreds of cloud services that are grouped various types of services. The four most common types of cloud services for infrastructure as a service(laaS) would be:

  1. Compute: Imagine having a virtual computer that can run application, programs and code.
  2. Storage: Imagine having a virtual hard-drive that can store files
  3. Networking: Image having a virtual network being able to define Internet connections or network isolations
  4. Database: Imagine a virtual database for stoing reporting data or a database for genetal purpose web-application

The term ‘cloud computing’ can be used to refer to all categories, even though it has ‘compute’ in the name.

Benefits of Cloud Computing

Cost-effective: You pay for what you consume, no up-front cost. Pay-as-you-go(PAYG) thousands of customers sharing the cost of the resources.

Global: Launch workloads anywhere in the world, just choose a region

Secure: Cloud provider takes care of physical security. Cloud services can by secure by default or you have the ability to configure access down to granular level.

Reliable: Data backup, disaster recovery, and data replication, and fault tolerance.

Scalable: Increase or decrease resources and services based on demand

Elastic: Automate scaling during spikes and drop in demand

Current: The underlying hardware and managed software is patched, upgraded and replaced by the cloud provider without interruption to you.

Next blog I will write some fundamentals about Microsoft Azure, which is the cloud provider service of Microsoft.

If you are interested in or have any problems with cloud computing, feel free to contact me.

Or you can connect with me through my LinkedIn.

How to design a Data Warehouse(Part 1)

Last blog I wrote why we need a Data Warehouse.

First, what is the data warehouse?

It is a centralized relational database that pulls together data from different sources (CRM, marketing stack, etc.) for better business insights.

It stores current and historical data are used for reporting and analysis.

However, here is the problem:

How we can design a Data Warehouse?

1 Define Business Requirements

Because it touches all areas of a company, all departments need to be onboard with the design. Each department needs to understand what the benefits of data warehouse and what results they can expect from it.

What objectives we can focus on:

  1. Determine the scope of the whole project
  2. Find out what data is useful for analysis and where our dat is current siloed
  3. Create a backup plan in case of failure
  4. Security: monitoring, etc.

2 Choose a data warehouse platform

There are four types of data warehouse platforms:

  1. Traditional database management systems: Row-based relational platforms, e.g., Microsoft SQL Server
  2. Specialized Analytics DBMS: Columnar data stores designed specifically for managing and running analytics, e.g., Teradata
  3. Out-of-box data warehouse appliances: the combination with a software and hardware with a DBMS pre-installed, e.g., Oracle Exadata
  4. Cloud-hosted data warehouse tools

We can choose the suitable one for the company according to budget, employees and infrastructure.

We can choose between cloud or on-premise?

Cloud solution pros:

  1. Scalability: easy, cost-effective, simple and flexible to scale with cloud services
  2. Low entry cost: no servers, hardware and operational cost
  3. Connectivity: easy to connect to other cloud services
  4. Security: cloud providers supply security patches and protocols to keep customers safe

Choices:

  1. Amazon Redshift
  2. Microsoft Azure SQL Data Warehouse
  3. Google Bigquery
  4. Snowflake Computing

On-premise solution pros:

  1. Reliability: With good staff and exceptional hardware, on-premise solutions can be highly available and reliable
  2. Security: Organizations have full control of the security and access

Choices:

  1. Oracle Database
  2. Microsoft SQL Server
  3. MySQL
  4. IBM DB2
  5. PostgreSQL

What we can choose between on-premise and cloud solution, in the big picture, it depends on our budget and existing system.

If we look for control, then we can choose on-premise solution. Conversely, if we look for scalability, we can choose a cloud service.

3 Set up the physical environments

There are three physical environments in Data Warehouse: development, testing and production.

  1. We need to test changes before they move into the production environment
  2. Running tests against data typically uses extreme data sets or random sets of data from the production environment.
  3. Data integrity is much easier to track and issues are easier to contain if we have three environments running.

4 Data Modelling

It is the most complex phase of Data Warehouse design. It is the process of visualizing data distribution in the warehouse.

  1. Visualize the relationships between data
  2. Set standardized naming conventions
  3. Create relationships between data sets
  4. Establish compliance and security processes

There are bunches of data modeling techniques that businesses use for data warehouse design. Here are top 3 popular ones:

  1. Snowflake Schema
  2. Star Schema
  3. Galaxy Schema

4 Choosing the ETL solution

ETL, stands for Extract, Transform and Load is the process we pull data from the storage solutions to warehouse.

We need to build an easy, replicable and consistent data pipeline because a poor ETL process can break the entire data warehouse.

Wrapping up

This post explored the first 4 steps about designing a Data Warehouse in the company. In the future, I will write the next steps.

If you are interested in or have any problems with Data Warehouse, feel free to contact me.

Or you can connect with me through my LinkedIn.

Tutorial: Using BigQuery to Analyze CrUX Data

Last blog I gave some examples of how we can use the Chrome User Experience report (CrUX) to gain some insights about site speed. This blog I will continue to show you how to use bigquery to compare your site with the competitors.

Prerequisite:

  1.  Log into Google Cloud,
  2. Create a project for the CrUX work
  3. Avigate to BigQuery console
  4. Add the chrome-ux-report dataset and explore the way the tables are structured in ‘preview’

Step one: Figure out what is the origin of your site and the competitor site

like syntax is preferred (Take care of the syntax difference between Standard SQL and T-SQL)

 -- created by: Jacqui Wu
  -- data source: Chrome-ux-report(202003)
  -- last update: 12/05/2020
  
SELECT
  DISTINCT origin
FROM
  `chrome-ux-report.all.202003`
WHERE
  origin LIKE '%yoursite'

Step two: Figure out what should be queried in the select clause?

What we can query from CrUX?

The specific elements that Google is sharing are:

  1. “Origin”, which consists of the protocol and hostname, as we used in step one, which can make sure the URL link
  2. Effective Connection Type (4G, 3G, etc), which can be queried as the network
  3. Form Factor (desktop, mobile, tablet), which can be queried as the device
  4. Percentile Histogram data for First Paint, First Contentful Paint, DOM Content Loaded and onLoad (these are all nested, so if we want to query them, we need to unnest them)

Here I create a SQL query of FCP percentage in different sites, which measures the time from navigation to the time when the browser renders the first bit of content from the DOM.

This is an important milestone for users because it provides feedback that the page is actually loading.

SQL queries: 

  -- created by: Jacqui Wu
  -- data source: Chrome-ux-report(202003) in diffrent sites
  -- last update: 12/05/2020
  -- Comparing fcp metric in Different Sites

SELECT origin, form_factor.name AS device, effective_connection_type.name AS conn, "first contentful paint" AS metric, bin.start/1000 AS bin, SUM(bin.density) AS volume
FROM(  
SELECT origin, form_factor, effective_connection_type, first_contentful_paint.histogram.bin as bins
FROM `chrome-ux-report.all.202003`
WHERE origin IN ("your site URL link", "competitor A site URL link", "competitor B site URL link")
)
CROSSS JOIN UNNEST(bins) AS bin
GROUP BY origin, device, conn, bin

Step 3: Export the results to the Data Studio(Google visualization tool)

Here are some tips may be useful

  1. Line chart is preferred for comparing different sites in Visual Selection
  2. Set x-axis to bin(which we already calculate it to seconds) and y-axis to percentage of fcp
  3. Set filter(origin, device, conn) in Filtering section

Wrapping up

This post explored the data pipeline we can use CrUX report to analyze the site performance. In the future, I will write more about CrUX.

If you are interested in or have any problems with CrUX or Business Intelligence, feel free to contact me.

Or you can connect with me through my LinkedIn.

How to use CrUX to analyze your site?

What is CrUX?

CrUX stands for the Chrome User Experience Report. It provides real world and real user metrics gathered from the millions of Google Chrome users who load millions of websites (include yours) each month. Of course, they all opt-in to syncing their browsing history and have usage statistic reporting enabled.

According to Google, its goal is ‘capture the full range of external factors that shape and contribute to the final user experience’.

In this post, I will walk you through how to use it to get insights of your site’s performance.

Why we need CrUX?

We all know faster site results in a better user experience and a better customer loyalty, compared to the sites of competitors. It results in the revenue increasing. Google confirmed some details about how they understand the speed. They are available in CrUX.

What are CrUX metrics?

  1. FP(First Paint): when everything loads on the page
  2. FCP(First Content loaded): when some text or an image loaded
  3. DCL(DOM content loaded): when DOM is loaded
  4. ONLOAD: when any additional scripts have loaded
  5. FID(First Input Delay): the time between when a user interacts with your site to when the server actually responds to that

How to generate the CrUX report on PageSpeed Insights?

PageSpeed Insights is a tool for people to understand what a page’s performance is and how to improve it.

It uses the lighthouse to audit the given page and identify opportunities to improve performance. It also integrates with the CrUX to show how real users experience performance on the page.

Take Yahoo as the example, after a few seconds, lighthouse audits will be performed and we will see sections for field and lab data.

In the field data section, we can see FCP and FID (please see the table below as we can see the FCP and FID values).

MetricFastAverageSlow
FCP0-1000ms1000ms-2500ms2500ms+
FID0-50ms50-250ms250ms+

We can see the Yahoo site is in ‘average’ according to the table. To achieve the ‘fast’, both FCP and FID must be categorized as fast.

Also, a percentile can be shown in each metric. For FCP, the 75th percentile is used and for FID, it is the 95th. For example, 75% of FCP experiences on the page are 1.5s or less.

How to use it in BigQuery?

In BigQuery, we can also extract insights about UX on our site.

SELECT origin, form_factor.name AS device, effective_connection_type.name  AS conn, 
       ROUND(SUM(onload.density),4) as density
FROM `chrome-ux-report.all.201907`,
    UNNEST (onload.histogram.bin) as onload
WHERE origin IN ("https://www.yahoo.com")
GROUP BY origin, device, conn

Then we can see the result in BigQuery.

The raw data is organized like a histogram, with bins have a start time, end time and density value. For example, we can query for the percent of ‘fast’ FCP experiences, where ‘fast’ is defined as happening under a second.

We can compare Yahoo with bing. Here is how the query look:

SELECT
  origin,
  SUM(fcp.density) AS fast_fcp
FROM
  `chrome-ux-report.all.201907`,
  UNNEST (first_contentful_paint.histogram.bin) AS fcp
WHERE
  fcp.start<1000
  AND origin IN ('https://www.bing.com',
    'https://www.yahoo.com')
GROUP BY
  origin

Wrapping up

This post explored some methods to get site insights with CrUX report. In the future, I will write more about CrUX.

If you are interested in or have any problems with CrUX or Business Intelligence, feel free to contact me.

Or you can connect with me through my LinkedIn.

Google Analytics cheat sheets

Data-driven VS data-informed

Data-driven: Decisions made only based upon statistics, which can be misleading.

Data-informed: Decisions made by combining statistics with insights and our knowledge of human wants & needs. 

We will be able to use data and human creativity to come up with innovative solutions in a business.

When we click into Google Analytics, we can see a large amount of lines, full of data, strange names.

But don’t chill out, we can break things

In this blog, I will illustrate Analytics +Art = Creative Data Scientist

Agenda

  1. Installing And Customising Google Analytics
  2. Learning Dashboard
  3. Analysing Behaviours
  4. User Acquisition
  5. Generate And Share Reports

1. Installing And Customising Google Analytics

How to Setup Google Analytics & Install on Website - YouTube

When we install the Google Analytics, there are several terms we should pay attention:

  1. Tracking code: Which is a basic code snippet for a website. It starts as UA, which stands for Universal Analytics.
  2. Data collection:  Turn it on and it allows us to get data of users.
  3. User-ID:  Allow us to tracker users. Generate the unique user Id and make sure the right ID is assigned to the right user and associate the data in Google Analytics.
  4. Session setting:  Any time a user has loaded up your site on their device. We can set the session time here.
  5. Referral exclusive list: Mostly, we set them as our own site URL.

2. Learning Dashboards:

Admin page: account, property, view

Under the account, there are several properties.

Take one account as the example, there are many URLs associated with this account.

In other words, we have a site. Under this site, we have a whole bunch of other properties that we’ve associated here.

The tracking Id is the same, except for a number at the very end.

Google Analytics Admin Page

Google Ads and Google AdSense: 

Ads are the words we buy from Google. They are links or text that appear on top of Google search pages. AdSense are the ads google sells that we can insert on our own site. We use AdSense if we are a publisher and we want to monetise our content.

Set the ‘Bot Filtering’ in the view setting under the view page of Admin.

It excludes all hits from known bots and spiders. Google, Yahoo and etc. They have programs to analyze and index the content. Bots is short for robots. Spiders are because these little programs crawl, web, spiders.

It is not a must way to have it turn on, but we have a big site, for a professional who wants to do the analytics, we should gain insights about what the humans (content is for humans) excluding the robots. If we are interested in what users are doing on our site, maybe we can do this easy way to turn it on.

Because the robots just crawl the data from our sites.

Sidebar

Then we can have a look at the sidebar.

  1. Customisation:  Suggest to install a custom report to give it a try.
  2. Real-time: What users are doing the right very second.
  3. Audience:  We will see who, what, when, where, from where. A big module.
  4. Acquisition:  Where traffic comes from, how marketing efforts working. We can have a look at channel, we will see direct, paid search, organic search, (other), referral, email, social. And this tells us again, how people find me? How is my marketing working? Are we just wasting our money?
  5. Behaviour: This is kind of fascinating, think of this as being the security camera in the store. We are watching our uses picking up items or checking out or running out of the exits.
  6. Conversiond: It is the happy part. It is where we track and figure out how well our sites are turning our visitors into customers.

3. Analysing Audience Behaviour

(1) Conversion vs Engagement:

Conversion: A one-time interaction. Granted, this is a powerful interaction, but it is the end goal of a chain of events.

Engagement: Repeated use, that results in an emotional, psychological and sometimes near-physical tie that users have to products, e.g., apple fans.

Build a hypothesis via the audience overview

There are a lot of opportunities to grow if we were to take this site to have it available in other languages.

(2) Active users

From the line in active users, we can see whether nothing is effective on the traffic. Or there is really no marketing being done.

(3) Cohort analysis

Cohort is a group of users that all share a common characteristic, in this case, the acquisition date is the day they came to your site which here is known as day 0. Metrics here are used to analyse the user behaviour.

We can see how is going on day 1?

We can see how many people came back the next day. 

  1. Track individual users with user explorer
  2. How to use segmentation to refine demographics and interests

It helps us to know who our audience is and what type of contents we are trying to expose them—>impact the design choices.

(4) Demographics

If it is young people who use smartphones mostly, we should simplify the navigation choices.

(5) Interests

Target-rich environment for the site can be a place which is the combination of top 3 interests

we can create a segment ‘tablet traffic’ to give it a try and we can find whether there are some differences between the all users.

(6) Geo

Language & Location. We can set the segment like ‘converters’ to compare with the ‘all users’ to find some differences.

(7) Behaviours:

We can set the ‘mobile and tablet traffic’ as the segment to compare with ‘all users’. We find after how many seconds, people are paying more attention. The numbers are trending up. Whether we got their attention for a long span of time. 

(8) Technology

Browser & OS:  Flash version is if we want to do ads on the website, we need to make sure that they actually display.

Network: This can be a really big deal if we are working in users and areas where we know they have very slow connections. And do we need to simplify a new page for them? This is called adaptive design.

(9) Mobile

If something is strange but not significant, we can just move on.

  1. Benchmarking and users flow
  2. Page Analytics (a plugin we can find in Chrome store)

4. User Acquisition

(1) Learning about channels, sources and mediums

There are many questions here:

Well, how do they get the site?

Sources, searched and referrals

SEO and what users are looking for

Social statistics and …

Channels:  The general, top-level categories that our traffic is coming from, such as search, referral or social

Sources: A subcategory of a channel. For example, search is a channel. Inside that channel, Yahoo Search is a source.

Mediums:  By which the traffic from a source is coming to our site. That is, if the traffic is coming from Google, is it organic search or paid search?  

(2) Differentiating between channels-organic search and direct

(direct): direct traffic is where someone comes directly to our site, i.e., type the address into the browser bar or they click on the bookmark.

‘not provided’: the data comes through Google is now encrypted to keep governments or hackers or spies from getting value from it. 

(3) Unlocking Mysterious Dark Social Traffic

There are 6 ways that the dark social traffic can come to the site.

  1. Email, messages. The traffic is from someone’s email program. This is not tracked by Google Analytics because GA lives in browsers.
  2. Links in docs: it is in an application that is not tracked by Google Analytics
  3. Bit.ly, Ow.ly, etc.
  4. Mobile social: twitter etc. 
  5. From https to http
  6. Search errors

(4) Drilling down to track who goes where

From source/mediums: trigger email

(5) Spotting the ‘Ghost Spam’ in referrals

Ghost spam: 

It isn’t really hurting anything. In fact if we really are organized and we are the only one looking at the reports. We can leave them alone and nothing happens. 

There are noxious visits to my site made with the nefarious intent of getting us to click on the links and visit the site of the spammer. They are not actual visits. These sessions and pageviews are from bots that either hit our site and execute the Google Analytics scripts or bypass the server and hit the Google Analytics directly.

Firstly, we need to find out the ghost referrals: 

They came from host names that are not our site.

Check it through:  Acquisition–>All traffic–>Referrals.

We can see there is some websites that have 100% bounce rate and 0s average session duration.

We can also see some things e.g., xxxxxx.com / referral in Acquisition–>All traffic–>Sources/Medium

How to remove the ghost spam and fake traffic from Google Analytics?

Next blog we will talk more about it.

If you are interested in or have any problems with Business Intelligence or Google Analytics, feel free to contact me.

Or you can connect with me through my LinkedIn.