Why Machine Learning is So Popular and What is Learning

The oldest and strongest emotion of mankind is fear, and the oldest and the strongest kind of fear is the fear of unknown.

In order to solve this fear, human can only explore the relationship between input and output from the limited examples.

Then they can predict a situation that has never been seen before.

This is what learning is.

Learning: Exploring the pattern between input and output from limited examples.

So, the dataset is so important. Not only it needs to be large but also it is supposed to be precise.

This is why children have always been curious because they want to get all the examples to explore the world.

But until high school, the learning method we are exposed to is actually Transfer Learning.

It is the precedure that transfering the knowledge of the other people (e.g., teachers, parents) to your own brain.

But in this process, we have made too many mistakes.

Because Transfer Learning is killing the desire of human nature to learn.

At the same time, many people think this is what learning looks like.

They think Machine Learning is to put some patterns into the machine by programming.

In fact, machine learning is to let the machine learn to discover some patterns.

And the pattern found is called knowledge.

Actually, it is why Deep Learning is so important in Machine Learning and human learning methods.

Next blog we will explore what deep learning is and why human learning should act like deep learning.

If you are interested in or have any problems with Machine Learning, feel free to contact me .

Or you can connect with me through my LinkedIn.

Stored Procedure in SQL

This article I will give you some basic syntax about stored procedure in SQL.

Stored Procedure:

Stored Procedure is the prepared SQL codes that we can save, so the code can be reused over and over again.

It is suitable for some SQL queries we need to use frequently.

There are three kinds of stored procedure:

  1. No parameter
  2. One parameter
  3. multiple parameters

Stored Procedure (No parameter) Syntax

CREATE PROCEDURE procedure_name
AS
sql_statement
GO
EXEC procedure_name

Stored Procedure (One parameter) Syntax

CREATE PROCEDURE [dbo].[oneparameter]
@ProductCatergoryID int
AS
SELECT *FROM Production.ProductCategoryID
GO
EXEC oneparameter @ProductCatergoryID = '4'

Stored Procedure (multiparameter) Syntax

CREATE PROCEDURE [dbo].[multiparameter]
@ProductCategoryID int, @Name varchar(50)
AS
SELECT *FROM Production.ProductCategory pc
WHERE pc.ProductCategoryID = @ProductCategoryID and pc.Name = @Name
GO
EXEC multiparameter @ProductCategoryID = '4', @Name = 'Accessories'

I will record all knowledge I touch in my Business Intelligence journey.

Next blog will be around Data Warehouse.

If you are interested in or have any problems with Business Intelligence, feel free to contact me .

Or you can connect with me through my LinkedIn.

Business Intelligence Tutorial

This blog is Business Intelligence tutorial, which contains lots of definitions and terms.

There are three steps in Business Intelligence: Data collecting, Data Warehousing and Reporting.

Data Collecting:

  1. Structured: Standardized and easy for computers to read and query.
  2. Semistructured
  3. Unstructured: Not stored in rows and columns, so it can’t easily read by computers.

As we talked in previous blog, company data can be found in several locations, such as CRM programs, which is also shown in the picture below.

Data Warehouse:

Data warehouse uses a process (ETL, i.e., extract, transform and load) to standardize data, which allows it can be queried.

How does information get to a central location?

ETL———————>Data Warehouse

  1. Extract: unstructutred data is tagged with metadata to make it easier to find
  2. Transform: normalize data
  3. Load: tranfer data to central warehouse or data mart

Turning Data into Powerpoints (Business Intelligence Reporting)

  1. Data visualization: Graphic display of results
  2. Dashboard: Interfaces that represent specific analyses

If you are interested in or have any problems with Business Intelligence, feel free to contact me .

Or you can connect with me through my LinkedIn.

Why Communication is So Important In IT Industry?

Many people have an impression on IT developers: Coding all the time.

Actually, it is a stereotype. In IT Industry, developers spend more time on communication rather than computers.

So, if you want to become an IT professional, you must be good at communication.

Even you more enjoy coding.

Think about how much time you will spend on interacting with others?

  1. Check the email box…
  2. Agile Scrum Meeting…
  3. Communicate with colleagues about the needs…

How we can improve our communication?

1 Don’t judge or criticize

People are good at putting themselves in the centre. Everyone think himself/herself is the most important.

It is wrong and we can’t belittle others.

We should respect them firstly and then others will accept our opinion.

2 Think what others need

The key in communication is not think our own interests, like what I need and what I want.

Instead, we should consider what others need and what they want.

Why they want this function?…

Why this part makes them feel bored?…

3 Avoid fighting

As IT developers, sometimes we are easy to think others will accept the logical thinking.

Actually, most people are sensitive and follow their feelings.

From a book (How to Win Friends and Influence People) written by Dale Carnegie, the only way to solve the fighting is to avoid it.

If you never think about how to communicate with others, this is the right time!

Improve the ability of soft skills, and you will find it is valuable for your life.

People will like you and value you back.

From my BI journey, I learn more about soft skills, English at work and tech chit chats.If you are interested in or have any problems with BI, feel free to contact me .

Or you can connect with me through my LinkedIn.

Some Cheat Sheets about Business Intelligence

There are several parts of my Business Intelligence journey.

So I write this blog for recording some cheat sheets.

Installation:

Firstly, some tools need to be installed on the laptop.

All of them are Microsoft softwares, so I prepare a surface laptop 2.

  1. SQL Server 2016 Developer Edition: Seclect the features we need to install, such as SQL Server Integration Services(SSIS), SQL Server Analysis Services(SSAS) and SQL Server Reporting Services(SSRS).
  2. SQL Server Management Studio(SSMS)
  3. SQL Server Data Tools(SSDT)
  4. Report Builder

What is Business Intelligence(BI)?

I have mentioned it in the previous blog. It is to transform raw data into meaningful information for analysis.

BI Process:

  1. Import data from different data sources
  2. Produce reports or graghs through SSIS, SSAS and SSRS
  3. Make business decisions

Some terms:

ETL: Extraction, Transformation and Loading

OLTP: Online Transaction Processing

Cube: Multi-dimensional data structure built using dimensions and facts, using MDX (Multi dimensional expression) language

SQL and Relational Database: which I have mentioned them in the previous blog

Primary Key: A single attribute or an unique id in a Relational Database

Foreign Key: A set of attributes that references a candidate key in a Relational Database

Some differences:

Traditional Normalized Database VS Data Warehouse: Formal one has no duplicate data but latter one has duplicate data for efficiency.

Null VS Blank(where clause): For null, the clause is where xxxxx is null; For blank, the clause is where xxxxx is ”

Delete VS truncate: truncate table xxxxx ->no rollback&faster; delete from xxxxx -> rollback

SQL Language:

Learning sources:

  1. w3schools.com
  2. SQL Server tutorial for beginner

Some common commands:

One table:

SELECT…FROM

WHERE CLAUSE

ORDER BY CLAUSE

Two tables:

Inner Join

Other Joins, e.g., LEFT OUTER JOIN

GROUP BY

If you are interested in or have any problems with Business Intellgence, feel free to contact me .

Or you can connect with me through my LinkedIn.

What is the Difference between Business Analysis, Data Analysis and Data Science?

The terms and job titles in the data field are extraordinarily large, such as business analysis, data analysis, data science, etc.

They often stun everyone, so this article we will talk about all of them, especially the differences.

Business Analysis VS Data Analysis

Generally, data analysis means that using data to analyse a XXX problem. The pipeline includes data collecting, data warehousing, data cleaning, data visualisation, etc.

As we can see, there is a blank in the sentence.

Actually, data analysis can be applied to different areas, and that is also the meaning of ‘XXX’ .

It can be academic or commercial.

If ‘XXX’ is commercial, then data analysis will equal to business analysis. In other words, the business analysis is an application of data analysis.

There is a main difference between data analysis and business analysis, which is the data sources.

The data source of data analysis positions is often based on the company websites, Apps, ERP system, etc.

And those job descriptions will require candidates master SQL, Python or R.

However, the data collected from those platforms often exists several problems, mainly the data quality, such as some missing data or data noise.

Those are caused by weak IT infrastructures and ‘econnoisseurs’ who want to register more user accounts.

However, the data sources of business analysis positions not just include those internal data collected by companies but also include a large amount of external sources. For example:

  1. Industry studies
  2. Qualitative interviews
  3. Quantitative interviews
  4. Internal data

Business Analysis VS Data Science

Because of Alpha Go, Artificial Intelligence is well-known to everyone.

However, the successful area of AI is not related to data analysis.

They focus on the computer vision and natural language processing, and their industrial applications are mainly in security and supply chain.

However, the usage of algorithms in commercial industry is limited because some departments in the company are hard to be represented by data and algorithm models.

This causes that algorithm can only solve the specific problems.

  1. Firstly, algorithm is related to users directly. The famous example is risk control. Because the factors contributing to the user credits are easy to be built by an algorithm model, commonly logistic regression.
  2. Secondly, forecasting algorithms. Business Intelligence is highly required to do forecasting.
  3. Thirdly, dimensionality reduction algorithms. It is often to reduce dimensions so it is easy to evaluate a problem, such as new products.

Summary

In summary, algorithms are useful in business intelligence, however, it cannot replace it.

Business Intelligence is an application in commercial problems and algorithm models are tools to solve specific problems.

If you are interested in or have any problems with Business Intellgence, feel free to contact me .

Or you can connect with me through my LinkedIn.

The Simplest Way to Install SQL Server 2017 on macOS

This article we will show how to install Microsoft SQL Server 2017 on macOS, as we talk it in previous blog.

Which method we use?

Prior to SQL Server 2017, if we want to install it on macOS, a virtual machine (like Parallels Desktop) is essential. Then we can install a Windows system in it and then we install and run SQL Server.

Luckily, from SQL Server 2017, we can choose to install SQL Server on Docker containers.

How to install SQL Server on Docker?

1 What is Docker?

Docker is a platform which can make softwares to run in it. It is called a container, which is an isolated environment.

2 Download and install Docker

If you haven’t installed Docker on your Mac, the next step is to install Docker.

Go to Docker page to download the .dmg file and then double click and install it according to the instructions.

3 Run Docker and increase the memory

Run Docker as you used to run other softwares and the next step is to increase the memory. This is because the default value of Docker memory is 2 GB but SQL needs at least 3.25 GB.

I recommend we set the memory value to 4 GB.

  1. Click Docker icon on top menu of your Mac
  2. Click Preferences
  3. Set Memory under Advanced to 4 GB.

Click Apply&Restart

4 Download Microsoft SQL Server 2017

The next step is to download SQL Server 2017 from Terminal, which is an easy way.

Copy and paste the command in Terminal of your Mac:

docker pull microsoft/mssql-server-linux

Through it, the latest version of MS Server SQL can be downloaded.

5 Run a Docker image

Copy, change and paste the command in Terminal of your Mac:

docker run -d --name xxxxxxx -e 'ACCEPT_EULA=Y' -e 'SA_PASSWORD=xxxx\xxxx' -p 1433:1433 microsoft/mssql-server-linux

We need to change the name and password as your own here.

This step is to run an instance of Docker image. A Docker image is a file, which is used to execute codes in a Docker container.

If you want to check, copy and paste the following command to see whether the Docker container is running:

docker ps

If it works, it will show like this:

6 Install the sql-cli command line tool

The next step is to install the sql-cli command line tool. It can allow you to run commands against your SQL Server instance.

Copy and paste the command in Terminal of your Mac:

npm install -g sql-cli

If an error happened and shows you do not have the permissions to access this file as the current user, just try add sudo in your command:

sudo npm install -g sql-cli

7 Connect to SQL Server

After sql-cli is installed, now we can connect to SQL Server using the mssql command.

mssql -u xxxxxxx -p xxxxxxx

Here xxxxxxx and xxxxxxx means your name and password.

Then you will see:


Now you’ve successfully connected to your instance of SQL Server.

Next blog we will continue our journey with SQL Server and Azure Data Studio. Maybe you will want to read more about Azure and Microsoft Learn in the previous blog.

If you are interested in or have any problems with Business Intelligence, feel free to contact me .

Or you can connect with me through my LinkedIn.

Welcome to SQL (SQL 101)

If you want to find a BI Analyst job in New Zealand, you may be not a master of Python, R, etc.

However, SQL knowledge is essential.

Here is a BI Analyst job advertisement on LinkedIn, which is a full-time role.

St John is also an accredited employer.

If you have a skill that is needed by a New Zealand accredited employer and they offer you full-time work when you are abroed, you may be able to get a Talent (Accredited Employer) Work Visa.

From the job description, we can find SQL knowledge is important.

What is SQL?

Firstly, I want to talk about Database(DB). We all know DB is a kind of software to store a large amount of data, mainly Relational DB.

Structured Query Language(SQL) is always linked with DB. SQL can be used to operate data, such as querying and updating.

The relationship between DB, data and SQL is like: DB is a plate, data is dishes on the plate and SQL is your fork.

Currently, most of the websites and Apps are based on SQL and DB.

There are several popular DB on the world, e.g., SQLite, MySQL, Postgres, Oracle and Microsoft SQL Server.

Among them, MSSQL Server is the most popular in New Zealand.

Luckily, all of them can support SQL although they have different characteristics.

It is like if you have the fork, you can operate dishes on different plates.

What is Relational DB?

A DB is composed of several tables (like tables in Excel) and tables are composed of rows and columns.

We can query and obtain some results from the DB through SQL language.

Maybe some people will be confused about: What is the difference between DB and Excel? Excel has already existed on the world, why SQL is created?

That is because DB = Tables + Relations between tables.

Tables in Excel can’t meet the data operating requirements in companies because of the complex relations between tables.

How to kick-start SQL?

After knowing the terms of SQL and Relational DB, the next step is to set up the environment with SQL Server, which we will talk in the next blog.

If you are interested in or have any problems with Business Intelligence, also feel free to contact me .

Or you can connect with me through my LinkedIn.

How Business Intelligence is Making the World a Better Place

What is Business Intelligence(BI)?

In general, BI refers to that companies slice and dice data to look for patterns. Then they make business decisions by the data insights.

When I ask other people this question: Do you know what is Business Intelligence?

I usually got the answer “No”.

However, I found most of people know what is Big Data. Of course, this is a buzzword.

Even those people who are not familiar with this area, they can tell me Big Data means the data is big…

This is a good thing. Even sometimes our understanding for terms is not precise.

It is common because a glossary consists of many knowledge systems.

It is hard to explain it in one sentence.

For example, Business Intelligence is a professional area, including ETL, Data Warehousing, etc. You can’t explain all of them to others.

Simply, BI is a pipeline which can transform data into information and then it creates business value.

How can we create business value using BI?

Reporting, which is common in New Zealand.

Reporting means using data visualisation techniques to visualise company’s business data.

It mainly focuses on finance, supply chain, marketing and human source departments.

For example, sales and marketing department cares about sales amount, order quantity, return rate, etc.

Then BI can be used to filter and extract analysis results. Tables and figures differ according to their subjects and they relate to each other.

Analysis about “abnormal” data.

Through data visualisation results, the abnormal data means some data indicators are beyond our predictions.

Sometimes it is positive that we are pursuing.

For example, average user registration amount is 1000 per month.

However, we find one month the number reaches 3000.

It is a kind of positive abnormality. Of course, sometimes the abnormal data can be negative.

We can analyse which factor contributes to the abnormality through one or more dimensions data tables and figures.

Those results can help to make better business plans.

Conclusions:

BI is actually a business and management area.

It comes from the presentation of visual analysis reports. Then it tries to find business problems (positive or negative) to improve business operations.

BI jobs in New Zealand:

According to Business Analyst on careernz website, BI analyst is in high demand and its salary is $81K-$104K per year.

Many companies hire BI analysts in New Zealand, including:

  1. Government departments or tertiary universities/institutes
  2. Finance companies
  3. Marketing companies
  4. Professional firms such as law and accounting

Next blog I will talk about which techniques are popular in New Zealand and how to learn them.

If you are interested in or have any problems with Business Intelligence, feel free to contact me.

Or you can connect with me through my LinkedIn.

Why Do People Think Enter X Challenge Innovation Competition is a Good Idea?

Except some IT competitions, I also enter X Challenge Innovation Competition in university.

X Challenge is a competition that gives every AUT student the opportunity and guidance to develop an idea for a business, cause or project. 

Each student can experience how to launch and run a business, which starts from an innovative idea. 

It consists of two stages: The Idea and The Accelerator.

You can enter either one or both, as an individual or as a team. When entering as a team at least one member must be a current AUT student.

I enter The Idea.

Next is my milestones for X Challenge:

01/04/2019: Gather my team.

Choosing a partner who matches the marketing and business ability that I lack will shape the future of the business more than any other.

In this stage, I choose Mia Lu.

She is a marketing and multi-media specialist who holds a double major in Marketing and Economics, with 4+ years of experience in digital marketing and communications. Except the marketing professional ability, design is also her passion and speciality, i.e., graphic design and UX design.

More important, she is an amazing women with high execution and communication skills.

07/04/2019: Decide what we want to build. Brainstorming something the world needs and all the members agreed on the idea.

Finally, our idea is Named Entity Recognition as a Service.

Here is the rationale of our idea. This tool can prevent privacy information leakage for individual Online Social Networks users.

It mainly involves in Deep Learning and Natural Language Processing techniques.

17/04/2019: Demonstrate that people want your product and figure out how to scale. We celebrate once 300 people who are not our friends and family tell us that using our product was useful.

25/04/2019: Submit the Google form to AUT. The form consists of one-page idea, including the problem we solve, our product advantages, markets and customers, direct or indirect competitors and team member description.

01/05/2019: Our team is chosen as a winner of The Idea.

My feelings:

Through the innovation competition, I realise how to change a technical idea to a business product.

And I realise how to act as a leader, especially collaborating with other team member.

Team support is the key to success!

If you are interested in or have any problems with business plan or competitions, feel free to contact me .

Or you can connect with me through my LinkedIn.