cloud

Upload Data To BigQuery With A Smile

Big QueryWhat is BigQuery?

In our world of ‘Big Data’ it can be time consuming and expensive to query massive datasets without the right infrastructure. Google BigQuery solves this problem by enabling super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure.

What You need to do?

  1. Move your data into BigQuery – This is what we will do in this post.
  2. Let Google BigQuery handle the hard work.
  3. Query your big data with a smile in this cost/effective way.

How to upload data to Big Query?

There are two main approaches: stream you data or upload it directly from Google cloud storage. Let’s have a look at the steps to leverage Google cloud storage in order to upload data into BigQuery.

The main steps you need to follow:

  1. You will need to prepare your data. In this stage, you need to analyze and think what will be the best format (both JSON and CSV are supported).
  2. In our example, we will show you how to work with CSV files and even better, we will upload them to Google Cloud Storage and later with a BigQuery job we will make sure our data is being pulled automatically into BigQuery.
  3. Run a ‘sanity’ check to see that the new data is in good shape (optional step).

Tips

  • Upload your the data to a project with a good name (The default project names are not too clear in most cases).
  • Consider breaking your data (e.g monthly tables instead of a unique big one) because it will make life easier in the future to update, query and maintain the data source.
  • Have an example dataset with data that reflect the popular cases. This could be great to give developer an option to ‘play’ with the data and see its value.
  • Think on some good and bold example. A few sample queries are crucial to get people started on a dataset.

Continue reading

Advertisement
Standard
cloud, webdev

What’s New On Google Cloud Platform

This year at #DevConTLV the main theme was around cloud and databases. In my talk I did my best to emphasis, that in the past 15 years, Google has been building out the world’s fastest, most powerful, highest quality cloud infrastructure on the planet and it is opening it so external developers could enjoy it. There are many services like: App Engine, BigQuery and VMs on Compute engine that all come with the same idea. You (=the developer) should focus on what you good at and not by reinventing the wheel again (and again) by trying to find the secret in memcache optimizations. It’s my 4th year in this event and I can say that it is (without doubt) one of the best developer conferences in Tel Aviv. So if you are around next time, please try to join us. It’s great fun to talk with so many talented developers and the talks that I’ve been to, where very good and informative. Continue reading

Standard
cloud

Big Query Basics

Big QueryIn today talk, I’ve covered one of the coolest tools in the “big data” world – Big Query. What is BigQuery? Well, querying massive datasets can be time consuming and expensive without the right hardware and infrastructure. BigQuery solves this problem by enabling super-fast, SQL-like queries, using the processing power of Google’s infrastructure. If you wish to play with a cool a BQ demo, just click on the picture on the left.

When to use it?

There are few bold use cases in the analytic landscape that Big Query is the right tool for the job:

  • It’s a great option when your MapReduce based analysis (=Hadoop system) can be slow for ad-hoc queries.
  • If you don’t wish to manage large data centers and tuning Hadoop all day long.
  • It seems that analytics tools should be services at 2014, no? Continue reading
Standard
cloud, JavaScript

Big Query And Google Spreadsheet Integration

big queryThere are many options to extend the powerful spreadsheets that Google offers. One of the cool, new ways to leverage its power is by using a spreadsheet as your ‘front-end’ to a big data processing power (=Big Query). In our world, there is a need to get results as fast as possible and since our data sources grow fast. It’s nice to have a tool that let us ‘see’ (and share) results quickly and easily.

What is BigQuery?

Google BigQuery is a web service that lets you do interactive analysis of massive datasets. When we saying massive we are talking here on billions of rows (or more). It is a scalable and easy to use tool that gives developers and businesses an easy way into powerful data analytics on demand.

As for Google Docs and their powerful sharing capabilities – I guess we don’t need to elaborate here. So, let’s see what are the steps that will let us get data from BigQuery into our Google spreadsheet.

Integrate BigQuery To Google Spreadsheet Continue reading

Standard
Chrome, HTML5, JavaScript, webdev

Google Cloud And Mobile Web

Screen Shot 2013-09-26 at 9.41.45 AM

Two talks in one day…

That’s what I did yesterday for Google Developer Group (GDG) Athens, Greece. It was a great opportunity to talk about the new cool aspects of google cloud platform (Yep, checkout things like: NodeJS on Compute Engine, App engine new support for technologies, Monte Carlo Simulations with App Script, Cloud storage, Big Query and many more). We talked for the first 25min on the new aspects of developing mobile web sites (and/or mobile web apps). In one word – go check out yeoman.io. Of course, there are many more aspects that are in the slides, so feel free to browse them and please let me know if I’ve missed something important that you are using in your mobile web project. Continue reading

Standard
Chrome, HTML5, JavaScript

Chrome & Google Cloud Quick Update (GDL-IL)

Google APIsSome of the topic we touch during the show today:

Standard
HTML5, JavaScript, webdev

Big Query Power With JavaScript

Big Query and App script logoThis week on Google developers live Israel we wanted to show the power of Big Query. What is Big Query? Well, in todays world when everyone like to use the term “big data” you need to have the capabilities to querying massive datasets. This can be time consuming and expensive without the right knowledge, hardware and infrastructure. Google BigQuery solves this problem by enabling super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure. In order to get started quickly and ‘test the water’ there is a powerful online tool that let you query pre-existing datasets like: wikipedia, Github etc’. If you like to type in command line, there is also a command line tool. Before you start your first project you should signup for BigQuery (yes! it’s open now for all). You should log in to the Google APIs Console and make sure you set a new project and allow Big Query API on it. You should also, enable billing if you have not done so in the past. Lastly, head to bigquery.cloud.google.com and click on one of the public datasets that are on the left sidebar. Continue reading

Standard