cloud

Upload Data To BigQuery With A Smile

Big QueryWhat is BigQuery?

In our world of ‘Big Data’ it can be time consuming and expensive to query massive datasets without the right infrastructure. Google BigQuery solves this problem by enabling super-fast, SQL-like queries against append-only tables, using the processing power of Google’s infrastructure.

What You need to do?

  1. Move your data into BigQuery – This is what we will do in this post.
  2. Let Google BigQuery handle the hard work.
  3. Query your big data with a smile in this cost/effective way.

How to upload data to Big Query?

There are two main approaches: stream you data or upload it directly from Google cloud storage. Let’s have a look at the steps to leverage Google cloud storage in order to upload data into BigQuery.

The main steps you need to follow:

  1. You will need to prepare your data. In this stage, you need to analyze and think what will be the best format (both JSON and CSV are supported).
  2. In our example, we will show you how to work with CSV files and even better, we will upload them to Google Cloud Storage and later with a BigQuery job we will make sure our data is being pulled automatically into BigQuery.
  3. Run a ‘sanity’ check to see that the new data is in good shape (optional step).

Tips

  • Upload your the data to a project with a good name (The default project names are not too clear in most cases).
  • Consider breaking your data (e.g monthly tables instead of a unique big one) because it will make life easier in the future to update, query and maintain the data source.
  • Have an example dataset with data that reflect the popular cases. This could be great to give developer an option to ‘play’ with the data and see its value.
  • Think on some good and bold example. A few sample queries are crucial to get people started on a dataset.

Continue reading

Standard
Chrome, HTML5, JavaScript, mobile, webdev

Israel’s Alerts – Mobile Web App Example

Alerts IL - mobile web app

If you wish to have a window on the side of your screen and/or leave your mobile device on a page that will show you an updated list of all the alerts (Tzeva Adom and others) – Here is a web app for you. It is based on Foundation CSS framework as it’s ‘mobile first ‘ and got a very powerful grid system.

Code

Check out the GitHub repo at https://github.com/greenido/AlertsIL Continue reading

Standard
Chrome, JavaScript, webdev

(Try To) Predict The World Cup With Monte Carlo Simulation

brazil 2014 world cupWho will win this world cup? Brazil without Neymar? Lionel Messi? Or are we going to see the Nederland making a big surprise?

If you like coding, statistics and problems that are not trivial… You found the right place. In this post, I will try to show an example to a way for solving this type of questions. For the readers that don’t remember what is Monte Carlo simulation (don’t be shy) – You might want to check a previous post that I wrote last summer and give you an intro to the world of Monte Carlo on Apps script and Google compute engine.

Monte Carlo experiments (simulations) are a broad class of computational algorithms that rely on repeated random sampling to obtain numerical results. In most cases, we will run our simulations many times over in order to obtain the distribution of an unknown probabilistic entity.  This tool is often used in physical and mathematical problems and are most useful when it is difficult or impossible to obtain a closed-form expression, or infeasible to apply a deterministic algorithm.
In the real world, we see it being mainly used in three distinct problem classes:
* Optimization.
* Numerical integration.
* Generation of draws from a probability distribution.

 

OK, there is no time. The game starts in two days.

The Idea

Monte Carlo simulations tend to follow a particular pattern. Here is how we will use this tool: Continue reading

Standard
Chrome, HTML5, JavaScript, mobile, webdev

Polymer And Web Components At I/O 2014

This talk cover three main subjects:

  • State of the union – What has change in the past 3 years
  • Problems solved by web components – When it comes to build complex web applications, there are many ways we can do things better. In this section we will see how/why.
  • Thinking in components – What are the implications when you coming to design and build web apps.

== Psst… It’s in raw format – I will polish it after I/O is over ==

State of the union

Once many years ago… We had HTML4 with all its problems. Later, well few years back, HTML5 was the answer for everything. It specifically, tried to show developers what can be done on the web modern platform and moreover APIs browser vendors could add to help developers boost their productivity. In the past 18 months we saw a lot of things like: templates, shadow dom, data binding etc’. It’s been a long journey. Now, we have Polymer. Continue reading

Standard
cloud

Google I/O 2014 – What’s interesting in the cloud?

google-io-logo 2014-06-24 14.30.42This year there are many interesting sessions on Google Cloud. Here are some that I think you might enjoy watching the live stream (or jump into the room if you are at Moscone center). There are many more great sessions, so the best advice is to check the schedule and plan in advance which one you wish to follow live. You can always catch the other sessions later on developers.google.com channel. Continue reading

Standard
Chrome, HTML5, JavaScript, mobile, webdev

HTML5 In Mobile (Hebrew)

The goal of this talk is to harness front-end developers with relevant knowledge and tools they can use  in their current jobs.

The main points in the TL;DR

Continue reading

Standard
JavaScript, webdev

2014 World Cup Matches And Results

brazil 2014 world cupThe world cup is the biggest sporting event on the planet. During the last world cup in 2010, 3.2 billion people watched at least one of the games. As I know, many groups are going to ‘make it interesting’ by placing bets on the results, I thought it would be good to have all the data on a spreadsheet. Google sheets are ideal for that as they offer a great ability to share, comment and ‘play’ with the data. You can see the demo spreadsheet: http://goo.gl/ZjdfqE and fork the code from: https://github.com/greenido/WorldCup

For now, we are fetching the info from these end points: Continue reading

Standard