Tag Archives: M2M

Five Sources of Big Data

Some time ago I’ve described how to think when you build solutions from Big Data in the post Six Graphs of Big Data. Today I am going to look in the opposite direction, where Big Data come from? I see distinctive five sources of the data: Transactional, Crowdsourced, Social, Search and Machine. All the details are below.

Transactional Data

This is old good data, most familiar and usual for the geeks and managers. It’s plenty of RBDMSes, running or archived, on-premise and in the cloud. The majority of transactional data belong to corporations because the data was authored/created mainly by businesses. It was a golden era of Oracle and SQL Server (and some others). At some point the RDBMS technology appeared to be incapable of handling more transactional data, thus we got Teradata (and others) to fix the problem. But there was no significant shift for the way we work with those data sources. Data warehouses and analytic cubes are trending, but they were used for years already. Financial systems/modules of the enterprise architectures will continue to rely on transactional data solutions from Oracle or IBM.

Crowdsourced Data

This data source has emerged from the activity rather than from a type of technology. The phenomenon of Wikipedia confirmed that crowdsourcing really works. Much time passed since Wikipedia adoption by the masses… We got other fine data sources built by the crowds, for example, OpenStreetMaps, Flickr, Picasa, Instagram.

Interesting things happen with the rise of personal genetic testing (verifying DNA for millions of known markers via 23andme). This leads to public crowdsourced databases. More samples available, e.g. amateur astronomy. Volunteers do author useful data. The size of the crowdsourced data is increasing.

What differentiates it from transactional/enterprise data? It’s a price. Usually, crowdsourced data is free for use, with one of creative commons licenses. Often, the motivation for the creation of such data set is the digitization of our world or making a free alternative to paid content. With the rise of nanofactories, we will see the growth of 3D models of every physical product. By using crowdsourced models we will print the goods at home (or elsewhere).

Social Data

With the rise of Friendster–>MySpace–>Facebook and then others (Linkedin, Twitter, etc.) we got a new type of data — Social. It should not be mixed for Crowdsourced data, because of completely different nature of it. The social data is the digitization of ourselves as persons and our behavior. Social data is very well complementing the Crowdsourced data. Eventually, there will be digital representation of everyone… So far social profiles are good enough for meaningful use. Social data is dynamic, it is possible to analyze it in real-time. E.g. put Tweets or Facebook posts thru the Google Predictive API to grab emotions. I’m sure everybody intuitively understands this type of data source.

Search Data

This is my favorite. Not obvious for many of you, while really strong data source. Just recall how much do you search on Amazon or eBay? How do you search on Wikis (not messing up with Wikipedia)? Quora gets plenty of search requests. StackOverflow is a good source of search data within Information Technology. There are intranet searches within Confluence and SharePoint. If those search logs are analyzed properly, then it is clear about potential usefulness and business application. E.g. Intention Graph and Interest Graph are related to the search data.

There is a problem with “walled gardens” for search data… This problem is big, bigger than for social data, because public profiles are fully or partially available, while searches are kept behind the walls.

Machine Data

This is also my favorite. In the Internet of Things every physical thing will be connected. New things are designed to be connectable. Old things are got connected via M2M. Consumers adopted wearable technology. I’ve posted about it earlier. Go to Wearable Technology and Wearable Technology, Part II.

The cost of data gathering is decreasing. The cost of wireless data transfer is decreasing. The bandwidth of wireless transfer is increasing dramatically. Fraunhofer and KIT completed 100Gbps transmission. It’s fourteen times faster than the most robust 802.11ac. The moral is — measure everything, just gather data until it becomes Big Data, then analyze it properly and operate proactively. Machine data is probably the most important data source for Big Data during the next years. We will digitize the world and ourselves via devices. Open Street Map got competitors, the fleet of eBees described Matterhorn with million of spatial points. More to expect from machines.

Tagged , , , , , , , , , , , , , , , , , , , , ,

Wearable Technology

Right time to wrap a short story about the modern technology. Today I will talk about what will be massively adopted in three years from now. It is Wearable Technology. Everything is very simple – the current smartphones will not last long, because their design sucks. The “brick design” is not human friendly. The bigger smartphones are even worse than smaller ones. It is a result of immature technology to produce better hardware. But things have started to change.

The Momentum

The documented research for wearables for consumer market is dated back to the end of 1990x, which constitutes approx 15-17 years. Usually it took about 25 years for new technology to win us over. But taking into consideration the exponential acceleration of our technology progress, the time frame is shrinking. The adoption has began and you will see what will happen in three years. Just to make a relevant analogy, recall what would you know about the tablets before April 2010? There was nothing sustainable on the market. Old tablets from Microsoft don’t count because of poor sustainability… And look at your hands right now, look around, many people hold iPads, and they use them for both work and life needs. That happened in 3 years. The hardware changes in 3 years. Google Glass is exciting, they tease us today, but everybody will be with Google and some competitors glasses in just few years.

Software + Services?

Now, take few minutes to read the spec of Google Glass. It is Android. It is a platform for the apps. Because web browser is too heavy for such limited environment. Sorry SaaS, you will have to wait until the mini ear of apps will come to the logical end. But today it is a perfect time for the apps, embedded or Android or similar. Take another make, Suunto. They converted wearable accessories into the app platforms. You can download the app for your favorite Ambit from the app zone. Let’s move further within quantified-self trend. Next stop is a health. There are nice wearable products for nutrition, fitness and sleep monitoring. From very sexy fashionable Jawbone to more classical looking Fitbit to sporty Nike Fuel to modest calories burn tracker BodyMedia. Those wearable buddies simply don’t have neither room nor battery capacity for web browser (read for fat HTML and JavaScript processing). Of course they synchronize with smartphones, then with backend. It is new model: Software + Software + Services, where first software is embedded, second is mobile app and services are in the cloud.

Could wearable gadgets work without the smartphone? Yes and no. They all work in offline mode, silently gathering/measuring you. Then you connect via Bluetooth to smartphone or with USB wire to laptop and sync the data with the cloud services. Theoretically the wearable gadget can bypass the phone and work directly with the services. It is called Internet of Things, when each “thing” is connected to the Internet, or vice verse, the Internet is constituted from the connected “things”. The more things connected, the better real world is duplicated/reflected into the information system. The goal is to replicate the physical World into digital.

If we take different class of devices, which connects other machines to the internet, we have to mention M2M. It is very simple concept, to connect smth disconnected, you have to put additional machine, aware of the first machine on one side and Internet on another side. Those M2M devices are usually working over HTTP with backend and via USB or COM (with native protocol) with original disconnected machine. Hence, there is a significant piece of software on M2M man-in-the-middle. There could be high-level logic running there, e.g. predictive or prescriptive analytics, which can work even in offline mode, if the accumulated or sync’ed data is sufficient that that. Software + Services is what we need to understand to design Internet of Things for machines and for people. Wearable technology will run software next three years for sure.


The “brick design” is bad, but what is better? Any other anatomic designs are way better. Wait a bit and there will be new stuff built out of Graphene. It has ideal properties to be an excellent component of chips and circuits. There are billion euro grants for this decade to bring Graphene into our lives. Old good Nokia got 1.35b euro to develop Graphene. May be they will make the Morph concept a reality!

Who worries about the chips? They are almost transparent today. The first transparent integrated circuit was built in far 2006. Today, achieving of 80% of transparency with chips is not a problem. Having Graphene as a basis, guarantee that modern devices will be easily bendable. All those tablets still lose to paper because they are not bendable. We almost achieved the paper resolution (dpi) on tablets, but still struggle with every-day usability.

Quantified Self

Everybody is going to be measured. If you want to measure yourself, just use wearable technology and track the rest of your activity with software products. Just put smart wristband, health monitor, GPS or whatever onto yourself, your bike and forget about it. The data will be continuously recorded. If you don’t like to track yourself, you still cannot avoid it, because big stores will track you via cell phone triangulation. They use fake signaling to make your phone revealing itself, thus identifying your location and movement inside the store. You are tracked via surveillance cameras too… but it is not wearable, hence out of the scope of this topic.

Don’t worry about the wearable guys, they will produce tons of data about you. While using tools to record other meaningful activities might be painful. Here is a sample how the man experienced it with Google Calendar. I foresee that non-wearable tracking will be resolved via image/object recognition. Retail industry needs it for descriptive, predictive and prescriptive analytics very much.

Internet of Things

Those things that could not be designed to work with cloud directly, will be classified as smart accessories to some smarter device – smartphone. At the beginning there will be smartphone as man-in-the-middle for almost all wearable devices. Like AliveCor ECG for iPhone. But then, those devices will be able to bypass the phone. In other words, we are using smartphone as M2M intermediary, to connect wearable machine to the Internet. We got Internet of Things right away, just not as big as it could be, and with heavy use of the phone for M2M. In the near future we will get even bigger Internet of Things without the phones, as wearable gadgets will be designed as connected right out of the box.

Ideally, there will be no need for visible computer (either laptop or smartphone or tablet) in common situations. The number of computers/machines will be bigger, and the size will be smaller. And many of them will be Wearable and connected.

Tagged , , , , , , , , ,