From the Big Data keynote at InCSIghts 2012

Editor's Notes

1. LONDON — Britain often conjures images of unpredictable weather, with downpours sometimes followed by sunshine within the same hour — several times a day. Such randomness has prompted Tesco, the country’s largest grocery chain, to create its own weather team in hopes of better forecasting temperatures and how consumer demand changes with them. After three years of research, the six-person team has created its own software that calculates how shopping patterns change “for every degree of temperature and every hour of sunshine,” Tesco said last month. “Rapidly changing weather can be a real challenge,” Jonathan Church, a Tesco spokesman, said in a statement. “The system successfully predicted temperature drops during July that led to a major increase in demand for soup, winter vegetables and cold-weather puddings.”
2. Big data: Embracing the elephant in the room(Case study – GlaxoSmithKline)By Steve Hemsley GlaxoSmithKline’s (GSK) CRM consultant James Parker believes the benefits of big data must be sold into every department within an organisation, rather than being seen purely as a marketing function.GSK, which owns brands from Sensodyne to Lucozade, is already tracking consumers online and repurposing the data to benefit particular brands. “Consumers are providing signals of what they are doing, seeing, thinking and sharing at a point in time and this is valuable to help drive relationships,” he says.The company is working with St Ives Group’s data management division Occam, which says GSK is aiming to build direct relationships with 1 million consumers in a year using social media as a base for research and multichannel marketing. Targeted offers and promotions will drive people to particular brand websites where external data is integrated with information already held by the marketing teams.“We identify people who are mentioning a particular brand of ours but also track everything else they are talking about in the public domain to build a customer profile,” says Parker.
3. Poor Use CasesOLTP. Outside VoltDB, complex multi-object transactions are generally not supported. Programmers are supposed to denormalize, use documents, or use other complex strategies like compensating transactions.Data integrity. Most of the NoSQL systems rely on applications to enforce data integrity where SQL uses a declarative approach. Relational databases are still the winner for data integrity.Data independence. Data outlasts applications. In NoSQL applications drive everything about the data. One argument for the relational model is as a repository of facts that can last for the entire lifetime of the enterprise, far past the expected life-time of any individual application.SQL. If you require SQL then very few NoSQL system will provide a SQL interface, but more systems are starting to provide SQLish interfaces.Ad-hoc queries. If you need to answer real-time questions about your data that you can’t predict in advance, relational databases are generally still the winner. Complex relationships. Some NoSQL systems support relationships, but a relational database is still the winner at relating.Maturity and stability. Relational databases still have the edge here. People are familiar with how they work, what they can do, and have confidence in their reliability. There are also more programmers and toolsets available for relational databases. So when in doubt, this is the road that will be traveled.
4. Bigness. Ability to support large volumes of data at a reasonable price on commodity hardware. Massive write performance. This is probably the canonical usage based on Google's influence. High volume. Facebook needs to store 135 billion messages a month. Twitter, for example, has the problem of storing 7 TB/data per day with the At 80 MB/s it takes a day to store 7TB so writes need to be distributed over a cluster.Fast key-value access. Hashing on a key in main memory is the fastest way to access a value.Flexible schema and flexible datatypes. NoSQL products support a whole range of new data types column-oriented, graph, advanced data structures, document-oriented, and key-value. Complex objects can be easily stored without a lot of mapping. Developers love avoiding complex schemas and ORM frameworks. Lack of structure allows for much more flexibility. We also have program and programmer friendly compatible datatypes likes JSON. Schema migration. Schemalessness makes it easier to deal with schema migrations without so much worrying. Schemas are in a sense dynamic, because they are imposed by the application at run-time, so different parts of an application can have a different view of the schema.Write availability. Do your writes need to succeed no mater what? Then we can get into partitioning, CAP, eventual consistency and all that jazz.Easier maintainability, administration and operations. This is very product specific, but many NoSQL vendors are trying to gain adoption by making it easy for developers to adopt them. They are spending a lot of effort on ease of use, minimal administration, and automated operations. This can lead to lower operations costs as special code doesn't have to be written to scale a system that was never intended to be used that way.No single point of failure. Not every product is delivering on this, but we are seeing a definite convergence on relatively easy to configure and manage high availability with automatic load balancing and cluster sizing. A perfect cloud partner.Generally available parallel computing. We are seeing MapReduce baked into products, which makes parallel computing something that will be a normal part of development in the future.Programmer ease of use. Accessing your data should be easy. While the relational model is intuitive for end users, like accountants, it's not very intuitive for developers. Programmers grok keys, values, JSON, Javascript stored procedures, HTTP, and so on. NoSQL is for programmers. This is a developer led coup. The response to a database problem can't always be to hire a really knowledgeable DBA, get your schema right, denormalize a little, etc., programmers would prefer a system that they can make work for themselves. It shouldn't be so hard to make a product perform. Money is part of the issue. If it costs a lot to scale a product then won't you go with the cheaper product, that you control, that's easier to use, and that's easier to scale?Use the right data model for the right problem. Different data models are used to solve different problems. Much effort has been put into, for example, wedging graph operations into a relational model, but it doesn't work. Isn't it better to solve a graph problem in a graph database? We are now seeing a general strategy of trying find the best fit between a problem and solution.
5. Microsoft: We have built the SQLServer to Hadoop connector that has been production, we are also doing testing of MongoDB on Azure.Salesforce Radian6: Done some salesforce integration work for them, I think.EMC: We have done some work on Greenplum but that was before buyout – we are trying to start an engagement with them. They are a partner though on content side (documentum)Hbase: Several projects have used Hbase – Email analytics, American AirlinesMongo DB: Perf Testing on Azure, Data store for social media analytics platform -  Content Savvy,  multi-media data store for QyukiPIG: Use it heavily on Yahoo, SciDB: Doing QA and code contributions, we will be using it on Vo as wellAsterData: Done some BI work for them.
6. Persistent has intellectual assets from the very core of the stack up through the solution specific layersBig Data ConsultingRoadmap for deploying Big Data in EnterpriseWorkshop for use case discovery.Sample Proof of Concepts.Hadoop ProgrammingDeveloping custom applications to fit in MR paradigm exposed by HadoopPorting existing application to utilize MR on HadoopTuning existing MR jobs to improve performanceBig Data Application DevelopmentKnowledge of tools based on the Hadoop platform - Nutch, Hive, Pig, JAQLExperienced in building applications using these toolsWriting connectors for loading data on HDFS
7. PeBAL – Peal is Persistent’sbig dataanalytics library. It helps quicker implementation of map reduce applications. Organized as JAQL functions, PeBAL implements several graph, set, text extraction, indexing and correlation algorithms. PeBAL functions are schema agnostic. All PeBAL functions are tried and tested against well defined use cases. Lot of common problemsMany of them solved – but not in Map ReduceDon’t reinvent the wheel. Use readymade solution which isWell writtenWell testedWell used in multiple applicationsWell (good) performanceWould PigLatin, Hive, JAQL solve this for you?No All of them are programming languages and not librariesThe language features are very tied to SQL like interfaceWe need something on top of them
8. A large US airlines, typically wanted to identify influencers for targeting marketing programs as well user segmentation
9. [Too Tedious to Define and Answer]Could be extended to ticketing systems, call center recordings, live communitiesAreas that this could helpCustomer serviceTrainingProduct Management
10. [Data Available but not Warehoused]75 thousand people die in the US because of correctable flaws in Health care systemData is in unstructured format, volume is huge, analysis has to be done quickly – solution built using IBM BigInsights
11. [Don’t believe it’s possible]Intended Users: Marketing Team, Product DevelopmentRequires: Persistent’sDatasift to BigInsights Connector, Persistent BigData Analytics Library (Pebal), Persistent Text Analytics Framework, IBM BigInsightsCompatible with: Twitter, Facebook,Data Volumes: In TerabytesAlso do work on Cognos Consumer Insight:A social media monitoring and analytics platform that allows several use cases like – brand reputation management, campaign tracking, use intent tracking to be implemented on data available on social media as well as Internet sources like blogs, news, message forrumsetc.TechnologyUsed: Hadoop (IBM Big Insights), JQAL, Text Analytics using System TDomain: Digital Marketing, Social Media Business Need for using Big Data:  Cost effective for large datasets, Ability to handle unstructured/semi-structured data.