Splunk App for Stream for Enhanced Operational Intelligence from Wire Data

Editor's Notes

1. Please skip the first section “Intro to wire data’ if the customer is familiar with the wire data collection. Typically this section may not needed to be explained for network or security teams.
2. If your customer is network engineer or admin or in network security and is familiar with wire data, please feel free to skip this segment
3. Wire Data is machine data, recorded as events, that we capture from the network using packet sniffing technology from a host’s network interface for a variety of standard protocols. It is an authoritative record of what is happening with and to your operations in real-time. It is a record of all communication between machines and applications We say that wire data is poly structured since certain protocols are more rigid than others. For example, DNS has little to no variance within the fields/attributes within the protocol while HTTP may have a great degree of variance or additional information within its fields.
4. While wire data is a golden source of operational performance information, it is very challenging to deal with. It is high-volume, running to petabytes of raw data a day; it is high-velocity, with 10Gb/s becoming the new standard capacity in datacenters and ever increasing capacity in the cloud; it is high-variety, with a multitude of application protocols and styles of transactions in use. Wire data can also be difficult to get tin a scalable and affordable manner. There are typically many of places (insturment) on the wire within a single data center where valuable application and operational data can be obtained. This easily extends to hundreds of points distributed across a global enterprise. Tap and SPAN ports can be expensive. And in some cases you may not even have the access to this data. Also you need accurate representation of the wire data is required to maximize its operational value. Finally you need to manage all these physical probes. And in some cases in complex network environments, this access can be limited to networking team. Which means often times the data is in silo and teams that need these data such as application owners may not have it readily available as it is within network operations ownership.
5. When you capture this wire data, you can get very deep insights across various use cases including transaction payloads, application performance, infrastructure bottlenecks, security vulnerabilities, customer payloads and usage metrics, troubleshooting and analytics. Second, capturing wire data has no impact on workloads as it Is passive and non intrusive and it does not require semantic logging by customer or byte-code instrumentation. Finally it is comprehensive as we get real-time insights into everything where we can correlate it with log data, database, Hadoop and systems data.
6. Splunk App for Stream is a free App that enables you to capture, visualize and analyze data in much more granular way then ever before. You can see everything – ALL user and applications behavior ],response times from every layer, DNS information, storage traffic, network traffic, your websites content, connections. Once this data is in Splunk you can correlate it with other data for much more comprehensive visibility. First Splunk App for Stream is a way of get wire data into Splunk Enterprise. By adding this comprehensive source of machine data, it enables you to extend Operational Intelligence use cases across IT security and the business. It is a software only solution with the ability that can be installed on VM on any host, it enables real-time insights into multi-cloud environments. And as such, it is easy to install anywhere on most of standard machines, it is a passive very efficient way to capture data.
7. What can you get out of wire data that you don’t already get from other machine data? Many different things as shown here much more than what specific application chose to log. Anything from data that appeal to the admin level user – the things as how long it takes for this page to load or round trip time. Than application owners can get information valuable for them, what are the error messages we are getting from particular application so that they can further investigate the applications issues. Finally, wire data contains information relevent for business users, what are customers buying, are they abandoning carts, where are this purchases coming from. And this is just a small example….there is way more. There is a small amount of overlap between wire data and other data that we’ve captured so far but it requires deeper and more intrusive instrumentation Optional text For example, web server logs typically record status codes such as HTTP 200 response, indicating whether a web page was rendered properly to a client. However, what is missing is transaction payload information – that means, it will not be able to show which of these HTTP 200 responses were for pages with a “service unavailable” message. This information is contained in wire data or transaction payload and is not logged by the server. Can you get this from log data – yes, if you instrument the code. And that is the beauty of wire data – it does not require any instrumentation of the application.
8. With this app users can capture application transaction times, transaction paths, network performance, and even database queries. Correlating wire data with other application and infrastructure data in Splunk software such as logs, metrics and events, As a result users are getting insights about app, service or network availability, performance and usage of their services. IT admins can pinpoint root-cause, proactively monitor the performance and availability of their individual technology silos, map dependencies of infrastructure to applications and trend performance to establish baselines. For security, wire data extends itself into rapid incident investigation. more complete threat detection, expanded monitoring and compliance. For business, wire data also captures user interactions and process insights for a deeper understanding of the user experience to support multiple business analytics use cases. The Splunk App for stream enables efficient, cloud-ready wire data collection with a single software solution. This provides real-time visibility into any public, private or hybrid cloud infrastructure through insights from wire data. Additionally, customers can now securely decrypt SSL encrypted data for data completeness. Capture only the relevant wire data for analytics, through filters and aggregation rules. The app provides the ability to control and manage wire data volumes with fine-grained precision by selecting or deselecting protocols and associated attributes within the App interface Lastly, can be rapidly deployed to collect wire data in real time to gain network visibility that is otherwise unavailable from cloud implementations and hard to achieve with traditional datacenters. Now, customers can quickly respond to any issue with a simple interface-driven installation, centralized deployment and configuration across IT environments of all sizes.
9. Let me go over Splunk Stream utilization in CanDeal. CanDeal is a Canadian online exchange for Canadian dollar debt securities. They provides their investors access to liquidity for Canadian Government Bonds and money market instruments. Stream is deployed at CanDeal across variety of different use cases – security, IT operations even application development. Their teams can collaborate together at CanDeal – in the past, due to strict restriction to who has access to financial data, developers could not get to production MySQL environment as raw visibility for packet data was something they never had access before. Now security team gives them visibility and they can control and they can access any time without the need to wait which significantly improves turnaround times and visibility into issues. preprod testing can also be quickly done. As a result they have improved collaboration among all different teams. In the past, they spent hours just collecting data, shuttling pcap files which created tremendous lag time. Customer satisfaction: In real time they can detect proxy issues, SSL mismatching, misconfigured routes, [Security]Splunk Stream helps Candeal to get huge value in their security practice. They now able to get indicators of compromise by bringing data from STIX into Splunk (utilizing Splice) and cross-correlate against data they are getting from Stream (HTTP, DNS, etc).Since they have a full user and applications behavior, they are now able to quickly investigate and mitigate ATS, analyze potential data exfiltration and other risks in their environment. In the past it was very hard and time consuming to grab data from various pcap and it was fragmented and further it was not indexed in Splunk. [Executive] They are able to create executive reports and present to executives which they could not do with tools they had in practice in the past.
10. In this example, the Stream is deployed in of the large national banks out of Texas. They had acquired branches around the country and in the process integrating them with the hq datacenters. They have several months to do the integration. They are using Stream to better understand the traffic that is going across key links not only within the country but also international. Stream gives them very granular visibilty into any traffic, they can understadn top talkers vs top communicators. They can apply analysis to trigger an alert if the traffic utilization is over specific threshold. And the data is used by new IT personnel. What they are getting from Stream that they cannot get from these other tools Is Splunk analytics behind. With other tools they can get some data but the granularity is not there. And many of the tools don’t look at client perspective. Example: With Stream and Splunk this customer can perform granular analytics they could not do with other tools. “ With other tools I can look at my conversations or all my bytes coming across are, you know, 50 percent of that is, you know, one host, you have thrown a load on that. I can alert when the bandwidth is 85 percent, right? I can do that all day long with other tools But I can't necessarily go look at the traffic and alert on, "Hey, this is I.P. address is taking all the bandwidth. That and much more I can do with Stream”.
11. This is a company that has deployed Splunk in financial industry and specifically in SaaS based payment processing. They are deploying Stream to monitor wire data traffic in their internal communication as they can easily detect anomalies in traffic. For example, they are able to look into database traffic mySQL and postgres traffic and detect issues with user authentication and more. They are looking at what type of data is being sent at their SQL and postgres servers. One of the biggest value for them is that they are able to apply Splunk statistical analysis on wire data and normalize the quiries so that they can prevent external infiltration and avoid malicious attacks. Both in real-time and historically, they are able to set baselines in the amount and type of their database communication . By doing that they were able prevent injection of malicious queries, ensuring there were no attacks on their servers. They were able to integrate wire data in existing security dashboards and proactively look for any abnormalities in communication. They are also able to look for unexpected traffic such as IRC communication or look for exposed passwords in the user authentication. Protocols: MySQL, postgres , LDAP, RADIUS, IRC, SMB, FTP.
12. This is a customer from one of the banking institutions in US. They have deployed Stream to monitor data on DMZ and on egress at the points where there is visibility across all the traffic. They wanted to simplify the data collection for forensics purposes. They did not want to search multiple tools to get the data they are looking for. The value for Stream is how fast can they resolve and close security cases. They got Stream because they wanted to get to the so called “higher level” data. For example, logs from firewalls offered them a very basic info example such as this user tried to connect to this or that external website or that external user wanted to connect to this resource from the outside. They get IP destination port and that is it. From Stream we are getting better understanding of the traffic. Now they can answer these question: This user from the outside tried to issue an SQL injection. Once they have the IP address from firewall they can search the Stream and they can get the better view of what the user did. [The way they did it before was to get the pcap from the user based on the firewall log IP information. Now they don’t need to go and get the pcap to get into very minor detail. We can just look into Splunk and see that is actually what happened.] They are looking into lots of things from their IDS including alerts and things . SQL injection, exploit attempt, etc. If it is something new, we go and check Stream out for more details. Before Stream one example would be as we would be going into IDS alert and bring that into a pcap and then look at pcap into another tool to see what happened, it would take me an hour. With Stream, if get data, enter source and destination IP the get this instantly. Then they can further determine whether I need to investigate more or not. With Stream it goes down to 5 min which is 90% reduction. It is much easier to get data now. ” For them the ability to look at meta data for HTTP level data, and see the things such as the user agent, the response is valuable and very useful for someone in security domain
13. New functionality: Custom Content Extraction Enables extraction, A simple GUI to create and apply rules to extract valuable insights on-the-fly without storgint complete payload or manually parsing the payload data Security: Quickly and easily analyze web traffic for potential security risks with a rules-based GUI extraction –Look for potential data exfiltration, including exposed assets, user credentials such as clear text passwords, or personal identifiable information such as credit card numbers –Prevent data loss, provide easier forensics and reduce troubleshooting times Digital Intelligence: Get real-time granular insights into key business indicators from web traffic payload for efficient business analytics, including marketing and transactional data –Visibility into shopping carts, user interactions, and other important business IT Operations/Applications Visibility: Monitor web services performance through protocols such as SOAP or JSON-RPC by extracting per-API response times or other information from payload data in real time
14. Here is the current list of protocols that are supported. We also now support Windows OS and also have improved performance. Here we see currently supported protocols and platforms. Talk with your customers and them if there is any other protocol they find extremely useful that they would like to be added. And also ask them why would need particular protocol to be added.
15. And finally, events are generated based on the Stream configuration from “App for Stream” and passed on to the UF as modular input data (streaming standard output) in JSON format.
16. We can get wire data directly from the “wire” by installing our wire data collector (the TA) on a dedicated, physical server. This server then receives a passive network copy from a SPAN/(TAP) or packet broker which would transport the “real” wire data of interest to the software.
17. Alternatively, the data collector can live directly on the systems of interest as a lightweight agent, where the systems can be either physical or virtual. In both cases the data collectors are actually TAs and therefore need to cohabitate with a forwarder.
18. Thank you. Open up for Questions
19. So let’s start with IT Operations – You can capture IT relevant data set from network and enrich it with existing data in Splunk such as infrastructure and application logs and events .You capture the content of database queries, granular IP conversations, transaction traces, applications response times. As a result, they will have granular visibility into infrastructure performance, resources utilization, or solve capacity bottlenecks. They can have visibility into applications availability, performance and usage and relation of it to underlying infrastructure components. IT admins can establish better baselines and trending for application performance and usage, and enable better IT and business decision making. This all results in faster resolutions of problems with fewer people.
20. With the Splunk App for Stream, customers can now unlock the full potential of their machine data by adding wire data to the Splunk software platform. Correlate application and infrastructure data such as logs, events, metrics with wire data to gain valuable insights into application and infrastructure performance, find the root cause of operational issues, understand transaction paths, resolve system downtime, identify infrastructure relationships, assess security threats and understand customer behavior. Enhance operational intelligence for IT, security and the business with wire data analytics, enabled by Splunk software. The Splunk App for Stream captures wire data from endpoints and key network locations to provide additional insight into how applications are performing, without requiring any instrumentation. Wire data collected by the Splunk App for Stream provides granular data on transaction response times, transaction traces, transaction paths, network performance and even database queries. Wire data effectively complements the kind of metrics often gathered by traditional APM tools, which often focus on specific transaction components. Also, the Splunk App for Stream does not require instrumentation of the application itself, so you can gather performance information across the application without developers instrumenting the application or modifying application logs.  
21. Stream brings huge benefits for your security practitioners.. It is particularly interesting as you are most likely used to packet sniffing for forensic and real time analysis. Data captured contains all user activity and behavior as well as applications behavior. With Stream security customers can perform deep protocol inspection understanding at a very granular level what is going in. This can be used both in real time to understand risks or to perform response to an incident. In addition, security investigators can observe daily or seasonal traffic patterns so that they can immediately react when these become anomalous– they can respond to insider threats. See when someone is emailing IP out or if someone is trying to mimic the database queries to trying to gain access to your internal databases. Stream extracts both header and payload information for very deep granular insights for incident response and threat prevention. It is very important to mention that it can be deployed anywhere into end points, without you need to buy having to by expensive appliances. Very important when customer is a breach conditions. Backup Protocol header and data decoding: HTTP, DNS and email protocols (e.g. IMAP, POP3 and SMTP) are the dominant attack and exfiltration vectors for some of the most damaging breaches. Streams can be deployed to acquire header information (HTTP and email) and payload information (DNS) to drive sophisticated analytics for threat detection, incident response, intelligence gathering and threat prevention. Rapid deployment and response: When incident investigation or analysis or tracking down malware requires additional real-time information from network traffic, threat responders can leverage Stream’s simple and rapid deployment via Splunk to start getting wire data from the system of interest to Splunk. This is useful under breach conditions – where a known infiltration may be in progress.
22. Customer Experience & Digital Analytics: The Splunk App for Stream allows organizations to capture all web interactions for a deeper understanding of user experience, to improve customer satisfaction, prevent drop-offs, improve conversions and boost online revenues. Wire data provides insights into key metrics such as time spent on page, bounce rates, time on site, navigation paths, product performance etc., without the need to tag individual pages. This is especially valuable to ensure the success of marketing campaigns. Business Process Analytics: Business processes such as order management in retail, provisioning in telecoms, trade execution in financial services etc. span many different applications. Collecting relevant data across applications and correlating it is critical for end-to-end process visibility. Wire data implicitly has this information, without requiring specific instrumentation. With the Splunk App for Stream, business operations teams can easily access this data and use Splunk Enterprise to gain real-time business insights across the complete process.
23. FaQ: The first, Explosion in the data volume We have built in filters and aggregation rules in the app so that our customers can fine tune data gathering Indexing volume can vary depending on the enabled protocols and uses and the amount of network traffic. To address that, our customers should try out the app on smaller environments and understand the usage types and what kind of data is useful to them. OK. We got that one sorted out. But hey, it is still lots of data. How will I know what is useful to me.As part of the App, we built the Stream Examples App which contains prebuilt sets of sample searches that will portray several scenarios. This includes, security scenario, shopping cart, application latencies. And as customers start using the app, we will add more.
24. \
25. Thank you. Open up for Questions