Google Machine Learning APIs - puppies or muffins?
•
6 likes•2,107 views
Want to take advantage of machine learning without building and training your own models? The Google Cloud Vision and Speech APIs expose the machine learning functionality behind Google Photos, Google Images, and the speech recognition in “Ok, Google.” Developers can now build these features into their apps with a simple REST API call. This deck gives an overview of Google Cloud Vision API, Cloud Speech API, and the Cloud Natural Language API.
Report
Share
Google Machine Learning APIs - puppies or muffins?
- 1. Puppies or muffins? Easily leverage machine learning in your apps Sara Robinson @SRobTweets Bret McGowen @bretmcg
- 2. 2@SRobTweets @bretmcg Who are we? Developer Advocate, Google Cloud Platform Sara Robinson / @SRobTweets ● New York, NY ● Swift fan (Taylor and language) ● Harry Potter aficionado Developer Advocate, Google Cloud Platform Bret McGowen / @bretmcg ● New York, NY ● U2 fan (band and plane) ● Lord of the Rings aficionado
- 3. What we’ll cover 01 02 03 04 05 A (very) brief overview of machine learning Machine learning at Google Vision API Speech API Natural Language API
- 4. 01 A (very) brief overview of machine learning
- 8. 02 Machine Learning at Google
- 13. Combined vision and translation
- 17. Google Cloud Platform 17 Doesn't seem THAT hard...
- 19. Google Cloud Platform 19 What if I'm not a machine learning expert?
- 20. 20@SRobTweets @bretmcg The Machine Learning Spectrum TensorFlow Cloud Machine Learning Machine Learning APIs BYOML skills (Friendly Machine Learning) Pre-packaged ML
- 21. 02 The Cloud Vision API Complex image detection with a simple REST request
- 23. 03 Making an API request
- 24. Making a request { "requests":[ { "image": { "content": "base64ImageString" // Alternatively, you can pass a Google Cloud Storage url here }, "features": [ { "type": "LABEL_DETECTION", "maxResults": 10 }, { "type": "FACE_DETECTION", "maxResults": 10 }, // More feature detection types... ] } ]
- 25. Google Cloud Platform 25 Let’s see some JSON responses
- 26. 26 { "labelAnnotations" : [ { "mid" : "/m/01wydv", "score" : 0.92442685, "description" : "beignet" }, { "mid" : "/m/0270h", "score" : 0.90845567, "description" : "dessert" }, { "mid" : "/m/033nb2", "score" : 0.74553984, "description" : "profiterole" }, { "mid" : "/m/01dk8s", "score" : 0.71415579, "description" : "powdered sugar" } ] } Label Detection 26
- 27. "landmarkAnnotations": [ { "mid": "/m/0c7ln", "description": "Navy Pier", "score": 36, "boundingPoly": { "vertices": [ { "x": 275, "y": 102 }, //... ] }, "locations": [ { "latLng": { "latitude": 41.888685, "longitude": -87.601311 } } ], //... }, { "mid": "/m/01_d4", "description": "Chicago", "score": 31, Landmark Detection 27
- 28. 28 ... "itemListElement": [ { "@type": "EntitySearchResult", "result": { "@id": "kg:/m/0c7ln", "name": "Navy Pier", "@type": [ "Thing", "Place", "LandmarksOrHistoricalBuildings", "TouristAttraction" ], ... "detailedDescription": { "articleBody": "Navy Pier is a 3,300-foot-long pier on the Chicago shoreline of Lake Michigan. It is located in the Streeterville neighborhood of the Near North Side community area.", "url": "http://en.wikipedia.org/wiki/Navy_Pier" ... Knowledge Graph sidebar GET https://kgsearch.googleapis.com/v1/entities:search?ids=%2Fm%2F0b__kbm&key={API_KEY}
- 29. 29 "textAnnotations": [ { "locale": "en", "description": "U.S. COAST GUARD AUXILIARYn242039n", "boundingPoly": { "vertices": [ { "x": 429, "y": 307 }, { "x": 1178, "y": 307 }, { "x": 1178, "y": 770 }, { "x": 429, "y": 770 } ] } }, // ... ] Text Detection
- 30. 30 "faceAnnotations" : [ { "headwearLikelihood" : "VERY_LIKELY", "surpriseLikelihood" : "VERY_UNLIKELY", "rollAngle" : 2.8030474, "angerLikelihood" : "VERY_UNLIKELY", "landmarks" : [ { "type" : "LEFT_EYE", "position" : { "x" : 221.60617, "y" : 638.263, "z" : 0.0017568493 } }, ... ], "boundingPoly" : { "vertices" : [ { "x" : 89, "y" : 436 }, ... Face Detection "detectionConfidence" : 0.98838496, "joyLikelihood" : "VERY_LIKELY", "panAngle" : -1.0822374, "sorrowLikelihood" : "VERY_UNLIKELY", "tiltAngle" : -2.5003448, "underExposedLikelihood" : "VERY_UNLIKELY", "blurredLikelihood" : "VERY_UNLIKELY"
- 31. 31 "faceAnnotations" : [ { "headwearLikelihood" : "VERY_UNLIKELY", "surpriseLikelihood" : "VERY_UNLIKELY", rollAngle" : -4.6490049, "angerLikelihood" : "VERY_UNLIKELY", "landmarks" : [ { "type" : "LEFT_EYE", "position" : { "x" : 691.97974, "y" : 373.11096, "z" : 0.000037421443 } }, ... ], "boundingPoly" : { "vertices" : [ { "x" : 743, "y" : 449 }, ... Face Detection "detectionConfidence" : 0.93568963, "joyLikelihood" : "VERY_LIKELY", "panAngle" : 4.150538, "sorrowLikelihood" : "VERY_UNLIKELY", "tiltAngle" : -19.377356, "underExposedLikelihood" : "VERY_UNLIKELY", "blurredLikelihood" : "VERY_UNLIKELY"
- 32. 32 "labelAnnotations" : [ { "mid" : "/m/01fklc", "score" : 0.9337945, "description" : "pink" }, { "mid" : "/m/09g5pq", "score" : 0.83878618, "description" : "people" }, { "mid" : "/m/017ftj", "score" : 0.71847415, "description" : "sunglasses" }, { "mid" : "/m/019nj4", "score" : 0.69381392, "description" : "smile" } ] Putting it all together: face + label + landmark "landmarkAnnotations" : [ { "boundingPoly" : { "vertices" : [ { "x" : 153, "y" : 64 }, ... ] }, "mid" : "/m/0c7zy", "score" : 0.56636304, "description" : "Petra", "locations" : [ { "latLng" : { "longitude" : 35.449361, "latitude" : 30.323975 } } ] } ]
- 33. 33 "safeSearchAnnotation" : { "spoof" : "VERY_UNLIKELY", "medical" : "VERY_UNLIKELY", "adult" : "VERY_UNLIKELY", "violence" : "VERY_UNLIKELY" } But wait...is it appropriate?
- 34. 04 Live Demo
- 35. 03 The Speech API Speech to text transcription in over 80 languages
- 36. 36@SRobTweets @bretmcg What can I do with the Speech API? ● Speech to text transcription in over 80 languages ● Supports streaming and non-streaming recognition ● Filters inappropriate content
- 37. 37@SRobTweets @bretmcg Translation Response "responses": [{ "results": [{ "alternatives": [ { "transcript": "how old is the Brooklyn Bridge", "confidence": 0.987629 }], "isFinal": true } ] }]
- 38. 38@SRobTweets @bretmcg Let’s make a recording! 1. Make a recording using SoX, a command line utility for audio files 2. Base64 encode the recording 3. Build our API request in a JSON file 4. Send the JSON request to the Speech API Bash script for this: bit.ly/speech-request-script
- 39. 04 Cloud Natural Language API Perform sentiment analysis and entity recognition on text
- 40. 40@SRobTweets @bretmcg What can I do with the Natural Language API? Three methods: 1. Analyze entities - The Cubs are an MLB team from Chicago 2. Analyze sentiment - I love Chicago 3. Analyze syntax - Michelle Obama is married to Barack Obama
- 41. 41@SRobTweets @bretmcg Analyze Entities Chicago is The Wrigley Building, Chicago is The Union Stockyard, Chicago is One town that won't let you down It's my kind of town -- Frank Sinatra
- 42. 42@SRobTweets @bretmcg Analyze Entities Chicago is The Wrigley Building, Chicago is The Union Stockyard, Chicago is One town that won't let you down It's my kind of town -- Frank Sinatra
- 43. 43@SRobTweets @bretmcg Analyze Entities Chicago is The Wrigley Building, Chicago is The Union Stockyard, Chicago is One town that won't let you down It's my kind of town -- Frank Sinatra
- 44. 44@SRobTweets @bretmcg Analyze Sentiment Chicago is the best city in the world. { “documentSentiment”: { “polarity”: 1, “magnitude”: 0.8 } }
- 45. 45@SRobTweets @bretmcg Analyze Syntax “The Chicago Cubs are an American professional baseball team based in Chicago, Illinois.”
- 47. 47@SRobTweets @bretmcg Let’s see a demo! Twitter Streaming NL API BigQuery bit.ly/nl-olympics
- 48. 48@SRobTweets @bretmcg APIs we covered ● Vision: cloud.google.com/vision ● Speech: cloud.google.com/speech ● Natural Language: cloud.google.com/natural-language Related APIs: ● Translate: cloud.google.com/translate ● Prediction: cloud.google.com/prediction ● Knowledge Graph API: developers.google.com/knowledge-graph
- 49. Thank You Sara Robinson @SRobTweets Bret McGowen @bretmcg