1. Intro
1.1. What am I going to talk about
1.1.1. What is AI?
1.1.1.1. Machine Learning
1.1.1.2. GenAI
1.1.2. This talk focuses on GenAI
1.2. Goals of this presentation
1.2.1. GenAI is not just chatbots
1.2.2. Think outside the box
1.2.2.1. How could you use GenAI in your current projects?
1.2.3. GenAI is like a good'ish intern
1.2.4. Don't trust, and verify
1.2.4.1. Always have a fallback solution
1.2.4.2. Careful of hallucinations
1.2.4.2.1. Caused by lack of data
1.2.4.2.2. Caused when not specific enough
1.3. Full Disclosure
1.3.1. I am not an expert
1.3.2. I am still very early in my education with AI
1.3.3. I want to help you shortcut your education
1.4. GenAI
1.4.1. LLMs
1.4.1.1. How they are made
1.4.1.1.1. Lots of data
1.4.1.1.2. Don't even attempt to build one yourself
1.4.1.2. Tokens predictor
1.4.1.2.1. Tokens are chunks of text
1.4.1.2.2. It predicts the next token using the vast amount of data it knows
1.4.1.2.3. Simulate humans
1.4.1.3. Common ones
1.4.1.3.1. Commercial
1.4.1.3.2. Open Source
1.4.2. GenAI tools we use at OSS
1.4.2.1. Dialogflow CX
1.4.2.1.1. Generative Playbooks
1.4.2.2. VertexAI
1.4.2.2.1. Set of APIs to make it simpler to code with GenAI
1.4.2.2.2. Large collection of models to choose from
1.4.2.2.3. SDKs
1.4.2.3. LangChain
1.4.2.3.1. SDKs
2. Prompt Engineering
2.1. More than just chatbots
2.1.1. Classification
2.1.1.1. Classify the following picture into one of these predefined type of packaging: ["envelope", "box", "bag"]
2.1.2. Named Entity Recognition
2.1.2.1. Identify and list all the items on and around the desk on this picture
2.1.3. Sentiment Analysis
2.1.3.1. Perform a sentiment analysis of the following picture of a person coming to visit the office today.
2.2. Use GenAI as applications
2.2.1. Data In / Data Out
2.2.2. Natural language
2.2.3. JSON schema output
2.3. More accurate output
2.3.1. Patterns
2.3.1.1. Persona Pattern
2.3.1.1.1. You are a...
2.3.1.1.2. Your mission is...
2.3.1.2. Audience Pattern
2.3.1.2.1. Assume that I am an employee working in an office
2.3.2. Single subject prompt
2.3.2.1. Bad prompt
2.3.2.1.1. What is the desk with the best view outside at the San Jose office and will be the weather in San Jose tomorrow?
2.3.3. Zero, One, Few Shots
2.3.3.1. Used to give the model some examples of the desired output
2.3.3.2. To get a more accurate output
2.3.3.3. Can be used to specify your JSON schema
2.3.3.4. Examples
2.3.3.4.1. [Picture of a box] => "box"
2.3.3.4.2. [Picture of a desk] => ["dual monitor", "mecanical keyboard", "webcam", "phone"]
2.3.3.4.3. [Wedding picture of Cordero] => "He looks like someone who you don't want to mess with, you better be nice to him."
3. Retrival Augmented Generation
3.1. Private data
3.1.1. How do you make LLMs aware of your private data?
3.1.1.1. And stay private
3.1.1.2. And stay secured
3.1.2. RAG is the solution
3.1.2.1. Additional data appended to the large language model source of data
3.1.3. Why RAG?
3.1.3.1. No need to train the data again
3.1.3.2. Decouples your private data from the LLMs
3.1.3.3. Easy to switch to a different LLM
3.1.3.4. Easy to rebuild your private data
3.1.4. Sources
3.1.4.1. SQL
3.1.4.2. BigQuery
3.1.4.3. Website
3.1.4.4. PDFs
3.2. Embeddings
3.2.1. Numerical representation of the data
3.2.2. Groups similar data together
3.2.3. Similar to how search engines indexes data
3.3. Vector Database
3.3.1. Database of embeddings
3.3.2. Fast retrival of embeddings
3.3.3. Similar to how search engines stores data
3.4. Grounding
3.4.1. Verifiable source of information
3.4.2. Provide a link to the source to the user
3.4.3. Reduces hallucinations
3.4.4. Enhances trustworthiness
4. VertexAI
4.1. Model Garden
4.1.1. Let's you experiment with lots of models
4.1.2. Open Source Models
4.1.2.1. Llama
4.1.2.2. Stable Diffusion
4.1.3. Partner Models
4.1.3.1. Anthropic Claude
4.2. Gemini
4.2.1. Multi Modal
4.2.1.1. Text
4.2.1.1.1. Data Analysis
4.2.1.1.2. Summarization
4.2.1.1.3. Translation
4.2.1.2. Image
4.2.1.2.1. Object recognition
4.2.1.2.2. Digital content understanding
4.2.1.3. Audio
4.2.1.3.1. Summarization
4.2.1.3.2. Transcription
4.2.1.3.3. Translation
4.2.1.4. Video
4.2.1.4.1. Captioning
4.2.1.4.2. Summarization
4.2.2. 1 Million Tokens Input
4.2.2.1. 1 hour of video
4.2.2.2. 11 hours of audio
4.2.2.3. 30k lines of code
4.2.2.4. 700k words
4.3. Fine Tuning
4.3.1. Temperature
4.3.1.1. Controls Randomness of the output
4.3.1.1.1. 0.0 = Least random (very strict output)
4.3.1.1.2. 1.0 = Most random (very creative output)
4.3.2. Tokens
4.3.2.1. Top-K
4.3.2.1.1. Limit the total of available next probable token
4.3.2.1.2. Lower = Less Random
4.3.2.2. Top-P
4.3.2.2.1. Limit the total of available next probably token to a sum of their probabilities
4.3.2.2.2. Lower = Less Random
4.4. Safety Settings
4.4.1. Block explicit and dangerous content
4.4.2. Responsible AI
4.5. Function calling
4.5.1. Functions to get data to help the model generate a response
4.5.1.1. API call
4.5.1.2. Any code that returns data that the model can use
4.5.2. Uses parameters extracted from the prompt
4.5.2.1. Defines parameters to extract using an OpenAPI schema
4.5.3. Example: What is the weather like in Atlanta?
4.5.3.1. 1. Define the parameters to extract into a declaration, in this case a "location" that is of "type" "string".
4.5.3.2. 2. Create a parameter extraction tool that uses the location declaration
4.5.3.3. 3. Process the prompt with the location parameter extraction tool which will produce an output containing a location string of "Atlanta"
4.5.3.4. 4. Send the "Atlanta" parameter to a weather API
4.5.3.5. 5. Give the model the JSON data that came back from the API for the parameter "Atlanta"
4.5.3.6. 6. Output the generated response from the model back to the user: "Today it is a very hot and humid 90 degrees (or 32 celcius for everywhere else in the world) in Atlanta."
4.6. Demo
4.6.1. Shipping Box Label to Employee matcher