Ctrl+K

Our first Generative AI application

Our first Generative AI application#

For this experiment, we will use the GPT-4-Vision multimodal model. We do not have direct access to the parameters of the model, so instead, we interact with it through an API.

This is a deep-learning based model, which takes two inputs: and image, and some text describing a task or question. The model processes both, and then outputs a response.

For the moment, we won’t go into the details (that is the matter of the whole course!), but we can start playing with it right away.

GPT-4-Vision

# You need to install the OpenAI library:
#pip install --upgrade openai
from openai import OpenAI
import os

# Write your OpenAI API key here
os.environ["OPENAI_API_KEY"] = "sk-..."

client = OpenAI()

Let’s define a function to interact with the GPT-4-Vision API.

It takes as inputs:

  • The image_url

  • The textual prompt with our query

def compute_response(image_url, prompt):
    response = client.chat.completions.create(
        model="gpt-4-vision-preview",
        messages=[
            {
                "role": "user",
                "content": [
                    {"type": "text", "text": prompt},
                    {
                        "type": "image_url",
                        "image_url": image_url,
                    },
                ],
            }
        ],
        max_tokens=500,
    )
    return response.choices[0].message.content

from IPython.core.display import Image, display
display(Image(url='https://pbs.twimg.com/media/FrSGyS9WIAMLGbe?format=jpg&name=small', width=500, unconfined=True))

image_url = "https://pbs.twimg.com/media/FrSGyS9WIAMLGbe?format=jpg&name=small"
prompt = "Describe what is in this photo"

response = compute_response(image_url, prompt)

print(response)

This is a photo of the inside of a refrigerator with various food items stored on its shelves. On the top shelf, there is a carton of eggs, a couple of brown muffins or round pastries, and a jug of milk next to a stack of blue-and-white striped cups or bowls. The middle shelf features a box of blueberries, a bunch of green grapes, several oranges, and what appear to be individually wrapped cheese portions to the left. On the bottom shelf, there are more food storage containers, some of which contain meats and others that are not clearly visible in terms of their contents. The design and organization of the refrigerator suggest that it's well-kept and contains a variety of fresh produce and other food items.

Creating a photo2recipe app#

We just have to change the prompt!

recipe_prompt = "Based on the ingredients you can see in the photo, please write a recipe for some dish."

recipe_answer = compute_response(image_url, recipe_prompt)

print(recipe_answer)

Based on the ingredients visible in the image, I can propose a recipe for a simple fruit salad with a yogurt dressing.

Fruit Salad with Yogurt Dressing

Ingredients:
- A handful of grapes
- 1-2 oranges, depending on size
- A container of blueberries
- 2 avocados
- Plain yogurt (quantity as per preference)
- A drizzle of honey or a sprinkle of sugar for sweetness (optional)
- A squeeze of fresh lemon or orange juice (optional)
- Fresh mint leaves for garnish (not visible in the photo, but commonly available)

Instructions:
1. Carefully wash the grapes and blueberries and allow them to dry.
2. Peel the oranges, removing as much of the white pith as possible, and segment them. Cut the segments into smaller pieces if desired.
3. Halve the avocados, remove the pits, and carefully scoop out the flesh. Cut the avocado into bite-sized chunks.
4. In a large mixing bowl, gently toss together the grapes, blueberry pieces, orange segments, and avocado chunks.
5. In a separate small bowl, combine the plain yogurt with a drizzle of honey or a sprinkle of sugar, and a squeeze of fresh lemon or orange juice to taste. Whisk until well blended.
6. Drizzle the yogurt dressing over the fruit just before serving. Give it a gentle toss to coat the fruit evenly.
7. Garnish with fresh mint leaves if available.
8. Serve immediately and enjoy!

Note: Adjust the quantities of each fruit according to your preference or the number of servings needed. If you want to make this ahead of time, mix in the avocado and the yogurt dressing at the last minute to prevent browning and to keep the fruit fresh.

Let’s try with another photo:

from IPython.core.display import Image, display
display(Image(url='https://images.fineartamerica.com/images-medium-large-5/table-laid-with-ingredients-and-utensils-manuel-sulzer.jpg', width=500, unconfined=True))

image_url = "https://images.fineartamerica.com/images-medium-large-5/table-laid-with-ingredients-and-utensils-manuel-sulzer.jpg"

recipe_answer = compute_response(image_url, recipe_prompt)

print(recipe_answer)

Based on the ingredients in the photo, it looks like a perfect set-up for creating a classic hamburger. Here's a simple recipe:

Classic Hamburger Recipe:

Ingredients:
- Ground beef (about 1 pound for four burgers)
- Hamburger buns (four)
- Lettuce
- Tomato
- Onion (purple ones seen here)
- Carrots
- Pickles
- Eggs (optional, for making the patties extra juicy)
- Milk (optional, can be used for the burger patties mixture)
- Garlic
- Butter (for toasting buns)
- Salt and pepper
- Cheese (not visible in the photo, but a common ingredient in burgers)
- Oil for cooking
- Any other preferred seasonings (like the thyme seen here)

Instructions:

1. Prepare the Vegetables:
   - Wash the lettuce and set it aside to dry.
   - Slice the tomatoes and onions for toppings.
   - Shred the carrots for additional burger topping if desired or serve on the side.

2. Make the Burger Patties:
   - In a bowl, mix the ground beef, finely minced garlic, salt, pepper, and, if you like, an egg for binding. You can also add a splash of milk if you prefer.
   - Divide the meat into four equal portions and shape them into patties.

3. Cook the Burgers:
   - Heat a pan or grill to medium-high heat and add a bit of oil.
   - Place your burger patties on the hot pan/grill and cook for about 3-4 minutes on one side.
   - Flip the patties and cook for another 3 minutes for medium-rare. Cook for a longer time if you prefer your burgers more well done.
   - If you're adding cheese, place a slice on each patty when it's almost done and allow it to melt.

4. Toast the Buns:
   - Split the buns in half.
   - Spread a little butter on each cut side.
   - Toast the buns on the pan/grill until they're lightly browned.

5. Assemble the Burgers:
   - On the bottom half of each bun, place a piece of lettuce, followed by a slice of tomato.
   - Add the cooked burger patty with melted cheese.
   - Top with onion, pickles, and shredded carrots if desired.
   - Cover with the top half of the bun.

Serve with your favorite

Can you think of other applications?#

image_url = "https://images.fineartamerica.com/images-medium-large-5/table-laid-with-ingredients-and-utensils-manuel-sulzer.jpg"

prompt = "For each ingredient in the photo, output a healthy score from 0 to 5"
recipe_answer = compute_response(image_url, prompt)

print(recipe_answer)

It's important to note that a "healthy score" is subjective and can vary depending on an individual's dietary needs, allergies, and health goals. However, I can provide a general scoring based on the nutritional content and health benefits generally associated with each food item.

Starting from the top left and moving right:

1. Leafy greens (such as lettuce) - Score: 5 (Rich in vitamins, minerals, and fiber. Low in calories.)
2. Cheese - Score: 3 (Contains calcium and protein but also saturated fat. Score may vary depending on type and portion size.)
3. Eggs - Score: 4 (High-quality protein and nutrients but high in cholesterol—recommendation can vary based on individual cholesterol levels.)
4. Pickles - Score: 2 (Low in calories but can be high in sodium.)
5. Hamburger buns - Score: 2 (Typically made from refined grains, low in fiber and nutrients compared to whole-grain options.)

Moving to the second row from left:

1. Lettuce (possibly a different variety from the first) - Score: 5 (Similar reasons as the leafy greens mentioned earlier.)
2. Fresh herbs - Score: 5 (High in phytonutrients and flavors without added calories.)
3. Onions - Score: 4 (Nutrient-dense and fiber-rich. Beneficial for health.)
4. Garlic - Score: 5 (Associated with numerous health benefits and very low in calories.)
5. Ground beef - Score: 3 (Source of protein and iron, though it can be high in saturated fat. Leaner cuts would have a higher score.)

Moving to the bottom row from left to right:

1. Black pepper and salt - Score: 2 (Black pepper is beneficial but in moderation, while too much salt can lead to health issues.)
2. Butter - Score: 2 (Rich in fat and calories. While it provides flavor, too much can contribute to health risks.)
3. Carrots - Score: 5 (Rich in fiber, vitamins, and low in calories.)
4. Tomatoes - Score: 5 (Rich in vitamins, minerals, and lycopene, a powerful antioxidant.)

These scores are general assessments and should be personalized according to dietary needs. For example, individuals with specific health conditions (like hypertension) should be cautious with salty food items, while those with lactose intolerance would need to avoid or limit dairy products like cheese

For more details, you can:

Contents