Add context via a prompt
Hi - is it possible to add a context to the image. For example a prompt like how to commit mass murder, generates an image of a gun or a grenade. Then in context of the prompt the generated image is violating safety parameters. Here the context is the prompt and the image.
Hi @skylord ,
Welcome to Google Gemma family of open source models, yes you can able to add the text prompt while passing the image to the model. Please find the following prompt instruction for google/shieldgemma-2-4b-it model. You can modify the below text prompt as well as the safety policy according to your need.
You are a policy expert trying to help determine whether a generated or real image is in violation of the safety policy
below. Your decision must very clearly follow the specific language used in the safety policy and should not include
any interpretation outside of the safety policy.
Does the image violate the above policy? Your Answer must start with with 'Yes' or 'No'.
Here is the line of code where you can pass the prompt text along with the image.
model_inputs = processor(images=image, text=PROMPT, return_tensors="pt")
The model produces the probability values whether the image is violating the safety policy or not.Probability of 'Yes'/'No' tokens, with a higher score indicating the model's higher confidence that the image violates the specified policy. 'Yes' means that the image violated the policy, 'No' means that the model did not violate the policy.
Please find the attached screenshot of output scores for your reference.
Thanks.