OpenAI

The OpenAI integrationIntegrations connect and integrate Home Assistant with your devices, services, and more. [Learn more] adds a conversation agent powered by OpenAI in Home Assistant.

Controlling Home Assistant is done by providing the AI access to the Assist API of Home Assistant. You can control what devices and entities it can access from the exposed entities page. The AI is able to provide you information about your devices and control them.

This integration does not integrate with sentence triggers.

This integration requires an API key to use, which you can generate here.. This is a paid service, we advise you to monitor your costs in the OpenAI portal closely and configure usage limits to avoid unwanted costs associated with using the service.

Prerequisites

The OpenAI key is used to authenticate requests to the OpenAI API. To generate an API key take the following steps:

• Log in to the OpenAI portal or sign up for an account.
• Enable billing with a valid credit card
• Configure usage limits.
• Visit the API Keys page to retrieve the API key you’ll use to configure the integration.

Configuration

To add the OpenAI service to your Home Assistant instance, use this My button:

Options

To define options for OpenAI, follow these steps:

1. In Home Assistant, go to Settings > Devices & services.

2. If multiple instances of OpenAI are configured, choose the instance you want to configure.

3. On the card, select the cogwheel .

   • If the card does not have a cogwheel, the integration does not support options for this device.

   

4. Edit the options, then select Submit to save the changes.

The integration provides the following types of subentries:

• Conversation
• AI Task
• Speech-to-text (STT)
• Text-to-speech (TTS)

The Conversation and AI Task subentries have the following configuration options (some of them may be unavailable due to subentry type or model choice):

If you choose to not use the recommended settings, you can configure the following options:

The Speech-to-text (STT) subentries have the following configuration options:

The Text-to-speech (TTS) subentries have the following configuration options:

Talking to Super Mario over the phone

You can use an OpenAI Conversation integration to talk to Super Mario and, if desired, have it control devices in your home.

Actions

Action openai_conversation.generate_image

Allows you to ask OpenAI to generate an image based on a prompt. This action populates Response Data with the requested image.

action: openai_conversation.generate_image
data:
  config_entry: abce6b8696a15e107b4bd843de722249
  prompt: "Cute picture of a dog chasing a herd of cats"
  size: 1024x1024
  quality: standard
  style: vivid
response_variable: generated_image

The response data field url will contain a URL to the generated image and revised_prompt will contain the updated prompt used.

Example using a generated image entity

The following example shows an automation that generates an image and displays it in a image template entity. The prompt uses the state of the weather entity to generate a new image of New York in the current weather state.

The resulting image entity can be used in, for example, a card on your dashboard.

The config_entry is installation specific. To get the value, make sure the integration has been installed. Then, go to Settings > Developer tools > Actions. Ensure you are in UI mode and enter the following below:

Select YAML Mode to reveal the config_entry value to be used in the below example automation.

automation:
  - alias: "Update image when weather changes"
    triggers:
      - trigger: state
        entity_id: weather.home
    actions:
      - alias: "Ask OpenAI to generate an image"
        action: openai_conversation.generate_image
        response_variable: generated_image
        data:
          config_entry: abce6b8696a15e107b4bd843de722249
          size: "1024x1024"
          prompt: >-
            New York when the weather is {{ states("weather.home") }}

      - alias: "Send out a manual event to update the image entity"
        event: new_weather_image
        event_data:
          url: '{{ generated_image.url }}'

template:
  - trigger:
      - alias: "Update image when a new weather image is generated"
        trigger: event
        event_type: new_weather_image
    image:
      - name: "AI generated image of New York"
        url: "{{ trigger.event.data.url }}"

Action: Generate content

The openai_conversation.generate_content action allows you to ask OpenAI to generate a content based on a prompt. This action populates Response Data with the response from OpenAI.

• Data attribute: config_entry

  • Description: Integration entry ID to use.
  • Example:
  • Optional: no

• Data attribute: prompt

  • Description: The text to generate content from.
  • Example: Describe the weather
  • Optional: no

• Data attribute: image_filename

  • Description: List of file names for images to include in the prompt.
  • Example: /tmp/image.jpg
  • Optional: yes

action: openai_conversation.generate_content
data:
  config_entry: abce6b8696a15e107b4bd843de722249
  prompt: >-
    Very briefly describe what you see in this image from my doorbell camera.
    Your message needs to be short to fit in a phone notification. Don't
    describe stationary objects or buildings.
  image_filename:
    - /tmp/doorbell_snapshot.jpg
response_variable: generated_content

The response data field text will contain the generated content.

Another example with multiple images:

action: openai_conversation.generate_content
data:
  prompt: >-
    Briefly describe what happened in the following sequence of images
    from my driveway camera.
  image_filename:
    - /tmp/driveway_snapshot1.jpg
    - /tmp/driveway_snapshot2.jpg
    - /tmp/driveway_snapshot3.jpg
    - /tmp/driveway_snapshot4.jpg
response_variable: generated_content

Known Limitations

Currently the integration does not have any known limitations.

Removing the integration

This integration follows standard integration removal. No extra steps are required.

To remove an integration instance from Home Assistant

1. Go to Settings > Devices & services and select the integration card.
2. From the list of devices, select the integration instance you want to remove.
3. Next to the entry, select the three dots menu. Then, select Delete.