[Bug]: ChatMessage not recognizing ImageBlock in LlamaIndex 0.12.11

[n1vedhar1]

Bug Description

The ChatMessage class in LlamaIndex 0.12.11 is not recognizing ImageBlock, regardless of the Python version used. Instead, it processes the content as though it only contains TextBlock. This behavior was observed in both Python 3.9.6 and Python 3.13.1.

In earlier versions of LlamaIndex (e.g., 0.12.10), this issue did not occur when using Python 3.9.6. The problem seems to have been introduced in version 0.12.11, significantly impacting workflows that rely on non-text blocks in ChatMessage.

Version

0.12.11

Steps to Reproduce

Steps to Reproduce
1. Set up a virtual environment with Python 3.13.1.
2. Install LlamaIndex and other dependencies.
3. Create a ChatMessage instance with both TextBlock and ImageBlock in the content.
4. Observe the behavior when processing the ChatMessage.
eg:

from llama_index.llms.openai import OpenAI
from llama_index.core.llms import (
    ChatMessage,
    ImageBlock,
    TextBlock,
    MessageRole,
)

msg = ChatMessage(
    role=MessageRole.USER,
    blocks=[
        TextBlock(text="Describe the images as an alternative text"),
        ImageBlock(url="https://letsenhance.io/static/8f5e523ee6b2479e26ecc91b9c25261e/1015f/MainAfter.jpg"),
    ],
)
openai_llm = OpenAI(model="gpt-4o", max_new_tokens=300)
response = openai_llm.chat(messages=[msg])

Relevant Logs/Tracbacks

>>> response = openai_llm.chat(messages=[msg])
DEBUG:httpx:load_ssl_context verify=True cert=None trust_env=True http2=False
DEBUG:httpx:load_verify_locations cafile='/Users/nivedhari/agents/.venv/lib/python3.13/site-packages/certifi/cacert.pem'
DEBUG:openai._base_client:Request options: {'method': 'post', 'url': '/chat/completions', 'files': None, 'json_data': {'messages': [{'role': 'user', 'content': 'Describe the images as an alternative text'}], 'model': 'gpt-4o', 'stream': False, 'temperature': 0.1}}
DEBUG:openai._base_client:Sending HTTP Request: POST https://api.openai.com/v1/chat/completions
DEBUG:httpcore.connection:connect_tcp.started host='api.openai.com' port=443 local_address=None timeout=60.0 socket_options=None
DEBUG:httpcore.connection:connect_tcp.complete return_value=<httpcore._backends.sync.SyncStream object at 0x105ac8ec0>
DEBUG:httpcore.connection:start_tls.started ssl_context=<ssl.SSLContext object at 0x1058a9250> server_hostname='api.openai.com' timeout=60.0
DEBUG:httpcore.connection:start_tls.complete return_value=<httpcore._backends.sync.SyncStream object at 0x10588a0d0>
DEBUG:httpcore.http11:send_request_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_headers.complete
DEBUG:httpcore.http11:send_request_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:send_request_body.complete
DEBUG:httpcore.http11:receive_response_headers.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_headers.complete return_value=(b'HTTP/1.1', 200, b'OK', [(b'Date', b'Tue, 21 Jan 2025 12:06:50 GMT'), (b'Content-Type', b'application/json'), (b'Transfer-Encoding', b'chunked'), (b'Connection', b'keep-alive'), (b'access-control-expose-headers', b'X-Request-ID'), (b'openai-organization', b'codemancers'), (b'openai-processing-ms', b'1171'), (b'openai-version', b'2020-10-01'), (b'x-ratelimit-limit-requests', b'5000'), (b'x-ratelimit-limit-tokens', b'800000'), (b'x-ratelimit-remaining-requests', b'4999'), (b'x-ratelimit-remaining-tokens', b'799971'), (b'x-ratelimit-reset-requests', b'12ms'), (b'x-ratelimit-reset-tokens', b'2ms'), (b'x-request-id', b'req_54e43647fa6faec6db5180e22e9b5fa1'), (b'strict-transport-security', b'max-age=31536000; includeSubDomains; preload'), (b'CF-Cache-Status', b'DYNAMIC'), (b'Set-Cookie', b'__cf_bm=lkKDQbrqeNo11O8om3n2FwoWM9xNNhfnifOXB3OTK3U-1737461210-1.0.1.1-U3_FTZ8pBvCwDpNRUXAY9S3HCMcPR3qs3N40VRGax5bzxBf2pNyYR.YcFMTUwc2mViwtZG3XanFsjh6ksaiKRg; path=/; expires=Tue, 21-Jan-25 12:36:50 GMT; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'X-Content-Type-Options', b'nosniff'), (b'Set-Cookie', b'_cfuvid=FHPHA3Fy5dW7VH.JoAD61XbhpouiDd42KESwKk8YiBA-1737461210162-0.0.1.1-604800000; path=/; domain=.api.openai.com; HttpOnly; Secure; SameSite=None'), (b'Server', b'cloudflare'), (b'CF-RAY', b'90572e1f4dd77fe6-MAA'), (b'Content-Encoding', b'gzip'), (b'alt-svc', b'h3=":443"; ma=86400')])
INFO:httpx:HTTP Request: POST https://api.openai.com/v1/chat/completions "HTTP/1.1 200 OK"
DEBUG:httpcore.http11:receive_response_body.started request=<Request [b'POST']>
DEBUG:httpcore.http11:receive_response_body.complete
DEBUG:httpcore.http11:response_closed.started
DEBUG:httpcore.http11:response_closed.complete

[dosubot]

Hey @n1vedhar1! I'm here to help you with any bugs, questions, or contributions you have for LlamaIndex. Let's squash those bugs together!

The issue you're experiencing with the ChatMessage class not recognizing ImageBlock in LlamaIndex 0.12.11 could be related to how the ChatMessage processes its content. In version 0.12.11, the ChatMessage class is designed to handle different block types, such as TextBlock and ImageBlock, by storing them in a list called blocks. When a ChatMessage is created, it should correctly identify and process both TextBlock and ImageBlock instances ^[1][2].

If the ImageBlock is not being recognized, it might be due to a bug or a change in how the ChatMessage class processes these blocks in version 0.12.11. You might want to verify that the ImageBlock is correctly initialized and added to the blocks list. Additionally, ensure that the OpenAI LLM is configured to handle multimodal content, as it interfaces with ChatMessage through functions that convert the message into a format suitable for OpenAI's API ^[3][4].

If the problem persists, consider checking for any updates or patches to LlamaIndex that might address this issue, or review the implementation details to ensure compatibility with the current version.

To continue talking to Dosu, mention @dosu.

---

^{Help Dosu learn! Give it feedback: Great Response | Irrelevant Answer | Incorrect Sources | Too Verbose | Hallucination | Other}

[rail-gpt]

Hi, I'm facing the same issue. Tried with llama-index 0.12.5 and 0.12.12, same issue my image block isn't recognized. I'm using python 3.13