Pass s3:// file URLs directly to API in BedrockConverseModel
#3663
Conversation
DouweM
changed the title
s3:// file URLs directly to API in BedrockConverseModel
| if item.url.startswith('s3://'): | ||
| source = {'s3Location': {'uri': item.url}} | ||
| else: | ||
| downloaded_item = await download_item(item, data_format='bytes', type_format='extension') |
There was a problem hiding this comment.
download_item currently has logic gating for gs:// URLs; let's check s3:// URLs there as well
There was a problem hiding this comment.
It seems the existing code in download_item checks for gs:// and youtube URLs:
if item.url.startswith('gs://'):
raise UserError('Downloading from protocol "gs://" is not supported.')
elif isinstance(item, VideoUrl) and item.is_youtube:
raise UserError('Downloading YouTube videos is not supported.')
What check do you mean for s3:// here?
There was a problem hiding this comment.
Same check raising an error saying that download_item does not support s3:// URLs
There was a problem hiding this comment.
Ah ok need to stop supporting download altogether. Updated.
Kept the check pretty simple. Not sure whether we should go for a proper url parsing here since the expectation is just bucketOwner param.
There was a problem hiding this comment.
If it's not drastically more code, I'd prefer proper URL parsing
There was a problem hiding this comment.
Not drastically more code. Used Python's urlib.
| format = item.media_type.split('/')[1] | ||
| assert format in ('jpeg', 'png', 'gif', 'webp'), f'Unsupported image format: {format}' | ||
| image: ImageBlockTypeDef = {'format': format, 'source': {'bytes': downloaded_item['data']}} | ||
| image: ImageBlockTypeDef = {'format': format, 'source': cast(DocumentSourceTypeDef, source)} |
There was a problem hiding this comment.
We shouldn't need to cast if hint the type of source to be source: DocumentSourceTypeDef
There was a problem hiding this comment.
Good catch, I only replaced any with the type previously 🤦
Updated to use proper hints.
4 participants
Closes #3621