Enhance Pandas Query Engine Output Processor #16052

MatinKhajavi · 2024-09-17T04:40:42Z

Description

This PR enhances the default_output_processor function to temporarily adjust pandas display settings (max_colwidth, max_rows, max_columns) based on the provided output_kwargs.
Adding max_rows and max_columns ensures that the full DataFrame is displayed, helping the LLM to have the complete context and not a truncated version with ..., which can otherwise hinder accurate interpretation.
It also ensures the display options are restored to their original values (and not reset) after execution to prevent side effects on global settings. This change provides better control over the formatting of DataFrame outputs in various contexts.

Fixes # (issue)

New Package?

Did I fill in the tool.llamahub section in the pyproject.toml and provide a detailed README.md for my new integration or package?

Yes
No

Version Bump?

Did I bump the version in the pyproject.toml file of the package I am updating? (Except for the llama-index-core package)

Yes
No

Type of Change

Please delete options that are not relevant.

Bug fix (non-breaking change which fixes an issue)
New feature (non-breaking change which adds functionality)
Breaking change (fix or feature that would cause existing functionality to not work as expected)
This change requires a documentation update

How Has This Been Tested?

Please describe the tests that you ran to verify your changes. Provide instructions so we can reproduce. Please also list any relevant details for your test configuration

Added new unit/integration tests
Added new notebook (that tests end-to-end)
I stared at the code and made sure it makes sense
Changes are minimal and I ran the PandasQueryEngine with the new changes and it works.

Suggested Checklist:

I have performed a self-review of my own code
I have commented my code, particularly in hard-to-understand areas
I have made corresponding changes to the documentation
I have added Google Colab support for the newly added notebooks.
My changes generate no new warnings
I have added tests that prove my fix is effective or that my feature works
New and existing unit tests pass locally with my changes
I ran make format; make lint to appease the lint gods

masci

Thank you!

* v0.11.5 (run-llama#15834) * Box bug fix 202409 (run-llama#15836) * fix document matadata for box file * fix BoxSearchOptions class initialization * bump versions to 0.2.1 * Add oreilly course cookbooks (run-llama#15845) * correct course name (run-llama#15859) * clone llama-deploy docs during docs builds (run-llama#15794) * chore(deps-dev): bump cryptography from 43.0.0 to 43.0.1 in /llama-index-core (run-llama#15840) * Refresh Opensearch index after delete operation (run-llama#15854) * Update ImageReader file loading logic (run-llama#15848) * Minor bug in SQQE (run-llama#15861) * v0.11.6 (run-llama#15863) * remove widgets from recent notebooks (run-llama#15864) * Fix image document deserialization issue (run-llama#15857) * Bugfix: Remove warnings during data insertion for Hybrid Search due to duplicated columns in the schema (run-llama#15846) * Add TablestoreVectorStore. (run-llama#15657) * Removed unused `llama-index-llms-anthropic` dependency from Bedrock Converse (run-llama#15869) Removed unused `llama-index-llms-anthropic` dependency. Incremented to `0.3.0`. * Update bedrock.py (run-llama#15879) Fix minor issue in error message * Add tool calling example in NVIDIA notebook (run-llama#15842) * NVIDIA: Completion fixes (run-llama#15820) * Fix PGVectorStore with latest pydantic, update pydantic imports (run-llama#15886) * add type ignore for streaming agents (run-llama#15887) * [wip] force openai structured output (run-llama#15706) * feature: Make SentenceSplitter's secondary_chunking_regex optional (run-llama#15882) * Bug fix for KuzuPropertyGraphStore: Allow upserting relations even when chunks are absent (run-llama#15889) * Missing tablestore (run-llama#15890) * v0.11.7 (run-llama#15891) * nit: extend workflow tutorial (run-llama#15901) * Modify generate_qa_embedding_pairs to use LLM from Settings (run-llama#15904) * docs: document how to disable global timeout along with more docs (run-llama#15912) document how to disable global timeout along with more docs * fix: model_name property and pydantic v2 in Azure Inference package (run-llama#15907) * fix: model_name property and pydantic v2 * fix: tests * Add InternalServerError to retry decorator (run-llama#15921) * Add InternalServerError to retry decorator * Bump version * docs: fix broken link (run-llama#15924) * Adding vector store for Azure Cosmos DB NoSql (run-llama#14158) * Initial changes * initial changes fixed with example * Initial commit with all the code and test changes * fixing test cases * adding jupyter notebook * cleaning files * resolving comments * fix linting * more linting * add a basic readme --------- Co-authored-by: Aayush Kataria <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> * update docs for custom embeddings (run-llama#15929) * Add doc.id to Langchain format conversions (run-llama#15928) * Update RankLLM with new rerankers (run-llama#15892) * Feature Request run-llama#15810 :: Add DynamoDBChatStore (run-llama#15917) * Adding support for `MetadataFilters` to WordLift Vector Store (run-llama#15905) * Repair and route colab links to official repo (run-llama#15900) * feat: Add a retry policy to steps (run-llama#15757) * Fix error handling in sharepoint reader (run-llama#15868) * feat: add llama-index llms alibabacloud_aisearch integration (run-llama#15850) * Opensearch Serverless filtered query support using knn_score script (run-llama#15899) * Mistral AI LLM Integration fixes (run-llama#15906) * Fix RagCLI (run-llama#15931) * v0.11.8 (run-llama#15933) * docs: document retry policy (run-llama#15938) * docs: update AI21 docs and readme (run-llama#15937) * docs: update ai21 docs and readme with jamba 1.5 and tool calling * docs: update ai21 docs and readme with jamba 1.5 and tool calling * docs: update ai21 docs and readme with jamba 1.5 and tool calling * Retry retriable errors in neo4j integrations (run-llama#15915) * Add four alibabacloud-aisearch llama-index integrations: rerank, node_parser, readers, embeddings (run-llama#15934) * Differentiate sync and async calls in OpenSearchVectorClient (run-llama#15945) * feat(postgres): add support for engine parameters (run-llama#15951) * feat(postgres): add support for engine parameters - Introduced engine_params to support passing parameters to create_engine. - Updated create_engine and create_async_engine calls to include engine_params. - Initialized engine_params in the constructor. * style(lint): reformat for readability * refactor(postgres): rename engine_params to create_engine_kwargs * refactor(postgres): rename engine_params to create_engine_kwargs * chore: bump version to 0.2.3 * fix(postgres): rename engine_params to create_engine_kwargs * update falkordb client (run-llama#15940) * update falkordb client * bump version * update version * fix * fix: Error when parsing output if tool name contains non-English characters (run-llama#15956) * fix attribute error in PGVectorStore (run-llama#15961) * Catch nest_asyncio errors (run-llama#15975) * NUDGE (run-llama#15954) * Add support for o1 openai models (run-llama#15979) * force temp to 1.0 for o1 (run-llama#15983) * Update concepts.md - fix link for Structured Data Extraction page (run-llama#15982) * Fix the import path example for SimpleMongoReader (run-llama#15988) * Do not pass system prompt from fn calling runner to fn calling worker (run-llama#15986) * Add callback manager to retriever query engine from args (run-llama#15990) * v0.11.9 (run-llama#15992) * Fix: get all documents from Elasticsearch KVStore (run-llama#16006) * Fix Pydantic numeric validation in openai/base.py (run-llama#15993) * feat: add quip reader (run-llama#16000) * [docs/example] Human in loop workflow example (run-llama#16011) * start choose own adventure hitl nb * hitl example * note on alternative implementation * add module guides and run prepare_for_build * fix: removed author info from quip reader (run-llama#16012) fix: removed author info * Fix Pydantic models definition (run-llama#16008) * chore(deps): bump litellm from 1.43.7 to 1.44.8 in /llama-index-integrations/embeddings/llama-index-embeddings-litellm (run-llama#16013) * Hotfix: Fix Citations Text (run-llama#16015) * Jacques/opik integration (run-llama#16007) * update llamacloud index with image nodes (run-llama#15996) * chore: add o1 models pinned versions (run-llama#16025) * add sparse embedding abstraction (run-llama#16018) * Implement `get_nodes` on `PGVectorStore` (run-llama#16026) * Implement `get_nodes` on `PGVectorStore` * Bump version num in pyproject.toml * Update docstring in base.py in MilvusVectorStore adding COSINE as available similarity metric (run-llama#16031) Update docstring in base.py adding COSINE as similarity metric available According to Milvus Documentation, **COSINE** as similarity metric is supported (Both Milvus and Milvus Lite) but in Llama-Index docs was missing. [Link to Milvus official docs](https://milvus.io/docs/metric.md?tab=floating#Similarity-Metrics) I've checked in the [code](https://github.com/run-llama/llama_index/blob/723c2533ed4b7b43b7d814c89af1838f0f1994c2/llama-index-integrations/vector_stores/llama-index-vector-stores-milvus/llama_index/vector_stores/milvus/base.py#L256) , so indeed COSINE is supported, so no more further changes are needed. * Fix: unnecessary warning issue in HuggingFace LLM when tokenizer is provided as argument(run-llama#16035) (run-llama#16037) * Attempt #3 of context/result refactor (run-llama#16036) * temporarily limit lancedb version (run-llama#16045) * fix: new Data Connector adaption for DashVector (run-llama#16028) * v0.11.10 (run-llama#16046) * Fix serde issue for huggingface inference API embedding (run-llama#16053) * wip * wip * fix: fix regression in OctoAI llm provider after 0.11 (run-llama#16002) * fix OctoAI llm provider * adjust code to the latest client * resolve conflicts * fix message format conversion * do not pass max_tokens if None * add chat test * bump version * update LanceDB integration (run-llama#16057) * Adding docstring inside preprocess base code. (run-llama#16060) * Add default to optional args for BedrockEmbedding (run-llama#16067) * wip * wip * Fix optional type for Cohere embedding (run-llama#16068) wip * fix vertex pydantic arguments (run-llama#16069) * Fix result order (run-llama#16078) * Fix result order * Bump version * fix elasticsearch embedding async function (run-llama#16083) * docs: add decorators to the api reference (run-llama#16081) * fix incorrect parameters in VertexAIIndex (run-llama#16080) * feat: update JinaEmbedding for v3 release (run-llama#15971) * Mistralai enable custom endpoint from env (run-llama#16084) * Implement async for multi modal ollama (run-llama#16091) * Remove circular package dep (run-llama#16070) * [chore] fix incorrect `handler.context` for `handler.ctx` in docs (run-llama#16101) fix * fix bug missing import (run-llama#16096) * Add more workflow references to docs (run-llama#16102) * fix cohere tests (run-llama#16104) * Use response synthesizer in context chat engines (run-llama#16017) * Fix mongodb hybrid search, also pass hybrid_top_k in vector retriever (run-llama#16105) * bump openai agent deps (run-llama#16112) * Fixed up test_vectorstore. (run-llama#16113) * Issue 16071: wordpress requires username, password (run-llama#16072) * Issue 16071: wordpress requires username, password * Adding changes suggested in PR template * Use Optional typing keyword * feat: add configurable base_url field in rerank (run-llama#16050) * add base_url * version bump * add default --------- Co-authored-by: Massimiliano Pippi <[email protected]> Co-authored-by: Logan <[email protected]> * Enhance Pandas Query Engine Output Processor (run-llama#16052) * Enhance output processor to temporarily adjust display options. * Make format changes. * Update pyproject.toml --------- Co-authored-by: Logan <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> * fix workflow docs (run-llama#16117) cr * Update OpenVINO LLM pyproject.toml (run-llama#16130) * Wordpress: Allow control of whether Pages and/or Posts are retrieved (run-llama#16128) * Async achat put operation (run-llama#16127) * [fix] handler.stream_events() doesn't yield `StopEvent` (run-llama#16115) * Improved TLM Rag cookbook (run-llama#16109) * Update chat message class for multi-modal (run-llama#15969) * Add support for Path for SimpleDirectoryReader (run-llama#16108) * Add TopicNodeParser based on MedGraphRAG paper (run-llama#16131) * Sql markdown response (run-llama#16103) * v0.11.11 (run-llama#16134) * Correct Pydantic warning(s) issed for llama-index-llms-ibm (run-llama#16141) Fix Pydantic warnings in llam-index-llms-ibm * feat: add drive link to google drive reader (run-llama#16156) * Introducing new VoyageAI models (run-llama#16150) * Add `required_exts` option to SharePoint reader (run-llama#16152) * User-defined schema in MilvusVectorStore (run-llama#16151) * safe format prompt variables in strings with JSON (run-llama#15734) * account for tools in prompt helper (run-llama#16157) * v0.11.12 (run-llama#16159) --------- Co-authored-by: Logan <[email protected]> Co-authored-by: Ravi Theja <[email protected]> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Yannic Spreen-Ledebur <[email protected]> Co-authored-by: Adnan Alkattan <[email protected]> Co-authored-by: Laurie Voss <[email protected]> Co-authored-by: saipjkai <[email protected]> Co-authored-by: ScriptShi <[email protected]> Co-authored-by: Bryce Freshcorn <[email protected]> Co-authored-by: Graham Tibbitts <[email protected]> Co-authored-by: Rashmi Pawar <[email protected]> Co-authored-by: Jerry Liu <[email protected]> Co-authored-by: Caroline Binley <[email protected]> Co-authored-by: Prashanth Rao <[email protected]> Co-authored-by: Asi Greenholts <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]> Co-authored-by: Facundo Santiago <[email protected]> Co-authored-by: gsa9989 <[email protected]> Co-authored-by: Aayush Kataria <[email protected]> Co-authored-by: Ryan Nguyen <[email protected]> Co-authored-by: David Riccitelli <[email protected]> Co-authored-by: GICodeWarrior <[email protected]> Co-authored-by: Javier Torres <[email protected]> Co-authored-by: 才胜 <[email protected]> Co-authored-by: George Dittmar <[email protected]> Co-authored-by: Tarun Jain <[email protected]> Co-authored-by: miri-bar <[email protected]> Co-authored-by: Tomaz Bratanic <[email protected]> Co-authored-by: kobiche <[email protected]> Co-authored-by: Arthur Moura Carvalho <[email protected]> Co-authored-by: Avi Avni <[email protected]> Co-authored-by: Cheese <[email protected]> Co-authored-by: Zac Wellmer <[email protected]> Co-authored-by: Ragul Kachiappan <[email protected]> Co-authored-by: Vinicius <[email protected]> Co-authored-by: Bidimpata-Kerim Aramyan-Tshimanga <[email protected]> Co-authored-by: Laura Ceconi <[email protected]> Co-authored-by: N1eo <[email protected]> Co-authored-by: Chirag Agrawal <[email protected]> Co-authored-by: Andrei Fajardo <[email protected]> Co-authored-by: David Oplatka <[email protected]> Co-authored-by: Jacques Verré <[email protected]> Co-authored-by: Harpinder <[email protected]> Co-authored-by: Richard Liu <[email protected]> Co-authored-by: Jorge Barrachina Gutiérrez <[email protected]> Co-authored-by: Sourabh <[email protected]> Co-authored-by: OceanPresent <[email protected]> Co-authored-by: Simon Suo <[email protected]> Co-authored-by: Ayush Chaurasia <[email protected]> Co-authored-by: preprocess-co <[email protected]> Co-authored-by: guodong <[email protected]> Co-authored-by: polarbear567 <[email protected]> Co-authored-by: Aaron Ji <[email protected]> Co-authored-by: enrico-stauss <[email protected]> Co-authored-by: Selim Çavaş <[email protected]> Co-authored-by: Casey Clements <[email protected]> Co-authored-by: Jonathan Springer <[email protected]> Co-authored-by: Anirudh31415926535 <[email protected]> Co-authored-by: Matin Khajavi <[email protected]> Co-authored-by: Ethan Yang <[email protected]> Co-authored-by: Matthew Turk <[email protected]> Co-authored-by: Andrew Kim <[email protected]> Co-authored-by: José Henrique Luckmann <[email protected]> Co-authored-by: Emanuel Ferreira <[email protected]> Co-authored-by: fzowl <[email protected]> Co-authored-by: João Martins <[email protected]> Co-authored-by: Petros Mitseas <[email protected]>

* Enhance output processor to temporarily adjust display options. * Make format changes. * Update pyproject.toml --------- Co-authored-by: Logan <[email protected]> Co-authored-by: Massimiliano Pippi <[email protected]>

MatinKhajavi added 2 commits September 16, 2024 21:24

Enhance output processor to temporarily adjust display options.

e9dc343

Make format changes.

9e3126e

dosubot bot added the size:S This PR changes 10-29 lines, ignoring generated files. label Sep 17, 2024

Merge branch 'main' into main

31cf2c3

masci self-assigned this Sep 18, 2024

logan-markewich and others added 3 commits September 19, 2024 10:10

Merge branch 'main' into main

7718fb1

Merge branch 'main' into main

05f0a7c

Update pyproject.toml

a5c2d11

masci approved these changes Sep 20, 2024

View reviewed changes

dosubot bot added the lgtm This PR has been approved by a maintainer label Sep 20, 2024

masci enabled auto-merge (squash) September 20, 2024 08:44

masci merged commit 2e99701 into run-llama:main Sep 20, 2024
10 checks passed

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Enhance Pandas Query Engine Output Processor #16052

Enhance Pandas Query Engine Output Processor #16052

MatinKhajavi commented Sep 17, 2024

masci left a comment

Enhance Pandas Query Engine Output Processor #16052

Enhance Pandas Query Engine Output Processor #16052

Conversation

MatinKhajavi commented Sep 17, 2024

Description

New Package?

Version Bump?

Type of Change

How Has This Been Tested?

Suggested Checklist:

masci left a comment

Choose a reason for hiding this comment