Events lost when sending them to Application Insights using azure-monitor-opentelemetry==1.8.3

[leireroman12]

Last week, we migrated to azure-monitor-opentelemetry==1.8.3.
In our project, we use it to connect to Application Insights from a Databricks job. We also use azure-monitor-events-extension==0.1.0 to send events to customEvents table.
We install these two libraries using an init script in the job cluster.

This is the code we use to send events to Application Insights.

from azure.monitor.events.extension import track_event
from azure.monitor.opentelemetry import configure_azure_monitor
from opentelemetry._logs import get_logger_provider

def create_azure_connection():
    connection_string = dbutils.secrets.get('nonprod-keyvault-scope', 'application-insights-connection-string')
    configure_azure_monitor(connection_string=connection_string)

def send_custom_event(event_name, message_dict):
    create_azure_connection()
    track_event(event_name, message_dict)
    get_logger_provider().force_flush()
    print(f"Event '{event_name}' tracked")

We send one event at the start of the Databricks job and another at the end.

send_custom_event('Start Cloud ETL Process', attributes)
send_custom_event('Finish Cloud ETL Process', attributes)

We have several ETLs running, and there are events that have not arrived to Application Insights, even though the send_custom_event() function has been executed.

In one of the ETLs, we have received all the Start events but not all the Finish events, despite seeing the print ‘Event “Finish Cloud ETL Process” tracked’. In another ETL, we received the Finish message but not the Start message. So there is no clear pattern as to when events are ‘lost’.

This is the result of running pip freeze from a cluster that has this init script:

annotated-types==0.7.0
asgiref==3.11.0
asttokens==2.0.5
astunparse==1.6.3
azure-core==1.31.0
azure-core-tracing-opentelemetry==1.0.0b12
azure-identity==1.25.1
azure-monitor-events-extension==0.1.0
azure-monitor-opentelemetry==1.8.3
azure-monitor-opentelemetry-exporter==1.0.0b46
azure-storage-blob==12.23.0
azure-storage-file-datalake==12.17.0
black==24.4.2
blinker==1.7.0
boto3==1.34.69
botocore==1.34.69
cachetools==5.3.3
certifi==2024.6.2
cffi==1.16.0
chardet==4.0.0
charset-normalizer==2.0.4
click==8.1.7
cloudpickle==2.2.1
comm==0.2.1
contourpy==1.2.0
cryptography==42.0.5
cycler==0.11.0
Cython==3.0.11
databricks-sdk==0.30.0
dbus-python==1.3.2
debugpy==1.6.7
decorator==5.1.1
Deprecated==1.2.14
distlib==0.3.8
distro==1.9.0
distro-info==1.7+build1
docstring-to-markdown==0.11
executing==0.8.3
facets-overview==1.1.1
filelock==3.15.4
findspark==2.0.1
fonttools==4.51.0
gitdb==4.0.11
GitPython==3.1.37
google-api-core==2.20.0
google-auth==2.35.0
google-cloud-core==2.4.1
google-cloud-storage==2.18.2
google-crc32c==1.6.0
google-resumable-media==2.7.2
googleapis-common-protos==1.65.0
grpcio==1.60.0
grpcio-status==1.60.0
httplib2==0.20.4
idna==3.7
importlib-metadata==6.0.0
ipyflow-core==0.0.201
ipykernel==6.28.0
ipython==8.25.0
ipython-genutils==0.2.0
ipywidgets @ file:///databricks/.virtualenv-def/ipywidgets-7.7.2-2databricksnojsdeps-py2.py3-none-any.whl#sha256=903ead20c8d40de671853515fcad2f34b43ebf3eff80e4df3f876b8dd64c903b
isodate==0.6.1
jedi==0.19.1
jmespath==1.0.1
joblib==1.4.2
jupyter_client==8.6.0
jupyter_core==5.7.2
kiwisolver==1.4.4
launchpadlib==1.11.0
lazr.restfulclient==0.14.6
lazr.uri==1.0.6
matplotlib==3.8.4
matplotlib-inline==0.1.6
mccabe==0.7.0
mlflow-skinny==2.19.0
msal==1.34.0
msal-extensions==1.3.1
msrest==0.7.1
mypy==1.10.0
mypy-extensions==1.0.0
nest-asyncio==1.6.0
nodeenv==1.9.1
numpy==1.26.4
oauthlib==3.2.2
opentelemetry-api==1.39.0
opentelemetry-instrumentation==0.60b0
opentelemetry-instrumentation-asgi==0.60b0
opentelemetry-instrumentation-dbapi==0.60b0
opentelemetry-instrumentation-django==0.60b0
opentelemetry-instrumentation-fastapi==0.60b0
opentelemetry-instrumentation-flask==0.60b0
opentelemetry-instrumentation-psycopg2==0.60b0
opentelemetry-instrumentation-requests==0.60b0
opentelemetry-instrumentation-urllib==0.60b0
opentelemetry-instrumentation-urllib3==0.60b0
opentelemetry-instrumentation-wsgi==0.60b0
opentelemetry-resource-detector-azure==0.1.5
opentelemetry-sdk==1.39.0
opentelemetry-semantic-conventions==0.60b0
opentelemetry-util-http==0.60b0
packaging==24.1
pandas==1.5.3
parso==0.8.3
pathspec==0.10.3
patsy==0.5.6
pexpect==4.8.0
pillow==10.3.0
platformdirs==3.10.0
plotly==5.22.0
pluggy==1.0.0
prompt-toolkit==3.0.43
proto-plus==1.24.0
protobuf==4.24.1
psutil==5.9.0
psycopg2==2.9.3
ptyprocess==0.7.0
pure-eval==0.2.2
pyarrow==15.0.2
pyasn1==0.4.8
pyasn1-modules==0.2.8
pyccolo==0.0.65
pycparser==2.21
pydantic==2.8.2
pydantic_core==2.20.1
pyflakes==3.2.0
Pygments==2.15.1
PyGObject==3.48.2
PyJWT==2.7.0
pyodbc==5.0.1
pyparsing==3.0.9
pyright==1.1.294
python-apt==2.7.7+ubuntu5
python-dateutil==2.9.0.post0
python-lsp-jsonrpc==1.1.2
python-lsp-server==1.10.0
pytoolconfig==1.2.6
pytz==2024.1
PyYAML==6.0.1
pyzmq==25.1.2
requests==2.32.2
requests-oauthlib==2.0.0
rope==1.12.0
rsa==4.9
s3transfer==0.10.2
scikit-learn==1.4.2
scipy==1.13.1
seaborn==0.13.2
setuptools==74.0.0
six==1.16.0
smmap==5.0.0
sqlparse==0.5.1
ssh-import-id==5.11
stack-data==0.2.0
statsmodels==0.14.2
tenacity==8.2.2
threadpoolctl==2.2.0
tokenize-rt==4.2.1
tomli==2.0.1
tornado==6.4.1
traitlets==5.14.3
types-protobuf==3.20.3
types-psutil==5.9.0
types-pytz==2023.3.1.1
types-PyYAML==6.0.0
types-requests==2.31.0.0
types-setuptools==68.0.0.0
types-six==1.16.0
types-urllib3==1.26.25.14
typing_extensions==4.11.0
ujson==5.10.0
unattended-upgrades==0.1
urllib3==1.26.16
virtualenv==20.26.2
wadllib==1.3.6
wcwidth==0.2.5
whatthepatch==1.0.2
wheel==0.43.0
wrapt==1.14.1
yapf==0.33.0
zipp==3.17.0

Do you know if it's because of the new version of azure-monitor-opentelemetry? Is anyone else experiencing this?
Before migrating from version 1.8.1 to 1.8.3 of azure-monitor-opentelemetry, we were receiving all events correctly.

[rads-1996]

@leireroman12 Could you please address the following questions -

1. Which version of azure-monitor-opentelemetry, azure-monitor-opentelemetry-exporter were you using before you upgraded to the latest version?
2. Did you update any other packages as well?
3. Have you set any environment variables?
4. For these ETLs, do you see any exceptions or error messages in the traces?
5. When you received the print message - ‘Event “Finish Cloud ETL Process” tracked’, you did not see the logs for those in application insights, correct? Similarly, for the Start Cloud event, did you observe the same behavior, i.e. seeing the print message but not the logs in AI?
6. Do you mind sharing the traces from the transaction search for any of the ETL runs?
7. After you upgraded to the latest version of azure-monitor-opentelemetry, did you restart your application/process?

[github-actions]

Hi @leireroman12. Thank you for opening this issue and giving us the opportunity to assist. To help our team better understand your issue and the details of your scenario please provide a response to the question asked above or the information requested above. This will help us more accurately address your issue.

[leireroman12]

@leireroman12 Could you please address the following questions -

1. Which version of azure-monitor-opentelemetry, azure-monitor-opentelemetry-exporter were you using before you upgraded to the latest version?

We were using azure-monitor-opentelemetry==1.8.1, which used azure-monitor-opentelemetry-exporter==1.0.0b45 underneath.
This is a screenshot of the stdout from the Databricks cluster job where an ETL was run before the upgrade:

2. Did you update any other packages as well?

No, I just updated azure-monitor-opentelemetry

3. Have you set any environment variables?

No

4. For these ETLs, do you see any exceptions or error messages in the traces?

No

5. When you received the print message - ‘Event “Finish Cloud ETL Process” tracked’, you did not see the logs for those in application insights, correct? Similarly, for the Start Cloud event, did you observe the same behavior, i.e. seeing the print message but not the logs in AI?

In all ETLs, I see the two messages: “Event ‘Start Cloud ETL Process’ tracked” and “Event ‘Finish Cloud ETL Process’ tracked”. However, there are some events that have not reached Application Insights (despite having seen the message in Databricks, as I mentioned).

Today, for example, we received all events correctly in Application Insights. We don't know if it was a coincidence, because the only thing we changed was adding the Databricks notification for when the job finishes successfully, which has nothing to do with the OpenTelemetry library or sending events to AI.

6. Do you mind sharing the traces from the transaction search for any of the ETL runs?

Example of the 2 events of an ETL in customEvents table in AI (I guess this is what you are asking for):

"timestamp [UTC]",name,itemType,customDimensions,customMeasurements,"operation_Name","operation_Id","operation_ParentId","operation_SyntheticSource","session_Id","user_Id","user_AuthenticatedId","user_AccountId","application_Version","client_Type","client_Model","client_OS","client_IP","client_City","client_StateOrProvince","client_CountryOrRegion","client_Browser","cloud_RoleName","cloud_RoleInstance",appId,appName,iKey,sdkVersion,itemId,itemCount,"_ResourceId"
"12/11/2025, 4:30:12.255 AM","Start Cloud ETL Process",customEvent,"{""DWH"":""sqldb-ana-dev-neu-demo"",""DWH_ID"":""4d22214d-f83a-4ca7-9c82-c22f49ae9856"",""DWH_size"":""GP_S_Gen5_1"",""JOB_ID"":""469100572173435"",""RUN_ID"":""121606667000485"",""ETL_TEMPLATE"":""databricks-etl-demo.json"",""Environment"":""dev"",""code.file.path"":""/databricks/python/lib/python3.12/site-packages/azure/monitor/events/extension/_events.py"",""code.function.name"":""track_event"",""code.line.number"":""36"",""size_files_processed"":""0.00 KiB"",""files_processed"":""()"",""unexpected_files"":""()"",""list_plants_expected"":""()"",""list_plants_processed"":""{}"",""list_plants_not_processed"":""()"",""node_type_id"":""Standard_D4ads_v5"",""min_workers"":""0"",""max_workers"":""0""}",,,00000000000000000000000000000000,0000000000000000,,,,,,,PC,Other,Linux,"98.71.61.183",Dublin,Dublin,Ireland,Other,"unknown_service","4e37842b-732e-4699-8151-b1524a04a626","f082d345-b893-4d5b-87fa-24392fd8159a","/subscriptions/a003f0d1-aede-4610-9a71-f162b4a39f33/resourcegroups/rg-ana-nonprod/providers/microsoft.insights/components/appi-ana-nonprod-neu","60cf0e61-a9ec-4b80-8aae-c1a5832500a6","ulm_py3.12.3:otel1.39.0:ext1.0.0b46","2036079e-d64a-11f0-9c4c-6045bdde6a46",1,"/subscriptions/a003f0d1-aede-4610-9a71-f162b4a39f33/resourcegroups/rg-ana-nonprod/providers/microsoft.insights/components/appi-ana-nonprod-neu"
"12/11/2025, 4:39:50.465 AM","Finish Cloud ETL Process",customEvent,"{""DWH"":""sqldb-ana-dev-neu-demo"",""DWH_ID"":""4d22214d-f83a-4ca7-9c82-c22f49ae9856"",""ETL_duration"":""0:09:52.668904"",""DWH_size"":""GP_S_Gen5_1"",""JOB_ID"":""469100572173435"",""RUN_ID"":""121606667000485"",""ETL_TEMPLATE"":""databricks-etl-demo.json"",""Environment"":""dev"",""ETL_duration_seconds"":""592.668904"",""ETL_return_code"":""0"",""Database_connection_time"":""40.646049"",""update_dates"":""460.943"",""start_etl"":""30.573508"",""index_maintenance"":""35.551"",""update_statistics"":""57.843"",""end"":""10.188"",""init_cluster"":""285.472492"",""code.file.path"":""/databricks/python/lib/python3.12/site-packages/azure/monitor/events/extension/_events.py"",""code.function.name"":""track_event"",""code.line.number"":""36""}",,,00000000000000000000000000000000,0000000000000000,,,,,,,PC,Other,Linux,"98.71.61.183",Dublin,Dublin,Ireland,Other,"unknown_service","4e37842b-732e-4699-8151-b1524a04a626","f082d345-b893-4d5b-87fa-24392fd8159a","/subscriptions/a003f0d1-aede-4610-9a71-f162b4a39f33/resourcegroups/rg-ana-nonprod/providers/microsoft.insights/components/appi-ana-nonprod-neu","60cf0e61-a9ec-4b80-8aae-c1a5832500a6","ulm_py3.12.3:otel1.39.0:ext1.0.0b46","77aa2927-d64b-11f0-89f4-7c1e5275b5f5",1,"/subscriptions/a003f0d1-aede-4610-9a71-f162b4a39f33/resourcegroups/rg-ana-nonprod/providers/microsoft.insights/components/appi-ana-nonprod-neu"

7. After you upgraded to the latest version of azure-monitor-opentelemetry, did you restart your application/process?

ETLs run on a Databricks job cluster, so libraries are installed using pip in a new, clean environment for each run.