Databricks - Meridian #1405
Description
In Databricks after this command:
mmm= model.Meridian(
input_data = data_test, #your input data already created
model_spec = model_spec, #model spec as created
)
#running sample prior analysis - default 500
mmm.sample_prior(500)
I get the following message:
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/context.py:694: UserWarning: Consider setting custom ROI priors, as kpi_type was specified as `non_revenue` with no `revenue_per_kpi` being set. Otherwise, the total media contribution prior will be used with `p_mean=0.4` and `p_sd=0.2`. Further documentation available at https://developers.google.com/meridian/docs/advanced-modeling/unknown-revenue-kpi-custom#set-total-paid-media-contribution-prior
warnings.warn(
WARNING: All log messages before absl::InitializeLog() is called are written to STDERR
I0000 00:00:1767869755.709581 10314 service.cc:152] XLA service 0x44085000 initialized for platform CUDA (this does not guarantee that XLA will be used). Devices:
I0000 00:00:1767869755.709697 10314 service.cc:160] StreamExecutor device (0): Tesla T4, Compute Capability 7.5
I0000 00:00:1767869755.722564 10314 cuda_dnn.cc:529] Loaded cuDNN version 90501
I0000 00:00:1767869755.880594 10314 device_compiler.h:188] Compiled cluster using XLA! This line is logged at most once for the lifetime of the process.
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/model.py:74: UserWarning: In a nationally aggregated model, the `media_effects_dist` will be reset to `normal`.
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/knots.py:505: RuntimeWarning: overflow encountered in cast
np.float32(math.comb(ncol, design_mat.shape[1]))
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. tau_g_excl_baseline has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_m has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_rf has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_om has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. eta_orf has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_c has been automatically set to Deterministic(0).
warnings.warn(
/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/prior_distribution.py:1265: UserWarning: Hierarchical distribution parameters must be deterministically zero for national models. xi_n has been automatically set to Deterministic(0).
warnings.warn(
And the after I run this command
mmm.sample_posterior(
n_chains = 10, #number of MCMC chains
n_adapt=1000, #NUmber of adaptation (warm up) chains - these are discarded - 1000->2000 sensible range
n_burnin=500, #Number of burn in iterations - additional discard - 500-> sensible range
n_keep=1000, #Number of samples kept - 1000+ per chain sensible
seed=0 #can enter seed to ensure chains start fro same place for reproducibility. Vary to test robustness
)
the notebook returns this message:
DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads:
DEBUG:ThreadMonitor:Logging Thread-16 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-8 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-7 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-4 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 2323, in run
readable, writable, errored = select.select(
DEBUG:ThreadMonitor:Logging MainThread stack frames:
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 52, in <module>
main()
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 48, in main
DatabricksKernelApp.launch_instance(config=databricks_kernel_config())
File "/databricks/python/lib/python3.12/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/databricks/python/lib/python3.12/site-packages/tornado/platform/asyncio.py", line 205, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
self._run_once()
File "/usr/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once
handle._run()
File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue
await self.process_one()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 534, in process_one
await dispatch(*args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell
await result
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 362, in execute_request
await super().execute_request(stream, ident, parent)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 778, in execute_request
reply_content = await reply_content
File "/databricks/python_shell/lib/dbruntime/kernel.py", line 534, in do_execute
reply_content = await super().do_execute(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 449, in do_execute
res = shell.run_cell(
File "/databricks/python/lib/python3.12/site-packages/ipykernel/zmqshell.py", line 549, in run_cell
return super().run_cell(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell
result = self._run_cell(
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell
result = runner(coro)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 128, in _pseudo_sync_runner
coro.send(None)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/root/.ipykernel/10314/command-2103895953871510-648632545", line 1, in <module>
mmm.sample_posterior(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/model.py", line 1013, in sample_posterior
self.posterior_sampler_callable(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/posterior_sampler.py", line 571, in __call__
mcmc = backend.xla_windowed_adaptive_nuts(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 833, in __call__
result = self._call(*args, **kwds)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 889, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 696, in _initialize
self._concrete_variable_creation_fn = tracing_compilation.trace_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 178, in trace_function
concrete_function = _maybe_define_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 283, in _maybe_define_function
concrete_function = _create_concrete_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 310, in _create_concrete_function
traced_func_graph = func_graph_module.func_graph_from_py_func(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/func_graph.py", line 1060, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 599, in wrapped_fn
out = weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/backend/__init__.py", line 1058, in _tf_xla_windowed_adaptive_nuts
return experimental.mcmc.windowed_adaptive_nuts(**kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/mcmc/windowed_sampling.py", line 715, in windowed_adaptive_nuts
return _windowed_adaptive_impl(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/mcmc/windowed_sampling.py", line 900, in _windowed_adaptive_impl
step_broadcast, batch_shape, shard_axis_names) = _setup_mcmc(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/mcmc/windowed_sampling.py", line 230, in _setup_mcmc
init_position = initialization.retry_init(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/mcmc/initialization.py", line 285, in retry_init
values, successes, _ = brs.batched_las_vegas_algorithm(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/internal/batched_rejection_sampler.py", line 78, in batched_las_vegas_algorithm
values, good_values_mask = batched_las_vegas_trial_fn(init_seed)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/mcmc/initialization.py", line 280, in trial
values = proposal_fn(*args, seed=seed, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/distributions/distribution.py", line 1205, in sample
return self._call_sample_n(sample_shape, seed, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/distributions/transformed_distribution.py", line 338, in _call_sample_n
return self.bijector.forward(x, **bijector_kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/bijectors/bijector.py", line 1327, in forward
return self._call_forward(x, name, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/bijectors/bijector.py", line 1309, in _call_forward
return self._cache.forward(x, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/internal/cache_util.py", line 334, in forward
return self._lookup(x, self._forward_name, self._inverse_name, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/internal/cache_util.py", line 493, in _lookup
self._invoke(input, forward_name, kwargs, attrs))
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/internal/cache_util.py", line 532, in _invoke
return getattr(self.bijector, fn_name)(input, **kwargs)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/experimental/distributions/joint_distribution_pinned.py", line 841, in <lambda>
return lambda *args: vectorized_fn_of_pins(self._jd.pins, *args)
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/tensorflow_probability/python/internal/vectorization_util.py", line 327, in vectorized_fn
batched_result = tf.vectorized_map(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 578, in vectorized_map
return pfor(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 228, in pfor
outputs = f()
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 833, in __call__
result = self._call(*args, **kwds)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 889, in _call
self._initialize(args, kwds, add_initializers_to=initializers)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 696, in _initialize
self._concrete_variable_creation_fn = tracing_compilation.trace_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 178, in trace_function
concrete_function = _maybe_define_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 283, in _maybe_define_function
concrete_function = _create_concrete_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/tracing_compilation.py", line 310, in _create_concrete_function
traced_func_graph = func_graph_module.func_graph_from_py_func(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/func_graph.py", line 1060, in func_graph_from_py_func
func_outputs = python_func(*func_args, **func_kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 599, in wrapped_fn
out = weak_wrapped_fn().__wrapped__(*args, **kwds)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/autograph_util.py", line 41, in autograph_handler
return api.converted_call(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 439, in converted_call
result = converted_f(*effective_args, **kwargs)
File "/tmp/__autograph_generated_file_ev423ff.py", line 17, in tf__f
retval_ = ag__.converted_call(ag__.ld(_pfor_impl), (ag__.ld(loop_fn), ag__.ld(iters)), dict(fallback_to_while_loop=ag__.ld(fallback_to_while_loop), parallel_iterations=ag__.ld(parallel_iterations), warn=ag__.ld(warn)), fscope)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 377, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 459, in _call_unconverted
return f(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 309, in _pfor_impl
loop_fn_outputs = f(loop_var)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 690, in wrapper
return converted_call(f, args, kwargs, options=options)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 377, in converted_call
return _call_unconverted(f, args, kwargs, options)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/autograph/impl/api.py", line 459, in _call_unconverted
return f(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 559, in loop_fn
return fn(gathered_elems)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/nest.py", line 628, in map_structure
return nest_util.map_structure(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/nest_util.py", line 1065, in map_structure
return _tf_core_map_structure(func, *structure, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/nest_util.py", line 1103, in _tf_core_map_structure
return _tf_core_pack_sequence_as(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 558, in <lambda>
lambda x: _gather_from_tensor_or_composite(x, i), elems)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 448, in _gather_from_tensor_or_composite
return _broadcasting_gather(x, i)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/parallel_for/control_flow_ops.py", line 437, in _broadcasting_gather
return result
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 155, in error_handler
del filtered_tb
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/dispatch.py", line 1260, in op_dispatch_handler
return dispatch_target(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/deprecation.py", line 588, in new_func
return func(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/array_ops.py", line 4993, in gather
return gen_array_ops.gather_v2(params, indices, axis, name=name)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/gen_array_ops.py", line 4123, in gather_v2
return _result
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/op_def_library.py", line 776, in _apply_op_helper
with g.as_default(), ops.name_scope(name) as scope:
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/op_def_library.py", line 416, in _ExtractInputsAndAttrs
for input_arg in op_def.input_arg:
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/profiler/trace.py", line 183, in wrapped
return func(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/ops.py", line 736, in convert_to_tensor
return tensor_conversion_registry.convert(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/tensor_conversion_registry.py", line 259, in convert
return ret
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/constant_tensor_conversion.py", line 29, in _constant_tensor_conversion_function
return constant_op.constant(v, dtype=dtype, name=name)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/ops/weak_tensor_ops.py", line 142, in wrapper
return op(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/constant_op.py", line 276, in constant
return _constant_impl(value, dtype, shape, name, verify_shape=False,
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/constant_op.py", line 294, in _constant_impl
return const_tensor
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/ops.py", line 299, in _create_graph_constant
return const_tensor
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/func_graph.py", line 614, in _create_op_internal
return super()._create_op_internal( # pylint: disable=protected-access
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/ops.py", line 2716, in _create_op_internal
return ret
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/ops.py", line 1209, in from_node_def
return self
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 155, in error_handler
del filtered_tb
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/framework/ops.py", line 1069, in _create_c_op
return c_op
2026-01-08 10:57:27.093004: I tensorflow/compiler/mlir/tensorflow/utils/dump_mlir_util.cc:269] disabling MLIR crash reproducer, set env var `MLIR_CRASH_REPRODUCER_DIRECTORY` to enable.
2026-01-08 10:57:30.433829: W tensorflow/compiler/tf2xla/kernels/assert_op.cc:39] Ignoring Assert operator mcmc_retry_init/assert_equal_1/Assert/AssertGuard/Assert
DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads:
DEBUG:ThreadMonitor:Logging Thread-16 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-8 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-7 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-4 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 2323, in run
readable, writable, errored = select.select(
DEBUG:ThreadMonitor:Logging MainThread stack frames:
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 52, in <module>
main()
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 48, in main
DatabricksKernelApp.launch_instance(config=databricks_kernel_config())
File "/databricks/python/lib/python3.12/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/databricks/python/lib/python3.12/site-packages/tornado/platform/asyncio.py", line 205, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
self._run_once()
File "/usr/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once
handle._run()
File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue
await self.process_one()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 534, in process_one
await dispatch(*args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell
await result
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 362, in execute_request
await super().execute_request(stream, ident, parent)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 778, in execute_request
reply_content = await reply_content
File "/databricks/python_shell/lib/dbruntime/kernel.py", line 534, in do_execute
reply_content = await super().do_execute(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 449, in do_execute
res = shell.run_cell(
File "/databricks/python/lib/python3.12/site-packages/ipykernel/zmqshell.py", line 549, in run_cell
return super().run_cell(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell
result = self._run_cell(
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell
result = runner(coro)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 128, in _pseudo_sync_runner
coro.send(None)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/root/.ipykernel/10314/command-2103895953871510-648632545", line 1, in <module>
mmm.sample_posterior(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/model.py", line 1013, in sample_posterior
self.posterior_sampler_callable(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/posterior_sampler.py", line 571, in __call__
mcmc = backend.xla_windowed_adaptive_nuts(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 833, in __call__
result = self._call(*args, **kwds)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 919, in _call
return self._concrete_variable_creation_fn._call_flat( # pylint: disable=protected-access
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py", line 1322, in _call_flat
return self._inference_function.call_preflattened(args)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 216, in call_preflattened
flat_outputs = self.call_flat(*args)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 251, in call_flat
outputs = self._bound_context.call_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/context.py", line 1688, in call_function
outputs = execute.execute(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
DEBUG:ThreadMonitor:Logging python thread stack frames for MainThread and py4j threads:
DEBUG:ThreadMonitor:Logging Thread-16 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-8 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-7 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 766, in run_closure
_threading_Thread_run(self)
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 521, in run
self.wait_for_commands()
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/clientserver.py", line 593, in wait_for_commands
command = smart_decode(self.stream.readline())[:-1]
File "/usr/lib/python3.12/socket.py", line 707, in readinto
return self._sock.recv_into(b)
DEBUG:ThreadMonitor:Logging Thread-4 (run) stack frames:
File "/usr/lib/python3.12/threading.py", line 1030, in _bootstrap
self._bootstrap_inner()
File "/usr/lib/python3.12/threading.py", line 1073, in _bootstrap_inner
self.run()
File "/usr/lib/python3.12/threading.py", line 1010, in run
self._target(*self._args, **self._kwargs)
File "/databricks/spark/python/lib/py4j-0.10.9.9-src.zip/py4j/java_gateway.py", line 2323, in run
readable, writable, errored = select.select(
DEBUG:ThreadMonitor:Logging MainThread stack frames:
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 52, in <module>
main()
File "/databricks/python_shell/scripts/db_ipykernel_launcher.py", line 48, in main
DatabricksKernelApp.launch_instance(config=databricks_kernel_config())
File "/databricks/python/lib/python3.12/site-packages/traitlets/config/application.py", line 1075, in launch_instance
app.start()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelapp.py", line 739, in start
self.io_loop.start()
File "/databricks/python/lib/python3.12/site-packages/tornado/platform/asyncio.py", line 205, in start
self.asyncio_loop.run_forever()
File "/usr/lib/python3.12/asyncio/base_events.py", line 641, in run_forever
self._run_once()
File "/usr/lib/python3.12/asyncio/base_events.py", line 1987, in _run_once
handle._run()
File "/usr/lib/python3.12/asyncio/events.py", line 88, in _run
self._context.run(self._callback, *self._args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 545, in dispatch_queue
await self.process_one()
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 534, in process_one
await dispatch(*args)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 437, in dispatch_shell
await result
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 362, in execute_request
await super().execute_request(stream, ident, parent)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/kernelbase.py", line 778, in execute_request
reply_content = await reply_content
File "/databricks/python_shell/lib/dbruntime/kernel.py", line 534, in do_execute
reply_content = await super().do_execute(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/ipykernel/ipkernel.py", line 449, in do_execute
res = shell.run_cell(
File "/databricks/python/lib/python3.12/site-packages/ipykernel/zmqshell.py", line 549, in run_cell
return super().run_cell(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3075, in run_cell
result = self._run_cell(
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3130, in _run_cell
result = runner(coro)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/async_helpers.py", line 128, in _pseudo_sync_runner
coro.send(None)
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3334, in run_cell_async
has_raised = await self.run_ast_nodes(code_ast.body, cell_name,
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3517, in run_ast_nodes
if await self.run_code(code, result, async_=asy):
File "/databricks/python/lib/python3.12/site-packages/IPython/core/interactiveshell.py", line 3577, in run_code
exec(code_obj, self.user_global_ns, self.user_ns)
File "/root/.ipykernel/10314/command-2103895953871510-648632545", line 1, in <module>
mmm.sample_posterior(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/model.py", line 1013, in sample_posterior
self.posterior_sampler_callable(
File "/local_disk0/.ephemeral_nfs/envs/pythonEnv-c4b29fa2-3467-4161-9990-97166558da21/lib/python3.12/site-packages/meridian/model/posterior_sampler.py", line 571, in __call__
mcmc = backend.xla_windowed_adaptive_nuts(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/util/traceback_utils.py", line 150, in error_handler
return fn(*args, **kwargs)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 833, in __call__
result = self._call(*args, **kwds)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/polymorphic_function.py", line 919, in _call
return self._concrete_variable_creation_fn._call_flat( # pylint: disable=protected-access
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/concrete_function.py", line 1322, in _call_flat
return self._inference_function.call_preflattened(args)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 216, in call_preflattened
flat_outputs = self.call_flat(*args)
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/polymorphic_function/atomic_function.py", line 251, in call_flat
outputs = self._bound_context.call_function(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/context.py", line 1688, in call_function
outputs = execute.execute(
File "/databricks/python/lib/python3.12/site-packages/tensorflow/python/eager/execute.py", line 53, in quick_execute
tensors = pywrap_tfe.TFE_Py_Execute(ctx._handle, device_name, op_name,
The model performance is meaningless, I am wondering if there is an issue between Databricks and Meridian.
This is my machine set up:
