compile error for mode reduction on QuadMeshCurvilinear

[dcherian]

ALL software version info

(this library, plus any other relevant software, e.g. bokeh, python, notebook, OS, browser, etc should be added within the dropdown below.)

Software Version Info

Datashader 0.18.1

Description of expected behavior and the observed behavior

The mode reduction works with QuadMeshRectilinear but fails for QuadMeshCurvilinear

Complete, minimal, self-contained example code that reproduces the issue

import numpy as np
import xarray as xr
import datashader as dsh
import datashader.transfer_functions as tf

ds = xr.Dataset(
    {"foo": (("x", "y"), np.arange(120).reshape(30, 4))},
    coords={"x": np.arange(30), "y": np.arange(4)},
).drop_indexes(("x", "y"))

# -----------> commenting out the bottom line fixes it
ds["x"], ds["y"] = xr.broadcast(ds.x, ds.y)

cvs = dsh.Canvas(x_range=(0, 3), y_range=(0, 4))
mesh = cvs.quadmesh(ds, x="x", y="y", agg=dsh.reductions.mode("foo"))
tf.shade(mesh)

Stack traceback and/or browser JavaScript console output

TypingError: Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Failed in nopython mode pipeline (step: nopython frontend)
Untyped global name '_2': Cannot determine Numba type of <class 'function'>

File "<string>", line 2:
<source missing, REPL[/exec](http://localhost:8888/exec) in use?>

During: Pass nopython_type_inference
During: resolving callee type: type(CPUDispatcher(<function append at 0x31afee480>))
During: typing of call at <perform_extend> (55)

During: resolving callee type: type(CPUDispatcher(<function append at 0x31afee480>))
During: typing of call at <perform_extend> (118)

During: resolving callee type: type(CPUDispatcher(<function append at 0x31afee480>))
During: typing of call at <perform_extend> (55)


File "<perform_extend>", line 55:
<source missing, REPL[/exec](http://localhost:8888/exec) in use?>

During: Pass nopython_type_inference
During: resolving callee type: type(CPUDispatcher(<function perform_extend at 0x31761bce0>))
During: typing of call at <extend_cpu> (26)

During: resolving callee type: type(CPUDispatcher(<function perform_extend at 0x31761bce0>))
During: typing of call at <extend_cpu> (26)


File "<extend_cpu>", line 26:
<source missing, REPL[/exec](http://localhost:8888/exec) in use?>

During: Pass nopython_type_inference

[hoxbro]

Just adding some quick comments.

Running the code with NUMBA_DISABLE_JIT=1 python example.py, we get the following traceback

Traceback (most recent call last):
  File "/home/shh/projects/holoviz/repos/datashader/example.py", line 16, in <module>
    mesh = cvs.quadmesh(ds, x="x", y="y", agg=dsh.reductions.mode("foo"))
  File "/home/shh/projects/holoviz/repos/datashader/datashader/core.py", line 885, in quadmesh
    return bypixel(source, self, glyph, agg)
  File "/home/shh/projects/holoviz/repos/datashader/datashader/core.py", line 1351, in bypixel
    return bypixel.pipeline(source, schema, canvas, glyph, agg, antialias=antialias)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/utils.py", line 118, in __call__
    return lk[typ](head, *rest, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/data_libraries/xarray.py", line 28, in xarray_pipeline
    return glyph_dispatch(
        glyph, xr_ds, schema, canvas, summary, antialias=antialias, cuda=cuda)
  File "/home/shh/projects/holoviz/repos/datashader/datashader/utils.py", line 121, in __call__
    return lk[cls](head, *rest, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/data_libraries/pandas.py", line 59, in default
    extend(bases, source, x_st + y_st, x_range + y_range)
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/glyphs/quadmesh.py", line 672, in extend
    do_extend(
    ~~~~~~~~~^
        plot_height, plot_width, xs, ys, *aggs_and_cols
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/glyphs/quadmesh.py", line 615, in extend_cpu
    perform_extend(
    ~~~~~~~~~~~~~~^
        i, j, plot_height, plot_width, xs, ys,
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
        xverts, yverts, yincreasing, eligible, intersect, *aggs_and_cols
        ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
    )
    ^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/glyphs/quadmesh.py", line 559, in perform_extend
    append(j, i, xi, yi, *aggs_and_cols)
    ~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "<string>", line 2, in append
TypeError: mode._append() takes 3 positional arguments but 4 were given

Commenting out the following lines. We get the following traceback:

datashader/datashader/glyphs/quadmesh.py

Lines 556 to 559 in b4d561a

	if intersections % 2 == 1:
	# If odd number of intersections, point
	# is inside quad
	append(j, i, xi, yi, *aggs_and_cols)

Traceback (most recent call last):
  File "/home/shh/projects/holoviz/repos/datashader/example.py", line 16, in <module>
    mesh = cvs.quadmesh(ds, x="x", y="y", agg=dsh.reductions.mode("foo"))
  File "/home/shh/projects/holoviz/repos/datashader/datashader/core.py", line 885, in quadmesh
    return bypixel(source, self, glyph, agg)
  File "/home/shh/projects/holoviz/repos/datashader/datashader/core.py", line 1351, in bypixel
    return bypixel.pipeline(source, schema, canvas, glyph, agg, antialias=antialias)
           ~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/utils.py", line 118, in __call__
    return lk[typ](head, *rest, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/data_libraries/xarray.py", line 28, in xarray_pipeline
    return glyph_dispatch(
        glyph, xr_ds, schema, canvas, summary, antialias=antialias, cuda=cuda)
  File "/home/shh/projects/holoviz/repos/datashader/datashader/utils.py", line 121, in __call__
    return lk[cls](head, *rest, **kwargs)
           ~~~~~~~^^^^^^^^^^^^^^^^^^^^^^^
  File "/home/shh/projects/holoviz/repos/datashader/datashader/data_libraries/pandas.py", line 61, in default
    return finalize(bases,
                    cuda=cuda,
    ...<2 lines>...
                    dims=[glyph.y_label, glyph.x_label],
                    attrs=dict(x_range=x_range, y_range=y_range))
  File "/home/shh/projects/holoviz/repos/datashader/datashader/reductions.py", line 1794, in _finalize
    raise NotImplementedError("mode is currently implemented only for rasters")
NotImplementedError: mode is currently implemented only for rasters

The mode reduction strongly suggests that _append is not supported for it. I haven't looked into whether it is the case for your MRE.

datashader/datashader/reductions.py

Lines 1764 to 1794 in b4d561a

	class mode(Reduction):
	"""Mode (most common value) of all the values encountered in ``column``.
	Useful for categorical data where an actual value must always be returned,
	not an average or other numerical calculation.
	Currently only supported for rasters, externally to this class.
	Implementing it for other glyph types would be difficult due to potentially
	unbounded data storage requirements to store indefinite point or line
	data per pixel.
	Parameters
	----------
	column : str
	Name of the column to aggregate over. If the data type is floating point,
	``NaN`` values in the column are skipped.
	"""
	def out_dshape(self, in_dshape, antialias, cuda, partitioned):
	return dshape(Option(ct.float64))
	@staticmethod
	def _append(x, y, agg):
	raise NotImplementedError("mode is currently implemented only for rasters")
	@staticmethod
	def _combine(aggs):
	raise NotImplementedError("mode is currently implemented only for rasters")
	@staticmethod
	def _finalize(bases, **kwargs):
	raise NotImplementedError("mode is currently implemented only for rasters")

[dcherian]

Currently only supported for rasters, externally to this class.

Ooops. I did read that but didn't process it fully. I'm OK with closing this as wontfix.

[hoxbro]

At a minimum, we should propagate the NotImplementedError.

[jbednar]

Thanks. I'd guess that mode could be implemented for QuadMesh the same as it was for Raster, but if it's not a priority, won't fix is fine.

[dcherian]

I'd like that but the code is inscrutable to me! :)

[jbednar]

Same! The implementation for rasters isn't too bad, in raster.py:_downsample_2d_mode, but the raster code is all quite separate from the other glyph types, even though a quadmesh could in principle have a similar implementation.