Numpy Recarray Append_fields: Can't Append Numpy Array Of Datetimes

July 26, 2023 Post a Comment

I have a recarray containing various fields and I want to append an array of datetime objects on to it. However, it seems like the append_fields function in numpy.lib.recfunctions

Solution 1:

The problem

In [143]: recfun.append_fields(obs,'test',np.array([None,[],1]))
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent calllast)
<ipython-input-143-5c3de23b09f7>in<module>()
----> 1 recfun.append_fields(obs,'test',np.array([None,[],1]))/usr/local/lib/python3.5/dist-packages/numpy/lib/recfunctions.py in append_fields(base, names, data, dtypes, fill_value, usemask, asrecarray)
    615     if dtypes isNone:
    616         data = [np.array(a, copy=False, subok=True) for a in data]
--> 617         data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)]618else:
    619         if not isinstance(dtypes, (tuple, list)):

/usr/local/lib/python3.5/dist-packages/numpy/lib/recfunctions.py in<listcomp>(.0)
    615     if dtypes isNone:
    616         data = [np.array(a, copy=False, subok=True) for a in data]
--> 617         data = [a.view([(name, a.dtype)]) for (name, a) in zip(names, data)]618else:
    619         if not isinstance(dtypes, (tuple, list)):

/usr/local/lib/python3.5/dist-packages/numpy/core/_internal.py in _view_is_safe(oldtype, newtype)
    363364     if newtype.hasobject or oldtype.hasobject:
--> 365         raise TypeError("Cannot change data-type for object array.")366return367 

TypeError: Cannot change data-type for object array.

So the problem is in this a.view([(name, a.dtype)]) expression. It tries to make a single field structured array from a. That works with dtypes like int and str, but fails with object. That failure is in the core view handling, so isn't likely to change.

In [148]: x=np.arange(3)

In [149]: x.view([('test', x.dtype)])
Out[149]: 
array([(0,), (1,), (2,)], 
      dtype=[('test', '<i4')])

In [150]: x=np.array(['one','two'])

In [151]: x.view([('test', x.dtype)])
Out[151]: 
array([('one',), ('two',)], 
      dtype=[('test', '<U3')])

In [152]: x=np.array([[1],[1,2]])

In [153]: x
Out[153]: array([[1], [1, 2]], dtype=object)

In [154]: x.view([('test', x.dtype)])
...
TypeError: Cannot change data-typefor object array.

The fact that recfunctions requires a separate load indicates that it is somewhat of a backwater, that isn't used a lot, and not under active development. I haven't examined the code in detail, but I suspect a fix would be a kludge.

A fix

Here's a way of adding a new field from scratch. It performs the same basic actions as append_fields:

Define a new dtype, using the obs and the new field name and dtype:

In [158]: obs.dtype.descr
Out[158]: [('WIND_WAVE_HGHT', '<f4'), ('WIND_WAVE_PERD', '<f4')]

In [159]: obs.dtype.descr+[('TEST',object)]
Out[159]: [('WIND_WAVE_HGHT', '<f4'), ('WIND_WAVE_PERD', '<f4'), ('TEST', object)]

In [160]: dt1  =np.dtype(obs.dtype.descr+[('TEST',object)])

Make an empty target array, and fill it by copying data by field name:

In [161]: newobs = np.empty(obs.shape, dtype=dt1)    
In [162]: for n in obs.dtype.names:
     ...:     newobs[n]=obs[n]

In [167]: dates
Out[167]: 
array([datetime.datetime(2001, 1, 1, 0, 0),
       datetime.datetime(2001, 1, 1, 0, 0),
       datetime.datetime(2001, 1, 1, 0, 0)], dtype=object)

In [168]: newobs['TEST']=dates

In [169]: newobs
Out[169]: 
array([( 0.1       ,  10., datetime.datetime(2001, 1, 1, 0, 0)),
       ( 0.2       ,  11., datetime.datetime(2001, 1, 1, 0, 0)),
       ( 0.30000001,  12., datetime.datetime(2001, 1, 1, 0, 0))], 
      dtype=[('WIND_WAVE_HGHT', '<f4'), ('WIND_WAVE_PERD', '<f4'), ('TEST', 'O')])

datetime64 alternative

With the native numpy datetimes, append works

In [179]: dates64 = dates.astype('datetime64[D]')

In [180]: recfun.append_fields(obs,'test',dates64,usemask=False)
Out[180]: 
array([( 0.1       ,  10., '2001-01-01'),
       ( 0.2       ,  11., '2001-01-01'), ( 0.30000001,  12., '2001-01-01')], 
      dtype=[('WIND_WAVE_HGHT', '<f4'), ('WIND_WAVE_PERD', '<f4'), ('test', '<M8[D]')])

append_fields has some bells-n-whistles that my version doesn't - fill values, masked arrays, recarray, etc.

structured dates array

I could create a structured array with the dates

In [197]: sdates = np.array([(i,) for i in dates],dtype=[('test',object)])
In [198]: sdates
Out[198]: 
array([(datetime.datetime(2001, 1, 1, 0, 0),),
       (datetime.datetime(2001, 1, 1, 0, 0),),
       (datetime.datetime(2001, 1, 1, 0, 0),)], 
      dtype=[('test', 'O')])