How Can I Avoid That Np.datetime64 Gets Auto Converted To Datetime When Adding It To A Numpy Array?

For the following example, the elements with dtype np.datetime64 get automatically converted to datetime.datetime when they get added to another numpy array. How can I avoid this

Solution 1:

Sometimes we have to make a 'blank' object array, and fill it piece by piece.

In [57]: d = np.empty((5,2), object)
In [58]: d
Out[58]: 
array([[None, None],
       [None, None],
       [None, None],
       [None, None],
       [None, None]], dtype=object)

We can fill it by columns, but the result is as with the concatenate (don't use np.append):

In [59]: d[:,0] = c.ravel()
In [60]: d[:,1] = a.ravel()
In [61]: d
Out[61]: 
array([[0, datetime.datetime(2018, 4, 1, 15, 30)],
       [1, datetime.datetime(2018, 4, 1, 15, 31)],
       [2, datetime.datetime(2018, 4, 1, 15, 32)],
       [3, datetime.datetime(2018, 4, 1, 15, 33)],
       [4, datetime.datetime(2018, 4, 1, 15, 34)]], dtype=object)

As with a.astype(object) it has 'unboxed' the dates.

But if I assign elements one by one:

In [62]: for i in range(5):
    ...:     d[i,1]=a[i,0]
    ...:     
In [63]: d
Out[63]: 
array([[0, numpy.datetime64('2018-04-01T15:30:00')],
       [1, numpy.datetime64('2018-04-01T15:31:00')],
       [2, numpy.datetime64('2018-04-01T15:32:00')],
       [3, numpy.datetime64('2018-04-01T15:33:00')],
       [4, numpy.datetime64('2018-04-01T15:34:00')]], dtype=object)

But what's the value of such an array?

I can add a timedelta to the original time array:

In [67]: a + np.array(10, 'timedelta64[m]')
Out[67]: 
array([['2018-04-01T15:40:00'],
       ['2018-04-01T15:41:00'],
       ['2018-04-01T15:42:00'],
       ['2018-04-01T15:43:00'],
       ['2018-04-01T15:44:00']], dtype='datetime64[s]')

but I can't do the same thing to the object array column:

In [68]: d[:,1] + np.array(10, 'timedelta64[m]')
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent calllast)
<ipython-input-68-f82827d3d355>in<module>()
----> 1 d[:,1] + np.array(10, 'timedelta64[m]')

TypeError: ufunc add cannot use operands with types dtype('O') and dtype('<m8[m]')

I have to explicitly iterate on the objects:

In [70]: for i in range(5):
    ...:     d[i,1] += np.array(i*10, 'timedelta64[m]')
    ...:     
In [71]: d
Out[71]: 
array([[0, numpy.datetime64('2018-04-01T15:30:00')],
       [1, numpy.datetime64('2018-04-01T15:41:00')],
       [2, numpy.datetime64('2018-04-01T15:52:00')],
       [3, numpy.datetime64('2018-04-01T16:03:00')],
       [4, numpy.datetime64('2018-04-01T16:14:00')]], dtype=object)

Solution 2:

Use a record array instead of dtype=object

Fix this by constructing an array that can properly handle columns with different types. The dead simplest way to do this is to make a record array, like so:

rarr = np.rec.fromarrays([a, c], names=('date', 'val'))

print(rarr)
# output#     rec.array([[('2018-04-01T15:30:00', 0)],#                [('2018-04-01T15:31:00', 1)],#                [('2018-04-01T15:32:00', 2)],#                [('2018-04-01T15:33:00', 3)],#                [('2018-04-01T15:34:00', 4)]],#               dtype=[('date', '<M8[s]'), ('val', '<i8')])

print(rarr.date)
# output#     array([['2018-04-01T15:30:00'],#            ['2018-04-01T15:31:00'],#            ['2018-04-01T15:32:00'],#            ['2018-04-01T15:33:00'],#            ['2018-04-01T15:34:00']], dtype='datetime64[s]')

As hpaulj points out, no matter what you do you can't add to (or otherwise easily manipulate) a datetime64 column in an array of dtype=object. However, this is easy to do with a record array:

print(rarr.date + np.array(10, 'timedelta64[m]'))
# output#     array([['2018-04-01T15:40:00'],#            ['2018-04-01T15:41:00'],#            ['2018-04-01T15:42:00'],#            ['2018-04-01T15:43:00'],#            ['2018-04-01T15:44:00']], dtype='datetime64[s]')