csp naive datetime discussion

[robambalu]

This has come up a few times, so noting down discussion points here for further discussions in the future

CSP converts all datetimes to naive UTC datetime objects. Two motiviations for why we do this:

• all time can be stored in a nice and efficient 64-bit int. DateTime objects are stored and used throughout the c++ engine, the smaller and more efficient we can keep this the better
• timezone aware timestamps generally lead to more bugs and confusion than keeping everything consistent. So we can tackle both of these issues in one shot, by keeping everything as naive UTC timestamps.

Now there are some occasions where you do in fact want TZ aware datetimes (in general we should always advise against using TZ at all and always conform to UTC unless TZ is actually needed by the inputs/outputs or the internal graph computation). However, there are some cases where it is needed, and as such we have no way to support it right now. Some challenges:

• Keeping the python TZ object around with the DateTime, while the DateTime object itself would have to be passed through the c++ engine which is python agnostic
• Ensuring the overhead of the potentially rarely used TZ support wont impact the more common non-TZ support
• How would one either migrate the existing code to be tz-aware, or make it opt in/out (which may lead to even more confusion where some datetimes have TZ and some do not)

Some comments:

• For the first point we can potentially store the TZ object as another DialectGeneric type along with the DateTime. This DG TZ type would have an interface to access various TZ related operations that the c++ engine would need to process TZ info. Alternatively we can store the TZ as our own c++ TZ object and convert python TZ <-> out TZ object as needed. One challenge with this would be conserving the exact python tz object (ie if user passes pytz but our conversion back to python uses zoneinfo, thats not great)
• To avoid unnecessary overhead we can keep this to strictly opt in on data times by introducing a new type, ie csp.datetimeTz (so ts[datetimeTz] and datetimeTz as a struct member). Users who absolutely need Tz preserved can define their types as such
• We can discuss whether in general csp should be changed to always stamp UTC on its own timestamps (csp.now(), result arrays from runs, etc). This would be a major breaking change though, but maybe a good time to do it before the 1.0 release