next encoder

[ruslandoga]

Trying out some ideas for a more efficient RowBinary encoder.

Idea 1: speed up encoding of DateTime.utc_now() (datetimes that are close to "now" by pre-generating unix offsets)

Idea 2: macro to generate single-call encoding function for an Ecto.Schema struct, e.g.

defmodule EventWriteBuffer do
  require Ch.RowBinary
  
  # https://github.com/plausible/analytics/blob/master/lib/plausible/clickhouse_event_v2.ex
  Ch.RowBinary.define_encoder(schema: Plausible.ClickhouseEventV2, name: :encode)
  # generates something like
  # def encode(%Plausible.ClickhouseEventV2{site_id: site_id, hostname: hostname, pathname: pathname, user_id: user_id, session_id: session_id, timestamp: timestamp, ...etc...}) do
  #   [<<site_id::64-unsigned>>, encode_string(hostname), encode_string(pathname), <<user_id::64-unsigned, session_id::64-unsigned, to_unix(timestamp)::16-unsigned>>, ...etc...]
  # end
  
  # instead of https://github.com/plausible/analytics/blob/1c5c4a25aae0a713c2452810a35e9bacedecd332/lib/plausible/event/write_buffer.ex#L25-L33
  def insert(event) do
    :ok = Plausible.Ingestion.WriteBuffer.insert(__MODULE__, encode(event))
    {:ok, event}
  end
end

[ruslandoga]

Current results from CI

Operating System: Linux
CPU Information: AMD EPYC 7763 64-Core Processor
Number of Available Cores: 4
Available memory: 15.61 GB
Elixir 1.18.0
Erlang 27.2
JIT enabled: true

Benchmark suite executing with the following configuration:
warmup: 2 s
time: 5 s
memory time: 0 ns
reduction time: 0 ns
parallel: 1
inputs: pageview
Estimated total run time: 14 s

Benchmarking current encoder with input pageview ...
Benchmarking next encoder with input pageview ...
Calculating statistics...
Formatting results...

##### With input pageview #####
Name                      ips        average  deviation         median         99th %
next encoder           3.96 M        0.25 μs ±15263.71%       0.190 μs        0.39 μs
current encoder        0.70 M        1.43 μs  ±1440.09%        1.32 μs        2.58 μs

Comparison: 
next encoder           3.96 M
current encoder        0.70 M - 5.65x slower +1.17 μs