brick_simple:{add,set,replace} APIs do not return the server-side timstamp for success

[norton]

These APIs only return 'ok' on success but some applications want/need the server-side created timestamp value to be returned as well. To keep backwards compatibility, a new flag could be added to enable this feature.

[norton]

Prototype for this change request is on branch norton-server-timestamp.

[tatsuya6502]

Reviewed the prototype and merged into the dev branch:
hibari/gdss-brick#10 (comment)

[tatsuya6502]

nodetool-admin is also needed to be updated.

/home/tatsuya/workhub/dev/hibari/hibari% make check-package 
cleaning: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...

...

./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari-admin bootstrap
ok
checking package: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./tmp/hibari/bin/hibari-admin client-add hibari@127.0.0.1
RPC(add_client_monitor) to 'hibari@127.0.0.1' failed due to unknown response: {ok,
                                                                               1362803565835405}
make: *** [check-package] Error 1

[tatsuya6502]

Updated nodetool-admin. Commit: hibari/hibari@f2b5341

/home/tatsuya/workhub/dev/hibari/hibari% make check-package 

...

bootstrapping package: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
tar -C ./tmp -xzf ../hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64.tgz
./tmp/hibari/bin/hibari start
./tmp/hibari/bin/hibari-admin bootstrap
ok
checking package: hibari-0.3.0-dev-x86_64-unknown-linux-gnu-64 ...
./tmp/hibari/bin/hibari-admin client-add hibari@127.0.0.1
ok
./tmp/hibari/bin/hibari-admin client-list
['hibari@127.0.0.1']
./tmp/hibari/bin/hibari-admin client-delete hibari@127.0.0.1
ok
./tmp/hibari/bin/hibari checkpoint
ok
./tmp/hibari/bin/hibari stop
ok
/home/tatsuya/workhub/dev/hibari/hibari% 

[tatsuya6502]

I found that multi-node admin server can't be bootstrapped (hibari/gdss-admin#8).

brick_admin and brick_sb modules rely on brick_squorum (simple quorum) client, and the quorum scheme doesn't work with server side timestamp. In squorum, each bricks are configured as standalone role so they will assign different timespamp for a key-value and cause a quorum_error.

To avoid this, brick_squorum:set/6 and brick_squorum:setmulti/2 should explicitly assign timestamp at client (admin server) side.

[tatsuya6502]

To avoid this, brick_squorum:set/6 and brick_squorum:setmulti/2 should explicitly assign timestamp at client (admin server) side.

Updated but not tested. I'll try to run 3-node admin server config later tonight or early tomorrow.
hibari/gdss-admin@a8fc2ed

[tatsuya6502]

I found that QuickCheck no longer fails with quorum_error.

[tatsuya6502]

There is one place that brick_admin directly adds a key-value to the quorum nodes. I updated it to use the same client-side timestamp. (Perhaps this should utilize squorum:set/6, but I fixed in place for now.)

Commits: hibari/gdss-admin@1ec1391, hibari/gdss-admin@f7747e2

[tatsuya6502]

OK. I verified that it no longer fails to bootstrap 3-node admin server. Closing this issue.

[clus@hibaribench ~]$ grep ADMIN hibari.config 
ADMIN_NODES=(hibari1 hibari2 hibari3)
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh bootstrap hibari hibari.config
ok
hibari@hibari1 => hibari@hibari1 hibari@hibari2 hibari@hibari3 hibari@hibari4 hibari@hibari5 hibari@hibari6
[clus@hibaribench ~]$ ./clus/priv/clus-hibari.sh ping hibari hibari.config