Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

feat(portability): add support for FreeBSD #1321

Merged
merged 81 commits into from
Aug 23, 2024
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
81 commits
Select commit Hold shift + click to select a range
b82939a
feat(portability): add support for FreeBSD
bitcoin-tools Aug 20, 2024
2b30d8a
Create Dockerfile_freebsd
bitcoin-tools Aug 20, 2024
c0064f2
Update validation.yaml
bitcoin-tools Aug 20, 2024
39c3bae
fix broken case syntax
bitcoin-tools Aug 20, 2024
d686ba9
fix case syntax
bitcoin-tools Aug 20, 2024
046506b
fix case syntax
bitcoin-tools Aug 20, 2024
a50cf0b
fix broken syntax
bitcoin-tools Aug 20, 2024
65b8a7a
Delete docker/Dockerfile_freebsd
bitcoin-tools Aug 20, 2024
52c0ade
Update validation.yaml
bitcoin-tools Aug 20, 2024
5a7bc12
Update nodebuilder
bitcoin-tools Aug 20, 2024
dfc937f
fix space
bitcoin-tools Aug 20, 2024
c09e95a
debug: use temporary branch name for deps
bitcoin-tools Aug 21, 2024
05e5cea
Merge branch 'master' into 251-add-freebsd-support
bitcoin-tools Aug 21, 2024
7e817ee
Create runtime_dependencies_freebsd.txt
bitcoin-tools Aug 21, 2024
af50a6b
Create build_dependencies_freebsd.txt
bitcoin-tools Aug 21, 2024
47e0ad3
fix typo
bitcoin-tools Aug 21, 2024
689450b
remove tar and procps runtime dependencies
bitcoin-tools Aug 21, 2024
586cda2
fix zmq package name
bitcoin-tools Aug 21, 2024
4c992fd
add qt5-widgets dependency
bitcoin-tools Aug 21, 2024
c88b1f1
Update nodebuilder
bitcoin-tools Aug 21, 2024
6105c75
Update nodebuilder
bitcoin-tools Aug 21, 2024
30ffc19
Update nodebuilder
bitcoin-tools Aug 21, 2024
0d16c0a
Update nodebuilder
bitcoin-tools Aug 21, 2024
3530062
add debug steps
bitcoin-tools Aug 21, 2024
4b1a15a
fix lint and add debug for sys cores
bitcoin-tools Aug 21, 2024
85d8e37
try another way to pass in jobs parameter
bitcoin-tools Aug 21, 2024
7188344
try without quotes
bitcoin-tools Aug 21, 2024
43a11b2
add debug step to redirect configure to file
bitcoin-tools Aug 21, 2024
45c658f
add qt5 build tools dep
bitcoin-tools Aug 21, 2024
8822e9f
Update build_dependencies_freebsd.txt
bitcoin-tools Aug 21, 2024
9e61be6
add gettext dep
bitcoin-tools Aug 21, 2024
861f369
fix FreeBSD compile bug by using gmake
bitcoin-tools Aug 21, 2024
890acc1
fix syntax
bitcoin-tools Aug 21, 2024
b60b57e
fix bug in FreeBSD make install
bitcoin-tools Aug 21, 2024
e61cff2
suppress stderr from make
bitcoin-tools Aug 21, 2024
016a670
feat(ci): add FreeBSD test on vm
bitcoin-tools Aug 21, 2024
d267c2d
add needs to freebsd
bitcoin-tools Aug 21, 2024
9682d34
remove --ref argument
bitcoin-tools Aug 21, 2024
e2b824c
fix freebsd df issue
bitcoin-tools Aug 21, 2024
9b985df
fix syntax
bitcoin-tools Aug 21, 2024
786c0a8
fix whitespace
bitcoin-tools Aug 21, 2024
49131d9
exclude freebsd from memory checks
bitcoin-tools Aug 22, 2024
6335e33
fix whitespace
bitcoin-tools Aug 22, 2024
a25942a
remove worthless comments
bitcoin-tools Aug 22, 2024
3c16c62
remove debug lines
bitcoin-tools Aug 22, 2024
6e920c8
set freebsd to download linux tarbal
bitcoin-tools Aug 22, 2024
f02df5f
don't disable sleep or suspend on freebsd
bitcoin-tools Aug 22, 2024
27f55c0
force compile on FreeBSD
bitcoin-tools Aug 22, 2024
0de3bf9
Update nodebuilder
bitcoin-tools Aug 22, 2024
0dd9756
Update validation.yaml
bitcoin-tools Aug 22, 2024
3c5a32c
fix date bug in FreeBSD
bitcoin-tools Aug 22, 2024
b3e6de5
get the environment stats, don't pre-install curl, and set no sync
bitcoin-tools Aug 22, 2024
b027085
remove no sync flag
bitcoin-tools Aug 22, 2024
0770347
capture the Bitcoin Core log file as an artifact
bitcoin-tools Aug 22, 2024
a843ab3
improve the pkg commands
bitcoin-tools Aug 22, 2024
9a7c97a
feat(ci): add console output CI testing
bitcoin-tools Aug 22, 2024
c7282b2
avoid git ref validation in FreeBSD
bitcoin-tools Aug 22, 2024
e70fdbd
add FreeBSD support to test_nodebuilder
bitcoin-tools Aug 22, 2024
c772b77
search for the bitcoin data directory
bitcoin-tools Aug 22, 2024
f34350c
fix console output tests for FreeBSD
bitcoin-tools Aug 22, 2024
d7a43a7
Merge branch 'master' into 251-add-freebsd-support
bitcoin-tools Aug 22, 2024
b76a5ab
Update test_nodebuilder
bitcoin-tools Aug 22, 2024
fad5fdd
fix syntax
bitcoin-tools Aug 22, 2024
bdd40ea
refactor: simplify the target kernel check
bitcoin-tools Aug 22, 2024
0e153de
fix find command
bitcoin-tools Aug 22, 2024
d7ebe8c
try to save the debug logs
bitcoin-tools Aug 22, 2024
c288f4f
fix syntax bug
bitcoin-tools Aug 22, 2024
0d5c8f7
search for where debug.log is
bitcoin-tools Aug 22, 2024
0b3f1a7
fix whitespace
bitcoin-tools Aug 22, 2024
8c7ffcc
fix typo
bitcoin-tools Aug 22, 2024
c6d10f5
fix path for upload artifact
bitcoin-tools Aug 22, 2024
b99a3d9
bitcoin-tools Aug 23, 2024
ab1732e
refactor code to improve readability
bitcoin-tools Aug 23, 2024
9a6c653
set freebsd dependencies url to master
bitcoin-tools Aug 23, 2024
3a35628
fix syntax
bitcoin-tools Aug 23, 2024
65f3b1f
fix syntax
bitcoin-tools Aug 23, 2024
dc5d46e
add case for test failure on
bitcoin-tools Aug 23, 2024
95ebd34
fix yaml issue
bitcoin-tools Aug 23, 2024
3d2de9b
capture debug logs on rerun too
bitcoin-tools Aug 23, 2024
5d09c29
remove the test failure debug run
bitcoin-tools Aug 23, 2024
c7c3bc4
Update validation.yaml
bitcoin-tools Aug 23, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
34 changes: 34 additions & 0 deletions .github/workflows/validation.yaml
Original file line number Diff line number Diff line change
Expand Up @@ -282,6 +282,40 @@ jobs:
if: steps.test-nodebuilder.outcome == 'failure'
run: printf '%s\n' "Review the step 'Test the console output' above." && exit 1

run-nodebuilder-freebsd:
name: Test source on FreeBSD
needs: [changes, shell-lint, yaml-lint]
if: >-
${{
github.event.action == 'prerelease' ||
github.event.action == 'published' ||
needs.changes.outputs.ci == 'true' ||
needs.changes.outputs.dependencies == 'true' ||
needs.changes.outputs.shell == 'true'
}}
runs-on: ubuntu-latest
steps:
- uses: actions/checkout@v4
- name: Test the console output
uses: vmactions/freebsd-vm@v1
timeout-minutes: 180
with:
prepare: |
uname -a
cat /etc/os-release
nproc
sysctl -a | grep ' memory' | head -2
df -h
date -u
run: |
/bin/sh -x ./test/test_nodebuilder --ref "${GITHUB_SHA}"
[ -f "${HOME}/.bitcoin/debug.log" ] && cp "${HOME}/.bitcoin/debug.log" .
- name: Save Bitcoin Core log as artifact
uses: actions/upload-artifact@v4
with:
name: freebsd-source-bitcoin-debug.log
path: /home/runner/work/nodebuilder/nodebuilder/debug.log

run-nodebuilder-docker:
name: Docker image for ${{ matrix.container }}
needs: [changes, dockerfile-lint, shell-lint, yaml-lint]
Expand Down
113 changes: 88 additions & 25 deletions nodebuilder
Original file line number Diff line number Diff line change
Expand Up @@ -66,19 +66,37 @@ compile_bitcoin_from_source() {
./autogen.sh > /dev/null 2> "${STDERR_COMPILE_LOG_FILE}"
grep -v 'build-aux' "${STDERR_COMPILE_LOG_FILE}" >&2 || true
log_info 'Configuring the build environment.'
if [ "${TARGET_KERNEL}" = 'Darwin' ]; then
case "${TARGET_KERNEL}" in
'Darwin')
./configure CC=clang CXX=clang++ --without-bdb --enable-suppress-external-warnings > /dev/null
else
;;
'FreeBSD')
./configure --without-bdb --enable-suppress-external-warnings MAKE=gmake > "${HOME}/configure_results"
;;
*)
./configure --without-bdb --enable-suppress-external-warnings > /dev/null
fi
;;
esac
log_info 'Compiling source code. Please wait.'
make --jobs "${SYS_CORES_PLUS_ONE}" > /dev/null 2>&1
if [ "${TARGET_KERNEL}" = 'FreeBSD' ]; then
gmake --jobs "${SYS_CORES_PLUS_ONE}" > /dev/null 2>&1
else
make --jobs "${SYS_CORES_PLUS_ONE}" > /dev/null 2>&1
fi
log_info 'Running compile checks. Please wait.'
make --jobs "${SYS_CORES_PLUS_ONE}" check > /dev/null 2> "${STDERR_COMPILE_LOG_FILE}"
if [ "${TARGET_KERNEL}" = 'FreeBSD' ]; then
gmake --jobs "${SYS_CORES_PLUS_ONE}" check > /dev/null 2> "${STDERR_COMPILE_LOG_FILE}"
else
make --jobs "${SYS_CORES_PLUS_ONE}" check > /dev/null 2> "${STDERR_COMPILE_LOG_FILE}"
fi
# exclude the two lines before and after 'Ran 3 tests in ' in make check's stdout
sed -n '1N;2N;/Ran 3 tests in /{N;N;d;};P;N;D' "${STDERR_COMPILE_LOG_FILE}" >&2
log_info 'Installing Bitcoin Core.'
make install > /dev/null 2>&1 || sudo make install > /dev/null
if [ "${TARGET_KERNEL}" = 'FreeBSD' ]; then
gmake install > /dev/null 2>&1 || sudo gmake install > /dev/null
else
make install > /dev/null 2>&1 || sudo make install > /dev/null
fi
cd - > /dev/null
rm "${STDERR_COMPILE_LOG_FILE}"
rm -rf "${COMPILE_DIRECTORY:?}"/
Expand Down Expand Up @@ -193,6 +211,9 @@ ensure_curl_dependency() {
Darwin)
throw_error "Please install the 'curl' dependency."
;;
FreeBSD)
sudo pkg install -y curl > /dev/null
;;
*)
if command -v apk > /dev/null; then
apk --quiet add curl
Expand Down Expand Up @@ -239,6 +260,9 @@ ensure_sudo_dependency() {
clear-linux-os)
swupd bundle-add sudo > /dev/null
;;
FreeBSD)
pkg install -y sudo > /dev/null
;;
*)
if command -v apk > /dev/null; then
apk --quiet add sudo
Expand Down Expand Up @@ -285,6 +309,9 @@ ensure_xargs_dependency() {
clear-linux-os)
swupd bundle-add findutils > /dev/null
;;
FreeBSD)
sudo pkg install -y findutils > /dev/null
;;
*)
if command -v apk > /dev/null; then
apk --quiet add findutils
Expand All @@ -309,11 +336,14 @@ ensure_xargs_dependency() {
}

get_free_space_in_mib() {
if [ "${TARGET_KERNEL}" = 'Darwin' ]; then
case "${TARGET_KERNEL}" in
'Darwin' | 'FreeBSD')
/bin/df -m "${HOME}" | awk '{print $4}' | sed 1d
else
;;
*)
df --output=avail --block-size='1MiB' "${HOME}" | sed 1d
fi
;;
esac
}

get_log_timestamp() {
Expand All @@ -328,8 +358,8 @@ get_memory_metric_in_mib() {
}

get_operating_system() {
if [ "${TARGET_KERNEL}" = 'Darwin' ]; then
printf '%s\n' 'Darwin'
if [ "${TARGET_KERNEL}" = 'Darwin' ] || [ "${TARGET_KERNEL}" = 'FreeBSD' ]; then
printf '%s\n' "${TARGET_KERNEL}"
else
OS_RELEASE_ID="$(grep '^ID=' /etc/os-release | cut -d= -f2)" && readonly OS_RELEASE_ID
OS_RELEASE_ID_LIKE="$(grep '^ID_LIKE=' /etc/os-release | cut -d= -f2)" && readonly OS_RELEASE_ID_LIKE
Expand Down Expand Up @@ -379,6 +409,9 @@ install_build_dependencies() {
Darwin)
install_build_dependencies_darwin
;;
FreeBSD)
install_build_dependencies_freebsd
;;
*)
install_build_command_function=''
for package_manager in \
Expand Down Expand Up @@ -481,6 +514,15 @@ install_build_dependencies_emerge() {
fi
}

install_build_dependencies_freebsd() {
readonly BUILD_DEPENDENCIES_URL="${FREEBSD_DEPENDENCIES_BASE_URL}/build_dependencies_freebsd.txt"
command -v torsocks > /dev/null 2>&1 &&
dependencies=$(torsocks curl --fail --silent --show-error --location --retry 2 "${BUILD_DEPENDENCIES_URL}") ||
dependencies=$(curl --fail --silent --show-error --location --retry 5 "${BUILD_DEPENDENCIES_URL}")
[ -z "${dependencies:-}" ] && throw_error "The list of dependencies is empty."
printf '%s\n' "${dependencies}" | xargs sudo pkg install --yes > /dev/null
}

install_build_dependencies_pacman() {
readonly BUILD_DEPENDENCIES_URL="${DEPENDENCIES_BASE_URL}/build_dependencies_pacman.txt"
command -v torsocks > /dev/null 2>&1 &&
Expand Down Expand Up @@ -537,6 +579,9 @@ install_runtime_dependencies() {
Darwin)
install_runtime_dependencies_darwin
;;
FreeBSD)
install_runtime_dependencies_freebsd
;;
*)
install_runtime_command_function=''
for package_manager in \
Expand Down Expand Up @@ -633,6 +678,15 @@ install_runtime_dependencies_emerge() {
printf '%s\n' "${dependencies}" | xargs emerge --autounmask-write --jobs "${SYS_CORES_COUNT}" --load-average "${SYS_CORES_PLUS_ONE}" --quiet --quiet-build --quiet-fail
}

install_runtime_dependencies_freebsd() {
readonly RUNTIME_DEPENDENCIES_URL="${FREEBSD_DEPENDENCIES_BASE_URL}/runtime_dependencies_freebsd.txt"
command -v torsocks > /dev/null 2>&1 &&
dependencies=$(torsocks curl --fail --silent --show-error --location --retry 2 "${RUNTIME_DEPENDENCIES_URL}") ||
dependencies=$(curl --fail --silent --show-error --location --retry 5 "${RUNTIME_DEPENDENCIES_URL}")
[ -z "${dependencies}" ] && throw_error 'The list of dependencies is empty.'
printf '%s\n' "${dependencies}" | xargs sudo pkg install --yes > /dev/null
}

install_runtime_dependencies_pacman() {
readonly RUNTIME_DEPENDENCIES_URL="${DEPENDENCIES_BASE_URL}/runtime_dependencies_pacman.txt"
readonly STDERR_DEPENDENCIES_LOG_FILE="${TEMP_DIRECTORY}/stderr_runtime_dependencies_log"
Expand Down Expand Up @@ -687,6 +741,9 @@ install_system_updates() {
clear-linux-os)
install_system_updates_swupd
;;
FreeBSD)
install_system_updates_freebsd
;;
*)
if command -v apk > /dev/null; then
install_system_updates_apk
Expand Down Expand Up @@ -765,6 +822,12 @@ install_system_updates_emerge() {

emerge --depclean --quiet || throw_error "Failed to clean up obsolete packages"
}

install_system_updates_freebsd() {
sudo pkg update > /dev/null
sudo pkg upgrade --yes > /dev/null
}

install_system_updates_pacman() {
sudo pacman -Syu --noconfirm --quiet > /dev/null
}
Expand Down Expand Up @@ -1007,9 +1070,10 @@ readonly BITCOIN_CORE_REPO='https://github.com/bitcoin/bitcoin'
readonly NODEBUILDER_REPO='https://github.com/bitcoin-tools/nodebuilder'
readonly NODEBUILDER_DEPENDENCIES_TAG='v1.7.1-beta'
readonly DEPENDENCIES_BASE_URL="${NODEBUILDER_REPO}/raw/${NODEBUILDER_DEPENDENCIES_TAG}/resources/dependencies"
readonly FREEBSD_DEPENDENCIES_BASE_URL="${NODEBUILDER_REPO}/raw/master/resources/dependencies"

case "${TARGET_KERNEL}" in
Linux)
Linux | FreeBSD)
SYS_CORES_COUNT="$(nproc)"
;;
Darwin)
Expand All @@ -1018,9 +1082,6 @@ Darwin)
MINGW* | Cygwin)
throw_error 'Windows is not supported. Use Linux, macOS, or Windows Subsystem for Linux instead.'
;;
FreeBSD)
throw_error "FreeBSD is not supported. Please open an issue at ${NODEBUILDER_REPO}."
;;
*)
throw_error "${TARGET_KERNEL} is not supported. Please run on Linux or macOS."
;;
Expand All @@ -1042,20 +1103,20 @@ fi

clear_the_terminal
case "${TARGET_KERNEL}" in
Linux)
Linux | FreeBSD)
TARGET_BITCOIN_TARBALL_OS='linux-gnu'
BITCOIN_DATA_DIRECTORY="${HOME}/.bitcoin"
log_info 'Detected: running Linux.'
log_info "Detected: running ${TARGET_KERNEL}."
is_running_in_container && log_info 'Detected: running in Docker.'
;;
Darwin)
TARGET_BITCOIN_TARBALL_OS='apple-darwin'
BITCOIN_DATA_DIRECTORY="${HOME}/Library/Application Support/Bitcoin"
log_info "Detected: running macOS."
log_info 'Detected: running macOS.'
[ "${TARGET_ARCHITECTURE}" = 'arm64' ] && display_macos_warning
;;
*)
throw_error 'Unknown target kernel type.'
throw_error "Unknown target kernel type '${TARGET_KERNEL}'."
;;
esac

Expand Down Expand Up @@ -1114,7 +1175,8 @@ if [ "${CURRENT_BITCOIN_VERSION_PADDED}" = "${TARGET_BITCOIN_VERSION_PADDED}" ];
log_info "Found Bitcoin Core ${target_bitcoin_version}."
elif [ "${compile_bitcoin_flag:-false}" = 'true' ] ||
[ "${TARGET_OPERATING_SYSTEM}" = 'alpine' ] ||
[ "${TARGET_OPERATING_SYSTEM}" = 'gentoo' ]; then
[ "${TARGET_OPERATING_SYSTEM}" = 'gentoo' ] ||
[ "${TARGET_OPERATING_SYSTEM}" = 'FreeBSD' ]; then
log_info "Failed to find Bitcoin Core ${target_bitcoin_version}."
compile_bitcoin_from_source
current_bitcoin_version="${target_bitcoin_version}"
Expand Down Expand Up @@ -1372,9 +1434,9 @@ elif ! grep -q -i '^prune=' "${BITCOIN_CORE_CONFIG_FILE}"; then
fi
fi

# PRUNING
case "${TARGET_KERNEL}" in
Darwin) ;; #TODO: Add macOS memory check
Darwin) ;;
FreeBSD) ;;
*)
PHYSICAL_MEMORY_TOTAL_IN_MIB="$(get_memory_metric_in_mib 'MemTotal')" && readonly PHYSICAL_MEMORY_TOTAL_IN_MIB
PHYSICAL_MEMORY_FREE_IN_MIB="$(get_memory_metric_in_mib 'MemAvailable')" && readonly PHYSICAL_MEMORY_FREE_IN_MIB
Expand Down Expand Up @@ -1431,10 +1493,11 @@ ibd_status=$(echo "${blockchain_info}" | jq '.initialblockdownload')

if [ "${ibd_status}" = 'true' ]; then
if ! is_running_in_container; then
log_info 'Disabling system sleep, suspend, and hibernate.'
if [ "${TARGET_KERNEL}" = 'Darwin' ]; then
log_info 'Disabling system sleep, suspend, and hibernate.'
caffeinate -sw "$(cat "${BITCOIN_DATA_DIRECTORY}/bitcoind.pid")" &
else
elif [ "${TARGET_KERNEL}" = 'Linux' ]; then
log_info 'Disabling system sleep, suspend, and hibernate.'
sudo systemctl mask sleep.target suspend.target hibernate.target hybrid-sleep.target > /dev/null 2>&1
fi
log_info "Close this Terminal window by clicking on the 'X'."
Expand Down Expand Up @@ -1488,7 +1551,7 @@ while [ "${ibd_status}" = 'true' ]; do
printf 'Sync progress: %.3f %%\n' "${sync_progress_percent}"
printf 'Blocks remaining: %d\n' "$((headers - blocks))"

if [ "${TARGET_KERNEL}" = 'Darwin' ]; then
if [ "${TARGET_KERNEL}" = 'Darwin' ] || [ "${TARGET_KERNEL}" = 'FreeBSD' ]; then
current_chain_tip_timestamp="$(/bin/date -r "${last_block_time}" | cut -c 5-)"
else
current_chain_tip_timestamp="$(date -d @"${last_block_time}" | cut -c 5-)"
Expand Down
16 changes: 16 additions & 0 deletions resources/dependencies/build_dependencies_freebsd.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
autoconf
automake
boost-libs
gettext
gmake
libevent
libqrencode
libtool
libzmq4
pkgconf
python3
sqlite3
qt5-buildtools
qt5-gui
qt5-linguisttools
qt5-widgets
6 changes: 6 additions & 0 deletions resources/dependencies/runtime_dependencies_freebsd.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,6 @@
curl
git
gnupg
gzip
jq
sudo
Loading
Loading