Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Add hcloud support #261

Merged
merged 1 commit into from
Dec 9, 2022
Merged

Add hcloud support #261

merged 1 commit into from
Dec 9, 2022

Conversation

bastelfreak
Copy link
Member

@bastelfreak bastelfreak commented Dec 7, 2022

This adds support for the Hetzner Cloud platform. They currently support
(2022-12-07) the folling images:

bastelfreak@bastelfreak-nb ~ $ hcloud image list --type system
ID         TYPE     NAME              DESCRIPTION       IMAGE SIZE   DISK SIZE   CREATED                         DEPRECATED
3          system   centos-7          CentOS 7          -            5 GB        Mon Jan 15 12:34:45 CET 2018    -
168855     system   ubuntu-18.04      Ubuntu 18.04      -            5 GB        Wed May  2 13:02:30 CEST 2018   -
5924233    system   debian-10         Debian 10         -            5 GB        Mon Jul  8 08:35:48 CEST 2019   -
15512617   system   ubuntu-20.04      Ubuntu 20.04      -            5 GB        Thu Apr 23 19:55:14 CEST 2020   -
45557056   system   debian-11         Debian 11         -            5 GB        Mon Aug 16 13:12:01 CEST 2021   -
45778012   system   centos-stream-8   CentOS Stream 8   -            5 GB        Thu Aug  5 07:07:23 CEST 2021   -
45780948   system   rocky-8           Rocky Linux 8     -            5 GB        Thu Aug 19 08:30:23 CEST 2021   -
59752342   system   centos-stream-9   CentOS Stream 9   -            5 GB        Thu Jan 27 08:52:03 CET 2022    -
67794396   system   ubuntu-22.04      Ubuntu 22.04      -            5 GB        Thu Apr 21 15:32:38 CEST 2022   -
69726282   system   fedora-36         Fedora 36         -            5 GB        Wed May 11 07:50:00 CEST 2022   -
76766499   system   rocky-9           Rocky Linux 9     -            5 GB        Wed Jul 20 15:55:52 CEST 2022   -
bastelfreak@bastelfreak-nb ~ $

@bastelfreak bastelfreak requested a review from ekohl December 7, 2022 15:28
@bastelfreak bastelfreak marked this pull request as draft December 7, 2022 15:28
@bastelfreak
Copy link
Member Author

BEAKER_PUPPET_COLLECTION=puppet6 BEAKER_setfile=ubuntu1804-64{hostname=ubuntu1804-64.example.com} BEAKER_HYPERVISOR=hcloud bundle exec rake beaker

brings me:

To use retry middleware with Faraday v2.0+, install `faraday-retry` gem
/usr/bin/ruby -I/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib:/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-support-3.12.0/lib /home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/exe/rspec --pattern spec/acceptance/\*\*/\*_spec.rb
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec/helpers/serverspec.rb:43: warning: already initialized constant Module::VALID_OPTIONS_KEYS
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/specinfra-2.83.4/lib/specinfra/configuration.rb:4: warning: previous definition of VALID_OPTIONS_KEYS was here

Hosts file 'ubuntu1804-64{hostname=ubuntu1804-64.example.com}' does not exist.
Trying as beaker-hostgenerator input.

Hypervisor for ubuntu1804-64.example.com is hcloud
Beaker::Hypervisor, found some hcloud boxes to create
Provisioning hcloud
Generating SSH keypair
Creating hcloud SSH key
provisioning ubuntu1804-64.example.com
invalid input in fields 'source_server', 'image'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/hcloud-1.1.0/lib/hcloud/typhoeus_ext.rb:54:in `check_for_error'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/hcloud-1.1.0/lib/hcloud/client.rb:150:in `block in request'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/callbacks.rb:146:in `block in execute_callbacks'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/callbacks.rb:145:in `each'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/callbacks.rb:145:in `execute_callbacks'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/operations.rb:35:in `finish'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/easy_factory.rb:164:in `block in set_callback'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/ethon-0.16.0/lib/ethon/easy/response_callbacks.rb:74:in `block in complete'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/ethon-0.16.0/lib/ethon/easy/response_callbacks.rb:74:in `each'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/ethon-0.16.0/lib/ethon/easy/response_callbacks.rb:74:in `complete'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/ethon-0.16.0/lib/ethon/easy/operations.rb:34:in `perform'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/operations.rb:16:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/cacheable.rb:18:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/block_connection.rb:31:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/stubbable.rb:25:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/typhoeus-1.4.0/lib/typhoeus/request/before.rb:26:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/hcloud-1.1.0/lib/hcloud/client.rb:117:in `prepare_request'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/hcloud-1.1.0/lib/hcloud/abstract_resource.rb:9:in `prepare_request'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/hcloud-1.1.0/lib/hcloud/server_resource.rb:18:in `create'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/bundler/gems/beaker-hcloud-6d0512902e85/lib/beaker/hypervisor/hcloud.rb:76:in `create_server'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/bundler/gems/beaker-hcloud-6d0512902e85/lib/beaker/hypervisor/hcloud.rb:28:in `block in provision'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/bundler/gems/beaker-hcloud-6d0512902e85/lib/beaker/hypervisor/hcloud.rb:27:in `each'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/bundler/gems/beaker-hcloud-6d0512902e85/lib/beaker/hypervisor/hcloud.rb:27:in `provision'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-4.38.1/lib/beaker/hypervisor.rb:37:in `create'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-4.38.1/lib/beaker/network_manager.rb:72:in `block in provision'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-4.38.1/lib/beaker/network_manager.rb:71:in `each_key'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-4.38.1/lib/beaker/network_manager.rb:71:in `provision'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec/beaker_shim.rb:35:in `provision'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec/spec_helper.rb:57:in `block in <top (required)>'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core.rb:98:in `configure'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec/spec_helper.rb:5:in `<top (required)>'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec.rb:5:in `require'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec.rb:5:in `<module:BeakerRSpec>'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/beaker-rspec-7.1.0/lib/beaker-rspec.rb:1:in `<top (required)>'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/voxpupuli-acceptance-1.2.0/lib/voxpupuli/acceptance/spec_helper_acceptance.rb:12:in `require'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/voxpupuli-acceptance-1.2.0/lib/voxpupuli/acceptance/spec_helper_acceptance.rb:12:in `configure_beaker'
/home/bastelfreak/code/puppet-hdm/spec/spec_helper_acceptance.rb:8:in `<top (required)>'
/home/bastelfreak/code/puppet-hdm/spec/acceptance/docker_spec.rb:3:in `require'
/home/bastelfreak/code/puppet-hdm/spec/acceptance/docker_spec.rb:3:in `<top (required)>'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/configuration.rb:2117:in `load'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/configuration.rb:2117:in `load_file_handling_errors'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/configuration.rb:1617:in `block in load_spec_files'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/configuration.rb:1615:in `each'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/configuration.rb:1615:in `load_spec_files'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:102:in `setup'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:86:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:71:in `run'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib/rspec/core/runner.rb:45:in `invoke'
/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/exe/rspec:4:in `<main>'
/usr/bin/ruby -I/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/lib:/home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-support-3.12.0/lib /home/bastelfreak/code/puppet-hdm/.vendor/ruby/3.0.0/gems/rspec-core-3.12.0/exe/rspec --pattern spec/acceptance/\*\*/\*_spec.rb failed

Copy link
Member

@ekohl ekohl left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

We also do some transformations specific to a hypervisor:
https://github.com/voxpupuli/beaker-hostgenerator/blob/master/lib/beaker-hostgenerator/hypervisor/docker.rb for example. It looks to be very consistent NAME-VERSION without an architecture so perhaps that's an easier solution.

@bastelfreak bastelfreak force-pushed the hcloud branch 2 times, most recently from 2dde06e to 2963286 Compare December 7, 2022 15:40
@bastelfreak
Copy link
Member Author

aha!

$ bundle exec beaker-hostgenerator --hypervisor hcloud ubuntu2004-64
---
HOSTS:
  ubuntu2004-64-1:
    platform: ubuntu-20.04-amd64
    image: ubuntu-20.04
    hypervisor: hcloud
    roles:
    - agent
CONFIG:
  nfs_server: none
  consoleport: 443

@codecov
Copy link

codecov bot commented Dec 7, 2022

Codecov Report

Base: 0.51% // Head: 0.50% // Decreases project coverage by -0.00% ⚠️

Coverage data is based on head (8351ba4) compared to base (bc18fc8).
Patch coverage: 0.00% of modified lines in pull request are covered.

❗ Current head 8351ba4 differs from pull request most recent head 4f97db4. Consider uploading reports for the commit 4f97db4 to get more accurate results

Additional details and impacted files
@@            Coverage Diff            @@
##           master    #261      +/-   ##
=========================================
- Coverage    0.51%   0.50%   -0.01%     
=========================================
  Files          14      15       +1     
  Lines        2533    2556      +23     
=========================================
  Hits           13      13              
- Misses       2520    2543      +23     
Impacted Files Coverage Δ
lib/beaker-hostgenerator/hypervisor.rb 0.00% <0.00%> (ø)
lib/beaker-hostgenerator/hypervisor/hcloud.rb 0.00% <0.00%> (ø)

Help us with your feedback. Take ten seconds to tell us how you rate us. Have a feature suggestion? Share it here.

☔ View full report at Codecov.
📢 Do you have feedback about the report comment? Let us know in this issue.

@bastelfreak bastelfreak marked this pull request as ready for review December 7, 2022 15:46
@bastelfreak
Copy link
Member Author

This is now working at: voxpupuli/puppet-hdm#18
a problem I noticed: We generate an FQDN like this: ubuntu1804-64.example.com. We will create two instances with that name, one for Puppet 6 and one for Puppet 7. At Hetzner, a project cannot contain multiple virtual machines with the same FQDN. If we assume we've one project for the whole Vox Pupuli organisation, the name needs to contain puppet+PR info, or a random string? I don't think that the name is actually important for the running tests.

@bastelfreak bastelfreak changed the title Add hcloud images Add hcloud support Dec 7, 2022
@bastelfreak bastelfreak changed the title Add hcloud support Add hcloud support Dec 7, 2022
@ekohl
Copy link
Member

ekohl commented Dec 7, 2022

a problem I noticed: We generate an FQDN like this: ubuntu1804-64.example.com. We will create two instances with that name, one for Puppet 6 and one for Puppet 7. At Hetzner, a project cannot contain multiple virtual machines with the same FQDN. If we assume we've one project for the whole Vox Pupuli organisation, the name needs to contain puppet+PR info, or a random string? I don't think that the name is actually important for the running tests.

FQDN generation is something we now do here: https://github.com/voxpupuli/puppet_metadata/blob/1d92f203598362d9719971cb5b6cab6c085f9689/lib/puppet_metadata/beaker.rb#L48

I'd actually prefer some option to set this and an env var like BEAKER_HOSTGENERATOR_DOMAIN where beaker-hostgenerator generates the FQDN.

Then for the hcloud hypervisor it could actually add some randomness to it. For reference, libvirt has this: https://github.com/voxpupuli/beaker-vagrant/blob/50d671f73862e7a2271b1b361036794489a64672/lib/beaker/hypervisor/vagrant_libvirt.rb#L7-L11

@bastelfreak
Copy link
Member Author

@ekohl I considered modifying the image name in the hcloud hypervisor file. but hcloud only supports a tiny subset of all the operating systems beaker-hostgenerator has. Filtering that might add a bit too much complexity. Or do you think we shouldn't filter that and generate all operating systems for hcloud?

@ekohl
Copy link
Member

ekohl commented Dec 8, 2022

Or do you think we shouldn't filter that and generate all operating systems for hcloud?

I think this. Just like we generate a docker image name on the assumption that it exists. It may exist in the future and otherwise you get an error from the hypervisor. For example, it's reasonable to assume that at some point there will be an image for Ubuntu 24.04 and it's nice not to need any code change for it when it comes out.

@bastelfreak bastelfreak force-pushed the hcloud branch 3 times, most recently from 1b8bb91 to a505106 Compare December 8, 2022 19:18
@bastelfreak
Copy link
Member Author

bastelfreak commented Dec 8, 2022

@ekohl I refactored the image generation. I just need to think about the FQDN now.

edit: so should the hcloud.rb modify the host_name attribute:

$ git grep host_name
lib/beaker-hostgenerator/generator.rb:        host_name = "#{platform}-#{nodeid[ostype]}"
lib/beaker-hostgenerator/generator.rb:        host_name = arbitrary_settings.delete('hostname') if
lib/beaker-hostgenerator/generator.rb:        config['HOSTS'][host_name] = host_config

Or should we implement this in pupet_metedata/beaker-hcloud? beaker-hcloud?

lib/beaker-hostgenerator/hypervisor/hcloud.rb Outdated Show resolved Hide resolved
lib/beaker-hostgenerator/hypervisor/hcloud.rb Outdated Show resolved Hide resolved
lib/beaker-hostgenerator/hypervisor.rb Outdated Show resolved Hide resolved
@bastelfreak bastelfreak force-pushed the hcloud branch 3 times, most recently from 2d7b202 to d81ab4b Compare December 9, 2022 17:13
when 'ubuntu'
"#{os}-#{version[0, 2]}.#{version[2,2]}"
when 'centos'
version.to_i == 7 ? "#{os}-#{version}" : "#{os}-stream-#{version}"
Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@ekohl I reversed the logic and also added a test case for centos7/8 to verify it's working correctly.

This adds support for the Hetzner Cloud platform. They currently support
(2022-12-07) the folling images:

```
bastelfreak@bastelfreak-nb ~ $ hcloud image list --type system
ID         TYPE     NAME              DESCRIPTION       IMAGE SIZE   DISK SIZE   CREATED                         DEPRECATED
3          system   centos-7          CentOS 7          -            5 GB        Mon Jan 15 12:34:45 CET 2018    -
168855     system   ubuntu-18.04      Ubuntu 18.04      -            5 GB        Wed May  2 13:02:30 CEST 2018   -
5924233    system   debian-10         Debian 10         -            5 GB        Mon Jul  8 08:35:48 CEST 2019   -
15512617   system   ubuntu-20.04      Ubuntu 20.04      -            5 GB        Thu Apr 23 19:55:14 CEST 2020   -
45557056   system   debian-11         Debian 11         -            5 GB        Mon Aug 16 13:12:01 CEST 2021   -
45778012   system   centos-stream-8   CentOS Stream 8   -            5 GB        Thu Aug  5 07:07:23 CEST 2021   -
45780948   system   rocky-8           Rocky Linux 8     -            5 GB        Thu Aug 19 08:30:23 CEST 2021   -
59752342   system   centos-stream-9   CentOS Stream 9   -            5 GB        Thu Jan 27 08:52:03 CET 2022    -
67794396   system   ubuntu-22.04      Ubuntu 22.04      -            5 GB        Thu Apr 21 15:32:38 CEST 2022   -
69726282   system   fedora-36         Fedora 36         -            5 GB        Wed May 11 07:50:00 CEST 2022   -
76766499   system   rocky-9           Rocky Linux 9     -            5 GB        Wed Jul 20 15:55:52 CEST 2022   -
bastelfreak@bastelfreak-nb ~ $
```
@bastelfreak bastelfreak merged commit a39f10e into voxpupuli:master Dec 9, 2022
@bastelfreak bastelfreak deleted the hcloud branch December 9, 2022 17:41
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants