Installation problems

[Dbg-atlas]

I've been trying to install Keras and Tensorflow from R following the instructions in the book, but I'm encountering an issue that I can't solve even following the discussion here.

First I installed the libraries on Windows using RStudio and everything worked perfectly. However, I only got it to work on CPU, not GPU. To facilitate the installation of dependencies to use the GPU I followed the instructions in the book and used Windows Subsystem for Linux to install Linux on my computer.

Once I did this, using Visual Studio Code with WSL to run the same code, the installation does not work. The end result is as follows:

Error: Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
 /home/dberm/.pyenv/versions/3.9.16/bin/python3.9
 /usr/bin/python3.10

My full code so far:

install.packages("keras")

Installing package into ‘/home/dberm/R/x86_64-pc-linux-gnu-library/4.1’
(as ‘lib’ is unspecified)
trying URL 'https://cloud.r-project.org/src/contrib/keras_2.11.1.tar.gz'
Content type 'application/x-gzip' length 3527604 bytes (3.4 MB)

downloaded 3.4 MB

• installing source package ‘keras’ ...
  ** package ‘keras’ successfully unpacked and MD5 sums checked
  ** using staged installation
  ** R
  ** inst
  ** byte-compile and prepare package for lazy loading
  ** help
  *** installing help indices
  ** building package indices
  ** installing vignettes
  ** testing if installed package can be loaded from temporary location
  ** testing if installed package can be loaded from final location
  ** testing if installed package keeps a record of temporary installation path
• DONE (keras)

The downloaded source packages are in
‘/tmp/RtmpDj8V9u/downloaded_packages’

library(reticulate)
virtualenv_create("r-reticulate", python = install_python())

virtualenv: r-reticulate

library(keras)
install_keras(envname = "r-reticulate")

Using virtual environment 'r-reticulate' ...

• '/home/dberm/.virtualenvs/r-reticulate/bin/python' -m pip install --upgrade --no-user --ignore-installed 'tensorflow==2.11.' 'tensorflow-hub' 'tensorflow-datasets' 'scipy' 'requests' 'Pillow' 'h5py' 'pandas' 'pydot'
  Collecting tensorflow==2.11.
  Using cached tensorflow-2.11.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (588.3 MB)
  Collecting tensorflow-hub
  Using cached tensorflow_hub-0.13.0-py2.py3-none-any.whl (100 kB)
  Collecting tensorflow-datasets
  Using cached tensorflow_datasets-4.8.3-py3-none-any.whl (5.4 MB)
  Collecting scipy
  Using cached scipy-1.10.1-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (34.5 MB)
  Collecting requests
  Using cached requests-2.28.2-py3-none-any.whl (62 kB)
  Collecting Pillow
  Using cached Pillow-9.4.0-cp39-cp39-manylinux_2_28_x86_64.whl (3.4 MB)
  Collecting h5py
  Using cached h5py-3.8.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.7 MB)
  Collecting pandas
  Using cached pandas-1.5.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (12.2 MB)
  Collecting pydot
  Using cached pydot-1.4.2-py2.py3-none-any.whl (21 kB)
  Collecting astunparse>=1.6.0
  Using cached astunparse-1.6.3-py2.py3-none-any.whl (12 kB)
  Collecting opt-einsum>=2.3.2
  Using cached opt_einsum-3.3.0-py3-none-any.whl (65 kB)
  Collecting typing-extensions>=3.6.6
  Using cached typing_extensions-4.5.0-py3-none-any.whl (27 kB)
  Collecting google-pasta>=0.1.1
  Using cached google_pasta-0.2.0-py3-none-any.whl (57 kB)
  Collecting grpcio<2.0,>=1.24.3
  Using cached grpcio-1.51.3-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (4.8 MB)
  Collecting tensorflow-estimator<2.12,>=2.11.0
  Using cached tensorflow_estimator-2.11.0-py2.py3-none-any.whl (439 kB)
  Collecting termcolor>=1.1.0
  Using cached termcolor-2.2.0-py3-none-any.whl (6.6 kB)
  Collecting libclang>=13.0.0
  Using cached libclang-15.0.6.1-py2.py3-none-manylinux2010_x86_64.whl (21.5 MB)
  Collecting packaging
  Using cached packaging-23.0-py3-none-any.whl (42 kB)
  Collecting tensorboard<2.12,>=2.11
  Using cached tensorboard-2.11.2-py3-none-any.whl (6.0 MB)
  Collecting wrapt>=1.11.0
  Using cached wrapt-1.15.0-cp39-cp39-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (78 kB)
  Collecting flatbuffers>=2.0
  Using cached flatbuffers-23.3.3-py2.py3-none-any.whl (26 kB)
  Collecting tensorflow-io-gcs-filesystem>=0.23.1
  Using cached tensorflow_io_gcs_filesystem-0.31.0-cp39-cp39-manylinux_2_12_x86_64.manylinux2010_x86_64.whl (2.4 MB)
  Collecting numpy>=1.20
  Using cached numpy-1.24.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (17.3 MB)
  Collecting protobuf<3.20,>=3.9.2
  Using cached protobuf-3.19.6-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)
  Collecting absl-py>=1.0.0
  Using cached absl_py-1.4.0-py3-none-any.whl (126 kB)
  Collecting keras<2.12,>=2.11.0
  Using cached keras-2.11.0-py2.py3-none-any.whl (1.7 MB)
  Collecting setuptools
  Using cached setuptools-67.6.0-py3-none-any.whl (1.1 MB)
  Collecting gast<=0.4.0,>=0.2.1
  Using cached gast-0.4.0-py3-none-any.whl (9.8 kB)
  Collecting six>=1.12.0
  Using cached six-1.16.0-py2.py3-none-any.whl (11 kB)
  Collecting promise
  Using cached promise-2.3.tar.gz (19 kB)
  Preparing metadata (setup.py) ... done
  Collecting tqdm
  Using cached tqdm-4.65.0-py3-none-any.whl (77 kB)
  Collecting dm-tree
  Using cached dm_tree-0.1.8-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (153 kB)
  Collecting tensorflow-metadata
  Using cached tensorflow_metadata-1.12.0-py3-none-any.whl (52 kB)
  Collecting toml
  Using cached toml-0.10.2-py2.py3-none-any.whl (16 kB)
  Collecting etils[enp,epath]>=0.9.0
  Using cached etils-1.1.1-py3-none-any.whl (115 kB)
  Collecting psutil
  Using cached psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)
  Collecting click
  Using cached click-8.1.3-py3-none-any.whl (96 kB)
  Collecting charset-normalizer<4,>=2
  Using cached charset_normalizer-3.1.0-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (199 kB)
  Collecting idna<4,>=2.5
  Using cached idna-3.4-py3-none-any.whl (61 kB)
  Collecting certifi>=2017.4.17
  Using cached certifi-2022.12.7-py3-none-any.whl (155 kB)
  Collecting urllib3<1.27,>=1.21.1
  Using cached urllib3-1.26.15-py2.py3-none-any.whl (140 kB)
  Collecting pytz>=2020.1
  Using cached pytz-2022.7.1-py2.py3-none-any.whl (499 kB)
  Collecting python-dateutil>=2.8.1
  Using cached python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)
  Collecting pyparsing>=2.1.4
  Using cached pyparsing-3.0.9-py3-none-any.whl (98 kB)
  Collecting wheel<1.0,>=0.23.0
  Using cached wheel-0.40.0-py3-none-any.whl (64 kB)
  Collecting zipp
  Using cached zipp-3.15.0-py3-none-any.whl (6.8 kB)
  Collecting importlib_resources
  Using cached importlib_resources-5.12.0-py3-none-any.whl (36 kB)
  Collecting tensorboard-data-server<0.7.0,>=0.6.0
  Using cached tensorboard_data_server-0.6.1-py3-none-manylinux2010_x86_64.whl (4.9 MB)
  Collecting markdown>=2.6.8
  Using cached Markdown-3.4.2-py3-none-any.whl (93 kB)
  Collecting tensorboard-plugin-wit>=1.6.0
  Using cached tensorboard_plugin_wit-1.8.1-py3-none-any.whl (781 kB)
  Collecting google-auth<3,>=1.6.3
  Using cached google_auth-2.16.2-py2.py3-none-any.whl (177 kB)
  Collecting google-auth-oauthlib<0.5,>=0.4.1
  Using cached google_auth_oauthlib-0.4.6-py2.py3-none-any.whl (18 kB)
  Collecting werkzeug>=1.0.1
  Using cached Werkzeug-2.2.3-py3-none-any.whl (233 kB)
  Collecting googleapis-common-protos<2,>=1.52.0
  Using cached googleapis_common_protos-1.59.0-py2.py3-none-any.whl (223 kB)
  Collecting rsa<5,>=3.1.4
  Using cached rsa-4.9-py3-none-any.whl (34 kB)
  Collecting pyasn1-modules>=0.2.1
  Using cached pyasn1_modules-0.2.8-py2.py3-none-any.whl (155 kB)
  Collecting cachetools<6.0,>=2.0.0
  Using cached cachetools-5.3.0-py3-none-any.whl (9.3 kB)
  Collecting requests-oauthlib>=0.7.0
  Using cached requests_oauthlib-1.3.1-py2.py3-none-any.whl (23 kB)
  Collecting importlib-metadata>=4.4
  Using cached importlib_metadata-6.1.0-py3-none-any.whl (21 kB)
  Collecting MarkupSafe>=2.1.1
  Using cached MarkupSafe-2.1.2-cp39-cp39-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)
  Collecting pyasn1<0.5.0,>=0.4.6
  Using cached pyasn1-0.4.8-py2.py3-none-any.whl (77 kB)
  Collecting oauthlib>=3.0.0
  Using cached oauthlib-3.2.2-py3-none-any.whl (151 kB)
  Building wheels for collected packages: promise
  Building wheel for promise (setup.py) ... error
  error: subprocess-exited-with-error

  × python setup.py bdist_wheel did not run successfully.
  │ exit code: 1
  ╰─> [34 lines of output]
  Traceback (most recent call last):
  File "", line 2, in
  File "", line 34, in
  File "/tmp/pip-install-c9bisgzm/promise_af6168e0cb2e42adaeca9a8025bee6aa/setup.py", line 28, in
  setup(
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/init.py", line 108, in setup
  return distutils.core.setup(**attrs)
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/core.py", line 172, in setup
  ok = dist.parse_command_line()
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 475, in parse_command_line
  args = self._parse_command_opts(parser, args)
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 1119, in _parse_command_opts
  nargs = _Distribution._parse_command_opts(self, parser, args)
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_distutils/dist.py", line 534, in _parse_command_opts
  cmd_class = self.get_command_class(command)
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/dist.py", line 966, in get_command_class
  self.cmdclass[command] = cmdclass = ep.load()
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/setuptools/_vendor/importlib_metadata/init.py", line 208, in load
  module = import_module(match.group('module'))
  File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/importlib/init.py", line 127, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
  File "", line 1030, in _gcd_import
  File "", line 1007, in _find_and_load
  File "", line 986, in _find_and_load_unlocked
  File "", line 680, in _load_unlocked
  File "", line 850, in exec_module
  File "", line 228, in _call_with_frames_removed
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/bdist_wheel.py", line 28, in
  from .macosx_libfile import calculate_macosx_platform_tag
  File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/wheel/macosx_libfile.py", line 43, in
  import ctypes
  File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
  from _ctypes import Union, Structure, Array
  ModuleNotFoundError: No module named '_ctypes'
  [end of output]

  note: This error originates from a subprocess, and is likely not a problem with pip.
  ERROR: Failed building wheel for promise
  Running setup.py clean for promise
  Failed to build promise
  Installing collected packages: tensorboard-plugin-wit, pytz, pyasn1, libclang, flatbuffers, dm-tree, zipp, wrapt, wheel, urllib3, typing-extensions, tqdm, toml, termcolor, tensorflow-io-gcs-filesystem, tensorflow-estimator, tensorboard-data-server, six, setuptools, rsa, pyparsing, pyasn1-modules, psutil, protobuf, Pillow, packaging, oauthlib, numpy, MarkupSafe, keras, idna, grpcio, gast, etils, click, charset-normalizer, certifi, cachetools, absl-py, werkzeug, tensorflow-hub, scipy, requests, python-dateutil, pydot, promise, opt-einsum, importlib_resources, importlib-metadata, h5py, googleapis-common-protos, google-pasta, google-auth, astunparse, tensorflow-metadata, requests-oauthlib, pandas, markdown, tensorflow-datasets, google-auth-oauthlib, tensorboard, tensorflow
  Running setup.py install for promise ... done
  DEPRECATION: promise was installed using the legacy 'setup.py install' method, because a wheel could not be built for it. pip 23.1 will enforce this behaviour change. A possible replacement is to fix the wheel build issue reported above. Discussion can be found at Deprecate call to setup.py install when building a wheel failed for source distributions without pyproject.toml pypa/pip#8368
  Successfully installed MarkupSafe-2.1.2 Pillow-9.4.0 absl-py-1.4.0 astunparse-1.6.3 cachetools-5.3.0 certifi-2022.12.7 charset-normalizer-3.1.0 click-8.1.3 dm-tree-0.1.8 etils-1.1.1 flatbuffers-23.3.3 gast-0.4.0 google-auth-2.16.2 google-auth-oauthlib-0.4.6 google-pasta-0.2.0 googleapis-common-protos-1.59.0 grpcio-1.51.3 h5py-3.8.0 idna-3.4 importlib-metadata-6.1.0 importlib_resources-5.12.0 keras-2.11.0 libclang-15.0.6.1 markdown-3.4.2 numpy-1.24.2 oauthlib-3.2.2 opt-einsum-3.3.0 packaging-23.0 pandas-1.5.3 promise-2.3 protobuf-3.19.6 psutil-5.9.4 pyasn1-0.4.8 pyasn1-modules-0.2.8 pydot-1.4.2 pyparsing-3.0.9 python-dateutil-2.8.2 pytz-2022.7.1 requests-2.28.2 requests-oauthlib-1.3.1 rsa-4.9 scipy-1.10.1 setuptools-67.6.0 six-1.16.0 tensorboard-2.11.2 tensorboard-data-server-0.6.1 tensorboard-plugin-wit-1.8.1 tensorflow-2.11.1 tensorflow-datasets-4.8.3 tensorflow-estimator-2.11.0 tensorflow-hub-0.13.0 tensorflow-io-gcs-filesystem-0.31.0 tensorflow-metadata-1.12.0 termcolor-2.2.0 toml-0.10.2 tqdm-4.65.0 typing-extensions-4.5.0 urllib3-1.26.15 werkzeug-2.2.3 wheel-0.40.0 wrapt-1.15.0 zipp-3.15.0

Installation complete.

tensorflow::tf_config()

Valid installation of TensorFlow not found.

Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10

Python exception encountered:
Traceback (most recent call last):
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/init.py", line 37, in
from tensorflow.python.tools import module_util as _module_util
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.virtualenvs/r-reticulate/lib/python3.9/site-packages/tensorflow/python/init.py", line 24, in
import ctypes
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
File "/home/dberm/.pyenv/versions/3.9.16/lib/python3.9/ctypes/init.py", line 8, in
from _ctypes import Union, Structure, Array
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 119, in _find_and_load_hook
return _run_hook(name, _hook)
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 93, in _run_hook
module = hook()
File "/home/dberm/R/x86_64-pc-linux-gnu-library/4.1/reticulate/python/rpytools/loader.py", line 117, in _hook
return find_and_load(name, import)
ModuleNotFoundError: No module named '_ctypes'

You can install TensorFlow using the install_tensorflow() function.

If I try to use tensorflow::install_tensorflow() before tensorflow::tf_config() the result is the same. I also tried the instructions from the RStudio website, with the same results:

install.packages("tensorflow")

library(reticulate)
path_to_python <- install_python()
virtualenv_create("r-reticulate", python = path_to_python)

library(tensorflow)
install_tensorflow(envname = "r-reticulate")

install.packages("keras")
library(keras)
install_keras(envname = "r-reticulate")

tensorflow::tf_config()

As far as I can tell by the output, tensorflow::tf_config() is looking for Tensorflow here:

Python environments searched for 'tensorflow' package:
/home/dberm/.pyenv/versions/3.9.16/bin/python3.9
/usr/bin/python3.10

while Tensorflor should be in envname = "r-reticulate", but I'm not used to work with Python enviroments so I don't know to change this behaviour.

I also read here this comment:

Additionally, you must be running an Arm native build of R, not the x86 build running under Rosetta.

Which I don't know if applies to my case or is just for Intel Mac.

I don't know what else to try as I have been several days stuck here and I lack the required Python knowledge to make any progress. Any help would be appreciated.