Unable to write to file using script in container

[hawkspar]

Expected Behavior

I'm expecting spython.main.Client.execute to behave exactly like singularity exec from the command line when running a dummy python script to write to a file.

Actual Behavior

I can create a file using the command line but not using spython.main.Client.execute from inside a python script.

Steps to Reproduce

Consider running singularity exec --bind ./local_dir:instance_dir any_image_with_python.sif python -c "with open('instance_dir/test.tx','w') as f: f.write('test')" and compare the result with the same command in a python script :

from spython.main import Client

Client.execute('any_instance_with_python.sif',["python","-c","with open('instance_dir/test.tx','w') as f: f.write('test')"],bind=[./local_dir:instance_dir`])

I am confident these are supposed to be identical because I've run the above with the quiet optional argument set to False. Yet one created a file, the other doesn't.

Context

• singularity version: 4.1.4-1
• spython version: 0.3.13
• python version: 3.9.18

Failure Logs

I would love to have those. The python script just continues without a sound...

Possible Fix

My guess is this is a permission issue which requires increasing permissions for the script ?

[vsoch]

It's probably something to do with how the command is executed - perhaps try running that via a script instead?

[hawkspar]

You mean a bash script ? That's a good idea. I'll also try running the python command from the python shell.

…

On Thu, 15 Aug 2024, 02:14 Vanessasaurus, ***@***.***> wrote: It's probably something to do with how the command is executed - perhaps try running that via a script instead? — Reply to this email directly, view it on GitHub <#222 (comment)>, or unsubscribe <https://github.com/notifications/unsubscribe-auth/APA6REP3UZHGFQHIYAIMPO3ZRPXGFAVCNFSM6AAAAABMQFWARGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGEYTEMJXHA> . You are receiving this because you authored the thread.Message ID: ***@***.***>

[hawkspar]

Hello all, As stated before, I originally ran a python script. The problem remains when running a test case on the command line with the -c option. I wrote a one liner bash script looking like `python3 -c "with open(..."`. This script produces a file when running `singularity exec` on it and doesn't, without raising an error, if ran using `Client.execute` from the spython library. I have no clue where this could be coming from. The commands are supposed to be identical... On Thu, 15 Aug 2024, 09:04 Quentin Chevalier, ***@***.***> wrote:

…

You mean a bash script ? That's a good idea. I'll also try running the python command from the python shell. On Thu, 15 Aug 2024, 02:14 Vanessasaurus, ***@***.***> wrote: > It's probably something to do with how the command is executed - perhaps > try running that via a script instead? > > — > Reply to this email directly, view it on GitHub > <#222 (comment)>, > or unsubscribe > <https://github.com/notifications/unsubscribe-auth/APA6REP3UZHGFQHIYAIMPO3ZRPXGFAVCNFSM6AAAAABMQFWARGVHI2DSMVQWIX3LMV43OSLTON2WKQ3PNVWWK3TUHMZDEOJQGEYTEMJXHA> > . > You are receiving this because you authored the thread.Message ID: > ***@***.***> >

[vsoch]

As suggested, you need to write into a script. The library here uses subprocess, which expects a list of commands (using shlex split) and here is what is happening:

# From the command line
 python -c "print('hello')"

hello

From Python

import subprocess
p = subprocess.Popen(['python', '-c', "'print(\"hello\")'"], stdout=subprocess.PIPE, stderr=subprocess.PIPE)
p.communicate()
# note that there is no output
(b'', b'')

The function is here:

singularity-cli/spython/utils/terminal.py

Lines 158 to 167 in 2b22233

	def run_command(
	cmd,
	sudo=False,
	capture=True,
	no_newline_regexp="Progess",
	quiet=False,
	sudo_options=None,
	environ=None,
	background=False,
	):

. If you'd like to suggest a way to use subprocess to get your desired output, I'd be happy to review a PR or put in a quick one myself. I suspect there is some kind of fork or other so it's not picked up here, but I haven't looked into it. The only way I can grep to see output (not capture it) would be to do:

os.system('python -c \"print(\'hello\')\"')
hello

But that is doing os.system and would not be good to put into the library. Thanks.