Skip to content

Fix ForEach-Object -Parallel when passing in script block variable - #16564

Merged
Ilya (iSazonov) merged 8 commits into
PowerShell:masterfrom
PaulHigin:fix-foreachparallel-using
Dec 14, 2021
Merged

Fix ForEach-Object -Parallel when passing in script block variable#16564
Ilya (iSazonov) merged 8 commits into
PowerShell:masterfrom
PaulHigin:fix-foreachparallel-using

Conversation

@PaulHigin

@PaulHigin Paul Higinbotham (PaulHigin) commented Dec 7, 2021

Copy link
Copy Markdown
Contributor

PR Summary

This change fixes #16445, where using variables were not being evaluated correctly when the -Parallel script block is passed in as a variable.

PR Context

ForeEach-Object -Parallel using variable processing walks the script block AST to determine if the using variable was defined in the current scope, and does this by tracking the number of 'ForEach-Object -Parallel' commands in the call chain.
However, the provided script block is different, depending on whether it is passed in as a variable or via parameter binding.

$myScript = '"Hello!"'
1 | ForEach-Object -Parallel ([scriptblock]::Create($myScript))
# Script block Ast does not contain the 'ForEach-Object' command in the call chain

1 | ForEach-Object -Parallel { "Hello!" }
# Script block Ast does contain the 'ForEach-Object' command in the call chain

This change takes into account the two forms of script blocks

PR Checklist

@ghost ghost assigned Ilya (iSazonov) Dec 7, 2021
@iSazonov Ilya (iSazonov) added the CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log label Dec 7, 2021
Comment thread src/System.Management.Automation/engine/hostifaces/PSTask.cs Outdated
Comment thread src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs Outdated
@iSazonov

Copy link
Copy Markdown
Collaborator

Could you please clarify why do we need to support $using if Foreach-Object execute the script block in current scope.

@PaulHigin

Copy link
Copy Markdown
Contributor Author

Ilya (@iSazonov) The $using keyword is only supported for ForEach-Object and the -Parallel parameter set, in which case the script block is executed in multiple runspaces in parallel. Since runspaces are isolation units, the $using keyword is needed to pass in variables from the current session to each parallel runspace sessions, similar to how it is used for remoting. Otherwise they would need to be passed in via the argument list.

@iSazonov

Copy link
Copy Markdown
Collaborator

Do you mean Foreach-Object execute the scriptblock in current scope and Foreach-Object execute the scriptblock in another scope?
image

If so, we got a very confusing UX. :-(
Ex.: an user has large scriptblock, add -Parallel switch and now the scriptblock doesn't work.

@PaulHigin

Copy link
Copy Markdown
Contributor Author

?? This is all discussed in the RFC created for this feature.

PowerShell/PowerShell-RFC#194

@iSazonov

Copy link
Copy Markdown
Collaborator

?? This is all discussed in the RFC created for this feature.

PowerShell/PowerShell-RFC#194

:-) Yes, and I objected to this $using support. I believe that the feedback now clearly shows the UX and performance issues. We should consider designing a new cmdlet to address these requests.

@PaulHigin

Copy link
Copy Markdown
Contributor Author

Dongbo Wang (@daxian-dbw) Please review.

Comment thread src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs Outdated
@ghost ghost added the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Dec 10, 2021
@daxian-dbw

Copy link
Copy Markdown
Member

Ilya (@iSazonov) ForEach-Object -Parallel runs the given script block in a separate Runspsace, so of course the script block will not run in the current scope.

@daxian-dbw

Copy link
Copy Markdown
Member

Paul Higinbotham (@PaulHigin) Can you please also review #12378 and see if that can be closed after your fix.

@ghost ghost removed the Waiting on Author The PR was reviewed and requires changes or comments from the author before being accept label Dec 10, 2021
@PaulHigin

Copy link
Copy Markdown
Contributor Author

Dongbo Wang (@daxian-dbw) #12378 is a different issue.

Comment thread src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs Outdated
Comment thread src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs Outdated
@iSazonov
Ilya (iSazonov) merged commit cadd576 into PowerShell:master Dec 14, 2021
TrapGodBrim pushed a commit to TrapGodBrim/PowerShell that referenced this pull request Jan 19, 2022
…owerShell#16564)

ForeEach-Object -Parallel using variable processing walks the script block AST to determine if the using variable was defined in the current scope, and does this by tracking the number of 'ForEach-Object -Parallel' commands in the call chain.
However, the provided script block is different, depending on whether it is passed in as a variable or via parameter binding.

$myScript = '"Hello!"'
1 | ForEach-Object -Parallel ([scriptblock]::Create($myScript))
# Script block Ast does not contain the 'ForEach-Object' command in the call chain

1 | ForEach-Object -Parallel { "Hello!" }
# Script block Ast does contain the 'ForEach-Object' command in the call chain

This change takes into account the two forms of script blocks
@ghost

Copy link
Copy Markdown

🎉v7.3.0-preview.2 has been released which incorporates this pull request.:tada:

Handy links:

@TravisEz13

Copy link
Copy Markdown
Member

@PowerShell/powershell-maintainers
We only take security fixes and regression to LTS. This is appropriate for 7.3. Rejecting back-port.

Allow some bake time and reconsider to verify that this is stable.

@SeeminglyScience

Copy link
Copy Markdown
Contributor

Travis Plunk (@TravisEz13) Just to clarify this one was actually a regression (worked in 7.1.5, broke in 7.2, thank you Jordan Borean (@jborean93)). Unsure if that was discussed in the meeting but if not, it may be worth another pass.

@PaulHigin

Copy link
Copy Markdown
Contributor Author

The progressive changes include not only the regression, but also new behavior. So I feel backporting this to LTS branches is risky, and that we should at least wait for it to bake longer in preview versions.

@PaulHigin

Copy link
Copy Markdown
Contributor Author

We should reconsider backporting this as it appears to be affecting users of 7.2.x.

#17571

@adityapatwardhan

Copy link
Copy Markdown
Member

/backport to release/v7.2.6

@github-actions

github-actions Bot commented Aug 2, 2022

Copy link
Copy Markdown
Contributor

Started backporting to release/v7.2.6: https://github.com/PowerShell/PowerShell/actions/runs/2784832290

GitHub
PowerShell for every system! Contribute to PowerShell/PowerShell development by creating an account on GitHub.

@github-actions

github-actions Bot commented Aug 2, 2022

Copy link
Copy Markdown
Contributor

Aditya Patwardhan (@adityapatwardhan) backporting to release/v7.2.6 failed, the patch most likely resulted in conflicts:

$ git am --3way --ignore-whitespace --keep-non-patch changes.patch

Applying: First try at fix
Applying: Fix scope function to correctly account for provided script block type.
Applying: Fix tests
Applying: Update src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs
Applying: PR comments
.git/rebase-apply/patch:26: trailing whitespace.
            
warning: 1 line adds whitespace errors.
Using index info to reconstruct a base tree...
M	src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs
Falling back to patching base and 3-way merge...
Auto-merging src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs
CONFLICT (content): Merge conflict in src/System.Management.Automation/engine/runtime/ScriptBlockToPowerShell.cs
error: Failed to merge in the changes.
hint: Use 'git am --show-current-patch=diff' to see the failed patch
Patch failed at 0005 PR comments
When you have resolved this problem, run "git am --continue".
If you prefer to skip this patch, run "git am --skip" instead.
To restore the original branch and stop patching, run "git am --abort".
Error: The process '/usr/bin/git' failed with exit code 128

Please backport manually!

@iSazonov

Ilya (iSazonov) commented Aug 5, 2022

Copy link
Copy Markdown
Collaborator

Aditya Patwardhan (@adityapatwardhan) Friendly ping!

@adityapatwardhan

Copy link
Copy Markdown
Member

This is in the backlog.. Currently working on fixing the build break in 7.2.6

@adityapatwardhan

Copy link
Copy Markdown
Member

/backport to release/v7.2.6

1 similar comment
@adityapatwardhan

Copy link
Copy Markdown
Member

/backport to release/v7.2.6

Aditya Patwardhan (adityapatwardhan) pushed a commit to adityapatwardhan/PowerShell that referenced this pull request Aug 8, 2022
…owerShell#16564)

ForeEach-Object -Parallel using variable processing walks the script block AST to determine if the using variable was defined in the current scope, and does this by tracking the number of 'ForEach-Object -Parallel' commands in the call chain.
However, the provided script block is different, depending on whether it is passed in as a variable or via parameter binding.

$myScript = '"Hello!"'
1 | ForEach-Object -Parallel ([scriptblock]::Create($myScript))
# Script block Ast does not contain the 'ForEach-Object' command in the call chain

1 | ForEach-Object -Parallel { "Hello!" }
# Script block Ast does contain the 'ForEach-Object' command in the call chain

This change takes into account the two forms of script blocks
@ghost

Copy link
Copy Markdown

🎉v7.2.6 has been released which incorporates this pull request.:tada:

Handy links:

Thatgfsj (Thatgfsj) pushed a commit to Thatgfsj/PowerShell that referenced this pull request Aug 6, 2026
…owerShell#16564)

ForeEach-Object -Parallel using variable processing walks the script block AST to determine if the using variable was defined in the current scope, and does this by tracking the number of 'ForEach-Object -Parallel' commands in the call chain.
However, the provided script block is different, depending on whether it is passed in as a variable or via parameter binding.

$myScript = '"Hello!"'
1 | ForEach-Object -Parallel ([scriptblock]::Create($myScript))
# Script block Ast does not contain the 'ForEach-Object' command in the call chain

1 | ForEach-Object -Parallel { "Hello!" }
# Script block Ast does contain the 'ForEach-Object' command in the call chain

This change takes into account the two forms of script blocks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

Backport-7.2.x-Done CL-General Indicates that a PR should be marked as a general cmdlet change in the Change Log Medium

Projects

None yet

Development

Successfully merging this pull request may close these issues.

ForEach-Object -Parallel does not accept a scriptblock variable containing a $using

6 participants