curl to post generate gives error: The parameter post_content reads from STDIN

[eduardoarandah]

Bug Report

• Yes, I reviewed the contribution guidelines.
• Yes, more specifically, I reviewed the guidelines on how to write clear bug reports.

Describe the current, buggy behavior

curl -Ns http://loripsum.net/api/5 | wp post generate --post_content --count=10

Gives me:

Error: The parameter post_content reads from STDIN.

Curl works fine:

Describe how other contributors can replicate this bug

• Install wordpress
• Run command

// You can also use code snippets if needed.

Describe what you would expect as the correct outcome

Generate posts

Let us know what environment you are running this on

OS:     Linux 4.19.84-microsoft-standard #1 SMP Wed Nov 13 11:44:37 UTC 2019 x86_64
Shell:  /usr/bin/zsh
PHP binary:     /usr/bin/php7.3
PHP version:    7.3.15-1+ubuntu18.04.1+deb.sury.org+1
php.ini used:   /etc/php/7.3/cli/php.ini
WP-CLI root dir:        phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir:      phar://wp-cli.phar/vendor
WP_CLI phar path:       /home/lalo/www/bedrock
WP-CLI packages dir:    /home/lalo/.wp-cli/packages/
WP-CLI global config:
WP-CLI project config:  /home/lalo/www/bedrock/wp-cli.yml
WP-CLI version: 2.4.0

Provide a possible solution

If you happen to have a suggestion on how to fix this bug, please tell us in here.

Just leave this section out if you don't know how to fix it.

Provide additional context/Screenshots

this works:

echo "hello world" | wp post generate --post_content --count=10

This works:

[Luc45]

Same issue here. Thanks for the proposed solution.

[schlessera]

I cannot replicate the above, the command just works fine for me for both v2.4.0 and for latest dev.

Can you post the output of the following commands for me to get more information about your environment?

• ls -ls $(which wp)
• ls -ls $(which curl)

I suspect there's a script in-between that messes up the IO.

[Luc45]

I cannot replicate the above, the command just works fine for me for both v2.4.0 and for latest dev.
Can you post the output of the following commands for me to get more information about your environment?

Howdy,

www-data@e82a29c5348e:~$ ls -ls $(which wp)
5440 -rwxr-xr-x 1 root root 5568133 Feb 18 08:10 /usr/local/bin/wp
www-data@e82a29c5348e:~$ ls -ls $(which curl)
236 -rwxr-xr-x 1 root root 239848 Nov 30 15:49 /usr/bin/curl
www-data@e82a29c5348e:~$ curl --version
curl 7.68.0 (x86_64-pc-linux-gnu) libcurl/7.68.0 OpenSSL/1.1.1f zlib/1.2.11 brotli/1.0.9 libidn2/2.3.0 libpsl/0.21.0 (+libidn2/2.3.0) libssh/0.9.3/openssl/zlib nghttp2/1.41.0 librtmp/2.3
Release-Date: 2020-01-08
Protocols: dict file ftp ftps gopher http https imap imaps ldap ldaps pop3 pop3s rtmp rtsp scp sftp smb smbs smtp smtps telnet tftp 
Features: AsynchDNS brotli GSS-API HTTP2 HTTPS-proxy IDN IPv6 Kerberos Largefile libz NTLM NTLM_WB PSL SPNEGO SSL TLS-SRP UnixSockets
www-data@e82a29c5348e:~$ wp --version
WP-CLI 2.4.0

This is running inside of Docker

[Luc45]

www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 > foo.txt
www-data@e82a29c5348e:~/multi$ cat foo.txt 
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Estne, quaeso, inquam, sitienti in bibendo voluptas? Sapiens autem semper beatus est et est aliquando in dolore; Hic nihil fuit, quod quaereremus. Habes, inquam, Cato, formam eorum, de quibus loquor, philosophorum. </p>
(...)

www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 | grep Lorem
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Ut alios omittam, hunc appello, quem ille unum secutus est. Conferam tecum, quam cuique verso rem subicias; Sed haec quidem liberius ab eo dicuntur et saepius. Illum mallem levares, quo optimum atque humanissimum virum, Cn. Sit hoc ultimum bonorum, quod nunc a me defenditur; Duo Reges: constructio interrete. Paria sunt igitur. Sed haec quidem liberius ab eo dicuntur et saepius. Itaque in rebus minime obscuris non multus est apud eos disserendi labor. Sed quanta sit alias, nunc tantum possitne esse tanta. Sin tantum modo ad indicia veteris memoriae cognoscenda, curiosorum. </p>

www-data@e82a29c5348e:~/multi$ curl -Ns http://loripsum.net/api/5 | wp post generate --post_content --count=10
Error: The parameter `post_content` reads from STDIN.

[Luc45]

I'm available for a quick anydesk call if that helps

[Luc45]

STDIN seems to be populated in PHP:

www-data@e82a29c5348e:~/single_tests$ echo "foo" | /usr/bin/php -r 'echo file_get_contents("php://stdin");'
foo
www-data@e82a29c5348e:~/single_tests$ echo "foo" | /usr/bin/php -r 'echo stream_get_contents(STDIN);'
foo

This works:

www-data@e82a29c5348e:~/single_tests$ echo "Foo" | wp post generate --post_content --count=10
Generating posts  100% [=====================================================================================] 0:00 / 0:00

So it means it's something with my CURL? Even though this works, too.

www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | cat /dev/stdin
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Negat enim summo bono afferre incrementum diem. Hoc etsi multimodis reprehendi potest, tamen accipio, quod dant. ALIO MODO. Aliter homines, aliter philosophos loqui putas oportere? Cum ageremus, inquit, vitae beatum et eundem supremum diem, scribebamus haec. Atqui haec patefactio quasi rerum opertarum, cum quid quidque sit aperitur, definitio est. Duo Reges: constructio interrete. Ut enim consuetudo loquitur, id solum dicitur honestum, quod est populari fama gloriosum. Utrum igitur tibi litteram videor an totas paginas commovere? Scio enim esse quosdam, qui quavis lingua philosophari possint; </p>

And, this works:

curl -Ns http://loripsum.net/api/5 | php -r 'echo file_get_contents("php://stdin");'
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Haec et tu ita posuisti, et verba vestra sunt. Cur deinde Metrodori liberos commendas? </p>
(...)

Something in this code is bailing the execution before it can try to read from STDIN, only when feeded by CURL:

entity-command/src/WP_CLI/Entity/Utils.php

Line 12 in 209d1c5

public static function has_stdin() {

www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(0)

www-data@e82a29c5348e:~/single_tests$ echo "Foo" | php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)

Aha! The problem is the timeout: stream_select( $read, $write, $except, 0 ); => stream_select( $read, $write, $except, 10 );

www-data@e82a29c5348e:~/single_tests$ curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)

I've added a timeout of 10s, the stream returns as expected with Curl now.

10s is just the timeout, but it seems it returns as soon as ready, if it will be ready at all:

www-data@e82a29c5348e:~/single_tests$ time curl -Ns http://loripsum.net/api/5 | php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)

real    0m0.450s
user    0m0.008s
sys     0m0.013s

[Luc45]

@schlessera Can you please take a look at the debugging process and proposed solution above? ☝️

Essentially, increasing the timeout here:

entity-command/src/WP_CLI/Entity/Utils.php

Line 17 in 209d1c5

$streams = stream_select( $read, $write, $except, 0 );

One downside of the timeout approach is that if there's no STDIN, it will take 10s for the command to fail:

www-data@e82a29c5348e:~/single_tests$ time php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 10 );fclose( $handle ); var_dump($streams);' 
Resource id #5Command line code:1:
int(0)

real    0m10.057s
user    0m0.027s
sys     0m0.021s

I'm not sure if there are better ways to tackle this. Any timeout below 1s is returning an empty stream for me, it seems the timeout is directly related to the time it takes for curl to feed the data, so it's a combination of issues between curl and the no timeout.

So that's why pre-computing the result of curl works, in the example of the original thread:

content=$(curl -Ns http://loripsum.net/api/5)
www-data@e82a29c5348e:~/single_tests$ time echo $content | php -r 'echo $handle  = fopen( "php://stdin", "r" );$read    = array( $handle );$write   = null;$except  = null;$streams = stream_select( $read, $write, $except, 0 );fclose( $handle ); var_dump($streams);'
Resource id #5Command line code:1:
int(1)

real    0m0.018s
user    0m0.007s
sys     0m0.013s

What I find most curious is that the STDIN is there when PHP starts execution, so the timeout should be unnecessary:

www-data@e82a29c5348e:~/single_tests$ time curl -Ns http://loripsum.net/api/50 | php -r 'echo substr(file_get_contents("php://stdin"), 0, 100);'
<p>Lorem ipsum dolor sit amet, consectetur adipiscing elit. Quamquam haec quidem praeposita recte et
real    0m1.056s
user    0m0.004s
sys     0m0.033s

Maybe just read file_get_contents("php://stdin") and if it's empty bail? Instead of using stream_select

[zzap]

I'm getting the same error.

System:

OS:     Linux 5.4.0-126-generic #142-Ubuntu SMP Fri Aug 26 12:12:57 UTC 2022 x86_64
Shell:  /bin/bash
PHP binary:     /usr/bin/php8.1
PHP version:    8.1.10
php.ini used:   /etc/php/8.1/cli/php.ini
MySQL binary:   /usr/bin/mysql
MySQL version:  mysql  Ver 8.0.30-0ubuntu0.20.04.2 for Linux on x86_64 ((Ubuntu))
SQL modes:
WP-CLI root dir:        phar://wp-cli.phar/vendor/wp-cli/wp-cli
WP-CLI vendor dir:      phar://wp-cli.phar/vendor
WP_CLI phar path:       /home/milana
WP-CLI packages dir:    /home/milana/.wp-cli/packages/
WP-CLI global config:   /home/milana/.wp-cli/config.yml
WP-CLI project config:
WP-CLI version: 2.6.0

Command:

curl -N https://baconipsum.com/api/?type=meat-and-filler | wp post generate --post_content --count=36

Error:

  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0Error: The parameter `post_content` reads from STDIN.
100  1536  100  1536    0     0   1545      0 --:--:-- --:--:-- --:--:--  1543
curl: (23) Failed writing body (0 != 1536)

But this one wors:

curl -N http://loripsum.net/api/5 | wp post generate --post_content --count=10

Could be the API issue

[Luc45]

@zzap Please try this:

curl -N https://baconipsum.com/api/?type=meat-and-filler > foo.txt && cat foo.txt | wp post generate --post_content && rm foo.txt

With the approach above, we are essentially pre-caching the remote response in a local file and then feeding it to WP CLI, essentially minimizing delays that could be causing this issue.

I don't know exactly what's happening, but for a definitive solution I'd play around with increasing the stream_select timeout in

entity-command/src/WP_CLI/Entity/Utils.php

Line 17 in 209d1c5

$streams = stream_select( $read, $write, $except, 0 );

from 0 to 1 or more

[GeoJunkie]

I encountered this issue, as well. The workaround @Luc45 suggested worked.

Would a permanent solution be to add an optional --timeout parameter? Looking at the stream_select() docs:

The seconds and microseconds together form the timeout parameter, seconds specifies the number of seconds while microseconds the number of microseconds. The timeout is an upper bound on the amount of time that stream_select() will wait before it returns. If seconds and microseconds are both set to 0, stream_select() will not wait for data - instead it will return immediately, indicating the current status of the streams.

So the 0 in the seconds parameter means it will not work for anything except a file redirect or other instantaneous piping, which is good for the majority of use cases, but breaks things like the example in the command help: curl -N https://loripsum.net/api/5 | wp post generate --post_content --count=10

The --timeout option can default to 0, creating the default behavior, but can be updated for cases where there will be a delay from STDIN via curl or other lengthy command.