Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Use CLI archiving in more places for more reliable archiving (by not using the 'which' tool) #17576

Merged
merged 10 commits into from Jun 1, 2021

Conversation

diosmosis
Copy link
Member

Description:

Some shells do not define which. For those systems, cli archiving will be unavailable and the system check will reflect this, even though everything needed to use cli archiving might be there. Fixed in this PR by just calling ps and awk and checking for a success code (which we already did for ps anyway).

Review

  • Functional review done
  • Potential edge cases thought about (behavior of the code with strange input, with strange internal state or possible interactions with other Matomo subsystems)
  • Usability review done (is anything maybe unclear or think about anything that would cause people to reach out to support)
  • Security review done see checklist
  • Code review done
  • Tests were added if useful/possible
  • Reviewed for breaking changes
  • Developer changelog updated if needed
  • Documentation added if needed
  • Existing documentation updated if needed

@diosmosis diosmosis added the Needs Review PRs that need a code review label May 17, 2021
@diosmosis diosmosis added this to the 4.4.0 milestone May 17, 2021
@@ -210,7 +210,7 @@ public static function isSupported()
return false;
}

if (!self::commandExists('ps') || !self::returnsSuccessCode('ps') || !self::commandExists('awk')) {
if (!self::returnsSuccessCode('ps --version') || !self::returnsSuccessCode('awk --version')) {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fyi haven't tested it but locally on my Mac I get this error:

 ps --version
ps: illegal option -- -
usage: ps [-AaCcEefhjlMmrSTvwXx] [-O fmt | -o fmt] [-G gid[,gid...]]
          [-g grp[,grp...]] [-u [uid,uid...]]
          [-p pid[,pid...]] [-t tty[,tty...]] [-U user[,user...]]
       ps [-L]

so this doesn't seem to work on mac.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I guess we can just use ps/awk then.

@@ -210,7 +210,7 @@ public static function isSupported()
return false;
}

if (!self::commandExists('ps') || !self::returnsSuccessCode('ps') || !self::commandExists('awk')) {
if (!self::returnsSuccessCode('ps') || !self::returnsSuccessCode('awk')) {
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

it works for the ps, but does not work for awk on my mac.
probably because the response for awk is:

➜  ~/dev/sites/matomo git:(l3-81-no-which) ✗ awk
usage: awk [-F fs] [-v var=value] [-f progfile | 'prog'] [file ...]

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@flamisz what is the return code for awk on mac?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@diosmosis I'm on Debian (so GNU awk) and for me awk also doesn't return 0 (I guess because it doesn't run without an arg)

awk ; echo $?
Usage: awk [POSIX or GNU style options] -f progfile [--] file ...
Usage: awk [POSIX or GNU style options] [--] 'program' file ...
POSIX options:          GNU long options: (standard)
        -f progfile             --file=progfile
        -F fs                   --field-separator=fs
        -v var=val              --assign=var=val
Short options:          GNU long options: (extensions)
        -b                      --characters-as-bytes
        -c                      --traditional
        -C                      --copyright
        -d[file]                --dump-variables[=file]
        -D[file]                --debug[=file]
        -e 'program-text'       --source='program-text'
        -E file                 --exec=file
        -g                      --gen-pot
        -h                      --help
        -i includefile          --include=includefile
        -l library              --load=library
        -L[fatal|invalid|no-ext]        --lint[=fatal|invalid|no-ext]
        -M                      --bignum
        -N                      --use-lc-numeric
        -n                      --non-decimal-data
        -o[file]                --pretty-print[=file]
        -O                      --optimize
        -p[file]                --profile[=file]
        -P                      --posix
        -r                      --re-interval
        -s                      --no-optimize
        -S                      --sandbox
        -t                      --lint-old
        -V                      --version

To report bugs, see node `Bugs' in `gawk.info'
which is section `Reporting Problems and Bugs' in the
printed version.  This same information may be found at
https://www.gnu.org/software/gawk/manual/html_node/Bugs.html.
PLEASE do NOT try to report bugs by posting in comp.lang.awk,
or by using a web forum such as Stack Overflow.

gawk is a pattern scanning and processing language.
By default it reads standard input and writes standard output.

Examples:
        awk '{ sum += $1 }; END { print sum }' file
        awk -F: '{ print $1 }' /etc/passwd
1

So there also doesn't seem to be a POSIX command we could use.
So either we try something like awk 'BEGIN {print "ping"}' or go back to command -v

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

what about using something like awk -Wversion 2>/dev/null || awk --version ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@sgiehl's suggestion or a variant of it sounds like the best idea to me. I'll run some tests w/ my macbook

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@Findus23 👍 I was specifically thinking of having multiple commands w/ ||.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

On Mac it awk returns status code 1. @diosmosis is there an issue for this? It may be acceptable that it won't work for some people where they don't have which and we may need to change nothing since it at least mostly works for most users and may be better to not regress anything?

Copy link
Member Author

@diosmosis diosmosis May 18, 2021

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsteur the issue is L3-81

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍 commented there

Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The command awk --version or awk -V works for GNU Awk, but mawk doesn't support --version or -V
@sgiehl's suggestion to use awk -Wversion 2>/dev/null || awk --version works for both gawk and mawk for me.

@diosmosis
Copy link
Member Author

@sgiehl @Findus23 @tsteur updated the test commands

@mattab mattab modified the milestones: 4.4.0, 4.3.0 May 26, 2021
@@ -210,7 +210,9 @@ public static function isSupported()
return false;
}

if (!self::commandExists('ps') || !self::returnsSuccessCode('ps') || !self::commandExists('awk')) {
$awkTestCommand = 'awk -Wversion 2>/dev/null || awk --version || awk \'BEGIN {print "ping"}\' || awk';
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The returnsSuccessCode() actually appends a > /dev/null 2>&1; echo $? to the provided command. shouldn't the commands we provide here be wrapped in brackets so the appended > /dev/null 2>&1 does not only apply to the last command?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw @diosmosis should be fine detecting AWK like this. Be good to check out comment from @sgiehl

Had just another thought before why not put the command we use awk '! /defunct/ {print $1} in a constant and check if we get expected output?

like run echo " 537 s000 Ss 0:00.05 login -pfl theuser /bin/bash -c exec -la bash /bin/bash" | awk '! /defunct/ {print $1}' we expect to get back 537? This way we would know if the command is there and behaves the way we expect maybe? Not 100% sure if it's a good idea but this way might be best to detect if archiving will work. Of course no longer such a good idea if we were to use multiple / different awk commands.

Same could be maybe done for ps instead of checking if command exists we use ps x 2 and we check whether we get a 0 exit code since this is also what the class actually executes. I remember in past there were sometimes issues with the arguments where PS worked but not specific arguments and this way maybe would detect if we get what we expect or so.

Or maybe even easier could we simply execute getListOfRunningProcesses and/or getRunningProcesses and check whether we get a 0 response code (can't check for !empty maybe if no processes are detected). Just a thought to maybe simplify this.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@tsteur @sgiehl updated the check + removed the || use.

core/CliMulti/Process.php Outdated Show resolved Hide resolved
Co-authored-by: Stefan Giehl <stefan@matomo.org>
@Starker3
Copy link
Contributor

Starker3 commented Jun 1, 2021

Not sure if this was ready to be tested yet, but using the most recent version of Process.php in this PR resulted in the Managing Processes via CLI to no longer be supported on two different testing instances where it was previously working. (Ubuntu & Debian)


private static function awkExistsAndRunsCorrectly()
{
$testResult = `echo " 537 s000 Ss 0:00.05 login -pfl theuser /bin/bash -c exec -la bash /bin/bash" | awk '! /defunct/ {print $1}'`;
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

btw for consistency can we use shell_exec here as well?

@tsteur
Copy link
Member

tsteur commented Jun 1, 2021

fyi works nicely here (on a Mac).

@tsteur
Copy link
Member

tsteur commented Jun 1, 2021

@Starker3 maybe execute below 2 commands and let us know what it outputs and the exit code:

ps x 2>/dev/null
echo " 537 s000 Ss 0:00.05 login -pfl theuser /bin/bash -c exec -la bash /bin/bash" | awk '! /defunct/ {print $1}'

@Starker3
Copy link
Contributor

Starker3 commented Jun 1, 2021

ps x 2>/dev/null

This outputs a list of processes with PID TTY STAT TIME COMMAND
and exists with exit code 0 (Both systems)

echo " 537 s000 Ss 0:00.05 login -pfl theuser /bin/bash -c exec -la bash /bin/bash" | awk '! /defunct/ {print $1}'

This outputs 537 with exit code 0 (Both systems)

@diosmosis
Copy link
Member Author

@Starker3 that's what they're supposed to return, but it's still saying it's not supported in the diagnostic?

@Starker3
Copy link
Contributor

Starker3 commented Jun 1, 2021

@diosmosis I used this version of the Process.php file when testing: https://github.com/matomo-org/matomo/blob/92256f4dcc13091c624cbe4b0f30b7b7f25dc212/core/CliMulti/Process.php
And on two different instances it returned Not supported in the diagnostic (Via web and cli)
Both instances using the normal Process.php show that Managing Processes via CLI works.

@diosmosis
Copy link
Member Author

@Starker3 would you be able to try w/ the latest? That ones missing a quote and the end of the command (which should have resulted in a fatal error).

@Starker3
Copy link
Contributor

Starker3 commented Jun 1, 2021

@diosmosis I just tested using this one: https://github.com/matomo-org/matomo/blob/a7d5f34a9f91be84ed958261c32a92d983317a81/core/CliMulti/Process.php
And the results is the same: Setup Cron - Managing processes via CLI: INFORMATIONAL not supported (optional)

@diosmosis
Copy link
Member Author

@Starker3 ok, I can reproduce, looking into it.

@diosmosis
Copy link
Member Author

@Starker3 @tsteur there was a trim() missing. Added it + added some output in the diagnostic explaining why cli archiving isn't supported to make debugging this issue easier for support.

@Starker3
Copy link
Contributor

Starker3 commented Jun 1, 2021

@diosmosis The latest changes work for me. Tested on Debian GNU/Linux 10

@diosmosis
Copy link
Member Author

@tsteur updated again. multiple reasons displayed + constants used.

@tsteur
Copy link
Member

tsteur commented Jun 1, 2021

👍 Looks good @diosmosis worked for me

@diosmosis diosmosis merged commit c3bf4c8 into 4.x-dev Jun 1, 2021
@diosmosis diosmosis deleted the l3-81-no-which branch June 1, 2021 23:03
@mattab mattab changed the title Do not use which when determining if ps and awk are available Use CLI archiving in more places for more reliable archiving (by not using the 'which' tool) Jul 27, 2021
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
Needs Review PRs that need a code review
Projects
None yet
Development

Successfully merging this pull request may close these issues.

None yet

7 participants